Re: File corruption on LVM2 on top of software RAID1

Re: File corruption on LVM2 on top of software RAID1

[Chris Adams]

Once upon a time, "Simon Matter" <simon.matter@invoca.ch> said:
>In my tests I get corrupt files on LVM2 which is on top of software raid1.
>(This is a common setup even mentioned in the software RAID HOWTO and has
>worked for me on RedHat 9 / kernel 2.4 for a long time now and it's my
>favourite configuration). Now, I tested two different distributions, three
>kernels, three different filesystems and three different hardware. I can
>always reproduce it with the following easy scripts:

See:

http://bugzilla.kernel.org/show_bug.cgi?id=4946
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=152162

There's a one-line patch in there; see if that fixes the problem for
you.

-- 
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: File corruption on LVM2 on top of software RAID1

[Simon Matter]

> Once upon a time, "Simon Matter" <simon.matter@invoca.ch> said:
>>In my tests I get corrupt files on LVM2 which is on top of software
>> raid1.
>>(This is a common setup even mentioned in the software RAID HOWTO and has
>>worked for me on RedHat 9 / kernel 2.4 for a long time now and it's my
>>favourite configuration). Now, I tested two different distributions,
>> three
>>kernels, three different filesystems and three different hardware. I can
>>always reproduce it with the following easy scripts:
>
> See:
>
> http://bugzilla.kernel.org/show_bug.cgi?id=4946
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=152162
>
> There's a one-line patch in there; see if that fixes the problem for
> you.

Hi Chris,

Thank you for your help, indeed this oneliner fixes the corruption. I have
tested with 2.6.12.3 + bio_clone fix and also built an updated RedHat EL4
kernel with the fix included. No corruption anymore.
This fix should be applied to most kernels shipped with the latest Linux
distributions. At least those products called Enterprise something should
hurry in the interest of their customers. Do you think this will happen
anytime soon?

-- 
Simon Matter
Invoca Systems

Re: File corruption on LVM2 on top of software RAID1

[Andrew Morton]

"Simon Matter" <simon.matter@invoca.ch> wrote:
>
> While looking at some data corruption vulnerability reports on
>  Securityfocus I wonder why this issue does not get any attention from
>  distributors. I have an open bugzilla report with RedHat, have an open
>  customer service request with RedHat, have mailed peoples directly. No
>  real feedback.
>  I'm now in the process of restoring intergrity of my data with the help of
>  backups and mirrored data. Maybe I just care too much about other peoples
>  data, but I know that this bug will corrupt files on hundreds or thousands
>  of servers today and most people simply don't know it. Did I miss
>  something?

I guess the bug hit really rarely and maybe the reports were lost in the
permanent background noise of dodgy hardware.

We only found and fixed it last week, due to much sleuthing by Matthew
Stapleton.  I assume vendor updates are in the pipeline.

Re: File corruption on LVM2 on top of software RAID1

[Simon Matter]

> "Simon Matter" <simon.matter@invoca.ch> wrote:
>>
>> Hi,
>>
>> Please CC me as I'm not subscribed to the kernel list.
>>
>> I had a hard time identifying a serious problem in the 2.6 linux kernel.
>
> Yes, it is a serious problem.
>
>> It all started while evaluating RHEL4 for new servers. My data integrity
>> tests gave me bad results - which I couldn't believe - and my first idea
>> was - of course - bad hardware. I ordered new SCSI disks instead of the
>> IDE disks, took another server, spent some money again, tried again and
>> again. That's all long ago now...
>>
>> In my tests I get corrupt files on LVM2 which is on top of software
>> raid1.
>> (This is a common setup even mentioned in the software RAID HOWTO and
>> has
>> worked for me on RedHat 9 / kernel 2.4 for a long time now and it's my
>> favourite configuration). Now, I tested two different distributions,
>> three
>> kernels, three different filesystems and three different hardware. I can
>> always reproduce it with the following easy scripts:
>>
>
> Thanks for doing that.
>
> There's one fix against 2.6.12.3 which is needed, but 2.6.9 didn't have
> the
> bug which this fix addresses.  So unless 2.6.9 has a different problem,
> this won't help.

Sorry for the confusion, the 2.6.9 in question is a "RedHat Enterprise
Kernel" which includes lots of backported patches and features. So it is
no plain 2.6.9.

The real problem is that most current distributions include this bug in
one or the other way. They either have one of the affected versions in the
kernel package or have an older kernel with the bug backported. What I
know for sure is that the following distributions are affected:
RedHat Enterprise 4
Fedora Core 4
SuSE 9.3
Debian with the latest "unstable" 2.6 kernel
I couldn't verify Mandrake, Gentoo and others, but I expect them to be at
risk too.

While looking at some data corruption vulnerability reports on
Securityfocus I wonder why this issue does not get any attention from
distributors. I have an open bugzilla report with RedHat, have an open
customer service request with RedHat, have mailed peoples directly. No
real feedback.
I'm now in the process of restoring intergrity of my data with the help of
backups and mirrored data. Maybe I just care too much about other peoples
data, but I know that this bug will corrupt files on hundreds or thousands
of servers today and most people simply don't know it. Did I miss
something?

Simon

>
> But still, you should include this in testing please.
>
>
> diff -puN fs/bio.c~bio_clone-fix fs/bio.c
> --- devel/fs/bio.c~bio_clone-fix	2005-07-28 00:39:40.000000000 -0700
> +++ devel-akpm/fs/bio.c	2005-07-28 01:02:34.000000000 -0700
> @@ -261,6 +261,7 @@ inline void __bio_clone(struct bio *bio,
>  	 */
>  	bio->bi_vcnt = bio_src->bi_vcnt;
>  	bio->bi_size = bio_src->bi_size;
> +	bio->bi_idx = bio_src->bi_idx;
>  	bio_phys_segments(q, bio);
>  	bio_hw_segments(q, bio);
>  }
> _
>
>

Re: File corruption on LVM2 on top of software RAID1

[Andrew Morton]

Andrew Morton <akpm@osdl.org> wrote:
>
> There's one fix against 2.6.12.3 which is needed, but 2.6.9 didn't have the
>  bug which this fix addresses.

aargh, I see that it did fix it.

Don't blame me.  Blame people who screw up list threading by reading a
mail->news gateway and hitting "reply".

Re: File corruption on LVM2 on top of software RAID1

[Andrew Morton]

"Simon Matter" <simon.matter@invoca.ch> wrote:
>
> Hi,
> 
> Please CC me as I'm not subscribed to the kernel list.
> 
> I had a hard time identifying a serious problem in the 2.6 linux kernel.

Yes, it is a serious problem.

> It all started while evaluating RHEL4 for new servers. My data integrity
> tests gave me bad results - which I couldn't believe - and my first idea
> was - of course - bad hardware. I ordered new SCSI disks instead of the
> IDE disks, took another server, spent some money again, tried again and
> again. That's all long ago now...
> 
> In my tests I get corrupt files on LVM2 which is on top of software raid1.
> (This is a common setup even mentioned in the software RAID HOWTO and has
> worked for me on RedHat 9 / kernel 2.4 for a long time now and it's my
> favourite configuration). Now, I tested two different distributions, three
> kernels, three different filesystems and three different hardware. I can
> always reproduce it with the following easy scripts:
> 
> LOGF=/root/diff.log
> while true; do
>   rm -rf /home/XXX2
>   rsync -a /home/XXX/ /home/XXX2
>   date >> $LOGF
>   diff -r /home/XXX /home/XXX2 >> $LOGF
> done
> 
> the files in /home/XXX are ~15G of ISO images and rpms.
> 
> diff.log looks like this:
> Wed Aug  3 13:45:57 CEST 2005
> Binary files /home/XXX/ES3-U3/rhel-3-U3-i386-es-disc3.iso and
> /home/XXX2/ES3-U3/rhel-3-U3-i386-es-disc3.iso differ
> Wed Aug  3 14:09:14 CEST 2005
> Binary files /home/XXX/8.0/psyche-i386-disc1.iso and
> /home/XXX2/8.0/psyche-i386-disc1.iso differ
> Wed Aug  3 14:44:17 CEST 2005
> Binary files /home/XXX/7.3/valhalla-i386-disc3.iso and
> /home/XXX2/7.3/valhalla-i386-disc3.iso differ
> Wed Aug  3 15:15:05 CEST 2005
> Wed Aug  3 15:45:40 CEST 2005
> 
> 
> Tested software:
> 1) RedHat EL4
> kernel-2.6.9-11.EL
> vanilla 2.6.12.3 kernel
> filesystems: EXT2, EXT3, XFS
> 
> 2) NOVELL/SUSE 9.3
> kernel-default-2.6.11.4-21.7
> filesystem: EXT3
> 
> Tested Hardware:
> 1)
> - ASUS P2B-S board
> - CPU PIII 450MHz
> - Intel 440BX/ZX/DX Chipset
> - 4x128M memory (ECC enabled)
> - 2x IDE disks Seagate Barracuda 400G, connected to onboard "Intel PIIX4
> Ultra 33"
> - Promise Ultra100TX2 adapter for additional tests
> 
> 2)
> - DELL PowerEdge 1400
> - CPU PIII 800MHz
> - ServerWorks OSB4 Chipset
> - 4x256M memory (ECC enabled)
> - 2x U320 SCSI disks Maxtor Atlas 10K 146G
> - onboard Adaptec aic7899 Ultra160 SCSI adapter
> 
> 3)
> - DELL PowerEdge 1850
> - CPU P4 XEON 2.8GHz
> - 2G memory
> - 2x U320 SCSI disks Maxtor Atlas 10K 73G SCA
> - onboard LSI53C1030 SCSI adapter
> 
> I've put some files toghether from the last test on the PE1850 server:
> http://www.invoca.ch/bugs/linux-2.6-corruption-on-lvm2-on-raid1/
> 
> I've also filed a bug with RedHat:
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=164696
> 
> I have spent a lot of time on this bug because I consider it very serious.
> I'm not a kernel hacker but if there is anything I can do to fix this, let
> me know.
> 

Thanks for doing that.

There's one fix against 2.6.12.3 which is needed, but 2.6.9 didn't have the
bug which this fix addresses.  So unless 2.6.9 has a different problem,
this won't help.

But still, you should include this in testing please.



diff -puN fs/bio.c~bio_clone-fix fs/bio.c
--- devel/fs/bio.c~bio_clone-fix	2005-07-28 00:39:40.000000000 -0700
+++ devel-akpm/fs/bio.c	2005-07-28 01:02:34.000000000 -0700
@@ -261,6 +261,7 @@ inline void __bio_clone(struct bio *bio,
 	 */
 	bio->bi_vcnt = bio_src->bi_vcnt;
 	bio->bi_size = bio_src->bi_size;
+	bio->bi_idx = bio_src->bi_idx;
 	bio_phys_segments(q, bio);
 	bio_hw_segments(q, bio);
 }
_

File corruption on LVM2 on top of software RAID1

[Simon Matter]

Hi,

Please CC me as I'm not subscribed to the kernel list.

I had a hard time identifying a serious problem in the 2.6 linux kernel.
It all started while evaluating RHEL4 for new servers. My data integrity
tests gave me bad results - which I couldn't believe - and my first idea
was - of course - bad hardware. I ordered new SCSI disks instead of the
IDE disks, took another server, spent some money again, tried again and
again. That's all long ago now...

In my tests I get corrupt files on LVM2 which is on top of software raid1.
(This is a common setup even mentioned in the software RAID HOWTO and has
worked for me on RedHat 9 / kernel 2.4 for a long time now and it's my
favourite configuration). Now, I tested two different distributions, three
kernels, three different filesystems and three different hardware. I can
always reproduce it with the following easy scripts:

LOGF=/root/diff.log
while true; do
  rm -rf /home/XXX2
  rsync -a /home/XXX/ /home/XXX2
  date >> $LOGF
  diff -r /home/XXX /home/XXX2 >> $LOGF
done

the files in /home/XXX are ~15G of ISO images and rpms.

diff.log looks like this:
Wed Aug  3 13:45:57 CEST 2005
Binary files /home/XXX/ES3-U3/rhel-3-U3-i386-es-disc3.iso and
/home/XXX2/ES3-U3/rhel-3-U3-i386-es-disc3.iso differ
Wed Aug  3 14:09:14 CEST 2005
Binary files /home/XXX/8.0/psyche-i386-disc1.iso and
/home/XXX2/8.0/psyche-i386-disc1.iso differ
Wed Aug  3 14:44:17 CEST 2005
Binary files /home/XXX/7.3/valhalla-i386-disc3.iso and
/home/XXX2/7.3/valhalla-i386-disc3.iso differ
Wed Aug  3 15:15:05 CEST 2005
Wed Aug  3 15:45:40 CEST 2005


Tested software:
1) RedHat EL4
kernel-2.6.9-11.EL
vanilla 2.6.12.3 kernel
filesystems: EXT2, EXT3, XFS

2) NOVELL/SUSE 9.3
kernel-default-2.6.11.4-21.7
filesystem: EXT3

Tested Hardware:
1)
- ASUS P2B-S board
- CPU PIII 450MHz
- Intel 440BX/ZX/DX Chipset
- 4x128M memory (ECC enabled)
- 2x IDE disks Seagate Barracuda 400G, connected to onboard "Intel PIIX4
Ultra 33"
- Promise Ultra100TX2 adapter for additional tests

2)
- DELL PowerEdge 1400
- CPU PIII 800MHz
- ServerWorks OSB4 Chipset
- 4x256M memory (ECC enabled)
- 2x U320 SCSI disks Maxtor Atlas 10K 146G
- onboard Adaptec aic7899 Ultra160 SCSI adapter

3)
- DELL PowerEdge 1850
- CPU P4 XEON 2.8GHz
- 2G memory
- 2x U320 SCSI disks Maxtor Atlas 10K 73G SCA
- onboard LSI53C1030 SCSI adapter

I've put some files toghether from the last test on the PE1850 server:
http://www.invoca.ch/bugs/linux-2.6-corruption-on-lvm2-on-raid1/

I've also filed a bug with RedHat:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=164696

I have spent a lot of time on this bug because I consider it very serious.
I'm not a kernel hacker but if there is anything I can do to fix this, let
me know.

Simon