Question regarding ext3 extents+mballoc+delalloc

Question regarding ext3 extents+mballoc+delalloc

[Holger Kiehl]

Hello

Looking at ways to increase write performance on my system using ext3
Andreas Dilger pointed me to delalloc+mballoc+extent patches. Downloaded
those from ftp://ftp.clusterfs.com/pub/people/alex/2.6.16.8 and run some
benchmark, here some results using bonnie++:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
2.6.16.19       16G 59223  91 264155  45 111459  36 57313  99 317944  63  1478   7
                     58814  92 276803  47 110418  36 57105  99 317534  65  1525   5
                     58299  92 274523  48 110290  36 56723  99 318839  65  1502   4

And here the results when mounting without extents,mballoc,delalloc option:


Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
2.6.16.19       16G 38621  98 194816  94 87776  49 37921  92 239128  54  1402   5
                     47000  99 194276  94 89232  49 38628  92 240539  55  1399   5
                     45873  98 178195  90 89726  50 38482  92 240490  55  1381   4

So using delalloc+mballoc+extent gives an approx. 30% increase in performance.

Using my own benchmark afdbench where many process copy thousands of files
around, the results are as follows:

With extents,mballoc,delalloc

      5488.36 files per second 14.19 MB/s

Without extents,mballoc,delalloc

      4829.84 files per second 12.72 MB/s

The performance increase is approx. 12%.

So the question is, why are these patches not included into the kernel?
I did some very extensive testing for several days and could not discover
any disadvantage using those patches. I must add that I did not test
crashes to see if data is lost. Are there any disadvantages using these
patches?

Please CC me since I am not subscribed to the list.

Thanks,
Holger
-- 

Re: Question regarding ext3 extents+mballoc+delalloc

[Andreas Dilger]

On Jun 06, 2006  10:23 +0000, Holger Kiehl wrote:
> Looking at ways to increase write performance on my system using ext3
> Andreas Dilger pointed me to delalloc+mballoc+extent patches. Downloaded
> those from ftp://ftp.clusterfs.com/pub/people/alex/2.6.16.8 and run some
> benchmark, here some results using bonnie++:

[note: this is WITH extents,mballoc,delalloc enabled]

> Version  1.03 ------Sequential Output------ --Sequential Input-  --Random-
>                -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine   Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> 2.6.16.19  16G 59223  91 264155 45 111459 36  57313 99 317944 63  1478   7
>                58814  92 276803 47 110418 36  57105 99 317534 65  1525   5
>                58299  92 274523 48 110290 36  56723 99 318839 65  1502   4
> 
> And here the results when mounting without extents,mballoc,delalloc option:
I was confused initially until I saw ^^^^^^^

> Version  1.03 ------Sequential Output------ --Sequential Input-  --Random-
>                -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine   Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> 2.6.16.19  16G 38621  98 194816 94  87776 49  37921 92 239128 54  1402   5
>                47000  99 194276 94  89232 49  38628 92 240539 55  1399   5
>                45873  98 178195 90  89726 50  38482 92 240490 55  1381   4
> 
> So using delalloc+mballoc+extent gives an approx. 30% increase in 
> performance.

Note also that there is a 50% reduction in CPU usage for writes (27% for
rewrites).  This is important when you are trying to maximize IO from a
single server.  I'm not sure why the read CPU usage increased, though it
may just be a result of increased memcpy due to the higher read throughput
(32% increase in read performance, 18% increase in CPU usage).

> So the question is, why are these patches not included into the kernel?
> I did some very extensive testing for several days and could not discover
> any disadvantage using those patches. I must add that I did not test
> crashes to see if data is lost. Are there any disadvantages using these
> patches?

One of the main reasons this isn't in the kernel yet is that the extents
on-disk format is incompatible with the current ext3 on-disk format.
That is OK for Lustre because the storage servers are essentially
"appliances" that are used in well-controlled environments, but this
isn't so good when random users get involved.  The patches couldn't be
merged until there was some consensus reached about the extents on-disk
format.


There is work currently underway with Red Hat, IBM, CFS, and Bull
to merge the extents support into the kernel.org ext3 code and the
official e2fsprogs, and this will likely also be in the upcoming RHEL5.
Once this is done it will be possible to merge the mballoc and delalloc
changes also.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: Question regarding ext3 extents+mballoc+delalloc

[Holger Kiehl]

On Tue, 6 Jun 2006, Andreas Dilger wrote:

> On Jun 06, 2006  10:23 +0000, Holger Kiehl wrote:
>> Looking at ways to increase write performance on my system using ext3
>> Andreas Dilger pointed me to delalloc+mballoc+extent patches. Downloaded
>> those from ftp://ftp.clusterfs.com/pub/people/alex/2.6.16.8 and run some
>> benchmark, here some results using bonnie++:
>
> [note: this is WITH extents,mballoc,delalloc enabled]
>
>> Version  1.03 ------Sequential Output------ --Sequential Input-  --Random-
>>                -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine   Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>> 2.6.16.19  16G 59223  91 264155 45 111459 36  57313 99 317944 63  1478   7
>>                58814  92 276803 47 110418 36  57105 99 317534 65  1525   5
>>                58299  92 274523 48 110290 36  56723 99 318839 65  1502   4
>>
>> And here the results when mounting without extents,mballoc,delalloc option:
> I was confused initially until I saw ^^^^^^^
>
Sorry for the confusion.

>> Version  1.03 ------Sequential Output------ --Sequential Input-  --Random-
>>                -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine   Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>> 2.6.16.19  16G 38621  98 194816 94  87776 49  37921 92 239128 54  1402   5
>>                47000  99 194276 94  89232 49  38628 92 240539 55  1399   5
>>                45873  98 178195 90  89726 50  38482 92 240490 55  1381   4
>>
>> So using delalloc+mballoc+extent gives an approx. 30% increase in
>> performance.
>
> Note also that there is a 50% reduction in CPU usage for writes (27% for
> rewrites).  This is important when you are trying to maximize IO from a
> single server.  I'm not sure why the read CPU usage increased, though it
> may just be a result of increased memcpy due to the higher read throughput
> (32% increase in read performance, 18% increase in CPU usage).
>
My main concern is write speed and here these patches help a lot. I have
not tested it, but I think this brings ext3 performance to that of ext2.

>> So the question is, why are these patches not included into the kernel?
>> I did some very extensive testing for several days and could not discover
>> any disadvantage using those patches. I must add that I did not test
>> crashes to see if data is lost. Are there any disadvantages using these
>> patches?
>
> One of the main reasons this isn't in the kernel yet is that the extents
> on-disk format is incompatible with the current ext3 on-disk format.
> That is OK for Lustre because the storage servers are essentially
> "appliances" that are used in well-controlled environments, but this
> isn't so good when random users get involved.  The patches couldn't be
> merged until there was some consensus reached about the extents on-disk
> format.
>
>
> There is work currently underway with Red Hat, IBM, CFS, and Bull
> to merge the extents support into the kernel.org ext3 code and the
> official e2fsprogs, and this will likely also be in the upcoming RHEL5.
> Once this is done it will be possible to merge the mballoc and delalloc
> changes also.
>
Just to ensure that I understand this correctly. The on-disk format is not
final and it will still change. This means if I use it now I will have
to reformat the disk when ever the format is changed.

As you mention e2fsprogs also needs to be updated:

    # dumpe2fs -h /dev/md7
    dumpe2fs 1.38 (30-Jun-2005)
    dumpe2fs: Filesystem has unsupported feature(s) while trying to open /dev/md7
    Couldn't find valid filesystem superblock.

Are there any patches down loadable, that add support to the e2fsprogs?

It would really be nice if consensus about the extents on-disk format
could be reached, so more people could benefit from it. Asking when this
will be reached, does not make sense?

Thanks,
Holger

Re: Question regarding ext3 extents+mballoc+delalloc

[Andreas Dilger]

On Jun 07, 2006  06:50 +0000, Holger Kiehl wrote:
> On Tue, 6 Jun 2006, Andreas Dilger wrote:
> >One of the main reasons this isn't in the kernel yet is that the extents
> >on-disk format is incompatible with the current ext3 on-disk format.
> >That is OK for Lustre because the storage servers are essentially
> >"appliances" that are used in well-controlled environments, but this
> >isn't so good when random users get involved.  The patches couldn't be
> >merged until there was some consensus reached about the extents on-disk
> >format.
>
> Just to ensure that I understand this correctly. The on-disk format is not
> final and it will still change. This means if I use it now I will have
> to reformat the disk when ever the format is changed.

Actually, in the end the format of the current extents is the one that will
be supported by kernel.org kernels.  It may be that there will be another
extents format at some time in the future, but the current format was deemed
good enough for now.

There are still a couple of minor changes that are being worked on, but
a filesystem with the current extents format will be usable in the future.

> As you mention e2fsprogs also needs to be updated:
> 
>    # dumpe2fs -h /dev/md7
>    dumpe2fs 1.38 (30-Jun-2005)
>    dumpe2fs: Filesystem has unsupported feature(s) while trying to open 
>    /dev/md7
>    Couldn't find valid filesystem superblock.
> 
> Are there any patches down loadable, that add support to the e2fsprogs?

Yes, these were posted as part of a whole patch series by Bull to ext2-devel.
You can also download the CFS e2fsprogs-1.38-cfs2 RPMs from
ftp://ftp.lustre.org/pub/lustre/other/e2fsprogs/

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.