Re: [RFC 0/13] extents and 48bit ext3

Re: [RFC 0/13] extents and 48bit ext3

[linux]

> We seem to be lagging behind "the industry" in some areas - handling large
> devices, high bandwidth IO, sophisticated on-disk data structures, advanced
> manageability, etc.

Er... I would like to point out that "sophisticated on-disk data
structures" are, in and of themselves, a Bad Thing.  It's only when
they provide some desirable capability that they earn their cost in
implementation difficulty, code size, and bug rate.


ZFS is interesting, and I Really Really Like its reliability guarantees,
but I notice that, due to the append-only nature of its operation,
it's extraordinarily difficult to move data once it's been written.
This makes migrating a file system off of old nasty disks to big new
disks rather annoying.  If you know before you add the new drives, you
can physically mirror the old disks and avoid changing block pointers,
but I'd wish for something more flexible.

Because block pointers are physical, and all checksummed, moving a
single block requires rewriting the root block of every snapshot that
contains that block.  Now, you can keep an index of "old block X is now
in new location Y" while walking the entire file system until you're
sure that all the old pointers are gone, but it's hard to preallocate
that index, because you also have to know that "old pointer block X
has been recreated at new location Y, but its contents are different;
only the logical content is the same", and there's no obvious way to
bound the number of such forwarding notes that need to be made.

You must have such an index, or you can't preserve sharing while you
migrate the data.

H'm... for sane efficiency, you also need to keep track of all metadata
blocks that have been examined and NOT changed, so when you hit them again
traversing the file system structure DAG, you know that you can stop.
Between the two, this amounts to every metadata block on the file system.
Wow!

Well, at least that gives you an upper limit on the size needed.
One block forwarding entry per data block on the migrated-from disk,
plus one index-forwarding entry (which may be larger, if it contains
the new block checksum) for each index block on the entire file system.

Ouch.

(And, of course, all of this has to be done on a live file system.)

Re: [RFC 0/13] extents and 48bit ext3

[Theodore Tso]

On Sat, Jun 10, 2006 at 04:07:14PM +0200, Olivier Galibert wrote:
> > Incorrect, because unless you explicitly enable the use of extents,
> > the mere act of using a new kernel such as might be found on knoppix
> > will not result in the filesystem utilizing the extent feature.
> 
> And how shall the rescue/live CD know whether to use the feature?

Because there will be a bit the superblock that the user will have to
explicitly enable in order to get extents, so a new kernel on the
rescue/live CD will no whether or not extents are allowed --- just as
today, you have to explicitly enable hashed tree directory indexing
with the command, tune2fs -O dir_index /dev/hdXXX.

							- Ted

Re: [RFC 0/13] extents and 48bit ext3

[Linus Torvalds]

On Sat, 10 Jun 2006, Kyle Moffett wrote:
> 
> One possible solution to the version-confusion that would avoid duplicating
> features would be to merge the fs/ext{2,3} to fs/ext, then make fs/ext
> register itself as a filesystem under "ext2", "ext3", and "ext4".

But the thing is, technical people don't actually care about the version 
confusion.

The real issue is that ext3 is a stable filesystem, and the ext4 stuff 
buys fundamentally and absolutely _nothing_ for the vast majority of uses. 
Except pain.

So the real reason for the split would be the _user_ split. There are 
people who want big filesystems, and there are people who don't care. 

It's that simple.

> I've heard quite some griping about the amount of duplicated code 
> between ext2 and ext3;

That's a total piece of bullshit. Nobody seriously gripes about the 
duplication, and the ones that do have absolutely no idea what that split 
bought us. Ignore them.

			Linus

Re: [RFC 0/13] extents and 48bit ext3

[Kyle Moffett]

On Jun 9, 2006, at 11:25:31, Jeff Garzik wrote:
> Overall, I'm surprised that ext3 developers don't see any of the  
> problems related to progressive, stealth filesystem upgrades.
>
> Users are never given a clear indication of when their metadata is  
> being upgraded, there is no clear "line of demarcation" they cross,  
> when they start using extents.
>
> Since there is no user-visible fs upgrade event, users do not have  
> a clear picture of what features are being used -- which means they  
> are kept in the dark about which kernels are OK to use on their data.
>
> Do you guys honestly expect users to keep track of which kernels  
> added specific ext3 features?
>
> This is why other enterprise filesystems have clear "fs version 1",  
> "fs version 2" points across which a user migrates.  ext3's feature- 
> flags approach just means that there are a million combinations of  
> potential old-and-new features, in-tree and third party, all of  
> which must be supported.

One possible solution to the version-confusion that would avoid  
duplicating features would be to merge the fs/ext{2,3} to fs/ext,  
then make fs/ext register itself as a filesystem under "ext2",  
"ext3", and "ext4".  Then have each name imply a specific set of  
features and compatibility.  That would allow the same performance  
optimizations to affect all 3 even as you make metadata changes in  
the latest version.  I've heard quite some griping about the amount  
of duplicated code between ext2 and ext3; why cause those problems  
again with an "ext4"?  There would probably be some fs/ext/ext{2,3,4} 
_foo.c files that could be compiled in or out depending on configured  
FS support, but I would guess that would make it easier on users and  
developers alike.

Cheers,
Kyle Moffett

Re: [RFC 0/13] extents and 48bit ext3

[Olivier Galibert]

On Fri, Jun 09, 2006 at 09:06:51PM -0400, Theodore Tso wrote:
> On Sat, Jun 10, 2006 at 02:49:32AM +0200, Sven-Haegar Koch wrote:
> > I see a different problem with "ext3 + extends is not ext3 anymore" when 
> > the feature goes mainstream:
> > - user with old distri, no extends in use, no kernel support for them
> > - user has some kind of problem
> > - uses new rescue disk (aka knoppix at the time of problem) - that then
> >   is current stuff, and certainly uses extents - fixes problem on disk
> >   (may be a simple as running lilo/grub from chroot, happens often for me)
> > - tries to boot back into his distri -> *boom* he lost
> 
> Incorrect, because unless you explicitly enable the use of extents,
> the mere act of using a new kernel such as might be found on knoppix
> will not result in the filesystem utilizing the extent feature.

And how shall the rescue/live CD know whether to use the feature?

  OG.

Re: [RFC 0/13] extents and 48bit ext3

[Bernd Eckenfels]

Sven-Haegar Koch <haegar@sdinet.de> wrote:
> I see a different problem with "ext3 + extends is not ext3 anymore" when 
> the feature goes mainstream:
> - user with old distri, no extends in use, no kernel support for them
> - user has some kind of problem
> - uses new rescue disk (aka knoppix at the time of problem) - that then
>   is current stuff, and certainly uses extents - fixes problem on disk
>   (may be a simple as running lilo/grub from chroot, happens often for me)
> - tries to boot back into his distri -> *boom* he lost

I dont see a need to enable extends on small filesystems or on existing ones
(from within any tool). Why would that happen? Do you had this problem with
sparse_super or dir_index?

Gruss
Bernd

Re: [RFC 0/13] extents and 48bit ext3

[Nathan Scott]

On Fri, Jun 09, 2006 at 03:15:53PM -0700, Andrew Morton wrote:
> Sonny Rao <sonny@burdell.org> wrote:
> > On Fri, Jun 09, 2006 at 10:35:43AM -0700, Andrew Morton wrote:
> > <snip> 
> > > All that being said, Linux's filesystems are looking increasingly crufty
> > > and we are getting to the time where we would benefit from a greenfield
> > > start-a-new-one.  
> > 
> > I'm curious about this comment; in what way are they _collectively_
> > looking crufty ? 
> 
> We seem to be lagging behind "the industry" in some areas - handling large
> devices, high bandwidth IO, sophisticated on-disk data structures, advanced
> manageability, etc.

Er, no.  I'm not aware of many filesystems that are in the same
league as XFS on those first three specific points.  It certainly
has "ondisk sophistication" very well covered, trust me. ;)

We are definately not lagging on handling large devices nor high
bandwidth I/O anyway - XFS serves up very close to the hardware
capabilities for high end hardware and it scales well.  One could
come up with a different list of areas where Linux filesystems
might be lagging, but that list above ain't right.

> I mean, although ZFS is a rampant layering violation and we can do a lot of
> the things in there (without doing it all in the fs!) I don't think we can
> do all of it.

*nod*.

cheers.

-- 
Nathan

Re: [RFC 0/13] extents and 48bit ext3

[Theodore Tso]

On Sat, Jun 10, 2006 at 02:49:32AM +0200, Sven-Haegar Koch wrote:
> I see a different problem with "ext3 + extends is not ext3 anymore" when 
> the feature goes mainstream:
> - user with old distri, no extends in use, no kernel support for them
> - user has some kind of problem
> - uses new rescue disk (aka knoppix at the time of problem) - that then
>   is current stuff, and certainly uses extents - fixes problem on disk
>   (may be a simple as running lilo/grub from chroot, happens often for me)
> - tries to boot back into his distri -> *boom* he lost

Incorrect, because unless you explicitly enable the use of extents,
the mere act of using a new kernel such as might be found on knoppix
will not result in the filesystem utilizing the extent feature.

There's a lot FUD being spread by people who haven't been bothering to
understand what is being proposed, and that's disappointing.

						- Ted

Re: [RFC 0/13] extents and 48bit ext3

[Sven-Haegar Koch]

On Fri, 9 Jun 2006, Michael Poole wrote:

> Jeff Garzik writes:
>
>> Andrew Morton wrote:
>>> Ted&co have been pretty good at avoiding compatibility problems.
>>
>> Well, extents and 48bit make that track record demonstrably worse.
>>
>> Users are now forced to remember that, if they write to their
>> filesystem after using either $mmver or $korgver kernels, they are
>> locked out of using older kernels.
>
> Users are also forced to remember that, if they use certain new
> distros or programs, they are locked out of using older kernels.  They
> are forced to remember that if they have certain newer hardware, they
> are locked out of using older kernels.  They are forced to remember
> that if they use ext3 (or XFS or JFS) _at all_ they are locked out of
> using older kernels.  Why single out this particular aspect of limited
> forward compatibility to harp on so much?

I see a different problem with "ext3 + extends is not ext3 anymore" when 
the feature goes mainstream:
- user with old distri, no extends in use, no kernel support for them
- user has some kind of problem
- uses new rescue disk (aka knoppix at the time of problem) - that then
   is current stuff, and certainly uses extents - fixes problem on disk
   (may be a simple as running lilo/grub from chroot, happens often for me)
- tries to boot back into his distri -> *boom* he lost

c'ya
sven

-- 

The Internet treats censorship as a routing problem, and routes around it.
(John Gilmore on http://www.cygnus.com/~gnu/)

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Andreas Dilger wrote:
> I'm not so strongly against ext4 that I won't follow that route if needed,
> but it essentially means that ext3 will be orphaned.

Not orphaned but scaled back over time.  IMO there's only so much 
developer and brain and test bandwidth for "the main Linux filesystem."

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Andreas Dilger]

On Jun 09, 2006  15:15 -0700, Andrew Morton wrote:
> We seem to be lagging behind "the industry" in some areas - handling large
> devices, high bandwidth IO, sophisticated on-disk data structures, advanced
> manageability, etc.
> 
> I mean, although ZFS is a rampant layering violation and we can do a lot of
> the things in there (without doing it all in the fs!) I don't think we can
> do all of it.
> 
> We're continuing to nurse along a few basically-15-year-old filesystems
> while we do have the brains, manpower and processes to implement a new,
> really great one.
> 
> It's just this feeling I have ;)

I think many people share this feeling (me included), hence the linux
filesystem meeting next week...  The problem is that even getting a
half-decent disk filesystem is many years of work, and large disks are
here before then.  The ZFS code took 10 years to get to its current state,
I understand, so I don't anticipate we will get there overnight.

The question is whether we can get to this state more easily by starting
on a known-good base (ext3) or by starting from scratch.  My opinion is
strongly in the "start from a known-good base" camp, and make incremental
improvements to that base instead of discarding everything and starting
again.

I think the real frontier for future filesystem development is in the
ZFS direction where the filesystem can be robust in the face of data
errors without having a single fail-stop mode of error handling.  While
ext2 and ext3 have been OK in this regard they can definitely be improved
without discarding the rest of the code and the millions of hours of
testing that has gone into it.

I'm not so strongly against ext4 that I won't follow that route if needed,
but it essentially means that ext3 will be orphaned.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: [RFC 0/13] extents and 48bit ext3

[Andrew Morton]

Sonny Rao <sonny@burdell.org> wrote:
>
> On Fri, Jun 09, 2006 at 10:35:43AM -0700, Andrew Morton wrote:
> <snip> 
> > All that being said, Linux's filesystems are looking increasingly crufty
> > and we are getting to the time where we would benefit from a greenfield
> > start-a-new-one.  
> 
> I'm curious about this comment; in what way are they _collectively_
> looking crufty ? 

We seem to be lagging behind "the industry" in some areas - handling large
devices, high bandwidth IO, sophisticated on-disk data structures, advanced
manageability, etc.

I mean, although ZFS is a rampant layering violation and we can do a lot of
the things in there (without doing it all in the fs!) I don't think we can
do all of it.

We're continuing to nurse along a few basically-15-year-old filesystems
while we do have the brains, manpower and processes to implement a new,
really great one.

It's just this feeling I have ;)

Re: [RFC 0/13] extents and 48bit ext3

[Sonny Rao]

On Fri, Jun 09, 2006 at 10:35:43AM -0700, Andrew Morton wrote:
<snip> 
> All that being said, Linux's filesystems are looking increasingly crufty
> and we are getting to the time where we would benefit from a greenfield
> start-a-new-one.  

I'm curious about this comment; in what way are they _collectively_
looking crufty ? 

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Michael Poole wrote:
> Jeff Garzik writes:
> 
>> Andrew Morton wrote:
>>> Ted&co have been pretty good at avoiding compatibility problems.
>> Well, extents and 48bit make that track record demonstrably worse.
>>
>> Users are now forced to remember that, if they write to their
>> filesystem after using either $mmver or $korgver kernels, they are
>> locked out of using older kernels.
> 
> Users are also forced to remember that, if they use certain new
> distros or programs, they are locked out of using older kernels.  They
> are forced to remember that if they have certain newer hardware, they
> are locked out of using older kernels.  They are forced to remember
> that if they use ext3 (or XFS or JFS) _at all_ they are locked out of
> using older kernels.  Why single out this particular aspect of limited
> forward compatibility to harp on so much?

Because it's called backwards compat, when it isn't?
Because it is very difficult to find out which set of kernels you are 
locked out of?
Because the filesystem upgrade is stealthy, occurring as it does on the 
first data write?

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Michael Poole]

Jeff Garzik writes:

> Andrew Morton wrote:
> > Ted&co have been pretty good at avoiding compatibility problems.
> 
> Well, extents and 48bit make that track record demonstrably worse.
> 
> Users are now forced to remember that, if they write to their
> filesystem after using either $mmver or $korgver kernels, they are
> locked out of using older kernels.

Users are also forced to remember that, if they use certain new
distros or programs, they are locked out of using older kernels.  They
are forced to remember that if they have certain newer hardware, they
are locked out of using older kernels.  They are forced to remember
that if they use ext3 (or XFS or JFS) _at all_ they are locked out of
using older kernels.  Why single out this particular aspect of limited
forward compatibility to harp on so much?

Michael Poole

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Jeff Garzik wrote:
> I disagree completely...  it would be an obvious win:  people who want 
> stability get that, people who want new features get that too.

And developers have a better outlet for their wacky developmental urges...

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Andrew Morton wrote:
> On Fri, 09 Jun 2006 13:07:37 -0400
> Jeff Garzik <jeff@garzik.org> wrote:
> 
>> I would propose the obvious...  'cp -a ext3 ext4', apply the extent and 
>> 48bit patches, and then do the obvious search-n-replace.
> 
> Most of ext3 is JBD.  At least, in terms of complexity.  And I don't think
> there's anything in this proposal which affects JBD, apart from changing
> the blocksize.
> 
> Cloning JBD for this exercise would, I suspect, be the wrong thing to do -
> the two clones would be pretty much identical, apart from some scalar
> types.
> 
> I did suggest a couple of years ago that we should clone the ext3 part and
> have both ext3 and ext4 use the same JBD layer - I don't know what happened
> to that idea.

The JBD API is reasonably distinct, so IMO this would be a logical next 
step.  I would hope they could use the same JBD, so, I strongly agree...


> There has been steady, cautious but significant improvement happening in
> ext3 over the past few years.  I'd expect that to continue, although
> perhaps at a lower rate.  Having to apply the same changes to two
> filesystems would be an obvious loss.

I disagree completely...  it would be an obvious win:  people who want 
stability get that, people who want new features get that too.


> It comes down to looking at the patches, and I haven't done that in quite
> some time.  Ideally the new functionality would all be under CONFIG_foo,
> but I do not know if that is being proposed here?
> 
>> We need to draw a line in the sand.  If we don't, no one ever will.
> 
> You speak as if this is something which has happened before, or that it will
> happen again.
> 
> All that being said, Linux's filesystems are looking increasingly crufty
> and we are getting to the time where we would benefit from a greenfield
> start-a-new-one.  That new one might even be based on reiser4 - has anyone
> looked?  It's been sitting around for a couple of years.

reiser4 actually has this same problem, but worse.  It has pluggable 
metadata even to the point of supporting plugin-style metadata development.

If we can successfully devolve a filesystem to metadata and algorithm 
plugins, that should be done at the VFS level, and not called "reiser4".

But in the absence of a different VFS API, I think it is the most 
practical of all the options to open the floodgates to ext4 rather than 
ext3.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Andrew Morton]

On Fri, 09 Jun 2006 13:07:37 -0400
Jeff Garzik <jeff@garzik.org> wrote:

> I would propose the obvious...  'cp -a ext3 ext4', apply the extent and 
> 48bit patches, and then do the obvious search-n-replace.

Most of ext3 is JBD.  At least, in terms of complexity.  And I don't think
there's anything in this proposal which affects JBD, apart from changing
the blocksize.

Cloning JBD for this exercise would, I suspect, be the wrong thing to do -
the two clones would be pretty much identical, apart from some scalar
types.

I did suggest a couple of years ago that we should clone the ext3 part and
have both ext3 and ext4 use the same JBD layer - I don't know what happened
to that idea.

There has been steady, cautious but significant improvement happening in
ext3 over the past few years.  I'd expect that to continue, although
perhaps at a lower rate.  Having to apply the same changes to two
filesystems would be an obvious loss.

It comes down to looking at the patches, and I haven't done that in quite
some time.  Ideally the new functionality would all be under CONFIG_foo,
but I do not know if that is being proposed here?

> We need to draw a line in the sand.  If we don't, no one ever will.

You speak as if this is something which has happened before, or that it will
happen again.

All that being said, Linux's filesystems are looking increasingly crufty
and we are getting to the time where we would benefit from a greenfield
start-a-new-one.  That new one might even be based on reiser4 - has anyone
looked?  It's been sitting around for a couple of years.

Re: [RFC 0/13] extents and 48bit ext3

[Alan Cox]

Ar Gwe, 2006-06-09 am 09:42 -0600, ysgrifennodd Matthew Wilcox:
> Hang on, you're going too far.  You have to enable extents with the
> extent mount option.  Otherwise you don't get to use them.  The user
> does, in fact, have a clear division, although maybe the blinky signs
> aren't quite luminous enough.

<mba marketing>
I'd rather the blinky sign was "ext4". It makes it clear it is a
progression and it also gives everyone something to put in the features
box and talk to the press about 8)
</mba>

> I still think making ext3 bigger than 16TB is just silly.

We recently fixed a 'If the disk is 4TB in size the geometry reporting
breaks and parted crashes' bug. The stuff is out there and people want
to run ext3 on it or an ext3 derivative they feel they trust. Does it
matter whether it is the most optimal solution, that'll sort itself out
as ext3.5/ext4, reiser4, jfs, xfs etc get picked and demanded by users

Alan

Re: [RFC 0/13] extents and 48bit ext3

[Alan Cox]

Ar Iau, 2006-06-08 am 22:49 -0400, ysgrifennodd Jeff Garzik:
> People (including me) still switch back and forth between ext2 and ext3 
> mounts of the same filesystem on occasion.  I think creating an "ext4" 
> would allow for greater developer flexibility in implementing new 
> features and ditching old ones -- while also emphasizing to the user 
> that switching back and forth between ext4 and ext[23] is a bad idea.

I would agree with this, particularly as ext3 and ext4 are quite small
in the kernel side of things and people needing 48bit extents are
probably not trying to run on 8MB of flash.

Alan

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Andrew Morton wrote:
> On Fri, 09 Jun 2006 11:40:03 -0400
> Jeff Garzik <jeff@garzik.org> wrote:
> 
>> Users are now forced to remember that, if they write to their filesystem 
>> after using either $mmver or $korgver kernels, they are locked out of 
>> using older kernels.
> 
> The same happens if we create ext4 - earlier kernels don't support that,
> either.
> 
> I suppose we could call it ext4, although that wouldn't make much
> difference operationally.  The developers would probably choose to generate
> ext4 from the same codebase as ext3 for maintainability reasons, rather
> than choosing to copy-n-modify.  We'd need to see the patches to be able to
> finally make that judgement.

I would propose the obvious...  'cp -a ext3 ext4', apply the extent and 
48bit patches, and then do the obvious search-n-replace.

I guarantee that developer momentum would take over from there.  Rather 
than fundamentally change ext3, let's let it stabilize.


>> And as features continue to be added in this manner, this problem gets 
>> _exponentially_ worse.
> 
> "continue to be added"?  afaik this is the first time this has happened,
> and there's no plan to do it again.

ext3 developers are _fundamentally changing_ the block allocation 
structure [in a good way].  If they can get away with it once, they will 
continue to modify ext3, adding btrees and other new gadgets.  That's 
just human nature.  For example, htree was a minor disaster, 
deployment-wise, on the distro vendor side.

I think extents and 48bit are so fundamental that it's silly to attempt 
to minimize the impact from the user's perspective, and moreover, I 
think Linux benefits more if ext3 is _not_ kept on life support this way.

We need to draw a line in the sand.  If we don't, no one ever will.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Andrew Morton]

On Fri, 09 Jun 2006 11:40:03 -0400
Jeff Garzik <jeff@garzik.org> wrote:

> Users are now forced to remember that, if they write to their filesystem 
> after using either $mmver or $korgver kernels, they are locked out of 
> using older kernels.

The same happens if we create ext4 - earlier kernels don't support that,
either.

I suppose we could call it ext4, although that wouldn't make much
difference operationally.  The developers would probably choose to generate
ext4 from the same codebase as ext3 for maintainability reasons, rather
than choosing to copy-n-modify.  We'd need to see the patches to be able to
finally make that judgement.

> 
> And as features continue to be added in this manner, this problem gets 
> _exponentially_ worse.

"continue to be added"?  afaik this is the first time this has happened,
and there's no plan to do it again.

Re: [RFC 0/13] extents and 48bit ext3

[Linus Torvalds]

On Fri, 9 Jun 2006, Jeff Garzik wrote:
>
> Linus Torvalds wrote:
> > 
> > On Fri, 9 Jun 2006, Jeff Garzik wrote:
> > > Overall, I'm surprised that ext3 developers don't see any of the problems
> > > related to progressive, stealth filesystem upgrades.
> > 
> > Hey, they're used to it - they've been doing it for a long time.
> 
> Agreed, but my argument is that extents are a Big Deal.

I'm not arguing against you - I'm arguing with you.

I just tried to explain what you saw as "surprising" - the fact that ext3 
developers don't see this as a problem at all. They don't see it as a 
problem, because it's how they have always worked, since before ext3 was 
ext3, and it was just a crazy extension to ext2.

And yes, it's a serious problem. Ext3 is pretty damn messy. It's not as 
messy as some, but it sure has potential.

		Linus

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Linus Torvalds wrote:
> Quite frankly, at this point, there's no way in hell I believe we can do 
> major surgery on ext3. It's the main filesystem for a lot of users, and 
> it's just not worth the instability worries unless it's something very 
> obviously transparent.
> 
> I wouldn't mind an ext4 (that hopefully drops some of the features of 
> ext3, and might not downgrade to ext2 on errors, for example).

Certainly agreed, for all of this :)

I think that the lack of ext4 means people keep trying to stuff the 
wrong things into ext3.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Matthew Wilcox wrote:
> On Fri, Jun 09, 2006 at 11:40:03AM -0400, Jeff Garzik wrote:
>> Users are now forced to remember that, if they write to their filesystem 
>> after using either $mmver or $korgver kernels, they are locked out of 
>> using older kernels.
>>
>> From the user's perspective, ext3 has no clear "metadata version 1", 
>> "metadata version 2" division.  Thus they are now forced to keep a 
>> matrix of kernel versions and ext3 feature flag support, to know which 
>> kernels are usable with which data.  It is a support nightmare.
> 
> Hang on, you're going too far.  You have to enable extents with the
> extent mount option.  Otherwise you don't get to use them.  The user
> does, in fact, have a clear division, although maybe the blinky signs
> aren't quite luminous enough.

...and how are distros going to deploy this?  They are going to turn on 
extents by default.

And do we honestly think that is a scalable option _anyway_?  That will 
slowly bloat fstab and mount command lines with an ever-increasing list 
of options.

It's IMO better experience for the user, and gives the developers more 
freedom.Look, I _really_ want extents.  I am a big fan.  But I think 
that extents are good time to make a clean break, and let ext3 live as 
it is.  And it will let ext3 stabilize.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Fri, 9 Jun 2006, Jeff Garzik wrote:
>> Overall, I'm surprised that ext3 developers don't see any of the problems
>> related to progressive, stealth filesystem upgrades.
> 
> Hey, they're used to it - they've been doing it for a long time.

Agreed, but my argument is that extents are a Big Deal.

think about The Experience:  Suddenly users that could use 2.4.x and 
2.6.x are locked into 2.6.18+, by the simple and common act of writing 
to a file.

No bells and whistles go off...

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Matthew Wilcox]

On Fri, Jun 09, 2006 at 11:40:03AM -0400, Jeff Garzik wrote:
> Users are now forced to remember that, if they write to their filesystem 
> after using either $mmver or $korgver kernels, they are locked out of 
> using older kernels.
> 
> From the user's perspective, ext3 has no clear "metadata version 1", 
> "metadata version 2" division.  Thus they are now forced to keep a 
> matrix of kernel versions and ext3 feature flag support, to know which 
> kernels are usable with which data.  It is a support nightmare.

Hang on, you're going too far.  You have to enable extents with the
extent mount option.  Otherwise you don't get to use them.  The user
does, in fact, have a clear division, although maybe the blinky signs
aren't quite luminous enough.

I still think making ext3 bigger than 16TB is just silly.

Re: [RFC 0/13] extents and 48bit ext3

[Linus Torvalds]

On Fri, 9 Jun 2006, Jeff Garzik wrote:
>
> Overall, I'm surprised that ext3 developers don't see any of the problems
> related to progressive, stealth filesystem upgrades.

Hey, they're used to it - they've been doing it for a long time.

In fact, ext3 wouldn't be ext3 unless I (and perhaps a few others) had 
insisted on it. People wanted to try to upgrade ext2 in place.

And they've been upgrading it in-place for a long time.

Now, there are unquestionably advantages to that approach too, but as you 
say, there are absolutely tons of disadvantages too. Bugs get much much 
subtler, and more disastrous for old users that don't even want the new 
features.

Quite frankly, at this point, there's no way in hell I believe we can do 
major surgery on ext3. It's the main filesystem for a lot of users, and 
it's just not worth the instability worries unless it's something very 
obviously transparent.

I wouldn't mind an ext4 (that hopefully drops some of the features of 
ext3, and might not downgrade to ext2 on errors, for example).

			Linus

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Andrew Morton wrote:
> Ted&co have been pretty good at avoiding compatibility problems.

Well, extents and 48bit make that track record demonstrably worse.

Users are now forced to remember that, if they write to their filesystem 
after using either $mmver or $korgver kernels, they are locked out of 
using older kernels.

 From the user's perspective, ext3 has no clear "metadata version 1", 
"metadata version 2" division.  Thus they are now forced to keep a 
matrix of kernel versions and ext3 feature flag support, to know which 
kernels are usable with which data.  It is a support nightmare.

At no point is a user ever told, in big capital letters, "IF YOU WRITE 
TO THIS FILESYSTEM, YOU CAN'T BOOT OLDER KERNELS."  There is no "click 
OK to continue with this dramatic event."

And as features continue to be added in this manner, this problem gets 
_exponentially_ worse.


On the project management side of things, I see no indication that this 
momentum slow -- which implies to me that people will keep slapping new 
stuff into ext3, rather than directing energy towards a newer, cleaner 
ext-NG filesystem.

Dragging around back-compat really constrains freedom, and you have to 
have some sort of "pressure relief valve" (a massive, wildly 
incompatible update) eventually.

In my mind, it's analagous to locking developers into developing and 
deploying new features into a stable branch of software.  The hacks just 
get worse and worse, as you bend over backwards for back-compat.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Overall, I'm surprised that ext3 developers don't see any of the 
problems related to progressive, stealth filesystem upgrades.

Users are never given a clear indication of when their metadata is being 
upgraded, there is no clear "line of demarcation" they cross, when they 
start using extents.

Since there is no user-visible fs upgrade event, users do not have a 
clear picture of what features are being used -- which means they are 
kept in the dark about which kernels are OK to use on their data.

Do you guys honestly expect users to keep track of which kernels added 
specific ext3 features?

This is why other enterprise filesystems have clear "fs version 1", "fs 
version 2" points across which a user migrates.  ext3's feature-flags 
approach just means that there are a million combinations of potential 
old-and-new features, in-tree and third party, all of which must be 
supported.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Mingming Cao]

Valdis.Kletnieks@vt.edu wrote:
> On Thu, 08 Jun 2006 18:20:54 PDT, Mingming Cao said:
> 
>>Current ext3 filesystem is limited to 8TB(4k block size), this is
>>practically not enough for the increasing need of bigger storage as
>>disks in a few years (or even now).
>>
>>To address this need, there are co-effort from RedHat, ClusterFS, IBM
>>and BULL to move ext3 from 32 bit filesystem to 48 bit filesystem,
>>expanding ext3 filesystem limit from 8TB today to 1024 PB. The 48 bit
>>ext3 is build on top of extent map changes for ext3, originally from
>>Alex Tomas. In short, the new ext3 on-disk extents format is:
> 
> 
> which implies matching changes to mkfs.ext2 and possibly mount..
> 
> 
Alexandre Ratchov and Laurent Vivier from BULL have been done some work 
in e2fsprog to support extents and 48/64 bit ext3, although the patches 
have not been thoroughly reviewed and discussed yet...

http://marc.theaimsgroup.com/?l=ext2-devel&m=114848122624510&w=2

>>Appreciate any comments and feedbacks!
> 
> 
> Somebody else was recently discussing a set of patches to ext3 for
> extents+delalloc+mballoc patches - is this work compatible with that?
> 
Yes, the extents patch you mentioned is the same one included in the 
series. The delalloc (support delayed allocation for ext3) and mballoc ( 
support multiple block allocation based on extents) are considered a 
future to add, as this series is intend to address the capability issue 
and on-disk format only.

> Also, a pointer to the matching userspace patches would help anybody
> who's gung-ho enough to test the code....
>

Thanks for your interest!

We have tested patch 1-4 (which basically not touching any on-disk 
format) and they have been in mm tree. Extent patch itself have been 
tested for a long time by ClusterFS and IBM, as it's actually being 
posted a while back.

At this point the whole series pass compile, but not being tested yet. 
This post as a RFC is intend to collect comments and feedbacks. BULL 
team has done some test on the 2.6.16 version of the series with the 
e2fsprog changes they posted though. I will upload the matching 
e2fsprogs changes to ext2.sf.net/48bitsext3 shortly..

Mingming

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Please fix your mailer to stop creating bogus Mail-Followup-To headers, 
headers which exclude the original poster, and cause compliant MUAs to 
incorrectly build To/CC.


Andreas Dilger wrote:
> On Jun 08, 2006  22:49 -0400, Jeff Garzik wrote:
>> One of my common complaints about massive ext3 updates such as this is 
>> the ever-growing "which ext3 filesystem am I mounting?" problem.
>>
>> I really think extents and 48bit-ness should imply
>> 	cp -a fs/ext3 fs/ext4
>> and go from there.
> 
> The problem with this approach (as seen with ext2 and ext3) is that one
> tree or the other gets stale w.r.t. bug fixes and now we have the case
> where ext2 has a noticably different implementation in some areas and
> bug fixes are no longer trivial to apply to both trees.
> 
> I think all of the ext3 maintainers think this split was a bad idea in
> hindsight, and having an ext3 mode where it can mount without a journal
> would be much more desirable.

Please look beyond just ext2/3.  Other filesystems which have "version 
1", "version 2", "version 3", ... formats are all nasty as hell.  The 
end-result bloated code essentially supports several filesystems, all 
within the same code base, and its a nightmare of ugliness.

Further, its not only bloated, but slow.  The code inevitably winds up 
in one of two forms:

	if (spiffy new-feature metadata)
		...
	else if (updated metadata)
		...
	else /* original metadata */
		...

_or_ you add a level of indirection, by creating internal-to-the-fs 
pointer operations.

Stuffing more and more features into fs/ext3 means you are following the 
path that leads to reiser4...  where EVERYTHING under the hood is 
mutable, all within fs/ext3.


>> IMHO the ext3 back-compat situation is already really hairy, with all 
>> the features added since the original ext3 release.
> 
> While partially true, ext2/ext3 has a very good history w.r.t. compatibility
> (with one exception being the EAs on symlinks problem that slipped through
> with selinux).
> 
> Yes, the extents format will be incompatible with older ext3, but it isn't
> enabled by default so it will be completely up to the sysadmin when they
> make their filesystem incompatible.  They also won't impact any existing
> files.  The earlier extents support gets into a kernel.org kernel the
> more systems will be able to mount a filesystem with the changes when
> they becomes widely used.
> 
> All of the other features that are going to be introduced will only going
> to be applicable for format time (filesystems larger than 16TB), or if
> exceeding limits of the current ext3 support (e.g. files larger than 2TB
> in size).

Yet more progressive incompatibility, yet more

	if (metadata v2)
		...
	else /* metadata v1 */
		...

Why do you insist upon calling the end result ext3, when the truth is 
that you are slowing rewriting ext3?

As time progresses, more and more admins must ask themselves the 
question "what flavor of ext3 filesystem is on my hard drive?"

Here's a key question for ext3 developers, which I bet has no answer: 
when is it enough?  Is the plan to continually introduce incompatible 
features into ext3, over time, ad infinitum?


>> People (including me) still switch back and forth between ext2 and ext3 
>> mounts of the same filesystem on occasion.  I think creating an "ext4" 
>> would allow for greater developer flexibility in implementing new 
>> features and ditching old ones -- while also emphasizing to the user 
>> that switching back and forth between ext4 and ext[23] is a bad idea.
> 
> While this is partly true, one of the big benefits is that you can
> transparently upgrade your system to use the new features and improve
> performance without a long outage window.  Having a completely separate

Changing the name to ext4 doesn't erase this capability.


> ext4 filesystem doesn't improve the compatibility story at all.  There
> has been renewed discussion on implementing "mounting ext3 without a
> journal", just for a recovery mode, because ext2 will not be modified
> to get all of these features (running e2fsck on a huge filesystem each
> reboot would be insane).

So now you are going backwards, and implementing ext2-within-ext3?

Are you ready to admit, yet, that ext3 is 100% mutable in the minds of 
ext3 developers?  Why not implement the minix filesystem format within 
ext3, at this point?  We could call it a "plugin", I bet.


>> Overall, after applying extent (and 48bit) patches, I think it is wrong 
>> to keep calling it ext3.  That will break some existing user 
>> assumptions, and continue to restrict developers' freedom to implement 
>> nifty new features.
> 
> Just FYI, all of the ext3 developers are on board with this patch series
> and it has been discussed and reviewed for many weeks already, it isn't
> just being pushed by one party.

That is completely irrelevant to this thread.

If all the ext3 developers are on board, that just implies that there is 
no clear definition of what "ext3" really means.  With this patch 
series, and with future plans described here and elsewhere, the name 
"ext3" will become more and more meaningless.  It could mean _any_ of 
several filesystem metadata variants, and the admin will have no clue 
which variant they are talking to until they try to mount the blkdev 
(and possibly fail the mount).

At SOME point, clueful developers will say "we should better concentrate 
our energy on a new filesystem."

But I see no one at all defining that "some point."

At some point you are beating a dead horse.  At some point, you are 
pushing features into a filesystem that was never designed to support 
said features.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Andreas Dilger]

On Jun 09, 2006  10:13 +0100, Christoph Hellwig wrote:
> the block numbers are't the big problem concerning scalability, there's
> a lot more to it, like btree(-like) structures in the allocator, parallel
> alloocator algorithms and a better allocation group concept.

All of the allocator changes are already written and well tested, and gave
ext3 a 30% performance improvement while at the same time reducing CPU
usage by 50% - not trivial.  See Holger Kiehl's post
http://marc.theaimsgroup.com/?l=linux-kernel&m=114958967600822&w=4

> If you guys want big storage on linux please help improving the filesystems
> design for that, e.g. jfs or xfs instead of showhorning it onto ext3 thus
> both making ext3 less reliable for us desktop/small server users and not get
> the full thing for the big storage people either.

XFS = 108844 lines, and a complete mess to understand.
ext3+jbd = 27749 lines (includes ~6000 lines extent/allocation changes)

Also, ext3 is just much more robust in the face of on-disk corruption
than xfs or jfs because of its "static" layout, and e2fsck is way
better than the alternatives.  Despite their "big storage" designs ext3
is still competitive in performance, especially with the allocation
improvements.  See also:

http://marc.theaimsgroup.com/?l=ext2-devel&m=108194477207334&w=4
http://marc.theaimsgroup.com/?l=linux-fsdevel&m=110112879929869&w=4
	http://samba.org/~tridge/xattr_results/xfs-ext3-tuning.png

If the XFS or JFS maintainers want to fix their filesystems, they are free
to do so, the ext3 maintainers (all of them, btw) want these changes.


The extent code (prior to some minor cleanups for landing on the vanilla
kernel) has already seen many millions of hours of testing in very heavy
IO environments, so it isn't something that was just written.  If extents
aren't enabled it amounts to a couple of extra conditionals in the
allocation path and basically no modifications to the existing code, so
you can safely avoid this code if you feel the need to.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: [RFC 0/13] extents and 48bit ext3

[Andrew Morton]

On Fri, 9 Jun 2006 10:13:27 +0100
Christoph Hellwig <hch@infradead.org> wrote:

> On Thu, Jun 08, 2006 at 06:20:54PM -0700, Mingming Cao wrote:
> > Current ext3 filesystem is limited to 8TB(4k block size), this is
> > practically not enough for the increasing need of bigger storage as
> > disks in a few years (or even now).
> > 
> > To address this need, there are co-effort from RedHat, ClusterFS, IBM
> > and BULL to move ext3 from 32 bit filesystem to 48 bit filesystem,
> > expanding ext3 filesystem limit from 8TB today to 1024 PB. The 48 bit
> > ext3 is build on top of extent map changes for ext3, originally from
> > Alex Tomas. In short, the new ext3 on-disk extents format is:
> 
> What a horrible idea!  The nice things about ext3 are:
> 
>  - the rather simple and thus reliable implementation

JBD isn't simple.  I don't think there's a need in this project to make
algorithmic changes in either JBD or htree, thankfully.

>  - the lack of incompatible ondisk changes

Ted&co have been pretty good at avoiding compatibility problems.

> and the block numbers are't the big problem concerning scalability, there's
> a lot more to it, like btree(-like) structures in the allocator, parallel
> alloocator algorithms and a better allocation group concept.

The performance testing results I've seen for a few of the components of
this project have been rather good, and that's the bottom line.

I don't know how the end result would compare in a bakeoff against XFS, and
I doubt if we know how much XFS performance would be improved if this
effort were diverted into that project.

But I don't think it's all as clear-cut as you imply.

> If you guys want big storage on linux please help improving the filesystems
> design for that, e.g. jfs or xfs instead of showhorning it onto ext3 thus
> both making ext3 less reliable for us desktop/small server users and not get
> the full thing for the big storage people either.

There have been pretty big changes in ext3 post-2.6.early and we've been OK
at avoiding breakage thus far.  It all comes down to how well the new
codepaths manage to avoid altering the existing ones.

That being said, ext3 isn't exactly ....  modern.  One day we'll need
something better.

Re: [RFC 0/13] extents and 48bit ext3

[Christoph Hellwig]

On Thu, Jun 08, 2006 at 06:20:54PM -0700, Mingming Cao wrote:
> Current ext3 filesystem is limited to 8TB(4k block size), this is
> practically not enough for the increasing need of bigger storage as
> disks in a few years (or even now).
> 
> To address this need, there are co-effort from RedHat, ClusterFS, IBM
> and BULL to move ext3 from 32 bit filesystem to 48 bit filesystem,
> expanding ext3 filesystem limit from 8TB today to 1024 PB. The 48 bit
> ext3 is build on top of extent map changes for ext3, originally from
> Alex Tomas. In short, the new ext3 on-disk extents format is:

What a horrible idea!  The nice things about ext3 are:

 - the rather simple and thus reliable implementation
 - the lack of incompatible ondisk changes

and the block numbers are't the big problem concerning scalability, there's
a lot more to it, like btree(-like) structures in the allocator, parallel
alloocator algorithms and a better allocation group concept.

If you guys want big storage on linux please help improving the filesystems
design for that, e.g. jfs or xfs instead of showhorning it onto ext3 thus
both making ext3 less reliable for us desktop/small server users and not get
the full thing for the big storage people either.

Re: [RFC 0/13] extents and 48bit ext3

[Andreas Dilger]

On Jun 08, 2006  22:49 -0400, Jeff Garzik wrote:
> One of my common complaints about massive ext3 updates such as this is 
> the ever-growing "which ext3 filesystem am I mounting?" problem.
> 
> I really think extents and 48bit-ness should imply
> 	cp -a fs/ext3 fs/ext4
> and go from there.

The problem with this approach (as seen with ext2 and ext3) is that one
tree or the other gets stale w.r.t. bug fixes and now we have the case
where ext2 has a noticably different implementation in some areas and
bug fixes are no longer trivial to apply to both trees.

I think all of the ext3 maintainers think this split was a bad idea in
hindsight, and having an ext3 mode where it can mount without a journal
would be much more desirable.

> IMHO the ext3 back-compat situation is already really hairy, with all 
> the features added since the original ext3 release.

While partially true, ext2/ext3 has a very good history w.r.t. compatibility
(with one exception being the EAs on symlinks problem that slipped through
with selinux).

Yes, the extents format will be incompatible with older ext3, but it isn't
enabled by default so it will be completely up to the sysadmin when they
make their filesystem incompatible.  They also won't impact any existing
files.  The earlier extents support gets into a kernel.org kernel the
more systems will be able to mount a filesystem with the changes when
they becomes widely used.

All of the other features that are going to be introduced will only going
to be applicable for format time (filesystems larger than 16TB), or if
exceeding limits of the current ext3 support (e.g. files larger than 2TB
in size).

> People (including me) still switch back and forth between ext2 and ext3 
> mounts of the same filesystem on occasion.  I think creating an "ext4" 
> would allow for greater developer flexibility in implementing new 
> features and ditching old ones -- while also emphasizing to the user 
> that switching back and forth between ext4 and ext[23] is a bad idea.

While this is partly true, one of the big benefits is that you can
transparently upgrade your system to use the new features and improve
performance without a long outage window.  Having a completely separate
ext4 filesystem doesn't improve the compatibility story at all.  There
has been renewed discussion on implementing "mounting ext3 without a
journal", just for a recovery mode, because ext2 will not be modified
to get all of these features (running e2fsck on a huge filesystem each
reboot would be insane).


> Overall, after applying extent (and 48bit) patches, I think it is wrong 
> to keep calling it ext3.  That will break some existing user 
> assumptions, and continue to restrict developers' freedom to implement 
> nifty new features.

Just FYI, all of the ext3 developers are on board with this patch series
and it has been discussed and reviewed for many weeks already, it isn't
just being pushed by one party.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: [RFC 0/13] extents and 48bit ext3

[Andreas Dilger]

On Jun 08, 2006  22:40 -0400, Valdis.Kletnieks@vt.edu wrote:
> On Thu, 08 Jun 2006 18:20:54 PDT, Mingming Cao said:
> > To address this need, there are co-effort from RedHat, ClusterFS, IBM
> > and BULL to move ext3 from 32 bit filesystem to 48 bit filesystem,
> > expanding ext3 filesystem limit from 8TB today to 1024 PB. The 48 bit
> > ext3 is build on top of extent map changes for ext3, originally from
> > Alex Tomas. In short, the new ext3 on-disk extents format is:
> 
> which implies matching changes to mkfs.ext2 and possibly mount..

The extents format doesn't need any support from mke2fs.  Currently this
is activated by a mount option "-o extents", so it won't be used until
a system administrator actively enables it.

> > Appreciate any comments and feedbacks!
> 
> Somebody else was recently discussing a set of patches to ext3 for
> extents+delalloc+mballoc patches - is this work compatible with that?

Yes, completely compatible (author is the same person).  We have all been
working to get these improvements into the vanilla kernel so that everyone
can benefit from the improved performance.  These patches are just the
start - the mballoc and delalloc patches are follow-on patches, but they
do not affect the on-disk format just the in-memory implementation of
block allocation.

> Also, a pointer to the matching userspace patches would help anybody
> who's gung-ho enough to test the code....

They were posted to the ext2-devel mailing list previously, or you can
download a patched RPM at ftp://ftp.lustre.org/pub/lustre/other/e2fsprogs/
(the extent support is making its way into the official e2fsprogs also).

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: [RFC 0/13] extents and 48bit ext3

[Jeff Garzik]

Mingming Cao wrote:
> Current ext3 filesystem is limited to 8TB(4k block size), this is
> practically not enough for the increasing need of bigger storage as
> disks in a few years (or even now).
> 
> To address this need, there are co-effort from RedHat, ClusterFS, IBM
> and BULL to move ext3 from 32 bit filesystem to 48 bit filesystem,
> expanding ext3 filesystem limit from 8TB today to 1024 PB. The 48 bit
> ext3 is build on top of extent map changes for ext3, originally from
> Alex Tomas. In short, the new ext3 on-disk extents format is:

One of my common complaints about massive ext3 updates such as this is 
the ever-growing "which ext3 filesystem am I mounting?" problem.

I really think extents and 48bit-ness should imply
	cp -a fs/ext3 fs/ext4
and go from there.

IMHO the ext3 back-compat situation is already really hairy, with all 
the features added since the original ext3 release.

The alternative is continual bloating of ext3, and on filesystems, 
inodes which are progressively upgraded -- meaning any use of a prior 
kernel implies that you can only read a subset of your [meta]data, if 
the back-compat code doesn't block the mount entirely.

People (including me) still switch back and forth between ext2 and ext3 
mounts of the same filesystem on occasion.  I think creating an "ext4" 
would allow for greater developer flexibility in implementing new 
features and ditching old ones -- while also emphasizing to the user 
that switching back and forth between ext4 and ext[23] is a bad idea.

Overall, after applying extent (and 48bit) patches, I think it is wrong 
to keep calling it ext3.  That will break some existing user 
assumptions, and continue to restrict developers' freedom to implement 
nifty new features.

	Jeff

Re: [RFC 0/13] extents and 48bit ext3

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

On Thu, 08 Jun 2006 18:20:54 PDT, Mingming Cao said:
> Current ext3 filesystem is limited to 8TB(4k block size), this is
> practically not enough for the increasing need of bigger storage as
> disks in a few years (or even now).
> 
> To address this need, there are co-effort from RedHat, ClusterFS, IBM
> and BULL to move ext3 from 32 bit filesystem to 48 bit filesystem,
> expanding ext3 filesystem limit from 8TB today to 1024 PB. The 48 bit
> ext3 is build on top of extent map changes for ext3, originally from
> Alex Tomas. In short, the new ext3 on-disk extents format is:

which implies matching changes to mkfs.ext2 and possibly mount..

> Appreciate any comments and feedbacks!

Somebody else was recently discussing a set of patches to ext3 for
extents+delalloc+mballoc patches - is this work compatible with that?

Also, a pointer to the matching userspace patches would help anybody
who's gung-ho enough to test the code....


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

[RFC 0/13] extents and 48bit ext3

[Mingming Cao]

Current ext3 filesystem is limited to 8TB(4k block size), this is
practically not enough for the increasing need of bigger storage as
disks in a few years (or even now).

To address this need, there are co-effort from RedHat, ClusterFS, IBM
and BULL to move ext3 from 32 bit filesystem to 48 bit filesystem,
expanding ext3 filesystem limit from 8TB today to 1024 PB. The 48 bit
ext3 is build on top of extent map changes for ext3, originally from
Alex Tomas. In short, the new ext3 on-disk extents format is:

On disk extents format:
/*
  * this is extent on-disk structure
  * it's used at the bottom of the tree
  */
struct ext3_extent {
        __le32  ee_block;       /* first logical block extent covers */
        __le16  ee_len;         /* number of blocks covered by extent */
        __le16  ee_start_hi;    /* high 16 bits of physical block */
        __le32  ee_start;       /* low 32 bigs of physical block */
};

A series of patches have been posted to ext2-devel list in last month
and have been reviewed.  This is updated full series of patches to
support 48 bit ext3 based on extent map. Patches are against 2.6.17-rc6
kernel, and could be found at
http://ext2.sourceforge.net/48bitext3/patches/patches-2.6.17-
rc6-06082006/

[patch 1/13] percpu_counter_longlong.patch
percpu count data type changes to support 64 bit ext3 free blocks count

[patch 2/13] ext3_check_sector_t_overflow.patch
sector_t overflow check for 32bit/48bit ext3 at mount/resize time

[patch 3/13] ext3_fsblk_t_fixes.patch
Define ext3 filesystem and group block types (ext3_fsblk_t,
ext3_grpblk_t, and fix in-kernel ext3 block types (from int type to
ext3_fsblk_t) to support 32bit ext3.

[patch 4/13] ext3_convert_blks_to_fsblk_t.patch
convert the rest of ext3 filesystem blocks to ext3_fsblk_t

patches 1-4 are currently in mm tree

[patch 5/13] sector_fmt.patch
sector_t type format string for all arch.

[patch 6/13] ext3_fsblk_sector_t.patch
support >32bit bit fs block type in kernel (convert ext3_fsblk_t to
sector_t)

[patch 7/13] 64bit_jbd_core.patch
Core 64 bit JBD changes

[patch 8/13] sector_t-jbd.patch
JBD layer in-kernel block variables type fixes to support >32
bit block number and convert to sector_t type.

#extent map patches
[patch 9/13] ext3-extents.patch
core extent map support

[patch 10/13] ext3-extents-48bit.patch
Add full 48 bit physical block support based on extents.

[patch 11/13] ext3-extents-ext3_fsblk_t.patch
convert block types in extents to ext3_fsblk_t

[patch 12/13]ext3_48bit_i_file_acl.pat
48 bit on-disk i_file_acl to support xttar for 48 bit ext3

[patch 13/13] 64bit-metadata
On-disk and in-kernel super block changes to support >32
bit free blocks numbers.


Appreciate any comments and feedbacks!

Mingming