Re: File System Performance

[Richard Gooch <rgooch@ras.ucalgary.ca>]

Andrew Morton writes:
> Ben Israel wrote:
> > 
> > Andrew
> > 
> >  Thank you very much for your response. I would like to know what ever I can
> > about File Systems that achieve near Raw Disk Transfer Speeds on large file
> > system modifications.
> 
> This is very filesystem-dependent.   Probably the XFS mailing list
> linux-xfs@oss.sgi.com is the place to ask for XFS.
> 
> > What does the "Orlov allocator" do differently?
> 
> The problem which ext2 experiences with your particular workload
> is that it attempts to balance the creation of new directories
> across the filesystem's various "block groups".  There are tens
> or hundreds of these.  ext2 also tries to place files in the same
> block group as their parent directory.
> 
> Consequently, when you untar (or just copy) a directory tree,
> it gets placed all over the partition.  So both reading it and
> writing it requires a lot of seeking.
> 
> That's a fast-growth workload.  A slow-growth workload is
> the normal sort of day-to-day activity where you reqularly
> create and delete files.  The occupancy of the fs grows slowly.
> 
> We are seeing some degradation in performance in the slow-growth
> case with the Orlov algorithm.  But fast-growth is increased a lot.
> 
> The Orlov algorithm (or at least Al's version of it) tries to
> spread top-level directories evenly across the partition, under
> the assumption that these contain unrelated information.  But for
> directories at a lower level, it is more aggressive about placing
> directories in the same block group as their parent.   It _will_
> move into a different block group, but only when it sees that the
> current one is filling up.

Here's an idea: add a "--compact" option to tar, so that it creates
*all* inodes (files and directories alike) in the base directory, and
then renames newly created entries to shuffle them into their correct
positions. That should limit the number of block groups that are used,
right?

It would probably also be a good idea to do that for cp as well, so
that when I do a "cp -al" of a virgin kernel tree, I can keep all the
directory inodes together. It will make a cold diff even faster.

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca