Q: sparse file creation in existing data?

Q: sparse file creation in existing data?

[Ph. Marek]

Hi everybody,

though looking and grepping through the sources I couldn't find a way (via
fcntl() or whatever) to allow an existing file to get holes.

I found that cp has a parameter --sparse (or suchlike) - but strace shows
it doing a open(,O_TRUNC) which has a bit of impact on previous filedata :-)


What I'd like to do is something like

  fh=open( ... , O_RDWR);
  lseek(fh, position ,SEEK_START); 
// where position is a multiple of fs block size
  fcntl(fh,F_MAKESPARSE,16384);

to create a 16kB hole in a file.
If the underlying fs doesn't support holes, I'd get ENOSYS or something.


What I'd like to use that for:

I imagine having a file on hd (eg. tar) and not enough space to decompress.
So with SOME space at least I'd open the file and stream it's data to tar,
after each few kB read I'd free some space - so this could eventually succeed.

I also thought about simple reversing the filedata - so I'd read off the
end of the file and truncate() downwards - but that would mean reversing
the whole file which could take some time on creation and would solve only
this specific case.


Ideas, anyone?


Regards,

Phil

Re: Q: sparse file creation in existing data?

[Dr. Michael Weller]

On Fri, 29 Jun 2001, Ph. Marek wrote:

> Hi everybody,
> 
> though looking and grepping through the sources I couldn't find a way (via
> fcntl() or whatever) to allow an existing file to get holes.

Indeed, I don't think there is any such syscall.

> I found that cp has a parameter --sparse (or suchlike) - but strace shows
> it doing a open(,O_TRUNC) which has a bit of impact on previous filedata :-)

Holes are not made by specific syscalls, just by not writing data to
certain places. In principle, the write syscall could check if it just
writes zeroes and create a hole instead. Of course,
this check would be much slower than just writing the data to disk.
Probably a 'write_zeroes' syscall to create a hole (if possible) would be
better.

While I see your purpose and the advantage, this is a pretty specific
question and doesn't look like std. unix semantics. I doubt you'll get
people to add such a syscall (maybe if you implement it yourself, people
will accept it for addition, but I doubt even that).

For your specific problem I'd suggest the following approach:

write a new filter prog, kind of a 'destructive cat' command.

Open the file for read-modify (non destructive)
Let it read some blocks (number controllable on commandline) from the
beginning and pipe them to stdout. Then.. read in same amount (well;
block aligned) from the end of file and copy it over the beginning,
truncate the file accordingly. After some time reading and truncating
will meet in the middle of the file, then you read the blocks back in
on reverse (and truncate them after reading)

It shouldn't be too difficult to write and result in a multipurpose
commandline utility.

It does some disk i/o but I don't think it will too bad due to disk
buffers of the kernel and read-ahead. Theoretically, the kernel, buffers
etc. could detect you are just moving file blocks to different places and
does this in a 'zero-copy' fashion by just moving some inode entries
around, but I doubt that it is soo smart.

Drawback: If you have choosen the read block size too large (hmm.. you
might want to read some data from beginning, some from the end, copy over
beginning, then truncate, and only then pipe to stdout: this should
eliminate this 'buffer/diskspace overrun')  of 'destructive cat' too big,
or the archive is compressed and doesn't fit uncompressed, the backup was
destructed. This is by far not as safe as any backup tool should be.

A major advantage, however, is that this tool will run w/o a kernel patch,
on any Unix on any filesystem, even those w/o holes (I think truncate
works in general, maybe not samba?) 

Just my 0.02$ (you asked for ideas, didn't you?)

Michael.

--

Michael Weller: eowmob@exp-math.uni-essen.de, eowmob@ms.exp-math.uni-essen.de,
or even mat42b@spi.power.uni-essen.de. If you encounter an eowmob account on
any machine in the net, it's very likely it's me.

Re: Q: sparse file creation in existing data?

[Ph. Marek]

>For your specific problem I'd suggest the following approach:
>
>write a new filter prog, kind of a 'destructive cat' command.
>
>Open the file for read-modify (non destructive)
>Let it read some blocks (number controllable on commandline) from the
>beginning and pipe them to stdout. Then.. read in same amount (well;
>block aligned) from the end of file and copy it over the beginning,
>truncate the file accordingly. After some time reading and truncating
>will meet in the middle of the file, then you read the blocks back in
>on reverse (and truncate them after reading)
>
>It shouldn't be too difficult to write and result in a multipurpose
>commandline utility.
>
>It does some disk i/o but I don't think it will too bad due to disk
>buffers of the kernel and read-ahead. Theoretically, the kernel, buffers
>etc. could detect you are just moving file blocks to different places and
>does this in a 'zero-copy' fashion by just moving some inode entries
>around, but I doubt that it is soo smart.
>
>Drawback: If you have choosen the read block size too large (hmm.. you
>might want to read some data from beginning, some from the end, copy over
>beginning, then truncate, and only then pipe to stdout: this should
>eliminate this 'buffer/diskspace overrun')  of 'destructive cat' too big,
>or the archive is compressed and doesn't fit uncompressed, the backup was
>destructed. This is by far not as safe as any backup tool should be.
>
>A major advantage, however, is that this tool will run w/o a kernel patch,
>on any Unix on any filesystem, even those w/o holes (I think truncate
>works in general, maybe not samba?) 
I didn't even get this idea - it sound's very io-intensive. 
Of course, if you look at it from the right angle - you'd
- read the first halve,
- read the 2nd halve,
- and write the 2nd halve to the 1st,
- and then read the 1st again.

That's two times the amount of io.
If I assume using tar it's because of writing the data only one third more
io than necessary.

Hmmm, on second thought ... But I'd like it better to have a fcntl for
hole-making :-)
Maybe I'll implement this myself.


Thanks,

Phil

Re: Q: sparse file creation in existing data?

[Daniel Phillips]

On Friday 29 June 2001 14:55, Ph. Marek wrote:
> Hmmm, on second thought ... But I'd like it better to have a fcntl for
> hole-making :-)
> Maybe I'll implement this myself.

A far superior interface would be:

    ssize_t sys_clear(unsigned int fd, size_t count)

A stub implementation would just write zeroes.  You would need a generic way 
of determining whether holes are supported for a particular file - this is 
where an fcntl would be appropriate.  It would also be nice to know this 
before opening/creating a file, perhaps by fcntling the directory.

But don't expect to have a real, hole-creating implementation any time soon.  
Taming the truncate races is hard enough as it is with a single boundary at 
the end of a file.  Taking care of multiple boundaries inside the file is 
far, far harder.  Talk to Al Viro or the Ext3 team if you want the whole ugly 
story.

--
Daniel

Re: Q: sparse file creation in existing data?

[Andreas Dilger]

Phil writes:
> though looking and grepping through the sources I couldn't find a way (via
> fcntl() or whatever) to allow an existing file to get holes.
> 
> What I'd like to do is something like
> 
>   fh=open( ... , O_RDWR);
>   lseek(fh, position ,SEEK_START); 
> // where position is a multiple of fs block size
>   fcntl(fh,F_MAKESPARSE,16384);
> 
> to create a 16kB hole in a file.
> If the underlying fs doesn't support holes, I'd get ENOSYS or something.

Peter Braam <braam@clusterfilesystem.com> implemented such a syscall, and
support for it in ext2, in the 2.2 kernel.  It is called "sys_punch"
(punching holes in a file).  I'm not sure how applicable the patch is
in the 2.4 world (probably not at all, unfortunately).  I did a port of
this code to 2.2 ext3 a long time ago, but have not kept it updated.
I'm not sure it was 100% race free (Al would probably say not), but it
worked well enough while I was using it.

The basic premise is that it will write zero's to partial blocks at the
beginning and end of the punch region, and make holes of any intervening
blocks.  It did NOT do checks to see if a partial block was entirely
zero filled and make it a hole (although that would be a possible feature).

It could be used as a replacement for the truncate code, because then
truncate is simply a special case of punch, namely punch(0, end).

> What I'd like to use that for:
> 
> I imagine having a file on hd (eg. tar) and not enough space to decompress.
> So with SOME space at least I'd open the file and stream it's data to tar,
> after each few kB read I'd free some space - so this could eventually succeed.
> 
> I also thought about simple reversing the filedata - so I'd read off the
> end of the file and truncate() downwards - but that would mean reversing
> the whole file which could take some time on creation and would solve only
> this specific case.

I'm not sure I would agree with your application, but I do agree that there
are some uses for it.  In Peter's case he used it for implementing a
cacheing HFS storage system, but he also wanted to use it for InterMezzo
(a cacheing network filesystem) to do several things:
- delete entries from a transaction log, in a way that actually reduces
  the physical size of the log.  The log itself could be always appended
  to the end (for 64-bit file size), but transactions/blocks would be
  punched from the beginning.
- delete blocks from the beginning/middle of large files cached on the client.
  This is useful if the file size is too large to fit into the cache, or if
  you are doing some sort of LRU replacement of blocks in the cache.

I don't think any of this has been implemented in InterMezzo yet.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: Q: sparse file creation in existing data?

[Pavel Machek]

Hi!

> It could be used as a replacement for the truncate code, because then
> truncate is simply a special case of punch, namely punch(0, end).

I do not think so. Truncate leaves you with filesize 0, while punch schould
leave you with filesize of original file.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: Q: sparse file creation in existing data?

[Andreas Dilger]

Pavel writes:
> > It could be used as a replacement for the truncate code, because then
> > truncate is simply a special case of punch, namely punch(0, end).
> 
> I do not think so. Truncate leaves you with filesize 0, while punch schould
> leave you with filesize of original file.

I can't recall the exact details (it's been a year since I looked at the
code), but I believe that the VFS sets the file size before the call to
truncate, so it didn't matter on the exact semantics of the punch call.
Truncate still existed, but it was just a wrapper calling the punch code
to punch out all of the existing blocks.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert