Re: ext4 features

Re: ext4 features

[Bodo Eggert]

Petr Tesarik <ptesarik@suse.cz> wrote:

> Salvaging files would be done with a separate tool. Of course, if you
> delete more files with the same name in the same directory, you'd need
> to tell that tool which one of them you want to salvage. Yes, I really
> mean you'd have more than one deleted file with the same name in the
> directory.

> Anyway, I doubt we want such feature for ext4, because to make things
> efficient, you'd need to provide some kind of pointer from the deleted
> (but not yet purged) blocks to the corresponding file. Hard links are
> also problematic and there is a whole lot of other troubles I haven't
> even thought of.

You can add the original directory inode of the deleted file to an attribute.
The name can be stored in the same way, so you can have multiple files with
the same name in the trashcan directory. (Heuristics for the in-trashcan-name
can be e.g. the original inode number or "n-$ORIGINALNAME" if it's unique,
but just using the number will be good enough.)

Using the directory inode you can follow the '..'-chain to the root in order
to present a nice 'original' name, but you should also be able to undelete
files into any directory (even chroots) if both the trashcan and the target
directory are visible and you've got apropiate permissions.

BTW: If you're using the trashcan on linked files, you will need to store
multiple sources for an inode.
-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

http://david.woodhou.se/why-not-spf.html

Re: ext4 features

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

On Tue, 04 Jul 2006 14:28:05 +0200, Bodo Eggert said:
> BTW: If you're using the trashcan on linked files, you will need to store
> multiple sources for an inode.

No, if it's a multiply linked file, you want all unlinks except the last
to operate the way they always have, and only the LAST link moves to the
trashcan.

Otherwise, if you have a file with two links, and one moves to the trashcan,
it's then subject to reaping and overwriting - which will come as a great
surprise to the still-existing link....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: ext4 features (salvage)

[Bodo Eggert]

Matthew Frost <artusemrys@sbcglobal.net> wrote:

> We silently keep files around in many filesystems, at least until
> whatever reclamation process runs.  The delete event doesn't itself
> generally purge the data from disk.  However, this is a matter of simple
> tools doing simple things.  Designing an intentional structure around
> not actually deleting deleted files, but keeping them around just in
> case may be lauded as "user-friendly", but it is counter-intuitive.  It
> is cleverness over clarity, good design smothered under feature demand.

[...]

> If you have to add a "really delete, I mean it" command, you're breaking
> fundamental assumptions.

Even without the patch, you can't guarantee that nobody will create a link
just before the supposed-to-be-final unlink(), or copies the file around.
Having undelete() will just make this risk very obvious.

BTW: If you really want to delete files, use shred.


[...]
> Why add non-free space to the free space count, when we're intentionally
> keeping those files?  If you have to be counter-intuitive, why go the
> second counter of hiding it from the user who "wants us to keep and
> index his deleted files"?

It is free space, but it can be unfreed in order to
a) restore the data (on demand)
b) store new data (automatically, no tool needed)

If b happens before a, it's called bad luck.

This concept is fundamentally different from keeping a recycle bin
of 10 % disksize for each of the 20 users (adding up to 200 % of the
disk size), which is well-known from a colorfull "operating" system.
-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

http://david.woodhou.se/why-not-spf.html

Re: ext4 features

[Bodo Eggert]

Lew Palm <lew@tzi.de> wrote:
> Jeffrey V. Merkey wrote:

>> The old novell model is simple. When someone unlinks a file, don't
>> delete it, just mv it to another special directory called DELETED.SAV.
>> Then setup the
>> fs space allocation to reuse these files when the drive fills up by
>> oldest files first. It's very simple. Then you have a salvagable file
>> system.
> 
> A complete foolproof car is a car with a maximum speed of 0 mph.
> As a user I give commands to my computer, for example an order to delete a
> file. And this is what I expect it to do.

You don't delete a file but a filename, and that's what your system will
still do. 

> If I want it to move a file to another position in the filesystem, I would
> use another command. I don't want my operating system to josh me, that's why
> I use Linux.
> Stealthy keeping of deleted files somewhere is a security black hole.

Depending on unlinked file to be inaccessible and never have been copied
just because you called unlink is the real security hole.

> But accidents happen. Hardware perishes, users are making mistakes, sometimes
> coffee is pouring...
> That's why we backup important data regulary.

And the salvaging fs will do exactly this whenever you unlink() the final
reference. You could also use a userspace library catching each unlink call,
but it would also have to intercept each write() call for each user and
try to reclaim the backup copies on disk-full and would-have-to-fragment
events. Off cause there are no userspace-visible would-have-to-fragment
events, so besides being ugly a userspace solution would not be able to
completely provide the same service.

> A not-really-deleting-filesystem wouldn't relieve us of that duty, but would
> make a system more insecure and ambiguous.

It's just a marginal shift. If you can't trust yourself, you've lost. If
you can't trust the current root, you're screwed, too. If you can't trust
a future root, the time window in which the file can be recovered will
slightly increase and the needed knowledge will be reduced. Otherwise, there
is no change.
-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

http://david.woodhou.se/why-not-spf.html

Re: ext4 features

[Bodo Eggert]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> On Wed, 2006-07-05 at 22:32 -0400, Bill Davidsen wrote:

>> But with timestamps I need remember only one number, the time of my last
>> backup. Skipping over the question of "who's idea of time" inherent in
>> network filesystems. I compare all ctimes with the time of the last
>> backup and do incremental on the newer ones. If we use versioning I have
>> to remember the version for each file! In practice I really question if
>> the benefit justified keeping all that metadata between backups. And if
>> I delete a file and create another by the same name, what is it's version?
> 
> You are completely missing the point. Our background is that all NFS
> clients are required to use the mtime and ctime timestamps in order to
> figure out if their cached data is valid. They need to do this extremely
> frequently (in fact, every time you open() the file).

If the changes to these files are very infrequent compared to nanoseconds,
you'll only need the version during some nanoseconds, and only during
runtime. Having a second-change-within-one-timeframe-flag(*) instead of
versions will be enough to make NFS mostly happy and only penalize your
users for one nanosecond, and it won't force version-keeping into the
filesystem. And besides that, all other filesystems will profit even
without having nanosecond resolution nor versioning (but they'll suffer
for up to a whole second).


*) TODO: Create a nice name
-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

http://david.woodhou.se/why-not-spf.html

Re: ext4 features

[Trond Myklebust]

On Fri, 2006-07-07 at 11:38 +0200, Bodo Eggert wrote:

> If the changes to these files are very infrequent compared to nanoseconds,
> you'll only need the version during some nanoseconds, and only during
> runtime. Having a second-change-within-one-timeframe-flag(*) instead of
> versions will be enough to make NFS mostly happy and only penalize your
> users for one nanosecond, and it won't force version-keeping into the
> filesystem. And besides that, all other filesystems will profit even
> without having nanosecond resolution nor versioning (but they'll suffer
> for up to a whole second).

NFS never required file versioning. You're talking to the wrong person.
It does, however, need a change attribute that logs all changes to the
file. The above flag does not suffice to provide that.

Trond

Re: ext4 features

[Bodo Eggert]

On Fri, 7 Jul 2006, Trond Myklebust wrote:
> On Fri, 2006-07-07 at 11:38 +0200, Bodo Eggert wrote:

> > If the changes to these files are very infrequent compared to nanoseconds,
> > you'll only need the version during some nanoseconds, and only during
> > runtime. Having a second-change-within-one-timeframe-flag(*) instead of
> > versions will be enough to make NFS mostly happy and only penalize your
> > users for one nanosecond, and it won't force version-keeping into the
> > filesystem. And besides that, all other filesystems will profit even
> > without having nanosecond resolution nor versioning (but they'll suffer
> > for up to a whole second).
> 
> NFS never required file versioning. You're talking to the wrong person.
> It does, however, need a change attribute that logs all changes to the
> file. The above flag does not suffice to provide that.

ACK, that's why I'm suggesting this doublechange-flag.
-- 
In the beginning, God created the earth and rested.
Then God created Man and rested.
Then God created Woman.
Since then, neither God nor Man has rested.

Re: ext4 features (salvage)

[Matthew Frost]

(Stupid mailer + user error = not sent to list)

Petr Tesarik wrote:
 > On Tue, 2006-07-04 at 13:35 +0200, Peter Zijlstra wrote:
 >> On Tue, 2006-07-04 at 11:22 +0200, Petr Tesarik wrote:
 >>> Yes and no. A simple mv is better done in userspace, but what I'd
 >>> _really_ appreciate would be a true kernel salvage (similar to the way
 >>> NetWare does things). That means marking the file as deleted in the
 >>> directory, marking its blocks as deleted but avoiding the use of those
 >>> blocks. The kernel would then prefer allocating new blocks from
 >>> elsewhere but once the filesystem runs out of space, it would start
 >>> allocating from the deleted files area and marking the blocks as 
well as
 >>> the corresponding files purged.
 >>>
 >>> Salvaging files would be done with a separate tool. Of course, if you
 >>> delete more files with the same name in the same directory, you'd need
 >>> to tell that tool which one of them you want to salvage. Yes, I really
 >>> mean you'd have more than one deleted file with the same name in the
 >>> directory.
 >> Wouldn't such a scheme interfere with the block allocator algorithms,
 >> and hence increase the risk of fragmentation? Schemes like this realy
 >> put my hairs on end,
 >
 > Yes, they would interfere. That's why I'm not proposing to add them to
 > ext4 in the first place.
 >
 >>   1) if you don't want to lose your data, make backups;
 >
 > Generally, I agree.
 >
 >>   2) if I mean to delete a file, I want it gone proper. Silently keeping
 >>      it about is not unix like;
 >
 > Yes, this is a problem. Although you would of course have a tool for
 > purging the files unconditionally, some programs may need the assumption
 > that an unlinked file is gone forever.
 >
 > Regarding the second clause, well, Linux is not Unix-like in many
 > respects and we want it like that. That's a weak argument.

We silently keep files around in many filesystems, at least until
whatever reclamation process runs.  The delete event doesn't itself
generally purge the data from disk.  However, this is a matter of simple
tools doing simple things.  Designing an intentional structure around
not actually deleting deleted files, but keeping them around just in
case may be lauded as "user-friendly", but it is counter-intuitive.  It
is cleverness over clarity, good design smothered under feature demand.

In the ways in which it counts, in the sensible, useful, elegantly
simple ways, the "Do one thing and do it well" ways, Linux tries to be
Unix-like.  We want stupid programs.  A filesystem that decides that it
knows better than the user is not desirable.  Filesystem programmers
that decide that they know better than the user are likewise sub-optimal.

Protect my data against accidental failure.  Do not protect it against me.

If you have to add a "really delete, I mean it" command, you're breaking
fundamental assumptions.

 >
 >>   3) don't aid third parties in recovering your removed data. If I want
 >>      them to have it I'll give it to them.
 >
 > See 2. Explicit purging is of course possible. (Novell Netware also had
 > a "purge" command.)
 >
 > Anyway, it seems that there is some functionality which many users want
 > but which can't be provided in user space:
 >
 >   - if files are moved to the recycle-bin-or-whatever-you-call-it, their
 > size is added to disk free space and

Why add non-free space to the free space count, when we're intentionally
keeping those files?  If you have to be counter-intuitive, why go the
second counter of hiding it from the user who "wants us to keep and
index his deleted files"?

 >   - automatically purging least recently deleted files.
 >
 > Regards,
 > Petr Tesarik

Matt

Re: ext4 features (salvage)

[Helge Hafting]

On Tue, Jul 04, 2006 at 02:31:56PM +0200, Petr Tesarik wrote:
> On Tue, 2006-07-04 at 16:08 +0400, Lex Lyamin wrote:
> > you mean that blocks are naturaly free, but we cant use them because
> > someone may made them free by accident, but we cant use them...
> > 
> > hmm...
> > great idea!
> > 
> > wait, its not.
> > because of we cant use those blocks we cant optimise way we write one
> > disk , and if we have defragmenter we cant  make use of them either.
> > and if (just if) this is online defragmenter, it cant use them too. 
> 
> Well, the way I saw it done was that you had no guarantee that any
> deleted file could be salvaged. Sometimes you even could salvage a file
> but not another one which was deleted later. Users seemed to be content
> with that, because in most situations it did help them restore files
> they deleted and within a few seconds realized that they didn't want to.
> 
> This means that the allocator MAY purge any deleted block at any moment,
> although it tends to allocate blocks from areas of disk which haven't
> been used recently.
> 
> And the benefits? The performance of such a filesystem could be better
> than snapshots, while allowing to cope with one of the most common human
> errors.

The most common error?  A few years ago I restored a file from
backup, because I deleted it in error.  I can't even remember
the second-last time I had that problem.

I'd say this error is among the easiest to avoid. :-)
Even a little performance loss won't justify it for me.

Now, there may be clumsier users than me, but they tend to
be using GUI "file managers" which do implement a "wastebasket"
for all internal deletion.

Helge Hafting

Re: ext4 features (salvage)

[Petr Tesarik]

On Tue, 2006-07-04 at 16:08 +0400, Lex Lyamin wrote:
> you mean that blocks are naturaly free, but we cant use them because
> someone may made them free by accident, but we cant use them...
> 
> hmm...
> great idea!
> 
> wait, its not.
> because of we cant use those blocks we cant optimise way we write one
> disk , and if we have defragmenter we cant  make use of them either.
> and if (just if) this is online defragmenter, it cant use them too. 

Well, the way I saw it done was that you had no guarantee that any
deleted file could be salvaged. Sometimes you even could salvage a file
but not another one which was deleted later. Users seemed to be content
with that, because in most situations it did help them restore files
they deleted and within a few seconds realized that they didn't want to.

This means that the allocator MAY purge any deleted block at any moment,
although it tends to allocate blocks from areas of disk which haven't
been used recently.

And the benefits? The performance of such a filesystem could be better
than snapshots, while allowing to cope with one of the most common human
errors.

Regards,
Petr Tesarik

> for what purpose ?
> are not we trying  play out solution to problem from level 3 on level
> 2 ?
> does the soulion really belong to this level ?
> would people pay with performance for "feature" which *probably* will
> help them to restore their files ? 

Re: ext4 features (salvage)

[Petr Tesarik]

On Tue, 2006-07-04 at 13:35 +0200, Peter Zijlstra wrote:
> On Tue, 2006-07-04 at 11:22 +0200, Petr Tesarik wrote:
> > Yes and no. A simple mv is better done in userspace, but what I'd
> > _really_ appreciate would be a true kernel salvage (similar to the way
> > NetWare does things). That means marking the file as deleted in the
> > directory, marking its blocks as deleted but avoiding the use of those
> > blocks. The kernel would then prefer allocating new blocks from
> > elsewhere but once the filesystem runs out of space, it would start
> > allocating from the deleted files area and marking the blocks as well as
> > the corresponding files purged.
> > 
> > Salvaging files would be done with a separate tool. Of course, if you
> > delete more files with the same name in the same directory, you'd need
> > to tell that tool which one of them you want to salvage. Yes, I really
> > mean you'd have more than one deleted file with the same name in the
> > directory.
> 
> Wouldn't such a scheme interfere with the block allocator algorithms,
> and hence increase the risk of fragmentation? Schemes like this realy
> put my hairs on end,

Yes, they would interfere. That's why I'm not proposing to add them to
ext4 in the first place.

>   1) if you don't want to lose your data, make backups; 

Generally, I agree.

>   2) if I mean to delete a file, I want it gone proper. Silently keeping
>      it about is not unix like;

Yes, this is a problem. Although you would of course have a tool for
purging the files unconditionally, some programs may need the assumption
that an unlinked file is gone forever.

Regarding the second clause, well, Linux is not Unix-like in many
respects and we want it like that. That's a weak argument.

>   3) don't aid third parties in recovering your removed data. If I want
>      them to have it I'll give it to them.

See 2. Explicit purging is of course possible. (Novell Netware also had
a "purge" command.)

Anyway, it seems that there is some functionality which many users want
but which can't be provided in user space: 

  - if files are moved to the recycle-bin-or-whatever-you-call-it, their
size is added to disk free space and
  - automatically purging least recently deleted files.

Regards,
Petr Tesarik