filesystem with integrity checking

filesystem with integrity checking

[crimil tradalo]

I am looking for a filesystem that supports integrity checking of
files with some kind of checksum/hash.

For example, for each block an SHA hash is stored when written and
verified when read. If the underlying hardware has corrupted the data,
or returned a different block than expected, the filesystem can detect
the error.

Obviously, ideal would be integration with the RAID layer so that
valid data could be found in the redundancy, similar to ZFS, but I
don't expect this to exist.

The last time I saw this discussed, the consensus seemed to be that it
wasn't worth the CPU time. I hope that as CPUs have gotten faster and
data sets larger, more people are noticing how frequent data
corruption occurs and are beginning to respect the value of an OS
ensuring the integrity of data on permanent storage. Everyone here
must have experienced data corruption that is either seemingly-random
or traced to bad hardware. I'd like to detect such corruption as soon
as it happens, rather than after it has caused a catastrophic loss of
data.

If there is none that currently supports this, I am curious if any
intend to support it eventually. I see it mentioned in BTRFS
documentation, but considering the low odds of a new filesystem
actually becoming viable, I'm more interested in a practical solution.

My understanding is that the only filesystems that offer this are ZFS
and WAFL, neither of which is going to be useful on Linux of course.

Re: filesystem with integrity checking

[Jeff Moyer]

crimil tradalo <crimiltradalo@gmail.com> writes:

> I am looking for a filesystem that supports integrity checking of
> files with some kind of checksum/hash.
>
> For example, for each block an SHA hash is stored when written and
> verified when read. If the underlying hardware has corrupted the data,
> or returned a different block than expected, the filesystem can detect
> the error.
>
> Obviously, ideal would be integration with the RAID layer so that
> valid data could be found in the redundancy, similar to ZFS, but I
> don't expect this to exist.
>
> The last time I saw this discussed, the consensus seemed to be that it
> wasn't worth the CPU time. I hope that as CPUs have gotten faster and
> data sets larger, more people are noticing how frequent data
> corruption occurs and are beginning to respect the value of an OS
> ensuring the integrity of data on permanent storage. Everyone here
> must have experienced data corruption that is either seemingly-random
> or traced to bad hardware. I'd like to detect such corruption as soon
> as it happens, rather than after it has caused a catastrophic loss of
> data.
>
> If there is none that currently supports this, I am curious if any
> intend to support it eventually. I see it mentioned in BTRFS
> documentation, but considering the low odds of a new filesystem
> actually becoming viable, I'm more interested in a practical solution.
>
> My understanding is that the only filesystems that offer this are ZFS
> and WAFL, neither of which is going to be useful on Linux of course.

You may want to take a look at the DIF/DIX work that Martin K. Petersen
is doing:
  http://oss.oracle.com/projects/data-integrity/

Cheers,
Jeff