David Howells <dhowells@redhat.com> wrote:

> I have patches that store a bitmap in an xattr, with 1 bit per 256K block of
> cachefile, for a 512 byte xattr able to map up to 1G of file size.  I need to
> revisit this at some point soon.  It may be better to switch to storing an
> extent list, but:
[...]
> Trying to store a million+ extent records in a single xattr would kind of
> suck.  Now, it might be possible to collapse that somewhat if a lot of the
> extents are virtually sequential within the file, since I don't care about
> physical sequentiality.
>
> Also, can you tell if you have a preferred cache granularity?  Would a 256K
> granularity, say, use too much space?

Sorry for the slow answer.  256K would be fine for us: our cache gets
filled by random training reads of 64 to 256 KB.  The problem is the
size of our files.  Many cached objects are hundreds of GB, the largest
we've seen is 946 GB, so at 256K one object needs up to about 450 KB of
bitmap - far more than a 512 byte xattr.

An extent list would suit the case that hurts us.  An object only gets
slow once it's fully or nearly fully cached, and then it's one virtually
contiguous range, however many physical extents it has: for example
569 GB in 2,528,439 extents.  While such an object is filling, though,
it's scattered pieces at random offsets, so the list would stay long
for weeks before it collapses.

> > I can send the workaround if it's useful as a reference,
>
> If you could, that would be great!

Attached, as a reference only.  It's a naive implementation I wrote
with the help of an LLM, just to confirm where the time goes - I don't
know the netfs and cachefiles internals well enough to propose it as a
fix, and nobody has reviewed it.  It keeps the last SEEK_DATA/SEEK_HOLE
answer in the cachefiles object, so it's lost when the cookie is
withdrawn - 10 s after the file is closed, or on reboot - and each new
open pays one full walk again.  It also still asks the backing
filesystem where the data is, the opposite of where you're taking
cachefiles.  It's written against vfs-7.4.netfs at 9f8b64f66, on top of
your v12 series (cachefiles_query_occupancy()), and was only tested in
a VM.  We also have a port to the Ubuntu 6.8 kernel our nodes run,
under test but not deployed.

On our nodes we'll detect fragmented, fully cached objects and
defragment them, and in the long run we'll move away from such large
files, which aren't ideal for us in other ways either.  I'm still happy
to test whatever you do for this case.

Thanks,
Petr