Re: [PATCH v12 0/5] netfs, cachefiles: Changes for next, primarily occupancy tracking-related

[David Howells <dhowells@redhat.com>]

Petr Chmelar <petr.chmelar@innovatrics.com> wrote:

> The cover letter leaves non-contiguous objects, which still use
> SEEK_DATA/SEEK_HOLE, for a later series.  We hit that case badly in
> production, so here are numbers in case they help with its priority.

I know.  But it's a tricky problem to solve as I need to store a map of the
content somewhere.  xattrs are painful as I can only read or write them
atomically and as whole blobs - and also I'm not sure if I need to limit the
size of any given xattr to 512 bytes.  Further, the xattr wangling interface
inside the kernel assumes a flat virtual buffer rather than taking an I/O
iterator.

I have patches that store a bitmap in an xattr, with 1 bit per 256K block of
cachefile, for a 512 byte xattr able to map up to 1G of file size.  I need to
revisit this at some point soon.  It may be better to switch to storing an
extent list, but:

> On our NFS clients (Ubuntu, v6.8, cache on XFS), a sequential read of a
> fully cached 184 GB object with 1,080,732 extents ran at 0.4 MB/s.

Trying to store a million+ extent records in a single xattr would kind of
suck.  Now, it might be possible to collapse that somewhat if a lot of the
extents are virtually sequential within the file, since I don't care about
physical sequentiality.

Also, can you tell if you have a preferred cache granularity?  Would a 256K
granularity, say, use too much space?

> I expect the right fix is for cachefiles to track what it has written
> itself, as CONTENT_ALL now does for contiguous files, rather than
> something like the above.

Indeed - as mentioned above.

> I can send the workaround if it's useful as a reference,

If you could, that would be great!

> and I'm happy to test whatever you plan for this case on our workload.

Thanks.  Now I just need to reproduce your bug.  Note that I posted another
cachefiles patch that Christian hasn't picked up yet:

https://lore.kernel.org/linux-fsdevel/1091807.1791483394@warthog.procyon.org.uk/T/#u

David