Re: [PATCH v12 4/5] cachefiles: Don't rely on backing fs storage map for most use cases

[Petr Chmelar <petr.chmelar@innovatrics.com>]

Hi David,

With this patch, reading a file whose cache object is only partly
filled returns wrong data, mostly whole 64 KiB blocks of zeros.  A
second full read returns the same wrong blocks.  It reproduces over
NFS and 9p on vfs-7.4.netfs (9f8b64f66).  Over 9p it bisects to this
patch, 13e340e397 in that branch; over NFS its parent is clean and the
branch tip is not.

#regzbot introduced: 13e340e397

Reproducer: random 64 KiB reads covering half of a 256 MiB file of
random data, then a full read after dropping the page cache, compared
with the original per 64 KiB block:

  fio --name=half --filename=/mnt/part.bin --rw=randread --bs=64k \
      --ioengine=psync --fadvise_hint=random --size=256m --io_size=128m \
      --randrepeat=1
  sync; echo 3 > /proc/sys/vm/drop_caches
  cmp /mnt/part.bin /path/to/original

Setup: QEMU 8.2.2 (TCG, 4 vCPUs), cache on a fresh XFS on a loop device
(2 GiB for 9p, 4 GiB for NFS), cachefilesd 0.10.10 with brun 10%, bcull
7%, bstop 3%, fio 3.36.  Mounts:

  NFS: NFSv4.2 loopback export of a tmpfs, vers=4.2,fsc,actimeo=1
  9p:  trans=virtio,version=9p2000.L,cache=fscache,msize=1048576

Wrong 64 KiB blocks out of 4096, first full read / second full read:

  kernel                mount     wrong       all zeros
  v7.3-rc6              9p        0 / 0       0
  e0aa90eb8c            9p        0 / 0       0
  98e0cd700d (parent)   9p        0 / 0       0
  98e0cd700d (parent)   NFS       0 / 0       0
  13e340e397 (this)     9p        936 / 936   825
  8097a263bd (merge)    9p        936 / 936   825
  9f8b64f66             9p        936 / 936   825
  9f8b64f66             NFS       817 / 817   817
  9f8b64f66             9p loose  0 / 0       0

  "9p loose" is the 9p mount with cache=loose, without fscache.

On 9f8b64f66 over 9p, three runs gave the same counts and the same
first wrong blocks.  The first wrong blocks are 10, 13, 15, 24, 26 and
27 over 9p, and 50, 51, 53, 54, 55 and 56 over NFS.

Two more observations, in case they help:

 - Over 9p, after the random fill and both full reads, the backing
   object has 451 extents with this patch and 28 on v7.3-rc6 (37 on
   98e0cd700d).

 - Over NFS on 9f8b64f66, the fscache counters after the fill and both
   full reads are IO rd=371 wr=2189.  I haven't traced whether the
   zeros come from reads over holes in the backing file or from what
   was written to it.

Over NFS, a server-side rewrite of the file (invalidation) and a shrink
and regrow through the mount both read back correctly on 9f8b64f66 and
98e0cd700d.

I found this while testing a workaround for slow reads from fragmented
cache objects, which I describe in a reply to the cover letter:

  https://lore.kernel.org/r/20261011.cachefiles-seek-hole-walk.pchmelar@innovatrics.com

It reproduces without that workaround.  The reproducer and the
bisection were done with LLM assistance.

Thanks,
Petr