From: Michael Mulqueen <mike@method-b.uk>
To: David Howells <dhowells@redhat.com>, Pierre Barre <pierre@barre.sh>
Cc: ericvh@kernel.org, lucho@ionkov.net, asmadeus@codewreck.org,
linux_oss@crudebyte.com, v9fs@lists.linux.dev,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH] 9p: fix data corruption with writeback caching during concurrent stat
Date: Wed, 18 Nov 2026 12:22:00 +0000 [thread overview]
Message-ID: <e2c960ca-7b29-45e1-b650-849fc357f8b0@method-b.uk> (raw)
In-Reply-To: <1667424.1771432276@warthog.procyon.org.uk>
Hi,
I've encountered this bug and have spent some time confirming it and
writing a minimal reproducer script for it.
I've bisected it with my reproducer script and found it was introduced
in v6.4, specifically commit d9bc0d11e33b. I wasn't sure about the
etiquette of jumping straight into a discussion and tagging regzbot, so
I haven't, but I'm pretty confident that this is where the regression
was introduced. That is the commit that adds the writeback path -
before it, cache=mmap did not buffer writes through the page cache, so
a stale getattr had no unsent data to zero.
I cannot reproduce it on cache=readahead, only on the writeback path
that cache=mmap uses.
The reproducer writes with O_APPEND in one thread while another thread
stats the file and then reads the server copy back at the end. It
writes for a fixed period of 5 seconds, which is why the number of
records in the output differs, but it reproduces reliably on every run.
It needs diod installed and has to be run as root. Here's the output on
the current v7.3 release:
cache=mmap: appending for 5s while a reader stat()s the file...
-> 689268 NUL bytes in the server's copy, 13897 of 16642 records lost
cache=readahead: appending for 5s while a reader stat()s the file...
-> 0 NUL bytes in the server's copy, 0 of 10179 records lost
I have tried the fixes (Pierre Barre's v1 and v2, 27 December 2025, and
David Howells' sketch, 18 February 2026) discussed in this thread and
they appear to resolve it in the basic case. Pierre's apply cleanly to
v7.3; David's I rebased by hand (its step 5 is already present as
c60ae98c5aa6). My reproducer is too minimal to address all the points
raised in this thread, it only confirms that it's still present in the
latest kernel and that it regressed from v6.3 to v6.4. I'll try to find
some more time in the next couple of weeks to test the patches
comprehensively, but I wanted toshare what I already have in case it is
useful to anyone else looking into this.
--- repro-b.py --------------------------------------------------------
#!/usr/bin/env python3
import os, re, shutil, socket, subprocess, tempfile, threading, time
base = tempfile.mkdtemp(prefix="p9b.")
exp = base + "/export"
mnt = base + "/mnt"
os.mkdir(exp)
os.mkdir(mnt)
port = 5700 + os.getpid() % 200
srv = subprocess.Popen(
["diod", "-f", "-n", "-l", f"127.0.0.1:{port}", "-e", exp],
stderr=subprocess.PIPE)
for _ in range(100):
try:
socket.create_connection(("127.0.0.1", port), 0.2).close()
break
except OSError:
time.sleep(0.05)
else:
srv.terminate()
raise SystemExit("diod failed to start:\n"
+ srv.stderr.read().decode())
print(f"diod up on 127.0.0.1:{port}", flush=True)
def run(mode, seconds=5):
opts = (f"trans=tcp,port={port},version=9p2000.L,cache={mode},"
f"aname={exp},uname=root,access=any,ignoreqv")
subprocess.run(["mount", "-t", "9p", "-o", opts,
"127.0.0.1", mnt], check=True)
log = mnt + "/app.log"
open(log, "w").close()
print(f"cache={mode}: appending for {seconds}s while a reader "
f"stat()s the file...", flush=True)
stop = threading.Event()
def poll():
while not stop.is_set():
try:
os.stat(log)
except OSError:
pass
threading.Thread(target=poll, daemon=True).start()
fd = os.open(log, os.O_WRONLY | os.O_APPEND)
n = 0
end = time.time() + seconds
while time.time() < end:
os.write(fd, b"%08d " % n + b"x" * 90 + b"\n")
n += 1
os.fsync(fd)
os.close(fd)
stop.set()
data = open(exp + "/app.log", "rb").read()
nul = data.count(0)
intact = len(set(re.findall(rb"(\d{8}) x{90}", data)))
subprocess.run(["umount", "-l", mnt], stderr=subprocess.DEVNULL)
print(f" -> {nul} NUL bytes in the server's copy, "
f"{n - intact} of {n} records lost", flush=True)
try:
run("mmap")
run("readahead")
finally:
srv.terminate()
srv.wait()
shutil.rmtree(base, ignore_errors=True)
--- end repro-b.py ----------------------------------------------------
Cheers,
Mike
prev parent reply other threads:[~2026-11-18 12:22 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-12-27 8:37 Pierre Barre
2025-12-27 11:30 ` Christian Schoenebeck
2025-12-27 17:58 ` Pierre Barre
2026-02-18 14:04 ` David Howells
2026-02-18 16:31 ` David Howells
2026-11-18 12:22 ` Michael Mulqueen [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e2c960ca-7b29-45e1-b650-849fc357f8b0@method-b.uk \
--to=mike@method-b.uk \
--cc=asmadeus@codewreck.org \
--cc=dhowells@redhat.com \
--cc=ericvh@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux_oss@crudebyte.com \
--cc=lucho@ionkov.net \
--cc=pierre@barre.sh \
--cc=v9fs@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®