From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=82.195.75.108 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793977021; b=grEP08rm+FLB5VkS7Dt8/cZvkishkDk7zRccHj5qv3sYjoDtEELi4ZQWHHy2jmwl2kuK El4EhI0LzT0hS4ZuziYROe/IoXWNSz+Z+8CCptGEypUdaM2269Nq38VOVKmj4HEBUpNpB D87dZI+WLgYjgX+Q0HiZRwED3GvGDHhh/ALLYt9ek0lCAx3kKcjEZRukR4cudV8w2lOzI BpeW/ErclD7fGdgHHFrD5mcgAhijPJKN17NZt2gwniyQVF+JXK+LU9Mo+k61l+vrAeM9f pPeG16VnG4ZAr/70QRxqtK6uv9t68Di9LXp3166uGBmGojjEaRqI01KTYXtRJGYdLeA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793977021; h=DMARC-Filter:DKIM-Signature:Received:Date:From:To:Cc:Subject: Message-ID:References:MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To:X-Debian-User; bh=ouqotmQz82VhTw/HxlXhVKKyO5h22zOiImm6TZVgam4=; b=0nPfuLB58RhAikIFV8sluS977GYrWf5nz7kxxOZS9W+3mDvv4UrfQ6yEThBKdcZ37yUx nWco0c7Eg5IJqwHRiHfs8DO5WwfbBfSRa8hKy/pcrptrELgq1MGdKREbLDkgdIPOEltbs oxbzyHChwOHIODSvriZkHfTjRvPbR7ivbFUg0tbziA+80WlpRc3g/3rFdEMSQ5e7yeetf iaOivN2Q3U0SEhdO3fQDEqbEmE34OQTK10ZHhE4ycj4mt1mQgNxjsqoUzPVx8TuxMTYOY ywVDZGm6Pixjw8QyGDYYyE0swUY12lGfqWGSqqAKNn9AP+9bgwXIcxQJOf7762SLSJg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass header.d=debian.org header.i=@debian.org header.a=rsa-sha256 header.s=smtpauto.stravinsky header.b=u3dvrilR; arc=none smtp.remote-ip=82.195.75.108 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4FFE606A326; Fri, 6 Nov 2026 14:57:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org E4FFE606A326 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=debian.org header.i=@debian.org header.a=rsa-sha256 header.s=smtpauto.stravinsky header.b=u3dvrilR DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:In-Reply-To:Content-Transfer-Encoding: Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date: Reply-To:Content-ID:Content-Description; bh=ouqotmQz82VhTw/HxlXhVKKyO5h22zOiImm6TZVgam4=; b=u3dvrilRnNU7Ioj9YIzB3OOZvI yjkvNQuOwsrhDV00nJE23P+eZq7kGmMD/Hkdnv88wkAR/6NCkpliNOHRcPZoNJp0OVHv+B5EgmQCn 3mt9zC8It6d80EjxLBLtAaiO+MELt98bxX2pp7LD/+Uvx+dEOH2GnxrnNjd2LOo6lB3vp6jXkxBPu Q8PKKcAMftLp8WcM3xqES9WYW3zKVcwRDQZTJSKzGJVd7M2+723Od9wuXHgGVgka+1IZjjvhKMby+ MaOf1rO7fJjtytn+dm8ToT+yK9toUrwT9iP+EfURTGM6uJaoXtO3lt5mn2tGGOQBbUlvzUk2b8evs IRj+IZxA==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1xPLNB-009MYZ-0p; Fri, 06 Nov 2026 14:56:57 +0000 Date: Fri, 6 Nov 2026 06:56:52 -0800 From: Breno Leitao To: Qu Wenruo Cc: linux-btrfs@vger.kernel.org, David Sterba , Chris Mason , Qu Wenruo , Filipe Manana , Boris Burkov , Atish Patra , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: Re: bug: btrfs: direct IO fallback racing with in-flight direct IO fails fsync Message-ID: References: X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Debian-User: leitao On Fri, Nov 06, 2026 at 08:15:23AM +1030, Qu Wenruo wrote: > > > 在 2026/11/5 22:37, Breno Leitao 写道: > > Hi, > > > > Atish reported that arm64 production hosts (64K pages, 4K blocks) that > > keep VM disk images as NOCOW files on btrfs, opened with O_DIRECT, on > > 7.x based kernels, and it still reproduces in linux-next as of > > next-20261104: > > > > Page cache invalidation failure on direct I/O. Possible data corruption due to collision with buffered I/O! > > > > Besides the message, dio_warn_stale_pagecache() records -EIO in the > > mapping, so the next fsync() on the image fails. The message blames the > > application for mixing buffered and direct IO, but every write here is > > O_DIRECT: the buffered writes come from btrfs itself, when it falls back > > for writes that are not 4K aligned. > > > > It does not look like a regression, since the fallback never waited for > > in-flight direct IO. 64K pages make it easy to hit, as a folio then holds > > 16 blocks. Since e9b7b9d9e78e5b ("btrfs: use IOMAP_DIO_BOUNCE flag > > instead of falling back to buffered IO"), COW files are exposed too. > > > > I was able to come up with a reproducer that hits the issue within > > seconds with 64K pages. With XFS on a loop device backed by such a file, > > standing in for the VM, XFS shuts down on "log I/O error -5": > > > > https://github.com/leitao/btrfs-dio-fallback > > > > My robot identified this as the root cause: > > > > 1. Task A has an async O_DIRECT write in flight to one 4K block of a 64K > > folio. iomap invalidated the page cache before submitting it, and the > > inode lock was dropped. > > 2. Task B issues an O_DIRECT write whose offset, length or buffer is not > > 4K aligned. btrfs_direct_write() falls back to btrfs_buffered_write(), > > which does not wait for in-flight direct IO, and dirties another block > > of the same folio. > > 3. Task A's write completes. kiocb_invalidate_post_direct_write() finds > > the folio dirty, so invalidate_inode_pages2_range() fails with -EBUSY > > and dio_warn_stale_pagecache() records -EIO. The next fsync() fails. > > > > ext4 fixed the same race in 15cdefd0c052 ("ext4: drain in-flight DIO > > before buffered write fallback"), and ext2, ntfs and exfat are getting > > the same fix. Doing the same in btrfs_buffered_write() for the fallback > > case makes the reproducer clean, and I can send that patch if it looks > > like the right direction. > > > > This patch is a copy of the fixes above (from other FS) and make the > > reproducer happy. > > > > commit 16cd17d1e8205f9f86b8da0a9b1ccefdb2665764 > > Author: Breno Leitao > > Date: Thu Nov 5 02:59:39 2026 -0800 > > > > btrfs: drain in-flight direct IO before buffered write fallback > > When btrfs_direct_write() cannot do a write as direct IO, because the > > offset, the length or the user buffer is not aligned to the sectorsize, > > or the range has compressed extents or page cache, it falls back to > > btrfs_buffered_write(). That dirties the page cache without waiting for > > direct IO still in flight from other tasks: > > 1) Task A submits an async direct write to block 1 of a folio. iomap > > invalidates the page cache in the range and submits the bio, and the > > inode lock is dropped. > > 2) Task B issues a 512 byte O_DIRECT write to block 0 of the same folio. > > btrfs_direct_write() falls back to btrfs_buffered_write(), which > > dirties the folio. > > 3) Task A's write completes. kiocb_invalidate_post_direct_write() finds > > the folio dirty, invalidate_inode_pages2_range() fails with -EBUSY, > > and dio_warn_stale_pagecache() logs: > > [ 2.048568] Page cache invalidation failure on direct I/O. Possible data corruption due to collision with buffered I/O! > > [ 2.048574] File: /tmp/diorepro.JTCKpJ/block.img PID: 260 Comm: kworker/11:1 > > It also records -EIO in the mapping, so the next fsync() fails. > > The two writes do not touch the same bytes, they only share a folio. > > With 4K blocks on 64K pages a folio holds 16 blocks, so this is easy to > > hit there. Meta's arm64 hosts with 64K pages hit it with VM disk images > > that are NOCOW files on btrfs, opened with O_DIRECT. In a test, XFS on a > > loop device with direct IO on such a file gets EIO on a log write and > > shuts down within seconds. > > Wait for in-flight direct IO before the fallback dirties the page cache, > > as ext4 does since commit 15cdefd0c052 ("ext4: drain in-flight DIO > > before buffered write fallback"). btrfs_buffered_write() holds the inode > > lock exclusively there, so no new direct IO can start until the write is > > done. NOWAIT writes never get there, as btrfs_direct_write() returns > > -EAGAIN for them before falling back. > > A reproducer that keeps 4K libaio O_DIRECT writes in flight while other > > threads issue 512 byte O_DIRECT writes to the same folios of a NOCOW > > file got 1123 fdatasync() EIOs in 10 seconds on a 64K page VM, and none > > with this patch. > > Fixes: f85781fb505e ("btrfs: switch to iomap for direct IO") > > Reported-by: Atish Patra > > Signed-off-by: Breno Leitao > > > > diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c > > index f978c6524aa03f..2017a96814d857 100644 > > --- a/fs/btrfs/file.c > > +++ b/fs/btrfs/file.c > > @@ -1334,6 +1334,14 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter) > > if (ret < 0) > > return ret; > > + /* > > + * A direct write that fell back to buffered IO must not dirty folios > > + * while direct IO is in flight: its completion fails to invalidate > > + * them and records -EIO in the mapping. > > + */ > > + if (iocb->ki_flags & IOCB_DIRECT) > > + inode_dio_wait(inode); > > + > > Sashiko exposed a problem that the inode can already be enlarged before: > > https://sashiko.dev/#/patchset/auxv7Tg0xSq5yHmb%40gmail.com That seems important as well, and not hard to fix. Let me try to hack something. Thanks, --breno