From: Xiubo Li via B4 Relay <devnull+xiubo.li.clyso.com@kernel.org>
To: Ilya Dryomov <idryomov@gmail.com>,
Alex Markuze <amarkuze@redhat.com>,
Viacheslav Dubeyko <slava@dubeyko.com>
Cc: ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org,
Anthony D'Atri <anthony.datri@gmail.com>,
Xiubo Li <xiubo.li@clyso.com>
Subject: [PATCH 0/2] ceph: serialize O_APPEND writes across clients
Date: Sat, 21 Nov 2026 03:21:04 +0200 [thread overview]
Message-ID: <20261121-ceph-append-excl-v1-0-5870375edeb8@clyso.com> (raw)
When two clients append to one file, the filelock is in MIX and both
hold Fw, so each appends at the end of file it has cached and may
overwrite what the other one appended. Commit d2a8d446a09c ("ceph:
revalidate ki_pos for O_APPEND writes after cap acquisition") does
not cover this case: neither client waits for caps, and the other
client's size only arrives after it flushed its caps.
The MDS side [1] lets O_APPEND writers take turns on Fx, in request
order, and sends the current size together with Fx. It is enabled by
the new CEPHFS_FEATURE_APPEND_EXCL feature bit; with an MDS that does
not have it, the client behaves as before.
Patch 1 fixes a reordering of the cap messages of an inode that this
work exposed: a later cap update could overtake the flush carrying the
new size, the MDS then finished its Fw gather early and gave Fx with
the old size to the other client.
Patch 2 is the O_APPEND change itself.
Tested on a vstart cluster with [1], KASAN and lockdep enabled, two
mounts (noshare, or ceph-fuse + kernel), 4 writers on each, without
any file locking; a lost record means one append overwrote another:
+--------------------------------------+--------+------------------+
| case | runs | lost records |
| | | base / patched |
+--------------------------------------+--------+------------------+
| buffered, kernel + kernel | 4 | ~3300-4000 / 0 |
| O_DIRECT, kernel + kernel | 10 | ~4000 / 0 |
| buffered/O_DIRECT, ceph-fuse+kernel | 6 | - / 0 |
| 2 active MDS, export every 3 s | 4 | - / 0 |
| MDS failover during the writes | 2 | - / 0 |
+--------------------------------------+--------+------------------+
With an MDS without the feature the patched client loses records like
the base one, and does not hang. xfstests -g quick shows the same 8
failures (ceph/006, fscrypt key tests) with and without the series.
[1] https://github.com/ceph/ceph/pull/70279
Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
---
Xiubo Li (2):
ceph: send the cap messages of an inode in the order they were prepared
ceph: serialize O_APPEND writes across clients on Fx
fs/ceph/caps.c | 208 +++++++++++++++++++++++++++++++++++++++++++
fs/ceph/file.c | 33 +++++--
fs/ceph/inode.c | 6 ++
fs/ceph/mds_client.h | 4 +-
fs/ceph/super.h | 17 ++++
include/linux/ceph/ceph_fs.h | 1 +
6 files changed, 261 insertions(+), 8 deletions(-)
---
base-commit: 6a9830a8f414d3830c584782568d0233be969d09
change-id: 20261121-ceph-append-excl-64ce323d1a04
Best regards,
--
Xiubo Li <xiubo.li@clyso.com>
next reply other threads:[~2026-11-21 1:21 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-21 1:21 Xiubo Li via B4 Relay [this message]
2026-11-21 1:21 ` [PATCH 1/2] ceph: send the cap messages of an inode in the order they were prepared Xiubo Li via B4 Relay
2026-11-21 1:21 ` [PATCH 2/2] ceph: serialize O_APPEND writes across clients on Fx Xiubo Li via B4 Relay
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261121-ceph-append-excl-v1-0-5870375edeb8@clyso.com \
--to=devnull+xiubo.li.clyso.com@kernel.org \
--cc=amarkuze@redhat.com \
--cc=anthony.datri@gmail.com \
--cc=ceph-devel@vger.kernel.org \
--cc=idryomov@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=slava@dubeyko.com \
--cc=xiubo.li@clyso.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®