mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 0/2] ceph: serialize O_APPEND writes across clients
@ 2026-11-21  1:21 Xiubo Li via B4 Relay
  2026-11-21  1:21 ` [PATCH 1/2] ceph: send the cap messages of an inode in the order they were prepared Xiubo Li via B4 Relay
  2026-11-21  1:21 ` [PATCH 2/2] ceph: serialize O_APPEND writes across clients on Fx Xiubo Li via B4 Relay
  0 siblings, 2 replies; 3+ messages in thread
From: Xiubo Li via B4 Relay @ 2026-11-21  1:21 UTC (permalink / raw)
  To: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko
  Cc: ceph-devel, linux-kernel, Anthony D'Atri, Xiubo Li

When two clients append to one file, the filelock is in MIX and both
hold Fw, so each appends at the end of file it has cached and may
overwrite what the other one appended.  Commit d2a8d446a09c ("ceph:
revalidate ki_pos for O_APPEND writes after cap acquisition") does
not cover this case: neither client waits for caps, and the other
client's size only arrives after it flushed its caps.

The MDS side [1] lets O_APPEND writers take turns on Fx, in request
order, and sends the current size together with Fx.  It is enabled by
the new CEPHFS_FEATURE_APPEND_EXCL feature bit; with an MDS that does
not have it, the client behaves as before.

Patch 1 fixes a reordering of the cap messages of an inode that this
work exposed: a later cap update could overtake the flush carrying the
new size, the MDS then finished its Fw gather early and gave Fx with
the old size to the other client.

Patch 2 is the O_APPEND change itself.

Tested on a vstart cluster with [1], KASAN and lockdep enabled, two
mounts (noshare, or ceph-fuse + kernel), 4 writers on each, without
any file locking; a lost record means one append overwrote another:

  +--------------------------------------+--------+------------------+
  | case                                 | runs   | lost records     |
  |                                      |        | base / patched   |
  +--------------------------------------+--------+------------------+
  | buffered, kernel + kernel            | 4      | ~3300-4000 / 0   |
  | O_DIRECT, kernel + kernel            | 10     | ~4000 / 0        |
  | buffered/O_DIRECT, ceph-fuse+kernel  | 6      | - / 0            |
  | 2 active MDS, export every 3 s       | 4      | - / 0            |
  | MDS failover during the writes       | 2      | - / 0            |
  +--------------------------------------+--------+------------------+

With an MDS without the feature the patched client loses records like
the base one, and does not hang.  xfstests -g quick shows the same 8
failures (ceph/006, fscrypt key tests) with and without the series.

[1] https://github.com/ceph/ceph/pull/70279

Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
---
Xiubo Li (2):
      ceph: send the cap messages of an inode in the order they were prepared
      ceph: serialize O_APPEND writes across clients on Fx

 fs/ceph/caps.c               | 208 +++++++++++++++++++++++++++++++++++++++++++
 fs/ceph/file.c               |  33 +++++--
 fs/ceph/inode.c              |   6 ++
 fs/ceph/mds_client.h         |   4 +-
 fs/ceph/super.h              |  17 ++++
 include/linux/ceph/ceph_fs.h |   1 +
 6 files changed, 261 insertions(+), 8 deletions(-)
---
base-commit: 6a9830a8f414d3830c584782568d0233be969d09
change-id: 20261121-ceph-append-excl-64ce323d1a04

Best regards,
--  
Xiubo Li <xiubo.li@clyso.com>



^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-21  1:21 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-21  1:21 [PATCH 0/2] ceph: serialize O_APPEND writes across clients Xiubo Li via B4 Relay
2026-11-21  1:21 ` [PATCH 1/2] ceph: send the cap messages of an inode in the order they were prepared Xiubo Li via B4 Relay
2026-11-21  1:21 ` [PATCH 2/2] ceph: serialize O_APPEND writes across clients on Fx Xiubo Li via B4 Relay

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®