* [PATCH v3 01/20] drbd: mark as BROKEN during DRBD 9 rework
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 02/20] drbd: extend wire protocol definitions for DRBD 9 Christoph Böhmwalder
` (18 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder
Mark DRBD as BROKEN while the driver is being reworked for DRBD 9
multi-peer support. The following commits restructure the driver
extensively, and intermediate states do not compile. DRBD will be
re-enabled once the rework is complete.
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/block/drbd/Kconfig b/drivers/block/drbd/Kconfig
index 495a72da04c6..ab0bf1b7eb7d 100644
--- a/drivers/block/drbd/Kconfig
+++ b/drivers/block/drbd/Kconfig
@@ -9,6 +9,7 @@ comment "DRBD disabled because PROC_FS or INET not selected"
config BLK_DEV_DRBD
tristate "DRBD Distributed Replicated Block Device support"
depends on PROC_FS && INET
+ depends on BROKEN
select LRU_CACHE
select CRC32
help
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 02/20] drbd: extend wire protocol definitions for DRBD 9
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 01/20] drbd: mark as BROKEN during " Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 03/20] drbd: introduce DRBD 9 on-disk metadata format Christoph Böhmwalder
` (17 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Andreas Gruenbacher,
Joel Colledge
Extend drbd_protocol.h with the packet types and structures needed for
multi-peer operation.
Two-phase commit (2PC) messages coordinate distributed state changes
across all peers in a cluster.
Data-generation-tag (dagtag) messages order application writes relative
to resync IO, preventing stale overwrites during concurrent resync.
Peer-acknowledgement packets carry a node bitmask so each primary can
track which peers have persisted a write.
The connection-features handshake now carries sender and receiver node
IDs, establishing peer identity at the wire level.
New feature-flag bits advertise these capabilities during negotiation,
allowing DRBD to remain wire-compatible with 8.4 peers while enabling
the full DRBD 9 feature set when both ends support it.
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_protocol.h | 614 +++++++++++++++++++++++------
1 file changed, 484 insertions(+), 130 deletions(-)
diff --git a/drivers/block/drbd/drbd_protocol.h b/drivers/block/drbd/drbd_protocol.h
index be7dacac5a86..6c845f4934da 100644
--- a/drivers/block/drbd/drbd_protocol.h
+++ b/drivers/block/drbd/drbd_protocol.h
@@ -6,6 +6,17 @@
#ifndef __DRBD_PROTOCOL_H
#define __DRBD_PROTOCOL_H
+#ifdef __KERNEL__
+#include <linux/types.h>
+#else
+#include <stdint.h>
+#define u64 uint64_t
+#ifndef ARRAY_SIZE
+#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
+#endif
+#endif
+#include <linux/drbd.h>
+
enum drbd_packet {
/* receiver (data socket) */
P_DATA = 0x00,
@@ -28,12 +39,12 @@ enum drbd_packet {
P_AUTH_RESPONSE = 0x11,
P_STATE_CHG_REQ = 0x12,
- /* (meta socket) */
+ /* asender (meta socket */
P_PING = 0x13,
P_PING_ACK = 0x14,
P_RECV_ACK = 0x15, /* Used in protocol B */
P_WRITE_ACK = 0x16, /* Used in protocol C */
- P_RS_WRITE_ACK = 0x17, /* Is a P_WRITE_ACK, additionally call set_in_sync(). */
+ P_RS_WRITE_ACK = 0x17, /* Write ack for resync reply. */
P_SUPERSEDED = 0x18, /* Used in proto C, two-primaries conflict detection */
P_NEG_ACK = 0x19, /* Sent if local disk is unusable */
P_NEG_DREPLY = 0x1a, /* Local disk is broken... */
@@ -45,7 +56,7 @@ enum drbd_packet {
P_OV_REQUEST = 0x1e, /* data socket */
P_OV_REPLY = 0x1f,
- P_OV_RESULT = 0x20, /* meta socket */
+ P_OV_RESULT = 0x20, /* meta sock: Protocol < 122 version of P_OV_RESULT_ID */
P_CSUM_RS_REQUEST = 0x21, /* data socket */
P_RS_IS_IN_SYNC = 0x22, /* meta socket */
P_SYNC_PARAM89 = 0x23, /* data socket, protocol version 89 replacement for P_SYNC_PARAM */
@@ -55,32 +66,84 @@ enum drbd_packet {
P_DELAY_PROBE = 0x27, /* is used on BOTH sockets */
P_OUT_OF_SYNC = 0x28, /* Mark as out of sync (Outrunning), data socket */
P_RS_CANCEL = 0x29, /* meta: Used to cancel RS_DATA_REQUEST packet by SyncSource */
- P_CONN_ST_CHG_REQ = 0x2a, /* data sock: Connection wide state request */
- P_CONN_ST_CHG_REPLY = 0x2b, /* meta sock: Connection side state req reply */
- P_RETRY_WRITE = 0x2c, /* Protocol C: retry conflicting write request */
+ P_CONN_ST_CHG_REQ = 0x2a, /* data sock: state change request */
+ P_CONN_ST_CHG_REPLY = 0x2b, /* meta sock: state change reply */
+ P_RETRY_WRITE = 0x2c, /* meta sock: Protocol C: peer did not process this write, retry it.
+ * Was the 8.4 two-primaries conflict retry; reused, gated by
+ * DRBD_FF_WRITE_POSTPONE, as the answer to a write a sync
+ * target can not secure toward its sync source. */
P_PROTOCOL_UPDATE = 0x2d, /* data sock: is used in established connections */
- /* 0x2e to 0x30 reserved, used in drbd 9 */
+ P_TWOPC_PREPARE = 0x2e, /* data sock: prepare state change */
+ P_TWOPC_ABORT = 0x2f, /* data sock: abort state change */
- /* REQ_OP_DISCARD. We used "discard" in different contexts before,
- * which is why I chose TRIM here, to disambiguate. */
+ P_DAGTAG = 0x30, /* data sock: set the current dagtag */
+
+ /*
+ * REQ_DISCARD. We used "discard" in different contexts before,
+ * which is why I chose TRIM here, to disambiguate.
+ */
P_TRIM = 0x31,
/* Only use these two if both support FF_THIN_RESYNC */
P_RS_THIN_REQ = 0x32, /* Request a block for resync or reply P_RS_DEALLOCATED */
- P_RS_DEALLOCATED = 0x33, /* Contains only zeros on sync source node */
+ P_RS_DEALLOCATED = 0x33, /* Protocol < 122 version of P_RS_DEALLOCATED_ID */
- /* REQ_WRITE_SAME.
+ /*
+ * REQ_WRITE_SAME.
* On a receiving side without REQ_WRITE_SAME,
- * we may fall back to an opencoded loop instead. */
+ * we may fall back to an opencoded loop instead.
+ */
P_WSAME = 0x34,
-
- /* 0x35 already claimed in DRBD 9 */
+ P_TWOPC_PREP_RSZ = 0x35, /* PREPARE a 2PC resize operation*/
P_ZEROES = 0x36, /* data sock: zero-out, WRITE_ZEROES */
- /* 0x40 .. 0x48 already claimed in DRBD 9 */
+ /*
+ * place new packets for both 8.4 and 9 here,
+ * place new packets for 9-only in the next gap.
+ */
+
+ P_PEER_ACK = 0x40, /* meta sock: tell which nodes have acked a request */
+ P_PEERS_IN_SYNC = 0x41, /* data sock: Mark area as in sync */
+
+ P_UUIDS110 = 0x42, /* data socket */
+ P_PEER_DAGTAG = 0x43, /* data socket, used to trigger reconciliation resync */
+ P_CURRENT_UUID = 0x44, /* data socket */
+
+ P_TWOPC_YES = 0x45, /* meta sock: allow two-phase commit */
+ P_TWOPC_NO = 0x46, /* meta sock: reject two-phase commit */
+ P_TWOPC_COMMIT = 0x47, /* data sock: commit state change */
+ P_TWOPC_RETRY = 0x48, /* meta sock: retry two-phase commit */
+
+ P_CONFIRM_STABLE = 0x49, /* meta sock: similar to an unsolicited partial barrier ack */
+ /* protocol version 115,
+ * meta: cancel RS_DATA_REQUEST packet if already Ahead again,
+ * tell peer to stop sending resync requests...
+ */
+ P_RS_CANCEL_AHEAD = 0x4a,
+ P_DISCONNECT = 0x4b, /* data sock: Disconnect and stop connection attempts */
+
+ P_RS_DAGTAG_REQ = 0x4c, /* data sock: Request a block for resync, with dagtag dependency */
+ P_RS_CSUM_DAGTAG_REQ = 0x4d, /* data sock: Request a block for resync if checksum differs, with dagtag dependency */
+ P_RS_THIN_DAGTAG_REQ = 0x4e, /* data sock: Request a block for resync or reply P_RS_DEALLOCATED, with dagtag dependency */
+ P_OV_DAGTAG_REQ = 0x4f, /* data sock: Request a checksum for online verify, with dagtag dependency */
+ P_OV_DAGTAG_REPLY = 0x50, /* data sock: Reply with a checksum for online verify, with dagtag dependency */
+
+ P_WRITE_ACK_IN_SYNC = 0x51, /* meta sock: Application write ack setting bits in sync. */
+ P_RS_NEG_ACK = 0x52, /* meta sock: Local disk is unusable writing resync reply. */
+ P_OV_RESULT_ID = 0x53, /* meta sock: Online verify result with block ID. */
+ P_RS_DEALLOCATED_ID = 0x54, /* data sock: Contains only zeros on sync source node. */
+
+ P_FLUSH_REQUESTS = 0x55, /* data sock: Flush prior requests then send ack and/or forward */
+ P_FLUSH_FORWARD = 0x56, /* meta sock: Send ack after sending P_OUT_OF_SYNC for prior P_PEER_ACK */
+ P_FLUSH_REQUESTS_ACK = 0x57, /* data sock: Response to initiator of P_FLUSH_REQUESTS */
+ P_ENABLE_REPLICATION_NEXT = 0x58, /* data sock: whether to start replication on next resync start */
+ P_ENABLE_REPLICATION = 0x59, /* data sock: enable or disable replication during resync */
+
+ P_RS_DAGTAG_WAIT_REQ = 0x5a, /* data sock: Wait for the dagtag dependency only, answer without data */
+ P_RS_DAGTAG_REACHED = 0x5b, /* meta sock: The dagtag dependency of this request is reached. */
+ P_RS_DAGTAG_UNREACHABLE = 0x5c, /* meta sock: Can not reach the dagtag this request depends on. */
P_MAY_IGNORE = 0x100, /* Flag to test if (cmd > P_MAY_IGNORE) ... */
- P_MAX_OPT_CMD = 0x101,
/* special command ids for handshake */
@@ -105,24 +168,24 @@ enum drbd_packet {
* regardless of 32 or 64 bit arch!
*/
struct p_header80 {
- u32 magic;
- u16 command;
- u16 length; /* bytes of data after this header */
+ uint32_t magic;
+ uint16_t command;
+ uint16_t length; /* bytes of data after this header */
} __packed;
/* Header for big packets, Used for data packets exceeding 64kB */
struct p_header95 {
- u16 magic; /* use DRBD_MAGIC_BIG here */
- u16 command;
- u32 length;
+ uint16_t magic; /* use DRBD_MAGIC_BIG here */
+ uint16_t command;
+ uint32_t length;
} __packed;
struct p_header100 {
- u32 magic;
- u16 volume;
- u16 command;
- u32 length;
- u32 pad;
+ uint32_t magic;
+ uint16_t volume;
+ uint16_t command;
+ uint32_t length;
+ uint32_t pad;
} __packed;
/* These defines must not be changed without changing the protocol version.
@@ -132,10 +195,10 @@ struct p_header100 {
#define DP_HARDBARRIER 1 /* no longer used */
#define DP_RW_SYNC 2 /* equals REQ_SYNC */
#define DP_MAY_SET_IN_SYNC 4
-#define DP_UNPLUG 8 /* not used anymore */
+#define DP_UNPLUG 8 /* equals REQ_UNPLUG (compat) */
#define DP_FUA 16 /* equals REQ_FUA */
#define DP_FLUSH 32 /* equals REQ_PREFLUSH */
-#define DP_DISCARD 64 /* equals REQ_OP_DISCARD */
+#define DP_DISCARD 64 /* equals REQ_DISCARD */
#define DP_SEND_RECEIVE_ACK 128 /* This is a proto B write request */
#define DP_SEND_WRITE_ACK 256 /* This is a proto C write request */
#define DP_WSAME 512 /* equiv. REQ_WRITE_SAME */
@@ -147,58 +210,114 @@ struct p_header100 {
*/
struct p_data {
- u64 sector; /* 64 bits sector number */
- u64 block_id; /* to identify the request in protocol B&C */
- u32 seq_num;
- u32 dp_flags;
+ uint64_t sector; /* 64 bits sector number */
+ uint64_t block_id; /* to identify the request in protocol B&C */
+ uint32_t seq_num;
+ uint32_t dp_flags;
} __packed;
struct p_trim {
struct p_data p_data;
- u32 size; /* == bio->bi_size */
+ uint32_t size; /* == bio->bi_size */
} __packed;
struct p_wsame {
struct p_data p_data;
- u32 size; /* == bio->bi_size */
+ uint32_t size; /* == bio->bi_size */
} __packed;
/*
- * commands which share a struct:
- * p_block_ack:
- * P_RECV_ACK (proto B), P_WRITE_ACK (proto C),
+ * struct p_block_ack shared by commands:
+ * P_RECV_ACK (proto B)
+ * P_WRITE_ACK (proto C),
+ * P_WRITE_ACK_IN_SYNC,
* P_SUPERSEDED (proto C, two-primaries conflict detection)
- * p_block_req:
- * P_DATA_REQUEST, P_RS_DATA_REQUEST
+ * P_RS_WRITE_ACK
+ * P_NEG_ACK
+ * P_NEG_DREPLY
+ * P_NEG_RS_DREPLY
+ * P_RS_NEG_ACK
+ * P_OV_RESULT
+ * P_RS_IS_IN_SYNC
+ * P_RS_CANCEL
+ * P_RS_DEALLOCATED_ID
+ * P_RS_CANCEL_AHEAD
+ * P_RS_DAGTAG_REACHED
+ * P_RS_DAGTAG_UNREACHABLE
*/
struct p_block_ack {
- u64 sector;
- u64 block_id;
- u32 blksize;
- u32 seq_num;
+ uint64_t sector;
+ uint64_t block_id;
+ uint32_t blksize;
+ uint32_t seq_num;
+} __packed;
+
+/* For P_OV_RESULT_ID. */
+struct p_ov_result {
+ uint64_t sector;
+ uint64_t block_id;
+ uint32_t blksize;
+ uint32_t seq_num;
+ uint32_t result;
+ uint32_t pad;
} __packed;
+enum ov_result {
+ OV_RESULT_SKIP = 0,
+ OV_RESULT_IN_SYNC = 1,
+ OV_RESULT_OUT_OF_SYNC = 2,
+};
+
+struct p_block_req_common {
+ uint64_t sector;
+ uint64_t block_id;
+ uint32_t blksize;
+} __packed;
+
+/*
+ * struct p_block_req shared by commands:
+ * P_DATA_REQUEST
+ * P_RS_DATA_REQUEST
+ * P_OV_REQUEST
+ * P_OV_REPLY
+ * P_CSUM_RS_REQUEST
+ * P_RS_THIN_REQ
+ */
struct p_block_req {
- u64 sector;
- u64 block_id;
- u32 blksize;
- u32 pad; /* to multiple of 8 Byte */
+ /* Allow fields to be addressed directly or via req_common. */
+ union {
+ struct {
+ uint64_t sector;
+ uint64_t block_id;
+ uint32_t blksize;
+ } __packed;
+ struct p_block_req_common req_common;
+ };
+ uint32_t pad; /* to multiple of 8 Byte */
} __packed;
/*
- * commands with their own struct for additional fields:
- * P_CONNECTION_FEATURES
- * P_BARRIER
- * P_BARRIER_ACK
- * P_SYNC_PARAM
- * ReportParams
+ * struct p_rs_req shared by commands:
+ * P_RS_DAGTAG_REQ
+ * P_RS_CSUM_DAGTAG_REQ
+ * P_RS_THIN_DAGTAG_REQ
+ * P_OV_DAGTAG_REQ
+ * P_OV_DAGTAG_REPLY
+ * P_RS_DAGTAG_WAIT_REQ
*/
+struct p_rs_req {
+ struct p_block_req_common req_common;
+ uint32_t dagtag_node_id;
+ uint64_t dagtag;
+} __packed;
/* supports TRIM/DISCARD on the "wire" protocol */
#define DRBD_FF_TRIM 1
-/* Detect all-zeros during resync, and rather TRIM/UNMAP/DISCARD those blocks
- * instead of fully allocate a supposedly thin volume on initial resync */
+/*
+ * Detect all-zeros during resync, and rather TRIM/UNMAP/DISCARD those blocks
+ * instead of fully allocate a supposedly thin volume on initial resync
+ */
#define DRBD_FF_THIN_RESYNC 2
/* supports REQ_WRITE_SAME on the "wire" protocol.
@@ -247,54 +366,117 @@ struct p_block_req {
*/
#define DRBD_FF_WZEROES 8
+/* Supports synchronization of application and resync IO using data generation
+ * tags (dagtags). See Documentation/application-resync-synchronization.rst for
+ * details.
+ */
+#define DRBD_FF_RESYNC_DAGTAG 16
+
+/* V2 of p_twopc_request has a 32 bit flag field and the two fields for node
+ * Ids are reduced to 8 bit instead of 32 bits.
+ *
+ * The flag TWOPC_HAS_RECHABLE indicates that in the commit phase
+ * (P_TWOPC_COMMIT) the reachable_nodes mask is set.
+ *
+ * The old behavior sends the primary_nodes mask, mask, and val in
+ * phase 2 (P_TWOPC_COMMIT), where mask and val are the same values as
+ * in phase 1 (P_TWOPC_PREPARE).
+ */
+#define DRBD_FF_2PC_V2 32
+
+/* Starting with drbd-9.1.15, a node with a backing disk sends the new
+ * current-uuid also to diskless nodes when the initial resync is
+ * skipped.
+ *
+ * The peer needs to know about this detail to apply the necessary
+ * strictness regarding downgrading its view of the partner's disk
+ * state.
+ */
+#define DRBD_FF_RS_SKIP_UUID 64
+
+/* Support for resync_without_replication.
+ */
+#define DRBD_FF_RESYNC_WITHOUT_REPLICATION 128
+
+/* Support for bitmap block size != 4k. If you connect peers with
+ * different bitmap block sizes, the resync becomes more
+ * interesting, and we need to communicate the bitmap block size.
+ */
+#define DRBD_FF_BM_BLOCK_SHIFT 256
+
+/* Starting with drbd-9.2.19 a new kind of reconciliation resync
+ */
+#define DRBD_FF_RECONCILE_RECONNECT 512
+
+/* A sync target that received a write in a range the sync source is still to
+ * resync does not acknowledge it before the source holds that write too: it
+ * asks with P_RS_DAGTAG_WAIT_REQ, naming the write's position in the writer's
+ * stream, and the source answers P_RS_DAGTAG_REACHED once its copy of that
+ * stream has reached the position, or P_RS_DAGTAG_UNREACHABLE when it has
+ * lost the writer. In the unreachable case the target answers the writer
+ * P_RETRY_WRITE: the write counts as not processed, and the writer retries
+ * it once the cluster can order it again.
+ *
+ * Advertised by every node that answers dagtag wait requests, serves a resync
+ * to a target that secures its writes, sends the postpone answer, and retries
+ * a postponed write.
+ */
+#define DRBD_FF_WRITE_POSTPONE 1024
struct p_connection_features {
- u32 protocol_min;
- u32 feature_flags;
- u32 protocol_max;
+ uint32_t protocol_min;
+ uint32_t feature_flags;
+ uint32_t protocol_max;
+ uint32_t sender_node_id;
+ uint32_t receiver_node_id;
/* should be more than enough for future enhancements
* for now, feature_flags and the reserved array shall be zero.
*/
- u32 _pad;
- u64 reserved[7];
+ uint32_t _pad;
+ uint64_t reserved[6];
} __packed;
struct p_barrier {
- u32 barrier; /* barrier number _handle_ only */
- u32 pad; /* to multiple of 8 Byte */
+ uint32_t barrier; /* barrier number _handle_ only */
+ uint32_t pad; /* to multiple of 8 Byte */
} __packed;
struct p_barrier_ack {
- u32 barrier;
- u32 set_size;
+ uint32_t barrier;
+ uint32_t set_size;
+} __packed;
+
+struct p_confirm_stable {
+ uint64_t oldest_block_id;
+ uint64_t youngest_block_id;
+ uint32_t set_size;
+ uint32_t pad; /* to multiple of 8 Byte */
} __packed;
struct p_rs_param {
- u32 resync_rate;
+ uint32_t resync_rate;
/* Since protocol version 88 and higher. */
char verify_alg[];
} __packed;
struct p_rs_param_89 {
- u32 resync_rate;
+ uint32_t resync_rate;
/* protocol version 89: */
char verify_alg[SHARED_SECRET_MAX];
char csums_alg[SHARED_SECRET_MAX];
} __packed;
struct p_rs_param_95 {
- u32 resync_rate;
- struct_group(algs,
- char verify_alg[SHARED_SECRET_MAX];
- char csums_alg[SHARED_SECRET_MAX];
- );
- u32 c_plan_ahead;
- u32 c_delay_target;
- u32 c_fill_target;
- u32 c_max_rate;
+ uint32_t resync_rate;
+ char verify_alg[SHARED_SECRET_MAX];
+ char csums_alg[SHARED_SECRET_MAX];
+ uint32_t c_plan_ahead;
+ uint32_t c_delay_target;
+ uint32_t c_fill_target;
+ uint32_t c_max_rate;
} __packed;
enum drbd_conn_flags {
@@ -303,35 +485,90 @@ enum drbd_conn_flags {
};
struct p_protocol {
- u32 protocol;
- u32 after_sb_0p;
- u32 after_sb_1p;
- u32 after_sb_2p;
- u32 conn_flags;
- u32 two_primaries;
+ uint32_t protocol;
+ uint32_t after_sb_0p;
+ uint32_t after_sb_1p;
+ uint32_t after_sb_2p;
+ uint32_t conn_flags;
+ uint32_t two_primaries;
/* Since protocol version 87 and higher. */
char integrity_alg[];
} __packed;
+#define UUID_FLAG_DISCARD_MY_DATA ((u64)1 << 0)
+#define UUID_FLAG_CRASHED_PRIMARY ((u64)1 << 1)
+#define UUID_FLAG_INCONSISTENT ((u64)1 << 2)
+#define UUID_FLAG_SKIP_INITIAL_SYNC ((u64)1 << 3)
+
+#define UUID_FLAG_MASK_COMPAT_84 \
+ (UUID_FLAG_DISCARD_MY_DATA|\
+ UUID_FLAG_CRASHED_PRIMARY|\
+ UUID_FLAG_INCONSISTENT|\
+ UUID_FLAG_SKIP_INITIAL_SYNC)
+
+#define UUID_FLAG_NEW_DATAGEN ((u64)1 << 4)
+#define UUID_FLAG_STABLE ((u64)1 << 5)
+#define UUID_FLAG_GOT_STABLE ((u64)1 << 6) /* send UUIDs */
+#define UUID_FLAG_RESYNC ((u64)1 << 7) /* compare UUIDs and eventually start resync */
+#define UUID_FLAG_RECONNECT ((u64)1 << 8)
+#define UUID_FLAG_DISKLESS_PRIMARY ((u64)1 << 9) /* Use with UUID_FLAG_RESYNC if a diskless primary is the reason */
+#define UUID_FLAG_PRIMARY_LOST_QUORUM ((u64)1 << 10)
+#define UUID_FLAG_SYNC_TARGET ((u64)1 << 11) /* currently L_SYNC_TARGET to some peer */
+#define UUID_FLAG_HAS_UNALLOC ((u64)1 << 12) /* highest byte contains index of not allocated bitmap uuid */
+#define UUID_FLAG_BITMAP_AUTHORITATIVE ((u64)1 << 13) /* my out-of-sync bits toward you were set for blocks you lack, not by a resync or an invalidate; protocol 125 */
+
+#define UUID_FLAG_UNALLOC_SHIFT 56
+#define UUID_FLAG_UNALLOC_MASK ((u64)0xff << UUID_FLAG_UNALLOC_SHIFT)
+
struct p_uuids {
- u64 uuid[UI_EXTENDED_SIZE];
+ uint64_t current_uuid;
+ uint64_t bitmap_uuid;
+ uint64_t history_uuids[HISTORY_UUIDS_V08];
+ uint64_t dirty_bits;
+ uint64_t uuid_flags;
+} __packed;
+
+struct p_uuids110 {
+ uint64_t current_uuid;
+ uint64_t dirty_bits;
+ uint64_t uuid_flags;
+ uint64_t node_mask;
+ /* weak_nodes when UUID_FLAG_NEW_DATAGEN is set;
+ * authoritative nodes when UUID_FLAG_STABLE not set
+ */
+
+ uint64_t bitmap_uuids_mask; /* non zero bitmap UUIDS for these nodes */
+ uint64_t other_uuids[];
+ /* the first hweight(bitmap_uuids_mask) slots carry bitmap uuids.
+ * The node with the lowest node_id first.
+ * The remaining slots carry history uuids
+ */
} __packed;
-struct p_rs_uuid {
- u64 uuid;
+struct p_current_uuid {
+ uint64_t uuid;
+ uint64_t weak_nodes;
} __packed;
-/* optional queue_limits if (agreed_features & DRBD_FF_WSAME)
- * see also struct queue_limits, as of late 2015 */
+struct p_uuid {
+ uint64_t uuid;
+} __packed;
+
+/*
+ * optional queue_limits if (agreed_features & DRBD_FF_WSAME)
+ * see also struct queue_limits, as of late 2015
+ */
struct o_qlim {
/* we don't need it yet, but we may as well communicate it now */
- u32 physical_block_size;
+ uint32_t physical_block_size;
- /* so the original in struct queue_limits is unsigned short,
- * but I'd have to put in padding anyways. */
- u32 logical_block_size;
+ /*
+ * so the original in struct queue_limits is unsigned short,
+ * but I'd have to put in padding anyways.
+ */
+ uint32_t logical_block_size;
/* One incoming bio becomes one DRBD request,
* which may be translated to several bio on the receiving side.
@@ -339,88 +576,205 @@ struct o_qlim {
*/
/* various IO hints may be useful with "diskless client" setups */
- u32 alignment_offset;
- u32 io_min;
- u32 io_opt;
+ uint32_t alignment_offset;
+ uint32_t io_min;
+ uint32_t io_opt;
- /* We may need to communicate integrity stuff at some point,
- * but let's not get ahead of ourselves. */
+ /*
+ * We may need to communicate integrity stuff at some point,
+ * but let's not get ahead of ourselves.
+ */
- /* Backend discard capabilities.
+ /*
+ * Backend discard capabilities.
* Receiving side uses "blkdev_issue_discard()", no need to communicate
* more specifics. If the backend cannot do discards, the DRBD peer
* may fall back to blkdev_issue_zeroout().
*/
- u8 discard_enabled;
- u8 discard_zeroes_data;
- u8 write_same_capable;
- u8 _pad;
+ uint8_t discard_enabled;
+ uint8_t discard_zeroes_data;
+ uint8_t write_same_capable;
+
+ /* Bitmap block shift relative to 4k. If peers have differnt bitmap
+ * granularity, any resync related request needs to be aligned to the
+ * larger granularity: we can not clear partial bits.
+ * 0 to 8 to represent 4k to 1M.
+ * If DRBD_FF_BM_BLOCK_SHIFT is agreed on.
+ */
+ uint8_t bm_block_shift_minus_12;
} __packed;
struct p_sizes {
- u64 d_size; /* size of disk */
- u64 u_size; /* user requested size */
- u64 c_size; /* current exported size */
- u32 max_bio_size; /* Maximal size of a BIO */
- u16 queue_order_type; /* not yet implemented in DRBD*/
- u16 dds_flags; /* use enum dds_flags here. */
+ uint64_t d_size; /* size of disk */
+ uint64_t u_size; /* user requested size */
+ uint64_t c_size; /* current exported size */
+ uint32_t max_bio_size; /* Maximal size of a BIO */
+ uint16_t queue_order_type; /* not yet implemented in DRBD*/
+ uint16_t dds_flags; /* use enum dds_flags here. */
/* optional queue_limits if (agreed_features & DRBD_FF_WSAME) */
struct o_qlim qlim[];
} __packed;
struct p_state {
- u32 state;
+ uint32_t state;
} __packed;
struct p_req_state {
- u32 mask;
- u32 val;
+ uint32_t mask;
+ uint32_t val;
} __packed;
struct p_req_state_reply {
- u32 retcode;
+ uint32_t retcode;
+} __packed;
+
+struct p_twopc_request {
+ uint32_t tid; /* transaction identifier */
+ union {
+ struct { /* when DRBD_FF_2PC_V2 is set */
+ uint32_t flags;
+ uint16_t _pad;
+ int8_t s8_initiator_node_id; /* initiator of the transaction */
+ int8_t s8_target_node_id; /* target of the transaction (or -1) */
+ };
+ struct { /* original packet version */
+ uint32_t u32_initiator_node_id; /* initiator of the transaction */
+ uint32_t u32_target_node_id; /* target of the transaction (or -1) */
+ };
+ };
+ uint64_t nodes_to_reach;
+ union {
+ union { /* TWOPC_STATE_CHANGE */
+ struct { /* P_TWOPC_PREPARE */
+ uint64_t _compat_pad;
+ uint32_t mask;
+ uint32_t val;
+ };
+ struct { /* P_TWOPC_COMMIT */
+ uint64_t primary_nodes;
+ uint64_t reachable_nodes; /* when TWOPC_HAS_RECHABLE flag is set */
+ };
+ };
+ union { /* TWOPC_RESIZE */
+ struct { /* P_TWOPC_PREP_RSZ */
+ uint64_t user_size;
+ uint16_t dds_flags;
+ };
+ struct { /* P_TWOPC_COMMIT */
+ uint64_t diskful_primary_nodes;
+ uint64_t exposed_size;
+ };
+ };
+ };
+} __packed;
+
+#define TWOPC_HAS_FLAGS 0x80000000 /* For packet dissectors */
+#define TWOPC_HAS_REACHABLE 0x40000000 /* The reachable_nodes field is valid */
+#define TWOPC_PRI_INCAPABLE 0x20000000 /* The primary has no access to data */
+
+struct p_twopc_reply {
+ uint32_t tid; /* transaction identifier */
+ uint32_t initiator_node_id; /* initiator of the transaction */
+ uint64_t reachable_nodes;
+
+ union {
+ struct { /* TWOPC_STATE_CHANGE */
+ uint64_t primary_nodes;
+ uint64_t weak_nodes;
+ };
+ struct { /* TWOPC_RESIZE */
+ uint64_t diskful_primary_nodes;
+ uint64_t max_possible_size;
+ };
+ };
} __packed;
struct p_drbd06_param {
- u64 size;
- u32 state;
- u32 blksize;
- u32 protocol;
- u32 version;
- u32 gen_cnt[5];
- u32 bit_map_gen[5];
+ uint64_t size;
+ uint32_t state;
+ uint32_t blksize;
+ uint32_t protocol;
+ uint32_t version;
+ uint32_t gen_cnt[5];
+ uint32_t bit_map_gen[5];
} __packed;
struct p_block_desc {
- u64 sector;
- u32 blksize;
- u32 pad; /* to multiple of 8 Byte */
+ uint64_t sector;
+ uint32_t blksize;
+ uint32_t pad; /* to multiple of 8 Byte */
} __packed;
-/* Valid values for the encoding field.
- * Bump proto version when changing this. */
+/*
+ * Valid values for the encoding field.
+ * Bump proto version when changing this.
+ */
enum drbd_bitmap_code {
- /* RLE_VLI_Bytes = 0,
+ /*
+ * RLE_VLI_Bytes = 0,
* and other bit variants had been defined during
- * algorithm evaluation. */
+ * algorithm evaluation.
+ */
RLE_VLI_Bits = 2,
};
struct p_compressed_bm {
- /* (encoding & 0x0f): actual encoding, see enum drbd_bitmap_code
+ /*
+ * (encoding & 0x0f): actual encoding, see enum drbd_bitmap_code
* (encoding & 0x80): polarity (set/unset) of first runlength
* ((encoding >> 4) & 0x07): pad_bits, number of trailing zero bits
* used to pad up to head.length bytes
*/
- u8 encoding;
+ uint8_t encoding;
- u8 code[];
+ uint8_t code[];
} __packed;
struct p_delay_probe93 {
- u32 seq_num; /* sequence number to match the two probe packets */
- u32 offset; /* usecs the probe got sent after the reference time point */
+ uint32_t seq_num; /* sequence number to match the two probe packets */
+ uint32_t offset; /* usecs the probe got sent after the reference time point */
+} __packed;
+
+struct p_dagtag {
+ uint64_t dagtag;
+} __packed;
+
+struct p_peer_ack {
+ uint64_t mask;
+ uint64_t dagtag;
+} __packed;
+
+struct p_peer_block_desc {
+ uint64_t sector;
+ uint64_t mask;
+ uint32_t size;
+ uint32_t pad; /* to multiple of 8 Byte */
+} __packed;
+
+struct p_peer_dagtag {
+ uint64_t dagtag;
+ uint32_t node_id;
+} __packed;
+
+struct p_flush_requests {
+ uint64_t flush_sequence;
+} __packed;
+
+struct p_flush_forward {
+ uint64_t flush_sequence;
+ uint32_t initiator_node_id;
+} __packed;
+
+struct p_flush_ack {
+ uint64_t flush_sequence;
+ uint32_t primary_node_id;
+} __packed;
+
+struct p_enable_replication {
+ uint8_t enable;
+ uint8_t _pad1;
+ uint16_t _pad2;
} __packed;
/*
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 03/20] drbd: introduce DRBD 9 on-disk metadata format
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 01/20] drbd: mark as BROKEN during " Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 02/20] drbd: extend wire protocol definitions for DRBD 9 Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 04/20] drbd: add transport layer abstraction Christoph Böhmwalder
` (16 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder
Add a new header that captures the DRBD 9 on-disk metadata layout,
enabling state tracking for multiple peers.
It includes the per-device superblock and per-peer slot structures
needed to track bitmap UUIDs and sync state for each peer.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_meta_data.h | 170 ++++++++++++++++++++++++++++
1 file changed, 170 insertions(+)
create mode 100644 drivers/block/drbd/drbd_meta_data.h
diff --git a/drivers/block/drbd/drbd_meta_data.h b/drivers/block/drbd/drbd_meta_data.h
new file mode 100644
index 000000000000..400803267c83
--- /dev/null
+++ b/drivers/block/drbd/drbd_meta_data.h
@@ -0,0 +1,170 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef DRBD_META_DATA_H
+#define DRBD_META_DATA_H
+
+#ifdef __KERNEL__
+#define be_u64 __be64
+#define be_s64 __be64
+#define be_u32 __be32
+#define be_s32 __be32
+#define be_u16 __be16
+#else
+#define be_u64 struct { uint64_t be; }
+#define be_s64 struct { int64_t be; }
+#define be_u32 struct { uint32_t be; }
+#define be_s32 struct { int32_t be; }
+#define be_u16 struct { uint16_t be; }
+#endif
+
+/* how I came up with this magic?
+ * base64 decode "actlog==" ;)
+ */
+#define DRBD_AL_MAGIC 0x69cb65a2
+
+#define BM_BLOCK_SHIFT_4k 12 /* 4k per bit */
+#define BM_BLOCK_SHIFT_MIN BM_BLOCK_SHIFT_4k
+#define BM_BLOCK_SHIFT_MAX 20
+#define BM_BLOCK_SIZE_4k 4096
+#define BM_BLOCK_SIZE_MIN (1<<BM_BLOCK_SHIFT_MIN)
+#define BM_BLOCK_SIZE_MAX (1<<BM_BLOCK_SHIFT_MAX)
+
+struct peer_dev_md_on_disk_9 {
+ be_u64 bitmap_uuid;
+ be_u64 bitmap_dagtag;
+ be_u32 flags;
+ be_s32 bitmap_index;
+ be_u32 reserved_u32[2];
+} __packed;
+
+/* Changed block tracking: one bitmap slot, not owned by a peer, records every
+ * write to the backing device since the bookmark "id". Valid only with
+ * DRBD_MDFF_CBT in the features.
+ */
+struct cbt_bookmark_on_disk_9 {
+ be_u64 id; /* the slot records changes since this bookmark */
+ be_u64 bitmap_uuid; /* current UUID at cbt-start */
+ be_u64 bitmap_dagtag; /* the Primary's dagtag at cbt-start */
+ be_s64 since; /* time of cbt-start, seconds since the epoch */
+ be_u32 flags; /* CBT_BOOKMARK_* */
+ be_s32 bitmap_index;
+} __packed;
+
+struct meta_data_on_disk_9 {
+ be_u64 effective_size; /* last agreed size */
+ be_u64 current_uuid;
+ be_u64 members; /* only if MDF_HAVE_MEMBERS_MASK is in the flags */
+ /* Feature flags. Values prefixed with "DRBD_MDFF".
+ * Separate from "flags": Unknown features bits are cleared; unknown flags bits are kept.
+ */
+ be_u64 features;
+ be_u64 reserved_u64[2]; /* to have the magic at the same position as in v07, and v08 */
+ be_u64 device_uuid;
+ be_u32 flags; /* MDF */
+ be_u32 magic;
+ be_u32 md_size_sect;
+ be_u32 al_offset; /* offset to this block */
+ be_u32 al_nr_extents; /* important for restoring the AL */
+ be_u32 bm_offset; /* offset to the bitmap, from here */
+ be_u32 bm_bytes_per_bit; /* BM_BLOCK_SIZE */
+ be_u32 la_peer_max_bio_size; /* last peer max_bio_size */
+ be_u32 bm_max_peers;
+ be_s32 node_id;
+
+ /* see al_tr_number_to_on_disk_sector() */
+ be_u32 al_stripes;
+ be_u32 al_stripe_size_4k;
+
+ be_u32 reserved_u32[2];
+
+ struct peer_dev_md_on_disk_9 peers[DRBD_PEERS_MAX];
+ be_u64 history_uuids[HISTORY_UUIDS];
+ struct cbt_bookmark_on_disk_9 cbt_bookmark;
+
+ unsigned char padding_start[0];
+ unsigned char padding_end[0] __attribute__((aligned(4096)));
+} __packed;
+
+/* Attention, these two are defined in drbd_int.h as well! */
+#define AL_UPDATES_PER_TRANSACTION 64
+#define AL_CONTEXT_PER_TRANSACTION 919
+
+enum al_transaction_types {
+ AL_TR_UPDATE = 0,
+ AL_TR_INITIALIZED = 0xffff
+};
+/* all fields on disc in big endian */
+struct __packed al_transaction_on_disk {
+ /* don't we all like magic */
+ be_u32 magic;
+
+ /* to identify the most recent transaction block
+ * in the on disk ring buffer
+ */
+ be_u32 tr_number;
+
+ /* checksum on the full 4k block, with this field set to 0. */
+ be_u32 crc32c;
+
+ /* type of transaction, special transaction types like:
+ * purge-all, set-all-idle, set-all-active, ... to-be-defined
+ * see also enum al_transaction_types
+ */
+ be_u16 transaction_type;
+
+ /* we currently allow only a few thousand extents,
+ * so 16bit will be enough for the slot number.
+ */
+
+ /* how many updates in this transaction */
+ be_u16 n_updates;
+
+ /* maximum slot number, "al-extents" in drbd.conf speak.
+ * Having this in each transaction should make reconfiguration
+ * of that parameter easier.
+ */
+ be_u16 context_size;
+
+ /* slot number the context starts with */
+ be_u16 context_start_slot_nr;
+
+ /* Some reserved bytes. Expected usage is a 64bit counter of
+ * sectors-written since device creation, and other data generation tag
+ * supporting usage
+ */
+ be_u32 __reserved[4];
+
+ /* --- 36 byte used --- */
+
+ /* Reserve space for up to AL_UPDATES_PER_TRANSACTION changes
+ * in one transaction, then use the remaining byte in the 4k block for
+ * context information. "Flexible" number of updates per transaction
+ * does not help, as we have to account for the case when all update
+ * slots are used anyways, so it would only complicate code without
+ * additional benefit.
+ */
+ be_u16 update_slot_nr[AL_UPDATES_PER_TRANSACTION];
+
+ /* but the extent number is 32bit, which at an extent size of 4 MiB
+ * allows to cover device sizes of up to 2**54 Byte (16 PiB)
+ */
+ be_u32 update_extent_nr[AL_UPDATES_PER_TRANSACTION];
+
+ /* --- 420 bytes used (36 + 64*6) --- */
+
+ /* 4096 - 420 = 3676 = 919 * 4 */
+ be_u32 context[AL_CONTEXT_PER_TRANSACTION];
+};
+
+#define DRBD_AL_PMEM_MAGIC 0x6aa667a6 /* "al==pmem" */
+
+struct __packed al_on_pmem {
+ be_u32 magic;
+ be_u32 slots[];
+};
+
+#undef be_u64
+#undef be_u32
+#undef be_s32
+#undef be_u16
+
+#endif
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 04/20] drbd: add transport layer abstraction
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (2 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 03/20] drbd: introduce DRBD 9 on-disk metadata format Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 05/20] drbd: add TCP transport implementation Christoph Böhmwalder
` (15 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Roland Kammerer,
Joel Colledge, Moritz Tanner
DRBD 9 decouples all network I/O from the core driver by introducing a
transport abstraction layer. The core driver interacts with the network
only through a well-defined ops table, and concrete implementations
live in separate kernel modules that register themselves at load time.
The abstraction models connections as a set of paths, each representing
a local/remote address pair. A shared listener mechanism allows multiple
paths on the same resource to reuse a single listening socket.
The core exports callbacks which the transports can call back into,
keeping protocol logic in the core and handling wire details in the
transport.
This commit adds the header defining the interface and some
infrastructure. Actual transport implementations follow later in the
series.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Makefile | 1 +
drivers/block/drbd/drbd_transport.c | 545 +++++++++++++++++++
drivers/block/drbd/drbd_transport.h | 487 +++++++++++++++++
drivers/block/drbd/drbd_transport_template.c | 164 ++++++
4 files changed, 1197 insertions(+)
create mode 100644 drivers/block/drbd/drbd_transport.c
create mode 100644 drivers/block/drbd/drbd_transport.h
create mode 100644 drivers/block/drbd/drbd_transport_template.c
diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index 1a6dc0f482f5..d4fa578325dc 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -4,6 +4,7 @@ drbd-y += drbd_sender.o drbd_receiver.o drbd_req.o drbd_actlog.o
drbd-y += drbd_main.o drbd_strings.o drbd_nl.o
drbd-y += drbd_interval.o drbd_state.o
drbd-y += drbd_nl_gen.o
+drbd-y += drbd_transport.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
obj-$(CONFIG_BLK_DEV_DRBD) += drbd.o
diff --git a/drivers/block/drbd/drbd_transport.c b/drivers/block/drbd/drbd_transport.c
new file mode 100644
index 000000000000..567cc5b9b13a
--- /dev/null
+++ b/drivers/block/drbd/drbd_transport.c
@@ -0,0 +1,545 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2014, LINBIT HA-Solutions GmbH.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/spinlock.h>
+#include <linux/module.h>
+#include <net/ipv6.h>
+#include "drbd_transport.h"
+#include "drbd_int.h"
+
+static LIST_HEAD(transport_classes);
+static DECLARE_RWSEM(transport_classes_lock);
+
+static struct drbd_transport_class *__find_transport_class(const char *transport_name)
+{
+ struct drbd_transport_class *transport_class;
+
+ list_for_each_entry(transport_class, &transport_classes, list)
+ if (!strcmp(transport_class->name, transport_name))
+ return transport_class;
+
+ return NULL;
+}
+
+int drbd_register_transport_class(struct drbd_transport_class *transport_class, int version,
+ int drbd_transport_size)
+{
+ int rv = 0;
+
+ if (version != DRBD_TRANSPORT_API_VERSION) {
+ pr_err("DRBD_TRANSPORT_API_VERSION not compatible\n");
+ return -EINVAL;
+ }
+
+ if (drbd_transport_size != sizeof(struct drbd_transport)) {
+ pr_err("sizeof(drbd_transport) not compatible\n");
+ return -EINVAL;
+ }
+
+ down_write(&transport_classes_lock);
+ if (__find_transport_class(transport_class->name)) {
+ pr_err("transport class '%s' already registered\n", transport_class->name);
+ rv = -EEXIST;
+ } else {
+ list_add_tail(&transport_class->list, &transport_classes);
+ pr_info("registered transport class '%s' (version:%s)\n",
+ transport_class->name,
+ transport_class->module->version ?: "N/A");
+ }
+ up_write(&transport_classes_lock);
+ return rv;
+}
+
+void drbd_unregister_transport_class(struct drbd_transport_class *transport_class)
+{
+ down_write(&transport_classes_lock);
+ if (!__find_transport_class(transport_class->name)) {
+ pr_crit("unregistering unknown transport class '%s'\n",
+ transport_class->name);
+ BUG();
+ }
+ list_del_init(&transport_class->list);
+ pr_info("unregistered transport class '%s'\n", transport_class->name);
+ up_write(&transport_classes_lock);
+}
+
+static struct drbd_transport_class *get_transport_class(const char *name)
+{
+ struct drbd_transport_class *tc;
+
+ down_read(&transport_classes_lock);
+ tc = __find_transport_class(name);
+ if (tc && !try_module_get(tc->module))
+ tc = NULL;
+ up_read(&transport_classes_lock);
+ return tc;
+}
+
+struct drbd_transport_class *drbd_get_transport_class(const char *name)
+{
+ struct drbd_transport_class *tc = get_transport_class(name);
+
+ if (!tc) {
+ request_module("drbd_transport_%s", name);
+ tc = get_transport_class(name);
+ }
+
+ return tc;
+}
+
+void drbd_put_transport_class(struct drbd_transport_class *tc)
+{
+ /* convenient in the error cleanup path */
+ if (!tc)
+ return;
+ down_read(&transport_classes_lock);
+ module_put(tc->module);
+ up_read(&transport_classes_lock);
+}
+
+void drbd_print_transports_loaded(struct seq_file *seq)
+{
+ struct drbd_transport_class *tc;
+
+ down_read(&transport_classes_lock);
+
+ seq_puts(seq, "Transports (api:" __stringify(DRBD_TRANSPORT_API_VERSION) "):");
+ list_for_each_entry(tc, &transport_classes, list) {
+ seq_printf(seq, " %s (%s)", tc->name,
+ tc->module->version ? tc->module->version : "NONE");
+ }
+ seq_putc(seq, '\n');
+
+ up_read(&transport_classes_lock);
+}
+
+static bool addr_equal(const struct sockaddr_storage *addr1, const struct sockaddr_storage *addr2)
+{
+ if (addr1->ss_family != addr2->ss_family)
+ return false;
+
+ if (addr1->ss_family == AF_INET6) {
+ const struct sockaddr_in6 *v6a1 = (const struct sockaddr_in6 *)addr1;
+ const struct sockaddr_in6 *v6a2 = (const struct sockaddr_in6 *)addr2;
+
+ if (!ipv6_addr_equal(&v6a1->sin6_addr, &v6a2->sin6_addr))
+ return false;
+ else if (ipv6_addr_type(&v6a1->sin6_addr) & IPV6_ADDR_LINKLOCAL)
+ return v6a1->sin6_scope_id == v6a2->sin6_scope_id;
+ return true;
+ } else /* AF_INET, AF_SSOCKS, AF_SDP */ {
+ const struct sockaddr_in *v4a1 = (const struct sockaddr_in *)addr1;
+ const struct sockaddr_in *v4a2 = (const struct sockaddr_in *)addr2;
+
+ return v4a1->sin_addr.s_addr == v4a2->sin_addr.s_addr;
+ }
+}
+
+static bool addr_and_port_equal(const struct sockaddr_storage *addr1, const struct sockaddr_storage *addr2)
+{
+ if (!addr_equal(addr1, addr2))
+ return false;
+
+ if (addr1->ss_family == AF_INET6) {
+ const struct sockaddr_in6 *v6a1 = (const struct sockaddr_in6 *)addr1;
+ const struct sockaddr_in6 *v6a2 = (const struct sockaddr_in6 *)addr2;
+
+ return v6a1->sin6_port == v6a2->sin6_port;
+ } else /* AF_INET, AF_SSOCKS, AF_SDP */ {
+ const struct sockaddr_in *v4a1 = (const struct sockaddr_in *)addr1;
+ const struct sockaddr_in *v4a2 = (const struct sockaddr_in *)addr2;
+
+ return v4a1->sin_port == v4a2->sin_port;
+ }
+
+ return false;
+}
+
+static struct drbd_listener *find_listener(struct drbd_connection *connection,
+ const struct sockaddr_storage *addr)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_listener *listener;
+
+ list_for_each_entry(listener, &resource->listeners, list) {
+ if (addr_and_port_equal(&listener->listen_addr, addr)) {
+ if (kref_get_unless_zero(&listener->kref))
+ return listener;
+ }
+ }
+ return NULL;
+}
+
+int drbd_get_listener(struct drbd_path *path)
+{
+ struct drbd_transport *transport = path->transport;
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+ struct sockaddr *addr = (struct sockaddr *)&path->my_addr;
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_transport_class *tc = transport->class;
+ struct drbd_listener *listener;
+ bool needs_init = false;
+ int err;
+
+ spin_lock_bh(&resource->listeners_lock);
+ listener = find_listener(connection, (struct sockaddr_storage *)addr);
+ if (!listener) {
+ listener = kzalloc(tc->listener_instance_size, GFP_ATOMIC);
+ if (!listener) {
+ spin_unlock_bh(&resource->listeners_lock);
+ return -ENOMEM;
+ }
+ kref_init(&listener->kref);
+ INIT_LIST_HEAD(&listener->waiters);
+ listener->resource = resource;
+ listener->pending_accepts = 0;
+ spin_lock_init(&listener->waiters_lock);
+ init_completion(&listener->ready);
+ listener->listen_addr = *(struct sockaddr_storage *)addr;
+ listener->transport_class = NULL;
+
+ list_add(&listener->list, &resource->listeners);
+ needs_init = true;
+ }
+ spin_unlock_bh(&resource->listeners_lock);
+
+ if (needs_init) {
+ if (try_module_get(tc->module)) {
+ listener->transport_class = tc;
+ err = tc->ops.init_listener(transport, addr, path->net, listener);
+ } else {
+ err = -ENODEV;
+ }
+ listener->err = err;
+ complete_all(&listener->ready);
+ } else {
+ wait_for_completion(&listener->ready);
+ err = listener->err;
+ }
+
+ if (err) {
+ kref_put(&listener->kref, drbd_listener_destroy);
+ return err;
+ }
+
+ spin_lock_bh(&listener->waiters_lock);
+ kref_get(&path->kref);
+ list_add(&path->listener_link, &listener->waiters);
+ path->listener = listener;
+ spin_unlock_bh(&listener->waiters_lock);
+ /* After exposing the listener on a path, drbd_put_listenr() can destroy it. */
+
+ return 0;
+}
+
+void drbd_listener_destroy(struct kref *kref)
+{
+ struct drbd_listener *listener = container_of(kref, struct drbd_listener, kref);
+ struct drbd_transport_class *tc = listener->transport_class;
+ struct drbd_resource *resource = listener->resource;
+
+ spin_lock_bh(&resource->listeners_lock);
+ list_del(&listener->list);
+ spin_unlock_bh(&resource->listeners_lock);
+
+ if (tc) {
+ tc->ops.release_listener(listener);
+ module_put(tc->module);
+ }
+ kfree(listener);
+}
+
+void drbd_put_listener(struct drbd_path *path)
+{
+ struct drbd_listener *listener;
+
+ listener = xchg(&path->listener, NULL);
+ if (!listener)
+ return;
+
+ spin_lock_bh(&listener->waiters_lock);
+ list_del(&path->listener_link);
+ kref_put(&path->kref, drbd_destroy_path);
+ spin_unlock_bh(&listener->waiters_lock);
+ kref_put(&listener->kref, drbd_listener_destroy);
+}
+
+/**
+ * drbd_listener_try_get_ref() - Acquire a reference to path->listener.
+ * @path: The path whose listener should be pinned.
+ *
+ * Necessary to protect from a concurrently running del-path.
+ */
+struct drbd_listener *drbd_listener_try_get_ref(struct drbd_path *path)
+{
+ struct drbd_connection *connection =
+ container_of(path->transport, struct drbd_connection, transport);
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_listener *listener;
+
+ spin_lock_bh(&resource->listeners_lock);
+ listener = READ_ONCE(path->listener);
+ if (!listener || !kref_get_unless_zero(&listener->kref))
+ listener = NULL;
+ spin_unlock_bh(&resource->listeners_lock);
+ return listener;
+}
+
+struct drbd_path *drbd_find_path_by_addr(struct drbd_listener *listener, struct sockaddr_storage *addr)
+{
+ struct drbd_path *path;
+
+ list_for_each_entry(path, &listener->waiters, listener_link) {
+ if (addr_equal(&path->peer_addr, addr))
+ return path;
+ }
+
+ return NULL;
+}
+
+/* An incoming connection is routed by matching the listener (my_addr and
+ * my_port) and then the peer IP. Two paths that agree on all three are
+ * therefore indistinguishable to the receiver.
+ */
+bool drbd_path_conflicts_by_listener(struct drbd_path *existing,
+ struct drbd_path *candidate)
+{
+ return addr_and_port_equal(&existing->my_addr, &candidate->my_addr) &&
+ addr_equal(&existing->peer_addr, &candidate->peer_addr);
+}
+
+/**
+ * drbd_stream_send_timed_out() - Tells transport if the connection should stay alive
+ * @transport: DRBD transport to operate on.
+ * @stream: DATA_STREAM or CONTROL_STREAM
+ *
+ * When it returns true, the transport should return -EAGAIN to its caller of the
+ * send function. When it returns false the transport should keep on trying to
+ * get the packet through.
+ */
+bool drbd_stream_send_timed_out(struct drbd_transport *transport, enum drbd_stream stream)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+ bool drop_it;
+
+ drop_it = stream == CONTROL_STREAM || connection->cstate[NOW] < C_CONNECTED;
+
+ if (drop_it)
+ return true;
+
+ if (connection->transport.ko_count == 0) {
+ drbd_err(connection, "[%s/%d] sending time expired, pinging peer\n",
+ current->comm, current->pid);
+ drbd_queue_ping(connection);
+ return false;
+ }
+
+ drop_it = !--connection->transport.ko_count;
+ if (!drop_it) {
+ drbd_err(connection, "[%s/%d] sending time expired, ko = %u\n",
+ current->comm, current->pid, connection->transport.ko_count);
+ drbd_queue_ping(connection);
+ }
+
+ return drop_it;
+}
+
+bool drbd_should_abort_listening(struct drbd_transport *transport)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+ bool abort = false;
+
+ if (connection->cstate[NOW] <= C_DISCONNECTING)
+ abort = true;
+ if (signal_pending(current)) {
+ flush_signals(current);
+ smp_rmb();
+ if (get_t_state(&connection->receiver) == EXITING)
+ abort = true;
+ }
+
+ return abort;
+}
+
+/* Called by a transport if a path was established / disconnected */
+void drbd_path_event(struct drbd_transport *transport, struct drbd_path *path)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+
+ notify_path(connection, path, NOTIFY_CHANGE);
+}
+
+struct drbd_path *__drbd_next_path_ref(struct drbd_path *drbd_path,
+ struct drbd_transport *transport)
+{
+ rcu_read_lock();
+ if (!drbd_path) {
+ drbd_path = list_first_or_null_rcu(&transport->paths, struct drbd_path, list);
+ } else {
+ struct list_head *pos;
+ bool in_list;
+
+ pos = list_next_rcu(&drbd_path->list);
+ /* Ensure list head is read before flag. */
+ smp_rmb();
+ in_list = !test_bit(TR_UNREGISTERED, &drbd_path->flags);
+ kref_put(&drbd_path->kref, drbd_destroy_path);
+
+ if (pos == &transport->paths) {
+ drbd_path = NULL;
+ } else if (in_list) {
+ drbd_path = list_entry_rcu(pos, struct drbd_path, list);
+ } else {
+ /* No longer on the list, element might be freed, restart from the start */
+ drbd_path = list_first_or_null_rcu(&transport->paths,
+ struct drbd_path, list);
+ }
+ }
+ if (drbd_path)
+ kref_get(&drbd_path->kref);
+ rcu_read_unlock();
+
+ return drbd_path;
+}
+
+static struct bio *drbd_bio_alloc_next(struct bio *bio)
+{
+ return bio_alloc_bioset(bio->bi_bdev, bio->bi_max_vecs, bio->bi_opf, GFP_NOIO,
+ bio->bi_pool);
+}
+
+int drbd_bio_add_page(struct drbd_transport *transport, struct bio_list *bios,
+ struct page *page, unsigned int len, unsigned int offset)
+{
+ struct bio *bio = bios->tail;
+ struct bio *new_bio;
+ int r;
+
+ r = bio_add_page(bio, page, len, offset);
+ if (r)
+ return r;
+
+ new_bio = drbd_bio_alloc_next(bio);
+ if (!new_bio)
+ return -ENOMEM;
+
+ bio_list_add(bios, new_bio);
+ r = bio_add_page(new_bio, page, len, offset);
+ if (r)
+ return r;
+
+ return -ENOENT;
+}
+
+/* The coarsest queue dma_alignment (as a byte count) among the resource's
+ * attached backing devices; 0 if none is attached. Received payload submitted
+ * to any of them must honour it, see set_rx_alignment.
+ */
+static unsigned int drbd_resource_rx_alignment(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ unsigned int align = 0;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (!get_ldev_if_state(device, D_ATTACHING))
+ continue;
+ if (device->ldev) {
+ struct request_queue *q = bdev_get_queue(device->ldev->backing_bdev);
+
+ align = max_t(unsigned int, align, queue_dma_alignment(q) + 1);
+ }
+ put_ldev(device);
+ }
+ rcu_read_unlock();
+
+ return align;
+}
+
+void drbd_connection_update_rx_alignment(struct drbd_resource *resource,
+ struct drbd_connection *connection)
+{
+ struct drbd_transport_ops *tr_ops = &connection->transport.class->ops;
+
+ if (tr_ops->set_rx_alignment)
+ tr_ops->set_rx_alignment(&connection->transport,
+ drbd_resource_rx_alignment(resource));
+}
+
+void drbd_resource_update_rx_alignment(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource)
+ drbd_connection_update_rx_alignment(resource, connection);
+ rcu_read_unlock();
+}
+
+/* Like drbd_bio_add_page(), but never merges into the tail bvec: the range
+ * becomes a bvec of its own, even if it is physically contiguous with the
+ * previous one. For a transport whose received chunks may share a page (the
+ * chunk boundary falls mid-page), one bvec then owns exactly one page
+ * reference per page it touches, which is what drbd_peer_req_strip_bio()
+ * releases.
+ */
+int drbd_bio_add_page_nomerge(struct drbd_transport *transport, struct bio_list *bios,
+ struct page *page, unsigned int len, unsigned int offset)
+{
+ struct bio *bio = bios->tail;
+
+ if (bio->bi_vcnt >= bio->bi_max_vecs || bio->bi_iter.bi_size > UINT_MAX - len) {
+ struct bio *new_bio = drbd_bio_alloc_next(bio);
+
+ if (!new_bio)
+ return -ENOMEM;
+ bio_list_add(bios, new_bio);
+ bio = new_bio;
+ }
+ __bio_add_page(bio, page, len, offset);
+ return len;
+}
+
+void drbd_transport_lock(struct drbd_transport *transport)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+
+ mutex_lock(&connection->mutex[DATA_STREAM]);
+ mutex_lock(&connection->mutex[CONTROL_STREAM]);
+}
+EXPORT_SYMBOL_GPL(drbd_transport_lock);
+
+void drbd_transport_unlock(struct drbd_transport *transport)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+
+ mutex_unlock(&connection->mutex[CONTROL_STREAM]);
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+}
+EXPORT_SYMBOL_GPL(drbd_transport_unlock);
+
+/* Network transport abstractions */
+EXPORT_SYMBOL_GPL(drbd_register_transport_class);
+EXPORT_SYMBOL_GPL(drbd_unregister_transport_class);
+EXPORT_SYMBOL_GPL(drbd_get_listener);
+EXPORT_SYMBOL_GPL(drbd_put_listener);
+EXPORT_SYMBOL_GPL(drbd_listener_try_get_ref);
+EXPORT_SYMBOL_GPL(drbd_find_path_by_addr);
+EXPORT_SYMBOL_GPL(drbd_stream_send_timed_out);
+EXPORT_SYMBOL_GPL(drbd_should_abort_listening);
+EXPORT_SYMBOL_GPL(drbd_path_event);
+EXPORT_SYMBOL_GPL(drbd_listener_destroy);
+EXPORT_SYMBOL_GPL(__drbd_next_path_ref);
+EXPORT_SYMBOL_GPL(drbd_bio_add_page);
+EXPORT_SYMBOL_GPL(drbd_bio_add_page_nomerge);
diff --git a/drivers/block/drbd/drbd_transport.h b/drivers/block/drbd/drbd_transport.h
new file mode 100644
index 000000000000..cb568420e69a
--- /dev/null
+++ b/drivers/block/drbd/drbd_transport.h
@@ -0,0 +1,487 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2014, LINBIT HA-Solutions GmbH.
+ */
+
+#ifndef DRBD_TRANSPORT_H
+#define DRBD_TRANSPORT_H
+
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/wait.h>
+#include <linux/socket.h>
+
+/*
+ * Whenever touch this file in a non-trivial way, increase the
+ * DRBD_TRANSPORT_API_VERSION
+ * So that transport compiled against an older version of this
+ * header will no longer load in a module that assumes a newer
+ * version.
+ */
+#define DRBD_TRANSPORT_API_VERSION 23
+
+/*
+ * MSG_MSG_DONTROUTE and MSG_PROBE are not used by DRBD. I.e.
+ * we can reuse these flags for our purposes
+ */
+#define CALLER_BUFFER MSG_DONTROUTE
+#define GROW_BUFFER MSG_PROBE
+
+/*
+ * gfp_mask for allocating memory with no write-out.
+ *
+ * When drbd allocates memory on behalf of the peer, we prevent it from causing
+ * write-out because in a criss-cross setup, the write-out could lead to memory
+ * pressure on the peer, eventually leading to deadlock.
+ */
+#define GFP_TRY (__GFP_HIGHMEM | __GFP_NOWARN | __GFP_RECLAIM)
+
+#define tr_printk(level, transport, fmt, args...) ({ \
+ rcu_read_lock(); \
+ printk(level "drbd %s %s:%s: " fmt, \
+ (transport)->log_prefix, \
+ (transport)->class->name, \
+ rcu_dereference((transport)->net_conf)->name, \
+ ## args); \
+ rcu_read_unlock(); \
+ })
+
+#define tr_err(transport, fmt, args...) \
+ tr_printk(KERN_ERR, transport, fmt, ## args)
+#define tr_warn(transport, fmt, args...) \
+ tr_printk(KERN_WARNING, transport, fmt, ## args)
+#define tr_notice(transport, fmt, args...) \
+ tr_printk(KERN_NOTICE, transport, fmt, ## args)
+#define tr_info(transport, fmt, args...) \
+ tr_printk(KERN_INFO, transport, fmt, ## args)
+#define tr_dbg(transport, fmt, args...) ({ \
+ rcu_read_lock(); \
+ pr_debug("drbd %s %s:%s: " fmt, (transport)->log_prefix, \
+ (transport)->class->name, \
+ rcu_dereference((transport)->net_conf)->name, ## args); \
+ rcu_read_unlock(); \
+ })
+
+#define TR_ASSERT(x, exp) \
+ do { \
+ if (!(exp)) \
+ tr_err(x, "ASSERTION %s FAILED in %s\n", \
+ #exp, __func__); \
+ } while (0)
+
+struct drbd_resource;
+struct drbd_listener;
+struct drbd_transport;
+struct bio;
+
+enum drbd_stream {
+ DATA_STREAM,
+ CONTROL_STREAM
+};
+
+enum drbd_tr_hints {
+ CORK,
+ UNCORK,
+ NODELAY,
+ NOSPACE,
+ QUICKACK
+};
+
+enum { /* bits in the flags word */
+ NET_CONGESTED, /* The data socket is congested */
+ RESOLVE_CONFLICTS, /* Set on one node, cleared on the peer! */
+};
+
+enum drbd_tr_free_op {
+ CLOSE_CONNECTION,
+ DESTROY_TRANSPORT
+};
+
+enum drbd_tr_event {
+ CLOSED_BY_PEER,
+ TIMEOUT,
+};
+
+enum drbd_tr_path_flag {
+ TR_ESTABLISHED, /* updated by the transport */
+ TR_UNREGISTERED,
+ TR_TRANSPORT_PRIVATE = 32, /* flags starting here are used exclusively by the transport */
+};
+
+/*
+ * A transport might wrap its own data structure around this. Having
+ * this base class as its first member.
+ */
+struct drbd_path {
+ struct sockaddr_storage my_addr;
+ struct sockaddr_storage peer_addr;
+
+ struct kref kref;
+
+ struct net *net;
+ int my_addr_len;
+ int peer_addr_len;
+ unsigned long flags;
+
+ struct drbd_transport *transport;
+ struct list_head list; /* paths of a connection */
+ /* paths waiting for an incoming connection,
+ * head is in a drbd_listener
+ */
+ struct list_head listener_link;
+ struct drbd_listener *listener;
+
+ struct rcu_head rcu;
+};
+
+/*
+ * Each transport implementation should embed a struct drbd_transport
+ * into its instance data structure.
+ */
+struct drbd_transport {
+ struct drbd_transport_class *class;
+
+ struct list_head paths;
+
+ const char *log_prefix; /* resource name */
+ struct drbd_net_conf __rcu *net_conf; /* content protected by rcu */
+
+ /* These members are intended to be updated by the transport: */
+ unsigned int ko_count;
+ unsigned long flags;
+};
+
+struct drbd_transport_stats {
+ int unread_received;
+ int unacked_send;
+ int send_buffer_size;
+ int send_buffer_used;
+};
+
+struct drbd_const_buffer {
+ const u8 *buffer;
+ unsigned int avail;
+};
+
+/**
+ * struct drbd_transport_ops - Operations implemented by the transport.
+ *
+ * The user of this API guarantees that all of the following will be exclusive
+ * with respect to each other for a given transport instance:
+ * * init()
+ * * free()
+ * * prepare_connect()
+ * * finish_connect()
+ * * add_path() and the subsequent list_add_tail_rcu() for the paths list
+ * * may_remove_path(), remove_path() and the subsequent list_del_rcu() for the paths list
+ *
+ * The connection sequence is as follows:
+ * 1. prepare_connect(), with the above exclusivity guarantee
+ * 2. connect(), this may take a long time
+ * 3. finish_connect(), with the above exclusivity guarantee
+ */
+struct drbd_transport_ops {
+ int (*init)(struct drbd_transport *);
+ void (*free)(struct drbd_transport *, enum drbd_tr_free_op free_op);
+ int (*init_listener)(struct drbd_transport *, const struct sockaddr *, struct net *net,
+ struct drbd_listener *);
+ void (*release_listener)(struct drbd_listener *);
+ int (*prepare_connect)(struct drbd_transport *);
+ int (*connect)(struct drbd_transport *);
+ void (*finish_connect)(struct drbd_transport *);
+
+/**
+ * recv() - Receive data via the transport
+ * @transport: The transport to use
+ * @stream: The stream within the transport to use. Ether DATA_STREAM or CONTROL_STREAM
+ * @buf: The function will place here the pointer to the data area
+ * @size: Number of byte to receive
+ * @msg_flags: Bitmask of CALLER_BUFFER, GROW_BUFFER and MSG_DONTWAIT
+ *
+ * recv() returns the requests data in a buffer (owned by the transport).
+ * You may pass MSG_DONTWAIT as flags. Usually with the next call to recv()
+ * or recv_pages() on the same stream, the buffer may no longer be accessed
+ * by the caller. I.e. it is reclaimed by the transport.
+ *
+ * If the transport was not capable of fulfilling the complete "wish" of the
+ * caller (that means it returned a smaller size that size), the caller may
+ * call recv() again with the flag GROW_BUFFER, and *buf as returned by the
+ * previous call.
+ * Note1: This can happen if MSG_DONTWAIT was used, or if a receive timeout
+ * was we with set_rcvtimeo().
+ * Note2: recv() is free to re-locate the buffer in such a call. I.e. to
+ * modify *buf. Then it copies the content received so far to the new
+ * memory location.
+ *
+ * Last not least the caller may also pass an arbitrary pointer in *buf with
+ * the CALLER_BUFFER flag. This is expected to be used for small amounts
+ * of data only
+ *
+ * Upon success the function returns the bytes read. Upon error the return
+ * code is negative. A 0 indicates that the socket was closed by the remote
+ * side.
+ */
+ int (*recv)(struct drbd_transport *, enum drbd_stream, void **buf, size_t size, int flags);
+
+/**
+ * recv_bio() - Receive bulk data via the transport's DATA_STREAM into bios
+ * @peer_device: Identify the transport and the device
+ * @bios: the bio_list to add received data to
+ * @size: Number of bytes to receive
+ * @misalign_bits: Out: the bitwise OR of the offset of every bvec added to
+ * @bios, and of the length of every bvec but the last
+ *
+ * recv_bio() receives the requested amount of data from DATA_STREAM. It
+ * allocates pages by using drbd_alloc_pages() and adds them to bios in the
+ * bio_list.
+ *
+ * The core submits the bvecs to a backing device whose queue dma_alignment
+ * constrains their offset and length. Reporting @misalign_bits lets it test
+ * the payload with a single mask, and copy it into aligned pages when it has
+ * to. The length of the last bvec is the payload size the core asked for; no
+ * copy can change it, so it stays out of @misalign_bits.
+ *
+ * Upon success the function returns the bytes read. Upon error the return
+ * code is negative. A 0 indicates that the socket was closed by the remote
+ * side.
+ */
+ int (*recv_bio)(struct drbd_transport *, struct bio_list *bios, size_t size,
+ unsigned int *misalign_bits);
+
+ void (*stats)(struct drbd_transport *, struct drbd_transport_stats *stats);
+/**
+ * net_conf_change() - Notify about changed network configuration on the transport.
+ * @new_net_conf: The new network configuration that should be applied.
+ *
+ * net_conf_change() is called in the context of either the initial creation of the connection,
+ * or when the net_conf is changed via netlink. Note that assignment of the net_conf to the
+ * transport object happens after this function is called.
+ *
+ * On a negative (error) return value, it is expected that any changes are reverted and
+ * the old net_conf (if any) is still in effect.
+ *
+ * Upon success the function return 0. Upon error the function returns a negative value.
+ */
+ int (*net_conf_change)(struct drbd_transport *, struct drbd_net_conf *new_net_conf);
+ void (*set_rcvtimeo)(struct drbd_transport *, enum drbd_stream, long timeout);
+ long (*get_rcvtimeo)(struct drbd_transport *, enum drbd_stream);
+ int (*send_page)(struct drbd_transport *, enum drbd_stream, struct page *,
+ int offset, size_t size, unsigned int msg_flags);
+ int (*send_bio)(struct drbd_transport *, struct bio *bio, unsigned int msg_flags);
+ bool (*stream_ok)(struct drbd_transport *, enum drbd_stream);
+ bool (*hint)(struct drbd_transport *, enum drbd_stream, enum drbd_tr_hints hint);
+ void (*debugfs_show)(struct drbd_transport *, struct seq_file *m);
+
+/**
+ * set_rx_alignment() - Tell the transport the alignment received payload needs
+ * @bytes: Alignment in bytes (a power of two, at most PAGE_SIZE), or 0 if
+ * no backing device is attached.
+ *
+ * The DRBD core submits pages received via recv_bio() to its backing devices,
+ * whose queue dma_alignment constrains the offset and length of every bvec. A
+ * transport that places received payload at sub-page offsets uses this to
+ * align it accordingly. Purely a performance hint: the core copies misaligned
+ * payload into aligned pages itself. Called whenever the set of attached
+ * backing devices changes. Optional; must not sleep.
+ */
+ void (*set_rx_alignment)(struct drbd_transport *, unsigned int bytes);
+
+/**
+ * add_path() - Prepare path to be added
+ * @path: The path that is being added
+ *
+ * Called before the path is added to the paths list.
+ *
+ * Return: 0 if path may be added, error code otherwise.
+ */
+ int (*add_path)(struct drbd_path *path);
+
+/**
+ * may_remove_path() - Query whether path may currently be removed
+ * @path: The path to be removed
+ *
+ * Return: true is path may be removed, false otherwise.
+ */
+ bool (*may_remove_path)(struct drbd_path *path);
+
+/**
+ * remove_path() - Tear down a path that is being removed
+ * @path: The path that is being removed
+ *
+ * Clean up a path that is being removed. Called with the path still on the
+ * transport's paths list and with references still held, under the same
+ * exclusivity guarantee as may_remove_path(). The transport may sleep here,
+ * e.g. to gracefully shut down an established connection, before the caller
+ * removes the path from the list and drops the remaining references.
+ */
+ void (*remove_path)(struct drbd_path *path);
+};
+
+struct drbd_transport_class {
+ const char *name;
+ const int instance_size;
+ const int path_instance_size;
+ const int listener_instance_size;
+ struct drbd_transport_ops ops;
+
+ struct module *module;
+
+ struct list_head list;
+};
+
+/*
+ * An "abstract base class" for transport implementations. I.e. it
+ * should be embedded into a transport specific representation of a
+ * listening "socket"
+ */
+struct drbd_listener {
+ struct kref kref;
+ struct drbd_resource *resource;
+ struct drbd_transport_class *transport_class;
+ struct list_head list; /* link for resource->listeners */
+ struct list_head waiters; /* list head for paths */
+ spinlock_t waiters_lock;
+ int pending_accepts;
+ struct sockaddr_storage listen_addr;
+ struct completion ready;
+ int err;
+};
+
+/* drbd_main.c */
+void drbd_destroy_path(struct kref *kref);
+
+/* drbd_transport.c */
+int drbd_register_transport_class(struct drbd_transport_class *transport_class,
+ int version, int drbd_transport_size);
+void drbd_unregister_transport_class(struct drbd_transport_class *transport_class);
+struct drbd_transport_class *drbd_get_transport_class(const char *name);
+void drbd_put_transport_class(struct drbd_transport_class *tc);
+void drbd_print_transports_loaded(struct seq_file *seq);
+
+int drbd_get_listener(struct drbd_path *path);
+void drbd_put_listener(struct drbd_path *path);
+struct drbd_listener *drbd_listener_try_get_ref(struct drbd_path *path);
+struct drbd_path *drbd_find_path_by_addr(struct drbd_listener *listener,
+ struct sockaddr_storage *addr);
+bool drbd_path_conflicts_by_listener(struct drbd_path *existing,
+ struct drbd_path *candidate);
+bool drbd_stream_send_timed_out(struct drbd_transport *transport,
+ enum drbd_stream stream);
+bool drbd_should_abort_listening(struct drbd_transport *transport);
+void drbd_path_event(struct drbd_transport *transport, struct drbd_path *path);
+void drbd_listener_destroy(struct kref *kref);
+struct drbd_path *__drbd_next_path_ref(struct drbd_path *drbd_path,
+ struct drbd_transport *transport);
+int drbd_bio_add_page(struct drbd_transport *transport, struct bio_list *bios,
+ struct page *page, unsigned int len, unsigned int offset);
+int drbd_bio_add_page_nomerge(struct drbd_transport *transport, struct bio_list *bios,
+ struct page *page, unsigned int len, unsigned int offset);
+
+void drbd_transport_lock(struct drbd_transport *transport);
+void drbd_transport_unlock(struct drbd_transport *transport);
+
+/* Might restart iteration, if current element is removed from list!! */
+#define for_each_path_ref(path, transport) \
+ for (path = __drbd_next_path_ref(NULL, transport); \
+ path; \
+ path = __drbd_next_path_ref(path, transport))
+
+/* drbd_receiver.c*/
+struct page *drbd_alloc_pages(struct drbd_transport *transport, gfp_t gfp_mask, unsigned int size);
+struct page *drbd_alloc_pages_split(struct drbd_transport *transport, gfp_t gfp_mask, int *order);
+void drbd_free_page(struct drbd_transport *transport, struct page *page);
+void drbd_get_page(struct drbd_transport *transport, struct page *page);
+void drbd_control_data_ready(struct drbd_transport *transport,
+ struct drbd_const_buffer *pool);
+void drbd_control_event(struct drbd_transport *transport,
+ enum drbd_tr_event event);
+
+/*
+ * Some helper functions to deal with our page chains.
+ */
+/* Our transports may sometimes need to only partially use a page.
+ * We need to express that somehow. Use this struct, and "graft" it into
+ * struct page at page->lru.
+ *
+ * According to include/linux/mm.h:
+ * | A page may be used by anyone else who does a __get_free_page().
+ * | In this case, page_count still tracks the references, and should only
+ * | be used through the normal accessor functions. The top bits of page->flags
+ * | and page->virtual store page management information, but all other fields
+ * | are unused and could be used privately, carefully. The management of this
+ * | page is the responsibility of the one who allocated it, and those who have
+ * | subsequently been given references to it.
+ * (we do alloc_page(), that is equivalent).
+ *
+ * Red Hat struct page is different from upstream (layout and members) :(
+ * So I am not too sure about the "all other fields", and it is not as easy to
+ * find a place where sizeof(struct drbd_page_chain) would fit on all archs and
+ * distribution-changed layouts.
+ *
+ * But (upstream) struct page also says:
+ * | struct list_head lru; * ...
+ * | * Can be used as a generic list
+ * | * by the page owner.
+ *
+ * On 32bit, use unsigned short for offset and size,
+ * to still fit in sizeof(page->lru).
+ */
+
+/* grafted over struct page.lru */
+struct drbd_page_chain {
+ struct page *next; /* next page in chain, if any */
+#ifdef CONFIG_64BIT
+ unsigned int offset; /* start offset of data within this page */
+ unsigned int size; /* number of data bytes within this page */
+#else
+#if PAGE_SIZE > (1U<<16)
+#error "won't work."
+#endif
+ unsigned short offset; /* start offset of data within this page */
+ unsigned short size; /* number of data bytes within this page */
+#endif
+};
+
+static inline void dummy_for_buildbug(void)
+{
+ struct page *dummy;
+
+ BUILD_BUG_ON(sizeof(struct drbd_page_chain) > sizeof(dummy->lru));
+}
+
+#define page_chain_next(page) \
+ (((struct drbd_page_chain *)&(page)->lru)->next)
+#define page_chain_size(page) \
+ (((struct drbd_page_chain *)&(page)->lru)->size)
+#define page_chain_offset(page) \
+ (((struct drbd_page_chain *)&(page)->lru)->offset)
+#define set_page_chain_next(page, v) \
+ (((struct drbd_page_chain *)&(page)->lru)->next = (v))
+#define set_page_chain_size(page, v) \
+ (((struct drbd_page_chain *)&(page)->lru)->size = (v))
+#define set_page_chain_offset(page, v) \
+ (((struct drbd_page_chain *)&(page)->lru)->offset = (v))
+#define set_page_chain_next_offset_size(page, n, o, s) \
+ (*((struct drbd_page_chain *)&(page)->lru) = \
+ ((struct drbd_page_chain){ \
+ .next = (n), \
+ .offset = (o), \
+ .size = (s), \
+ }))
+
+#define page_chain_for_each(page) \
+ for (; page && ({ prefetch(page_chain_next(page)); 1; }); \
+ page = page_chain_next(page))
+#define page_chain_for_each_safe(page, n) \
+ for (; page && ({ n = page_chain_next(page); 1; }); page = n)
+
+#ifndef SK_CAN_REUSE
+/*
+ * This constant was introduced by Pavel Emelyanov <xemul@parallels.com> on
+ * Thu Apr 19 03:39:36 2012 +0000. Before the release of linux-3.5
+ * commit 4a17fd52 sock: Introduce named constants for sk_reuse
+ */
+#define SK_CAN_REUSE 1
+#endif
+
+#endif
diff --git a/drivers/block/drbd/drbd_transport_template.c b/drivers/block/drbd/drbd_transport_template.c
new file mode 100644
index 000000000000..02130d424e57
--- /dev/null
+++ b/drivers/block/drbd/drbd_transport_template.c
@@ -0,0 +1,164 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2014, LINBIT HA-Solutions GmbH.
+ */
+
+#include <linux/module.h>
+#include "drbd_transport.h"
+#include "drbd_int.h"
+
+
+MODULE_AUTHOR("xxx");
+MODULE_DESCRIPTION("xxx transport layer for DRBD");
+MODULE_LICENSE("GPL");
+
+
+struct drbd_xxx_transport {
+ struct drbd_transport transport;
+ /* xxx */
+};
+
+struct xxx_listener {
+ struct drbd_listener listener;
+ /* xxx */
+};
+
+struct xxx_waiter {
+ struct drbd_waiter waiter;
+ /* xxx */
+};
+
+static struct drbd_transport *xxx_create(struct drbd_connection *connection);
+static void xxx_free(struct drbd_transport *transport, enum drbd_tr_free_op free_op);
+static int xxx_connect(struct drbd_transport *transport);
+static int xxx_recv(struct drbd_transport *transport, enum drbd_stream stream, void *buf, size_t size, int flags);
+static void xxx_stats(struct drbd_transport *transport, struct drbd_transport_stats *stats);
+static void xxx_set_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream, long timeout);
+static long xxx_get_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream);
+static int xxx_send_page(struct drbd_transport *transport, enum drbd_stream stream, struct page *page,
+ int offset, size_t size, unsigned msg_flags);
+static bool xxx_stream_ok(struct drbd_transport *transport, enum drbd_stream stream);
+static bool xxx_hint(struct drbd_transport *transport, enum drbd_stream stream, enum drbd_tr_hints hint);
+
+
+static struct drbd_transport_class xxx_transport_class = {
+ .name = "xxx",
+ .create = xxx_create,
+ .list = LIST_HEAD_INIT(xxx_transport_class.list),
+};
+
+static struct drbd_transport_ops xxx_ops = {
+ .free = xxx_free,
+ .connect = xxx_connect,
+ .recv = xxx_recv,
+ .stats = xxx_stats,
+ .set_rcvtimeo = xxx_set_rcvtimeo,
+ .get_rcvtimeo = xxx_get_rcvtimeo,
+ .send_page = xxx_send_page,
+ .stream_ok = xxx_stream_ok,
+ .hint = xxx_hint,
+};
+
+
+static struct drbd_transport *xxx_create(struct drbd_connection *connection)
+{
+ struct drbd_xxx_transport *xxx_transport;
+
+ if (!try_module_get(THIS_MODULE))
+ return NULL;
+
+ xxx_transport = kzalloc_obj(struct drbd_xxx_transport);
+ if (!xxx_transport) {
+ module_put(THIS_MODULE);
+ return NULL;
+ }
+
+ xxx_transport->transport.ops = &xxx_ops;
+ xxx_transport->transport.connection = connection;
+
+ return &xxx_transport->transport;
+}
+
+static void xxx_free(struct drbd_transport *transport, enum drbd_tr_free_op free_op)
+{
+ struct drbd_xxx_transport *xxx_transport =
+ container_of(transport, struct drbd_xxx_transport, transport);
+
+ /* disconnect here */
+
+ if (free_op == DESTROY_TRANSPORT) {
+ kfree(xxx_transport);
+ module_put(THIS_MODULE);
+ }
+}
+
+static int xxx_send(struct drbd_transport *transport, enum drbd_stream stream, void *buf, size_t size, unsigned msg_flags)
+{
+ struct drbd_xxx_transport *xxx_transport =
+ container_of(transport, struct drbd_xxx_transport, transport);
+
+ return 0;
+}
+
+static int xxx_recv(struct drbd_transport *transport, enum drbd_stream stream, void *buf, size_t size, int flags)
+{
+ struct drbd_xxx_transport *xxx_transport =
+ container_of(transport, struct drbd_xxx_transport, transport);
+
+ return 0;
+}
+
+static void xxx_stats(struct drbd_transport *transport, struct drbd_transport_stats *stats)
+{
+}
+
+static int xxx_connect(struct drbd_transport *transport)
+{
+ struct drbd_xxx_transport *xxx_transport =
+ container_of(transport, struct drbd_xxx_transport, transport);
+
+ return true;
+}
+
+static void xxx_set_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream, long timeout)
+{
+}
+
+static long xxx_get_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream)
+{
+ return 0;
+}
+
+static bool xxx_stream_ok(struct drbd_transport *transport, enum drbd_stream stream)
+{
+ return true;
+}
+
+static int xxx_send_page(struct drbd_transport *transport, enum drbd_stream stream, struct page *page,
+ int offset, size_t size, unsigned msg_flags)
+{
+ return 0;
+}
+
+static bool xxx_hint(struct drbd_transport *transport, enum drbd_stream stream,
+ enum drbd_tr_hints hint)
+{
+ switch (hint) {
+ default: /* not implemented, but should not trigger error handling */
+ return true;
+ }
+ return true;
+}
+
+static int __init xxx_init(void)
+{
+ return drbd_register_transport_class(&xxx_transport_class);
+}
+
+static void __exit xxx_cleanup(void)
+{
+ drbd_unregister_transport_class(&xxx_transport_class);
+}
+
+module_init(xxx_init)
+module_exit(xxx_cleanup)
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 05/20] drbd: add TCP transport implementation
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (3 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 04/20] drbd: add transport layer abstraction Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 06/20] drbd: add DAX/PMEM support for metadata access Christoph Böhmwalder
` (14 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Roland Kammerer,
Moritz Tanner, Joel Colledge
Add the TCP transport implementation as a standalone kernel module
(drbd_transport_tcp) that registers itself with the DRBD transport
abstraction layer.
This extracts TCP replication details to a separate component
rather than being hard-wired in the main module, so alternative
transports can coexist and be selected at runtime.
The implementation manages two independent sockets per peer connection
(data and control streams) and supports dynamic path add/remove while a
connection is live. This is the foundation for DRBD 9 multi-path
operation.
It also integrates kernel-native TLS via the in-kernel handshake API,
enabling encrypted replication.
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Moritz Tanner <moritz.tanner@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Kconfig | 11 +
drivers/block/drbd/Makefile | 1 +
drivers/block/drbd/drbd_transport_tcp.c | 1805 +++++++++++++++++++++++
3 files changed, 1817 insertions(+)
create mode 100644 drivers/block/drbd/drbd_transport_tcp.c
diff --git a/drivers/block/drbd/Kconfig b/drivers/block/drbd/Kconfig
index ab0bf1b7eb7d..49b75cbe1def 100644
--- a/drivers/block/drbd/Kconfig
+++ b/drivers/block/drbd/Kconfig
@@ -72,3 +72,14 @@ config DRBD_FAULT_INJECTION
echo 5 > /sys/module/drbd/parameters/fault_rate
If unsure, say N.
+
+config BLK_DEV_DRBD_TCP
+ tristate "DRBD TCP transport"
+ depends on BLK_DEV_DRBD
+ default BLK_DEV_DRBD
+ help
+
+ TCP transport support for DRBD. This is the standard transport
+ for DRBD replication over TCP/IP networks.
+
+ If unsure, say Y.
diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index d4fa578325dc..9f05f74e619c 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -8,3 +8,4 @@ drbd-y += drbd_transport.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
obj-$(CONFIG_BLK_DEV_DRBD) += drbd.o
+obj-$(CONFIG_BLK_DEV_DRBD_TCP) += drbd_transport_tcp.o
diff --git a/drivers/block/drbd/drbd_transport_tcp.c b/drivers/block/drbd/drbd_transport_tcp.c
new file mode 100644
index 000000000000..542de836eab3
--- /dev/null
+++ b/drivers/block/drbd/drbd_transport_tcp.c
@@ -0,0 +1,1805 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2014, LINBIT HA-Solutions GmbH.
+ */
+
+#include <linux/module.h>
+#include <linux/errno.h>
+#include <linux/socket.h>
+#include <linux/pkt_sched.h>
+#include <linux/sched/signal.h>
+#include <linux/net.h>
+#include <linux/skbuff.h>
+#include <linux/file.h>
+#include <linux/tcp.h>
+#include <linux/highmem.h>
+#include <linux/bio.h>
+#include <linux/drbd.h>
+#include "drbd_nl_types.h"
+#include "drbd_config.h"
+#include <linux/tls.h>
+#include <net/tcp.h>
+#include <net/handshake.h>
+#include <net/tls.h>
+#include <net/tls_prot.h>
+#include "drbd_protocol.h"
+#include "drbd_transport.h"
+
+
+MODULE_AUTHOR("Philipp Reisner <philipp.reisner@linbit.com>");
+MODULE_AUTHOR("Lars Ellenberg <lars.ellenberg@linbit.com>");
+MODULE_AUTHOR("Roland Kammerer <roland.kammerer@linbit.com>");
+MODULE_DESCRIPTION("TCP (SDP, SSOCKS) transport layer for DRBD");
+MODULE_LICENSE("GPL");
+MODULE_VERSION(REL_VERSION);
+
+/* TCP keepalive has proven to be vital in many deployment scenarios.
+ * Without keepalive, after a device has seen a sufficiently long period of
+ * idle time, packets on our "bulk data" socket may be dropped because an
+ * overly "smart" network infrastructure decided that TCP session was stale.
+ * Note that we don't try to use this to detect "broken" tcp sessions here,
+ * these will still be handled by the DRBD effective network timeout via
+ * timeout / ko-count settings.
+ * We use this to try to keep "idle" TCP sessions "alive".
+ * Default to send a probe every 23 seconds.
+ */
+#define DRBD_KEEP_IDLE 23
+#define DRBD_KEEP_INTVL 23
+#define DRBD_KEEP_CNT 9
+static unsigned int drbd_keepcnt = DRBD_KEEP_CNT;
+module_param_named(keepcnt, drbd_keepcnt, uint, 0664);
+MODULE_PARM_DESC(keepcnt, "see tcp(7) tcp_keepalive_probes; set TCP_KEEPCNT for data sockets; default: 9");
+static unsigned int drbd_keepidle = DRBD_KEEP_IDLE;
+module_param_named(keepidle, drbd_keepidle, uint, 0664);
+MODULE_PARM_DESC(keepidle, "see tcp(7) tcp_keepalive_time; set TCP_KEEPIDLE for data sockets; default: 23s");
+static unsigned int drbd_keepintvl = DRBD_KEEP_INTVL;
+module_param_named(keepintvl, drbd_keepintvl, uint, 0664);
+MODULE_PARM_DESC(keepintvl, "see tcp(7) tcp_keepalive_intvl; set TCP_KEEPINTVL for data sockets; default: 23s");
+
+static struct workqueue_struct *dtt_csocket_recv;
+
+struct buffer {
+ void *base;
+ void *pos;
+};
+
+#define DTT_CONNECTING 1
+#define DTT_DATA_READY_ARMED 2
+
+struct drbd_tcp_transport {
+ struct drbd_transport transport; /* Must be first! */
+ spinlock_t control_recv_lock;
+ unsigned long flags;
+ struct socket *stream[2];
+ struct buffer rbuf[2];
+ struct timer_list control_timer;
+ struct work_struct control_data_ready_work;
+ void (*original_control_sk_state_change)(struct sock *sk);
+ void (*original_control_sk_data_ready)(struct sock *sk);
+ wait_queue_head_t wait; /* woken if a connection came in on any of our paths */
+};
+
+struct dtt_listener {
+ struct drbd_listener listener;
+ void (*original_sk_state_change)(struct sock *sk);
+ struct socket *s_listen;
+};
+
+/* Since each path might have a different local IP address, each
+ path might need its own listener. Therefore the drbd_waiter object
+ is embedded into the dtt_path and _not_ the dtt_waiter. */
+
+struct dtt_socket_container {
+ struct list_head list;
+ struct socket *socket;
+};
+
+struct dtt_path {
+ struct drbd_path path;
+
+ struct list_head sockets; /* sockets passed to me by other receiver threads */
+};
+
+
+static int dtt_init(struct drbd_transport *transport);
+static void dtt_free(struct drbd_transport *transport, enum drbd_tr_free_op free_op);
+static void dtt_socket_free(struct socket **sock);
+static int dtt_init_listener(struct drbd_transport *transport, const struct sockaddr *addr,
+ struct net *net, struct drbd_listener *drbd_listener);
+static void dtt_destroy_listener(struct drbd_listener *generic_listener);
+static int dtt_prepare_connect(struct drbd_transport *transport);
+static int dtt_connect(struct drbd_transport *transport);
+static void dtt_finish_connect(struct drbd_transport *transport);
+static int dtt_recv(struct drbd_transport *transport, enum drbd_stream stream, void **buf, size_t size, int flags);
+static int dtt_recv_bio(struct drbd_transport *transport, struct bio_list *bios, size_t size,
+ unsigned int *misalign_bits);
+static void dtt_stats(struct drbd_transport *transport, struct drbd_transport_stats *stats);
+static int dtt_net_conf_change(struct drbd_transport *transport,
+ struct drbd_net_conf *new_net_conf);
+static void dtt_set_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream, long timeout);
+static long dtt_get_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream);
+static int dtt_send_page(struct drbd_transport *transport, enum drbd_stream, struct page *page,
+ int offset, size_t size, unsigned int msg_flags);
+static int dtt_send_bio(struct drbd_transport *, struct bio *bio, unsigned int msg_flags);
+static bool dtt_stream_ok(struct drbd_transport *transport, enum drbd_stream stream);
+static bool dtt_hint(struct drbd_transport *transport, enum drbd_stream stream, enum drbd_tr_hints hint);
+static void dtt_debugfs_show(struct drbd_transport *transport, struct seq_file *m);
+static void dtt_update_congested(struct drbd_tcp_transport *tcp_transport);
+static int dtt_add_path(struct drbd_path *path);
+static bool dtt_may_remove_path(struct drbd_path *);
+static void dtt_remove_path(struct drbd_path *);
+static void dtt_control_timer_fn(struct timer_list *t);
+
+static struct drbd_transport_class tcp_transport_class = {
+ .name = "tcp",
+ .instance_size = sizeof(struct drbd_tcp_transport),
+ .path_instance_size = sizeof(struct dtt_path),
+ .listener_instance_size = sizeof(struct dtt_listener),
+ .ops = (struct drbd_transport_ops) {
+ .init = dtt_init,
+ .free = dtt_free,
+ .init_listener = dtt_init_listener,
+ .release_listener = dtt_destroy_listener,
+ .prepare_connect = dtt_prepare_connect,
+ .connect = dtt_connect,
+ .finish_connect = dtt_finish_connect,
+ .recv = dtt_recv,
+ .recv_bio = dtt_recv_bio,
+ .stats = dtt_stats,
+ .net_conf_change = dtt_net_conf_change,
+ .set_rcvtimeo = dtt_set_rcvtimeo,
+ .get_rcvtimeo = dtt_get_rcvtimeo,
+ .send_page = dtt_send_page,
+ .send_bio = dtt_send_bio,
+ .stream_ok = dtt_stream_ok,
+ .hint = dtt_hint,
+ .debugfs_show = dtt_debugfs_show,
+ .add_path = dtt_add_path,
+ .may_remove_path = dtt_may_remove_path,
+ .remove_path = dtt_remove_path,
+ },
+ .module = THIS_MODULE,
+ .list = LIST_HEAD_INIT(tcp_transport_class.list),
+};
+
+static int dtt_init(struct drbd_transport *transport)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ enum drbd_stream i;
+
+ spin_lock_init(&tcp_transport->control_recv_lock);
+ init_waitqueue_head(&tcp_transport->wait);
+ tcp_transport->transport.class = &tcp_transport_class;
+ for (i = DATA_STREAM; i <= CONTROL_STREAM ; i++) {
+ void *buffer = (void *)__get_free_page(GFP_KERNEL);
+ if (!buffer)
+ goto fail;
+ tcp_transport->rbuf[i].base = buffer;
+ tcp_transport->rbuf[i].pos = buffer;
+ }
+ timer_setup(&tcp_transport->control_timer, dtt_control_timer_fn, 0);
+
+ return 0;
+fail:
+ free_page((unsigned long)tcp_transport->rbuf[0].base);
+ return -ENOMEM;
+}
+
+static void dtt_free(struct drbd_transport *transport, enum drbd_tr_free_op free_op)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ enum drbd_stream i;
+ struct drbd_path *drbd_path;
+ /* free the socket specific stuff,
+ * mutexes are handled by caller */
+
+ clear_bit(DTT_DATA_READY_ARMED, &tcp_transport->flags);
+
+ if (tcp_transport->control_data_ready_work.func) {
+ cancel_work_sync(&tcp_transport->control_data_ready_work);
+ tcp_transport->control_data_ready_work.func = NULL;
+ }
+
+ if (tcp_transport->stream[CONTROL_STREAM] &&
+ tcp_transport->original_control_sk_state_change) {
+ write_lock_bh(&tcp_transport->stream[CONTROL_STREAM]->sk->sk_callback_lock);
+ tcp_transport->stream[CONTROL_STREAM]->sk->sk_state_change =
+ tcp_transport->original_control_sk_state_change;
+ write_unlock_bh(&tcp_transport->stream[CONTROL_STREAM]->sk->sk_callback_lock);
+ }
+
+ synchronize_rcu();
+ for (i = DATA_STREAM; i <= CONTROL_STREAM; i++) {
+ dtt_socket_free(&tcp_transport->stream[i]);
+ }
+
+ list_for_each_entry(drbd_path, &transport->paths, list) {
+ bool was_established = test_and_clear_bit(TR_ESTABLISHED, &drbd_path->flags);
+
+ if (free_op == CLOSE_CONNECTION && was_established)
+ drbd_path_event(transport, drbd_path);
+ }
+
+ timer_delete_sync(&tcp_transport->control_timer);
+
+ if (free_op == DESTROY_TRANSPORT) {
+ for (i = DATA_STREAM; i <= CONTROL_STREAM; i++) {
+ free_page((unsigned long)tcp_transport->rbuf[i].base);
+ tcp_transport->rbuf[i].base = NULL;
+ }
+ }
+}
+
+static int _dtt_send(struct drbd_tcp_transport *tcp_transport, struct socket *socket,
+ void *buf, size_t size, unsigned msg_flags)
+{
+ struct kvec iov;
+ struct msghdr msg;
+ int rv, sent = 0;
+
+ iov.iov_base = buf;
+ iov.iov_len = size;
+
+ msg.msg_name = NULL;
+ msg.msg_namelen = 0;
+ msg.msg_control = NULL;
+ msg.msg_controllen = 0;
+ msg.msg_flags = msg_flags | MSG_NOSIGNAL;
+
+ do {
+ rv = kernel_sendmsg(socket, &msg, &iov, 1, iov.iov_len);
+ if (rv == -EAGAIN) {
+ struct drbd_transport *transport = &tcp_transport->transport;
+ enum drbd_stream stream =
+ tcp_transport->stream[DATA_STREAM] == socket ?
+ DATA_STREAM : CONTROL_STREAM;
+
+ if (drbd_stream_send_timed_out(transport, stream))
+ break;
+ else
+ continue;
+ }
+ if (rv < 0)
+ break;
+ sent += rv;
+ iov.iov_base += rv;
+ iov.iov_len -= rv;
+ } while (sent < size);
+
+ if (rv <= 0)
+ return rv;
+
+ return sent;
+}
+
+static int dtt_recv_short(struct socket *socket, void *buf, size_t size, int flags)
+{
+ struct kvec iov = {
+ .iov_base = buf,
+ .iov_len = size,
+ };
+ union {
+ struct cmsghdr cmsg;
+ u8 buf[CMSG_SPACE(sizeof(u8))];
+ } u;
+ struct msghdr msg = {
+ .msg_control = &u,
+ .msg_controllen = sizeof(u),
+ };
+ int ret;
+
+ flags = flags ? flags : MSG_WAITALL | MSG_NOSIGNAL;
+
+ ret = kernel_recvmsg(socket, &msg, &iov, 1, size, flags);
+
+ if (msg.msg_controllen != sizeof(u)) {
+ u8 level, description;
+
+ switch (tls_get_record_type(socket->sk, &u.cmsg)) {
+ case 0:
+ fallthrough;
+ case TLS_RECORD_TYPE_DATA:
+ break;
+ case TLS_RECORD_TYPE_ALERT:
+ tls_alert_recv(socket->sk, &msg, &level, &description);
+ ret = (level == TLS_ALERT_LEVEL_FATAL) ? -EACCES : -EAGAIN;
+ break;
+ default:
+ /* discard this record type */
+ ret = -EAGAIN;
+ break;
+ }
+ }
+
+ return ret;
+}
+
+static int dtt_recv(struct drbd_transport *transport, enum drbd_stream stream, void **buf, size_t size, int flags)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct socket *socket = tcp_transport->stream[stream];
+ void *buffer;
+ int rv;
+
+ if (!socket)
+ return -ENOTCONN;
+
+ if (flags & CALLER_BUFFER) {
+ buffer = *buf;
+ rv = dtt_recv_short(socket, buffer, size, flags & ~CALLER_BUFFER);
+ } else if (flags & GROW_BUFFER) {
+ TR_ASSERT(transport, *buf == tcp_transport->rbuf[stream].base);
+ buffer = tcp_transport->rbuf[stream].pos;
+ TR_ASSERT(transport, (buffer - *buf) + size <= PAGE_SIZE);
+
+ rv = dtt_recv_short(socket, buffer, size, flags & ~GROW_BUFFER);
+ } else {
+ buffer = tcp_transport->rbuf[stream].base;
+
+ rv = dtt_recv_short(socket, buffer, size, flags);
+ if (rv > 0)
+ *buf = buffer;
+ }
+
+ if (rv > 0)
+ tcp_transport->rbuf[stream].pos = buffer + rv;
+
+ return rv;
+}
+
+
+static int dtt_recv_bio(struct drbd_transport *transport, struct bio_list *bios, size_t size,
+ unsigned int *misalign_bits)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct socket *socket = tcp_transport->stream[DATA_STREAM];
+ size_t remaining = size;
+ struct page *page;
+ int err;
+
+ *misalign_bits = 0;
+ if (!socket)
+ return -ENOTCONN;
+
+ do {
+ size_t len;
+
+ page = drbd_alloc_pages(transport, GFP_KERNEL, remaining);
+ if (!page)
+ return -ENOMEM;
+ len = min(PAGE_SIZE << compound_order(page), remaining);
+
+ err = dtt_recv_short(socket, page_address(page), len, 0);
+ if (err <= 0)
+ goto fail;
+ remaining -= err;
+ err = drbd_bio_add_page(transport, bios, page, len, 0);
+ if (err < 0)
+ goto fail;
+ } while (remaining > 0);
+
+ return size;
+
+fail:
+ drbd_free_page(transport, page);
+ return err;
+}
+
+static void dtt_stats(struct drbd_transport *transport, struct drbd_transport_stats *stats)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+
+ struct socket *socket = tcp_transport->stream[DATA_STREAM];
+
+ if (socket) {
+ struct sock *sk = socket->sk;
+ struct tcp_sock *tp = tcp_sk(sk);
+
+ stats->unread_received = tp->rcv_nxt - tp->copied_seq;
+ stats->unacked_send = tp->write_seq - tp->snd_una;
+ stats->send_buffer_size = sk->sk_sndbuf;
+ stats->send_buffer_used = sk->sk_wmem_queued;
+ }
+}
+
+static void dtt_setbufsize(struct socket *socket, unsigned int snd,
+ unsigned int rcv)
+{
+ struct sock *sk = socket->sk;
+
+ /* open coded SO_SNDBUF, SO_RCVBUF */
+ if (snd) {
+ sk->sk_sndbuf = snd;
+ sk->sk_userlocks |= SOCK_SNDBUF_LOCK;
+ /* Wake up sending tasks if we upped the value. */
+ sk->sk_write_space(sk);
+ } else {
+ sk->sk_userlocks &= ~SOCK_SNDBUF_LOCK;
+ }
+
+ if (rcv) {
+ sk->sk_rcvbuf = rcv;
+ sk->sk_userlocks |= SOCK_RCVBUF_LOCK;
+ } else {
+ sk->sk_userlocks &= ~SOCK_RCVBUF_LOCK;
+ }
+}
+
+static bool dtt_path_cmp_addr(struct dtt_path *path)
+{
+ struct drbd_path *drbd_path = &path->path;
+ int addr_size;
+
+ addr_size = min(drbd_path->my_addr_len, drbd_path->peer_addr_len);
+ return memcmp(&drbd_path->my_addr, &drbd_path->peer_addr, addr_size) > 0;
+}
+
+/* Translate transient socket errors (connect timeout, peer reset, etc.)
+ * to -EAGAIN so the caller restarts the connect cycle instead of treating
+ * the error as fatal. Other errors are returned unchanged.
+ */
+static int dtt_socket_err_to_eagain(int err)
+{
+ switch (err) {
+ case -ETIMEDOUT:
+ case -EINPROGRESS:
+ case -EINTR:
+ case -ERESTARTSYS:
+ case -ECONNREFUSED:
+ case -ECONNRESET:
+ case -ENETUNREACH:
+ case -EHOSTDOWN:
+ case -EHOSTUNREACH:
+ return -EAGAIN;
+ default:
+ return err;
+ }
+}
+
+static int dtt_try_connect(struct dtt_path *path, struct socket **ret_socket)
+{
+ struct drbd_transport *transport = path->path.transport;
+ const char *what;
+ struct socket *socket;
+ struct sockaddr_storage my_addr, peer_addr;
+ struct drbd_net_conf *nc;
+ int err;
+ int sndbuf_size, rcvbuf_size, connect_int;
+
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ if (!nc) {
+ rcu_read_unlock();
+ return -EIO;
+ }
+ sndbuf_size = nc->sndbuf_size;
+ rcvbuf_size = nc->rcvbuf_size;
+ connect_int = nc->connect_int;
+ rcu_read_unlock();
+
+ my_addr = path->path.my_addr;
+ if (my_addr.ss_family == AF_INET6)
+ ((struct sockaddr_in6 *)&my_addr)->sin6_port = 0;
+ else
+ ((struct sockaddr_in *)&my_addr)->sin_port = 0; /* AF_INET & AF_SCI */
+
+ /* In some cases, the network stack can end up overwriting
+ peer_addr.ss_family, so use a copy here. */
+ peer_addr = path->path.peer_addr;
+
+ what = "sock_create_kern";
+ err = sock_create_kern(path->path.net, my_addr.ss_family, SOCK_STREAM, IPPROTO_TCP, &socket);
+ if (err < 0) {
+ socket = NULL;
+ goto out;
+ }
+
+ socket->sk->sk_rcvtimeo =
+ socket->sk->sk_sndtimeo = connect_int * HZ;
+ dtt_setbufsize(socket, sndbuf_size, rcvbuf_size);
+
+ /* explicitly bind to the configured IP as source IP
+ * for the outgoing connections.
+ * This is needed for multihomed hosts and to be
+ * able to use lo: interfaces for drbd.
+ * Make sure to use 0 as port number, so linux selects
+ * a free one dynamically.
+ */
+ what = "bind before connect";
+ err = socket->ops->bind(socket, (struct sockaddr_unsized *) &my_addr,
+ path->path.my_addr_len);
+ if (err < 0)
+ goto out;
+
+ /* connect may fail, peer not yet available.
+ * stay C_CONNECTING, don't go Disconnecting! */
+ what = "connect";
+ err = socket->ops->connect(socket, (struct sockaddr_unsized *) &peer_addr,
+ path->path.peer_addr_len, 0);
+ if (err < 0) {
+ err = dtt_socket_err_to_eagain(err);
+ if (err == -EINVAL)
+ err = -EADDRNOTAVAIL;
+ }
+
+out:
+ if (err < 0) {
+ if (socket)
+ sock_release(socket);
+ if (err != -EAGAIN && err != -EADDRNOTAVAIL)
+ tr_err(transport, "%s failed, err = %d\n", what, err);
+ else
+ tr_dbg(transport, "connect %pISpc -> %pISpc: %d\n",
+ &path->path.my_addr, &peer_addr, err);
+ } else {
+ *ret_socket = socket;
+ }
+
+ return err;
+}
+
+typedef int (*tls_hello_func)(const struct tls_handshake_args *, gfp_t);
+
+struct tls_handshake_wait {
+ struct completion done;
+ struct sock *sk;
+ int status;
+};
+
+static void tls_handshake_done(void *data, int status, key_serial_t peerid)
+{
+ struct tls_handshake_wait *wait = data;
+
+ // Normalize the error to be negative: while the error _should_ be negative
+ // it is not guaranteed: the netlink interface allows any u32 value, which is
+ // then negated and cast to int, so who knows what will be returned.
+ if (status > 0)
+ status = -status;
+
+ /* tlshd reports a reset connection as -EACCES, like a rejected certificate.
+ * The node that rejects a certificate still has an established connection.
+ */
+ if (status && READ_ONCE(wait->sk->sk_state) != TCP_ESTABLISHED)
+ status = -ECONNRESET;
+
+ wait->status = status;
+ complete(&wait->done);
+}
+
+static int tls_init_hello(struct socket *sock, const char *peername,
+ key_serial_t keyring, key_serial_t privkey,
+ key_serial_t certificate, tls_hello_func hello,
+ struct tls_handshake_wait *tls_wait)
+{
+ int err;
+ struct tls_handshake_args tls_args = {
+ .ta_sock = sock,
+ .ta_done = tls_handshake_done,
+ .ta_data = tls_wait,
+ .ta_peername = peername,
+ .ta_keyring = keyring,
+ .ta_my_privkey = privkey,
+ .ta_my_cert = certificate,
+ };
+
+ if (IS_ERR(sock_alloc_file(sock, O_NONBLOCK, NULL)))
+ return -EIO;
+
+ tls_wait->sk = sock->sk;
+
+ do {
+ err = hello(&tls_args, GFP_KERNEL);
+ } while (err == -EAGAIN);
+
+ return err;
+}
+
+static int tls_wait_hello(struct tls_handshake_wait *csocket_tls_wait,
+ struct tls_handshake_wait *dsocket_tls_wait,
+ unsigned long timeout)
+{
+ unsigned long remaining = wait_for_completion_timeout(
+ &csocket_tls_wait->done, timeout);
+ if (!remaining)
+ return -ETIMEDOUT;
+
+ if (!wait_for_completion_timeout(&dsocket_tls_wait->done, remaining))
+ return -ETIMEDOUT;
+
+ /* A fatal error on either socket outranks a transient one. */
+ if (dtt_socket_err_to_eagain(csocket_tls_wait->status) != -EAGAIN)
+ return csocket_tls_wait->status ?: dsocket_tls_wait->status;
+
+ return dsocket_tls_wait->status ?: csocket_tls_wait->status;
+}
+
+
+static int dtt_send_first_packet(struct drbd_tcp_transport *tcp_transport, struct socket *socket,
+ enum drbd_packet cmd)
+{
+ struct p_header80 h;
+
+ h.magic = cpu_to_be32(DRBD_MAGIC);
+ h.command = cpu_to_be16(cmd);
+ h.length = 0;
+
+ return _dtt_send(tcp_transport, socket, &h, sizeof(h), 0);
+}
+
+static void dtt_dbg_socket(struct drbd_transport *transport, struct socket *socket,
+ const char *what)
+{
+ struct sockaddr_storage me = {}, peer = {};
+
+ socket->ops->getname(socket, (struct sockaddr *)&me, 0);
+ socket->ops->getname(socket, (struct sockaddr *)&peer, 2);
+ tr_dbg(transport, "%s %pISpc -> %pISpc, state %d\n",
+ what, &me, &peer, socket->sk->sk_state);
+}
+
+/**
+ * dtt_socket_free() - Free the socket
+ * @socket: pointer to the pointer to the socket.
+ */
+static void dtt_socket_free(struct socket **socket)
+{
+ if (!*socket)
+ return;
+
+ tls_handshake_cancel((*socket)->sk);
+ kernel_sock_shutdown(*socket, SHUT_RDWR);
+
+ if ((*socket)->file)
+ sockfd_put((*socket));
+ else
+ sock_release(*socket);
+
+ *socket = NULL;
+}
+
+/**
+ * dtt_socket_ok_or_free() - Free the socket if its connection is not okay
+ * @socket: pointer to the pointer to the socket.
+ */
+static bool dtt_socket_ok_or_free(struct drbd_transport *transport,
+ struct socket **socket, const char *what)
+{
+ if (!*socket)
+ return false;
+
+ if ((*socket)->sk->sk_state == TCP_ESTABLISHED)
+ return true;
+
+ dtt_dbg_socket(transport, *socket, what);
+ dtt_socket_free(socket);
+ return false;
+}
+
+static bool dtt_connection_established(struct drbd_transport *transport,
+ struct socket **dsocket,
+ struct socket **csocket,
+ struct dtt_path **first_path)
+{
+ struct drbd_net_conf *nc;
+ int timeout, good = 0;
+
+ if (!*dsocket || !*csocket)
+ return false;
+
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ timeout = (nc->sock_check_timeo ?: nc->ping_timeo) * HZ / 10;
+ rcu_read_unlock();
+ schedule_timeout_interruptible(timeout);
+
+ good += dtt_socket_ok_or_free(transport, dsocket, "drop data socket");
+ good += dtt_socket_ok_or_free(transport, csocket, "drop control socket");
+
+ if (good == 0) {
+ kref_put(&(*first_path)->path.kref, drbd_destroy_path);
+ *first_path = NULL;
+ }
+
+ return good == 2;
+}
+
+static struct dtt_path *dtt_wait_connect_cond(struct drbd_transport *transport)
+{
+ struct drbd_listener *listener;
+ struct drbd_path *drbd_path;
+ struct dtt_path *path = NULL;
+ bool rv = false;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(drbd_path, &transport->paths, list) {
+ path = container_of(drbd_path, struct dtt_path, path);
+
+ /* Pairs with xchg() in drbd_put_listener() */
+ listener = READ_ONCE(drbd_path->listener);
+ if (!listener)
+ continue;
+
+ spin_lock_bh(&listener->waiters_lock);
+ rv = listener->pending_accepts > 0 || !list_empty(&path->sockets);
+ spin_unlock_bh(&listener->waiters_lock);
+
+ if (rv)
+ break;
+ }
+ if (rv)
+ kref_get(&path->path.kref);
+ rcu_read_unlock();
+
+ return rv ? path : NULL;
+}
+
+static void unregister_state_change(struct sock *sock, struct dtt_listener *listener)
+{
+ write_lock_bh(&sock->sk_callback_lock);
+ sock->sk_state_change = listener->original_sk_state_change;
+ sock->sk_user_data = NULL;
+ write_unlock_bh(&sock->sk_callback_lock);
+}
+
+static bool dtt_connect_pending(struct drbd_transport *transport)
+{
+ struct dtt_path *path = dtt_wait_connect_cond(transport);
+
+ if (path)
+ kref_put(&path->path.kref, drbd_destroy_path);
+ return path != NULL;
+}
+
+static int dtt_accept_connection(struct drbd_transport *transport,
+ struct socket **socket,
+ struct dtt_path **ret_path, bool nowait)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct dtt_socket_container *socket_c;
+ struct sockaddr_storage peer_addr;
+ int connect_int, err = 0;
+ long timeo;
+ struct socket *s_estab = NULL;
+ struct drbd_net_conf *nc;
+ struct drbd_path *drbd_path2;
+ struct drbd_listener *drbd_listener;
+ struct dtt_listener *listener;
+ struct dtt_path *path = NULL;
+
+ /* Protect the listener from a concurrent del-path (drbd_put_listener()) */
+ drbd_listener = drbd_listener_try_get_ref(&(*ret_path)->path);
+ if (!drbd_listener)
+ return -EAGAIN;
+
+ listener = container_of(drbd_listener, struct dtt_listener, listener);
+
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ if (!nc) {
+ rcu_read_unlock();
+ err = -EINVAL;
+ goto out;
+ }
+ connect_int = nc->connect_int;
+ rcu_read_unlock();
+
+ timeo = connect_int * HZ;
+ timeo += get_random_u32_below(2) ? timeo / 7 : -timeo / 7; /* 28.5% random jitter */
+ if (nowait)
+ timeo = 0;
+
+retry:
+ if (path)
+ kref_put(&path->path.kref, drbd_destroy_path);
+ timeo = wait_event_interruptible_timeout(tcp_transport->wait,
+ (path = dtt_wait_connect_cond(transport)),
+ timeo);
+ if (!path) {
+ err = -EAGAIN;
+ goto out;
+ }
+
+ spin_lock_bh(&listener->listener.waiters_lock);
+ socket_c = list_first_entry_or_null(&path->sockets, struct dtt_socket_container, list);
+ if (socket_c) {
+ s_estab = socket_c->socket;
+ list_del(&socket_c->list);
+ kfree(socket_c);
+ } else if (listener->listener.pending_accepts > 0) {
+ listener->listener.pending_accepts--;
+ spin_unlock_bh(&listener->listener.waiters_lock);
+
+ s_estab = NULL;
+ err = kernel_accept(listener->s_listen, &s_estab, O_NONBLOCK);
+ if (err < 0) {
+ kref_put(&path->path.kref, drbd_destroy_path);
+ goto out;
+ }
+
+ /* The established socket inherits the sk_state_change callback
+ from the listening socket. */
+ unregister_state_change(s_estab->sk, listener);
+
+ s_estab->ops->getname(s_estab, (struct sockaddr *)&peer_addr, 2);
+
+ spin_lock_bh(&listener->listener.waiters_lock);
+ drbd_path2 = drbd_find_path_by_addr(&listener->listener, &peer_addr);
+ if (!drbd_path2) {
+ struct sockaddr_in6 *from_sin6;
+ struct sockaddr_in *from_sin;
+
+ switch (peer_addr.ss_family) {
+ case AF_INET6:
+ from_sin6 = (struct sockaddr_in6 *)&peer_addr;
+ tr_notice(transport, "Closing unexpected connection from "
+ "%pI6\n", &from_sin6->sin6_addr);
+ break;
+ default:
+ from_sin = (struct sockaddr_in *)&peer_addr;
+ tr_notice(transport, "Closing unexpected connection from "
+ "%pI4\n", &from_sin->sin_addr);
+ break;
+ }
+
+ goto retry_locked;
+ }
+ if (drbd_path2 != &path->path) {
+ struct dtt_path *path2 =
+ container_of(drbd_path2, struct dtt_path, path);
+ struct drbd_tcp_transport *path2_tcp_transport =
+ container_of(drbd_path2->transport,
+ struct drbd_tcp_transport, transport);
+
+ socket_c = kmalloc_obj(*socket_c, GFP_ATOMIC);
+ if (!socket_c) {
+ tr_info(transport, /* path2->transport, */
+ "No mem, dropped an incoming connection\n");
+ goto retry_locked;
+ }
+
+ socket_c->socket = s_estab;
+ s_estab = NULL;
+ list_add_tail(&socket_c->list, &path2->sockets);
+ wake_up(&path2_tcp_transport->wait);
+ goto retry_locked;
+ }
+ if (s_estab->sk->sk_state != TCP_ESTABLISHED) {
+ tr_dbg(transport, "drop accepted socket from %pISpc, state %d\n",
+ &peer_addr, s_estab->sk->sk_state);
+ goto retry_locked;
+ }
+ }
+ spin_unlock_bh(&listener->listener.waiters_lock);
+ *socket = s_estab;
+ if (*ret_path)
+ kref_put(&(*ret_path)->path.kref, drbd_destroy_path);
+ *ret_path = path;
+ err = 0;
+ goto out;
+
+retry_locked:
+ spin_unlock_bh(&listener->listener.waiters_lock);
+ dtt_socket_free(&s_estab);
+ goto retry;
+
+out:
+ kref_put(&drbd_listener->kref, drbd_listener_destroy);
+ return err;
+}
+
+static int dtt_receive_first_packet(struct drbd_tcp_transport *tcp_transport, struct socket *socket)
+{
+ struct drbd_transport *transport = &tcp_transport->transport;
+ struct p_header80 *h = tcp_transport->rbuf[DATA_STREAM].base;
+ const unsigned int header_size = sizeof(*h);
+ struct drbd_net_conf *nc;
+ int err;
+
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ if (!nc) {
+ rcu_read_unlock();
+ return -EIO;
+ }
+ socket->sk->sk_rcvtimeo = nc->ping_timeo * 4 * HZ / 10;
+ rcu_read_unlock();
+
+ err = dtt_recv_short(socket, h, header_size, 0);
+ if (err != header_size) {
+ if (err >= 0)
+ err = -EIO;
+ return err;
+ }
+ if (h->magic != cpu_to_be32(DRBD_MAGIC)) {
+ tr_err(transport, "Wrong magic value 0x%08x in receive_first_packet\n",
+ be32_to_cpu(h->magic));
+ return -EINVAL;
+ }
+ return be16_to_cpu(h->command);
+}
+
+
+static int dtt_control_tcp_input(read_descriptor_t *rd_desc, struct sk_buff *skb,
+ unsigned int offset, size_t len)
+{
+ struct drbd_transport *transport = rd_desc->arg.data;
+ unsigned int avail, consumed = 0;
+ struct skb_seq_state seq;
+
+ skb_prepare_seq_read(skb, offset, offset + len, &seq);
+ do {
+ struct drbd_const_buffer buffer;
+
+ /*
+ * skb_seq_read() returns the length of the block assigned to buffer. This might
+ * be more than is actually ready, so we ensure we only mark as available what
+ * is ready.
+ */
+ avail = skb_seq_read(consumed, &buffer.buffer, &seq);
+ if (!avail)
+ break;
+ buffer.avail = min_t(unsigned int, avail, len - consumed);
+ consumed += buffer.avail;
+ drbd_control_data_ready(transport, &buffer);
+ } while (consumed < len);
+ skb_abort_seq_read(&seq);
+
+ return consumed;
+}
+
+static void dtt_control_data_ready_work(struct work_struct *item)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(item, struct drbd_tcp_transport, control_data_ready_work);
+ struct socket *csocket = tcp_transport->stream[CONTROL_STREAM];
+ struct drbd_const_buffer drbd_buffer;
+ int n;
+
+ while (true) {
+ n = dtt_recv_short(csocket, tcp_transport->rbuf[CONTROL_STREAM].base, PAGE_SIZE,
+ MSG_DONTWAIT | MSG_NOSIGNAL);
+ if (n == 0) {
+ /* Orderly shutdown: the peer closed the meta socket, and
+ * the close surfaced here as EOF rather than (or before) a
+ * state-change callback. Do not silently drop it -- treat
+ * it as a peer-initiated disconnect, mirroring
+ * dtt_control_state_change(). Otherwise the connection
+ * lingers until the ping timeout, or forever on a transport
+ * that reports a close only through the receive path.
+ */
+ drbd_control_event(&tcp_transport->transport, CLOSED_BY_PEER);
+ break;
+ }
+ if (n < 0)
+ break; /* -EAGAIN: nothing more available right now */
+
+ drbd_buffer.buffer = tcp_transport->rbuf[CONTROL_STREAM].base;
+ drbd_buffer.avail = n;
+ drbd_control_data_ready(&tcp_transport->transport, &drbd_buffer);
+ }
+}
+
+static void dtt_control_data_ready(struct sock *sock)
+{
+ struct drbd_transport *transport = sock->sk_user_data;
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+
+ read_descriptor_t rd_desc = {
+ .count = 1,
+ .arg = { .data = transport },
+ };
+
+ if (!test_bit(DTT_DATA_READY_ARMED, &tcp_transport->flags)
+ && tcp_transport->original_control_sk_data_ready)
+ return tcp_transport->original_control_sk_data_ready(sock);
+
+ /* We have two different paths depending on if TLS is enabled or not.
+ * If TLS is enabled, we can't use read_sock, firstly because it's not implemented for the
+ * TLS protocol on most kernels, secondly the implementation that does exist is not safe
+ * to call from SOFTIRQ context. Instead, we schedule a work and increment the counter of
+ * "pending" ready events.
+ *
+ * In normal TCP mode, we can simply use tcp_read_sock, as that is safe to call from SOFTIRQ
+ * contexts.
+ */
+ mod_timer(&tcp_transport->control_timer, jiffies + sock->sk_rcvtimeo);
+ if (tcp_transport->control_data_ready_work.func) {
+ queue_work(dtt_csocket_recv, &tcp_transport->control_data_ready_work);
+ } else {
+ spin_lock_bh(&tcp_transport->control_recv_lock);
+ tcp_read_sock(sock, &rd_desc, dtt_control_tcp_input);
+ spin_unlock_bh(&tcp_transport->control_recv_lock);
+ }
+}
+
+static void dtt_control_state_change(struct sock *sock)
+{
+ struct drbd_transport *transport = sock->sk_user_data;
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+
+ switch (sock->sk_state) {
+ case TCP_FIN_WAIT1:
+ case TCP_CLOSE_WAIT:
+ case TCP_CLOSE:
+ case TCP_LAST_ACK:
+ case TCP_CLOSING:
+ drbd_control_event(transport, CLOSED_BY_PEER);
+ break;
+ default:
+ tr_warn(transport, "unhandled state %d\n", sock->sk_state);
+ }
+
+ tcp_transport->original_control_sk_state_change(sock);
+}
+
+static void dtt_incoming_connection(struct sock *sock)
+{
+ struct dtt_listener *listener = sock->sk_user_data;
+ struct drbd_path *drbd_path;
+ void (*state_change)(struct sock *sock);
+
+ state_change = listener->original_sk_state_change;
+ state_change(sock);
+
+ spin_lock(&listener->listener.waiters_lock);
+ listener->listener.pending_accepts++;
+ list_for_each_entry(drbd_path, &listener->listener.waiters, listener_link) {
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(drbd_path->transport,
+ struct drbd_tcp_transport, transport);
+ wake_up(&tcp_transport->wait);
+ }
+ spin_unlock(&listener->listener.waiters_lock);
+}
+
+static void dtt_control_timer_fn(struct timer_list *t)
+{
+ struct drbd_tcp_transport *tcp_transport = timer_container_of(tcp_transport, t,
+ control_timer);
+ struct drbd_transport *transport = &tcp_transport->transport;
+
+ drbd_control_event(transport, TIMEOUT);
+}
+
+static void dtt_destroy_listener(struct drbd_listener *generic_listener)
+{
+ struct dtt_listener *listener =
+ container_of(generic_listener, struct dtt_listener, listener);
+
+ if (!listener->s_listen)
+ return;
+ unregister_state_change(listener->s_listen->sk, listener);
+ sock_release(listener->s_listen);
+}
+
+static int dtt_init_listener(struct drbd_transport *transport,
+ const struct sockaddr *addr,
+ struct net *net,
+ struct drbd_listener *drbd_listener)
+{
+ int err, sndbuf_size, rcvbuf_size, addr_len;
+ struct sockaddr_storage my_addr;
+ struct dtt_listener *listener = container_of(drbd_listener, struct dtt_listener, listener);
+ struct socket *s_listen;
+ struct drbd_net_conf *nc;
+ const char *what = "";
+
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ if (!nc) {
+ rcu_read_unlock();
+ return -EINVAL;
+ }
+ sndbuf_size = nc->sndbuf_size;
+ rcvbuf_size = nc->rcvbuf_size;
+ rcu_read_unlock();
+
+ my_addr = *(struct sockaddr_storage *)addr;
+
+ err = sock_create_kern(net, my_addr.ss_family, SOCK_STREAM, IPPROTO_TCP, &s_listen);
+ if (err < 0) {
+ s_listen = NULL;
+ what = "sock_create_kern";
+ goto out;
+ }
+
+ s_listen->sk->sk_reuse = SK_CAN_REUSE; /* SO_REUSEADDR */
+ dtt_setbufsize(s_listen, sndbuf_size, rcvbuf_size);
+
+ addr_len = addr->sa_family == AF_INET6 ? sizeof(struct sockaddr_in6)
+ : sizeof(struct sockaddr_in);
+
+ err = s_listen->ops->bind(s_listen, (struct sockaddr_unsized *)&my_addr, addr_len);
+ if (err < 0) {
+ what = "bind before listen";
+ goto out;
+ }
+
+ listener->s_listen = s_listen;
+ write_lock_bh(&s_listen->sk->sk_callback_lock);
+ listener->original_sk_state_change = s_listen->sk->sk_state_change;
+ s_listen->sk->sk_state_change = dtt_incoming_connection;
+ s_listen->sk->sk_user_data = listener;
+ write_unlock_bh(&s_listen->sk->sk_callback_lock);
+
+ err = s_listen->ops->listen(s_listen, DRBD_PEERS_MAX * 2);
+ if (err < 0) {
+ what = "listen";
+ goto out;
+ }
+
+ listener->listener.listen_addr = my_addr;
+
+ return 0;
+out:
+ if (s_listen)
+ sock_release(s_listen);
+
+ if (err < 0 &&
+ err != -EAGAIN && err != -EINTR && err != -ERESTARTSYS && err != -EADDRINUSE &&
+ err != -EADDRNOTAVAIL)
+ tr_err(transport, "%s failed, err = %d\n", what, err);
+
+ return err;
+}
+
+static void dtt_cleanup_accepted_sockets(struct dtt_path *path)
+{
+ while (!list_empty(&path->sockets)) {
+ struct dtt_socket_container *socket_c =
+ list_first_entry(&path->sockets, struct dtt_socket_container, list);
+
+ list_del(&socket_c->list);
+ dtt_socket_free(&socket_c->socket);
+ kfree(socket_c);
+ }
+}
+
+static void dtt_finish_connect(struct drbd_transport *transport)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct dtt_path *path;
+
+ clear_bit(DTT_CONNECTING, &tcp_transport->flags);
+
+ list_for_each_entry(path, &transport->paths, path.list) {
+ drbd_put_listener(&path->path);
+ dtt_cleanup_accepted_sockets(path);
+ }
+}
+
+static struct dtt_path *dtt_next_path(struct dtt_path *path, struct drbd_transport *transport)
+{
+ struct drbd_path *drbd_path;
+
+ drbd_path = __drbd_next_path_ref(path ? &path->path : NULL, transport);
+
+ /* Loop when we reach the end. */
+ if (!drbd_path)
+ drbd_path = __drbd_next_path_ref(NULL, transport);
+
+ return drbd_path ? container_of(drbd_path, struct dtt_path, path) : NULL;
+}
+
+static int dtt_prepare_connect(struct drbd_transport *transport)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct dtt_path *path;
+ struct drbd_path *drbd_path;
+
+ list_for_each_entry(path, &transport->paths, path.list)
+ dtt_cleanup_accepted_sockets(path);
+
+ set_bit(DTT_CONNECTING, &tcp_transport->flags);
+
+ list_for_each_entry(drbd_path, &transport->paths, list) {
+ if (!drbd_path->listener) {
+ int err = drbd_get_listener(drbd_path);
+
+ if (err)
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+static int dtt_connect(struct drbd_transport *transport)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct dtt_path *connect_to_path, *first_path = NULL;
+ struct socket *dsocket, *csocket;
+ struct drbd_net_conf *nc;
+ bool tls, dsocket_is_server = false, csocket_is_server = false;
+ char peername[64];
+ key_serial_t tls_keyring, tls_privkey, tls_certificate;
+ int timeout, err;
+ bool ok, nowait = false;
+ struct tls_handshake_wait csocket_tls_wait = { .status = 0 };
+ struct tls_handshake_wait dsocket_tls_wait = { .status = 0 };
+
+ dsocket = NULL;
+ csocket = NULL;
+
+ connect_to_path = dtt_next_path(NULL, transport);
+ if (!connect_to_path) {
+ err = -EDESTADDRREQ;
+ goto out;
+ }
+
+ do {
+ struct socket *s = NULL;
+
+ /* The peer may already use these as its data and control socket */
+ if (!nowait && dtt_connect_pending(transport)) {
+ nowait = true;
+ goto retry;
+ }
+ nowait = false;
+
+ err = dtt_try_connect(connect_to_path, &s);
+ if (err < 0 && err != -EAGAIN)
+ goto out_release_sockets;
+
+ if (s) {
+ bool use_for_data;
+
+ if (first_path) {
+ if (first_path != connect_to_path) {
+ tr_info(transport, "initial paths crossed A - fail over\n");
+ dtt_socket_free(&dsocket);
+ dtt_socket_free(&csocket);
+ }
+
+ kref_put(&first_path->path.kref, drbd_destroy_path);
+ first_path = NULL;
+ }
+
+ kref_get(&connect_to_path->path.kref);
+ first_path = connect_to_path;
+
+ if (!dsocket && !csocket) {
+ use_for_data = dtt_path_cmp_addr(first_path);
+ } else if (!dsocket) {
+ use_for_data = true;
+ } else {
+ if (csocket) {
+ tr_err(transport, "Logic error in conn_connect()\n");
+ goto out_eagain;
+ }
+ use_for_data = false;
+ }
+
+ if (!use_for_data)
+ clear_bit(RESOLVE_CONFLICTS, &transport->flags);
+
+ err = dtt_send_first_packet(tcp_transport,
+ s,
+ use_for_data ? P_INITIAL_DATA : P_INITIAL_META);
+
+ if (err < 0) {
+ tr_warn(transport, "Error sending initial packet: %d\n", err);
+ dtt_socket_free(&s);
+ } else if (use_for_data) {
+ dsocket = s;
+ dsocket_is_server = false;
+ dtt_dbg_socket(transport, s, "data socket, client");
+ } else {
+ csocket = s;
+ csocket_is_server = false;
+ dtt_dbg_socket(transport, s, "control socket, client");
+ }
+ } else if (!first_path) {
+ connect_to_path = dtt_next_path(connect_to_path, transport);
+
+ /*
+ * The final path should not be removed while
+ * connecting, but handle the case for robustness.
+ */
+ err = -EDESTADDRREQ;
+ if (!connect_to_path)
+ goto out_release_sockets;
+ }
+
+ if (dtt_connection_established(transport, &dsocket, &csocket, &first_path))
+ break;
+
+retry:
+ s = NULL;
+ err = dtt_accept_connection(transport, &s, &connect_to_path, nowait);
+ if (s)
+ nowait = false;
+ if (err < 0 && err != -EAGAIN)
+ goto out_release_sockets;
+
+ if (s) {
+ int fp = dtt_receive_first_packet(tcp_transport, s);
+
+ dtt_dbg_socket(transport, s, fp == P_INITIAL_DATA ? "data socket, server" :
+ fp == P_INITIAL_META ? "control socket, server" :
+ "accepted socket without initial packet");
+
+ if (first_path) {
+ if (first_path != connect_to_path) {
+ tr_info(transport, "initial paths crossed P - fail over\n");
+ dtt_socket_free(&dsocket);
+ dtt_socket_free(&csocket);
+ }
+
+ kref_put(&first_path->path.kref, drbd_destroy_path);
+ first_path = NULL;
+ }
+
+ kref_get(&connect_to_path->path.kref);
+ first_path = connect_to_path;
+
+ dtt_socket_ok_or_free(transport, &dsocket, "drop data socket");
+ dtt_socket_ok_or_free(transport, &csocket, "drop control socket");
+ switch (fp) {
+ case P_INITIAL_DATA:
+ if (dsocket) {
+ tr_warn(transport, "initial packet S crossed\n");
+ dtt_socket_free(&dsocket);
+ dsocket = s;
+ dsocket_is_server = true;
+ goto randomize;
+ }
+ dsocket = s;
+ dsocket_is_server = true;
+ break;
+ case P_INITIAL_META:
+ set_bit(RESOLVE_CONFLICTS, &transport->flags);
+ if (csocket) {
+ tr_warn(transport, "initial packet M crossed\n");
+ dtt_socket_free(&csocket);
+ csocket = s;
+ csocket_is_server = true;
+ goto randomize;
+ }
+ csocket = s;
+ csocket_is_server = true;
+ break;
+ default:
+ tr_warn(transport, "Error receiving initial packet: %d\n", fp);
+ dtt_socket_free(&s);
+randomize:
+ if (get_random_u32_below(2))
+ goto retry;
+ }
+ }
+
+ if (drbd_should_abort_listening(transport))
+ goto out_eagain;
+
+ ok = dtt_connection_established(transport, &dsocket, &csocket, &first_path);
+ } while (!ok);
+
+ dtt_dbg_socket(transport, dsocket, dsocket_is_server ? "use data socket, server" :
+ "use data socket, client");
+ dtt_dbg_socket(transport, csocket, csocket_is_server ? "use control socket, server" :
+ "use control socket, client");
+
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ timeout = nc->timeout * HZ / 10;
+ tls = nc->tls;
+ memcpy(peername, nc->name, 64);
+ tls_keyring = nc->tls_keyring;
+ tls_privkey = nc->tls_privkey;
+ tls_certificate = nc->tls_certificate;
+ rcu_read_unlock();
+
+ write_lock_bh(&csocket->sk->sk_callback_lock);
+ clear_bit(DTT_DATA_READY_ARMED, &tcp_transport->flags);
+ tcp_transport->original_control_sk_data_ready = csocket->sk->sk_data_ready;
+ csocket->sk->sk_user_data = transport;
+ csocket->sk->sk_data_ready = dtt_control_data_ready;
+ write_unlock_bh(&csocket->sk->sk_callback_lock);
+
+ if (tls) {
+ csocket_tls_wait.done = COMPLETION_INITIALIZER_ONSTACK(csocket_tls_wait.done);
+ dsocket_tls_wait.done = COMPLETION_INITIALIZER_ONSTACK(dsocket_tls_wait.done);
+
+ err = tls_init_hello(
+ csocket, peername, tls_keyring, tls_privkey, tls_certificate,
+ csocket_is_server ? tls_server_hello_x509 : tls_client_hello_x509,
+ &csocket_tls_wait);
+ if (err < 0) {
+ tr_warn(transport, "Error from control socket tls handshake: %d\n", err);
+ err = dtt_socket_err_to_eagain(err);
+ goto out_release_sockets;
+ }
+
+ err = tls_init_hello(
+ dsocket, peername, tls_keyring, tls_privkey, tls_certificate,
+ dsocket_is_server ? tls_server_hello_x509 : tls_client_hello_x509,
+ &dsocket_tls_wait);
+ if (err < 0) {
+ tr_warn(transport, "Error from data socket tls handshake: %d\n", err);
+ err = dtt_socket_err_to_eagain(err);
+ goto out_release_sockets;
+ }
+
+ err = tls_wait_hello(&csocket_tls_wait, &dsocket_tls_wait, timeout);
+ if (err < 0) {
+ tr_warn(transport, "Error from tls handshake: %d\n", err);
+ err = dtt_socket_err_to_eagain(err);
+ goto out_release_sockets;
+ }
+
+ INIT_WORK(&tcp_transport->control_data_ready_work, dtt_control_data_ready_work);
+ }
+
+ TR_ASSERT(transport, first_path == connect_to_path);
+ set_bit(TR_ESTABLISHED, &connect_to_path->path.flags);
+ drbd_path_event(transport, &connect_to_path->path);
+
+ dsocket->sk->sk_reuse = SK_CAN_REUSE; /* SO_REUSEADDR */
+ csocket->sk->sk_reuse = SK_CAN_REUSE; /* SO_REUSEADDR */
+
+ /* We are a block device, we are in the write-out path,
+ * we may need memory to facilitate memory reclaim
+ */
+ dsocket->sk->sk_allocation = GFP_ATOMIC;
+ csocket->sk->sk_allocation = GFP_ATOMIC;
+
+ dsocket->sk->sk_use_task_frag = false;
+ csocket->sk->sk_use_task_frag = false;
+
+ sk_set_memalloc(dsocket->sk);
+ sk_set_memalloc(csocket->sk);
+
+ dsocket->sk->sk_priority = TC_PRIO_INTERACTIVE_BULK;
+ csocket->sk->sk_priority = TC_PRIO_INTERACTIVE;
+
+ /* NOT YET ...
+ * sock.socket->sk->sk_sndtimeo = transport->net_conf->timeout*HZ/10;
+ * sock.socket->sk->sk_rcvtimeo = MAX_SCHEDULE_TIMEOUT;
+ * first set it to the P_CONNECTION_FEATURES timeout,
+ * which we set to 4x the configured ping_timeout. */
+
+ /* we don't want delays.
+ * we use tcp_sock_set_cork where appropriate, though */
+ tcp_sock_set_nodelay(dsocket->sk);
+ tcp_sock_set_nodelay(csocket->sk);
+
+ tcp_transport->stream[DATA_STREAM] = dsocket;
+ tcp_transport->stream[CONTROL_STREAM] = csocket;
+
+ dsocket->sk->sk_sndtimeo = timeout;
+ csocket->sk->sk_sndtimeo = timeout;
+
+ sock_set_keepalive(dsocket->sk);
+
+ if (drbd_keepidle)
+ tcp_sock_set_keepidle(dsocket->sk, drbd_keepidle);
+ if (drbd_keepcnt)
+ tcp_sock_set_keepcnt(dsocket->sk, drbd_keepcnt);
+ if (drbd_keepintvl)
+ tcp_sock_set_keepintvl(dsocket->sk, drbd_keepintvl);
+
+ write_lock_bh(&csocket->sk->sk_callback_lock);
+ tcp_transport->original_control_sk_state_change = csocket->sk->sk_state_change;
+ csocket->sk->sk_state_change = dtt_control_state_change;
+ set_bit(DTT_DATA_READY_ARMED, &tcp_transport->flags);
+ write_unlock_bh(&csocket->sk->sk_callback_lock);
+
+ err = 0;
+ goto out;
+
+out_eagain:
+ err = -EAGAIN;
+
+out_release_sockets:
+ dtt_socket_free(&dsocket);
+ dtt_socket_free(&csocket);
+
+out:
+ if (first_path)
+ kref_put(&first_path->path.kref, drbd_destroy_path);
+ if (connect_to_path)
+ kref_put(&connect_to_path->path.kref, drbd_destroy_path);
+
+ return err;
+}
+
+static int dtt_net_conf_change(struct drbd_transport *transport,
+ struct drbd_net_conf *new_net_conf)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct drbd_net_conf *old_net_conf;
+ struct socket *data_socket = tcp_transport->stream[DATA_STREAM];
+ struct socket *control_socket = tcp_transport->stream[CONTROL_STREAM];
+
+ rcu_read_lock();
+ old_net_conf = rcu_dereference(transport->net_conf);
+ rcu_read_unlock();
+
+ if (old_net_conf && old_net_conf->tls != new_net_conf->tls &&
+ (data_socket || control_socket)) {
+ tr_warn(transport, "cannot switch tls (%s -> %s) while connected\n",
+ old_net_conf->tls ? "yes" : "no", new_net_conf->tls ? "yes" : "no");
+ return -EINVAL;
+ }
+
+ if (data_socket) {
+ dtt_setbufsize(data_socket, new_net_conf->sndbuf_size, new_net_conf->rcvbuf_size);
+ }
+
+ if (control_socket) {
+ dtt_setbufsize(control_socket, new_net_conf->sndbuf_size, new_net_conf->rcvbuf_size);
+ }
+
+ return 0;
+}
+
+static void dtt_set_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream, long timeout)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct socket *socket = tcp_transport->stream[stream];
+
+ if (!socket)
+ return;
+
+ socket->sk->sk_rcvtimeo = timeout;
+
+ if (stream == CONTROL_STREAM)
+ mod_timer(&tcp_transport->control_timer, jiffies + timeout);
+
+}
+
+static long dtt_get_rcvtimeo(struct drbd_transport *transport, enum drbd_stream stream)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct socket *socket = tcp_transport->stream[stream];
+
+ if (!socket)
+ return -ENOTCONN;
+
+ return socket->sk->sk_rcvtimeo;
+}
+
+static bool dtt_stream_ok(struct drbd_transport *transport, enum drbd_stream stream)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct socket *socket = tcp_transport->stream[stream];
+
+ return socket && socket->sk;
+}
+
+static void dtt_update_congested(struct drbd_tcp_transport *tcp_transport)
+{
+ struct socket *socket = tcp_transport->stream[DATA_STREAM];
+ struct sock *sock;
+
+ if (!socket)
+ return;
+
+ sock = socket->sk;
+ if (sock->sk_wmem_queued > sock->sk_sndbuf * 4 / 5)
+ set_bit(NET_CONGESTED, &tcp_transport->transport.flags);
+}
+
+static int dtt_send_page(struct drbd_transport *transport, enum drbd_stream stream,
+ struct page *page, int offset, size_t size, unsigned msg_flags)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct socket *socket = tcp_transport->stream[stream];
+ struct drbd_net_conf *nc;
+ struct msghdr msg;
+ struct bio_vec bvec;
+ int len = size;
+ int err = -EIO;
+ bool tls;
+
+ if (!socket)
+ return -ENOTCONN;
+
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ tls = nc && nc->tls;
+ rcu_read_unlock();
+
+ /* With kTLS the payload is encrypted into freshly allocated TLS records,
+ * so there is no zero-copy to preserve. Doing so would cause page ref bugs.
+ */
+ if (tls)
+ msg_flags &= ~MSG_SPLICE_PAGES;
+
+ msg = (struct msghdr) { .msg_flags = msg_flags | MSG_NOSIGNAL };
+
+ dtt_update_congested(tcp_transport);
+ do {
+ int sent;
+
+ if (tls) {
+ /* Map and send at most one page at a time. */
+ struct page *p = page + (offset >> PAGE_SHIFT);
+ unsigned int poff = offset & ~PAGE_MASK;
+ unsigned int chunk = min_t(int, len, PAGE_SIZE - poff);
+ void *addr = kmap_local_page(p);
+ struct kvec iov = { .iov_base = addr + poff, .iov_len = chunk };
+
+ iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &iov, 1, chunk);
+ sent = sock_sendmsg(socket, &msg);
+ kunmap_local(addr);
+ } else {
+ bvec_set_page(&bvec, page, len, offset);
+ iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len);
+
+ sent = sock_sendmsg(socket, &msg);
+ }
+ if (sent <= 0) {
+ if (sent == -EAGAIN) {
+ if (drbd_stream_send_timed_out(transport, stream))
+ break;
+ continue;
+ }
+ tr_warn(transport, "%s: size=%d len=%d sent=%d\n",
+ __func__, (int)size, len, sent);
+ if (sent < 0)
+ err = sent;
+ break;
+ }
+ len -= sent;
+ offset += sent;
+ /* NOTE: it may take up to twice the socket timeout to have it
+ * return -EAGAIN, the first timeout will likely happen with a
+ * partial send, masking the timeout. Maybe we want to export
+ * drbd_stream_should_continue_after_partial_send(transport, stream)
+ * and add that to the while() condition below.
+ */
+ } while (len > 0 /* THINK && peer_device->repl_state[NOW] >= L_ESTABLISHED */);
+ clear_bit(NET_CONGESTED, &tcp_transport->transport.flags);
+
+ if (len == 0)
+ err = 0;
+
+ return err;
+}
+
+static int dtt_send_bio(struct drbd_transport *transport, struct bio *bio, unsigned int msg_flags)
+{
+ struct bio_vec bvec;
+ struct bvec_iter iter;
+
+ bio_for_each_segment(bvec, bio, iter) {
+ int err;
+
+ err = dtt_send_page(transport, DATA_STREAM, bvec.bv_page,
+ bvec.bv_offset, bvec.bv_len,
+ msg_flags | (bio_iter_last(bvec, iter) ? 0 : MSG_MORE));
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
+static bool dtt_hint(struct drbd_transport *transport, enum drbd_stream stream,
+ enum drbd_tr_hints hint)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ bool rv = true;
+ struct socket *socket = tcp_transport->stream[stream];
+
+ if (!socket)
+ return false;
+
+ switch (hint) {
+ case CORK:
+ tcp_sock_set_cork(socket->sk, true);
+ break;
+ case UNCORK:
+ tcp_sock_set_cork(socket->sk, false);
+ break;
+ case NODELAY:
+ tcp_sock_set_nodelay(socket->sk);
+ break;
+ case NOSPACE:
+ sk_set_nospace(socket->sk);
+ break;
+ case QUICKACK:
+ tcp_sock_set_quickack(socket->sk, 2);
+ break;
+ default: /* not implemented, but should not trigger error handling */
+ return true;
+ }
+
+ return rv;
+}
+
+static void dtt_debugfs_show_stream(struct seq_file *m, struct socket *socket)
+{
+ struct sock *sk = socket->sk;
+ struct tcp_sock *tp = tcp_sk(sk);
+
+ seq_printf(m, "unread receive buffer: %u Byte\n",
+ tp->rcv_nxt - tp->copied_seq);
+ seq_printf(m, "unacked send buffer: %u Byte\n",
+ tp->write_seq - tp->snd_una);
+ seq_printf(m, "send buffer size: %u Byte\n", sk->sk_sndbuf);
+ seq_printf(m, "send buffer used: %u Byte\n", sk->sk_wmem_queued);
+}
+
+static void dtt_debugfs_show(struct drbd_transport *transport, struct seq_file *m)
+{
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ enum drbd_stream i;
+
+ /* BUMP me if you change the file format/content/presentation */
+ seq_printf(m, "v: %u\n\n", 0);
+
+ for (i = DATA_STREAM; i <= CONTROL_STREAM ; i++) {
+ struct socket *socket = tcp_transport->stream[i];
+
+ if (socket) {
+ seq_printf(m, "%s stream\n", i == DATA_STREAM ? "data" : "control");
+ dtt_debugfs_show_stream(m, socket);
+ }
+ }
+
+}
+
+static int dtt_add_path(struct drbd_path *drbd_path)
+{
+ struct drbd_transport *transport = drbd_path->transport;
+ struct drbd_tcp_transport *tcp_transport =
+ container_of(transport, struct drbd_tcp_transport, transport);
+ struct dtt_path *path = container_of(drbd_path, struct dtt_path, path);
+
+ clear_bit(TR_ESTABLISHED, &drbd_path->flags);
+ INIT_LIST_HEAD(&path->sockets);
+
+ if (!test_bit(DTT_CONNECTING, &tcp_transport->flags))
+ return 0;
+
+ return drbd_get_listener(drbd_path);
+}
+
+static bool dtt_may_remove_path(struct drbd_path *drbd_path)
+{
+ return !test_bit(TR_ESTABLISHED, &drbd_path->flags);
+}
+
+static void dtt_remove_path(struct drbd_path *drbd_path)
+{
+ drbd_put_listener(drbd_path);
+}
+
+static int __init dtt_initialize(void)
+{
+ dtt_csocket_recv = alloc_workqueue("dtt_csocket_recv",
+ WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
+ if (!dtt_csocket_recv)
+ return -ENOMEM;
+ return drbd_register_transport_class(&tcp_transport_class,
+ DRBD_TRANSPORT_API_VERSION,
+ sizeof(struct drbd_transport));
+}
+
+static void __exit dtt_cleanup(void)
+{
+ destroy_workqueue(dtt_csocket_recv);
+ drbd_unregister_transport_class(&tcp_transport_class);
+}
+
+module_init(dtt_initialize)
+module_exit(dtt_cleanup)
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 06/20] drbd: add DAX/PMEM support for metadata access
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (4 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 05/20] drbd: add TCP transport implementation Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 07/20] drbd: add optional compatibility layer for DRBD 8.4 Christoph Böhmwalder
` (13 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Dan Williams,
Vishal Verma, Dave Jiang, Alison Schofield, nvdimm
When DRBD's metadata device resides on persistent memory (PMEM/NVDIMM),
accessing it by reading and writing full blocks is unnecessarily
costly.
Add a DAX-based metadata path that directly maps the metadata region,
enabling byte-granular, IRQ-safe access without having to go through
the block layer.
The PMEM path also introduces a more efficient activity log layout:
instead of writing journal transactions, the in-memory LRU-cache hash
table is stored directly in persistent memory and updated in-place.
Similarly, the resync bitmap is accessed directly from PMEM rather than
being loaded into and flushed from DRAM.
This is compiled in only when CONFIG_DEV_DAX_PMEM is enabled.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Makefile | 1 +
drivers/block/drbd/drbd_dax_pmem.c | 175 +++++++++++++++++++++++++++++
drivers/block/drbd/drbd_dax_pmem.h | 43 +++++++
3 files changed, 219 insertions(+)
create mode 100644 drivers/block/drbd/drbd_dax_pmem.c
create mode 100644 drivers/block/drbd/drbd_dax_pmem.h
diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index 9f05f74e619c..8ec1ad360449 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -5,6 +5,7 @@ drbd-y += drbd_main.o drbd_strings.o drbd_nl.o
drbd-y += drbd_interval.o drbd_state.o
drbd-y += drbd_nl_gen.o
drbd-y += drbd_transport.o
+drbd-$(CONFIG_DEV_DAX_PMEM) += drbd_dax_pmem.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
obj-$(CONFIG_BLK_DEV_DRBD) += drbd.o
diff --git a/drivers/block/drbd/drbd_dax_pmem.c b/drivers/block/drbd/drbd_dax_pmem.c
new file mode 100644
index 000000000000..1fb22e57cf5d
--- /dev/null
+++ b/drivers/block/drbd/drbd_dax_pmem.c
@@ -0,0 +1,175 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2017, LINBIT HA-Solutions GmbH.
+ */
+
+/*
+ * In case DRBD's meta-data resides in persistent memory do a few things
+ * different.
+ *
+ * 1 direct access the bitmap in place. Do not load it into DRAM, do not
+ * write it back from DRAM.
+ * 2 Use a better fitting format for the on-disk activity log. Instead of
+ * writing transactions, the unmangled LRU-cache hash table is there.
+ */
+
+#include <linux/vmalloc.h>
+#include <linux/slab.h>
+#include <linux/dax.h>
+#include <linux/libnvdimm.h>
+#include <linux/blkdev.h>
+#include <linux/sizes.h>
+#include "drbd_int.h"
+#include "drbd_dax_pmem.h"
+#include "drbd_meta_data.h"
+
+/*
+ * Unit conversions for addressing the metadata area through DAX.
+ * DRBD keeps metadata offsets in sectors, dax_direct_access() works in pages.
+ */
+static pgoff_t md_sector_to_pgoff(sector_t sector)
+{
+ return sector >> (PAGE_SHIFT - SECTOR_SHIFT);
+}
+
+static long md_sectors_to_pages(sector_t nr_sectors)
+{
+ return nr_sectors >> (PAGE_SHIFT - SECTOR_SHIFT);
+}
+
+/* Signed, since the bitmap offset relative to the superblock may be negative. */
+static long md_sector_offset_to_bytes(long sector_offset)
+{
+ return sector_offset * SECTOR_SIZE;
+}
+
+static int map_superblock_for_dax(struct drbd_backing_dev *bdev, struct dax_device *dax_dev)
+{
+ long want = 1;
+ pgoff_t pgoff = md_sector_to_pgoff(bdev->md.md_offset);
+ void *kaddr;
+ long len;
+ int id;
+
+ id = dax_read_lock();
+ len = dax_direct_access(dax_dev, pgoff, want, DAX_ACCESS, &kaddr, NULL);
+ dax_read_unlock(id);
+
+ if (len < want)
+ return -EIO;
+
+ bdev->md_on_pmem = kaddr;
+
+ return 0;
+}
+
+/**
+ * drbd_dax_open() - Open device for dax and map metadata superblock
+ * @bdev: backing device to be opened
+ */
+int drbd_dax_open(struct drbd_backing_dev *bdev)
+{
+ struct dax_device *dax_dev;
+ int err;
+ u64 part_off;
+
+ dax_dev = fs_dax_get_by_bdev(bdev->md_bdev, &part_off, NULL, NULL);
+ if (!dax_dev)
+ return -ENODEV;
+
+ err = map_superblock_for_dax(bdev, dax_dev);
+ if (!err)
+ bdev->dax_dev = dax_dev;
+ else
+ put_dax(dax_dev);
+
+ return err;
+}
+
+void drbd_dax_close(struct drbd_backing_dev *bdev)
+{
+ put_dax(bdev->dax_dev);
+}
+
+/**
+ * drbd_dax_map() - Map metadata for dax
+ * @bdev: backing device whose metadata is to be mapped
+ */
+int drbd_dax_map(struct drbd_backing_dev *bdev)
+{
+ struct dax_device *dax_dev = bdev->dax_dev;
+ sector_t first_sector = drbd_md_first_sector(bdev);
+ sector_t al_sector = bdev->md.md_offset + bdev->md.al_offset;
+ long want = md_sectors_to_pages(drbd_md_last_sector(bdev) + 1 - first_sector);
+ pgoff_t pgoff = md_sector_to_pgoff(first_sector);
+ long md_offset_byte = md_sector_offset_to_bytes(bdev->md.md_offset - first_sector);
+ long al_offset_byte = md_sector_offset_to_bytes(al_sector - first_sector);
+ void *kaddr;
+ long len;
+ int id;
+
+ id = dax_read_lock();
+ len = dax_direct_access(dax_dev, pgoff, want, DAX_ACCESS, &kaddr, NULL);
+ dax_read_unlock(id);
+
+ if (len < want)
+ return -EIO;
+
+ bdev->md_on_pmem = kaddr + md_offset_byte;
+ bdev->al_on_pmem = kaddr + al_offset_byte;
+
+ return 0;
+}
+
+void drbd_dax_al_update(struct drbd_device *device, struct lc_element *al_ext)
+{
+ struct al_on_pmem *al_on_pmem = device->ldev->al_on_pmem;
+ __be32 *slot = &al_on_pmem->slots[al_ext->lc_index];
+
+ *slot = cpu_to_be32(al_ext->lc_new_number);
+ arch_wb_cache_pmem(slot, sizeof(*slot));
+}
+
+void drbd_dax_al_begin_io_commit(struct drbd_device *device)
+{
+ struct lc_element *e;
+
+ spin_lock_irq(&device->al_lock);
+
+ list_for_each_entry(e, &device->act_log->to_be_changed, list)
+ drbd_dax_al_update(device, e);
+
+ lc_committed(device->act_log);
+
+ spin_unlock_irq(&device->al_lock);
+}
+
+int drbd_dax_al_initialize(struct drbd_device *device)
+{
+ struct al_on_pmem *al_on_pmem = device->ldev->al_on_pmem;
+ __be32 *slots = al_on_pmem->slots;
+ /* The activity log area holds the magic followed by the slots. */
+ int i, al_slots = device->ldev->md.al_size_4k * (SZ_4K / sizeof(*slots)) - 1;
+
+ al_on_pmem->magic = cpu_to_be32(DRBD_AL_PMEM_MAGIC);
+ /*
+ * Initialize all slots rather than just the configured number in case
+ * the configuration is later changed.
+ */
+ for (i = 0; i < al_slots; i++) {
+ unsigned int extent_nr = i < device->act_log->nr_elements ?
+ lc_element_by_index(device->act_log, i)->lc_number :
+ LC_FREE;
+ slots[i] = cpu_to_be32(extent_nr);
+ }
+
+ return 0;
+}
+
+void *drbd_dax_bitmap(struct drbd_device *device, unsigned long want)
+{
+ struct drbd_backing_dev *bdev = device->ldev;
+ unsigned char *md_on_pmem = (unsigned char *)bdev->md_on_pmem;
+
+ return md_on_pmem + md_sector_offset_to_bytes(bdev->md.bm_offset);
+}
diff --git a/drivers/block/drbd/drbd_dax_pmem.h b/drivers/block/drbd/drbd_dax_pmem.h
new file mode 100644
index 000000000000..52b116300122
--- /dev/null
+++ b/drivers/block/drbd/drbd_dax_pmem.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2017, LINBIT HA-Solutions GmbH.
+ */
+#ifndef DRBD_DAX_H
+#define DRBD_DAX_H
+
+#include <linux/kconfig.h>
+
+#if IS_ENABLED(CONFIG_DEV_DAX_PMEM)
+
+int drbd_dax_open(struct drbd_backing_dev *bdev);
+void drbd_dax_close(struct drbd_backing_dev *bdev);
+int drbd_dax_map(struct drbd_backing_dev *bdev);
+void drbd_dax_al_update(struct drbd_device *device, struct lc_element *al_ext);
+void drbd_dax_al_begin_io_commit(struct drbd_device *device);
+int drbd_dax_al_initialize(struct drbd_device *device);
+void *drbd_dax_bitmap(struct drbd_device *device, unsigned long want);
+
+static inline bool drbd_md_dax_active(struct drbd_backing_dev *bdev)
+{
+ return bdev->dax_dev != NULL;
+}
+static inline struct meta_data_on_disk_9 *drbd_dax_md_addr(struct drbd_backing_dev *bdev)
+{
+ return bdev->md_on_pmem;
+}
+#else
+
+#define drbd_dax_open(B) do { } while (0)
+#define drbd_dax_close(B) do { } while (0)
+#define drbd_dax_map(B) (-ENOTSUPP)
+#define drbd_dax_al_begin_io_commit(D) do { } while (0)
+#define drbd_dax_al_initialize(D) (-EIO)
+#define drbd_dax_bitmap(D, L) (NULL)
+#define drbd_md_dax_active(B) (false)
+#define drbd_dax_md_addr(B) (NULL)
+
+#define arch_wb_cache_pmem(A, L) do { } while (0)
+
+#endif /* IS_ENABLED(CONFIG_DEV_DAX_PMEM) */
+
+#endif /* DRBD_DAX_H */
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 07/20] drbd: add optional compatibility layer for DRBD 8.4
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (5 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 06/20] drbd: add DAX/PMEM support for metadata access Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 08/20] drbd: add application/resync IO synchronization documentation Christoph Böhmwalder
` (12 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Joel Colledge
Introduce a Kconfig option, DRBD_COMPAT_84, that enables a
self-contained source module that encapsulates everything needed to
interoperate with the userspace and on-disk formats of DRBD 8.4 (the
version that has been shipped with the kernel until now).
This ensures compatibility with existing userspace tooling is kept.
The main DRBD code is deliberately kept as clean as possible from any
backward compatibility hacks, so that this acts as the "main switch" for
enabling compatibility with old DRBD userspace utilities.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Kconfig | 26 ++
drivers/block/drbd/Makefile | 1 +
drivers/block/drbd/drbd_legacy_84.c | 641 ++++++++++++++++++++++++++++
drivers/block/drbd/drbd_legacy_84.h | 116 +++++
4 files changed, 784 insertions(+)
create mode 100644 drivers/block/drbd/drbd_legacy_84.c
create mode 100644 drivers/block/drbd/drbd_legacy_84.h
diff --git a/drivers/block/drbd/Kconfig b/drivers/block/drbd/Kconfig
index 49b75cbe1def..1b8b3f6efc1b 100644
--- a/drivers/block/drbd/Kconfig
+++ b/drivers/block/drbd/Kconfig
@@ -83,3 +83,29 @@ config BLK_DEV_DRBD_TCP
for DRBD replication over TCP/IP networks.
If unsure, say Y.
+
+config DRBD_COMPAT_84
+ bool "Enable legacy (8.4) /proc/drbd and metadata compatibility"
+ depends on BLK_DEV_DRBD
+ default y
+ help
+
+ This option enables the DRBD-8.4 style representation of
+ DRBD devices in /proc/drbd. With DRBD 9.0, released in 2015,
+ we deprecated this interface. The replacement is `drbdsetup
+ status [--json] <resname>`. The new interface is optionally
+ machine-readable, extensible, and scales to 1000s of
+ resources. The deprecated interface had issues with all
+ three areas.
+ At the same time, this option also enables the DRBD driver
+ to read and write the deprecated 8.4 version of the
+ metadata.
+ Only when you create a DRBD resource in the legacy way
+ `drbdsetup new-resource <resname>` (no node-id positional
+ argument), then the resource is put into legacy mode, and
+ shows up in /proc/drbd, is capable of reading and writing
+ the 8.4 metadata, and can have only a single peer. Without
+ this compile-time option, creating a resource without a node
+ ID is not possible.
+
+ If unsure, say Y.
diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index 8ec1ad360449..f184b99d6a93 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -7,6 +7,7 @@ drbd-y += drbd_nl_gen.o
drbd-y += drbd_transport.o
drbd-$(CONFIG_DEV_DAX_PMEM) += drbd_dax_pmem.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
+drbd-$(CONFIG_DRBD_COMPAT_84) += drbd_legacy_84.o
obj-$(CONFIG_BLK_DEV_DRBD) += drbd.o
obj-$(CONFIG_BLK_DEV_DRBD_TCP) += drbd_transport_tcp.o
diff --git a/drivers/block/drbd/drbd_legacy_84.c b/drivers/block/drbd/drbd_legacy_84.c
new file mode 100644
index 000000000000..0c2f0851f58a
--- /dev/null
+++ b/drivers/block/drbd/drbd_legacy_84.c
@@ -0,0 +1,641 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2025, LINBIT HA-Solutions GmbH.
+ */
+
+#include "drbd_legacy_84.h"
+#include "drbd_meta_data.h"
+
+/* MDF_84_* masks and the flags table they encode are declared in
+ * drbd_legacy_84.h, shared with drbd_nl_84.c (GET_STATUS's disk_flags).
+ */
+
+struct meta_data_on_disk_84 {
+ u64 la_size_sect; /* last agreed size. */
+ u64 uuid[UI_SIZE]; /* UUIDs. */
+ u64 device_uuid;
+ u64 reserved_u64_1;
+ u32 flags; /* MDF */
+ u32 magic;
+ u32 md_size_sect;
+ u32 al_offset; /* offset to this block */
+ u32 al_nr_extents; /* important for restoring the AL (userspace) */
+ /* `-- act_log->nr_elements <-- ldev->dc.al_extents */
+ u32 bm_offset; /* offset to the bitmap, from here */
+ u32 bm_bytes_per_bit; /* 4k. Treat as magic number, must keep it compatible. */
+ u32 la_peer_max_bio_size; /* last peer max_bio_size */
+
+ /* see al_tr_number_to_on_disk_sector() */
+ u32 al_stripes;
+ u32 al_stripe_size_4k;
+
+ u8 reserved_u8[4096 - (7*8 + 10*4)];
+} __packed;
+
+
+static const char * const drbd_conn_s_names[] = {
+ [C_STANDALONE] = "StandAlone",
+ [C_DISCONNECTING] = "Disconnecting",
+ [C_UNCONNECTED] = "Unconnected",
+ [C_TIMEOUT] = "Timeout",
+ [C_BROKEN_PIPE] = "BrokenPipe",
+ [C_NETWORK_FAILURE] = "NetworkFailure",
+ [C_PROTOCOL_ERROR] = "ProtocolError",
+ [C_CONNECTING] = "WFConnection",
+ /* [C_WF_REPORT_PARAMS] = "WFReportParams", does no longer exist in drbd-9.x */
+ [C_TEAR_DOWN] = "TearDown",
+ [C_CONNECTED] = "WFReportParams", /* drbd-8.4 for "Negotiating" or "Off" */
+ [L_ESTABLISHED] = "Connected",
+ [L_STARTING_SYNC_S] = "StartingSyncS",
+ [L_STARTING_SYNC_T] = "StartingSyncT",
+ [L_WF_BITMAP_S] = "WFBitMapS",
+ [L_WF_BITMAP_T] = "WFBitMapT",
+ [L_WF_SYNC_UUID] = "WFSyncUUID",
+ [L_SYNC_SOURCE] = "SyncSource",
+ [L_SYNC_TARGET] = "SyncTarget",
+ [L_PAUSED_SYNC_S] = "PausedSyncS",
+ [L_PAUSED_SYNC_T] = "PausedSyncT",
+ [L_VERIFY_S] = "VerifyS",
+ [L_VERIFY_T] = "VerifyT",
+ [L_AHEAD] = "Ahead",
+ [L_BEHIND] = "Behind",
+};
+
+static const char write_ordering_chars[] = {
+ [WO_NONE] = 'n',
+ [WO_DRAIN_IO] = 'd',
+ [WO_BDEV_FLUSH] = 'f',
+ [WO_BIO_BARRIER] = 'b',
+};
+
+
+static int seq_print_device_proc_drbd(struct seq_file *m, struct drbd_device *device);
+
+atomic_t nr_drbd8_devices;
+
+void drbd_md_decode_84(struct meta_data_on_disk_84 *on_disk, struct drbd_md *md)
+{
+ struct drbd_peer_md *peer_md;
+ const int peer_node_id = 0; /* setup_node_ids_84() moves it later */
+ u32 on_disk_flags;
+ unsigned long flags;
+ int i;
+
+ md->effective_size = be64_to_cpu(on_disk->la_size_sect);
+ md->current_uuid = be64_to_cpu(on_disk->uuid[UI_CURRENT]);
+ md->prev_members = 0;
+ md->prev_features = 0; /* no features field in the drbd-8.4 meta-data */
+ md->device_uuid = be64_to_cpu(on_disk->device_uuid);
+ md->md_size_sect = be32_to_cpu(on_disk->md_size_sect);
+ md->al_offset = be32_to_cpu(on_disk->al_offset);
+
+ md->bm_offset = be32_to_cpu(on_disk->bm_offset);
+
+ on_disk_flags = be32_to_cpu(on_disk->flags);
+ md->flags = on_disk_flags & MDF_84_MASK;
+
+ md->max_peers = 1;
+ md->bm_block_size = be32_to_cpu(on_disk->bm_bytes_per_bit);
+ md->node_id = -1; /* no node_id in the drbd-8.4 meta-data */
+ md->al_stripes = be32_to_cpu(on_disk->al_stripes);
+ md->al_stripe_size_4k = be32_to_cpu(on_disk->al_stripe_size_4k);
+
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ peer_md = &md->peers[i];
+
+ peer_md->bitmap_uuid = 0;
+ peer_md->bitmap_dagtag = 0;
+ peer_md->flags = 0;
+ peer_md->bitmap_index = -1;
+ }
+
+ peer_md = &md->peers[peer_node_id];
+ peer_md->bitmap_index = 0;
+
+ flags = (on_disk_flags & MDF_84_PEER_MASK) | MDF_HAVE_BITMAP;
+ flags |= on_disk_flags & MDF_84_PEER_OUTDATED ? MDF_PEER_OUTDATED : 0;
+ flags |= on_disk_flags & MDF_84_CONNECTED_IND ? MDF_PEER_CONNECTED : 0;
+ peer_md->flags = flags;
+ /* An 8.4 bitmap fully records the divergence; set the UUID (and its
+ * divergence flag) after the other flags so it is not overwritten.
+ */
+ drbd_set_peer_bitmap_uuid(peer_md, be64_to_cpu(on_disk->uuid[UI_BITMAP]), 0);
+
+
+ for (i = UI_HISTORY_START; i < UI_HISTORY_END; i++)
+ md->history_uuids[i - UI_HISTORY_START] = be64_to_cpu(on_disk->uuid[i]);
+}
+
+void drbd_md_encode_84(struct drbd_device *device, struct meta_data_on_disk_84 *buffer)
+{
+ struct drbd_md *md = &device->ldev->md;
+ int peer_node_id = !md->node_id;
+ struct drbd_peer_md *peer_md = &md->peers[peer_node_id];
+ u32 flags = (md->flags & MDF_84_MASK) | (peer_md->flags & MDF_84_PEER_MASK);
+ int i;
+
+ if (device->bitmap == NULL)
+ flags |= MDF_PEER_FULL_SYNC;
+
+ flags |= test_bit(__MDF_PEER_OUTDATED, &peer_md->flags) ? MDF_84_PEER_OUTDATED : 0;
+ flags |= test_bit(__MDF_PEER_CONNECTED, &peer_md->flags) ? MDF_84_CONNECTED_IND : 0;
+ buffer->la_size_sect = cpu_to_be64(md->effective_size);
+ buffer->device_uuid = cpu_to_be64(md->device_uuid);
+ buffer->uuid[UI_CURRENT] = cpu_to_be64(md->current_uuid);
+ buffer->uuid[UI_BITMAP] = cpu_to_be64(peer_md->bitmap_uuid);
+ for (i = UI_HISTORY_START; i < UI_HISTORY_END; i++)
+ buffer->uuid[i] = cpu_to_be64(md->history_uuids[i - UI_HISTORY_START]);
+ buffer->reserved_u64_1 = 0;
+ buffer->flags = cpu_to_be32(flags);
+ buffer->magic = cpu_to_be32(DRBD_MD_MAGIC_84_UNCLEAN);
+ buffer->md_size_sect = cpu_to_be32(md->md_size_sect);
+ buffer->al_offset = cpu_to_be32(md->al_offset);
+ buffer->al_nr_extents = cpu_to_be32(device->act_log->nr_elements);
+ buffer->bm_offset = cpu_to_be32(md->bm_offset);
+ buffer->bm_bytes_per_bit = cpu_to_be32(BM_BLOCK_SIZE_4k); /* treat as magic number */
+ buffer->la_peer_max_bio_size = cpu_to_be32(device->device_conf.max_bio_size);
+
+ buffer->al_stripes = cpu_to_be32(md->al_stripes);
+ buffer->al_stripe_size_4k = cpu_to_be32(md->al_stripe_size_4k);
+}
+
+
+/*
+ * This is DRBD 8 userspace compatibility mode, so we do not have a node ID
+ * yet. We derive our own node ID from the peer node ID. drbdsetup gives us the
+ * peer-node-id, which it determines by comparing the IP addresses.
+ */
+int drbd_setup_node_ids_84(struct drbd_connection *connection, struct drbd_path *path,
+ unsigned int peer_node_id)
+{
+ int vnr, my_node_id, nr_legacy = 0, nr_v9 = 0;
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+
+ my_node_id = !peer_node_id;
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (test_bit(LEGACY_84_MD, &device->flags)) {
+ nr_legacy++;
+ } else {
+ nr_v9++;
+ if (get_ldev(device)) {
+ int md_my_node_id = device->ldev->md.node_id;
+
+ put_ldev(device);
+ if (my_node_id != md_my_node_id) {
+ drbd_err(connection, "inconsistent node_ids %d %d\n",
+ my_node_id, md_my_node_id);
+ return -ENOTUNIQ;
+ }
+ }
+ }
+ }
+
+ if (nr_legacy && nr_v9)
+ drbd_warn(connection, "legacy-84 and drbd-9 metadata in one resource\n");
+
+ drbd_info(connection, "drbd8 userspace compat mode: setting my node id to %d\n",
+ my_node_id);
+
+ /* setting up all node_ids*/
+ resource->res_opts.node_id = my_node_id;
+ connection->peer_node_id = peer_node_id;
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ peer_device = list_first_entry_or_null(&device->peer_devices,
+ struct drbd_peer_device, peer_devices);
+ peer_device->node_id = peer_node_id;
+ peer_device->bitmap_index = 0;
+
+ if (get_ldev(device)) {
+ const struct drbd_peer_md clear = { .bitmap_index = -1 };
+ struct drbd_md *md = &device->ldev->md;
+ struct drbd_peer_md *to = &md->peers[peer_node_id];
+ int i;
+
+ md->node_id = my_node_id;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ struct drbd_peer_md *from = &md->peers[i];
+
+ if (from->bitmap_index != -1) {
+ if (from != to) {
+ *to = *from;
+ *from = clear;
+ }
+ break;
+ }
+ }
+ put_ldev(device);
+ }
+ }
+
+ return 0;
+}
+
+
+/*
+ * Some resources may be operating in "DRBD 8 compatibility mode", where the
+ * user created the resource using the old drbd8-style drbdsetup command line
+ * syntax.
+ * This implies that the user probably also expects the old drbd8-style
+ * /proc/drbd output showing the device state.
+ * If the flag is set for a resource, we show the old-style output for that
+ * resource.
+ * If any resource is in DRBD 8 compatibility mode, this function returns true.
+ */
+bool drbd_show_legacy_device(struct seq_file *seq, void *v)
+{
+ struct drbd_device *device;
+ int i, prev_i = -1;
+
+ if (!atomic_read(&nr_drbd8_devices))
+ return false;
+
+ rcu_read_lock();
+ idr_for_each_entry(&drbd_devices, device, i) {
+ if (!device->resource->res_opts.drbd8_compat_mode)
+ continue;
+
+ if (prev_i != i - 1)
+ seq_putc(seq, '\n');
+ prev_i = i;
+
+ seq_print_device_proc_drbd(seq, device);
+ }
+ rcu_read_unlock();
+ return true;
+}
+
+static void seq_printf_with_thousands_grouping(struct seq_file *seq, long v)
+{
+ /* v is in kB/sec. We don't expect TiByte/sec yet.
+ * v can be negative: while the resync loses ground to incoming
+ * application writes the out-of-sync count grows, so the rate at
+ * which it shrinks is negative. Show that honestly.
+ */
+ if (v < 0) {
+ seq_putc(seq, '-');
+ v = -v;
+ }
+ if (unlikely(v >= 1000000)) {
+ /* cool: > GiByte/s */
+ seq_printf(seq, "%ld,", v / 1000000);
+ v %= 1000000;
+ seq_printf(seq, "%03ld,%03ld", v/1000, v % 1000);
+ } else if (likely(v >= 1000))
+ seq_printf(seq, "%ld,%03ld", v/1000, v % 1000);
+ else
+ seq_printf(seq, "%ld", v);
+}
+
+/* like bit_to_kb(), but accepts a signed bit count so we can express a
+ * negative kB/sec rate when the out-of-sync set is growing.
+ */
+static long signed_bit_to_kb(long bits, unsigned int bm_block_shift)
+{
+ if (bits < 0)
+ return -(long)bit_to_kb(-bits, bm_block_shift);
+ return bit_to_kb(bits, bm_block_shift);
+}
+
+void drbd_get_syncer_progress_84(struct drbd_peer_device *pd,
+ enum drbd_repl_state repl_state, unsigned long *rs_total,
+ unsigned long *bits_left, unsigned int *per_mil_done)
+{
+ /* this is to break it at compile time when we change that, in case we
+ * want to support more than (1<<32) bits on a 32bit arch.
+ */
+ typecheck(unsigned long, pd->rs_total);
+ *rs_total = pd->rs_total;
+
+ /* note: both rs_total and rs_left are in bits, i.e. in
+ * units of BM_BLOCK_SIZE.
+ * for the percentage, we don't care.
+ */
+
+ if (repl_state == L_VERIFY_S || repl_state == L_VERIFY_T)
+ *bits_left = atomic64_read(&pd->ov_left);
+ else
+ *bits_left = drbd_bm_total_weight(pd) - pd->rs_failed;
+ /* >> 10 to prevent overflow,
+ * +1 to prevent division by zero
+ */
+ if (*bits_left > *rs_total) {
+ /* D'oh. Maybe a logic bug somewhere. More likely just a race
+ * between state change and reset of rs_total.
+ */
+ *bits_left = *rs_total;
+ *per_mil_done = *rs_total ? 0 : 1000;
+ } else {
+ /* Make sure the division happens in long context.
+ * We allow up to one petabyte storage right now,
+ * at a granularity of 4k per bit that is 2**38 bits.
+ * After shift right and multiplication by 1000,
+ * this should still fit easily into a 32bit long,
+ * so we don't need a 64bit division on 32bit arch.
+ * Note: currently we don't support such large bitmaps on 32bit
+ * arch anyways, but no harm done to be prepared for it here.
+ */
+ unsigned int shift = *rs_total > UINT_MAX ? 16 : 10;
+ unsigned long left = *bits_left >> shift;
+ unsigned long total = 1UL + (*rs_total >> shift);
+ unsigned long tmp = 1000UL - left * 1000UL/total;
+ *per_mil_done = tmp;
+ }
+}
+
+static void drbd_syncer_progress(struct drbd_peer_device *pd, struct seq_file *seq,
+ enum drbd_repl_state repl_state)
+{
+ unsigned long dt, rt, rs_total, rs_left;
+ long db, dbdt;
+ unsigned int res;
+ int i, x, y;
+ int stalled = 0;
+ unsigned int bm_block_shift = pd->device->last_bm_block_shift;
+
+ drbd_get_syncer_progress_84(pd, repl_state, &rs_total, &rs_left, &res);
+
+ x = res/50;
+ y = 20-x;
+ seq_puts(seq, "\t[");
+ for (i = 1; i < x; i++)
+ seq_putc(seq, '=');
+ seq_putc(seq, '>');
+ for (i = 0; i < y; i++)
+ seq_putc(seq, '.');
+ seq_puts(seq, "] ");
+
+ if (repl_state == L_VERIFY_S || repl_state == L_VERIFY_T)
+ seq_puts(seq, "verified:");
+ else
+ seq_puts(seq, "sync'ed:");
+ seq_printf(seq, "%3u.%u%% ", res / 10, res % 10);
+
+ /* if more than a few GB, display in MB */
+ if (rs_total > (4UL << (30 - bm_block_shift)))
+ seq_printf(seq, "(%llu/%llu)M",
+ bit_to_kb(rs_left >> 10, bm_block_shift),
+ bit_to_kb(rs_total >> 10, bm_block_shift));
+ else
+ seq_printf(seq, "(%llu/%llu)K",
+ bit_to_kb(rs_left, bm_block_shift),
+ bit_to_kb(rs_total, bm_block_shift));
+
+ seq_puts(seq, "\n\t");
+
+ /* see drivers/md/md.c
+ * We do not want to overflow, so the order of operands and
+ * the * 100 / 100 trick are important. We do a +1 to be
+ * safe against division by zero. We only estimate anyway.
+ *
+ * dt: time from mark until now
+ * db: blocks written from mark until now
+ * rt: remaining time
+ */
+ /* Rolling marks. last_mark+1 may just now be modified. last_mark+2 is
+ * at least (DRBD_SYNC_MARKS-2)*DRBD_SYNC_MARK_STEP old, and has at
+ * least DRBD_SYNC_MARK_STEP time before it will be modified.
+ */
+ /* ------------------------ ~18s average ------------------------ */
+ i = (pd->rs_last_mark + 2) % DRBD_SYNC_MARKS;
+ dt = (jiffies - pd->rs_mark_time[i]) / HZ;
+ if (dt > 180)
+ stalled = 1;
+
+ if (!dt)
+ dt++;
+ db = (long)pd->rs_mark_left[i] - (long)rs_left;
+ if (db <= 0) {
+ /* out-of-sync is not shrinking: under this write load the
+ * resync is not converging, so there is no finish time.
+ */
+ seq_puts(seq, "finish: ∞");
+ } else {
+ rt = (dt * (rs_left / ((unsigned long)db/100+1)))/100; /* seconds */
+ seq_printf(seq, "finish: %lu:%02lu:%02lu",
+ rt / 3600, (rt % 3600) / 60, rt % 60);
+ }
+
+ dbdt = signed_bit_to_kb(db / (long)dt, bm_block_shift);
+ seq_puts(seq, " speed: ");
+ seq_printf_with_thousands_grouping(seq, dbdt);
+ seq_puts(seq, " (");
+ /* ------------------------- ~3s average ------------------------ */
+ if (1) {
+ /* this is what drbd_rs_should_slow_down() uses */
+ i = (pd->rs_last_mark + DRBD_SYNC_MARKS-1) % DRBD_SYNC_MARKS;
+ dt = (jiffies - pd->rs_mark_time[i]) / HZ;
+ if (!dt)
+ dt++;
+ db = (long)pd->rs_mark_left[i] - (long)rs_left;
+ dbdt = signed_bit_to_kb(db / (long)dt, bm_block_shift);
+ seq_printf_with_thousands_grouping(seq, dbdt);
+ seq_puts(seq, " -- ");
+ }
+
+ /* --------------------- long term average ---------------------- */
+ /* mean speed since syncer started we do account for PausedSync periods */
+ dt = (jiffies - pd->rs_start - pd->rs_paused) / HZ;
+ if (dt == 0)
+ dt = 1;
+ db = (long)rs_total - (long)rs_left;
+ dbdt = signed_bit_to_kb(db / (long)dt, bm_block_shift);
+ seq_printf_with_thousands_grouping(seq, dbdt);
+ seq_putc(seq, ')');
+
+ if (repl_state == L_SYNC_TARGET ||
+ repl_state == L_VERIFY_S) {
+ seq_puts(seq, " want: ");
+ seq_printf_with_thousands_grouping(seq, pd->c_sync_rate);
+ }
+ seq_printf(seq, " K/sec%s\n", stalled ? " (stalled)" : "");
+
+ {
+ /* 64 bit: we convert to sectors in the display below. */
+ unsigned long bm_bits = drbd_bm_bits(pd->device);
+ unsigned long bit_pos;
+ unsigned long long stop_sector = 0;
+
+ if (repl_state == L_VERIFY_S ||
+ repl_state == L_VERIFY_T) {
+ bit_pos = bm_bits - (unsigned long)atomic64_read(&pd->ov_left);
+ if (verify_can_do_stop_sector(pd))
+ stop_sector = pd->ov_stop_sector;
+ } else
+ bit_pos = pd->resync_next_bit;
+ /* Total sectors may be slightly off for oddly sized devices. So what. */
+ seq_printf(seq,
+ "\t%3d%% sector pos: %llu/%llu",
+ (int)(bit_pos / (bm_bits/100+1)),
+ (unsigned long long)bit_pos * sect_per_bit(bm_block_shift),
+ (unsigned long long)bm_bits * sect_per_bit(bm_block_shift));
+ if (stop_sector != 0 && stop_sector != ULLONG_MAX)
+ seq_printf(seq, " stop sector: %llu", stop_sector);
+ seq_putc(seq, '\n');
+ }
+}
+
+static const char *drbd_conn_str_84(enum drbd_conn_state s)
+{
+ /* enums are unsigned... */
+ return (int)s > (int)L_BEHIND ? "TOO_LARGE" : drbd_conn_s_names[s];
+}
+
+/*
+ * Select an 8.4-mode device's peer device (there is at most one: an
+ * 8.4-mode resource has one connection, which creates a peer device for
+ * every device) and fetch the raw state: the peer device's if one exists,
+ * else the device's own.
+ *
+ * The result is DRBD 9's unpacked union drbd_state, with DRBD 9's own
+ * disk/pdsk numbering and quorum bit still in place; drbd_pack_state_84()
+ * applies the v1 wire remap on top, seq_print_device_proc_drbd() feeds it
+ * to DRBD 9's own name lookups and must not.
+ *
+ * The caller holds rcu_read_lock() across this call and for as long as it
+ * keeps using *peer_device_r: the peer device is found without taking a
+ * reference.
+ *
+ * @peer_device_r: the selected peer device or NULL; may be NULL.
+ */
+static union drbd_state drbd_get_state_84(struct drbd_device *device,
+ struct drbd_peer_device **peer_device_r)
+{
+ struct drbd_peer_device *peer_device;
+
+ peer_device = list_first_or_null_rcu(&device->peer_devices, struct drbd_peer_device,
+ peer_devices);
+ if (peer_device_r)
+ *peer_device_r = peer_device;
+
+ return peer_device ? drbd_get_peer_device_state(peer_device, NOW)
+ : drbd_get_device_state(device, NOW);
+}
+
+/*
+ * Pack a device's current state into an 8.4 wire state word (union
+ * drbd_state as u32).
+ *
+ * conn/repl numbering needs no translation: DRBD 9 split 8.4's enum
+ * drbd_conns into drbd_conn_state and drbd_repl_state but kept the numeric
+ * values. disk/pdsk do: DRBD 9 inserted D_DETACHING at index 2, so
+ * drbd_disk_state_84() shifts everything above it by one, the same remap
+ * send_state() uses for a pre-9.0 peer. s.quorum (bit 23) is _pad on 8.4
+ * and must read as 0.
+ *
+ * The result carries 8.4 numbering: do not feed it to drbd_disk_str() and
+ * friends.
+ */
+u32 drbd_pack_state_84(struct drbd_device *device)
+{
+ union drbd_state s;
+
+ rcu_read_lock();
+ s = drbd_get_state_84(device, NULL);
+ rcu_read_unlock();
+
+ s.disk = drbd_disk_state_84(s.disk);
+ s.pdsk = drbd_disk_state_84(s.pdsk);
+ s.quorum = 0; /* bit 23 is _pad to 8.4 */
+
+ return s.i;
+}
+
+static int seq_print_device_proc_drbd(struct seq_file *m, struct drbd_device *device)
+{
+ unsigned int send_kb, recv_kb, pending_cnt, unacked_cnt, epochs;
+ struct drbd_connection *connection = NULL;
+ struct drbd_peer_device *peer_device;
+ union drbd_state state;
+ const char *sn;
+ bool have_ldev;
+ char wp;
+
+ /*
+ * Unpacked on purpose: state.disk/state.pdsk go to DRBD 9's own
+ * drbd_disk_str() below. The caller holds rcu_read_lock().
+ */
+ state = drbd_get_state_84(device, &peer_device);
+
+ if (peer_device) {
+ connection = peer_device->connection;
+ send_kb = peer_device->send_cnt/2;
+ recv_kb = peer_device->recv_cnt/2;
+ pending_cnt = atomic_read(&peer_device->ap_pending_cnt) +
+ atomic_read(&peer_device->rs_pending_cnt);
+ unacked_cnt = atomic_read(&peer_device->unacked_cnt);
+ } else {
+ connection = list_first_or_null_rcu(&device->resource->connections,
+ struct drbd_connection, connections);
+ send_kb = 0;
+ recv_kb = 0;
+ pending_cnt = 0;
+ unacked_cnt = 0;
+ }
+ if (connection) {
+ struct drbd_net_conf *nc = rcu_dereference(connection->transport.net_conf);
+
+ wp = nc ? nc->wire_protocol - DRBD_PROT_A + 'A' : ' ';
+ epochs = connection->epochs;
+ } else {
+ wp = 'C';
+ epochs = 0;
+ }
+
+ sn = drbd_conn_str_84(state.conn);
+ have_ldev = get_ldev_if_state(device, D_FAILED);
+
+ if (state.conn == C_STANDALONE &&
+ state.disk == D_DISKLESS &&
+ state.role == R_SECONDARY) {
+ seq_printf(m, "%2d: cs:Unconfigured\n", device->minor);
+ } else {
+ seq_printf(m,
+ "%2d: cs:%s ro:%s/%s ds:%s/%s %c %c%c%c%c%c%c\n"
+ " ns:%u nr:%u dw:%u dr:%u al:%u bm:%u "
+ "lo:%d pe:%d ua:%d ap:%d ep:%d wo:%c",
+ device->minor, sn,
+ drbd_role_str(state.role),
+ drbd_role_str(state.peer),
+ drbd_disk_str(state.disk),
+ drbd_disk_str(state.pdsk),
+ wp,
+ drbd_suspended(device) ? 's' : 'r',
+ state.aftr_isp ? 'a' : '-',
+ state.peer_isp ? 'p' : '-',
+ state.user_isp ? 'u' : '-',
+ '-' /* congestion reason... FIXME */,
+ test_bit(AL_SUSPENDED, &device->flags) ? 's' : '-',
+ send_kb,
+ recv_kb,
+ device->writ_cnt/2,
+ device->read_cnt/2,
+ device->al_writ_cnt,
+ device->bm_writ_cnt,
+ atomic_read(&device->local_cnt),
+ pending_cnt,
+ unacked_cnt,
+ atomic_read(&device->ap_bio_cnt[WRITE]) +
+ atomic_read(&device->ap_bio_cnt[READ]),
+ epochs,
+ write_ordering_chars[device->resource->write_ordering]
+ );
+ seq_printf(m, " oos:%llu\n", (peer_device && have_ldev) ?
+ device_bit_to_kb(device, drbd_bm_total_weight(peer_device)) : 0);
+ }
+ if (have_ldev) {
+ if (state.conn == L_SYNC_SOURCE ||
+ state.conn == L_SYNC_TARGET ||
+ state.conn == L_VERIFY_S ||
+ state.conn == L_VERIFY_T)
+ drbd_syncer_progress(peer_device, m, state.conn);
+
+ put_ldev(device);
+ }
+ /* drbd_proc_details 1 or 2 missing */
+
+ return 0;
+}
diff --git a/drivers/block/drbd/drbd_legacy_84.h b/drivers/block/drbd/drbd_legacy_84.h
new file mode 100644
index 000000000000..0cd7d4322dd6
--- /dev/null
+++ b/drivers/block/drbd/drbd_legacy_84.h
@@ -0,0 +1,116 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025, LINBIT HA-Solutions GmbH.
+ */
+
+#ifndef __DRBD_LEGACY_84_H
+#define __DRBD_LEGACY_84_H
+
+#include "drbd_int.h"
+
+struct meta_data_on_disk_84;
+
+/*
+ * drbd-8.4 drbd-9 md.flags drbd-9 peer-md.flags
+ * MDF_CONSISTENT 1 << 0 MDF_CONSISTENT = 1 << 0, MDF_PEER_CONNECTED = 1 << 0,
+ * MDF_PRIMARY_IND 1 << 1 MDF_PRIMARY_IND = 1 << 1, MDF_PEER_OUTDATED = 1 << 1,
+ * MDF_CONNECTED_IND 1 << 2 MDF_PEER_FENCING = 1 << 2,
+ * MDF_FULL_SYNC 1 << 3 MDF_PEER_FULL_SYNC = 1 << 3,
+ * MDF_WAS_UP_TO_DATE 1 << 4 MDF_WAS_UP_TO_DATE = 1 << 4, MDF_PEER_DEVICE_SEEN = 1 << 4,
+ * MDF_PEER_OUT_DATED 1 << 5 MDF_PEER_DIVERGENCE_BITMAP = 1 << 5
+ * MDF_CRASHED_PRIMARY 1 << 6 MDF_CRASHED_PRIMARY = 1 << 6, MDF_PEER_BITMAP_AUTHORITATIVE
+ * = 1 << 6
+ * MDF_AL_CLEAN 1 << 7 MDF_AL_CLEAN = 1 << 7,
+ * MDF_AL_DISABLED 1 << 8 MDF_AL_DISABLED = 1 << 8,
+ * MDF_PRIMARY_LOST_QUORUM = 1 << 9,
+ * MDF_HAVE_QUORUM = 1 << 10,
+ * MDF_NODE_EXISTS = 1 << 16,
+ */
+
+#define MDF_84_MASK (MDF_CONSISTENT | MDF_PRIMARY_IND | MDF_WAS_UP_TO_DATE | \
+ MDF_CRASHED_PRIMARY | MDF_AL_CLEAN | MDF_AL_DISABLED)
+#define MDF_84_PEER_MASK (MDF_PEER_FULL_SYNC)
+#define MDF_84_CONNECTED_IND (1<<2)
+#define MDF_84_PEER_OUTDATED (1<<5)
+
+/*
+ * Mask for the v1 DEVICE_STATISTICS dev_disk_flags and
+ * PEER_DEVICE_STATISTICS peer_dev_flags attributes. DRBD 9 keeps 8.4's
+ * bit positions for every bit 8.4 knows about, so masking off the newer
+ * bits is enough; nothing needs remapping.
+ */
+#define PEER_DEV_FLAGS_84_MASK (MDF_PEER_CONNECTED | MDF_PEER_OUTDATED | \
+ MDF_PEER_FENCING | MDF_PEER_FULL_SYNC)
+
+#ifdef CONFIG_DRBD_COMPAT_84
+extern atomic_t nr_drbd8_devices;
+
+void drbd_md_decode_84(struct meta_data_on_disk_84 *on_disk, struct drbd_md *md);
+void drbd_md_encode_84(struct drbd_device *device, struct meta_data_on_disk_84 *buffer);
+int drbd_setup_node_ids_84(struct drbd_connection *connection, struct drbd_path *path,
+ unsigned int peer_node_id);
+bool drbd_show_legacy_device(struct seq_file *seq, void *v);
+u32 drbd_pack_state_84(struct drbd_device *device);
+void drbd_get_syncer_progress_84(struct drbd_peer_device *pd,
+ enum drbd_repl_state repl_state, unsigned long *rs_total,
+ unsigned long *bits_left, unsigned int *per_mil_done);
+
+/*
+ * Remap a DRBD 9 enum drbd_state_rv value to one 8.4 userland's own SS_*
+ * string table can render truthfully. SS_UNKNOWN_ERROR (0) through
+ * SS_O_VOL_PEER_PRI (-20), every success code and every enum drbd_ret_code
+ * are numerically identical in both dialects. From SS_INTERRUPTED (-21)
+ * down, DRBD 9 has nine codes 8.4 numbers differently (SS_INTERRUPTED
+ * collides with 8.4's SS_OUTDATE_WO_CONN) or lacks entirely (8.4 ends at
+ * SS_AFTER_LAST_ERROR = -22); passed through raw they would print a wrong
+ * message or "unknown error code". Map each to the nearest 8.4 code whose
+ * string does not mislead: the interrupted, timed-out and retried
+ * handshake cases to SS_IN_TRANSIENT_STATE ("retry"), a blocking
+ * read-only opener to SS_DEVICE_IN_USE, a handshake-forced disconnect to
+ * SS_CW_FAILED_BY_PEER, and the quorum, weak-connectivity, bitmap
+ * negotiation and sentinel codes, which have no 8.4 meaning, to
+ * SS_UNKNOWN_ERROR. compat84_put_outcome() is the only place a
+ * drbd_state_rv reaches the v1 wire.
+ */
+static inline enum drbd_state_rv drbd_state_rv_84(enum drbd_state_rv rv)
+{
+ switch (rv) {
+ case SS_INTERRUPTED:
+ case SS_TIMEOUT:
+ case SS_HANDSHAKE_RETRY:
+ return SS_IN_TRANSIENT_STATE;
+ case SS_PRIMARY_READER:
+ return SS_DEVICE_IN_USE;
+ case SS_HANDSHAKE_DISCONNECT:
+ return SS_CW_FAILED_BY_PEER;
+ case SS_WEAKLY_CONNECTED:
+ case SS_NO_QUORUM:
+ case SS_ATTACH_NO_BITMAP:
+ case SS_AFTER_LAST_ERROR:
+ return SS_UNKNOWN_ERROR;
+ default:
+ return rv;
+ }
+}
+
+/*
+ * Fused v1 SIB_SYNC_PROGRESS event, called from update_on_disk_bitmap()
+ * (drbd_sender.c), mainline 8.4's own call site; already throttled there
+ * by drbd_lazy_bitmap_update_due().
+ */
+void compat84_notify_sync_progress(struct drbd_peer_device *peer_device);
+#else
+static inline void drbd_md_decode_84(struct meta_data_on_disk_84 *on_disk, struct drbd_md *md) {};
+static inline void drbd_md_encode_84(struct drbd_device *device,
+ struct meta_data_on_disk_84 *buffer) {};
+static inline int drbd_setup_node_ids_84(struct drbd_connection *connection, struct drbd_path *path,
+ unsigned int peer_node_id) { return 0; };
+static inline bool drbd_show_legacy_device(struct seq_file *seq, void *v) { return false; };
+static inline u32 drbd_pack_state_84(struct drbd_device *device) { return 0; };
+static inline void drbd_get_syncer_progress_84(struct drbd_peer_device *pd,
+ enum drbd_repl_state repl_state, unsigned long *rs_total,
+ unsigned long *bits_left, unsigned int *per_mil_done) {};
+static inline void compat84_notify_sync_progress(struct drbd_peer_device *peer_device) {};
+#endif /* CONFIG_DRBD_COMPAT_84 */
+
+#endif /* __DRBD_LEGACY_84_H */
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 08/20] drbd: add application/resync IO synchronization documentation
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (6 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 07/20] drbd: add optional compatibility layer for DRBD 8.4 Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 09/20] drbd: rework sender for DRBD 9 multi-peer Christoph Böhmwalder
` (11 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Joel Colledge
Add documentation describing how DRBD synchronizes application and
resync IO, including the dagtag-based locking scheme introduced with
the DRBD_FF_RESYNC_DAGTAG feature flag.
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
.../application-resync-synchronization.rst | 208 ++++++++++++++++++
.../admin-guide/blockdev/drbd/index.rst | 1 +
2 files changed, 209 insertions(+)
create mode 100644 Documentation/admin-guide/blockdev/drbd/application-resync-synchronization.rst
diff --git a/Documentation/admin-guide/blockdev/drbd/application-resync-synchronization.rst b/Documentation/admin-guide/blockdev/drbd/application-resync-synchronization.rst
new file mode 100644
index 000000000000..895312453229
--- /dev/null
+++ b/Documentation/admin-guide/blockdev/drbd/application-resync-synchronization.rst
@@ -0,0 +1,208 @@
+=========================================
+Application and Resync IO Synchronization
+=========================================
+
+This summary describes one aspect of the Distributed Replicated Block Device
+(DRBD) protocol. For a full definition of the protocol see the DRBD code.
+
+IO operations to the DRBD backing device can originate from 2 sources. Normal
+operations from a filesystem or other user of DRBD are called application IO.
+Resync operations in DRBD also perform reads and writes. These operations must
+be synchronized to ensure that the data on the backing device is correct.
+
+For instance, the following must be prevented:
+
+* Resync read obtains block version v1 on some node
+* Application writes block version v2 on all nodes
+* Resync write overwrites block version v2 with version v1
+
+In addition, care must be taken to ensure that bitmap bits are only cleared
+when this block is genuinely in sync between these nodes.
+
+Synchronization with resync extents
+===================================
+
+Older versions of DRBD without the feature flag ``DRBD_FF_RESYNC_DAGTAG``
+perform synchronization using "resync extents". These are also known as "bitmap
+extents". They are stored in an LRU cache. These extents are exclusive with the
+activity log extents.
+
+Synchronization with data generation tags
+=========================================
+
+When the feature flag ``DRBD_FF_RESYNC_DAGTAG`` is present, DRBD synchronizes
+resync and application IO using the concept of the "data generation tag"
+(dagtag). This is coupled with fine-grained locking of the request intervals,
+implemented internally using an interval tree.
+
+For a given node, the dagtag is the number of sectors written by the
+application on that node. The dagtag is used to determine which node has newer
+data in certain scenarios. DRBD keeps track of the dagtags of its peers.
+
+Whenever a resync request is made, the ``L_SYNC_TARGET`` node making the
+request sends the dagtag from the current Primary node, if any. The
+``L_SYNC_SOURCE`` node must wait until it has received the data corresponding
+to this dagtag before responding to the resync request. This is important for
+preventing the resync from writing older data over newer data.
+
+In addition, the request intervals are locked according to the following
+scheme:
+
+* Primary: The application IO interval is locked while the data is being
+ written to the backing disk. In addition, conflicting application IO is
+ prevented until the epoch containing the request is complete. However, this
+ is effectively a separate lock. Resync IO is not blocked while this lock is
+ held.
+* Secondary: The application IO interval is locked while the data is being
+ written to the backing disk.
+* Sync target: The ``P_RS_DATA_REQUEST`` interval is locked. The lock is taken
+ in two phases. Before sending the request, the interval is locked for
+ conflicts with other peers. Then the dagtag is recorded and the request is
+ sent. When the reply is received, the interval is additionally locked for
+ conflicts with ongoing IO, in particular writes from the same peer. The lock
+ is released when the reply has been received and the data written to the
+ backing disk.
+* Sync source: The ``P_RS_DATA_REQUEST`` interval is locked. The lock is taken
+ when the dagtag for the request is reached. It is released when the
+ ``P_RS_WRITE_ACK`` is received. This lock is a read lock; it is exclusive
+ with write locks, but not with other read locks.
+* Verify source: The online verify interval is marked but does not block any
+ other requests. The mark is set then the dagtag is recorded and the request
+ is sent. The mark is removed when the ``P_OV_REPLY`` has been received, the
+ dagtag from the reply has been reached and the data read. If any conflicting
+ writes occur while the mark is set, the sectors are skipped.
+* Verify target: The online verify interval is marked but does not block any
+ other requests. The mark is set when the dagtag for the request has been
+ reached. It is removed after reading the data. The latest dagtag received by
+ this node is sent with the ``P_OV_REPLY``. If any conflicting writes occur
+ while the mark is set, the sectors are skipped.
+* Sending ``P_PEERS_IN_SYNC``: Intervals are briefly locked while sending
+ ``P_PEERS_IN_SYNC`` to ensure that the bits remain in sync until the packet
+ has been sent.
+
+If a conflict occurs when an interval should be locked, the request is delayed
+until the conflict resolves. Internally this is implemented by storing the
+interval in the tree in an unlocked form. When an interval is removed from the
+tree, the tree is searched for any intervals which can now be released.
+
+Application IO defers to resync IO. That is, application IO is blocked by
+resync IO even when that resync IO has not yet obtained the lock for its
+interval. This is important for ensuring progress. In the normal case, resyncs
+only make one pass through the data. Hence they will eventually terminate.
+Application IO, on the other hand, can keep a given region busy for an
+arbitrary length of time. So resync IO must not wait indefinitely for
+application IO.
+
+Correctness of data
+-------------------
+
+We only consider the synchronization between application and resync IO here.
+
+The locking scheme prevents any writes from other peers to the resync request
+interval from when the request is initiated until the received data is written.
+After the lock is taken on the target, the dagtag is recorded and the request
+is sent to the source. The source then waits until it has reached this dagtag
+before reading. This ensures that the resync data is at least as new as the
+data on the target when the request was made.
+
+Conflicting application writes that reach the target while the resync request
+is in progress are held until the resync data has been written. Hence they
+overwrite the resync data. In the case where the source had already received
+this application write when it performed the resync read, the application write
+will overwrite the resync write on the target with identical data. This is
+harmless.
+
+Resync requests sent from the target are not exclusive with application writes
+from the same peer. However, since the resync and application data originate
+from the same node, they are transmitted in the correct order in the data
+stream. Application IO defers to received resync IO, ensuring that a resync
+write received before an application write is also submitted first.
+
+Correctness of bitmap bits
+--------------------------
+
+DRBD guarantees that bitmap bits are set, or the corresponding activity log
+extent is active, on at least one peer whenever 2 nodes are out of sync with
+each other. A resync is called "stable" when the target is a neighbor of the
+Primary node, if there is one. After a stable resync, all bitmap bits should be
+clear. In other situations, DRBD makes a best effort attempt to clear bits when
+appropriate.
+
+Hence we need to ensure that:
+
+1. Bits are set when out of sync
+2. Bits are only cleared when in sync
+3. Bits are cleared in a stable resync
+
+We are only considering the synchronization of application and resync IO here,
+so we only need to consider interactions between the 2 types. Requirement (1)
+holds due to the general design for how writes work in DRBD. Requirement (3)
+holds because there is no operation that sets bits in a stable resync. The
+potential issues with these interactions arise with requirement (2). We need to
+ensure that bits are never cleared that have become out of sync during the
+operation.
+
+On a Primary node, writes cause bits to be set and cleared when the
+corresponding ``P_BARRIER_ACK`` packets are received. On a Secondary node,
+writes cause bits to be set and cleared when the corresponding ``P_PEER_ACK``
+is received. On a sync target, bits are cleared when the resync data has been
+written. On a sync source, bits are cleared when ``P_RS_WRITE_ACK`` is
+received.
+
+The bits cleared by writes must always be in sync because the corresponding
+nodes have received the write. As demonstrated in the section "correctness of
+data", they cannot lose this data due to a resync.
+
+For a stable resync, bits will not become out of sync for the peer device on
+either side during the resync operation because both peers receive the
+application writes.
+
+On a sync target for an unstable resync, no application writes are received, so
+there will be no bits set that could be incorrectly cleared.
+
+On a sync source for an unstable resync, the interval is locked until
+``P_RS_WRITE_ACK`` is received. Hence, when the bit is cleared, the target has
+the same data for the interval as the source. That is, they are still in sync.
+
+For ``P_PEERS_IN_SYNC`` we consider only the 3 node case. There is only one
+configuration with an unstable resync with 3 nodes. That is a chain
+A - B - C with A being Primary and a sync from B to C. The only
+``P_PEERS_IN_SYNC`` packets that have an effect in this configuration are those
+from B to A indicating that C is in sync for some interval. B only sends this
+packet when no bitmap bits are set towards C for the interval. In addition,
+B must ensure that no application write causes bits to be incorrectly cleared
+on A towards C. This could occur when B has sent ``P_BARRIER_ACK`` for a write
+which is not yet represented in its bitmap towards C. So B must not send
+``P_PEERS_IN_SYNC`` for an interval where this may be the case. To do this, it
+checks that there is no activity in the activity log that overlaps with this
+interval. To ensure that no writes occur between this check and sending
+``P_PEERS_IN_SYNC``, it locks the interval temporarily.
+
+Deadlock safety
+---------------
+
+We can ignore the locking of application IO until the containing epoch is
+complete. No other lock acquisition depends on it. To put it another way, it
+operates on a level above the rest of the locking.
+
+The locking on Primary and Secondary while application IO is being written to
+the backing disk does not depend on any other lock acquisition. So it is
+guaranteed that a locked interval of this type will eventually be unlocked.
+
+Online verify does not block any other operations, so cannot be involved in
+causing a deadlock.
+
+Sending ``P_PEERS_IN_SYNC`` also cannot be involved in causing a deadlock
+because it does not depend on any other lock acquisition.
+
+Resync requests depend on the corresponding peer. If the connection is lost,
+the operation is aborted, so no deadlock will occur as a result of
+non-responsive peers.
+
+A node cannot be both sync source and sync target simultaneously. Hence there
+are no locks in the scheme which can block sync source reads indefinitely. So
+a resync request from a sync target will always eventually receive a reply,
+which allows it to perform the write and unlock its interval. This in turn
+guarantees that the sync source will receive an ack and unlock its interval.
+
+Hence the locking scheme itself is free from distributed deadlocks.
diff --git a/Documentation/admin-guide/blockdev/drbd/index.rst b/Documentation/admin-guide/blockdev/drbd/index.rst
index 561fd1e35917..5d8d6707e4f4 100644
--- a/Documentation/admin-guide/blockdev/drbd/index.rst
+++ b/Documentation/admin-guide/blockdev/drbd/index.rst
@@ -15,5 +15,6 @@ Description
.. toctree::
:maxdepth: 1
+ application-resync-synchronization
data-structure-v9
figures
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 09/20] drbd: rework sender for DRBD 9 multi-peer
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (7 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 08/20] drbd: add application/resync IO synchronization documentation Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 10/20] drbd: replace per-device state model with multi-peer data structures Christoph Böhmwalder
` (10 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Joel Colledge,
Andreas Gruenbacher, Roland Kammerer, Moritz Tanner
Split the single legacy worker thread into two: a per-connection
sender thread that drives the transfer log and replication data, and
a per-resource worker thread for background device tasks.
Resync and online-verify state moves entirely from the device to the
per-peer-device object.
Rewrite the resync request path around the interval tree instead of
the old resync-LRU extent locking, so conflict detection against
application I/O is precise rather than coarse-grained.
Resync requests carry DAG-tag ordering information if the peer supports
it, allowing the sync source to safely reorder replies.
Variable bitmap block sizes are handled correctly across peers with
different bitmap granularities.
Tighten locking throughout: replace the per-resource request spinlock
by a read-write state lock plus fine-grained per-connection and
per-device locks.
Move I/O completion tracking from per-device lists to per-connection
counters.
Switch the resync controller to nanosecond precision for better
throughput adaptation.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Co-developed-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_sender.c | 4445 ++++++++++++++++++++++--------
1 file changed, 3268 insertions(+), 1177 deletions(-)
diff --git a/drivers/block/drbd/drbd_sender.c b/drivers/block/drbd/drbd_sender.c
index b58984f02094..7d4d52517d28 100644
--- a/drivers/block/drbd/drbd_sender.c
+++ b/drivers/block/drbd/drbd_sender.c
@@ -6,25 +6,31 @@
* Copyright (C) 2008, LINBIT HA-Solutions GmbH.
*/
-#include <linux/module.h>
#include <linux/drbd.h>
+#include <linux/sched.h>
#include <linux/sched/signal.h>
#include <linux/wait.h>
#include <linux/mm.h>
-#include <linux/memcontrol.h>
+#include <linux/memcontrol.h> /* needed on kernels <4.3 */
#include <linux/mm_inline.h>
#include <linux/slab.h>
#include <linux/random.h>
-#include <linux/string.h>
-#include <linux/scatterlist.h>
+#include <linux/overflow.h>
#include <linux/part_stat.h>
#include "drbd_int.h"
#include "drbd_protocol.h"
#include "drbd_req.h"
+#include "drbd_meta_data.h"
+#include "drbd_legacy_84.h"
+
+void drbd_panic_after_delayed_completion_of_aborted_request(struct drbd_device *device);
static int make_ov_request(struct drbd_peer_device *, int);
static int make_resync_request(struct drbd_peer_device *, int);
+static bool should_send_barrier(struct drbd_connection *, unsigned int epoch);
+static void maybe_send_barrier(struct drbd_connection *, unsigned int);
+static void send_reconcile_current_uuid(struct drbd_peer_device *);
/* endio handlers:
* drbd_md_endio (defined here)
@@ -44,12 +50,15 @@ static int make_resync_request(struct drbd_peer_device *, int);
*/
void drbd_md_endio(struct bio *bio)
{
+ /* ldev_ref_transfer: ldev ref from bio submit in md I/O path */
struct drbd_device *device;
+ blk_status_t status = bio->bi_status;
+
device = bio->bi_private;
- device->md_io.error = blk_status_to_errno(bio->bi_status);
+ device->md_io.error = blk_status_to_errno(status);
- /* special case: drbd_md_read() during drbd_adm_attach() */
+ /* special case: drbd_md_read() during drbd_nl_attach_doit() */
if (device->ldev)
put_ldev(device);
bio_put(bio);
@@ -61,9 +70,9 @@ void drbd_md_endio(struct bio *bio)
* During normal operation, this only puts that extra reference
* down to 1 again.
* Make sure we first drop the reference, and only then signal
- * completion, or we may (in drbd_al_read_log()) cycle so fast into the
- * next drbd_md_sync_page_io(), that we trigger the
- * ASSERT(atomic_read(&device->md_io_in_use) == 1) there.
+ * completion, or a tight loop of metadata page IO (the next
+ * drbd_md_get_buffer() / drbd_md_sync_page_io()) could re-enter so fast
+ * that it finds device->md_io.in_use still held.
*/
drbd_md_put_buffer(device);
device->md_io.done = 1;
@@ -73,89 +82,168 @@ void drbd_md_endio(struct bio *bio)
/* reads on behalf of the partner,
* "submitted" by the receiver
*/
-static void drbd_endio_read_sec_final(struct drbd_peer_request *peer_req) __releases(local)
+static void drbd_endio_read_sec_final(struct drbd_peer_request *peer_req)
{
- unsigned long flags = 0;
struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ bool io_error;
- spin_lock_irqsave(&device->resource->req_lock, flags);
device->read_cnt += peer_req->i.size >> 9;
- list_del(&peer_req->w.list);
- if (list_empty(&device->read_ee))
- wake_up(&device->ee_wait);
- if (test_bit(__EE_WAS_ERROR, &peer_req->flags))
- __drbd_chk_io_error(device, DRBD_READ_ERROR);
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
-
- drbd_queue_work(&peer_device->connection->sender_work, &peer_req->w);
+ /* Resync and online-verify reads are DRBD's own IO at this device;
+ * of the reads that come through here, only the peer's is not.
+ */
+ if (!drbd_interval_is_application(&peer_req->i))
+ atomic_add(peer_req->i.size >> 9, &device->rs_sect_done);
+ /* The depth bound holds resync reads only, so only those were
+ * counted into the queue at submit.
+ */
+ if (drbd_interval_is_resync(&peer_req->i))
+ drbd_rs_depth_completed(device, peer_req->i.size >> 9,
+ peer_req->submit_jif);
+ io_error = test_bit(__EE_WAS_ERROR, &peer_req->flags);
+
+ drbd_queue_work(&connection->sender_work, &peer_req->w);
+ peer_req = NULL; /* peer_req may be freed. */
+
+ /*
+ * Decrement counter after queuing work to avoid a moment where
+ * backing_ee_cnt is zero and the sender work list is empty.
+ */
+ if (atomic_dec_and_test(&connection->backing_ee_cnt))
+ wake_up(&connection->ee_wait);
+
+ if (io_error)
+ drbd_handle_io_error(device, DRBD_READ_ERROR);
+
put_ldev(device);
}
+static int is_failed_barrier(int ee_flags)
+{
+ return (ee_flags & (EE_IS_BARRIER|EE_WAS_ERROR|EE_RESUBMITTED|EE_TRIM|EE_ZEROOUT))
+ == (EE_IS_BARRIER|EE_WAS_ERROR);
+}
+
+static bool drbd_peer_request_is_merged(struct drbd_peer_request *peer_req,
+ sector_t main_sector, sector_t main_sector_end)
+{
+ /*
+ * We do not send overlapping resync requests. So any request which is
+ * in the corresponding range and for which we have received a reply
+ * must be a merged request. EE_TRIM implies that we have received a
+ * reply.
+ */
+ return peer_req->i.sector >= main_sector &&
+ peer_req->i.sector + (peer_req->i.size >> SECTOR_SHIFT) <= main_sector_end &&
+ peer_req->i.type == INTERVAL_RESYNC_WRITE &&
+ (peer_req->flags & EE_TRIM);
+}
+
+int drbd_unmerge_discard(struct drbd_peer_request *peer_req_main, struct list_head *list)
+{
+ struct drbd_peer_device *peer_device = peer_req_main->peer_device;
+ struct drbd_peer_request *peer_req = peer_req_main;
+ sector_t main_sector = peer_req_main->i.sector;
+ sector_t main_sector_end = main_sector + (peer_req_main->i.size >> SECTOR_SHIFT);
+ int merged_count = 0;
+
+ list_for_each_entry_continue(peer_req, &peer_device->resync_requests, recv_order) {
+ if (!drbd_peer_request_is_merged(peer_req, main_sector, main_sector_end))
+ break;
+
+ merged_count++;
+ list_add_tail(&peer_req->w.list, list);
+ }
+
+ return merged_count;
+}
+
/* writes on behalf of the partner, or resync writes,
* "submitted" by the receiver, final stage. */
-void drbd_endio_write_sec_final(struct drbd_peer_request *peer_req) __releases(local)
+void drbd_endio_write_sec_final(struct drbd_peer_request *peer_req)
{
unsigned long flags = 0;
struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
struct drbd_connection *connection = peer_device->connection;
- struct drbd_interval i;
- int do_wake;
- u64 block_id;
- int do_al_complete_io;
+ enum drbd_interval_type type;
+ bool do_wake;
+
+ /* if this is a failed barrier request, disable use of barriers,
+ * and schedule for resubmission */
+ if (is_failed_barrier(peer_req->flags)) {
+ drbd_bump_write_ordering(device->resource, device->ldev, WO_BDEV_FLUSH);
+ spin_lock_irqsave(&connection->peer_reqs_lock, flags);
+ peer_req->flags = (peer_req->flags & ~EE_WAS_ERROR) | EE_RESUBMITTED;
+ peer_req->w.cb = w_e_reissue;
+ spin_unlock_irqrestore(&connection->peer_reqs_lock, flags);
+ drbd_queue_work(&connection->sender_work, &peer_req->w);
+ if (atomic_dec_and_test(&connection->active_ee_cnt))
+ wake_up(&connection->ee_wait);
+ return;
+ }
/* after we moved peer_req to done_ee,
* we may no longer access it,
* it may be freed/reused already!
- * (as soon as we release the req_lock) */
- i = peer_req->i;
- do_al_complete_io = peer_req->flags & EE_CALL_AL_COMPLETE_IO;
- block_id = peer_req->block_id;
- peer_req->flags &= ~EE_CALL_AL_COMPLETE_IO;
+ * (as soon as we release the peer_reqs_lock) */
+ type = peer_req->i.type;
+
+ if (type == INTERVAL_RESYNC_WRITE) {
+ atomic_add(peer_req->i.size >> 9, &device->rs_sect_done);
+ if (!(peer_req->flags & EE_TRIM))
+ drbd_rs_depth_completed(device, peer_req->i.size >> 9,
+ peer_req->submit_jif);
+ }
if (peer_req->flags & EE_WAS_ERROR) {
/* In protocol != C, we usually do not send write acks.
- * In case of a write error, send the neg ack anyways. */
- if (!__test_and_set_bit(__EE_SEND_WRITE_ACK, &peer_req->flags))
- inc_unacked(device);
+ * In case of a write error, send the neg ack anyways.
+ * This only applies to to application writes, not to resync. */
+ if (peer_req->i.type == INTERVAL_PEER_WRITE) {
+ if (!__test_and_set_bit(__EE_SEND_WRITE_ACK, &peer_req->flags))
+ inc_unacked(peer_device);
+ }
drbd_set_out_of_sync(peer_device, peer_req->i.sector, peer_req->i.size);
+ drbd_handle_io_error(device, DRBD_WRITE_ERROR);
}
- spin_lock_irqsave(&device->resource->req_lock, flags);
+ spin_lock_irqsave(&connection->peer_reqs_lock, flags);
device->writ_cnt += peer_req->i.size >> 9;
- list_move_tail(&peer_req->w.list, &device->done_ee);
+ atomic_inc(&connection->done_ee_cnt);
+ list_add_tail(&peer_req->w.list, &connection->done_ee);
+ if (peer_req->i.type == INTERVAL_RESYNC_WRITE && peer_req->flags & EE_TRIM) {
+ LIST_HEAD(merged);
+ int merged_count;
+
+ merged_count = drbd_unmerge_discard(peer_req, &merged);
+ list_splice_tail(&merged, &connection->done_ee);
+ atomic_add(merged_count, &connection->done_ee_cnt);
+ }
+ peer_req = NULL; /* may be freed after unlock */
+ spin_unlock_irqrestore(&connection->peer_reqs_lock, flags);
/*
- * Do not remove from the write_requests tree here: we did not send the
- * Ack yet and did not wake possibly waiting conflicting requests.
- * Removed from the tree from "drbd_process_done_ee" within the
- * appropriate dw.cb (e_end_block/e_end_resync_block) or from
- * _drbd_clear_done_ee.
+ * Do not remove from the requests tree here: we did not send the
+ * Ack yet.
+ * Removed from the tree from "drbd_finish_peer_reqs" within the
+ * appropriate callback (e_end_block/e_end_resync_block) or from
+ * cleanup functions if the connection is lost.
*/
- do_wake = list_empty(block_id == ID_SYNCER ? &device->sync_ee : &device->active_ee);
+ read_lock_irqsave(&connection->resource->state_rwlock, flags);
+ if (connection->cstate[NOW] == C_CONNECTED)
+ queue_work(connection->ack_sender, &connection->send_acks_work);
+ read_unlock_irqrestore(&connection->resource->state_rwlock, flags);
- /* FIXME do we want to detach for failed REQ_OP_DISCARD?
- * ((peer_req->flags & (EE_WAS_ERROR|EE_TRIM)) == EE_WAS_ERROR) */
- if (peer_req->flags & EE_WAS_ERROR)
- __drbd_chk_io_error(device, DRBD_WRITE_ERROR);
-
- if (connection->cstate >= C_WF_REPORT_PARAMS) {
- kref_get(&device->kref); /* put is in drbd_send_acks_wf() */
- if (!queue_work(connection->ack_sender, &peer_device->send_acks_work))
- kref_put(&device->kref, drbd_destroy_device);
- }
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
-
- if (block_id == ID_SYNCER)
- drbd_rs_complete_io(device, i.sector);
+ if (type == INTERVAL_RESYNC_WRITE)
+ do_wake = atomic_dec_and_test(&connection->backing_ee_cnt);
+ else
+ do_wake = atomic_dec_and_test(&connection->active_ee_cnt);
if (do_wake)
- wake_up(&device->ee_wait);
-
- if (do_al_complete_io)
- drbd_al_complete_io(device, &i);
+ wake_up(&connection->ee_wait);
put_ldev(device);
}
@@ -165,22 +253,49 @@ void drbd_endio_write_sec_final(struct drbd_peer_request *peer_req) __releases(l
*/
void drbd_peer_request_endio(struct bio *bio)
{
+ /* ldev_ref_transfer: ldev ref from bio submit in peer request I/O path */
struct drbd_peer_request *peer_req = bio->bi_private;
struct drbd_device *device = peer_req->peer_device->device;
bool is_write = bio_data_dir(bio) == WRITE;
bool is_discard = bio_op(bio) == REQ_OP_WRITE_ZEROES ||
bio_op(bio) == REQ_OP_DISCARD;
+ blk_status_t status = bio->bi_status;
+ unsigned long flags;
+ struct bio **pos;
- if (bio->bi_status && drbd_device_ratelimit(device, BACKEND))
+ if (status && drbd_device_ratelimit(device, BACKEND))
drbd_warn(device, "%s: error=%d s=%llus\n",
is_write ? (is_discard ? "discard" : "write")
- : "read", bio->bi_status,
+ : "read", status,
(unsigned long long)peer_req->i.sector);
- if (bio->bi_status)
+ if (status)
set_bit(__EE_WAS_ERROR, &peer_req->flags);
- bio_put(bio); /* no need for the bio anymore */
+ bio->bi_next = NULL; /* bi_next was used by the kernel during I/O; reinitialize */
+ /* Reset iter and restore sector and size for bio_for_each_segment(). */
+ bio->bi_iter = (struct bvec_iter) {
+ .bi_sector = peer_req->i.sector + to_drbd_peer_bio(bio)->sector_offset,
+ .bi_size = to_drbd_peer_bio(bio)->size,
+ };
+
+ spin_lock_irqsave(&device->peer_req_bio_completion_lock, flags);
+ if (bio_list_empty(&peer_req->bios)) {
+ bio_list_add(&peer_req->bios, bio);
+ } else {
+ /* Insert bio into the chain ordered by bi_sector */
+ for (pos = &peer_req->bios.head; *pos; pos = &(*pos)->bi_next) {
+ if (bio->bi_iter.bi_sector < (*pos)->bi_iter.bi_sector)
+ break;
+ }
+ bio->bi_next = *pos;
+ *pos = bio;
+ /* Update tail if we inserted at the end */
+ if (!bio->bi_next)
+ peer_req->bios.tail = bio;
+ }
+ spin_unlock_irqrestore(&device->peer_req_bio_completion_lock, flags);
+
if (atomic_dec_and_test(&peer_req->pending_bios)) {
if (is_write)
drbd_endio_write_sec_final(peer_req);
@@ -189,14 +304,15 @@ void drbd_peer_request_endio(struct bio *bio)
}
}
-static void
-drbd_panic_after_delayed_completion_of_aborted_request(struct drbd_device *device)
+/* Not static to increase the likelyhood that it will show up in a stack trace */
+void drbd_panic_after_delayed_completion_of_aborted_request(struct drbd_device *device)
{
panic("drbd%u %s/%u potential random memory corruption caused by delayed completion of aborted local request\n",
device->minor, device->resource->name, device->vnr);
}
-/* read, readA or write requests on R_PRIMARY coming from drbd_make_request
+
+/* read, readA or write requests on R_PRIMARY coming from drbd_submit_bio
*/
void drbd_request_endio(struct bio *bio)
{
@@ -206,6 +322,8 @@ void drbd_request_endio(struct bio *bio)
struct bio_and_error m;
enum drbd_req_event what;
+ blk_status_t status = bio->bi_status;
+
/* If this request was aborted locally before,
* but now was completed "successfully",
* chances are that this caused arbitrary data corruption.
@@ -216,7 +334,7 @@ void drbd_request_endio(struct bio *bio)
* situation, usually a hard-reset and failover is the only way out.
*
* By "aborting", basically faking a local error-completion,
- * we allow for a more graceful swichover by cleanly migrating services.
+ * we allow for a more graceful switchover by cleanly migrating services.
* Still the affected node has to be rebooted "soon".
*
* By completing these requests, we allow the upper layers to re-use
@@ -234,89 +352,200 @@ void drbd_request_endio(struct bio *bio)
* We assume that a delayed *error* completion is OK,
* though we still will complain noisily about it.
*/
- if (unlikely(req->rq_state & RQ_LOCAL_ABORTED)) {
+ if (unlikely(req->local_rq_state & RQ_LOCAL_ABORTED)) {
if (drbd_device_ratelimit(device, BACKEND))
drbd_emerg(device, "delayed completion of aborted local request; disk-timeout may be too aggressive\n");
- if (!bio->bi_status)
+ if (!status)
drbd_panic_after_delayed_completion_of_aborted_request(device);
}
/* to avoid recursion in __req_mod */
- if (unlikely(bio->bi_status)) {
- switch (bio_op(bio)) {
- case REQ_OP_WRITE_ZEROES:
- case REQ_OP_DISCARD:
- if (bio->bi_status == BLK_STS_NOTSUPP)
+ if (unlikely(status)) {
+ enum req_op op = bio_op(bio);
+
+ if (op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES) {
+ if (status == BLK_STS_NOTSUPP)
what = DISCARD_COMPLETED_NOTSUPP;
else
what = DISCARD_COMPLETED_WITH_ERROR;
- break;
- case REQ_OP_READ:
+ } else if (op == REQ_OP_READ) {
if (bio->bi_opf & REQ_RAHEAD)
what = READ_AHEAD_COMPLETED_WITH_ERROR;
else
what = READ_COMPLETED_WITH_ERROR;
- break;
- default:
+ } else {
what = WRITE_COMPLETED_WITH_ERROR;
- break;
}
} else {
what = COMPLETED_OK;
}
- req->private_bio = ERR_PTR(blk_status_to_errno(bio->bi_status));
- bio_put(bio);
+ bio_put(req->private_bio);
+ req->private_bio = ERR_PTR(blk_status_to_errno(status));
+
+ /* it is legal to fail read-ahead, no drbd_handle_io_error for READ_AHEAD_COMPLETED_WITH_ERROR */
+ if (what == WRITE_COMPLETED_WITH_ERROR)
+ drbd_handle_io_error(device, DRBD_WRITE_ERROR);
+ else if (what == READ_COMPLETED_WITH_ERROR)
+ drbd_handle_io_error(device, DRBD_READ_ERROR);
+
+ spin_lock_irqsave(&device->interval_lock, flags);
+ set_bit(INTERVAL_BACKING_COMPLETED, &req->i.flags);
+ if (req->local_rq_state & RQ_WRITE)
+ drbd_release_conflicts(device, &req->i);
+ spin_unlock_irqrestore(&device->interval_lock, flags);
/* not req_mod(), we need irqsave here! */
- spin_lock_irqsave(&device->resource->req_lock, flags);
+ read_lock_irqsave(&device->resource->state_rwlock, flags);
+ /* ldev_safe: bio endio, ldev ref held since drbd_request_prepare(), put_ldev() follows */
__req_mod(req, what, NULL, &m);
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
+ read_unlock_irqrestore(&device->resource->state_rwlock, flags);
put_ldev(device);
if (m.bio)
complete_master_bio(device, &m);
}
-void drbd_csum_ee(struct crypto_shash *tfm, struct drbd_peer_request *peer_req, void *digest)
+struct dagtag_find_result {
+ int err;
+ unsigned int node_id;
+ u64 dagtag;
+};
+
+static struct dagtag_find_result find_current_dagtag(struct drbd_resource *resource)
{
- SHASH_DESC_ON_STACK(desc, tfm);
- struct page *page = peer_req->pages;
- struct page *tmp;
- unsigned len;
- void *src;
+ struct drbd_connection *connection;
+ struct dagtag_find_result ret = { 0 };
+
+ read_lock_irq(&resource->state_rwlock);
+
+ if (resource->role[NOW] == R_PRIMARY) {
+ /* Sending data and sending resync requests are not
+ * synchronized with each other, so our peer may need to wait
+ * until it has received more data before it can reply to this
+ * request. */
+ ret.node_id = resource->res_opts.node_id;
+ ret.dagtag = resource->dagtag_sector;
+ } else {
+ for_each_connection(connection, resource) {
+ if (connection->peer_role[NOW] != R_PRIMARY)
+ continue;
- desc->tfm = tfm;
+ /* Do not depend on a stale dagtag. */
+ if (!test_bit(RECEIVED_DAGTAG, &connection->flags))
+ continue;
- crypto_shash_init(desc);
+ if (ret.dagtag) {
+ if (drbd_ratelimit())
+ drbd_err(resource, "Refusing to resync due to multiple remote primaries\n");
+ ret.err = 1;
+ break;
+ } else {
+ ret.node_id = connection->peer_node_id;
+ ret.dagtag = atomic64_read(&connection->last_dagtag_sector);
+ }
+ }
+ }
+
+ read_unlock_irq(&resource->state_rwlock);
+
+ return ret;
+}
+
+static void send_resync_request(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
+ struct dagtag_find_result dagtag_result;
+
+ if (!(connection->agreed_features & DRBD_FF_RESYNC_DAGTAG) &&
+ drbd_al_active(peer_device->device, peer_req->i.sector, peer_req->i.size)) {
+ dynamic_drbd_dbg(peer_device,
+ "Abort resync request at %llus+%u due to activity",
+ (unsigned long long) peer_req->i.sector, peer_req->i.size);
- src = kmap_atomic(page);
- while ((tmp = page_chain_next(page))) {
- /* all but the last page will be fully used */
- crypto_shash_update(desc, src, PAGE_SIZE);
- kunmap_atomic(src);
- page = tmp;
- src = kmap_atomic(page);
+ drbd_unsuccessful_resync_request(peer_req, false);
+ return;
}
- /* and now the last, possibly only partially used page */
- len = peer_req->i.size & (PAGE_SIZE - 1);
- crypto_shash_update(desc, src, len ?: PAGE_SIZE);
- kunmap_atomic(src);
- crypto_shash_final(desc, digest);
- shash_desc_zero(desc);
+ inc_rs_pending(peer_device);
+
+ dagtag_result = find_current_dagtag(peer_device->device->resource);
+ if (dagtag_result.err) {
+ change_cstate(peer_device->connection, C_DISCONNECTING, CS_HARD);
+ return;
+ }
+
+ if (peer_req->flags & EE_HAS_DIGEST) {
+ enum drbd_packet cmd = connection->agreed_features & DRBD_FF_RESYNC_DAGTAG ?
+ P_RS_CSUM_DAGTAG_REQ : P_CSUM_RS_REQUEST;
+
+ void *digest = drbd_prepare_drequest_csum(peer_req, cmd,
+ peer_req->digest->digest_size,
+ dagtag_result.node_id, dagtag_result.dagtag);
+ if (!digest)
+ return;
+
+ memcpy(digest, peer_req->digest->digest, peer_req->digest->digest_size);
+
+ /* We are now finished with the digest, so we can free it.
+ * If we don't, the reference will be lost when the block_id
+ * field of the union is used for the reply. */
+ peer_req->flags &= ~EE_HAS_DIGEST;
+ kfree(peer_req->digest);
+ peer_req->digest = NULL;
+
+ drbd_send_command(peer_device, cmd, DATA_STREAM);
+ } else {
+ enum drbd_packet cmd;
+
+ if (connection->agreed_features & DRBD_FF_RESYNC_DAGTAG)
+ cmd = peer_req->flags & EE_RS_THIN_REQ ? P_RS_THIN_DAGTAG_REQ : P_RS_DAGTAG_REQ;
+ else
+ cmd = peer_req->flags & EE_RS_THIN_REQ ? P_RS_THIN_REQ : P_RS_DATA_REQUEST;
+
+ drbd_send_rs_request(peer_device, cmd,
+ peer_req->i.sector, peer_req->i.size, peer_req->block_id,
+ dagtag_result.node_id, dagtag_result.dagtag);
+ }
}
-void drbd_csum_bio(struct crypto_shash *tfm, struct bio *bio, void *digest)
+void drbd_conflict_send_resync_request(struct drbd_peer_request *peer_req)
{
- SHASH_DESC_ON_STACK(desc, tfm);
- struct bio_vec bvec;
- struct bvec_iter iter;
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_req->peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ bool conflict;
+ bool canceled;
+
+ spin_lock_irq(&device->interval_lock);
+ clear_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &peer_req->i.flags);
+ canceled = test_bit(INTERVAL_CANCELED, &peer_req->i.flags);
+ conflict = drbd_find_conflict(device, &peer_req->i, CONFLICT_FLAG_IGNORE_SAME_PEER);
+ if (drbd_interval_empty(&peer_req->i))
+ drbd_insert_interval(&device->requests, &peer_req->i);
+ if (!conflict)
+ set_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags);
+ spin_unlock_irq(&device->interval_lock);
+
+ if (!conflict) {
+ send_resync_request(peer_req);
+ } else if (canceled) {
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ }
- desc->tfm = tfm;
+ if ((!conflict || canceled) && atomic_dec_and_test(&connection->backing_ee_cnt))
+ wake_up(&connection->ee_wait);
+}
- crypto_shash_init(desc);
+
+
+static void __drbd_csum_bio(struct bio *bio, struct shash_desc *desc)
+{
+ struct bio_vec bvec;
+ struct bvec_iter iter;
bio_for_each_segment(bvec, bio, iter) {
u8 *src;
@@ -325,6 +554,33 @@ void drbd_csum_bio(struct crypto_shash *tfm, struct bio *bio, void *digest)
crypto_shash_update(desc, src, bvec.bv_len);
kunmap_local(src);
}
+}
+
+void drbd_csum_bios(struct crypto_shash *tfm, struct bio_list *bios, void *digest)
+{
+ struct bio *bio;
+ SHASH_DESC_ON_STACK(desc, tfm);
+
+ desc->tfm = tfm;
+ crypto_shash_init(desc);
+
+ bio_list_for_each(bio, bios)
+ __drbd_csum_bio(bio, desc);
+
+ crypto_shash_final(desc, digest);
+ shash_desc_zero(desc);
+}
+
+/* This function needs to ignore bi_next */
+void drbd_csum_bio(struct crypto_shash *tfm, struct bio *bio, void *digest)
+{
+ SHASH_DESC_ON_STACK(desc, tfm);
+
+ desc->tfm = tfm;
+ crypto_shash_init(desc);
+
+ __drbd_csum_bio(bio, desc);
+
crypto_shash_final(desc, digest);
shash_desc_zero(desc);
}
@@ -334,120 +590,245 @@ static int w_e_send_csum(struct drbd_work *w, int cancel)
{
struct drbd_peer_request *peer_req = container_of(w, struct drbd_peer_request, w);
struct drbd_peer_device *peer_device = peer_req->peer_device;
- struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
int digest_size;
- void *digest;
int err = 0;
+ struct digest_info *di;
if (unlikely(cancel))
goto out;
+ /* Do not add to interval tree if already disconnected or resync aborted */
+ if (!repl_is_sync_target(peer_device->repl_state[NOW]))
+ goto out;
+
if (unlikely((peer_req->flags & EE_WAS_ERROR) != 0))
goto out;
digest_size = crypto_shash_digestsize(peer_device->connection->csums_tfm);
- digest = kmalloc(digest_size, GFP_NOIO);
- if (digest) {
- sector_t sector = peer_req->i.sector;
- unsigned int size = peer_req->i.size;
- drbd_csum_ee(peer_device->connection->csums_tfm, peer_req, digest);
- /* Free peer_req and pages before send.
- * In case we block on congestion, we could otherwise run into
- * some distributed deadlock, if the other side blocks on
- * congestion as well, because our receiver blocks in
- * drbd_alloc_pages due to pp_in_use > max_buffers. */
- drbd_free_peer_req(device, peer_req);
- peer_req = NULL;
- inc_rs_pending(peer_device);
- err = drbd_send_drequest_csum(peer_device, sector, size,
- digest, digest_size,
- P_CSUM_RS_REQUEST);
- kfree(digest);
- } else {
- drbd_err(device, "kmalloc() of digest failed.\n");
+
+ di = kmalloc(sizeof(*di) + digest_size, GFP_NOIO);
+ if (!di) {
err = -ENOMEM;
+ goto out;
}
+ di->digest_size = digest_size;
+ di->digest = (((char *)di)+sizeof(struct digest_info));
+
+ drbd_csum_bios(connection->csums_tfm, &peer_req->bios, di->digest);
+
+ /* Free pages before continuing.
+ * In case we block on congestion, we could otherwise run into
+ * some distributed deadlock, if the other side blocks on
+ * congestion as well, because our receiver blocks in
+ * drbd_alloc_pages due to pp_in_use > max_buffers. */
+ drbd_peer_req_strip_bio(peer_req);
+
+ /* Use the same drbd_peer_request for tracking resync request and for
+ * writing, if that is necessary. */
+ peer_req->digest = di;
+ peer_req->flags |= EE_HAS_DIGEST;
+
+ atomic_inc(&connection->backing_ee_cnt);
+ drbd_conflict_send_resync_request(peer_req);
+ return 0;
+
out:
- if (peer_req)
- drbd_free_peer_req(device, peer_req);
+ drbd_free_peer_req(peer_req);
if (unlikely(err))
- drbd_err(device, "drbd_send_drequest(..., csum) failed\n");
+ drbd_err(peer_device, "drbd_send_drequest(..., csum) failed\n");
return err;
}
-#define GFP_TRY (__GFP_HIGHMEM | __GFP_NOWARN)
-
static int read_for_csum(struct drbd_peer_device *peer_device, sector_t sector, int size)
{
+ struct drbd_connection *connection = peer_device->connection;
struct drbd_device *device = peer_device->device;
struct drbd_peer_request *peer_req;
if (!get_ldev(device))
return -EIO;
- /* GFP_TRY, because if there is no memory available right now, this may
- * be rescheduled for later. It is "only" background resync, after all. */
- peer_req = drbd_alloc_peer_req(peer_device, ID_SYNCER /* unused */, sector,
- size, size, GFP_TRY);
+ /* Do not wait if no memory is immediately available. */
+ peer_req = drbd_alloc_peer_req(peer_device, GFP_TRY & ~__GFP_RECLAIM,
+ size, REQ_OP_READ);
if (!peer_req)
goto defer;
- /*
- * This will be a resync write once we receive the data back from the
- * peer, assuming the checksums differ.
- */
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_add_tail(&peer_req->recv_order, &peer_device->resync_requests);
+ peer_req->flags |= EE_ON_RECV_ORDER;
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ peer_req->i.size = size;
+ peer_req->i.sector = sector;
+ /* This will be a resync write once we receive the data back from the
+ * peer, assuming the checksums differ. */
peer_req->i.type = INTERVAL_RESYNC_WRITE;
+ peer_req->requested_size = size;
+
peer_req->w.cb = w_e_send_csum;
- peer_req->opf = REQ_OP_READ;
- spin_lock_irq(&device->resource->req_lock);
- list_add_tail(&peer_req->w.list, &device->read_ee);
- spin_unlock_irq(&device->resource->req_lock);
- atomic_add(size >> 9, &device->rs_sect_ev);
+ atomic_inc(&connection->backing_ee_cnt);
+ drbd_rs_depth_queued(device, size >> 9);
+ /* ldev_ref_transfer: put_ldev in peer_req endio */
if (drbd_submit_peer_request(peer_req) == 0)
return 0;
+ atomic_sub(size >> 9, &device->rs_sect_queued);
/* If it failed because of ENOMEM, retry should help. If it failed
* because bio_add_page failed (probably broken lower level driver),
* retry may or may not help.
* If it does not, you may need to force disconnect. */
- spin_lock_irq(&device->resource->req_lock);
- list_del(&peer_req->w.list);
- spin_unlock_irq(&device->resource->req_lock);
- drbd_free_peer_req(device, peer_req);
defer:
put_ldev(device);
return -EAGAIN;
}
+static int make_one_resync_request(struct drbd_peer_device *peer_device, int discard_granularity, sector_t sector, int size)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_peer_request *peer_req;
+
+ /* Do not wait if no memory is immediately available. */
+ peer_req = drbd_alloc_peer_req(peer_device, GFP_TRY & ~__GFP_RECLAIM,
+ size, REQ_OP_WRITE);
+ if (!peer_req) {
+ drbd_err(device, "Could not allocate resync request\n");
+ return -EAGAIN;
+ }
+
+ peer_req->i.size = size;
+ peer_req->i.sector = sector;
+ peer_req->i.type = INTERVAL_RESYNC_WRITE;
+ peer_req->requested_size = size;
+
+ if (size == discard_granularity)
+ peer_req->flags |= EE_RS_THIN_REQ;
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_add_tail(&peer_req->recv_order, &peer_device->resync_requests);
+ peer_req->flags |= EE_ON_RECV_ORDER;
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ atomic_inc(&connection->backing_ee_cnt);
+ drbd_conflict_send_resync_request(peer_req);
+ return 0;
+}
+
int w_resync_timer(struct drbd_work *w, int cancel)
{
- struct drbd_device *device =
- container_of(w, struct drbd_device, resync_work);
+ struct drbd_peer_device *peer_device =
+ container_of(w, struct drbd_peer_device, resync_work);
- switch (device->state.conn) {
- case C_VERIFY_S:
- make_ov_request(first_peer_device(device), cancel);
+ switch (peer_device->repl_state[NOW]) {
+ case L_VERIFY_S:
+ make_ov_request(peer_device, cancel);
break;
- case C_SYNC_TARGET:
- make_resync_request(first_peer_device(device), cancel);
+ case L_SYNC_TARGET:
+ make_resync_request(peer_device, cancel);
+ break;
+ default:
+ if (atomic_read(&peer_device->rs_sect_in) >= peer_device->rs_in_flight) {
+ struct drbd_resource *resource = peer_device->device->resource;
+ unsigned long irq_flags;
+
+ begin_state_change(resource, &irq_flags, 0);
+ peer_device->resync_active[NEW] = false;
+ end_state_change(resource, &irq_flags, "resync-inactive");
+ }
break;
}
return 0;
}
+int w_send_dagtag(struct drbd_work *w, int cancel)
+{
+ struct drbd_connection *connection =
+ container_of(w, struct drbd_connection, send_dagtag_work);
+ struct drbd_resource *resource = connection->resource;
+ int err;
+ u64 dagtag_sector;
+
+ if (cancel)
+ return 0;
+
+ read_lock_irq(&resource->state_rwlock);
+ dagtag_sector = connection->send_dagtag;
+ /* It is OK to use the value outside the lock, because the work will be
+ * queued again if it is changed. */
+ read_unlock_irq(&resource->state_rwlock);
+
+ /* Only send if no request with a newer dagtag has been sent. This can
+ * occur if a write arrives after the state change and is processed
+ * before this work item. */
+ if (dagtag_newer_eq(connection->send.current_dagtag_sector, dagtag_sector))
+ return 0;
+
+ err = drbd_send_dagtag(connection, dagtag_sector);
+ if (err)
+ return err;
+
+ connection->send.current_dagtag_sector = dagtag_sector;
+ return 0;
+}
+
+int w_send_uuids(struct drbd_work *w, int cancel)
+{
+ struct drbd_peer_device *peer_device =
+ container_of(w, struct drbd_peer_device, propagate_uuids_work);
+
+ if (peer_device->repl_state[NOW] < L_ESTABLISHED ||
+ !test_bit(INITIAL_STATE_SENT, peer_device->flags))
+ return 0;
+
+ drbd_send_uuids(peer_device, 0, 0);
+
+ return 0;
+}
+
+bool drbd_any_flush_pending(struct drbd_resource *resource)
+{
+ unsigned long flags;
+ struct drbd_connection *primary_connection;
+ bool any_flush_pending = false;
+
+ spin_lock_irqsave(&resource->initiator_flush_lock, flags);
+ rcu_read_lock();
+ for_each_connection_rcu(primary_connection, resource) {
+ if (primary_connection->pending_flush_mask) {
+ any_flush_pending = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+ if (drbd_flush_connect_needs_restart(resource))
+ any_flush_pending = true;
+ spin_unlock_irqrestore(&resource->initiator_flush_lock, flags);
+
+ return any_flush_pending;
+}
+
+/* Caller holds initiator_flush_lock. */
+bool drbd_flush_connect_needs_restart(struct drbd_resource *resource)
+{
+ return resource->flush_connect_nodes &&
+ (resource->flush_connect_own_missing ||
+ resource->flush_connect_sequence != resource->current_flush_sequence);
+}
+
void resync_timer_fn(struct timer_list *t)
{
- struct drbd_device *device = timer_container_of(device, t,
- resync_timer);
+ struct drbd_peer_device *peer_device = timer_container_of(peer_device, t, resync_timer);
drbd_queue_work_if_unqueued(
- &first_peer_device(device)->connection->sender_work,
- &device->resync_work);
+ &peer_device->connection->sender_work,
+ &peer_device->resync_work);
}
static void fifo_set(struct fifo_buffer *fb, int value)
@@ -494,32 +875,55 @@ struct fifo_buffer *fifo_alloc(unsigned int fifo_size)
return fb;
}
-static int drbd_rs_controller(struct drbd_peer_device *peer_device, unsigned int sect_in)
+/* FIXME by choosing to calculate in nano seconds, we now have several do_div()
+ * in here, which I find very ugly.
+ */
+static int drbd_rs_controller(struct drbd_peer_device *peer_device, u64 sect_in, u64 duration_ns)
{
- struct drbd_device *device = peer_device->device;
- struct disk_conf *dc;
+ const u64 max_duration_ns = RS_MAKE_REQS_INTV_NS * 10;
+ struct drbd_peer_device_conf *pdc;
unsigned int want; /* The number of sectors we want in-flight */
int req_sect; /* Number of sectors to request in this turn */
int correction; /* Number of sectors more we need in-flight */
int cps; /* correction per invocation of drbd_rs_controller() */
int steps; /* Number of time steps to plan ahead */
int curr_corr;
- int max_sect;
+ u64 max_sect;
struct fifo_buffer *plan;
+ u64 duration_ms;
+
+ if (duration_ns == 0)
+ duration_ns = 1;
+ else if (duration_ns > max_duration_ns)
+ duration_ns = max_duration_ns;
+
+ if (duration_ns < RS_MAKE_REQS_INTV_NS) {
+ /* Scale sect_in so that it represents the number of sectors which
+ * would have arrived if the cycle had lasted the normal time
+ * (RS_MAKE_REQS_INTV). */
+ sect_in = sect_in * RS_MAKE_REQS_INTV_NS;
+ do_div(sect_in, duration_ns);
+ }
- dc = rcu_dereference(device->ldev->disk_conf);
- plan = rcu_dereference(device->rs_plan_s);
+ pdc = rcu_dereference(peer_device->conf);
+ plan = rcu_dereference(peer_device->rs_plan_s);
- steps = plan->size; /* (dc->c_plan_ahead * 10 * SLEEP_TIME) / HZ; */
+ steps = plan->size; /* (pdc->c_plan_ahead * 10 * RS_MAKE_REQS_INTV) / HZ; */
- if (device->rs_in_flight + sect_in == 0) { /* At start of resync */
- want = ((dc->resync_rate * 2 * SLEEP_TIME) / HZ) * steps;
+ if (drbd_rs_outstanding(peer_device) + sect_in == 0) { /* At start of resync */
+ want = ((pdc->resync_rate * 2 * RS_MAKE_REQS_INTV) / HZ) * steps;
} else { /* normal path */
- want = dc->c_fill_target ? dc->c_fill_target :
- sect_in * dc->c_delay_target * HZ / (SLEEP_TIME * 10);
+ if (pdc->c_fill_target) {
+ want = pdc->c_fill_target;
+ } else {
+ u64 tmp = sect_in * pdc->c_delay_target * NSEC_PER_SEC;
+
+ do_div(tmp, (duration_ns * 10));
+ want = tmp;
+ }
}
- correction = want - device->rs_in_flight - plan->total;
+ correction = want - drbd_rs_outstanding(peer_device) - plan->total;
/* Plan ahead */
cps = correction / steps;
@@ -534,36 +938,123 @@ static int drbd_rs_controller(struct drbd_peer_device *peer_device, unsigned int
if (req_sect < 0)
req_sect = 0;
- max_sect = (dc->c_max_rate * 2 * SLEEP_TIME) / HZ;
+ if (pdc->c_max_rate == 0) {
+ /* No rate limiting. */
+ max_sect = ~0ULL;
+ } else {
+ max_sect = (u64)pdc->c_max_rate * 2 * RS_MAKE_REQS_INTV_NS;
+ do_div(max_sect, NSEC_PER_SEC);
+ }
+
+ duration_ms = duration_ns;
+ do_div(duration_ms, NSEC_PER_MSEC);
+ dynamic_drbd_dbg(peer_device, "dur=%lluns (%llums) sect_in=%llu in_flight=%d wa=%u co=%d st=%d cps=%d cc=%d rs=%d mx=%llu dr=%u\n",
+ duration_ns, duration_ms, sect_in, peer_device->rs_in_flight, want, correction,
+ steps, cps, curr_corr, req_sect, max_sect,
+ READ_ONCE(peer_device->device->rs_drain_rate));
+
if (req_sect > max_sect)
req_sect = max_sect;
- /*
- drbd_warn(device, "si=%u if=%d wa=%u co=%d st=%d cps=%d pl=%d cc=%d rs=%d\n",
- sect_in, device->rs_in_flight, want, correction,
- steps, cps, device->rs_planed, curr_corr, req_sect);
- */
-
return req_sect;
}
+/* How many 4k blocks a pacing of rate_kib_s allows over duration_ns.
+ *
+ * The remainder is carried in rs_pacing_credit to maintain a stable
+ * rate whatever the actual round cadence is.
+ * duration_ns is clamped the way drbd_rs_controller() clamps it, so a
+ * round after a long pause grants at most ten intervals worth.
+ */
+static int rs_paced_number(struct drbd_peer_device *peer_device,
+ unsigned int rate_kib_s, u64 duration_ns)
+{
+ const u64 per_block = (BM_BLOCK_SIZE_4k / 1024) * (u64)NSEC_PER_SEC;
+ u64 credit, blocks;
+
+ if (duration_ns > RS_MAKE_REQS_INTV_NS * 10)
+ duration_ns = RS_MAKE_REQS_INTV_NS * 10;
+
+ credit = peer_device->rs_pacing_credit + rate_kib_s * duration_ns;
+ blocks = credit;
+ do_div(blocks, per_block);
+ if (blocks > INT_MAX)
+ blocks = INT_MAX;
+ peer_device->rs_pacing_credit = credit - blocks * per_block;
+
+ return blocks;
+}
+
+/* Say so, once per resync and net configuration, when the in-flight window is
+ * what bounds this resync and not the configured rate: the max-buffers clamp
+ * keeps trimming the request while the data actually arriving stays well below
+ * c-max-rate. max-buffers pages over the round trip is then the ceiling, and no
+ * rate setting will lift it.
+ */
+static void rs_hint_mxb_limited(struct drbd_peer_device *peer_device, int mxb,
+ unsigned int c_max_rate, unsigned int sect_in,
+ u64 duration_ns)
+{
+ u64 kib_s;
+
+ if (++peer_device->rs_mxb_limited < RS_MXB_LIMITED_ROUNDS)
+ return;
+ if (!c_max_rate || !duration_ns)
+ return;
+
+ kib_s = (u64)sect_in * NSEC_PER_SEC / 2;
+ kib_s = div64_u64(kib_s, duration_ns);
+ if (kib_s * 4 >= (u64)c_max_rate * 3)
+ return;
+
+ if (!test_and_set_bit(RS_MXB_LIMITED_LOGGED, peer_device->flags))
+ drbd_info(peer_device,
+ "resync bounded by max-buffers (%d), not by c-max-rate (%u KiB/s): %llu KiB/s arriving\n",
+ mxb, c_max_rate, kib_s);
+}
+
+/* Calculate how many 4k sized blocks we want to resync this time.
+ * Because peer nodes may have different bitmap granularity, and won't
+ * be able to clear "partial bits", make sure we try to request
+ * multiples of the bitmap block size (of myself or the peer, whichever
+ * is larger) in one go. Return value is scaled to our bm_block_size.
+ * If both peers operate at 4k granularity, or at identical granularity,
+ * this should not change behavior.
+ * If we operate at different granularities, we may need to improve our
+ * drbd_rs_controller() as well to get intended resync rates,
+ * especially if you configure for rather small c-max-rate.
+ */
static int drbd_rs_number_requests(struct drbd_peer_device *peer_device)
{
- struct drbd_device *device = peer_device->device;
+ struct drbd_net_conf *nc;
+ ktime_t duration, now;
unsigned int sect_in; /* Number of sectors that came in since the last turn */
+ unsigned int c_max_rate;
int number, mxb;
+ int effective_resync_request_size;
+ struct drbd_bitmap *bm = peer_device->device->bitmap;
+
+ sect_in = atomic_xchg(&peer_device->rs_sect_in, 0);
+ peer_device->rs_in_flight -= sect_in;
- sect_in = atomic_xchg(&device->rs_sect_in, 0);
- device->rs_in_flight -= sect_in;
+ now = ktime_get();
+ duration = ktime_sub(now, peer_device->rs_last_mk_req_kt);
+ peer_device->rs_last_mk_req_kt = now;
rcu_read_lock();
- mxb = drbd_get_max_buffers(device) / 2;
- if (rcu_dereference(device->rs_plan_s)->size) {
- number = drbd_rs_controller(peer_device, sect_in) >> (BM_BLOCK_SHIFT - 9);
- device->c_sync_rate = number * HZ * (BM_BLOCK_SIZE / 1024) / SLEEP_TIME;
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ mxb = nc ? nc->max_buffers : 0;
+ c_max_rate = rcu_dereference(peer_device->conf)->c_max_rate;
+ if (rcu_dereference(peer_device->rs_plan_s)->size) {
+ number = drbd_rs_controller(peer_device, sect_in, ktime_to_ns(duration));
+ number = sect_to_bit(number, BM_BLOCK_SHIFT_4k);
} else {
- device->c_sync_rate = rcu_dereference(device->ldev->disk_conf)->resync_rate;
- number = SLEEP_TIME * device->c_sync_rate / ((BM_BLOCK_SIZE / 1024) * HZ);
+ struct drbd_peer_device_conf *pdc = rcu_dereference(peer_device->conf);
+ unsigned int rate = pdc->resync_rate;
+
+ if (pdc->c_max_rate && pdc->c_max_rate < rate)
+ rate = pdc->c_max_rate;
+ number = rs_paced_number(peer_device, rate, ktime_to_ns(duration));
}
rcu_read_unlock();
@@ -571,40 +1062,354 @@ static int drbd_rs_number_requests(struct drbd_peer_device *peer_device)
* Otherwise we may cause the remote site to stall on drbd_alloc_pages(),
* potentially causing a distributed deadlock on congestion during
* online-verify or (checksum-based) resync, if max-buffers,
- * socket buffer sizes and resync rate settings are mis-configured. */
-
- /* note that "number" is in units of "BM_BLOCK_SIZE" (which is 4k),
+ * socket buffer sizes and resync rate settings are mis-configured.
+ * Note that "number" is in units of "bm_bytes_per_bit",
* mxb (as used here, and in drbd_alloc_pages on the peer) is
- * "number of pages" (typically also 4k),
- * but "rs_in_flight" is in "sectors" (512 Byte). */
- if (mxb - device->rs_in_flight/8 < number)
- number = mxb - device->rs_in_flight/8;
+ * "number of pages" (typically 4k), and "rs_in_flight" is in "sectors"
+ * (512 Byte). Convert everything to sectors and back.
+ */
+ {
+ int mxb_sect = mxb << (PAGE_SHIFT - 9);
+ int num_sect = bit_to_sect(number, BM_BLOCK_SHIFT_4k);
+
+ if (mxb_sect - drbd_rs_outstanding(peer_device) < num_sect) {
+ num_sect = mxb_sect - drbd_rs_outstanding(peer_device);
+ number = sect_to_bit(num_sect, BM_BLOCK_SHIFT_4k);
+ rs_hint_mxb_limited(peer_device, mxb, c_max_rate, sect_in,
+ ktime_to_ns(duration));
+ } else {
+ peer_device->rs_mxb_limited = 0;
+ }
+ }
- return number;
+ /* BM_BLOCK_SIZE_MAX/BM_BLOCK_SIZE_4k? Maybe. But do not round up unless we have to. */
+ effective_resync_request_size =
+ 1 << (max(bm->bm_block_shift, peer_device->bm_block_shift) - BM_BLOCK_SHIFT_4k);
+ number = ALIGN(number, effective_resync_request_size);
+ peer_device->c_sync_rate = number * HZ * (BM_BLOCK_SIZE_4k/1024) / RS_MAKE_REQS_INTV;
+ return number >> (bm->bm_block_shift - BM_BLOCK_SHIFT_4k);
}
-static int make_resync_request(struct drbd_peer_device *const peer_device, int cancel)
+static int resync_delay(bool request_ok, int number, int done)
{
- struct drbd_device *const device = peer_device->device;
- struct drbd_connection *const connection = peer_device ? peer_device->connection : NULL;
- unsigned long bit;
- sector_t sector;
- const sector_t capacity = get_capacity(device->vdisk);
- int max_bio_size;
- int number, rollback_i, size;
- int align, requeue = 0;
- int i = 0;
- int discard_granularity = 0;
+ if (request_ok && number > 0 && done > 0) {
+ /* Requests in-flight. Adjusting the standard delay to
+ * mitigate rounding and other errors, that cause 'done'
+ * to be different from the optimal 'number'. (usually
+ * in the range of 66ms to 133ms) */
+ return RS_MAKE_REQS_INTV * done / number;
+ }
+
+ return RS_MAKE_REQS_INTV;
+}
+
+void drbd_rs_all_in_flight_came_back(struct drbd_peer_device *peer_device, int rs_sect_in)
+{
+ unsigned int max_bio_size_kb = DRBD_MAX_BIO_SIZE / 1024;
+ struct drbd_device *device = peer_device->device;
+ unsigned int c_max_rate, interval, latency, m, amount_kb;
+ unsigned int rs_kib_in = rs_sect_in / 2;
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+ ktime_t latency_kt;
+ bool kickstart;
+
+ /*
+ * Not making requests, last reply in: w_resync_timer() clears
+ * resync_active. Keep the pacing below for SyncTarget and VerifyS,
+ * which make new requests from that work.
+ */
+ if (repl_state != L_SYNC_TARGET && repl_state != L_VERIFY_S) {
+ drbd_queue_work_if_unqueued(&peer_device->connection->sender_work,
+ &peer_device->resync_work);
+ return;
+ }
+
+ if (get_ldev(device)) {
+ max_bio_size_kb = queue_max_hw_sectors(device->rq_queue) / 2;
+ put_ldev(device);
+ }
+
+ rcu_read_lock();
+ c_max_rate = rcu_dereference(peer_device->conf)->c_max_rate;
+ rcu_read_unlock();
+
+ latency_kt = ktime_sub(ktime_get(), peer_device->rs_last_mk_req_kt);
+ latency = nsecs_to_jiffies(ktime_to_ns(latency_kt));
+
+ m = max_bio_size_kb > rs_kib_in ? max_bio_size_kb / rs_kib_in : 1;
+ if (c_max_rate != 0)
+ interval = rs_kib_in * m * HZ / c_max_rate;
+ else
+ interval = 0;
+ /* interval holds the ideal pace in which we should request max_bio_size */
+
+ if (repl_state == L_SYNC_TARGET) {
+ /* Only run resync_work early if we are definitely making
+ * progress. Otherwise we might continually lock a resync
+ * extent even when all the requests are canceled. This can
+ * cause application IO to be blocked for an indefinitely long
+ * time. */
+ if (test_bit(RS_REQUEST_UNSUCCESSFUL, peer_device->flags))
+ return;
+ }
+
+ amount_kb = c_max_rate / (HZ / RS_MAKE_REQS_INTV);
+ kickstart = rs_kib_in < amount_kb / 2 && latency < RS_MAKE_REQS_INTV / 2;
+ /* In case the latency of the link and remote IO subsystem is small and
+ the controller was clearly issuing a too small number of requests,
+ kickstart it by scheduling it immediately */
+
+ if (kickstart || interval <= latency) {
+ drbd_queue_work_if_unqueued(
+ &peer_device->connection->sender_work,
+ &peer_device->resync_work);
+ return;
+ }
+
+ if (interval < RS_MAKE_REQS_INTV)
+ mod_timer(&peer_device->resync_timer, jiffies + (interval - latency));
+}
+
+static void drbd_enable_peer_replication(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
+ unsigned long irq_flags;
+ struct drbd_peer_device *peer_device;
+
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ for_each_peer_device(peer_device, device)
+ peer_device->peer_replication[NEW] = true;
+ end_state_change(resource, &irq_flags, "enable-peer-replication");
+}
+
+/* Returns whether whole resync is finished. */
+static bool drbd_resync_check_finished(struct drbd_peer_device *peer_device)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_resource *resource = connection->resource;
+ bool resync_requests_complete;
+ unsigned long bitmap_weight;
+ unsigned long last_resync_pass_bits;
+ bool peer_replication;
+
+ /* Test whether resync pass finished */
+ if (drbd_bm_find_next(peer_device, peer_device->resync_next_bit) < DRBD_END_OF_BITMAP)
+ return false;
+
+ if (drbd_any_flush_pending(resource))
+ return false;
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ resync_requests_complete = list_empty(&peer_device->resync_requests);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ if (!resync_requests_complete)
+ return false;
+
+ last_resync_pass_bits = peer_device->last_resync_pass_bits;
+ bitmap_weight = drbd_bm_total_weight(peer_device);
+ peer_device->last_resync_pass_bits = bitmap_weight;
+
+ peer_replication = drbd_all_peer_replication(device, NOW);
+ dynamic_drbd_dbg(peer_device, "Resync pass complete last:%lu out-of-sync:%lu failed:%lu replication:%s\n",
+ last_resync_pass_bits, bitmap_weight, peer_device->rs_failed,
+ peer_replication ? "enabled" : "disabled");
+
+ if (!peer_replication) {
+ if (peer_device->rs_failed == 0 && bitmap_weight > 0 &&
+ bitmap_weight < last_resync_pass_bits / 2) {
+ /* Start next pass with replication still disabled */
+ peer_device->resync_next_bit = 0;
+ return false;
+ }
+
+ drbd_enable_peer_replication(device);
+ return false;
+ }
+
+ if (peer_device->rs_failed == 0 && bitmap_weight > 0) {
+ /* Start next pass. Replication is enabled. */
+ peer_device->resync_next_bit = 0;
+ return false;
+ }
+
+ drbd_resync_finished(peer_device, D_MASK);
+ return true;
+}
+
+static bool send_buffer_half_full(struct drbd_peer_device *peer_device)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_transport *transport = &connection->transport;
+ bool half_full = false;
+
+ mutex_lock(&connection->mutex[DATA_STREAM]);
+ if (transport->class->ops.stream_ok(transport, DATA_STREAM)) {
+ struct drbd_transport_stats transport_stats;
+ int queued, sndbuf;
+
+ transport->class->ops.stats(transport, &transport_stats);
+ queued = transport_stats.send_buffer_used;
+ sndbuf = transport_stats.send_buffer_size;
+ if (queued > sndbuf / 2) {
+ half_full = true;
+ transport->class->ops.hint(transport, DATA_STREAM, NOSPACE);
+ }
+ } else {
+ half_full = true;
+ }
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+
+ return half_full;
+}
+
+static int optimal_bits_for_alignment(unsigned long bit, int bm_block_shift)
+{
+ int max_bio_bits = DRBD_MAX_BIO_SIZE >> bm_block_shift;
+
+ /* under the assumption that we find a big block of out-of-sync blocks
+ in the bitmap, calculate the optimal request size so that the
+ request sizes get bigger, and each request is "perfectly" aligned.
+ (In case the backing device is a RAID5)
+ for an odd number, it returns 1.
+ for anything dividable by 2, it returns 2.
+ for 3 it returns 1 so that the next request size can be 4.
+ and so on...
+ */
+
+ /* Only consider the lower order bits up to the size of max_bio_bits.
+ * This prevents overflows when converting to int. */
+ bit = bit & (max_bio_bits - 1);
+
+ if (bit == 0)
+ return max_bio_bits;
+
+ return 1 << __ffs(bit);
+}
+
+static int round_to_powerof_2(int value)
+{
+ int l2 = fls(value) - 1;
+ int smaller = 1 << l2;
+ int bigger = smaller << 1;
+
+ if (value == 0)
+ return 0;
+
+ return value - smaller < bigger - value ? smaller : bigger;
+}
+
+static bool adjacent(sector_t sector1, int size, sector_t sector2)
+{
+ return sector1 + (size >> SECTOR_SHIFT) == sector2;
+}
+
+/* make_resync_request() - initiate resync requests as required
+ *
+ * Request handling flow:
+ *
+ * checksum resync
+ * make_resync_request --------+
+ * | v
+ * | read_for_csum
+ * | |
+ * | v
+ * | drbd_submit_peer_request
+ * | |
+ * | ... backing device
+ * | |
+ * | v
+ * | drbd_peer_request_endio
+ * | |
+ * | v
+ * | drbd_endio_read_sec_final
+ * | |
+ * V ... sender_work
+ * make_one_resync_request |
+ * | v
+ * +---------------- w_e_send_csum
+ * |
+ * v conflict
+ * drbd_conflict_send_resync_request -------+
+ * | ^ |
+ * | | ...
+ * | | |
+ * | | v
+ * v +---- drbd_do_submit_conflict
+ * send_resync_request
+ * |
+ * ... via peer
+ * |
+ * +----------------------------+
+ * | |
+ * v v
+ * receive_RSDataReply receive_rs_deallocated
+ * | |
+ * | ... using list resync_requests
+ * | |
+ * v v
+ * recv_resync_read drbd_process_rs_discards
+ * | |
+ * | v
+ * +----------------- drbd_submit_rs_discard
+ * |
+ * v conflict
+ * drbd_conflict_submit_resync_request -----+
+ * | ^ |
+ * | | ...
+ * | | |
+ * | | v
+ * v +---- drbd_do_submit_conflict
+ * drbd_submit_peer_request
+ * |
+ * ... backing device
+ * |
+ * v
+ * drbd_peer_request_endio
+ * |
+ * v
+ * drbd_endio_write_sec_final
+ * |
+ * ... done_ee
+ * |
+ * v
+ * drbd_finish_peer_reqs
+ * |
+ * v
+ * e_end_resync_block
+ * |
+ * v
+ * drbd_resync_request_complete
+ */
+static int make_resync_request(struct drbd_peer_device *peer_device, int cancel)
+{
+ int optimal_bits_alignment, optimal_bits_rate, discard_granularity = 0;
+ int number = 0, rollback_i, size = 0, i = 0, optimal_bits;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_bitmap *bm;
+ const sector_t capacity = get_capacity(device->vdisk);
+ bool request_ok = true;
+ unsigned long bit;
+ sector_t sector, prev_sector = 0;
+ unsigned int peer_bm_block_shift = peer_device->bm_block_shift;
+ unsigned int bm_block_shift, bits_per_peer_bit;
if (unlikely(cancel))
return 0;
- if (device->rs_total == 0) {
- /* empty resync? */
- drbd_resync_finished(peer_device);
+ if (test_bit(SYNC_TARGET_TO_BEHIND, peer_device->flags)) {
+ /* If a P_RS_CANCEL_AHEAD on control socket overtook the
+ * already queued data and state change to Ahead/Behind,
+ * don't add more resync requests, just wait it out.
+ * Keep polling: once we are Behind, w_resync_timer() clears
+ * resync_active; once all answers are in, nothing else
+ * re-arms this timer.
+ */
+ mod_timer(&peer_device->resync_timer, jiffies + RS_MAKE_REQS_INTV);
return 0;
}
-
if (!get_ldev(device)) {
/* Since we only need to access device->rsync a
get_ldev_if_state(device,D_FAILED) would be sufficient, but
@@ -613,6 +1418,15 @@ static int make_resync_request(struct drbd_peer_device *const peer_device, int c
drbd_err(device, "Disk broke down during resync!\n");
return 0;
}
+ bm = device->bitmap;
+
+ if (drbd_resync_check_finished(peer_device))
+ goto out_put_ldev;
+
+ if (send_buffer_half_full(peer_device)) {
+ /* We still want to reschedule ourselves, so do not return. */
+ goto skip_request;
+ }
if (connection->agreed_features & DRBD_FF_THIN_RESYNC) {
rcu_read_lock();
@@ -620,151 +1434,261 @@ static int make_resync_request(struct drbd_peer_device *const peer_device, int c
rcu_read_unlock();
}
- max_bio_size = queue_max_hw_sectors(device->rq_queue) << 9;
+ bm_block_shift = bm->bm_block_shift;
number = drbd_rs_number_requests(peer_device);
- if (number <= 0)
- goto requeue;
+ if (number < discard_granularity >> bm_block_shift)
+ number = discard_granularity >> bm_block_shift;
- for (i = 0; i < number; i++) {
- /* Stop generating RS requests when half of the send buffer is filled,
- * but notify TCP that we'd like to have more space. */
- mutex_lock(&connection->data.mutex);
- if (connection->data.socket) {
- struct sock *sk = connection->data.socket->sk;
- int queued = sk->sk_wmem_queued;
- int sndbuf = sk->sk_sndbuf;
- if (queued > sndbuf / 2) {
- requeue = 1;
- sk_set_nospace(sk);
- }
- } else
- requeue = 1;
- mutex_unlock(&connection->data.mutex);
- if (requeue)
- goto requeue;
+ /*
+ * Drain resync requests when we jump back to avoid conflicts that are
+ * resolved in an arbitrary order, leading to an unexpected ordering of
+ * requests being completed.
+ */
+ if (test_bit(RS_REQUEST_UNSUCCESSFUL, peer_device->flags) &&
+ peer_device->rs_in_flight > 0) {
+ /*
+ * The rs_in_flight counter does not include discards waiting
+ * to be merged. Hence we may jump back while there are
+ * discards waiting to be merged. In this situation, we may
+ * make a resync request that conflicts with a discard. Allow
+ * the discard to be merged here so that the conflict is
+ * resolved.
+ */
+ drbd_process_rs_discards(peer_device, false);
+ goto skip_request;
+ }
-next_sector:
- size = BM_BLOCK_SIZE;
- bit = drbd_bm_find_next(device, device->bm_resync_fo);
+ /* don't let rs_sectors_came_in() re-schedule us "early"
+ * just because the first reply came "fast", ... */
+ peer_device->rs_in_flight += bm_bit_to_sect(device->bitmap, number);
- if (bit == DRBD_END_OF_BITMAP) {
- device->bm_resync_fo = drbd_bm_bits(device);
- put_ldev(device);
- return 0;
+ bits_per_peer_bit = peer_bm_block_shift > bm_block_shift ?
+ 1 << (peer_bm_block_shift - bm_block_shift) : 1;
+
+ clear_bit(RS_REQUEST_UNSUCCESSFUL, peer_device->flags);
+ for (; i < number; i += bits_per_peer_bit) {
+ int err;
+
+ /* If we are aborting the requests or the peer is canceling
+ * them, there is no need to flood the connection with
+ * requests. Back off now. */
+ if (i > 0 && test_bit(RS_REQUEST_UNSUCCESSFUL, peer_device->flags)) {
+ request_ok = false;
+ goto request_done;
}
- sector = BM_BIT_TO_SECT(bit);
+ if ((number - i) < discard_granularity >> bm_block_shift)
+ goto request_done;
- if (drbd_try_rs_begin_io(peer_device, sector)) {
- device->bm_resync_fo = bit;
- goto requeue;
+ bit = drbd_bm_find_next(peer_device, peer_device->resync_next_bit);
+ if (bit == DRBD_END_OF_BITMAP) {
+ peer_device->resync_next_bit = drbd_bm_bits(device);
+ goto request_done;
}
- device->bm_resync_fo = bit + 1;
- if (unlikely(drbd_bm_test_bit(device, bit) == 0)) {
- drbd_rs_complete_io(device, sector);
- goto next_sector;
+ bit = ALIGN_DOWN(bit, bits_per_peer_bit);
+ sector = bm_bit_to_sect(bm, bit);
+
+ if (drbd_rs_c_min_rate_throttle(peer_device)) {
+ peer_device->resync_next_bit = bit;
+ goto request_done;
}
-#if DRBD_MAX_BIO_SIZE > BM_BLOCK_SIZE
- /* try to find some adjacent bits.
- * we stop if we have already the maximum req size.
- *
- * Additionally always align bigger requests, in order to
- * be prepared for all stripe sizes of software RAIDs.
+ /* read_for_csum() reads at this node's own backing device, so
+ * a checksum round stops at the depth bound, before the cursor
+ * moves past this request.
*/
- align = 1;
- rollback_i = i;
- while (i < number) {
- if (size + BM_BLOCK_SIZE > max_bio_size)
- break;
+ if (peer_device->use_csums && drbd_rs_depth_exceeded(peer_device)) {
+ peer_device->resync_next_bit = bit;
+ drbd_rs_depth_defer(peer_device);
+ goto request_done;
+ }
- /* Be always aligned */
- if (sector & ((1<<(align+3))-1))
- break;
+ if (adjacent(prev_sector, size, sector) && (number - i) < size >> bm_block_shift) {
+ /* When making requests in an out-of-sync area, ensure that the size
+ of successive requests does not decrease. This allows the next
+ make_resync_request call to start with optimal alignment. */
+ goto request_done;
+ }
+ prev_sector = sector;
+ size = bm_block_size(bm) * bits_per_peer_bit;
+ optimal_bits_alignment = optimal_bits_for_alignment(bit,
+ max(peer_bm_block_shift, bm_block_shift));
+ optimal_bits_rate = round_to_powerof_2(number - i);
+ optimal_bits = min(optimal_bits_alignment, optimal_bits_rate) - 1;
+
+ /* try to find some adjacent bits. */
+ rollback_i = i;
+ while (optimal_bits-- > 0) {
if (discard_granularity && size == discard_granularity)
break;
- /* do not cross extent boundaries */
- if (((bit+1) & BM_BLOCKS_PER_BM_EXT_MASK) == 0)
+ if (drbd_bm_count_bits(device, peer_device->bitmap_index,
+ bit + bits_per_peer_bit,
+ bit + bits_per_peer_bit * 2 - 1) == 0)
break;
- /* now, is it actually dirty, after all?
- * caution, drbd_bm_test_bit is tri-state for some
- * obscure reason; ( b == 0 ) would get the out-of-band
- * only accidentally right because of the "oddly sized"
- * adjustment below */
- if (drbd_bm_test_bit(device, bit+1) != 1)
- break;
- bit++;
- size += BM_BLOCK_SIZE;
- if ((BM_BLOCK_SIZE << align) <= size)
- align++;
- i++;
- }
- /* if we merged some,
- * reset the offset to start the next drbd_bm_find_next from */
- if (size > BM_BLOCK_SIZE)
- device->bm_resync_fo = bit + 1;
-#endif
+ size += bm_block_size(bm) * bits_per_peer_bit;
+ bit += bits_per_peer_bit;
+ i += bits_per_peer_bit;
+ }
+
+ /* set the offset to start the next drbd_bm_find_next from */
+ peer_device->resync_next_bit = bit + bits_per_peer_bit;
/* adjust very last sectors, in case we are oddly sized */
if (sector + (size>>9) > capacity)
size = (capacity-sector)<<9;
- if (device->use_csums) {
- switch (read_for_csum(peer_device, sector, size)) {
- case -EIO: /* Disk failure */
- put_ldev(device);
- return -EIO;
- case -EAGAIN: /* allocation failed, or ldev busy */
- drbd_rs_complete_io(device, sector);
- device->bm_resync_fo = BM_SECT_TO_BIT(sector);
- i = rollback_i;
- goto requeue;
- case 0:
- /* everything ok */
- break;
- default:
- BUG();
- }
- } else {
- int err;
-
- inc_rs_pending(peer_device);
- err = drbd_send_drequest(peer_device,
- size == discard_granularity ? P_RS_THIN_REQ : P_RS_DATA_REQUEST,
- sector, size, ID_SYNCER);
- if (err) {
- drbd_err(device, "drbd_send_drequest() failed, aborting...\n");
- dec_rs_pending(peer_device);
- put_ldev(device);
- return err;
- }
+ if (peer_device->use_csums)
+ err = read_for_csum(peer_device, sector, size);
+ else
+ err = make_one_resync_request(peer_device, discard_granularity, sector, size);
+
+ switch (err) {
+ case -EIO: /* Disk failure */
+ put_ldev(device);
+ return -EIO;
+ case -EAGAIN: /* allocation failed, or ldev busy */
+ set_bit(RS_REQUEST_UNSUCCESSFUL, peer_device->flags);
+ peer_device->resync_next_bit = bm_sect_to_bit(bm, sector);
+ i = rollback_i;
+ goto request_done;
+ case 0:
+ /* everything ok */
+ break;
+ default:
+ BUG();
}
}
- if (device->bm_resync_fo >= drbd_bm_bits(device)) {
- /* last syncer _request_ was sent,
- * but the P_RS_DATA_REPLY not yet received. sync will end (and
- * next sync group will resume), as soon as we receive the last
- * resync data block, and the last bit is cleared.
- * until then resync "work" is "inactive" ...
+request_done:
+ /* ... but do a correction, in case we had to break/goto request_done; */
+ peer_device->rs_in_flight -= (number - i) * bm_sect_per_bit(bm);
+
+ if (peer_device->resync_next_bit >= drbd_bm_bits(device)) {
+ /*
+ * Last resync request sent in this pass. There will be no
+ * replies for subsequent sectors so discard merging should
+ * stop here.
*/
- put_ldev(device);
- return 0;
+ drbd_last_resync_request(peer_device, false);
}
- requeue:
- device->rs_in_flight += (i << (BM_BLOCK_SHIFT - 9));
- mod_timer(&device->resync_timer, jiffies + SLEEP_TIME);
+skip_request:
+ /* Always reschedule ourselves as a form of polling to detect the end of a resync pass. */
+ mod_timer(&peer_device->resync_timer, jiffies + resync_delay(request_ok, number, i));
+
+ if (peer_device->rs_in_flight > 0 && request_ok) {
+ int rs_sect_in = atomic_read(&peer_device->rs_sect_in);
+
+ if (rs_sect_in >= peer_device->rs_in_flight) {
+ /*
+ * In case replies were received before correction to
+ * rs_in_flight, consider whether to schedule ourselves
+ * early.
+ */
+ drbd_rs_all_in_flight_came_back(peer_device, rs_sect_in);
+ }
+ }
+out_put_ldev:
put_ldev(device);
return 0;
}
+static void send_ov_request(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct dagtag_find_result dagtag_result;
+ enum drbd_packet cmd = peer_device->connection->agreed_features & DRBD_FF_RESYNC_DAGTAG ?
+ P_OV_DAGTAG_REQ : P_OV_REQUEST;
+
+ inc_rs_pending(peer_device);
+
+ dagtag_result = find_current_dagtag(peer_device->device->resource);
+ if (dagtag_result.err) {
+ change_cstate(peer_device->connection, C_DISCONNECTING, CS_HARD);
+ return;
+ }
+
+ drbd_send_rs_request(peer_device, cmd,
+ peer_req->i.sector, peer_req->i.size, peer_req->block_id,
+ dagtag_result.node_id, dagtag_result.dagtag);
+}
+
+static void drbd_conflict_send_ov_request(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+
+ spin_lock_irq(&device->interval_lock);
+ if (drbd_find_conflict(device, &peer_req->i, 0))
+ set_bit(INTERVAL_CONFLICT, &peer_req->i.flags);
+ drbd_insert_interval(&device->requests, &peer_req->i);
+ set_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags);
+ /* Mark as submitted now, since OV requests do not have a second
+ * conflict resolution stage when the reply is received. */
+ set_bit(INTERVAL_SUBMITTED, &peer_req->i.flags);
+ spin_unlock_irq(&device->interval_lock);
+
+ /* If there were conflicts we will skip the block. However, we send a
+ * request anyway because the protocol doesn't include any way to mark
+ * a block as skipped without having sent any request. */
+ send_ov_request(peer_req);
+}
+
+/* make_ov_request() - initiate online verify requests as required
+ *
+ * Request handling flow:
+ *
+ * make_ov_request
+ * |
+ * v
+ * drbd_conflict_send_ov_request
+ * |
+ * v
+ * send_ov_request
+ * |
+ * ... via peer
+ * |
+ * v
+ * receive_dagtag_ov_reply
+ * |
+ * v
+ * receive_common_ov_reply
+ * |
+ * v dagtag waiting
+ * drbd_peer_resync_read --------------+
+ * | |
+ * | ... dagtag_wait_ee
+ * | |
+ * | v
+ * +--------------- release_dagtag_wait
+ * |
+ * v
+ * drbd_conflict_submit_peer_read
+ * |
+ * v
+ * drbd_submit_peer_request
+ * |
+ * ... backing device
+ * |
+ * v
+ * drbd_peer_request_endio
+ * |
+ * v
+ * drbd_endio_read_sec_final
+ * |
+ * ... sender_work
+ * |
+ * v
+ * w_e_end_ov_reply
+ */
static int make_ov_request(struct drbd_peer_device *peer_device, int cancel)
{
struct drbd_device *device = peer_device->device;
+ struct drbd_bitmap *bm;
+ struct drbd_connection *connection = peer_device->connection;
int number, i, size;
sector_t sector;
const sector_t capacity = get_capacity(device->vdisk);
@@ -773,262 +1697,541 @@ static int make_ov_request(struct drbd_peer_device *peer_device, int cancel)
if (unlikely(cancel))
return 1;
+ if (!get_ldev(device))
+ return 0;
+
+ bm = device->bitmap;
number = drbd_rs_number_requests(peer_device);
+ sector = peer_device->ov_position;
- sector = device->ov_position;
+ /* don't let rs_sectors_came_in() re-schedule us "early"
+ * just because the first reply came "fast", ... */
+ peer_device->rs_in_flight += bm_bit_to_sect(bm, number);
for (i = 0; i < number; i++) {
+ struct drbd_peer_request *peer_req;
+
if (sector >= capacity)
- return 1;
+ break;
/* We check for "finished" only in the reply path:
* w_e_end_ov_reply().
* We need to send at least one request out. */
- stop_sector_reached = i > 0
- && verify_can_do_stop_sector(device)
- && sector >= device->ov_stop_sector;
+ stop_sector_reached = sector > peer_device->ov_start_sector
+ && verify_can_do_stop_sector(peer_device)
+ && sector >= peer_device->ov_stop_sector;
if (stop_sector_reached)
break;
- size = BM_BLOCK_SIZE;
-
- if (drbd_try_rs_begin_io(peer_device, sector)) {
- device->ov_position = sector;
- goto requeue;
- }
+ if (drbd_rs_c_min_rate_throttle(peer_device))
+ break;
+ size = bm_block_size(bm);
if (sector + (size>>9) > capacity)
size = (capacity-sector)<<9;
- inc_rs_pending(peer_device);
- if (drbd_send_ov_request(first_peer_device(device), sector, size)) {
- dec_rs_pending(peer_device);
+ /* Do not wait if no memory is immediately available.
+ * Don't allocate pages yet - we only need them when we
+ * receive the P_OV_REPLY and need to read local data.
+ * That is important to not exhaust max_buffers prematurely.
+ */
+ peer_req = drbd_alloc_peer_req(peer_device, GFP_TRY & ~__GFP_RECLAIM, size,
+ REQ_NO_BIO);
+ if (!peer_req) {
+ drbd_err(device, "Could not allocate online verify request\n");
+ put_ldev(device);
return 0;
}
- sector += BM_SECT_PER_BIT;
- }
- device->ov_position = sector;
- requeue:
- device->rs_in_flight += (i << (BM_BLOCK_SHIFT - 9));
- if (i == 0 || !stop_sector_reached)
- mod_timer(&device->resync_timer, jiffies + SLEEP_TIME);
+ peer_req->i.size = size;
+ peer_req->i.sector = sector;
+ peer_req->i.type = INTERVAL_OV_READ_SOURCE;
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_add_tail(&peer_req->recv_order, &connection->peer_reads);
+ peer_req->flags |= EE_ON_RECV_ORDER;
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ drbd_conflict_send_ov_request(peer_req);
+
+ sector += bm_sect_per_bit(bm);
+ }
+ /* ... but do a correction, in case we had to break; ... */
+ peer_device->rs_in_flight -= bm_bit_to_sect(bm, number-i);
+ peer_device->ov_position = sector;
+ if (stop_sector_reached)
+ goto out_ok;
+ /* ... and in case that raced with the receiver,
+ * reschedule ourselves right now */
+ if (i > 0 && atomic_read(&peer_device->rs_sect_in) >= peer_device->rs_in_flight)
+ drbd_queue_work_if_unqueued(
+ &peer_device->connection->sender_work,
+ &peer_device->resync_work);
+ else
+ mod_timer(&peer_device->resync_timer, jiffies + resync_delay(true, number, i));
+out_ok:
+ put_ldev(device);
return 1;
}
-int w_ov_finished(struct drbd_work *w, int cancel)
+struct resync_finished_work {
+ struct drbd_peer_device_work pdw;
+ enum drbd_disk_state new_peer_disk_state;
+};
+
+static int w_resync_finished(struct drbd_work *w, int cancel)
{
- struct drbd_device_work *dw =
- container_of(w, struct drbd_device_work, w);
- struct drbd_device *device = dw->device;
- kfree(dw);
- ov_out_of_sync_print(first_peer_device(device));
- drbd_resync_finished(first_peer_device(device));
+ struct resync_finished_work *rfw = container_of(
+ container_of(w, struct drbd_peer_device_work, w),
+ struct resync_finished_work, pdw);
+
+ if (!cancel)
+ drbd_resync_finished(rfw->pdw.peer_device, rfw->new_peer_disk_state);
+ kfree(rfw);
return 0;
}
-static int w_resync_finished(struct drbd_work *w, int cancel)
+static long ping_timeout(struct drbd_connection *connection)
{
- struct drbd_device_work *dw =
- container_of(w, struct drbd_device_work, w);
- struct drbd_device *device = dw->device;
- kfree(dw);
+ struct drbd_net_conf *nc;
+ long timeout;
- drbd_resync_finished(first_peer_device(device));
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ timeout = nc->ping_timeo * HZ / 10;
+ rcu_read_unlock();
- return 0;
+ return timeout;
+}
+
+static int send_ping_peer(struct drbd_connection *connection)
+{
+ bool was_pending = test_and_set_bit(PING_PENDING, &connection->flags);
+ int err = 0;
+
+ if (!was_pending) {
+ err = drbd_send_ping(connection);
+ if (err)
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ }
+
+ return err;
+}
+
+void drbd_ping_peer(struct drbd_connection *connection)
+{
+ long r, timeout = ping_timeout(connection);
+ int err;
+
+ err = send_ping_peer(connection);
+ if (err)
+ return;
+
+ r = wait_event_timeout(connection->resource->state_wait,
+ !test_bit(PING_PENDING, &connection->flags) ||
+ connection->cstate[NOW] < C_CONNECTED,
+ timeout);
+ if (r > 0)
+ return;
+
+ drbd_warn(connection, "PingAck did not arrive in time\n");
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+}
+
+/* caller needs to hold rcu_read_lock, state_rwlock, adm_mutex or conf_update */
+struct drbd_peer_device *peer_device_by_node_id(struct drbd_device *device, int node_id)
+{
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->node_id == node_id)
+ return peer_device;
+ }
+
+ return NULL;
}
-static void ping_peer(struct drbd_device *device)
+static void __outdate_peer_disk_by_mask(struct drbd_device *device, u64 nodes)
{
- struct drbd_connection *connection = first_peer_device(device)->connection;
+ struct drbd_peer_device *peer_device;
+ int node_id;
- clear_bit(GOT_PING_ACK, &connection->flags);
- request_ping(connection);
- wait_event(connection->ping_wait,
- test_bit(GOT_PING_ACK, &connection->flags) || device->state.conn < C_CONNECTED);
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (!(nodes & NODE_MASK(node_id)))
+ continue;
+ peer_device = peer_device_by_node_id(device, node_id);
+ if (peer_device && peer_device->disk_state[NEW] >= D_CONSISTENT)
+ __change_peer_disk_state(peer_device, D_OUTDATED);
+ }
}
-int drbd_resync_finished(struct drbd_peer_device *peer_device)
+/* An annoying corner case is if we are resync target towards a bunch
+ of nodes. One of the resyncs finished as STABLE_RESYNC, the others
+ as UNSTABLE_RESYNC. */
+static bool was_resync_stable(struct drbd_peer_device *peer_device)
{
struct drbd_device *device = peer_device->device;
+
+ if (test_bit(UNSTABLE_RESYNC, peer_device->flags) &&
+ !test_bit(STABLE_RESYNC, &device->flags))
+ return false;
+
+ set_bit(STABLE_RESYNC, &device->flags);
+ /* that STABLE_RESYNC bit gets reset if in any other ongoing resync
+ we receive something from a resync source that is marked with
+ UNSTABLE RESYNC. */
+
+ return true;
+}
+
+static u64 __cancel_other_resyncs(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ u64 target_m = 0;
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->repl_state[NEW] == L_PAUSED_SYNC_T) {
+ target_m |= NODE_MASK(peer_device->node_id);
+ __change_repl_state(peer_device, L_ESTABLISHED);
+ }
+ }
+
+ return target_m;
+}
+
+static void resync_again(struct drbd_device *device, u64 source_m, u64 target_m)
+{
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->resync_again) {
+ u64 m = NODE_MASK(peer_device->node_id);
+ enum drbd_repl_state new_repl_state =
+ source_m & m ? L_WF_BITMAP_S :
+ target_m & m ? L_WF_BITMAP_T :
+ L_ESTABLISHED;
+
+ if (new_repl_state != L_ESTABLISHED) {
+ peer_device->resync_again--;
+ begin_state_change_locked(device->resource, CS_VERBOSE);
+ __change_repl_state(peer_device, new_repl_state);
+ end_state_change_locked(device->resource, "resync-again");
+ }
+ }
+ }
+}
+
+static void init_resync_stable_bits(struct drbd_peer_device *first_target_pd)
+{
+ struct drbd_device *device = first_target_pd->device;
+ struct drbd_peer_device *peer_device;
+
+ clear_bit(UNSTABLE_RESYNC, first_target_pd->flags);
+
+ /* Clear the device wide STABLE_RESYNC flag when becoming
+ resync target on the first peer_device. */
+ for_each_peer_device(peer_device, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+
+ if (peer_device == first_target_pd)
+ continue;
+ if (repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T)
+ return;
+ }
+ clear_bit(STABLE_RESYNC, &device->flags);
+}
+
+static void after_reconciliation_resync(struct drbd_connection *connection)
+{
+ struct drbd_connection *lost_peer =
+ drbd_get_connection_by_node_id(connection->resource,
+ connection->after_reconciliation.lost_node_id);
+
+ if (lost_peer) {
+ if (lost_peer->cstate[NOW] < C_CONNECTED)
+ atomic64_set(&lost_peer->last_dagtag_sector,
+ connection->after_reconciliation.dagtag_sector);
+
+ kref_put(&lost_peer->kref, drbd_destroy_connection);
+ }
+
+ connection->after_reconciliation.lost_node_id = -1;
+}
+
+static void try_to_get_resynced_from_primary(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
+ struct drbd_connection *connection;
+
+ read_lock_irq(&resource->state_rwlock);
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->connection->peer_role[NEW] == R_PRIMARY &&
+ peer_device->disk_state[NEW] == D_UP_TO_DATE)
+ goto found;
+ }
+ peer_device = NULL;
+found:
+ read_unlock_irq(&resource->state_rwlock);
+
+ if (!peer_device)
+ return;
+
+ connection = peer_device->connection;
+ if (connection->agreed_pro_version < 118) {
+ drbd_warn(connection,
+ "peer is lower than protocol vers 118, reconnecting to get resynced\n");
+ change_cstate(connection, C_PROTOCOL_ERROR, CS_HARD);
+ return;
+ }
+
+ drbd_send_uuids(peer_device, 0, 0);
+ drbd_start_resync(peer_device, L_SYNC_TARGET, "resync-from-primary");
+}
+
+static void queue_resync_finished(struct drbd_peer_device *peer_device, enum drbd_disk_state new_peer_disk_state)
+{
struct drbd_connection *connection = peer_device->connection;
+ struct resync_finished_work *rfw;
+
+ rfw = kmalloc_obj(*rfw, GFP_ATOMIC);
+ if (!rfw) {
+ drbd_err(peer_device, "Warn failed to kmalloc(dw).\n");
+ return;
+ }
+
+ rfw->pdw.w.cb = w_resync_finished;
+ rfw->pdw.peer_device = peer_device;
+ rfw->new_peer_disk_state = new_peer_disk_state;
+ drbd_queue_work(&connection->sender_work, &rfw->pdw.w);
+}
+
+static void drbd_queue_final_peers_in_sync(struct drbd_peer_device *peer_device)
+{
+ sector_t last_end = peer_device->last_in_sync_end;
+ sector_t last_step = last_end & ~PEERS_IN_SYNC_STEP_SECT_MASK;
+ sector_t last_step_end = min(get_capacity(peer_device->device->vdisk),
+ last_step + PEERS_IN_SYNC_STEP_SECT);
+
+ /* Send for last request if it was part way through a step */
+ if (last_end > last_step)
+ drbd_queue_update_peers(peer_device, last_step, last_step_end);
+}
+
+void drbd_resync_finished(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state new_peer_disk_state)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+ enum drbd_repl_state old_repl_state = L_ESTABLISHED;
+ bool try_to_get_resynced_from_primary_flag = false;
+ u64 source_m = 0, target_m = 0;
unsigned long db, dt, dbdt;
unsigned long n_oos;
- union drbd_state os, ns;
- struct drbd_device_work *dw;
char *khelper_cmd = NULL;
int verify_done = 0;
+ bool aborted = false;
+ int bm_block_shift = device->last_bm_block_shift;
- /* Remove all elements from the resync LRU. Since future actions
- * might set bits in the (main) bitmap, then the entries in the
- * resync LRU would be wrong. */
- if (drbd_rs_del_all(device)) {
- /* In case this is not possible now, most probably because
- * there are P_RS_DATA_REPLY Packets lingering on the worker's
- * queue (or even the read operations for those packets
- * is not finished by now). Retry in 100ms. */
+ if (repl_state[NOW] == L_SYNC_SOURCE || repl_state[NOW] == L_PAUSED_SYNC_S) {
+ /* Make sure all queued w_update_peers() executed. */
+ if (current == device->resource->worker.task) {
+ queue_resync_finished(peer_device, new_peer_disk_state);
+ return;
+ } else {
+ drbd_flush_workqueue(&device->resource->work);
+ }
+ }
- schedule_timeout_interruptible(HZ / 10);
- dw = kmalloc_obj(struct drbd_device_work, GFP_ATOMIC);
- if (dw) {
- dw->w.cb = w_resync_finished;
- dw->device = device;
- drbd_queue_work(&connection->sender_work, &dw->w);
- return 1;
+ if (!down_write_trylock(&device->uuid_sem)) {
+ if (current == device->resource->worker.task) {
+ queue_resync_finished(peer_device, new_peer_disk_state);
+ return;
+ } else {
+ down_write(&device->uuid_sem);
}
- drbd_err(device, "Warn failed to drbd_rs_del_all() and to kmalloc(dw).\n");
}
- dt = (jiffies - device->rs_start - device->rs_paused) / HZ;
+ dt = (jiffies - peer_device->rs_start - peer_device->rs_paused) / HZ;
if (dt <= 0)
dt = 1;
-
- db = device->rs_total;
+ db = peer_device->rs_total;
/* adjust for verify start and stop sectors, respective reached position */
- if (device->state.conn == C_VERIFY_S || device->state.conn == C_VERIFY_T)
- db -= device->ov_left;
+ if (repl_state[NOW] == L_VERIFY_S || repl_state[NOW] == L_VERIFY_T)
+ db -= atomic64_read(&peer_device->ov_left);
- dbdt = Bit2KB(db/dt);
- device->rs_paused /= HZ;
+ dbdt = bit_to_kb(db/dt, bm_block_shift);
+ peer_device->rs_paused /= HZ;
- if (!get_ldev(device))
+ if (!get_ldev(device)) {
+ up_write(&device->uuid_sem);
goto out;
+ }
- ping_peer(device);
+ drbd_ping_peer(connection);
- spin_lock_irq(&device->resource->req_lock);
- os = drbd_read_state(device);
+ write_lock_irq(&device->resource->state_rwlock);
+ begin_state_change_locked(device->resource, CS_VERBOSE);
+ old_repl_state = repl_state[NOW];
- verify_done = (os.conn == C_VERIFY_S || os.conn == C_VERIFY_T);
+ verify_done = (repl_state[NOW] == L_VERIFY_S || repl_state[NOW] == L_VERIFY_T);
/* This protects us against multiple calls (that can happen in the presence
of application IO), and against connectivity loss just before we arrive here. */
- if (os.conn <= C_CONNECTED)
+ if (peer_device->repl_state[NOW] <= L_ESTABLISHED)
goto out_unlock;
- ns = os;
- ns.conn = C_CONNECTED;
+ /*
+ * This protects us against a race with the peer when finishing a
+ * resync at the same time as entering Ahead-Behind mode.
+ */
+ if (peer_device->repl_state[NOW] == L_BEHIND)
+ goto out_unlock;
+
+ peer_device->resync_active[NEW] = false;
+ __change_repl_state(peer_device, L_ESTABLISHED);
+
+ aborted = device->disk_state[NOW] == D_OUTDATED && new_peer_disk_state == D_INCONSISTENT;
+ {
+ char tmp[sizeof(" but 01234567890123456789 01234567890123456789k blocks skipped")] = "";
- drbd_info(device, "%s done (total %lu sec; paused %lu sec; %lu K/sec)\n",
- verify_done ? "Online verify" : "Resync",
- dt + device->rs_paused, device->rs_paused, dbdt);
+ if (verify_done && peer_device->ov_skipped)
+ snprintf(tmp, sizeof(tmp), " but %lu %lluk blocks skipped",
+ peer_device->ov_skipped, bit_to_kb(1, bm_block_shift));
+ drbd_info(peer_device, "%s %s%s (total %lu sec; paused %lu sec; %lu K/sec)\n",
+ verify_done ? "Online verify" : "Resync",
+ aborted ? "aborted" : "done", tmp,
+ dt + peer_device->rs_paused, peer_device->rs_paused, dbdt);
+ }
- n_oos = drbd_bm_total_weight(device);
+ n_oos = drbd_bm_total_weight(peer_device);
- if (os.conn == C_VERIFY_S || os.conn == C_VERIFY_T) {
+ if (repl_state[NOW] == L_VERIFY_S || repl_state[NOW] == L_VERIFY_T) {
if (n_oos) {
- drbd_alert(device, "Online verify found %lu %dk block out of sync!\n",
- n_oos, Bit2KB(1));
+ drbd_alert(peer_device, "Online verify found %lu %lluk blocks out of sync!\n",
+ n_oos, bit_to_kb(1, bm_block_shift));
khelper_cmd = "out-of-sync";
}
} else {
- D_ASSERT(device, (n_oos - device->rs_failed) == 0);
+ if (!aborted && peer_device->rs_failed == 0 && n_oos != 0)
+ drbd_warn(peer_device, "expected n_oos:%lu to be 0\n", n_oos);
- if (os.conn == C_SYNC_TARGET || os.conn == C_PAUSED_SYNC_T)
+ if (repl_state[NOW] == L_SYNC_TARGET || repl_state[NOW] == L_PAUSED_SYNC_T)
khelper_cmd = "after-resync-target";
- if (device->use_csums && device->rs_total) {
- const unsigned long s = device->rs_same_csum;
- const unsigned long t = device->rs_total;
+ if (peer_device->use_csums && peer_device->rs_total) {
+ const unsigned long s = peer_device->rs_same_csum;
+ const unsigned long t = peer_device->rs_total;
const int ratio =
(t == 0) ? 0 :
(t < 100000) ? ((s*100)/t) : (s/(t/100));
- drbd_info(device, "%u %% had equal checksums, eliminated: %luK; "
- "transferred %luK total %luK\n",
+ drbd_info(peer_device, "%u %% had equal checksums, eliminated: %lluK; "
+ "transferred %lluK total %lluK\n",
ratio,
- Bit2KB(device->rs_same_csum),
- Bit2KB(device->rs_total - device->rs_same_csum),
- Bit2KB(device->rs_total));
+ bit_to_kb(peer_device->rs_same_csum, bm_block_shift),
+ bit_to_kb(peer_device->rs_total - peer_device->rs_same_csum,
+ bm_block_shift),
+ bit_to_kb(peer_device->rs_total, bm_block_shift));
}
}
- if (device->rs_failed) {
- drbd_info(device, " %lu failed blocks\n", device->rs_failed);
+ if (peer_device->rs_failed) {
+ drbd_info(peer_device, " %lu failed blocks\n", peer_device->rs_failed);
- if (os.conn == C_SYNC_TARGET || os.conn == C_PAUSED_SYNC_T) {
- ns.disk = D_INCONSISTENT;
- ns.pdsk = D_UP_TO_DATE;
- } else {
- ns.disk = D_UP_TO_DATE;
- ns.pdsk = D_INCONSISTENT;
- }
+ /* The source may be Outdated; failed blocks must not raise it. */
+ if (repl_state[NOW] == L_SYNC_TARGET || repl_state[NOW] == L_PAUSED_SYNC_T)
+ __change_disk_state(device, D_INCONSISTENT);
+ else
+ __change_peer_disk_state(peer_device, D_INCONSISTENT);
} else {
- ns.disk = D_UP_TO_DATE;
- ns.pdsk = D_UP_TO_DATE;
+ if (repl_state[NOW] == L_SYNC_TARGET || repl_state[NOW] == L_PAUSED_SYNC_T) {
+ bool stable_resync = was_resync_stable(peer_device);
- if (os.conn == C_SYNC_TARGET || os.conn == C_PAUSED_SYNC_T) {
- if (device->p_uuid) {
- int i;
- for (i = UI_BITMAP ; i <= UI_HISTORY_END ; i++)
- _drbd_uuid_set(device, i, device->p_uuid[i]);
- drbd_uuid_set(device, UI_BITMAP, device->ldev->md.uuid[UI_CURRENT]);
- _drbd_uuid_set(device, UI_CURRENT, device->p_uuid[UI_CURRENT]);
- } else {
- drbd_err(device, "device->p_uuid is NULL! BUG\n");
+ if (stable_resync) {
+ enum drbd_disk_state new_disk_state = peer_device->disk_state[NOW];
+
+ if (new_disk_state < D_UP_TO_DATE &&
+ test_bit(SYNC_SRC_CRASHED_PRI, peer_device->flags)) {
+ try_to_get_resynced_from_primary_flag = true;
+ set_bit(CRASHED_PRIMARY, &device->flags);
+ }
+ __change_disk_state(device, new_disk_state);
}
- }
- if (!(os.conn == C_VERIFY_S || os.conn == C_VERIFY_T)) {
- /* for verify runs, we don't update uuids here,
- * so there would be nothing to report. */
- drbd_uuid_set_bm(device, 0UL);
- drbd_print_uuids(device, "updated UUIDs");
- if (device->p_uuid) {
+ if (device->disk_state[NEW] == D_UP_TO_DATE)
+ target_m = __cancel_other_resyncs(device);
+
+ if (stable_resync && test_bit(UUIDS_RECEIVED, peer_device->flags)) {
+ const int node_id = device->resource->res_opts.node_id;
+ int i;
+
+ u64 newer = drbd_uuid_resync_finished(peer_device);
+
+ __outdate_peer_disk_by_mask(device, newer);
+ drbd_print_uuids(peer_device, "updated UUIDs");
+
/* Now the two UUID sets are equal, update what we
* know of the peer. */
- int i;
- for (i = UI_CURRENT ; i <= UI_HISTORY_END ; i++)
- device->p_uuid[i] = device->ldev->md.uuid[i];
+ peer_device->current_uuid = drbd_current_uuid(device);
+ peer_device->bitmap_uuids[node_id] = drbd_bitmap_uuid(peer_device);
+ for (i = 0; i < ARRAY_SIZE(peer_device->history_uuids); i++)
+ peer_device->history_uuids[i] =
+ drbd_history_uuid(device, i);
+ } else {
+ if (!test_bit(UUIDS_RECEIVED, peer_device->flags))
+ drbd_err(peer_device, "BUG: uuids were not received!\n");
+
+ if (test_bit(UNSTABLE_RESYNC, peer_device->flags))
+ drbd_info(peer_device, "Peer was unstable during resync\n");
+ }
+ } else if (repl_state[NOW] == L_SYNC_SOURCE || repl_state[NOW] == L_PAUSED_SYNC_S) {
+ if (new_peer_disk_state != D_MASK)
+ __change_peer_disk_state(peer_device, new_peer_disk_state);
+ if (peer_device->connection->agreed_pro_version < 110) {
+ drbd_uuid_set_bitmap(peer_device, 0UL);
+ drbd_print_uuids(peer_device, "updated UUIDs");
}
}
}
- _drbd_set_state(device, ns, CS_VERBOSE, NULL);
out_unlock:
- spin_unlock_irq(&device->resource->req_lock);
+ end_state_change_locked(device->resource, "resync-finished");
- /* If we have been sync source, and have an effective fencing-policy,
- * once *all* volumes are back in sync, call "unfence". */
- if (os.conn == C_SYNC_SOURCE) {
- enum drbd_disk_state disk_state = D_MASK;
- enum drbd_disk_state pdsk_state = D_MASK;
- enum drbd_fencing_p fp = FP_DONT_CARE;
+ put_ldev(device);
- rcu_read_lock();
- fp = rcu_dereference(device->ldev->disk_conf)->fencing;
- if (fp != FP_DONT_CARE) {
- struct drbd_peer_device *peer_device;
- int vnr;
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- disk_state = min_t(enum drbd_disk_state, disk_state, device->state.disk);
- pdsk_state = min_t(enum drbd_disk_state, pdsk_state, device->state.pdsk);
- }
- }
- rcu_read_unlock();
- if (disk_state == D_UP_TO_DATE && pdsk_state == D_UP_TO_DATE)
- conn_khelper(connection, "unfence-peer");
- }
+ peer_device->rs_total = 0;
+ peer_device->rs_failed = 0;
+ peer_device->rs_paused = 0;
- put_ldev(device);
-out:
- device->rs_total = 0;
- device->rs_failed = 0;
- device->rs_paused = 0;
+ if (old_repl_state == L_SYNC_TARGET || old_repl_state == L_PAUSED_SYNC_T)
+ target_m |= NODE_MASK(peer_device->node_id);
+ else if (old_repl_state == L_SYNC_SOURCE || old_repl_state == L_PAUSED_SYNC_S)
+ source_m |= NODE_MASK(peer_device->node_id);
+ resync_again(device, source_m, target_m);
+ write_unlock_irq(&device->resource->state_rwlock);
+ up_write(&device->uuid_sem);
+ if (connection->after_reconciliation.lost_node_id != -1)
+ after_reconciliation_resync(connection);
+
+ drbd_queue_final_peers_in_sync(peer_device);
+
+out:
/* reset start sector, if we reached end of device */
- if (verify_done && device->ov_left == 0)
- device->ov_start_sector = 0;
+ if (verify_done && atomic64_read(&peer_device->ov_left) == 0)
+ peer_device->ov_start_sector = 0;
- drbd_md_sync(device);
+ drbd_md_sync_if_dirty(device);
if (khelper_cmd)
- drbd_khelper(device, khelper_cmd);
+ drbd_maybe_khelper(device, connection, khelper_cmd);
- return 1;
+ if (try_to_get_resynced_from_primary_flag)
+ try_to_get_resynced_from_primary(device);
}
+
/**
* w_e_end_data_req() - Worker callback, to send a P_DATA_REPLY packet in response to a P_DATA_REQUEST
* @w: work object.
@@ -1038,7 +2241,6 @@ int w_e_end_data_req(struct drbd_work *w, int cancel)
{
struct drbd_peer_request *peer_req = container_of(w, struct drbd_peer_request, w);
struct drbd_peer_device *peer_device = peer_req->peer_device;
- struct drbd_device *device = peer_device->device;
int err;
if (unlikely(cancel)) {
@@ -1050,256 +2252,441 @@ int w_e_end_data_req(struct drbd_work *w, int cancel)
err = drbd_send_block(peer_device, P_DATA_REPLY, peer_req);
} else {
drbd_err_ratelimit(peer_device, "Sending NegDReply. sector=%llus.\n",
- (unsigned long long)peer_req->i.sector);
+ (unsigned long long)peer_req->i.sector);
+
+ err = drbd_send_ack(peer_device, P_NEG_DREPLY, peer_req);
+ }
+ if (unlikely(err))
+ drbd_err(peer_device, "drbd_send_block() failed\n");
+
+out:
+ dec_unacked(peer_device);
+ drbd_free_peer_req(peer_req);
+
+ return err;
+}
+
+void
+drbd_resync_read_req_mod(struct drbd_peer_request *peer_req, enum drbd_interval_flags bit_to_set)
+{
+ const unsigned long done_mask = 1UL << INTERVAL_SENT | 1UL << INTERVAL_RECEIVED;
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ unsigned long nflags, oflags, new_flag;
+
+ new_flag = 1UL << bit_to_set;
+ if (!(new_flag & done_mask))
+ drbd_err(peer_device, "BUG: %s: Unexpected flag 0x%lx\n", __func__, new_flag);
+
+ do {
+ oflags = READ_ONCE(peer_req->i.flags);
+ nflags = oflags | new_flag;
+ } while (cmpxchg(&peer_req->i.flags, oflags, nflags) != oflags);
+
+ if (new_flag & oflags)
+ drbd_err(peer_device, "BUG: %s: Flag 0x%lx already set\n", __func__, new_flag);
+
+ if ((nflags & done_mask) == done_mask)
+ drbd_free_peer_req(peer_req);
+}
+
+static bool all_zero(struct drbd_peer_request *peer_req)
+{
+ struct bvec_iter iter;
+ struct bio_vec bvec;
+ struct bio *bio;
- err = drbd_send_ack(peer_device, P_NEG_DREPLY, peer_req);
+ bio_list_for_each(bio, &peer_req->bios) {
+ bio_for_each_segment(bvec, bio, iter) {
+ unsigned long *d = bvec_virt(&bvec);
+ unsigned int i, words = bvec.bv_len / sizeof(*d);
+
+ for (i = 0; i < words; i++) {
+ if (d[i])
+ return false;
+ }
+ }
}
- if (unlikely(err))
- drbd_err(device, "drbd_send_block() failed\n");
-out:
- dec_unacked(device);
- drbd_free_peer_req(device, peer_req);
+ return true;
+}
- return err;
+static bool al_resync_extent_active(struct drbd_device *device, sector_t sector, unsigned int size)
+{
+ sector_t resync_extent_sector = sector & ~LEGACY_BM_EXT_SECT_MASK;
+ sector_t end_sector = sector + (size >> SECTOR_SHIFT);
+ sector_t resync_extent_end_sector =
+ (end_sector + LEGACY_BM_EXT_SECT_MASK) & ~LEGACY_BM_EXT_SECT_MASK;
+ return drbd_al_active(device,
+ resync_extent_sector,
+ (resync_extent_end_sector - resync_extent_sector) << SECTOR_SHIFT);
}
-static bool all_zero(struct drbd_peer_request *peer_req)
+static int drbd_rs_reply(struct drbd_peer_device *peer_device, struct drbd_peer_request *peer_req, bool *expect_ack)
{
- struct page *page = peer_req->pages;
- unsigned int len = peer_req->i.size;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ int err;
+ bool eq = false;
- page_chain_for_each(page) {
- unsigned int l = min_t(unsigned int, len, PAGE_SIZE);
- unsigned int i, words = l / sizeof(long);
- unsigned long *d;
+ if (peer_req->flags & EE_HAS_DIGEST) {
+ struct digest_info *di = peer_req->digest;
+ int digest_size;
+ void *digest = NULL;
- d = kmap_atomic(page);
- for (i = 0; i < words; i++) {
- if (d[i]) {
- kunmap_atomic(d);
- return false;
+ /* quick hack to try to avoid a race against reconfiguration.
+ * a real fix would be much more involved,
+ * introducing more locking mechanisms */
+ if (connection->csums_tfm) {
+ digest_size = crypto_shash_digestsize(connection->csums_tfm);
+ D_ASSERT(device, digest_size == di->digest_size);
+ digest = kmalloc(digest_size, GFP_NOIO);
+ if (digest) {
+ drbd_csum_bios(connection->csums_tfm, &peer_req->bios, digest);
+ eq = !memcmp(digest, di->digest, digest_size);
+ kfree(digest);
}
}
- kunmap_atomic(d);
- len -= l;
+
+ peer_req->flags &= ~EE_HAS_DIGEST; /* This peer request no longer has a digest pointer */
+ kfree(di);
}
- return true;
+ if (eq) {
+ drbd_set_in_sync(peer_device, peer_req->i.sector, peer_req->i.size);
+ /* rs_same_csums unit is BM_BLOCK_SIZE */
+ /* ldev_safe: a bio that holds a ldev ref exists */
+ peer_device->rs_same_csum += peer_req->i.size >> device->ldev->md.bm_block_shift;
+ err = drbd_send_ack(peer_device, P_RS_IS_IN_SYNC, peer_req);
+ } else {
+ inc_rs_pending(peer_device);
+ /*
+ * If we send back as P_RS_DEALLOCATED,
+ * this is overestimating "in-flight" accounting.
+ * But needed to be properly balanced with
+ * the atomic_sub() in got_RSWriteAck.
+ */
+ atomic_add(peer_req->i.size >> 9, &connection->rs_in_flight);
+
+ /* After setting this, peer_req can be found by got_RSWriteAck. */
+ set_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags);
+
+ if (peer_req->flags & EE_RS_THIN_REQ && all_zero(peer_req)) {
+ err = drbd_send_rs_deallocated(peer_device, peer_req);
+ } else {
+ err = drbd_send_block(peer_device, P_RS_DATA_REPLY, peer_req);
+ }
+ *expect_ack = true;
+
+ drbd_peer_req_strip_bio(peer_req);
+
+ drbd_resync_read_req_mod(peer_req, INTERVAL_SENT);
+ peer_req = NULL;
+ }
+
+ return err;
}
/**
- * w_e_end_rsdata_req() - Worker callback to send a P_RS_DATA_REPLY packet in response to a P_RS_DATA_REQUEST
+ * w_e_end_rsdata_req() - Reply to a resync request.
* @w: work object.
- * @cancel: The connection will be closed anyways
+ * @cancel: The connection is being closed
+ *
+ * Worker callback to send P_RS_DATA_REPLY or a related packet after completing
+ * a resync read.
+ *
+ * Return: Error code or 0 on success.
*/
int w_e_end_rsdata_req(struct drbd_work *w, int cancel)
{
struct drbd_peer_request *peer_req = container_of(w, struct drbd_peer_request, w);
struct drbd_peer_device *peer_device = peer_req->peer_device;
- struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
int err;
+ bool expect_ack = false;
- if (unlikely(cancel)) {
- err = 0;
- goto out;
- }
-
- if (get_ldev_if_state(device, D_FAILED)) {
- drbd_rs_complete_io(device, peer_req->i.sector);
- put_ldev(device);
+ if (unlikely(cancel) || connection->cstate[NOW] < C_CONNECTED) {
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ dec_unacked(peer_device);
+ return 0;
}
- if (device->state.conn == C_AHEAD) {
+ if (peer_device->repl_state[NOW] == L_AHEAD) {
err = drbd_send_ack(peer_device, P_RS_CANCEL, peer_req);
} else if (likely((peer_req->flags & EE_WAS_ERROR) == 0)) {
- if (likely(device->state.pdsk >= D_INCONSISTENT)) {
- inc_rs_pending(peer_device);
- if (peer_req->flags & EE_RS_THIN_REQ && all_zero(peer_req))
- err = drbd_send_rs_deallocated(peer_device, peer_req);
- else
- err = drbd_send_block(peer_device, P_RS_DATA_REPLY, peer_req);
+ if (unlikely(peer_device->disk_state[NOW] < D_INCONSISTENT)) {
+ if (connection->agreed_features & DRBD_FF_RESYNC_DAGTAG) {
+ drbd_err_ratelimit(peer_device,
+ "Sending P_RS_CANCEL, partner DISKLESS!\n");
+ err = drbd_send_ack(peer_device, P_RS_CANCEL, peer_req);
+ } else {
+ /*
+ * A peer that does not support DRBD_FF_RESYNC_DAGTAG does not
+ * expect to receive P_RS_CANCEL after losing its disk.
+ */
+ drbd_err_ratelimit(peer_device,
+ "Not sending resync reply, partner DISKLESS!\n");
+ err = 0;
+ }
+ } else if (connection->agreed_pro_version >= 110 &&
+ !(connection->agreed_features & DRBD_FF_RESYNC_DAGTAG) &&
+ al_resync_extent_active(peer_device->device,
+ peer_req->i.sector, peer_req->i.size)) {
+ /* DRBD versions without DRBD_FF_RESYNC_DAGTAG lock
+ * 128MiB "resync extents" in the activity log whenever
+ * they make resync requests. Some of these versions
+ * also lock activity lock extents when receiving
+ * P_DATA. In particular, DRBD 9.0 and 9.1. This can
+ * cause a deadlock if we send resync replies in these
+ * extents as follows:
+ * * Node is SyncTarget towards us
+ * * Node locks a resync extent and sends P_RS_DATA_REQUEST
+ * * Node receives P_DATA write in this extent; write
+ * waits for resync extent to be unlocked
+ * * Node receives P_BARRIER (protocol A); receiver
+ * thread blocks waiting for write to complete
+ * * We reply to P_RS_DATA_REQUEST, but it is never
+ * processed because receiver thread is blocked
+ *
+ * Break the deadlock by canceling instead. This is
+ * sent on the control socket so it will be processed. */
+ dynamic_drbd_dbg(peer_device,
+ "Cancel resync request at %llus+%u due to activity",
+ (unsigned long long) peer_req->i.sector, peer_req->i.size);
+
+ err = drbd_send_ack(peer_device, P_RS_CANCEL, peer_req);
} else {
- drbd_err_ratelimit(peer_device,
- "Not sending RSDataReply, partner DISKLESS!\n");
- err = 0;
+ err = drbd_rs_reply(peer_device, peer_req, &expect_ack);
+
+ /* If expect_ack is true, peer_req may already have been freed. */
+ if (expect_ack)
+ peer_req = NULL;
}
} else {
drbd_err_ratelimit(peer_device, "Sending NegRSDReply. sector %llus.\n",
- (unsigned long long)peer_req->i.sector);
+ (unsigned long long)peer_req->i.sector);
err = drbd_send_ack(peer_device, P_NEG_RS_DREPLY, peer_req);
/* update resync data with failure */
drbd_rs_failed_io(peer_device, peer_req->i.sector, peer_req->i.size);
}
- if (unlikely(err))
- drbd_err(device, "drbd_send_block() failed\n");
-out:
- dec_unacked(device);
- drbd_free_peer_req(device, peer_req);
+ dec_unacked(peer_device);
+
+ if (!expect_ack) {
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ }
+
+ if (unlikely(err))
+ drbd_err(peer_device, "Sending resync reply failed\n");
return err;
}
-int w_e_end_csum_rs_req(struct drbd_work *w, int cancel)
+int w_e_end_ov_req(struct drbd_work *w, int cancel)
{
struct drbd_peer_request *peer_req = container_of(w, struct drbd_peer_request, w);
struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
- struct digest_info *di;
+ struct drbd_connection *connection = peer_device->connection;
int digest_size;
- void *digest = NULL;
- int err, eq = 0;
+ void *digest;
+ sector_t sector = peer_req->i.sector;
+ unsigned int size = peer_req->i.size;
+ struct dagtag_find_result dagtag_result;
+ int err = 0;
+ enum drbd_packet cmd = connection->agreed_features & DRBD_FF_RESYNC_DAGTAG ?
+ P_OV_DAGTAG_REPLY : P_OV_REPLY;
- if (unlikely(cancel)) {
- err = 0;
+ if (unlikely(cancel) || connection->cstate[NOW] < C_CONNECTED)
goto out;
- }
-
- if (get_ldev(device)) {
- drbd_rs_complete_io(device, peer_req->i.sector);
- put_ldev(device);
- }
- di = peer_req->digest;
+ if (!(connection->agreed_features & DRBD_FF_RESYNC_DAGTAG) &&
+ al_resync_extent_active(peer_device->device, peer_req->i.sector, peer_req->i.size)) {
+ /* A peer that does not support DRBD_FF_RESYNC_DAGTAG expects
+ * online verify to be exclusive with 128MiB "resync extents"
+ * in the activity log. If such a verify source sends a request
+ * but we receive an overlapping write before the request then
+ * we will read newer data for the verify transaction than the
+ * source did. So we may detect spurious out-of-sync blocks.
+ *
+ * In addition, we may trigger a deadlock in such a peer by
+ * sending a reply if it is waiting for writes to drain due to
+ * a P_BARRIER packet. See w_e_end_rsdata_req for details.
+ *
+ * Prevent these issues by canceling instead.
+ */
+ dynamic_drbd_dbg(peer_device,
+ "Cancel online verify request at %llus+%u due to activity",
+ (unsigned long long) peer_req->i.sector, peer_req->i.size);
- if (likely((peer_req->flags & EE_WAS_ERROR) == 0)) {
- /* quick hack to try to avoid a race against reconfiguration.
- * a real fix would be much more involved,
- * introducing more locking mechanisms */
- if (peer_device->connection->csums_tfm) {
- digest_size = crypto_shash_digestsize(peer_device->connection->csums_tfm);
- D_ASSERT(device, digest_size == di->digest_size);
- digest = kmalloc(digest_size, GFP_NOIO);
- }
- if (digest) {
- drbd_csum_ee(peer_device->connection->csums_tfm, peer_req, digest);
- eq = !memcmp(digest, di->digest, digest_size);
- kfree(digest);
- }
+ spin_lock_irq(&device->interval_lock);
+ set_bit(INTERVAL_CONFLICT, &peer_req->i.flags);
+ spin_unlock_irq(&device->interval_lock);
+ }
- if (eq) {
- drbd_set_in_sync(peer_device, peer_req->i.sector, peer_req->i.size);
- /* rs_same_csums unit is BM_BLOCK_SIZE */
- device->rs_same_csum += peer_req->i.size >> BM_BLOCK_SHIFT;
- err = drbd_send_ack(peer_device, P_RS_IS_IN_SYNC, peer_req);
+ if (test_bit(INTERVAL_CONFLICT, &peer_req->i.flags)) {
+ if (connection->agreed_pro_version < 110) {
+ if (drbd_ratelimit())
+ drbd_warn(peer_device, "Verify request conflicts but cannot cancel, "
+ "peer may report spurious out-of-sync\n");
} else {
- inc_rs_pending(peer_device);
- peer_req->block_id = ID_SYNCER; /* By setting block_id, digest pointer becomes invalid! */
- peer_req->flags &= ~EE_HAS_DIGEST; /* This peer request no longer has a digest pointer */
- kfree(di);
- err = drbd_send_block(peer_device, P_RS_DATA_REPLY, peer_req);
+ drbd_verify_skipped_block(peer_device, sector, size);
+ verify_progress(peer_device, sector, size);
+ drbd_send_ack(peer_device, P_RS_CANCEL, peer_req);
+ goto out;
}
- } else {
- err = drbd_send_ack(peer_device, P_NEG_RS_DREPLY, peer_req);
- drbd_err_ratelimit(device, "Sending NegDReply. I guess it gets messy.\n");
}
- if (unlikely(err))
- drbd_err(device, "drbd_send_block/ack() failed\n");
-out:
- dec_unacked(device);
- drbd_free_peer_req(device, peer_req);
-
- return err;
-}
-
-int w_e_end_ov_req(struct drbd_work *w, int cancel)
-{
- struct drbd_peer_request *peer_req = container_of(w, struct drbd_peer_request, w);
- struct drbd_peer_device *peer_device = peer_req->peer_device;
- struct drbd_device *device = peer_device->device;
- sector_t sector = peer_req->i.sector;
- unsigned int size = peer_req->i.size;
- int digest_size;
- void *digest;
- int err = 0;
- if (unlikely(cancel))
+ dagtag_result = find_current_dagtag(peer_device->device->resource);
+ if (dagtag_result.err)
goto out;
+ set_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags);
+
digest_size = crypto_shash_digestsize(peer_device->connection->verify_tfm);
- digest = kmalloc(digest_size, GFP_NOIO);
+ /* FIXME if this allocation fails, online verify will not terminate! */
+ digest = drbd_prepare_drequest_csum(peer_req, cmd, digest_size,
+ dagtag_result.node_id, dagtag_result.dagtag);
if (!digest) {
- err = 1; /* terminate the connection in case the allocation failed */
+ err = -ENOMEM;
goto out;
}
- if (likely(!(peer_req->flags & EE_WAS_ERROR)))
- drbd_csum_ee(peer_device->connection->verify_tfm, peer_req, digest);
+ if (!(peer_req->flags & EE_WAS_ERROR))
+ drbd_csum_bios(peer_device->connection->verify_tfm, &peer_req->bios, digest);
else
memset(digest, 0, digest_size);
- /* Free e and pages before send.
+ /* Free pages before send.
* In case we block on congestion, we could otherwise run into
* some distributed deadlock, if the other side blocks on
* congestion as well, because our receiver blocks in
* drbd_alloc_pages due to pp_in_use > max_buffers. */
- drbd_free_peer_req(device, peer_req);
- peer_req = NULL;
+ drbd_peer_req_strip_bio(peer_req);
+
inc_rs_pending(peer_device);
- err = drbd_send_drequest_csum(peer_device, sector, size, digest, digest_size, P_OV_REPLY);
+
+ err = drbd_send_command(peer_device, cmd, DATA_STREAM);
if (err)
- dec_rs_pending(peer_device);
- kfree(digest);
+ goto out_rs_pending;
+
+ dec_unacked(peer_device);
+ return 0;
+out_rs_pending:
+ dec_rs_pending(peer_device);
out:
- if (peer_req)
- drbd_free_peer_req(device, peer_req);
- dec_unacked(device);
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ dec_unacked(peer_device);
return err;
}
void drbd_ov_out_of_sync_found(struct drbd_peer_device *peer_device, sector_t sector, int size)
{
- struct drbd_device *device = peer_device->device;
- if (device->ov_last_oos_start + device->ov_last_oos_size == sector) {
- device->ov_last_oos_size += size>>9;
+ if (peer_device->ov_last_oos_start + peer_device->ov_last_oos_size == sector) {
+ peer_device->ov_last_oos_size += size>>9;
} else {
- device->ov_last_oos_start = sector;
- device->ov_last_oos_size = size>>9;
+ ov_out_of_sync_print(peer_device);
+ peer_device->ov_last_oos_start = sector;
+ peer_device->ov_last_oos_size = size>>9;
}
drbd_set_out_of_sync(peer_device, sector, size);
}
-int w_e_end_ov_reply(struct drbd_work *w, int cancel)
+void verify_progress(struct drbd_peer_device *peer_device,
+ const sector_t sector, const unsigned int size)
+{
+ bool stop_sector_reached =
+ (peer_device->repl_state[NOW] == L_VERIFY_S) &&
+ verify_can_do_stop_sector(peer_device) &&
+ (sector + (size>>9)) >= peer_device->ov_stop_sector;
+
+ unsigned long ov_left = atomic64_dec_return(&peer_device->ov_left);
+
+ /* let's advance progress step marks only for every other megabyte */
+ if ((ov_left & 0x1ff) == 0)
+ drbd_advance_rs_marks(peer_device, ov_left);
+
+ if (ov_left == 0 || stop_sector_reached)
+ drbd_peer_device_post_work(peer_device, RS_DONE);
+}
+
+static bool digest_equal(struct drbd_peer_request *peer_req)
{
- struct drbd_peer_request *peer_req = container_of(w, struct drbd_peer_request, w);
struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
struct digest_info *di;
void *digest;
+ int digest_size;
+ bool eq = false;
+
+ di = peer_req->digest;
+
+ digest_size = crypto_shash_digestsize(peer_device->connection->verify_tfm);
+ digest = kmalloc(digest_size, GFP_NOIO);
+ if (digest) {
+ drbd_csum_bios(peer_device->connection->verify_tfm, &peer_req->bios, digest);
+
+ D_ASSERT(device, digest_size == di->digest_size);
+ eq = !memcmp(digest, di->digest, digest_size);
+ kfree(digest);
+ }
+
+ return eq;
+}
+
+int w_e_end_ov_reply(struct drbd_work *w, int cancel)
+{
+ struct drbd_peer_request *peer_req = container_of(w, struct drbd_peer_request, w);
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
sector_t sector = peer_req->i.sector;
unsigned int size = peer_req->i.size;
- int digest_size;
- int err, eq = 0;
- bool stop_sector_reached = false;
+ u64 block_id = peer_req->block_id;
+ enum ov_result result;
+ bool al_conflict = false;
+ int err;
if (unlikely(cancel)) {
- drbd_free_peer_req(device, peer_req);
- dec_unacked(device);
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ dec_unacked(peer_device);
return 0;
}
- /* after "cancel", because after drbd_disconnect/drbd_rs_cancel_all
- * the resync lru has been cleaned up already */
- if (get_ldev(device)) {
- drbd_rs_complete_io(device, peer_req->i.sector);
- put_ldev(device);
- }
-
- di = peer_req->digest;
+ if (!(connection->agreed_features & DRBD_FF_RESYNC_DAGTAG) &&
+ al_resync_extent_active(peer_device->device, peer_req->i.sector, peer_req->i.size)) {
+ /* A peer that does not support DRBD_FF_RESYNC_DAGTAG expects
+ * online verify to be exclusive with 128MiB "resync extents"
+ * in the activity log. We may have received an overlapping
+ * write before issuing this read, which the peer did not have
+ * at the time of its read. So we may detect spurious
+ * out-of-sync blocks.
+ *
+ * Prevent this by skipping instead.
+ */
+ dynamic_drbd_dbg(peer_device,
+ "Skip online verify block at %llus+%u due to activity",
+ (unsigned long long) peer_req->i.sector, peer_req->i.size);
- if (likely((peer_req->flags & EE_WAS_ERROR) == 0)) {
- digest_size = crypto_shash_digestsize(peer_device->connection->verify_tfm);
- digest = kmalloc(digest_size, GFP_NOIO);
- if (digest) {
- drbd_csum_ee(peer_device->connection->verify_tfm, peer_req, digest);
+ al_conflict = true;
+ }
- D_ASSERT(device, digest_size == di->digest_size);
- eq = !memcmp(digest, di->digest, digest_size);
- kfree(digest);
- }
+ if (test_bit(INTERVAL_CONFLICT, &peer_req->i.flags) || al_conflict) {
+ /* DRBD versions without DRBD_FF_RESYNC_DAGTAG do not know about
+ * OV_RESULT_SKIP, but they treat it the same as OV_RESULT_IN_SYNC which is
+ * the best we can do here anyway. */
+ result = OV_RESULT_SKIP;
+ drbd_verify_skipped_block(peer_device, sector, size);
+ } else if (likely((peer_req->flags & EE_WAS_ERROR) == 0) && digest_equal(peer_req)) {
+ result = OV_RESULT_IN_SYNC;
+ ov_out_of_sync_print(peer_device);
+ } else {
+ result = OV_RESULT_OUT_OF_SYNC;
+ drbd_ov_out_of_sync_found(peer_device, sector, size);
}
/* Free peer_req and pages before send.
@@ -1307,30 +2694,15 @@ int w_e_end_ov_reply(struct drbd_work *w, int cancel)
* some distributed deadlock, if the other side blocks on
* congestion as well, because our receiver blocks in
* drbd_alloc_pages due to pp_in_use > max_buffers. */
- drbd_free_peer_req(device, peer_req);
- if (!eq)
- drbd_ov_out_of_sync_found(peer_device, sector, size);
- else
- ov_out_of_sync_print(peer_device);
-
- err = drbd_send_ack_ex(peer_device, P_OV_RESULT, sector, size,
- eq ? ID_IN_SYNC : ID_OUT_OF_SYNC);
-
- dec_unacked(device);
-
- --device->ov_left;
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ peer_req = NULL;
- /* let's advance progress step marks only for every other megabyte */
- if ((device->ov_left & 0x200) == 0x200)
- drbd_advance_rs_marks(peer_device, device->ov_left);
+ err = drbd_send_ov_result(peer_device, sector, size, block_id, result);
- stop_sector_reached = verify_can_do_stop_sector(device) &&
- (sector + (size>>9)) >= device->ov_stop_sector;
+ dec_unacked(peer_device);
- if (device->ov_left == 0 || stop_sector_reached) {
- ov_out_of_sync_print(peer_device);
- drbd_resync_finished(peer_device);
- }
+ verify_progress(peer_device, sector, size);
return err;
}
@@ -1338,199 +2710,95 @@ int w_e_end_ov_reply(struct drbd_work *w, int cancel)
/* FIXME
* We need to track the number of pending barrier acks,
* and to be able to wait for them.
- * See also comment in drbd_adm_attach before drbd_suspend_io.
+ * See also comment in drbd_nl_attach_doit before drbd_suspend_io.
*/
static int drbd_send_barrier(struct drbd_connection *connection)
{
struct p_barrier *p;
- struct drbd_socket *sock;
- sock = &connection->data;
- p = conn_prepare_command(connection, sock);
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
if (!p)
return -EIO;
+
p->barrier = connection->send.current_epoch_nr;
p->pad = 0;
+ WRITE_ONCE(connection->send.last_sent_epoch_nr, connection->send.current_epoch_nr);
connection->send.current_epoch_writes = 0;
connection->send.last_sent_barrier_jif = jiffies;
- return conn_send_command(connection, sock, P_BARRIER, sizeof(*p), NULL, 0);
-}
-
-static int pd_send_unplug_remote(struct drbd_peer_device *pd)
-{
- struct drbd_socket *sock = &pd->connection->data;
- if (!drbd_prepare_command(pd, sock))
- return -EIO;
- return drbd_send_command(pd, sock, P_UNPLUG_REMOTE, 0, NULL, 0);
+ return send_command(connection, -1, P_BARRIER, DATA_STREAM);
}
-int w_send_write_hint(struct drbd_work *w, int cancel)
+static bool need_unplug(struct drbd_connection *connection)
{
- struct drbd_device *device =
- container_of(w, struct drbd_device, unplug_work);
+ unsigned int i = connection->todo.unplug_slot;
- if (cancel)
- return 0;
- return pd_send_unplug_remote(first_peer_device(device));
-}
+ /* Nobody to hint to below C_CONNECTED. */
+ if (connection->cstate[NOW] < C_CONNECTED)
+ return false;
-static void re_init_if_first_write(struct drbd_connection *connection, unsigned int epoch)
-{
- if (!connection->send.seen_any_write_yet) {
- connection->send.seen_any_write_yet = true;
- connection->send.current_epoch_nr = epoch;
- connection->send.current_epoch_writes = 0;
- connection->send.last_sent_barrier_jif = jiffies;
- }
+ return dagtag_newer_eq(connection->send.current_dagtag_sector,
+ connection->todo.unplug_dagtag_sector[i]);
}
-static void maybe_send_barrier(struct drbd_connection *connection, unsigned int epoch)
+static void maybe_send_unplug_remote(struct drbd_connection *connection, bool send_anyways)
{
- /* re-init if first write on this connection */
- if (!connection->send.seen_any_write_yet)
+ if (need_unplug(connection)) {
+ /* Yes, this is non-atomic wrt. its use in drbd_unplug_fn.
+ * We save a spin_lock_irq, and worst case
+ * we occasionally miss an unplug event. */
+
+ /* Paranoia: to avoid a continuous stream of unplug-hints,
+ * in case we never get any unplug events */
+ connection->todo.unplug_dagtag_sector[connection->todo.unplug_slot] =
+ connection->send.current_dagtag_sector + (1ULL << 63);
+ /* advance the current unplug slot */
+ connection->todo.unplug_slot ^= 1;
+ } else if (!send_anyways)
return;
- if (connection->send.current_epoch_nr != epoch) {
- if (connection->send.current_epoch_writes)
- drbd_send_barrier(connection);
- connection->send.current_epoch_nr = epoch;
- }
-}
-
-int w_send_out_of_sync(struct drbd_work *w, int cancel)
-{
- struct drbd_request *req = container_of(w, struct drbd_request, w);
- struct drbd_device *device = req->device;
- struct drbd_peer_device *const peer_device = first_peer_device(device);
- struct drbd_connection *const connection = peer_device->connection;
- int err;
-
- if (unlikely(cancel)) {
- req_mod(req, SEND_CANCELED, peer_device);
- return 0;
- }
- req->pre_send_jif = jiffies;
-
- /* this time, no connection->send.current_epoch_writes++;
- * If it was sent, it was the closing barrier for the last
- * replicated epoch, before we went into AHEAD mode.
- * No more barriers will be sent, until we leave AHEAD mode again. */
- maybe_send_barrier(connection, req->epoch);
-
- err = drbd_send_out_of_sync(peer_device, req);
- req_mod(req, OOS_HANDED_TO_NETWORK, peer_device);
-
- return err;
-}
-
-/**
- * w_send_dblock() - Worker callback to send a P_DATA packet in order to mirror a write request
- * @w: work object.
- * @cancel: The connection will be closed anyways
- */
-int w_send_dblock(struct drbd_work *w, int cancel)
-{
- struct drbd_request *req = container_of(w, struct drbd_request, w);
- struct drbd_device *device = req->device;
- struct drbd_peer_device *const peer_device = first_peer_device(device);
- struct drbd_connection *connection = peer_device->connection;
- bool do_send_unplug = req->rq_state & RQ_UNPLUG;
- int err;
-
- if (unlikely(cancel)) {
- req_mod(req, SEND_CANCELED, peer_device);
- return 0;
- }
- req->pre_send_jif = jiffies;
-
- re_init_if_first_write(connection, req->epoch);
- maybe_send_barrier(connection, req->epoch);
- connection->send.current_epoch_writes++;
-
- err = drbd_send_dblock(peer_device, req);
- req_mod(req, err ? SEND_FAILED : HANDED_OVER_TO_NETWORK, peer_device);
-
- if (do_send_unplug && !err)
- pd_send_unplug_remote(peer_device);
-
- return err;
-}
-
-/**
- * w_send_read_req() - Worker callback to send a read request (P_DATA_REQUEST) packet
- * @w: work object.
- * @cancel: The connection will be closed anyways
- */
-int w_send_read_req(struct drbd_work *w, int cancel)
-{
- struct drbd_request *req = container_of(w, struct drbd_request, w);
- struct drbd_device *device = req->device;
- struct drbd_peer_device *const peer_device = first_peer_device(device);
- struct drbd_connection *connection = peer_device->connection;
- bool do_send_unplug = req->rq_state & RQ_UNPLUG;
- int err;
-
- if (unlikely(cancel)) {
- req_mod(req, SEND_CANCELED, peer_device);
- return 0;
- }
- req->pre_send_jif = jiffies;
-
- /* Even read requests may close a write epoch,
- * if there was any yet. */
- maybe_send_barrier(connection, req->epoch);
-
- err = drbd_send_drequest(peer_device, P_DATA_REQUEST, req->i.sector, req->i.size,
- (unsigned long)req);
-
- req_mod(req, err ? SEND_FAILED : HANDED_OVER_TO_NETWORK, peer_device);
-
- if (do_send_unplug && !err)
- pd_send_unplug_remote(peer_device);
-
- return err;
-}
-
-int w_restart_disk_io(struct drbd_work *w, int cancel)
-{
- struct drbd_request *req = container_of(w, struct drbd_request, w);
- struct drbd_device *device = req->device;
- if (bio_data_dir(req->master_bio) == WRITE && req->rq_state & RQ_IN_ACT_LOG)
- drbd_al_begin_io(device, &req->i);
+ if (connection->cstate[NOW] < C_CONNECTED)
+ return;
- req->private_bio = bio_alloc_clone(device->ldev->backing_bdev,
- req->master_bio, GFP_NOIO,
- &drbd_io_bio_set);
- req->private_bio->bi_private = req;
- req->private_bio->bi_end_io = drbd_request_endio;
- submit_bio_noacct(req->private_bio);
+ if (!conn_prepare_command(connection, 0, DATA_STREAM))
+ return;
- return 0;
+ send_command(connection, -1, P_UNPLUG_REMOTE, DATA_STREAM);
}
-static int _drbd_may_sync_now(struct drbd_device *device)
+static bool __drbd_may_sync_now(struct drbd_peer_device *peer_device)
{
- struct drbd_device *odev = device;
- int resync_after;
+ struct drbd_device *other_device = peer_device->device;
+ int ret = true;
+ rcu_read_lock();
while (1) {
- if (!odev->ldev || odev->state.disk == D_DISKLESS)
- return 1;
- rcu_read_lock();
- resync_after = rcu_dereference(odev->ldev->disk_conf)->resync_after;
- rcu_read_unlock();
+ struct drbd_peer_device *other_peer_device;
+ int resync_after;
+
+ if (!other_device->ldev || other_device->disk_state[NOW] == D_DISKLESS)
+ break;
+ resync_after = rcu_dereference(other_device->ldev->disk_conf)->resync_after;
if (resync_after == -1)
- return 1;
- odev = minor_to_device(resync_after);
- if (!odev)
- return 1;
- if ((odev->state.conn >= C_SYNC_SOURCE &&
- odev->state.conn <= C_PAUSED_SYNC_T) ||
- odev->state.aftr_isp || odev->state.peer_isp ||
- odev->state.user_isp)
- return 0;
+ break;
+ other_device = minor_to_device(resync_after);
+ if (!other_device)
+ break;
+ for_each_peer_device_rcu(other_peer_device, other_device) {
+ if ((other_peer_device->repl_state[NOW] >= L_SYNC_SOURCE &&
+ other_peer_device->repl_state[NOW] <= L_PAUSED_SYNC_T) ||
+ other_peer_device->resync_susp_dependency[NOW] ||
+ other_peer_device->resync_susp_peer[NOW] ||
+ other_peer_device->resync_susp_user[NOW]) {
+ ret = false;
+ goto break_unlock;
+ }
+ }
}
+break_unlock:
+ rcu_read_unlock();
+
+ return ret;
}
/**
@@ -1541,21 +2809,32 @@ static int _drbd_may_sync_now(struct drbd_device *device)
*/
static bool drbd_pause_after(struct drbd_device *device)
{
+ struct drbd_device *other_device;
bool changed = false;
- struct drbd_device *odev;
- int i;
+ int vnr;
+ /* FIXME seriously inefficient with many devices,
+ * while also ignoring the input "device" argument :-( */
rcu_read_lock();
- idr_for_each_entry(&drbd_devices, odev, i) {
- if (odev->state.conn == C_STANDALONE && odev->state.disk == D_DISKLESS)
+ idr_for_each_entry(&drbd_devices, other_device, vnr) {
+ struct drbd_peer_device *other_peer_device;
+
+ begin_state_change_locked(other_device->resource, CS_HARD);
+ if (other_device->disk_state[NOW] == D_DISKLESS) {
+ abort_state_change_locked(other_device->resource);
continue;
- if (!_drbd_may_sync_now(odev) &&
- _drbd_set_state(_NS(odev, aftr_isp, 1),
- CS_HARD, NULL) != SS_NOTHING_TO_DO)
+ }
+ for_each_peer_device_rcu(other_peer_device, other_device) {
+ if (other_peer_device->repl_state[NOW] == L_OFF)
+ continue;
+ if (!__drbd_may_sync_now(other_peer_device))
+ __change_resync_susp_dependency(other_peer_device, true);
+ }
+ if (end_state_change_locked(other_device->resource, "resync-after") !=
+ SS_NOTHING_TO_DO)
changed = true;
}
rcu_read_unlock();
-
return changed;
}
@@ -1563,24 +2842,35 @@ static bool drbd_pause_after(struct drbd_device *device)
* drbd_resume_next() - Resume resync on all devices that may resync now
* @device: DRBD device.
*
- * Called from process context only (admin command and worker).
+ * Called from process context only (admin command and sender).
*/
static bool drbd_resume_next(struct drbd_device *device)
{
+ struct drbd_device *other_device;
bool changed = false;
- struct drbd_device *odev;
- int i;
+ int vnr;
+ /* FIXME seriously inefficient with many devices,
+ * while also ignoring the input "device" argument :-( */
rcu_read_lock();
- idr_for_each_entry(&drbd_devices, odev, i) {
- if (odev->state.conn == C_STANDALONE && odev->state.disk == D_DISKLESS)
+ idr_for_each_entry(&drbd_devices, other_device, vnr) {
+ struct drbd_peer_device *other_peer_device;
+
+ begin_state_change_locked(other_device->resource, CS_HARD);
+ if (other_device->disk_state[NOW] == D_DISKLESS) {
+ abort_state_change_locked(other_device->resource);
continue;
- if (odev->state.aftr_isp) {
- if (_drbd_may_sync_now(odev) &&
- _drbd_set_state(_NS(odev, aftr_isp, 0),
- CS_HARD, NULL) != SS_NOTHING_TO_DO)
- changed = true;
}
+ for_each_peer_device_rcu(other_peer_device, other_device) {
+ if (other_peer_device->repl_state[NOW] == L_OFF)
+ continue;
+ if (other_peer_device->resync_susp_dependency[NOW] &&
+ __drbd_may_sync_now(other_peer_device))
+ __change_resync_susp_dependency(other_peer_device, false);
+ }
+ if (end_state_change_locked(other_device->resource, "resync-after") !=
+ SS_NOTHING_TO_DO)
+ changed = true;
}
rcu_read_unlock();
return changed;
@@ -1589,84 +2879,294 @@ static bool drbd_resume_next(struct drbd_device *device)
void resume_next_sg(struct drbd_device *device)
{
lock_all_resources();
- drbd_resume_next(device);
+ while (drbd_resume_next(device))
+ ; /* Iterate if some state changed. */
unlock_all_resources();
}
void suspend_other_sg(struct drbd_device *device)
{
lock_all_resources();
- drbd_pause_after(device);
+ while (drbd_pause_after(device))
+ ; /* Iterate if some state changed. */
+ unlock_all_resources();
+}
+
+static bool __device_is_resyncing(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device_rcu(peer_device, device) {
+ if (repl_holds_resync_slot(peer_device->repl_state[NOW]))
+ return true;
+ }
+ return false;
+}
+
+static bool __resource_has_resyncing_device(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr;
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (__device_is_resyncing(device))
+ return true;
+ }
+ return false;
+}
+
+/* Only admit a device whose unblocking would actually transition it into
+ * L_SYNC_*. Otherwise the slot would be consumed without progress: e.g.,
+ * a volume blocked by both max_parallel and resync-after would stay
+ * paused on the dependency, starving the volume it is waiting for.
+ */
+static bool device_held_by_max_parallel(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state s = peer_device->repl_state[NOW];
+
+ if (s != L_PAUSED_SYNC_T || !peer_device->resync_susp_max_parallel[NOW])
+ continue;
+
+ if (peer_device->resync_susp_user[NOW] ||
+ peer_device->resync_susp_peer[NOW] ||
+ peer_device->resync_susp_dependency[NOW] ||
+ peer_device->resync_susp_other_c[NOW])
+ continue;
+
+ return true;
+ }
+ return false;
+}
+
+static bool set_suspend_resync_device_max_parallel(struct drbd_device *device, bool value)
+{
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
+
+ begin_state_change_locked(resource, CS_HARD);
+ for_each_peer_device_rcu(peer_device, device) {
+ bool v = value && repl_is_sync_target(peer_device->repl_state[NOW]);
+
+ if (peer_device->resync_susp_max_parallel[NOW] != v)
+ __change_resync_susp_max_parallel(peer_device, v);
+ }
+ return end_state_change_locked(resource, "resync-max-parallel") != SS_NOTHING_TO_DO;
+}
+
+/* Re-evaluate the parallel-resync limit across all volumes and toggle
+ * resync_susp_max_parallel to enforce drbd_max_parallel_resyncs. limit == 0
+ * means unlimited (clear all max_parallel suspensions).
+ */
+void drbd_apply_resync_max_parallel(void)
+{
+ unsigned int limit = READ_ONCE(drbd_max_parallel_resyncs);
+ struct drbd_resource *resource;
+ struct drbd_device *device;
+ unsigned int running = 0;
+ int vnr;
+
+ lock_all_resources();
+ rcu_read_lock();
+
+ if (limit == 0) {
+ for_each_resource_rcu(resource, &drbd_resources) {
+ idr_for_each_entry(&resource->devices, device, vnr)
+ set_suspend_resync_device_max_parallel(device, false);
+ }
+ goto out;
+ }
+
+ for_each_resource_rcu(resource, &drbd_resources) {
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (__device_is_resyncing(device))
+ running++;
+ }
+ }
+
+ if (running > limit) {
+ unsigned int excess = running - limit;
+
+ for_each_resource_rcu(resource, &drbd_resources) {
+ if (excess == 0)
+ break;
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (excess == 0)
+ break;
+ if (__device_is_resyncing(device)) {
+ if (set_suspend_resync_device_max_parallel(device, true))
+ excess--;
+ }
+ }
+ }
+ } else if (running < limit) {
+ unsigned int slots = limit - running;
+
+ /* Pass 1: top up resources that already have a running resync,
+ * so we finish all volumes of one resource before starting
+ * volumes of the next.
+ */
+ for_each_resource_rcu(resource, &drbd_resources) {
+ if (slots == 0)
+ break;
+ if (!__resource_has_resyncing_device(resource))
+ continue;
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (slots == 0)
+ break;
+ if (device_held_by_max_parallel(device)) {
+ if (set_suspend_resync_device_max_parallel(device, false))
+ slots--;
+ }
+ }
+ }
+
+ /* Pass 2: any remaining slots go to volumes of fresh
+ * resources.
+ */
+ for_each_resource_rcu(resource, &drbd_resources) {
+ if (slots == 0)
+ break;
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (slots == 0)
+ break;
+ if (device_held_by_max_parallel(device)) {
+ if (set_suspend_resync_device_max_parallel(device, false))
+ slots--;
+ }
+ }
+ }
+ }
+
+out:
+ rcu_read_unlock();
unlock_all_resources();
}
-/* caller must lock_all_resources() */
-enum drbd_ret_code drbd_resync_after_valid(struct drbd_device *device, int o_minor)
+/* Caller already holds lock_all_resources(). Returns true if @device is
+ * allowed to start a new resync under the parallel-resync limit.
+ */
+static bool __device_max_parallel_admit(struct drbd_device *device)
+{
+ unsigned int limit = READ_ONCE(drbd_max_parallel_resyncs);
+ struct drbd_resource *resource;
+ struct drbd_device *d;
+ unsigned int running = 0;
+ int vnr;
+
+ if (limit == 0)
+ return true;
+
+ if (__device_is_resyncing(device))
+ return true; /* already counted */
+
+ for_each_resource_rcu(resource, &drbd_resources) {
+ idr_for_each_entry(&resource->devices, d, vnr) {
+ if (d == device)
+ continue;
+ if (__device_is_resyncing(d))
+ running++;
+ }
+ }
+ return running < limit;
+}
+
+/* caller must hold resources_mutex */
+enum drbd_ret_code drbd_resync_after_valid(struct drbd_device *device, int resync_after)
{
- struct drbd_device *odev;
- int resync_after;
+ struct drbd_device *other_device;
+ int rv = NO_ERROR;
- if (o_minor == -1)
+ if (resync_after == -1)
return NO_ERROR;
- if (o_minor < -1 || o_minor > MINORMASK)
+ if (resync_after < -1)
return ERR_RESYNC_AFTER;
+ other_device = minor_to_device(resync_after);
+
+ /* You are free to depend on diskless, non-existing,
+ * or not yet/no longer existing minors.
+ * We only reject dependency loops.
+ * We cannot follow the dependency chain beyond a detached or
+ * missing minor.
+ */
+ if (!other_device)
+ return NO_ERROR;
/* check for loops */
- odev = minor_to_device(o_minor);
+ rcu_read_lock();
while (1) {
- if (odev == device)
- return ERR_RESYNC_AFTER_CYCLE;
-
- /* You are free to depend on diskless, non-existing,
- * or not yet/no longer existing minors.
- * We only reject dependency loops.
- * We cannot follow the dependency chain beyond a detached or
- * missing minor.
- */
- if (!odev || !odev->ldev || odev->state.disk == D_DISKLESS)
- return NO_ERROR;
+ if (other_device == device) {
+ rv = ERR_RESYNC_AFTER_CYCLE;
+ break;
+ }
+
+ if (!other_device)
+ break;
+
+ if (!get_ldev_if_state(other_device, D_NEGOTIATING))
+ break;
+ resync_after = rcu_dereference(other_device->ldev->disk_conf)->resync_after;
+ put_ldev(other_device);
- rcu_read_lock();
- resync_after = rcu_dereference(odev->ldev->disk_conf)->resync_after;
- rcu_read_unlock();
/* dependency chain ends here, no cycles. */
if (resync_after == -1)
- return NO_ERROR;
+ break;
/* follow the dependency chain */
- odev = minor_to_device(resync_after);
+ other_device = minor_to_device(resync_after);
}
+ rcu_read_unlock();
+
+ return rv;
}
-/* caller must lock_all_resources() */
+/* caller must hold resources_mutex */
void drbd_resync_after_changed(struct drbd_device *device)
{
- int changed;
-
- do {
- changed = drbd_pause_after(device);
- changed |= drbd_resume_next(device);
- } while (changed);
+ while (drbd_pause_after(device) || drbd_resume_next(device))
+ ;
}
void drbd_rs_controller_reset(struct drbd_peer_device *peer_device)
{
struct drbd_device *device = peer_device->device;
+ /* ldev_safe: callers transition into a resync or verify state */
struct gendisk *disk = device->ldev->backing_bdev->bd_disk;
struct fifo_buffer *plan;
- atomic_set(&device->rs_sect_in, 0);
- atomic_set(&device->rs_sect_ev, 0);
- device->rs_in_flight = 0;
- device->rs_last_events =
- (int)part_stat_read_accum(disk->part0, sectors);
+ atomic_set(&peer_device->rs_sect_in, 0);
+ /* the end of the previous resync canceled its parked writes */
+ atomic_set(&peer_device->rs_sect_parked, 0);
+ peer_device->rs_last_mk_req_kt = ktime_get();
+ peer_device->rs_in_flight = 0;
+ peer_device->rs_pacing_credit = 0;
+ peer_device->rs_mxb_limited = 0;
+ clear_bit(RS_MXB_LIMITED_LOGGED, peer_device->flags);
+ peer_device->rs_last_events = (int)part_stat_read_accum(disk->part0, sectors)
+ - atomic_read(&device->rs_sect_done);
+
+ /* Measure this device now rather than trust what it did last time: it
+ * may not even be the same backing device any more.
+ */
+ device->rs_drain_mark = atomic_read(&device->rs_drain_sect);
+ device->rs_drain_since = ktime_get();
+ device->rs_drain_busy_ns = 0;
+ WRITE_ONCE(device->rs_drain_rate, 0);
+ WRITE_ONCE(device->rs_lat_ns, 0);
+ WRITE_ONCE(device->rs_drain_sample_sect, RS_DEPTH_MIN_SECT);
+ WRITE_ONCE(device->rs_lat_probe, RS_LAT_DRAIN);
+ device->rs_lat_due = jiffies + RS_LAT_PROBE_INTV;
+ device->rs_lat_wend = jiffies + RS_LAT_WINDOW;
+ device->rs_lat_wmin = 0;
/* Updating the RCU protected object in place is necessary since
this function gets called from atomic context.
It is valid since all other updates also lead to an completely
empty fifo */
rcu_read_lock();
- plan = rcu_dereference(device->rs_plan_s);
+ plan = rcu_dereference(peer_device->rs_plan_s);
plan->total = 0;
fifo_set(plan, 0);
rcu_read_unlock();
@@ -1674,76 +3174,160 @@ void drbd_rs_controller_reset(struct drbd_peer_device *peer_device)
void start_resync_timer_fn(struct timer_list *t)
{
- struct drbd_device *device = timer_container_of(device, t,
- start_resync_timer);
- drbd_device_post_work(device, RS_START);
+ struct drbd_peer_device *peer_device = timer_container_of(peer_device, t,
+ start_resync_timer);
+ drbd_peer_device_post_work(peer_device, RS_START);
+}
+
+bool drbd_stable_sync_source_present(struct drbd_peer_device *except_peer_device, enum which_state which)
+{
+ struct drbd_device *device = except_peer_device->device;
+ struct drbd_peer_device *peer_device;
+ u64 authoritative_nodes = 0;
+ bool rv = false;
+
+ if (!(except_peer_device->uuid_flags & UUID_FLAG_STABLE))
+ authoritative_nodes = except_peer_device->uuid_node_mask;
+
+ /* If a peer considers himself as unstable and sees me as an authoritative
+ node, then we have a stable resync source! */
+ if (authoritative_nodes & NODE_MASK(device->resource->res_opts.node_id))
+ return true;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state repl_state;
+ struct drbd_net_conf *nc;
+
+ if (peer_device == except_peer_device)
+ continue;
+
+ repl_state = peer_device->repl_state[which];
+
+ if (repl_state == L_ESTABLISHED ||
+ repl_state == L_WF_BITMAP_S ||
+ (repl_state >= L_SYNC_SOURCE && repl_state < L_AHEAD)) {
+ if (authoritative_nodes & NODE_MASK(peer_device->node_id)) {
+ rv = true;
+ break;
+ }
+
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ /* Restricting the clause the two_primaries not allowed, otherwise
+ we need to ensure here that we are neighbor of all primaries,
+ and that is a lot more challenging. */
+
+ if ((!nc->two_primaries &&
+ peer_device->connection->peer_role[which] == R_PRIMARY) ||
+ ((repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T) &&
+ peer_device->uuid_flags & UUID_FLAG_STABLE)) {
+ rv = true;
+ break;
+ }
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
}
-static void do_start_resync(struct drbd_device *device)
+static void do_start_resync(struct drbd_peer_device *peer_device)
{
- if (atomic_read(&device->unacked_cnt) || atomic_read(&device->rs_pending_cnt)) {
- drbd_warn(device, "postponing start_resync ...\n");
- device->start_resync_timer.expires = jiffies + HZ/10;
- add_timer(&device->start_resync_timer);
+ /*
+ * Postpone resync in any of these situations:
+ * - There is still activity from a previous resync according to rs_pending_cnt.
+ * - The resync as SyncTarget is still active.
+ * - The transition from Ahead to SyncSource if there is any activity on this peer device.
+ */
+ if (atomic_read(&peer_device->rs_pending_cnt) ||
+ peer_device->resync_active[NOW] ||
+ (peer_device->repl_state[NOW] == L_AHEAD &&
+ atomic_read(&peer_device->unacked_cnt))) {
+ drbd_warn(peer_device, "postponing start_resync ...\n");
+ mod_timer(&peer_device->start_resync_timer, jiffies + HZ/10);
return;
}
- drbd_start_resync(device, C_SYNC_SOURCE);
- clear_bit(AHEAD_TO_SYNC_SOURCE, &device->flags);
+ drbd_start_resync(peer_device, peer_device->start_resync_side, "postponed-resync");
+ clear_bit(AHEAD_TO_SYNC_SOURCE, peer_device->flags);
}
-static bool use_checksum_based_resync(struct drbd_connection *connection, struct drbd_device *device)
+static void handle_congestion(struct drbd_peer_device *peer_device)
{
- bool csums_after_crash_only;
+ struct drbd_resource *resource = peer_device->device->resource;
+ unsigned long irq_flags;
+ struct drbd_net_conf *nc;
+ enum drbd_on_congestion on_congestion;
+
rcu_read_lock();
- csums_after_crash_only = rcu_dereference(connection->net_conf)->csums_after_crash_only;
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ if (nc) {
+ on_congestion = nc->on_congestion;
+
+ begin_state_change(resource, &irq_flags, CS_VERBOSE | CS_HARD);
+ /* congestion may have cleared since it was detected */
+ if (atomic_read(&peer_device->connection->ap_in_flight) > 0) {
+ if (on_congestion == OC_PULL_AHEAD)
+ __change_repl_state(peer_device, L_AHEAD);
+ else if (on_congestion == OC_DISCONNECT)
+ __change_cstate(peer_device->connection, C_DISCONNECTING);
+ }
+ end_state_change(resource, &irq_flags, "congestion");
+ }
rcu_read_unlock();
- return connection->agreed_pro_version >= 89 && /* supported? */
- connection->csums_tfm && /* configured? */
- (csums_after_crash_only == false /* use for each resync? */
- || test_bit(CRASHED_PRIMARY, &device->flags)); /* or only after Primary crash? */
+
+ clear_bit(HANDLING_CONGESTION, peer_device->flags);
}
/**
* drbd_start_resync() - Start the resync process
- * @device: DRBD device.
- * @side: Either C_SYNC_SOURCE or C_SYNC_TARGET
+ * @peer_device: The DRBD peer device to start the resync on.
+ * @side: Direction of the resync; which side am I? Either L_SYNC_SOURCE or
+ * L_SYNC_TARGET.
+ * @tag: State change tag to print in status messages.
*
* This function might bring you directly into one of the
* C_PAUSED_SYNC_* states.
*/
-void drbd_start_resync(struct drbd_device *device, enum drbd_conns side)
+void drbd_start_resync(struct drbd_peer_device *peer_device, enum drbd_repl_state side,
+ const char *tag)
{
- struct drbd_peer_device *peer_device = first_peer_device(device);
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- union drbd_state ns;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ enum drbd_disk_state finished_resync_pdsk = D_UNKNOWN;
+ enum drbd_repl_state repl_state;
int r;
- if (device->state.conn >= C_SYNC_SOURCE && device->state.conn < C_AHEAD) {
- drbd_err(device, "Resync already running!\n");
+ read_lock_irq(&device->resource->state_rwlock);
+ repl_state = peer_device->repl_state[NOW];
+ read_unlock_irq(&device->resource->state_rwlock);
+ if (repl_state < L_ESTABLISHED) {
+ /* Connection closed meanwhile. */
return;
}
-
- if (!connection) {
- drbd_err(device, "No connection to peer, aborting!\n");
+ if (repl_state >= L_SYNC_SOURCE && repl_state < L_AHEAD) {
+ drbd_err(peer_device, "Resync already running!\n");
return;
}
- if (!test_bit(B_RS_H_DONE, &device->flags)) {
- if (side == C_SYNC_TARGET) {
- /* Since application IO was locked out during C_WF_BITMAP_T and
- C_WF_SYNC_UUID we are still unmodified. Before going to C_SYNC_TARGET
- we check that we might make the data inconsistent. */
- r = drbd_khelper(device, "before-resync-target");
+ if (!test_bit(B_RS_H_DONE, peer_device->flags)) {
+ if (side == L_SYNC_TARGET) {
+ r = drbd_maybe_khelper(device, connection, "before-resync-target");
+ if (r == DRBD_UMH_DISABLED)
+ goto skip_helper;
+
r = (r >> 8) & 0xff;
if (r > 0) {
drbd_info(device, "before-resync-target handler returned %d, "
"dropping connection.\n", r);
- conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_HARD);
+ change_cstate(connection, C_DISCONNECTING, CS_HARD);
return;
}
- } else /* C_SYNC_SOURCE */ {
- r = drbd_khelper(device, "before-resync-source");
+ } else /* L_SYNC_SOURCE */ {
+ r = drbd_maybe_khelper(device, connection, "before-resync-source");
+ if (r == DRBD_UMH_DISABLED)
+ goto skip_helper;
+
r = (r >> 8) & 0xff;
if (r > 0) {
if (r == 3) {
@@ -1752,185 +3336,118 @@ void drbd_start_resync(struct drbd_device *device, enum drbd_conns side)
} else {
drbd_info(device, "before-resync-source handler returned %d, "
"dropping connection.\n", r);
- conn_request_state(connection,
- NS(conn, C_DISCONNECTING), CS_HARD);
+ change_cstate(connection, C_DISCONNECTING, CS_HARD);
return;
}
}
}
}
- if (current == connection->worker.task) {
- /* The worker should not sleep waiting for state_mutex,
- that can take long */
- if (!mutex_trylock(device->state_mutex)) {
- set_bit(B_RS_H_DONE, &device->flags);
- device->start_resync_timer.expires = jiffies + HZ/5;
- add_timer(&device->start_resync_timer);
- return;
+skip_helper:
+
+ if (side == L_SYNC_TARGET && drbd_current_uuid(device) == UUID_JUST_CREATED) {
+ /* prepare to continue an interrupted initial resync later */
+ if (get_ldev(device)) {
+ const int my_node_id = device->resource->res_opts.node_id;
+ u64 peer_bitmap_uuid = peer_device->bitmap_uuids[my_node_id];
+
+ if (peer_bitmap_uuid) {
+ down_write(&device->uuid_sem);
+ _drbd_uuid_set_current(device, peer_bitmap_uuid);
+ up_write(&device->uuid_sem);
+ drbd_print_uuids(peer_device, "setting UUIDs to");
+ }
+ put_ldev(device);
}
- } else {
- mutex_lock(device->state_mutex);
}
+ if (down_trylock(&device->resource->state_sem)) {
+ /* Retry later and let the worker make progress in the
+ * meantime; two-phase commits depend on that. */
+ set_bit(B_RS_H_DONE, peer_device->flags);
+ peer_device->start_resync_side = side;
+ mod_timer(&peer_device->start_resync_timer, jiffies + HZ/5);
+ return;
+ }
+
+ /* Past the retries, so once per resync start, and before the state
+ * change: the first resync write may follow that at once, no matter
+ * which thread runs this.
+ */
+ if (side == L_SYNC_TARGET)
+ drbd_cbt_merge_resync_bits(peer_device);
+
lock_all_resources();
- clear_bit(B_RS_H_DONE, &device->flags);
- /* Did some connection breakage or IO error race with us? */
- if (device->state.conn < C_CONNECTED
- || !get_ldev_if_state(device, D_NEGOTIATING)) {
+ clear_bit(B_RS_H_DONE, peer_device->flags);
+ if (connection->cstate[NOW] < C_CONNECTED ||
+ !get_ldev_if_state(device, D_NEGOTIATING)) {
unlock_all_resources();
goto out;
}
- ns = drbd_read_state(device);
-
- ns.aftr_isp = !_drbd_may_sync_now(device);
-
- ns.conn = side;
-
- if (side == C_SYNC_TARGET)
- ns.disk = D_INCONSISTENT;
- else /* side == C_SYNC_SOURCE */
- ns.pdsk = D_INCONSISTENT;
-
- r = _drbd_set_state(device, ns, CS_VERBOSE, NULL);
- ns = drbd_read_state(device);
-
- if (ns.conn < C_CONNECTED)
+ begin_state_change_locked(device->resource, CS_VERBOSE);
+ __change_resync_susp_dependency(peer_device, !__drbd_may_sync_now(peer_device));
+ __change_resync_susp_max_parallel(peer_device, side == L_SYNC_TARGET &&
+ !__device_max_parallel_admit(device));
+ __change_repl_state(peer_device, side);
+ if (side == L_SYNC_TARGET)
+ init_resync_stable_bits(peer_device);
+ finished_resync_pdsk = peer_device->resync_finished_pdsk;
+ peer_device->resync_finished_pdsk = D_UNKNOWN;
+ r = end_state_change_locked(device->resource, tag);
+ repl_state = peer_device->repl_state[NOW];
+
+ if (repl_state < L_ESTABLISHED)
r = SS_UNKNOWN_ERROR;
- if (r == SS_SUCCESS) {
- unsigned long tw = drbd_bm_total_weight(device);
- unsigned long now = jiffies;
- int i;
-
- device->rs_failed = 0;
- device->rs_paused = 0;
- device->rs_same_csum = 0;
- device->rs_last_sect_ev = 0;
- device->rs_total = tw;
- device->rs_start = now;
- for (i = 0; i < DRBD_SYNC_MARKS; i++) {
- device->rs_mark_left[i] = tw;
- device->rs_mark_time[i] = now;
- }
+ if (r == SS_SUCCESS)
drbd_pause_after(device);
- /* Forget potentially stale cached per resync extent bit-counts.
- * Open coded drbd_rs_cancel_all(device), we already have IRQs
- * disabled, and know the disk state is ok. */
- spin_lock(&device->al_lock);
- lc_reset(device->resync);
- device->resync_locked = 0;
- device->resync_wenr = LC_FREE;
- spin_unlock(&device->al_lock);
- }
- unlock_all_resources();
-
- if (r == SS_SUCCESS) {
- wake_up(&device->al_wait); /* for lc_reset() above */
- /* reset rs_last_bcast when a resync or verify is started,
- * to deal with potential jiffies wrap. */
- device->rs_last_bcast = jiffies - HZ;
-
- drbd_info(device, "Began resync as %s (will sync %lu KB [%lu bits set]).\n",
- drbd_conn_str(ns.conn),
- (unsigned long) device->rs_total << (BM_BLOCK_SHIFT-10),
- (unsigned long) device->rs_total);
- if (side == C_SYNC_TARGET) {
- device->bm_resync_fo = 0;
- device->use_csums = use_checksum_based_resync(connection, device);
- } else {
- device->use_csums = false;
- }
-
- /* Since protocol 96, we must serialize drbd_gen_and_send_sync_uuid
- * with w_send_oos, or the sync target will get confused as to
- * how much bits to resync. We cannot do that always, because for an
- * empty resync and protocol < 95, we need to do it here, as we call
- * drbd_resync_finished from here in that case.
- * We drbd_gen_and_send_sync_uuid here for protocol < 96,
- * and from after_state_ch otherwise. */
- if (side == C_SYNC_SOURCE && connection->agreed_pro_version < 96)
- drbd_gen_and_send_sync_uuid(peer_device);
-
- if (connection->agreed_pro_version < 95 && device->rs_total == 0) {
- /* This still has a race (about when exactly the peers
- * detect connection loss) that can lead to a full sync
- * on next handshake. In 8.3.9 we fixed this with explicit
- * resync-finished notifications, but the fix
- * introduces a protocol change. Sleeping for some
- * time longer than the ping interval + timeout on the
- * SyncSource, to give the SyncTarget the chance to
- * detect connection loss, then waiting for a ping
- * response (implicit in drbd_resync_finished) reduces
- * the race considerably, but does not solve it. */
- if (side == C_SYNC_SOURCE) {
- struct net_conf *nc;
- int timeo;
-
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- timeo = nc->ping_int * HZ + nc->ping_timeo * HZ / 9;
- rcu_read_unlock();
- schedule_timeout_interruptible(timeo);
- }
- drbd_resync_finished(peer_device);
- }
- drbd_rs_controller_reset(peer_device);
- /* ns.conn may already be != device->state.conn,
- * we may have been paused in between, or become paused until
- * the timer triggers.
- * No matter, that is handled in resync_timer_fn() */
- if (ns.conn == C_SYNC_TARGET)
- mod_timer(&device->resync_timer, jiffies);
-
- drbd_md_sync(device);
- }
+ unlock_all_resources();
put_ldev(device);
out:
- mutex_unlock(device->state_mutex);
+ up(&device->resource->state_sem);
+ if (finished_resync_pdsk != D_UNKNOWN)
+ drbd_resync_finished(peer_device, finished_resync_pdsk);
}
static void update_on_disk_bitmap(struct drbd_peer_device *peer_device, bool resync_done)
{
struct drbd_device *device = peer_device->device;
- struct sib_info sib = { .sib_reason = SIB_SYNC_PROGRESS, };
- device->rs_last_bcast = jiffies;
+
+ peer_device->rs_last_writeout = jiffies;
if (!get_ldev(device))
return;
drbd_bm_write_lazy(device, 0);
- if (resync_done && is_sync_state(device->state.conn))
- drbd_resync_finished(peer_device);
-
- drbd_bcast_event(device, &sib);
- /* update timestamp, in case it took a while to write out stuff */
- device->rs_last_bcast = jiffies;
- put_ldev(device);
-}
-static void drbd_ldev_destroy(struct drbd_device *device)
-{
- lc_destroy(device->resync);
- device->resync = NULL;
- lc_destroy(device->act_log);
- device->act_log = NULL;
+ if (resync_done) {
+ if (is_verify_state(peer_device, NOW)) {
+ ov_out_of_sync_print(peer_device);
+ ov_skipped_print(peer_device);
+ drbd_resync_finished(peer_device, D_MASK);
+ } else if (is_sync_state(peer_device, NOW)) {
+ drbd_resync_finished(peer_device, D_MASK);
+ }
+ }
- __acquire(local);
- drbd_backing_dev_free(device, device->ldev);
- device->ldev = NULL;
- __release(local);
+ /* v1 dialect only: fused SIB_SYNC_PROGRESS DRBD_EVENT, at the same
+ * point and under the same throttle mainline's own
+ * update_on_disk_bitmap() broadcasts one from (drivers/block/drbd/
+ * drbd_worker.c). No-op stub without CONFIG_DRBD_COMPAT_84.
+ */
+ compat84_notify_sync_progress(peer_device);
- clear_bit(GOING_DISKLESS, &device->flags);
- wake_up(&device->misc_wait);
+ /* update timestamp, in case it took a while to write out stuff */
+ peer_device->rs_last_writeout = jiffies;
+ put_ldev(device);
}
static void go_diskless(struct drbd_device *device)
{
- struct drbd_peer_device *peer_device = first_peer_device(device);
- D_ASSERT(device, device->state.disk == D_FAILED);
+ D_ASSERT(device, device->disk_state[NOW] == D_FAILED ||
+ device->disk_state[NOW] == D_DETACHING);
/* we cannot assert local_cnt == 0 here, as get_ldev_if_state will
* inc/dec it frequently. Once we are D_DISKLESS, no one will touch
* the protected members anymore, though, so once put_ldev reaches zero
@@ -1949,96 +3466,255 @@ static void go_diskless(struct drbd_device *device)
* We still need to check if both bitmap and ldev are present, we may
* end up here after a failed attach, before ldev was even assigned.
*/
+ /* ldev_safe: ldev is only destroyed after state change to D_DISKLESS below */
if (device->bitmap && device->ldev) {
- /* An interrupted resync or similar is allowed to recounts bits
- * while we detach.
- * Any modifications would not be expected anymore, though.
- */
if (drbd_bitmap_io_from_worker(device, drbd_bm_write,
- "detach", BM_LOCKED_TEST_ALLOWED, peer_device)) {
- if (test_bit(WAS_READ_ERROR, &device->flags)) {
- drbd_md_set_flag(device, MDF_FULL_SYNC);
- drbd_md_sync(device);
+ "detach",
+ BM_LOCK_SET | BM_LOCK_CLEAR | BM_LOCK_BULK,
+ NULL)) {
+ if (test_bit(CRASHED_PRIMARY, &device->flags)) {
+ struct drbd_peer_device *peer_device;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device)
+ drbd_md_set_peer_flag(peer_device, __MDF_PEER_FULL_SYNC);
+ rcu_read_unlock();
+ }
+ }
+ }
+
+ drbd_md_sync_if_dirty(device);
+ change_disk_state(device, D_DISKLESS, CS_HARD, "go-diskless", NULL);
+}
+
+static int do_md_sync(struct drbd_device *device)
+{
+ drbd_warn(device, "md_sync_timer expired! Worker calls drbd_md_sync().\n");
+ drbd_md_sync(device);
+ return 0;
+}
+
+/* only called from drbd_worker thread, no locking */
+void __update_timing_details(
+ struct drbd_thread_timing_details *tdp,
+ unsigned int *cb_nr,
+ void *cb,
+ const char *fn, const unsigned int line)
+{
+ unsigned int i = *cb_nr % DRBD_THREAD_DETAILS_HIST;
+ struct drbd_thread_timing_details *td = tdp + i;
+
+ td->start_jif = jiffies;
+ td->cb_addr = cb;
+ td->caller_fn = fn;
+ td->line = line;
+ td->cb_nr = *cb_nr;
+
+ i = (i+1) % DRBD_THREAD_DETAILS_HIST;
+ td = tdp + i;
+ memset(td, 0, sizeof(*td));
+
+ ++(*cb_nr);
+}
+
+static bool all_responded(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ bool all_responded = true;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (!test_bit(CHECKING_PEER, &connection->flags))
+ continue;
+ if (connection->cstate[NOW] < C_CONNECTED) {
+ clear_bit(CHECKING_PEER, &connection->flags);
+ continue;
+ }
+ if (test_bit(PING_PENDING, &connection->flags)) {
+ all_responded = false;
+ continue;
+ } else {
+ clear_bit(CHECKING_PEER, &connection->flags);
+ }
+ }
+ rcu_read_unlock();
+
+ return all_responded;
+}
+
+void drbd_check_peers(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ long t, timeo = LONG_MAX;
+ unsigned long start;
+ bool check_ongoing;
+ u64 im;
+
+ check_ongoing = test_and_set_bit(CHECKING_PEERS, &resource->flags);
+ if (check_ongoing) {
+ wait_event(resource->state_wait,
+ !test_bit(CHECKING_PEERS, &resource->flags));
+ return;
+ }
+
+ start = jiffies;
+ for_each_connection_ref(connection, im, resource) {
+ if (connection->cstate[NOW] < C_CONNECTED)
+ continue;
+ set_bit(CHECKING_PEER, &connection->flags);
+ send_ping_peer(connection);
+ t = ping_timeout(connection);
+ if (t < timeo)
+ timeo = t;
+ }
+
+ while (!wait_event_timeout(resource->state_wait, all_responded(resource), timeo)) {
+ unsigned long waited = jiffies - start;
+
+ timeo = LONG_MAX;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (!test_bit(CHECKING_PEER, &connection->flags))
+ continue;
+ t = ping_timeout(connection);
+ if (waited >= t) {
+ drbd_warn(connection, "peer failed to send PingAck in time\n");
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ clear_bit(CHECKING_PEER, &connection->flags);
+ continue;
}
+ if (t - waited < timeo)
+ timeo = t - waited;
}
+ rcu_read_unlock();
}
- drbd_force_state(device, NS(disk, D_DISKLESS));
+ clear_bit(CHECKING_PEERS, &resource->flags);
+ wake_up_all(&resource->state_wait);
}
-static int do_md_sync(struct drbd_device *device)
+enum drbd_mint_outcome drbd_check_peers_new_current_uuid(struct drbd_device *device)
{
- drbd_warn(device, "md_sync_timer expired! Worker calls drbd_md_sync().\n");
- drbd_md_sync(device);
- return 0;
+ struct drbd_resource *resource = device->resource;
+
+ drbd_check_peers(resource);
+
+ /* gen-rotate reason: DEGRADE (peer disconnected; create deferred bump once quorate) */
+ if (!device->have_quorum[NOW] || !drbd_data_accessible(device, NOW))
+ return MINT_NOT_EVALUATED;
+
+ return drbd_uuid_new_current(device, false);
}
-/* only called from drbd_worker thread, no locking */
-void __update_timing_details(
- struct drbd_thread_timing_details *tdp,
- unsigned int *cb_nr,
- void *cb,
- const char *fn, const unsigned int line)
+/* The mint executor. A materialized obligation is about writes this node
+ * decided to complete without a lost replica: the settle round must not delay
+ * that generation, and the quorum and data gate must not decline it.
+ *
+ * Runs in the worker for a dispatched mint, and in the caller's thread where
+ * one waits for the outcome; both enter it through drbd_gen_obligation_mint_start().
+ * It sleeps.
+ */
+void drbd_gen_obligation_mint_run(struct drbd_device *device)
{
- unsigned int i = *cb_nr % DRBD_THREAD_DETAILS_HIST;
- struct drbd_thread_timing_details *td = tdp + i;
-
- td->start_jif = jiffies;
- td->cb_addr = cb;
- td->caller_fn = fn;
- td->line = line;
- td->cb_nr = *cb_nr;
+ enum drbd_mint_outcome outcome;
- i = (i+1) % DRBD_THREAD_DETAILS_HIST;
- td = tdp + i;
- memset(td, 0, sizeof(*td));
+ if (drbd_gen_obligation_materialized(device))
+ outcome = drbd_uuid_new_current(device, false);
+ else
+ outcome = drbd_check_peers_new_current_uuid(device);
- ++(*cb_nr);
+ drbd_gen_obligation_mint_done(device, outcome);
+ wake_up(&device->misc_wait);
}
static void do_device_work(struct drbd_device *device, const unsigned long todo)
{
if (test_bit(MD_SYNC, &todo))
do_md_sync(device);
- if (test_bit(RS_DONE, &todo) ||
- test_bit(RS_PROGRESS, &todo))
- update_on_disk_bitmap(first_peer_device(device), test_bit(RS_DONE, &todo));
if (test_bit(GO_DISKLESS, &todo))
go_diskless(device);
- if (test_bit(DESTROY_DISK, &todo))
- drbd_ldev_destroy(device);
+ if (test_bit(MAKE_NEW_CUR_UUID, &todo))
+ drbd_gen_obligation_mint_run(device);
+ if (test_bit(AUTO_GROW, &todo))
+ drbd_auto_grow(device);
+}
+
+static void do_peer_device_work(struct drbd_peer_device *peer_device, const unsigned long todo)
+{
+ if (test_bit(RS_PROGRESS, &todo))
+ drbd_broadcast_peer_device_state(peer_device);
+ if (test_bit(RS_DONE, &todo) ||
+ test_bit(RS_LAZY_BM_WRITE, &todo))
+ update_on_disk_bitmap(peer_device, test_bit(RS_DONE, &todo));
if (test_bit(RS_START, &todo))
- do_start_resync(device);
+ do_start_resync(peer_device);
+ if (test_bit(HANDLE_CONGESTION, &todo))
+ handle_congestion(peer_device);
+ if (test_bit(SEND_RECONCILE_UUID, &todo))
+ send_reconcile_current_uuid(peer_device);
}
#define DRBD_DEVICE_WORK_MASK \
((1UL << GO_DISKLESS) \
- |(1UL << DESTROY_DISK) \
|(1UL << MD_SYNC) \
- |(1UL << RS_START) \
- |(1UL << RS_PROGRESS) \
- |(1UL << RS_DONE) \
+ |(1UL << MAKE_NEW_CUR_UUID)\
+ |(1UL << AUTO_GROW) \
)
-static unsigned long get_work_bits(unsigned long *flags)
-{
- unsigned long old, new;
- do {
- old = *flags;
- new = old & ~DRBD_DEVICE_WORK_MASK;
- } while (cmpxchg(flags, old, new) != old);
- return old & DRBD_DEVICE_WORK_MASK;
-}
+#define DRBD_PEER_DEVICE_WORK_MASK \
+ ((1UL << RS_START) \
+ |(1UL << RS_LAZY_BM_WRITE) \
+ |(1UL << RS_PROGRESS) \
+ |(1UL << RS_DONE) \
+ |(1UL << HANDLE_CONGESTION) \
+ |(1UL << SEND_RECONCILE_UUID) \
+ )
-static void do_unqueued_work(struct drbd_connection *connection)
+static void __do_unqueued_peer_device_work(struct drbd_connection *connection)
{
struct drbd_peer_device *peer_device;
int vnr;
+ /* get_work_bits() sees only the first word of peer_device->flags,
+ * which has 32 bits on 32-bit kernels
+ */
+ BUILD_BUG_ON((u64)DRBD_PEER_DEVICE_WORK_MASK >> 32);
+
rcu_read_lock();
idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
struct drbd_device *device = peer_device->device;
- unsigned long todo = get_work_bits(&device->flags);
+ unsigned long todo = get_work_bits(DRBD_PEER_DEVICE_WORK_MASK, peer_device->flags);
+
+ if (!todo)
+ continue;
+
+ kref_get(&device->kref);
+ rcu_read_unlock();
+ do_peer_device_work(peer_device, todo);
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+}
+
+static void do_unqueued_peer_device_work(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ u64 im;
+
+ for_each_connection_ref(connection, im, resource)
+ __do_unqueued_peer_device_work(connection);
+}
+
+static void do_unqueued_device_work(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ unsigned long todo = get_work_bits(DRBD_DEVICE_WORK_MASK, &device->flags);
+
if (!todo)
continue;
@@ -2059,14 +3735,137 @@ static bool dequeue_work_batch(struct drbd_work_queue *queue, struct list_head *
return !list_empty(work_list);
}
-static void wait_for_work(struct drbd_connection *connection, struct list_head *work_list)
+static struct drbd_request *__next_request_for_connection(
+ struct drbd_connection *connection)
+{
+ struct drbd_request *req;
+
+ list_for_each_entry_rcu(req, &connection->resource->transfer_log, tl_requests) {
+ unsigned int s = req->net_rq_state[connection->peer_node_id];
+
+ if (likely(s & RQ_NET_QUEUED))
+ return req;
+ }
+ return NULL;
+}
+
+static struct drbd_request *tl_next_request_for_connection(
+ struct drbd_connection *connection, bool wait_ready)
+{
+ struct drbd_request *req;
+
+ /*
+ * No cstate check here, unlike drbd-9.2: a request that is
+ * RQ_NET_QUEUED holds an oos_send_ref and stays on the transfer log
+ * until process_one_request() retires it, and tl_walk(CONNECTION_LOST)
+ * skips a request that is already RQ_NET_DONE. The sender therefore
+ * still has to hand out queued requests below C_CONNECTED, even though
+ * every send in process_one_request() fails there.
+ */
+ if (connection->todo.req_next == NULL)
+ connection->todo.req_next = __next_request_for_connection(connection);
+
+ req = connection->todo.req_next;
+ if (req && wait_ready &&
+ !(req->net_rq_state[connection->peer_node_id] & RQ_NET_READY)) {
+ /*
+ * The oldest queued request only announces out-of-sync blocks
+ * and becomes ready once it is done. That may take a while,
+ * and it may even depend on the barrier ack for the epoch of
+ * a preceding write. Do not hold back the requests behind it;
+ * process the oldest ready one instead. Leave the request in
+ * "req_next" until it is ready.
+ */
+ req = READ_ONCE(connection->req_next_ready);
+ }
+
+ connection->todo.req = req;
+ if (req)
+ connection->send.seen_dagtag_sector = req->dagtag_sector;
+
+ /*
+ * Advancement of todo.req_next happens in advance_conn_req_next(),
+ * that of req_next_ready in advance_cache_ptr(), both called from
+ * mod_rq_state()
+ */
+
+ return req;
+}
+
+static void maybe_send_state_after_ahead(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (test_and_clear_bit(SEND_STATE_AFTER_AHEAD, peer_device->flags)) {
+ peer_device->todo.was_sending_out_of_sync = false;
+ rcu_read_unlock();
+ drbd_send_current_state(peer_device);
+ rcu_read_lock();
+ }
+ }
+ rcu_read_unlock();
+}
+
+/* This finds the next not yet processed request from
+ * connection->resource->transfer_log.
+ * It also moves all currently queued connection->sender_work
+ * to connection->todo.work_list.
+ */
+static bool check_sender_todo(struct drbd_connection *connection)
+{
+ rcu_read_lock();
+ tl_next_request_for_connection(connection, true);
+
+ /* FIXME can we get rid of this additional lock? */
+ spin_lock_irq(&connection->sender_work.q_lock);
+ list_splice_tail_init(&connection->sender_work.q, &connection->todo.work_list);
+ spin_unlock_irq(&connection->sender_work.q_lock);
+ rcu_read_unlock();
+
+ return connection->todo.req
+ || need_unplug(connection)
+ || !list_empty(&connection->todo.work_list);
+}
+
+/*
+ * Whether all requests that the sender has to process are in the transfer log
+ * and have been processed. Requests that only announce out-of-sync blocks do
+ * not count: they are not ready before they are done, and the sender has
+ * passed them.
+ *
+ * Replicated writes are ready as soon as they are added to the transfer log,
+ * so req_next_ready is set while any of them waits to be processed. A write
+ * that is being submitted has not yet reached the transfer log, so it is not
+ * covered by that. Take tl_update_lock to wait for it.
+ */
+static bool drbd_sender_processed_all_writes(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_request *req_next;
+ bool ready;
+
+ spin_lock_irq(&resource->tl_update_lock);
+ req_next = connection->todo.req_next;
+ ready = (req_next && (req_next->net_rq_state[connection->peer_node_id] & RQ_NET_READY)) ||
+ READ_ONCE(connection->req_next_ready);
+ spin_unlock_irq(&resource->tl_update_lock);
+
+ return !ready;
+}
+
+static void wait_for_sender_todo(struct drbd_connection *connection)
{
+ struct drbd_resource *resource = connection->resource;
DEFINE_WAIT(wait);
- struct net_conf *nc;
+ struct drbd_net_conf *nc;
int uncork, cork;
+ bool got_something = 0;
- dequeue_work_batch(&connection->sender_work, work_list);
- if (!list_empty(work_list))
+ got_something = check_sender_todo(connection);
+ if (got_something)
return;
/* Still nothing to do?
@@ -2076,50 +3875,72 @@ static void wait_for_work(struct drbd_connection *connection, struct list_head *
* Also, poke TCP, just in case.
* Then wait for new work (or signal). */
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
+ nc = rcu_dereference(connection->transport.net_conf);
uncork = nc ? nc->tcp_cork : 0;
rcu_read_unlock();
if (uncork) {
- mutex_lock(&connection->data.mutex);
- if (connection->data.socket)
- tcp_sock_set_cork(connection->data.socket->sk, false);
- mutex_unlock(&connection->data.mutex);
+ if (drbd_uncork(connection, DATA_STREAM))
+ return;
}
for (;;) {
- int send_barrier;
- prepare_to_wait(&connection->sender_work.q_wait, &wait, TASK_INTERRUPTIBLE);
- spin_lock_irq(&connection->resource->req_lock);
- spin_lock(&connection->sender_work.q_lock); /* FIXME get rid of this one? */
- if (!list_empty(&connection->sender_work.q))
- list_splice_tail_init(&connection->sender_work.q, work_list);
- spin_unlock(&connection->sender_work.q_lock); /* FIXME get rid of this one? */
- if (!list_empty(work_list) || signal_pending(current)) {
- spin_unlock_irq(&connection->resource->req_lock);
+ bool send_barrier = false;
+
+ prepare_to_wait(&connection->sender_work.q_wait, &wait,
+ TASK_INTERRUPTIBLE);
+ if (check_sender_todo(connection) || signal_pending(current))
break;
- }
/* We found nothing new to do, no to-be-communicated request,
* no other work item. We may still need to close the last
* epoch. Next incoming request epoch will be connection ->
* current transfer log epoch number. If that is different
- * from the epoch of the last request we communicated, it is
- * safe to send the epoch separating barrier now.
+ * from the epoch of the last request we communicated, we want
+ * to send the epoch separating barrier now.
*/
- send_barrier =
- atomic_read(&connection->current_tle_nr) !=
- connection->send.current_epoch_nr;
- spin_unlock_irq(&connection->resource->req_lock);
+ if (connection->cstate[NOW] >= C_CONNECTED)
+ send_barrier = should_send_barrier(connection,
+ atomic_read(&resource->current_tle_nr));
+
+ if (send_barrier) {
+ /* Ensure that we read the most recent
+ * resource->dagtag_sector value. */
+ smp_rmb();
+ /* If a request is currently being submitted it may not
+ * have been picked up by this sender, even though it
+ * belongs to the old epoch. Ensure that we are
+ * up-to-date with the most recently submitted dagtag
+ * to ensure that we do not send a barrier early in
+ * this case. If there is such a request then this
+ * sender will be woken, so it is OK to schedule().
+ *
+ * Requests that only announce out-of-sync blocks are
+ * not ready before they are done. The sender passes
+ * them, so it may not have seen the most recent
+ * dagtag although it has processed all replicated
+ * writes. Then check that no further write is queued
+ * or being submitted, so that we do not send a
+ * barrier early in this case either.
+ */
+ if (dagtag_newer_eq(connection->send.seen_dagtag_sector,
+ READ_ONCE(resource->dagtag_sector)) ||
+ drbd_sender_processed_all_writes(connection)) {
+ finish_wait(&connection->sender_work.q_wait, &wait);
+ maybe_send_barrier(connection,
+ connection->send.current_epoch_nr + 1);
+ continue;
+ }
+ }
- if (send_barrier)
- maybe_send_barrier(connection,
- connection->send.current_epoch_nr + 1);
+ if (test_and_clear_bit(SEND_STATE_AFTER_AHEAD_C, &connection->flags)) {
+ finish_wait(&connection->sender_work.q_wait, &wait);
+ maybe_send_state_after_ahead(connection);
+ continue;
+ }
- if (test_bit(DEVICE_WORK_PENDING, &connection->flags))
- break;
/* drbd_send() may have called flush_signals() */
- if (get_t_state(&connection->worker) != RUNNING)
+ if (get_t_state(&connection->sender) != RUNNING)
break;
schedule();
@@ -2131,44 +3952,322 @@ static void wait_for_work(struct drbd_connection *connection, struct list_head *
/* someone may have changed the config while we have been waiting above. */
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
+ nc = rcu_dereference(connection->transport.net_conf);
cork = nc ? nc->tcp_cork : 0;
rcu_read_unlock();
- mutex_lock(&connection->data.mutex);
- if (connection->data.socket) {
- if (cork)
- tcp_sock_set_cork(connection->data.socket->sk, true);
- else if (!uncork)
- tcp_sock_set_cork(connection->data.socket->sk, false);
+
+ if (cork)
+ drbd_cork(connection, DATA_STREAM);
+ else if (!uncork)
+ drbd_uncork(connection, DATA_STREAM);
+}
+
+static void re_init_if_first_write(struct drbd_connection *connection, unsigned int epoch)
+{
+ if (!connection->send.seen_any_write_yet) {
+ connection->send.current_epoch_nr = epoch;
+ connection->send.current_epoch_writes = 0;
+ connection->send.last_sent_barrier_jif = jiffies;
+ /* no barrier is pending; the first one of this connection is @epoch */
+ WRITE_ONCE(connection->send.last_sent_epoch_nr, epoch - 1);
+ WRITE_ONCE(connection->last_acked_epoch_nr, epoch - 1);
+ /* pairs with smp_load_acquire() in barriers_acked() */
+ smp_store_release(&connection->send.seen_any_write_yet, true);
}
- mutex_unlock(&connection->data.mutex);
}
-int drbd_worker(struct drbd_thread *thi)
+static bool should_send_barrier(struct drbd_connection *connection, unsigned int epoch)
+{
+ if (!connection->send.seen_any_write_yet)
+ return false;
+ return connection->send.current_epoch_nr != epoch;
+}
+static void maybe_send_barrier(struct drbd_connection *connection, unsigned int epoch)
+{
+ /* re-init if first write on this connection */
+ if (!should_send_barrier(connection, epoch))
+ return;
+
+ /*
+ * A request that only announces out-of-sync blocks is processed once
+ * it is done. By then, the sender may have closed its epoch already.
+ * Never step back to a closed epoch, the requests processed since
+ * belong to the current one.
+ */
+ if ((int)(epoch - connection->send.current_epoch_nr) < 0)
+ return;
+
+ if (connection->send.current_epoch_writes)
+ drbd_send_barrier(connection);
+ /* after last_sent_epoch_nr; pairs with smp_load_acquire() in barriers_acked() */
+ smp_store_release(&connection->send.current_epoch_nr, epoch);
+ if (wq_has_sleeper(&connection->resource->barrier_wait))
+ wake_up(&connection->resource->barrier_wait);
+}
+
+/* The reconcile peer we asserted UpToDate on its predecessor generation has
+ * settled UpToDate; the handshake is complete and it knows we are Primary.
+ * Relabel it forward to our real current generation now. Running on the sender,
+ * this P_CURRENT_UUID is ordered after the replayed transfer-log writes, and the
+ * peer is UpToDate, so it takes the adopt path in receive_current_uuid (rather
+ * than outdating). See diskless_with_peers_different_current_uuids().
+ */
+static void send_reconcile_current_uuid(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+
+ if (!test_and_clear_bit(RECONCILE_INJECT_CUR_UUID, peer_device->flags))
+ return;
+ drbd_send_current_uuid(peer_device, device->exposed_data_uuid,
+ drbd_weak_nodes_device(device));
+}
+
+static int process_one_request(struct drbd_connection *connection)
+{
+ struct bio_and_error m;
+ struct drbd_request *req = connection->todo.req;
+ struct drbd_device *device = req->device;
+ struct drbd_peer_device *peer_device =
+ conn_peer_device(connection, device->vnr);
+ unsigned int s = req->net_rq_state[peer_device->node_id];
+ bool do_send_unplug = req->local_rq_state & RQ_UNPLUG;
+ int err = 0;
+ enum drbd_req_event what;
+
+ /* pre_send_jif[] is used in net_timeout_reached() */
+ req->pre_send_jif[peer_device->node_id] = jiffies;
+ ktime_get_accounting(req->pre_send_kt[peer_device->node_id]);
+ if (drbd_req_is_write(req)) {
+ /* If a WRITE does not expect a barrier ack,
+ * we are supposed to only send an "out of sync" info packet */
+ if (s & RQ_EXP_BARR_ACK) {
+ u64 current_dagtag_sector =
+ req->dagtag_sector - (req->i.size >> 9);
+
+ re_init_if_first_write(connection, req->epoch);
+ maybe_send_barrier(connection, req->epoch);
+
+ if (current_dagtag_sector != connection->send.current_dagtag_sector)
+ drbd_send_dagtag(connection, current_dagtag_sector);
+
+ connection->send.current_epoch_writes++;
+ connection->send.current_dagtag_sector = req->dagtag_sector;
+
+ if (peer_device->todo.was_sending_out_of_sync) {
+ clear_bit(SEND_STATE_AFTER_AHEAD, peer_device->flags);
+ peer_device->todo.was_sending_out_of_sync = false;
+ drbd_send_current_state(peer_device);
+ }
+
+ err = drbd_send_dblock(peer_device, req);
+ what = err ? SEND_FAILED : HANDED_OVER_TO_NETWORK;
+ } else {
+ /* this time, no connection->send.current_epoch_writes++;
+ * If it was sent, it was the closing barrier for the last
+ * replicated epoch, before we went into AHEAD mode.
+ * No more barriers will be sent, until we leave AHEAD mode again.
+ * If this request became ready only after its epoch
+ * was closed, maybe_send_barrier() does nothing.
+ */
+ maybe_send_barrier(connection, req->epoch);
+
+ /* make sure the state change to L_AHEAD/L_BEHIND
+ * arrives before the first set-out-of-sync information */
+ if (!peer_device->todo.was_sending_out_of_sync) {
+ peer_device->todo.was_sending_out_of_sync = true;
+ drbd_send_current_state(peer_device);
+ }
+
+ /* When this flag is not set, sending OOS may be skipped */
+ if (s & RQ_NET_PENDING_OOS)
+ err = drbd_send_out_of_sync(peer_device,
+ req->i.sector, req->i.size);
+ /* This event has the appropriate effect even if OOS skipped or failed */
+ what = OOS_HANDED_TO_NETWORK;
+ }
+ } else {
+ maybe_send_barrier(connection, req->epoch);
+ err = drbd_send_drequest(peer_device,
+ req->i.sector, req->i.size, (unsigned long)req);
+ what = err ? SEND_FAILED : HANDED_OVER_TO_NETWORK;
+ }
+
+ read_lock_irq(&connection->resource->state_rwlock);
+ __req_mod(req, what, peer_device, &m);
+ read_unlock_irq(&connection->resource->state_rwlock);
+
+ check_sender_todo(connection);
+
+ if (m.bio)
+ complete_master_bio(device, &m);
+
+ do_send_unplug = do_send_unplug && what == HANDED_OVER_TO_NETWORK;
+ maybe_send_unplug_remote(connection, do_send_unplug);
+
+ return err;
+}
+
+static int process_sender_todo(struct drbd_connection *connection)
{
- struct drbd_connection *connection = thi->connection;
struct drbd_work *w = NULL;
+
+ /* Process all currently pending work items,
+ * or requests from the transfer log.
+ *
+ * Right now, work items do not require any strict ordering wrt. the
+ * request stream, so lets just do simple interleaved processing.
+ *
+ * Stop processing as soon as an error is encountered.
+ */
+ if (!connection->todo.req) {
+ update_sender_timing_details(connection, maybe_send_unplug_remote);
+ maybe_send_unplug_remote(connection, false);
+ } else if (list_empty(&connection->todo.work_list)) {
+ update_sender_timing_details(connection, process_one_request);
+ /* ldev_safe: have connection->todo.req which holds its own ldev ref */
+ return process_one_request(connection);
+ }
+
+ while (!list_empty(&connection->todo.work_list)) {
+ int err;
+
+ w = list_first_entry(&connection->todo.work_list, struct drbd_work, list);
+ list_del_init(&w->list);
+ update_sender_timing_details(connection, w->cb);
+ err = w->cb(w, connection->cstate[NOW] < C_CONNECTED);
+ if (err)
+ return err;
+
+ if (connection->todo.req) {
+ update_sender_timing_details(connection, process_one_request);
+ /* ldev_safe: have connection->todo.req which holds its own ldev ref */
+ err = process_one_request(connection);
+ }
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+int drbd_sender(struct drbd_thread *thi)
+{
+ struct drbd_connection *connection = thi->connection;
+ struct drbd_work *w;
struct drbd_peer_device *peer_device;
- LIST_HEAD(work_list);
int vnr;
+ int err;
+
+ /* Should we drop this? Or reset even more stuff? */
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ peer_device->send_cnt = 0;
+ peer_device->recv_cnt = 0;
+ }
+ rcu_read_unlock();
while (get_t_state(thi) == RUNNING) {
drbd_thread_current_set_cpu(thi);
- if (list_empty(&work_list)) {
- update_worker_timing_details(connection, wait_for_work);
- wait_for_work(connection, &work_list);
+ /* The only other places this thread can yield the CPU are
+ * wait_for_sender_todo() and a network send.
+ */
+ cond_resched();
+
+ if (list_empty(&connection->todo.work_list) &&
+ connection->todo.req == NULL) {
+ update_sender_timing_details(connection, wait_for_sender_todo);
+ wait_for_sender_todo(connection);
+ }
+
+ if (signal_pending(current)) {
+ flush_signals(current);
+ if (get_t_state(thi) == RUNNING) {
+ drbd_warn(connection, "Sender got an unexpected signal\n");
+ continue;
+ }
+ break;
+ }
+
+ if (get_t_state(thi) != RUNNING)
+ break;
+
+ err = process_sender_todo(connection);
+ if (err)
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ }
+
+ /* cleanup all currently unprocessed requests */
+ if (!connection->todo.req) {
+ rcu_read_lock();
+ tl_next_request_for_connection(connection, false);
+ rcu_read_unlock();
+ }
+ while (connection->todo.req) {
+ struct bio_and_error m;
+ struct drbd_request *req = connection->todo.req;
+ struct drbd_device *device = req->device;
+
+ peer_device = conn_peer_device(connection, device->vnr);
+ read_lock_irq(&connection->resource->state_rwlock);
+ /* ldev_safe: requests hold their own ldev refs */
+ __req_mod(req, SEND_CANCELED, peer_device, &m);
+ read_unlock_irq(&connection->resource->state_rwlock);
+ if (m.bio)
+ complete_master_bio(device, &m);
+
+ rcu_read_lock();
+ tl_next_request_for_connection(connection, false);
+ rcu_read_unlock();
+ }
+
+ /* cancel all still pending works */
+ do {
+ while (!list_empty(&connection->todo.work_list)) {
+ w = list_first_entry(&connection->todo.work_list, struct drbd_work, list);
+ list_del_init(&w->list);
+ w->cb(w, 1);
}
+ dequeue_work_batch(&connection->sender_work, &connection->todo.work_list);
+ } while (!list_empty(&connection->todo.work_list));
+
+ return 0;
+}
+
+int drbd_worker(struct drbd_thread *thi)
+{
+ LIST_HEAD(work_list);
+ struct drbd_resource *resource = thi->resource;
+ struct drbd_work *w;
+
+ while (get_t_state(thi) == RUNNING) {
+ drbd_thread_current_set_cpu(thi);
+
+ if (list_empty(&work_list)) {
+ bool w, d, p;
+
+ update_worker_timing_details(resource, dequeue_work_batch);
+ wait_event_interruptible(resource->work.q_wait,
+ (w = dequeue_work_batch(&resource->work, &work_list),
+ d = test_and_clear_bit(DEVICE_WORK_PENDING, &resource->flags),
+ p = test_and_clear_bit(PEER_DEVICE_WORK_PENDING, &resource->flags),
+ w || d || p));
+
+ if (p) {
+ update_worker_timing_details(resource, do_unqueued_peer_device_work);
+ do_unqueued_peer_device_work(resource);
+ }
- if (test_and_clear_bit(DEVICE_WORK_PENDING, &connection->flags)) {
- update_worker_timing_details(connection, do_unqueued_work);
- do_unqueued_work(connection);
+ if (d) {
+ update_worker_timing_details(resource, do_unqueued_device_work);
+ do_unqueued_device_work(resource);
+ }
}
if (signal_pending(current)) {
flush_signals(current);
if (get_t_state(thi) == RUNNING) {
- drbd_warn(connection, "Worker got an unexpected signal\n");
+ drbd_warn(resource, "Worker got an unexpected signal\n");
continue;
}
break;
@@ -2177,42 +4276,34 @@ int drbd_worker(struct drbd_thread *thi)
if (get_t_state(thi) != RUNNING)
break;
- if (!list_empty(&work_list)) {
+
+ while (!list_empty(&work_list)) {
w = list_first_entry(&work_list, struct drbd_work, list);
list_del_init(&w->list);
- update_worker_timing_details(connection, w->cb);
- if (w->cb(w, connection->cstate < C_WF_REPORT_PARAMS) == 0)
- continue;
- if (connection->cstate >= C_WF_REPORT_PARAMS)
- conn_request_state(connection, NS(conn, C_NETWORK_FAILURE), CS_HARD);
+ update_worker_timing_details(resource, w->cb);
+ w->cb(w, 0);
}
}
do {
- if (test_and_clear_bit(DEVICE_WORK_PENDING, &connection->flags)) {
- update_worker_timing_details(connection, do_unqueued_work);
- do_unqueued_work(connection);
+ if (test_and_clear_bit(DEVICE_WORK_PENDING, &resource->flags)) {
+ update_worker_timing_details(resource, do_unqueued_device_work);
+ do_unqueued_device_work(resource);
}
- if (!list_empty(&work_list)) {
+ if (test_and_clear_bit(PEER_DEVICE_WORK_PENDING, &resource->flags)) {
+ update_worker_timing_details(resource, do_unqueued_peer_device_work);
+ do_unqueued_peer_device_work(resource);
+ }
+ while (!list_empty(&work_list)) {
w = list_first_entry(&work_list, struct drbd_work, list);
list_del_init(&w->list);
- update_worker_timing_details(connection, w->cb);
+ update_worker_timing_details(resource, w->cb);
w->cb(w, 1);
- } else
- dequeue_work_batch(&connection->sender_work, &work_list);
- } while (!list_empty(&work_list) || test_bit(DEVICE_WORK_PENDING, &connection->flags));
-
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- D_ASSERT(device, device->state.disk == D_DISKLESS && device->state.conn == C_STANDALONE);
- kref_get(&device->kref);
- rcu_read_unlock();
- drbd_device_cleanup(device);
- kref_put(&device->kref, drbd_destroy_device);
- rcu_read_lock();
- }
- rcu_read_unlock();
+ }
+ dequeue_work_batch(&resource->work, &work_list);
+ } while (!list_empty(&work_list) ||
+ test_bit(DEVICE_WORK_PENDING, &resource->flags) ||
+ test_bit(PEER_DEVICE_WORK_PENDING, &resource->flags));
return 0;
}
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 10/20] drbd: replace per-device state model with multi-peer data structures
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (8 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 09/20] drbd: rework sender for DRBD 9 multi-peer Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 11/20] drbd: rewrite state machine for DRBD 9 multi-peer clusters Christoph Böhmwalder
` (9 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Joel Colledge,
Andreas Gruenbacher, Moritz Tanner, Roland Kammerer
Overhaul the internal header definitions to support DRBD 9's
multi-peer replication model.
The fundamental shift is that per-peer state (replication progress,
UUIDs, resync bookkeeping) moves from per-device to per-peer-device
scope, and all mutable state is now tracked as a [NOW]/[NEW] pair
on each object to support atomic, cluster-visible state transitions.
Redesign the locking model to match: remove the coarse per-resource
spinlock in favor of a resource-level rwlock for state, a
per-connection lock for peer request lists, and a per-device lock
for interval tree operations.
Replace direct socket members on the connection with the transport
abstraction.
Move the transfer log with its peer-ack machinery up to the resource
level so that writes can be serialized and acknowledged across all
peers consistently.
Move the state change API to a two-phase commit model at the
resource level, enabling cluster-wide coordinated transitions for
connect, disconnect, role change, and resize operations.
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Moritz Tanner <moritz.tanner@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_config.h | 45 +-
drivers/block/drbd/drbd_debugfs.h | 2 +
drivers/block/drbd/drbd_int.h | 3930 ++++++++++++++------
drivers/block/drbd/drbd_interval.h | 130 +-
drivers/block/drbd/drbd_polymorph_printk.h | 37 +-
drivers/block/drbd/drbd_req.h | 332 +-
drivers/block/drbd/drbd_state.h | 334 +-
drivers/block/drbd/drbd_state_change.h | 80 +-
drivers/block/drbd/drbd_strings.h | 4 +-
9 files changed, 3387 insertions(+), 1507 deletions(-)
diff --git a/drivers/block/drbd/drbd_config.h b/drivers/block/drbd/drbd_config.h
index d215365c6bb1..f61834fd2413 100644
--- a/drivers/block/drbd/drbd_config.h
+++ b/drivers/block/drbd/drbd_config.h
@@ -1,16 +1,47 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * drbd_config.h
- * DRBD's compile time configuration.
- */
+ drbd_config.h
+ DRBD's compile time configuration.
+*/
#ifndef DRBD_CONFIG_H
#define DRBD_CONFIG_H
-extern const char *drbd_buildtag(void);
+#include "drbd_protocol.h"
+
+const char *drbd_buildtag(void);
+
+#define REL_VERSION "9.3.5-rc.1"
+#define PRO_VERSION_MIN 118 /* 9.0.26 */
+#define PRO_VERSION_MAX 125
+
+#define PRO_FEATURES (DRBD_FF_TRIM | DRBD_FF_THIN_RESYNC | DRBD_FF_WSAME | DRBD_FF_WZEROES | \
+ DRBD_FF_RESYNC_DAGTAG | \
+ DRBD_FF_2PC_V2 | DRBD_FF_RS_SKIP_UUID | \
+ DRBD_FF_RESYNC_WITHOUT_REPLICATION | \
+ DRBD_FF_BM_BLOCK_SHIFT |\
+ DRBD_FF_RECONCILE_RECONNECT | DRBD_FF_WRITE_POSTPONE)
+
+#define PRO_VERSION_8_MIN 86
+#define PRO_VERSION_8_MAX 101
+
+/* We support two ranges of DRBD protocol version:
+ * 86-101: accepted DRBD 8 protocol versions as "rolling upgrade" path
+ * 102-109: never defined
+ * 110-117: _rejected_ because of bugs in the backward compat path
+ * in more recent DRBD versions. That is 9.0.0 to 9.0.25 inclusive.
+ * "Rolling" upgrade path for those versions:
+ * first upgrade to 9.0.latest, then connect to 9.1/9.2 or later.
+ * 118-PRO_VERSION_MAX: accepted DRBD 9 protocol versions.
+ * 125: P_UUIDS110 carries UUID_FLAG_BITMAP_AUTHORITATIVE, and standing
+ * out-of-sync bits at equal current UUIDs are reconciled by it.
+ *
+ * Note that we also reject connections with protocol version 121 and feature
+ * DRBD_FF_RESYNC_DAGTAG.
+ */
-#define REL_VERSION "8.4.11"
-#define PRO_VERSION_MIN 86
-#define PRO_VERSION_MAX 101
+/* Features supported in meta data format. */
+#define DRBD_MD_FEATURES (DRBD_MDFF_DIVERGENCE_BITMAP | DRBD_MDFF_BITMAP_AUTHORITATIVE | \
+ DRBD_MDFF_CBT)
#endif
diff --git a/drivers/block/drbd/drbd_debugfs.h b/drivers/block/drbd/drbd_debugfs.h
index e02960c6ec35..db4854750b90 100644
--- a/drivers/block/drbd/drbd_debugfs.h
+++ b/drivers/block/drbd/drbd_debugfs.h
@@ -15,6 +15,7 @@ void drbd_debugfs_cleanup(void);
void drbd_debugfs_resource_add(struct drbd_resource *resource);
void drbd_debugfs_resource_cleanup(struct drbd_resource *resource);
+void drbd_debugfs_resource_rename(struct drbd_resource *resource, const char *new_name);
void drbd_debugfs_connection_add(struct drbd_connection *connection);
void drbd_debugfs_connection_cleanup(struct drbd_connection *connection);
@@ -31,6 +32,7 @@ static inline void drbd_debugfs_cleanup(void) { }
static inline void drbd_debugfs_resource_add(struct drbd_resource *resource) { }
static inline void drbd_debugfs_resource_cleanup(struct drbd_resource *resource) { }
+static inline void drbd_debugfs_resource_rename(struct drbd_resource *resource, const char *new_name) { }
static inline void drbd_debugfs_connection_add(struct drbd_connection *connection) { }
static inline void drbd_debugfs_connection_cleanup(struct drbd_connection *connection) { }
diff --git a/drivers/block/drbd/drbd_int.h b/drivers/block/drbd/drbd_int.h
index 40be382b24d2..fb7553ab82a6 100644
--- a/drivers/block/drbd/drbd_int.h
+++ b/drivers/block/drbd/drbd_int.h
@@ -13,57 +13,127 @@
#include <linux/compiler.h>
#include <linux/types.h>
#include <linux/list.h>
+#include <linux/sched.h>
#include <linux/sched/signal.h>
#include <linux/bitops.h>
#include <linux/slab.h>
#include <linux/ratelimit.h>
-#include <linux/tcp.h>
#include <linux/mutex.h>
#include <linux/major.h>
#include <linux/blkdev.h>
#include <linux/backing-dev.h>
#include <linux/idr.h>
-#include <linux/dynamic_debug.h>
-#include <net/tcp.h>
#include <linux/lru_cache.h>
#include <linux/prefetch.h>
#include <linux/drbd.h>
+#include "drbd_nl_types.h"
#include "drbd_config.h"
-#include "drbd_nl_gen.h"
+
#include "drbd_strings.h"
#include "drbd_state.h"
+#include "drbd_state_change.h"
#include "drbd_protocol.h"
+#include "drbd_transport.h"
#include "drbd_polymorph_printk.h"
-extern struct genl_family drbd_nl_family;
+/*
+ * The version DRBD advertises in /proc/drbd and in the module's "version"
+ * modinfo field.
+ *
+ * Released drbd-utils 9.x do not probe the netlink family: they take the
+ * first token after "version:" in /proc/drbd (or, with the module not yet
+ * loaded, "modinfo -F version drbd"). For 8.4 they exec their bundled
+ * drbdadm-84/drbdsetup-84; for anything else they run the DRBD 9 tools,
+ * which exit with "API mismatch" unless "drbd" is at family version 2. A
+ * build serving the v1 family must therefore say 8.4. 8.4.11 is what the
+ * in-tree module has reported since 2018; the real release is still shown
+ * as "core:" in /proc/drbd, in the kernel log and in debugfs.
+ *
+ * Only where "drbd" is actually v1, which CONFIG_DRBD_COMPAT_84 means. The
+ * out-of-tree module can instead serve DRBD 9's own version 2 of the family
+ * next to the 8.4 metadata support; its Kbuild then defines
+ * DRBD_NL_FAMILY_V2. That is what LINBIT's packages ship, and an 8.4 token
+ * there would send drbd-utils 9.x to drbdsetup-84 against a v2 family.
+ */
+#if defined(CONFIG_DRBD_COMPAT_84) && !defined(DRBD_NL_FAMILY_V2)
+#define DRBD_PROC_VERSION "8.4.11"
+#else
+#define DRBD_PROC_VERSION REL_VERSION
+#endif
+
+/* module parameter, defined in drbd_main.c */
+extern unsigned int drbd_minor_count;
+extern unsigned int drbd_protocol_version_min;
+extern unsigned int drbd_max_parallel_resyncs;
+extern bool drbd_strict_names;
+
+static inline bool drbd_protocol_version_acceptable(unsigned int pv)
+{
+ return /* DRBD 9 */ (pv >= PRO_VERSION_MIN && pv <= PRO_VERSION_MAX) ||
+ /* DRBD 8 */ (pv >= PRO_VERSION_8_MIN && pv <= PRO_VERSION_8_MAX);
+}
-/* shared module parameters, defined in drbd_main.c */
#ifdef CONFIG_DRBD_FAULT_INJECTION
extern int drbd_enable_faults;
extern int drbd_fault_rate;
#endif
-extern unsigned int drbd_minor_count;
extern char drbd_usermode_helper[];
-extern int drbd_proc_details;
+enum {
+ /* drbd_khelper returns >= 0, we can use negative values as flags for drbd_maybe_khelper */
+ DRBD_UMH_DISABLED = INT_MIN,
+};
+#ifndef DRBD_MAJOR
+# define DRBD_MAJOR 147
+#endif
/* This is used to stop/restart our threads.
* Cannot use SIGTERM nor SIGKILL, since these
* are sent out by init on runlevel changes
* I choose SIGHUP for now.
+ *
+ * FIXME btw, we should register some reboot notifier.
*/
#define DRBD_SIGKILL SIGHUP
+/* For compatibility with protocol < 122 */
+#define ID_SKIP (4710ULL)
#define ID_IN_SYNC (4711ULL)
#define ID_OUT_OF_SYNC (4712ULL)
#define ID_SYNCER (-1ULL)
+static inline enum ov_result drbd_block_id_to_ov_result(u64 block_id)
+{
+ switch (block_id) {
+ case ID_IN_SYNC:
+ return OV_RESULT_IN_SYNC;
+ case ID_OUT_OF_SYNC:
+ return OV_RESULT_OUT_OF_SYNC;
+ default:
+ return OV_RESULT_SKIP;
+ }
+}
+
+static inline u64 drbd_ov_result_to_block_id(enum ov_result result)
+{
+ switch (result) {
+ case OV_RESULT_IN_SYNC:
+ return ID_IN_SYNC;
+ case OV_RESULT_OUT_OF_SYNC:
+ return ID_OUT_OF_SYNC;
+ default:
+ return ID_SKIP;
+ }
+}
+
#define UUID_NEW_BM_OFFSET ((u64)0x0001000000000000ULL)
struct drbd_device;
struct drbd_connection;
-struct drbd_peer_device;
+
+/* I want to be able to grep for "drbd $resource_name"
+ * and get all relevant log lines. */
/* Defines to control fault insertion */
enum {
@@ -77,11 +147,12 @@ enum {
DRBD_FAULT_BM_ALLOC = 7, /* bitmap allocation */
DRBD_FAULT_AL_EE = 8, /* alloc ee */
DRBD_FAULT_RECEIVE = 9, /* Changes some bytes upon receiving a [rs]data block */
+ DRBD_FAULT_BIO_TOO_SMALL = 10, /* Allocate smaller bios to trigger bio chaining */
DRBD_FAULT_MAX,
};
-extern unsigned int
+unsigned int
_drbd_insert_fault(struct drbd_device *device, unsigned int type);
static inline int
@@ -95,27 +166,31 @@ drbd_insert_fault(struct drbd_device *device, unsigned int type) {
#endif
}
-/* integer division, round _UP_ to the next integer */
-#define div_ceil(A, B) ((A)/(B) + ((A)%(B) ? 1 : 0))
-/* usual integer division */
-#define div_floor(A, B) ((A)/(B))
-
-extern struct ratelimit_state drbd_ratelimit_state;
-extern struct idr drbd_devices; /* RCU, updates: genl_lock() */
-extern struct list_head drbd_resources; /* RCU, updates: genl_lock() */
+/*
+ * our structs
+ *************************/
+extern struct idr drbd_devices; /* RCU, updates: drbd_devices_lock */
+extern struct list_head drbd_resources; /* RCU, updates: resources_mutex */
+extern struct mutex resources_mutex;
/* for sending/receiving the bitmap,
- * possibly in some encoding scheme */
+ * possibly in some encoding scheme.
+ * For compatibility, we transfer as if bm_block_size was 4k.
+ */
struct bm_xfer_ctx {
/* "const"
* stores total bits and long words
* of the bitmap, so we don't need to
* call the accessor functions over and again. */
+ unsigned long bm_bits_4k; /* unused on sending side */
unsigned long bm_bits;
unsigned long bm_words;
+ unsigned int scale; /* against BM_BLOCK_SHIFT_4k */
/* during xfer, current position within the bitmap */
unsigned long bit_offset;
+ /* receiving "partial" bits; unused on sending side. */
+ unsigned long bit_offset_4k;
unsigned long word_offset;
/* statistics; index: (h->command == P_BITMAP) */
@@ -123,8 +198,8 @@ struct bm_xfer_ctx {
unsigned bytes[2];
};
-extern void INFO_bm_xfer_stats(struct drbd_peer_device *peer_device,
- const char *direction, struct bm_xfer_ctx *c);
+void INFO_bm_xfer_stats(struct drbd_peer_device *peer_device,
+ const char *direction, struct bm_xfer_ctx *c);
static inline void bm_xfer_ctx_bit_to_word_offset(struct bm_xfer_ctx *c)
{
@@ -145,7 +220,7 @@ static inline void bm_xfer_ctx_bit_to_word_offset(struct bm_xfer_ctx *c)
#endif
}
-extern unsigned int drbd_header_size(struct drbd_connection *connection);
+unsigned int drbd_header_size(struct drbd_connection *connection);
/**********************************************************************/
enum drbd_thread_state {
@@ -160,7 +235,7 @@ struct drbd_thread {
struct task_struct *task;
struct completion stop;
enum drbd_thread_state t_state;
- int (*function) (struct drbd_thread *);
+ int (*function)(struct drbd_thread *thi);
struct drbd_resource *resource;
struct drbd_connection *connection;
int reset_cpu_mask;
@@ -179,31 +254,60 @@ static inline enum drbd_thread_state get_t_state(struct drbd_thread *thi)
struct drbd_work {
struct list_head list;
- int (*cb)(struct drbd_work *, int cancel);
+ int (*cb)(struct drbd_work *w, int cancel);
};
-struct drbd_device_work {
+struct drbd_peer_device_work {
struct drbd_work w;
- struct drbd_device *device;
+ struct drbd_peer_device *peer_device;
};
-#include "drbd_interval.h"
-
-extern int drbd_wait_misc(struct drbd_device *, struct drbd_interval *);
+enum drbd_stream;
-extern void lock_all_resources(void);
-extern void unlock_all_resources(void);
+#include "drbd_interval.h"
+void lock_all_resources(void);
+void unlock_all_resources(void);
+
+enum drbd_disk_state disk_state_from_md(struct drbd_device *device);
+bool want_bitmap(struct drbd_peer_device *peer_device);
+long twopc_timeout(struct drbd_resource *resource);
+long twopc_retry_timeout(struct drbd_resource *resource, int retries);
+u64 directly_connected_nodes(struct drbd_resource *resource,
+ enum which_state which);
+
+/* sequence arithmetic for dagtag (data generation tag) sector numbers.
+ * dagtag_newer_eq: true, if a is newer than b */
+#define dagtag_newer_eq(a, b) \
+ (typecheck(u64, a) && \
+ typecheck(u64, b) && \
+ ((s64)(a) - (s64)(b) >= 0))
+
+#define dagtag_newer(a, b) \
+ (typecheck(u64, a) && \
+ typecheck(u64, b) && \
+ ((s64)(a) - (s64)(b) > 0))
+
+/* An application I/O request.
+ *
+ * Fields marked as "immutable" may only be modified when the request is
+ * exclusively owned, e.g. when the request is created or is being retried.
+ */
struct drbd_request {
- struct drbd_work w;
+ /* "immutable" */
struct drbd_device *device;
/* if local IO is not allowed, will be NULL.
* if local IO _is_ allowed, holds the locally submitted bio clone,
* or, after local IO completion, the ERR_PTR(error).
- * see drbd_request_endio(). */
+ * see drbd_request_endio().
+ *
+ * Only accessed by app/submitter/endio - strictly sequential,
+ * no serialization required. */
struct bio *private_bio;
+ /* Fields sector and size are "immutable". Other fields protected
+ * by interval_lock. */
struct drbd_interval i;
/* epoch: used to check on "completion" whether this req was in
@@ -213,96 +317,152 @@ struct drbd_request {
* This corresponds to "barrier" in struct p_barrier[_ack],
* and to "barrier_nr" in struct drbd_epoch (and various
* comments/function parameters/local variable names).
+ *
+ * "immutable"
*/
unsigned int epoch;
- struct list_head tl_requests; /* ring list in the transfer log */
- struct bio *master_bio; /* master bio pointer */
+ /* Position of this request in the serialized per-resource change
+ * stream. Can be used to serialize with other events when
+ * communicating the change stream via multiple connections.
+ * Assigned from device->resource->dagtag_sector.
+ *
+ * Given that some IO backends write several GB per second meanwhile,
+ * lets just use a 64bit sequence space.
+ *
+ * "immutable"
+ */
+ u64 dagtag_sector;
+
+ /* list entry in transfer log (protected by RCU) */
+ struct list_head tl_requests;
+
+ /* list entry in submitter lists, peer ack list, or retry lists;
+ * protected by the locks for those lists */
+ struct list_head list;
+
+ /* master bio pointer; "immutable" */
+ struct bio *master_bio;
/* see struct drbd_device */
struct list_head req_pending_master_completion;
struct list_head req_pending_local;
- /* for generic IO accounting */
+ /* for generic IO accounting; "immutable" */
unsigned long start_jif;
- /* for DRBD internal statistics */
+ /* for request_timer_fn() */
+ unsigned long pre_submit_jif;
+ unsigned long pre_send_jif[DRBD_PEERS_MAX];
- /* Minimal set of time stamps to determine if we wait for activity log
- * transactions, local disk or peer. 32 bit "jiffies" are good enough,
- * we don't expect a DRBD request to be stalled for several month.
- */
+#ifdef CONFIG_DRBD_TIMING_STATS
+ /* for DRBD internal statistics */
+ ktime_t start_kt;
/* before actual request processing */
- unsigned long in_actlog_jif;
+ ktime_t in_actlog_kt;
/* local disk */
- unsigned long pre_submit_jif;
+ ktime_t pre_submit_kt;
/* per connection */
- unsigned long pre_send_jif;
- unsigned long acked_jif;
- unsigned long net_done_jif;
-
+ ktime_t pre_send_kt[DRBD_PEERS_MAX];
+ ktime_t acked_kt[DRBD_PEERS_MAX];
+ ktime_t net_done_kt[DRBD_PEERS_MAX];
+#endif
/* Possibly even more detail to track each phase:
- * master_completion_jif
+ * master_completion_kt
* how long did it take to complete the master bio
* (application visible latency)
- * allocated_jif
+ * allocated_kt
* how long the master bio was blocked until we finally allocated
* a tracking struct
- * in_actlog_jif
+ * in_actlog_kt
* how long did we wait for activity log transactions
*
- * net_queued_jif
+ * net_queued_kt
* when did we finally queue it for sending
- * pre_send_jif
+ * pre_send_kt
* when did we start sending it
- * post_send_jif
+ * post_send_kt
* how long did we block in the network stack trying to send it
- * acked_jif
+ * acked_kt
* when did we receive (or fake, in protocol A) a remote ACK
- * net_done_jif
+ * net_done_kt
* when did we receive final acknowledgement (P_BARRIER_ACK),
* or decide, e.g. on connection loss, that we do no longer expect
* anything from this peer for this request.
*
- * pre_submit_jif
- * post_sub_jif
+ * pre_submit_kt
+ * post_sub_kt
* when did we start submiting to the lower level device,
* and how long did we block in that submit function
- * local_completion_jif
+ * local_completion_kt
* how long did it take the lower level device to complete this request
*/
/* once it hits 0, we may complete the master_bio */
atomic_t completion_ref;
+ /* once it hits 0, we may remove the request from the interval tree and activity log */
+ refcount_t done_ref;
+ /* once it hits 0, we may remove from transfer log and send a corresponding peer ack */
+ refcount_t oos_send_ref;
/* once it hits 0, we may destroy this drbd_request object */
struct kref kref;
- unsigned rq_state; /* see comments above _req_mod() */
+ /* Creates a dependency chain between writes so that we know that a
+ * peer ack can be sent when done_ref reaches zero.
+ *
+ * If not NULL, when this drbd_request is done, one done_ref reference
+ * of ->done_next will be put.
+ *
+ * "immutable" */
+ struct drbd_request *next_write;
+
+ /* lock to protect state flags */
+ spinlock_t rq_lock;
+ unsigned int local_rq_state;
+ u16 net_rq_state[DRBD_NODE_ID_MAX];
+
+ /* for reclaim from transfer log */
+ struct rcu_head rcu;
+};
+
+/* Used to multicast peer acks. */
+struct drbd_peer_ack {
+ struct drbd_resource *resource;
+ struct list_head list;
+ /*
+ * Keeps track of which connections have not yet processed this peer
+ * ack. Peer acks are queued for connections on which they are not sent
+ * so that last_peer_ack_dagtag_seen is updated at the correct moment.
+ */
+ u64 queued_mask;
+ u64 pending_mask; /* Peer ack is sent to these nodes */
+ u64 mask; /* Nodes which successfully wrote the requests covered by this peer ack */
+ u64 dagtag_sector;
};
+/* Tracks received writes grouped in epochs. Protected by epoch_lock. */
struct drbd_epoch {
struct drbd_connection *connection;
+ struct drbd_peer_request *oldest_unconfirmed_peer_req;
struct list_head list;
unsigned int barrier_nr;
atomic_t epoch_size; /* increased on every request added. */
atomic_t active; /* increased on every req. added, and dec on every finished. */
+ atomic_t confirmed; /* adjusted for every P_CONFIRM_STABLE */
unsigned long flags;
};
/* drbd_epoch flag bits */
enum {
+ DE_BARRIER_IN_NEXT_EPOCH_ISSUED,
+ DE_BARRIER_IN_NEXT_EPOCH_DONE,
+ DE_CONTAINS_A_BARRIER,
DE_HAVE_BARRIER_NUMBER,
-};
-
-enum epoch_event {
- EV_PUT,
- EV_GOT_BARRIER_NR,
- EV_BECAME_LAST,
- EV_CLEANUP = 32, /* used as flag */
+ DE_IS_FINISHING,
};
struct digest_info {
@@ -313,23 +473,39 @@ struct digest_info {
struct drbd_peer_request {
struct drbd_work w;
struct drbd_peer_device *peer_device;
- struct drbd_epoch *epoch; /* for writes */
- struct page *pages;
- blk_opf_t opf;
+ struct list_head recv_order; /* see peer_requests, peer_reads, resync_requests */
+
+ union {
+ struct { /* read requests */
+ unsigned int depend_dagtag_node_id;
+ u64 depend_dagtag;
+ };
+ struct { /* resync target requests */
+ unsigned int requested_size;
+ };
+ struct { /* peer writes withheld from acknowledgment */
+ unsigned int wait_source_node_id;
+ };
+ };
+
+ struct bio_list bios;
atomic_t pending_bios;
struct drbd_interval i;
- /* see comments on ee flag bits below */
- unsigned long flags;
- unsigned long submit_jif;
+ unsigned long flags; /* see comments on ee flag bits below */
union {
- u64 block_id;
- struct digest_info *digest;
+ struct { /* regular peer_request */
+ struct drbd_epoch *epoch; /* for writes */
+ unsigned long submit_jif;
+ u64 block_id;
+ struct digest_info *digest;
+ u64 dagtag_sector;
+ };
+ struct { /* reused object for sending OOS to other nodes */
+ u64 send_oos_pending;
+ };
};
};
-/* Equivalent to bio_op and req_op. */
-#define peer_req_op(peer_req) \
- ((peer_req)->opf & REQ_OP_MASK)
/* ee flag bits.
* While corresponding bios are in flight, the only modification will be
@@ -338,9 +514,19 @@ struct drbd_peer_request {
* non-atomic modification to ee->flags is ok.
*/
enum {
- __EE_CALL_AL_COMPLETE_IO,
+ /* If successfully written,
+ * we may clear the corresponding out-of-sync bits */
__EE_MAY_SET_IN_SYNC,
+ /* Peer did not write this one, we must set-out-of-sync
+ * before actually submitting ourselves */
+ __EE_SET_OUT_OF_SYNC,
+
+ /* This peer request closes an epoch using a barrier.
+ * On successful completion, the epoch is released,
+ * and the P_BARRIER_ACK send. */
+ __EE_IS_BARRIER,
+
/* is this a TRIM aka REQ_OP_DISCARD? */
__EE_TRIM,
/* explicit zero-out requested, or
@@ -360,126 +546,384 @@ enum {
/* This ee has a pointer to a digest instead of a block id */
__EE_HAS_DIGEST,
- /* Conflicting local requests need to be restarted after this request */
- __EE_RESTART_REQUESTS,
-
/* The peer wants a write ACK for this (wire proto C) */
__EE_SEND_WRITE_ACK,
- /* Is set when net_conf had two_primaries set while creating this peer_req */
- __EE_IN_INTERVAL_TREE,
-
- /* for debugfs: */
- /* has this been submitted, or does it still wait for something else? */
- __EE_SUBMITTED,
-
- /* this is/was a write request */
- __EE_WRITE,
-
/* hand back using mempool_free(e, drbd_buffer_page_pool) */
__EE_RELEASE_TO_MEMPOOL,
/* this is/was a write same request */
__EE_WRITE_SAME,
- /* this originates from application on peer
- * (not some resync or verify or other DRBD internal request) */
- __EE_APPLICATION,
-
- /* If it contains only 0 bytes, send back P_RS_DEALLOCATED */
+ /* On target: Send P_RS_THIN_REQ.
+ * On source: If it contains only 0 bytes, send back P_RS_DEALLOCATED. */
__EE_RS_THIN_REQ,
+
+ /* Hold reference in activity log */
+ __EE_IN_ACTLOG,
+
+ /* SyncTarget: This is the last resync request. */
+ __EE_LAST_RESYNC_REQUEST,
+
+ /* This peer_req->recv_order is on some list protected by peer_reqs_lock */
+ __EE_ON_RECV_ORDER,
+
+ /* This peer_req->recv_order is on resource->send_oos protected by send_oos_lock */
+ __EE_ON_SEND_OOS,
+
+ /* A write on a sync target, not acknowledged until the sync source
+ * has it as well. See drbd_withhold_ack_until_source_has_write().
+ */
+ __EE_WAIT_FOR_SOURCE,
+
+ /* The sync source's answer overtook the local completion; recorded
+ * here under peer_reqs_lock, acted on by e_end_block().
+ */
+ __EE_SOURCE_REACHED,
+ __EE_SOURCE_UNREACHABLE,
+
+ /* The sync source can not get this write: answer the writer
+ * P_RETRY_WRITE instead of P_WRITE_ACK, so it retries the write.
+ */
+ __EE_POSTPONE,
+
+ /* The answer to the writer is decided; set under peer_reqs_lock by
+ * e_end_block() when it does not park the write. From here on the
+ * write can not be withheld or refused any more.
+ */
+ __EE_ACK_DECIDED,
+
+ /* SyncTarget: parked on the conflict submitter's list because the
+ * backing device already holds its share of resync work.
+ */
+ __EE_RS_DEPTH_PARKED,
};
-#define EE_CALL_AL_COMPLETE_IO (1<<__EE_CALL_AL_COMPLETE_IO)
#define EE_MAY_SET_IN_SYNC (1<<__EE_MAY_SET_IN_SYNC)
+#define EE_SET_OUT_OF_SYNC (1<<__EE_SET_OUT_OF_SYNC)
+#define EE_IS_BARRIER (1<<__EE_IS_BARRIER)
#define EE_TRIM (1<<__EE_TRIM)
#define EE_ZEROOUT (1<<__EE_ZEROOUT)
#define EE_RESUBMITTED (1<<__EE_RESUBMITTED)
#define EE_WAS_ERROR (1<<__EE_WAS_ERROR)
#define EE_HAS_DIGEST (1<<__EE_HAS_DIGEST)
-#define EE_RESTART_REQUESTS (1<<__EE_RESTART_REQUESTS)
#define EE_SEND_WRITE_ACK (1<<__EE_SEND_WRITE_ACK)
-#define EE_IN_INTERVAL_TREE (1<<__EE_IN_INTERVAL_TREE)
-#define EE_SUBMITTED (1<<__EE_SUBMITTED)
-#define EE_WRITE (1<<__EE_WRITE)
#define EE_RELEASE_TO_MEMPOOL (1<<__EE_RELEASE_TO_MEMPOOL)
#define EE_WRITE_SAME (1<<__EE_WRITE_SAME)
-#define EE_APPLICATION (1<<__EE_APPLICATION)
#define EE_RS_THIN_REQ (1<<__EE_RS_THIN_REQ)
+#define EE_IN_ACTLOG (1<<__EE_IN_ACTLOG)
+#define EE_LAST_RESYNC_REQUEST (1<<__EE_LAST_RESYNC_REQUEST)
+#define EE_RS_DEPTH_PARKED (1<<__EE_RS_DEPTH_PARKED)
+#define EE_ON_RECV_ORDER (1<<__EE_ON_RECV_ORDER)
+#define EE_ON_SEND_OOS (1<<__EE_ON_SEND_OOS)
+#define EE_WAIT_FOR_SOURCE (1<<__EE_WAIT_FOR_SOURCE)
+#define EE_SOURCE_REACHED (1<<__EE_SOURCE_REACHED)
+#define EE_SOURCE_UNREACHABLE (1<<__EE_SOURCE_UNREACHABLE)
+#define EE_POSTPONE (1<<__EE_POSTPONE)
+#define EE_ACK_DECIDED (1<<__EE_ACK_DECIDED)
+
+#define REQ_NO_BIO (REQ_OP_DRV_OUT) /* exception for drbd_alloc_peer_request(), DRBD private */
/* flag bits per device */
-enum {
- UNPLUG_REMOTE, /* sending a "UnplugRemote" could help */
+enum device_flag {
MD_DIRTY, /* current uuids and flags not yet on disk */
- USE_DEGR_WFC_T, /* degr-wfc-timeout instead of wfc-timeout. */
- CL_ST_CHG_SUCCESS,
- CL_ST_CHG_FAIL,
CRASHED_PRIMARY, /* This node was a crashed primary.
* Gets cleared when the state.conn
- * goes into C_CONNECTED state. */
- CONSIDER_RESYNC,
-
- MD_NO_FUA, /* Users wants us to not use FUA/FLUSH on meta data dev */
-
- BITMAP_IO, /* suspend application io;
- once no more io in flight, start bitmap io */
- BITMAP_IO_QUEUED, /* Started bitmap IO */
- WAS_IO_ERROR, /* Local disk failed, returned IO error */
- WAS_READ_ERROR, /* Local disk READ failed (set additionally to the above) */
+ * goes into L_ESTABLISHED state. */
+ MD_NO_FUA, /* meta data device does not support barriers,
+ so don't even try */
FORCE_DETACH, /* Force-detach from local disk, aborting any pending local IO */
- RESYNC_AFTER_NEG, /* Resync after online grow after the attach&negotiate finished. */
- RESIZE_PENDING, /* Size change detected locally, waiting for the response from
- * the peer, if it changed there as well. */
- NEW_CUR_UUID, /* Create new current UUID when thawing IO */
+ ABORT_MDIO, /* Interrupt ongoing meta-data I/O */
AL_SUSPENDED, /* Activity logging is currently suspended. */
- AHEAD_TO_SYNC_SOURCE, /* Ahead -> SyncSource queued */
- B_RS_H_DONE, /* Before resync handler done (already executed) */
- DISCARD_MY_DATA, /* discard_my_data flag per volume */
- READ_BALANCE_RR,
-
+ AL_ENABLE_PENDING, /* AL updates on, the in-core AL not yet on disk */
+ UNREGISTERED,
FLUSH_PENDING, /* if set, device->flush_jif is when we submitted that flush
* from drbd_flush_after_epoch() */
/* cleared only after backing device related structures have been destroyed. */
- GOING_DISKLESS, /* Disk is being detached, because of io-error, or admin request. */
+ GOING_DISKLESS, /* Disk is being detached, because of io-error, or admin request. */
/* to be used in drbd_device_post_work() */
- GO_DISKLESS, /* tell worker to schedule cleanup before detach */
- DESTROY_DISK, /* tell worker to close backing devices and destroy related structures. */
+ GO_DISKLESS, /* tell worker to schedule cleanup before detach */
MD_SYNC, /* tell worker to call drbd_md_sync() */
+ MAKE_NEW_CUR_UUID, /* tell worker to ping peers and eventually write new current uuid */
+ AUTO_GROW, /* tell worker to ask the cluster to grow into a bigger backend */
+
+ STABLE_RESYNC, /* One peer_device finished the resync stable! */
+ READ_BALANCE_RR,
+ PRIMARY_LOST_QUORUM,
+ TIEBREAKER_QUORUM, /* Tiebreaker keeps quorum; used to avoid too verbose logging */
+ DESTROYING_DEV,
+ TRY_TO_GET_RESYNC,
+ OUTDATE_ON_2PC_COMMIT,
+ RESTORE_QUORUM, /* Restore quorum when we have the same members as before */
+ RESTORING_QUORUM, /* sanitize_state() -> finish_state_change() */
+ LEGACY_84_MD,
+ BDEV_FROZEN, /* called bdev_freeze(), needs bdev_thaw() on resume-io */
+ CBT_BITMAPS_FLUSHED, /* cbt-flush wrote the bitmap during this suspend-io */
+ RS_DEPTH_WAITERS, /* a peer device has RS_DEPTH_DEFERRED set */
+ TWOPC_IO_SUSPENDED, /* application I/O suspended until the running 2PC ends */
+};
+
+/* The data-generation obligation of one volume. A divergence-start event
+ * (a peer's data lost, own disk failed, promotion, ...) obliges this volume
+ * to start a new data generation before it may admit further writes. The
+ * state, the auxiliary bits and the reason set are packed into one word,
+ * device->gen_obligation, so that a reader gets a consistent snapshot with a
+ * single READ_ONCE().
+ */
+enum drbd_gen_obl_state {
+ GEN_OBL_NONE, /* no obligation, writes admitted */
+ GEN_OBL_ARMED, /* obligation outstanding, writes not admitted */
+ GEN_OBL_MINTING, /* the new generation is being made right now */
+ GEN_OBL_UNCONFIRMED, /* diskless: the new generation is exposed and
+ * queued in order on every established channel,
+ * but no peer has confirmed durable receipt.
+ * Writes are admitted, their completion held.
+ */
+ GEN_OBL_DISCHARGED, /* met: new generation persisted or confirmed */
+ GEN_OBL_PARKED, /* retained while writers fail fast (io-error
+ * policy): the data set does not change, so no
+ * generation is owed for that span. Write
+ * admission never sees this state -- whenever
+ * it is set, device->cached_err_io is set as
+ * well, and drbd_submit_bio() errors the write
+ * long before inc_ap_bio_cond().
+ */
+};
+
+/* The reason set: one bit per class of divergence-start event. Several may
+ * accumulate before the obligation is met.
+ */
+#define GEN_OBL_PEER_DATA_LOST (1 << 0)
+#define GEN_OBL_PEER_RETURNED_DISKLESS (1 << 1)
+#define GEN_OBL_PEER_DISK_FAILED (1 << 2)
+#define GEN_OBL_OWN_DISK_FAILED (1 << 3)
+#define GEN_OBL_PROMOTED (1 << 4)
+#define GEN_OBL_AHEAD (1 << 5)
+#define GEN_OBL_PRE_110 (1 << 6)
+#define GEN_OBL_COMPLETION_DECIDED (1 << 7)
+#define GEN_OBL_REASON_COUNT 8
+
+/* Layout of device->gen_obligation. */
+#define GEN_OBL_STATE_MASK 0x0000000fu
+#define GEN_OBL_MATERIALIZED 0x00000010u /* a completion decision was taken
+ * although acks of a lost replica
+ * are missing
+ */
+#define GEN_OBL_REARM_PENDING 0x00000020u /* armed again while the mint of an
+ * earlier obligation was running
+ */
+#define GEN_OBL_REASON_SHIFT 16
+#define GEN_OBL_REASON_MASK 0xffff0000u
+
+/* A set of states, for the "allowed from" argument of a transition. */
+#define GEN_OBL_IN(state) (1u << (state))
+
+/* The states an obligation can materialize out of: it is recorded, and its
+ * mint has not started. PARKED is one of them -- a completion decision is
+ * about writes that already happened, so the io-error policy does not hold it
+ * back.
+ */
+#define GEN_OBL_MATERIALIZE_FROM (GEN_OBL_IN(GEN_OBL_ARMED) | GEN_OBL_IN(GEN_OBL_PARKED))
+
+/* Enough for any state name plus the whole reason set. */
+#define GEN_OBL_STR_MAX 160
+
+/* What a mint attempt did. The site that holds the obligation decides from
+ * this whether the obligation is met.
+ */
+enum drbd_mint_outcome {
+ MINT_MINTED, /* a new generation exists, persisted to stable
+ * storage before anyone else can see it
+ */
+ MINT_EXPOSED, /* diskless: the new generation is exposed and
+ * queued in order on every established channel,
+ * but no peer has confirmed durable receipt.
+ * The obligation is not owed any more; it is met
+ * once the confirm arrives.
+ */
+ MINT_UNNECESSARY, /* nothing owed: every absent peer already sees a
+ * current UUID of ours it does not have
+ */
+ MINT_DEFERRED, /* not now: the exposed generation of a diskless
+ * primary is still unconfirmed
+ */
+ MINT_FAILED, /* the metadata write failed; the new current UUID
+ * was rolled back
+ */
+ MINT_NOT_EVALUATED, /* the mint did not run: the quorum and data gate
+ * of the ping round declined
+ */
+};
+
+/* The new data generation did not happen, so whoever holds the obligation
+ * still owes it.
+ */
+static inline bool drbd_mint_still_owed(enum drbd_mint_outcome outcome)
+{
+ return outcome == MINT_DEFERRED || outcome == MINT_FAILED ||
+ outcome == MINT_NOT_EVALUATED;
+}
+
+/* flag bits per peer device */
+enum peer_device_flag {
+ CONSIDER_RESYNC,
+ RESYNC_AFTER_NEG, /* Resync after online grow after the attach&negotiate finished. */
+ RESIZE_PENDING, /* Size change detected locally, waiting for the response from
+ * the peer, if it changed there as well. */
RS_START, /* tell worker to start resync/OV */
RS_PROGRESS, /* tell worker that resync made significant progress */
+ RS_LAZY_BM_WRITE, /* -"- and bitmap writeout should be efficient now */
RS_DONE, /* tell worker that resync is done */
- UNREGISTERED, /* minor removed, device may still be referenced */
+ B_RS_H_DONE, /* Before resync handler done (already executed) */
+ DISCARD_MY_DATA, /* discard_my_data flag per volume */
+ USE_DEGR_WFC_T, /* degr-wfc-timeout instead of wfc-timeout. */
+ SEND_RECONCILE_UUID, /* worker: the reconcile peer has settled UpToDate;
+ * send it our current UUID (the relabel), ordered
+ * by the sender after the replayed transfer log.
+ * Worker-dispatched (see DRBD_PEER_DEVICE_WORK_MASK);
+ * must stay below 32 so get_work_bits() finds it in
+ * flags[0] on 32-bit kernels.
+ */
+ INITIAL_STATE_SENT,
+ INITIAL_STATE_RECEIVED,
+ RECONCILIATION_RESYNC,
+ RECONCILE_PENDING, /* post-loss reconcile owed, not yet armed;
+ * bridges loss -> RECONCILIATION_RESYNC
+ */
+ BITMAP_HELD_AT_PRIMARY_LOSS, /* had bits before the last lost primary */
+ UNSTABLE_RESYNC, /* Sync source went unstable during resync. */
+ PEER_UPGRADE_PREDICTED, /* Recorded the peer UpToDate before its P_STATE */
+ SEND_UUIDS_PEER_ATTACH, /* Peer attached: receiver or worker, whichever is
+ * first, sends P_UUIDS and P_STATE before we decide
+ */
+ SEND_STATE_AFTER_AHEAD,
+ GOT_NEG_ACK, /* got a neg_ack while primary, wait until peer_disk is lower than
+ D_UP_TO_DATE before becoming secondary! */
+ AHEAD_TO_SYNC_SOURCE, /* Ahead -> SyncSource queued */
+ SYNC_TARGET_TO_BEHIND, /* SyncTarget, wait for Behind */
+ HANDLING_CONGESTION, /* Set while testing for congestion and handling it */
+ HANDLE_CONGESTION, /* tell worker to change state due to congestion */
+ HOLDING_UUID_READ_LOCK, /* did a down_read(&device->uuid_sem) */
+ RS_SOURCE_MISSED_END, /* SyncSource did not got P_UUIDS110 */
+ RS_PEER_MISSED_END, /* Peer (which was SyncSource) did not got P_UUIDS110 after resync */
+ SYNC_SRC_CRASHED_PRI, /* Source of this resync was a crashed primary */
+ HAVE_SIZES, /* Cleared when connection gets lost; set when sizes received */
+ UUIDS_RECEIVED, /* Have recent UUIDs from the peer */
+ CURRENT_UUID_RECEIVED, /* Got a p_current_uuid packet */
+ PEER_QUORATE, /* Peer has quorum */
+ RS_REQUEST_UNSUCCESSFUL, /* Some resync request was unsuccessful in current cycle */
+ REPLICATION_NEXT, /* If unset, do not replicate writes when next Inconsistent */
+ PEER_REPLICATION_NEXT, /* We have instructed peer not to replicate writes */
+ CURRENT_UUID_UNCONFIRMED, /* Diskless primary optimistically advanced this
+ * peer's current_uuid (sent the new UUID, assumed
+ * it took). Until the peer confirms, the handshake
+ * trusts the peer's reported UUID and the rotated
+ * gen is not yet confirmed durable on this peer.
+ * Cleared only on the peer's own data evidence by
+ * drbd_peer_maybe_confirm_rotated_gen(): every held
+ * write of the gen durable on the peer, or, none
+ * held, a barrier ack at/past the gen's epoch.
+ */
+ RECONCILE_INJECT_CUR_UUID, /* This peer returned on our predecessor
+ * generation. We assert it UpToDate anyway
+ * (it holds a complete generation and the
+ * bridging writes are still replayable) and arm
+ * this so that, once it settles UpToDate, we
+ * relabel it forward to our current generation.
+ * See diskless_with_peers_different_current_uuids().
+ */
+ RS_MXB_LIMITED_LOGGED, /* the max-buffers hint has been logged for this
+ * resync and this net configuration.
+ */
+ RS_DEPTH_DEFERRED, /* a checksum resync round stopped at the resync
+ * depth bound; restart it when there is room
+ */
+ PEER_DEVICE_FLAG_COUNT, /* keep last: sizes peer_device->flags[] */
};
-struct drbd_bitmap; /* opaque for drbd_device */
+/* We could make these currently hardcoded constants configurable
+ * variables at create-md time (or even re-configurable at runtime?).
+ * Which will require some more changes to the DRBD "super block"
+ * and attach code.
+ *
+ * updates per transaction:
+ * This many changes to the active set can be logged with one transaction.
+ * This number is arbitrary.
+ * context per transaction:
+ * This many context extent numbers are logged with each transaction.
+ * This number is resulting from the transaction block size (4k), the layout
+ * of the transaction header, and the number of updates per transaction.
+ * See drbd_actlog.c:struct al_transaction_on_disk
+ * */
+#define AL_UPDATES_PER_TRANSACTION 64 // arbitrary
+#define AL_CONTEXT_PER_TRANSACTION 919 // (4096 - 36 - 6*64)/4
/* definition of bits in bm_flags to be used in drbd_bm_lock
* and drbd_bitmap_io and friends. */
enum bm_flag {
- /* currently locked for bulk operation */
- BM_LOCKED_MASK = 0xf,
-
- /* in detail, that is: */
- BM_DONT_CLEAR = 0x1,
- BM_DONT_SET = 0x2,
- BM_DONT_TEST = 0x4,
+ /*
+ * The bitmap can be locked to prevent others from clearing, setting,
+ * and/or testing bits. The following combinations of lock flags make
+ * sense:
+ *
+ * BM_LOCK_CLEAR,
+ * BM_LOCK_SET, | BM_LOCK_CLEAR,
+ * BM_LOCK_TEST | BM_LOCK_SET | BM_LOCK_CLEAR.
+ */
- /* so we can mark it locked for bulk operation,
- * and still allow all non-bulk operations */
- BM_IS_LOCKED = 0x8,
+ BM_LOCK_TEST = 0x1,
+ BM_LOCK_SET = 0x2,
+ BM_LOCK_CLEAR = 0x4,
+ BM_LOCK_BULK = 0x8, /* locked for bulk operation, allow all non-bulk operations */
- /* (test bit, count bit) allowed (common case) */
- BM_LOCKED_TEST_ALLOWED = BM_DONT_CLEAR | BM_DONT_SET | BM_IS_LOCKED,
+ BM_LOCK_ALL = BM_LOCK_TEST | BM_LOCK_SET | BM_LOCK_CLEAR | BM_LOCK_BULK,
- /* testing bits, as well as setting new bits allowed, but clearing bits
- * would be unexpected. Used during bitmap receive. Setting new bits
- * requires sending of "out-of-sync" information, though. */
- BM_LOCKED_SET_ALLOWED = BM_DONT_CLEAR | BM_IS_LOCKED,
+ BM_LOCK_SINGLE_SLOT = 0x10,
+ BM_ON_DAX_PMEM = 0x10000,
+};
- /* for drbd_bm_write_copy_pages, everything is allowed,
- * only concurrent bulk operations are locked out. */
- BM_LOCKED_CHANGE_ALLOWED = BM_IS_LOCKED,
+struct drbd_bitmap {
+ union {
+ struct page **bm_pages;
+ void *bm_on_pmem;
+ };
+ spinlock_t bm_lock; /* fine-grain lock (TODO: per slot) */
+ spinlock_t bm_all_slots_lock; /* all bitmap slots lock */
+
+ unsigned long bm_set[DRBD_PEERS_MAX]; /* number of bits set */
+ unsigned long bm_bits; /* bits per peer */
+ unsigned long bm_bits_4k; /* bits per peer, if we had bm_block_size of 4k */
+ size_t bm_words; /* platform specitif word size; not 32bit!! */
+ size_t bm_number_of_pages;
+ sector_t bm_dev_capacity;
+ struct mutex bm_change; /* serializes resize operations */
+
+ wait_queue_head_t bm_io_wait; /* used to serialize IO of single pages */
+
+ enum bm_flag bm_flags;
+ unsigned int bm_max_peers;
+ unsigned int bm_block_shift; /* ln2 of bytes per bit for this bitmap */
+
+ /* exclusively to be used by __al_write_transaction(),
+ * and drbd_bm_write_hinted() -> bm_rw() called from there.
+ * One activity log extent represents 4MB of storage, which are 1024
+ * bits (at 4k per bit), times at most DRBD_PEERS_MAX (currently 32).
+ * The bitmap is created interleaved, with a potentially odd number
+ * of peer slots determined at create-md time. Which means that one
+ * AL-extent may be associated with one or two bitmap pages.
+ */
+ unsigned int n_bitmap_hints;
+ unsigned int al_bitmap_hints[2*AL_UPDATES_PER_TRANSACTION];
+
+ /* debugging aid, in case we are still racy somewhere */
+ const char *bm_why;
+ char bm_task_comm[TASK_COMM_LEN];
+ pid_t bm_task_pid;
+ /* >= 0 if a slot-scoped lock is held (the locked slot's index);
+ * -1 if no lock is held or the lock is device-wide.
+ */
+ int bm_locked_slot_index;
};
struct drbd_work_queue {
@@ -488,29 +932,81 @@ struct drbd_work_queue {
wait_queue_head_t q_wait;
};
-struct drbd_socket {
- struct mutex mutex;
- struct socket *socket;
- /* this way we get our
- * send/receive buffers off the stack */
- void *sbuf;
- void *rbuf;
+/*
+ * Per-peer metadata slot. bitmap_uuid names the generation this slot's bitmap
+ * records divergence from (0 when unused); bitmap_dagtag is the dagtag at which
+ * it was set. A slot is in one of three states:
+ *
+ * - allocated bitmap (MDF_HAVE_BITMAP, bitmap_index >= 0): owns an on-disk
+ * bitmap tracking per-block divergence toward the peer.
+ * - diskless peer (MDF_NODE_EXISTS, bitmap_index == -1): the peer is known
+ * but we hold no bitmap of our own for it.
+ * - unallocated day0 slot (flags == 0, bitmap_index == -1): carries the day0
+ * UUID -- the generation the volume was created in -- as a common-ancestor
+ * reference for inception-based resync. Its bitmap is associated with it
+ * indirectly (a spare bitmap index, or implicitly all-set), never owned
+ * directly, so it always records full divergence.
+ *
+ * (The same notion of day0 as drbd_uuid_is_day0(), seen differently: there, the
+ * local current UUID is still day0; here, a bitmap tracks divergence since day0.)
+ */
+struct drbd_peer_md {
+ u64 bitmap_uuid;
+ u64 bitmap_dagtag;
+ unsigned long flags; /* enum mdf_peer_flag_bit, atomic bit operations */
+ s32 bitmap_index;
+ /* Nodes whose resync set placeholder bits in this slot; those bits are
+ * retired by a P_PEERS_IN_SYNC from one of them. Not in the meta data:
+ * after a restart the slot falls back to asking whether any other peer
+ * is connected, see drbd_bitmap_slot_decides().
+ */
+ u64 placeholder_src;
+ /* Whether placeholder_src names every bit standing in this slot. False
+ * for bits that were already in the meta data when the disk was
+ * attached, or that a slot inherited when it was allocated; their
+ * origin is not recorded there.
+ */
+ bool placeholder_src_complete;
+};
+
+/* Changed block tracking: a bitmap slot that no peer owns records every write
+ * to the backing device since the bookmark "id". flags are CBT_BOOKMARK_*.
+ */
+struct drbd_cbt_md {
+ u64 id;
+ u64 bitmap_uuid;
+ u64 bitmap_dagtag;
+ s64 since;
+ u32 flags;
+ s32 bitmap_index;
};
struct drbd_md {
u64 md_offset; /* sector offset to 'super' block */
- u64 la_size_sect; /* last agreed size, unit sectors */
+ u64 effective_size; /* last agreed size (sectors) */
+ u64 prev_members; /* read from the meta-data */
+ u64 members; /* current member mask for writing meta-data */
+ u64 prev_features; /* read from the meta-data; DRBD_MD_FEATURES is written */
spinlock_t uuid_lock;
- u64 uuid[UI_SIZE];
+ u64 current_uuid;
u64 device_uuid;
u32 flags;
+ s32 node_id;
u32 md_size_sect;
s32 al_offset; /* signed relative sector offset to activity log */
s32 bm_offset; /* signed relative sector offset to bitmap */
- /* cached value of bdev->disk_conf->meta_dev_idx (see below) */
+ u32 max_peers;
+ u32 bm_block_size;
+ u32 bm_block_shift; /* ilog2(bm_block_size) */
+
+ struct drbd_peer_md peers[DRBD_NODE_ID_MAX];
+ u64 history_uuids[HISTORY_UUIDS];
+ struct drbd_cbt_md cbt;
+
+ /* cached value of bdev->disk_conf->meta_dev_idx */
s32 meta_dev_idx;
/* see al_tr_number_to_on_disk_sector() */
@@ -525,8 +1021,13 @@ struct drbd_backing_dev {
struct block_device *md_bdev;
struct file *f_md_bdev;
struct drbd_md md;
- struct disk_conf *disk_conf; /* RCU, for updates: resource->conf_update */
+ struct drbd_disk_conf __rcu *disk_conf; /* RCU, for updates: resource->conf_update */
sector_t known_size; /* last known size of that backing device */
+#if IS_ENABLED(CONFIG_DEV_DAX_PMEM)
+ struct dax_device *dax_dev;
+ struct meta_data_on_disk_9 *md_on_pmem; /* address of md_offset */
+ struct al_on_pmem *al_on_pmem;
+#endif
};
struct drbd_md_io {
@@ -541,47 +1042,159 @@ struct drbd_md_io {
struct bm_io_work {
struct drbd_work w;
+ struct drbd_device *device;
struct drbd_peer_device *peer_device;
char *why;
enum bm_flag flags;
- int (*io_fn)(struct drbd_device *device, struct drbd_peer_device *peer_device);
- void (*done)(struct drbd_device *device, int rv);
+ int (*io_fn)(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+ void (*done)(struct drbd_device *device,
+ struct drbd_peer_device *peer_device,
+ int rv);
};
struct fifo_buffer {
+ /* singly linked list to accumulate multiple such struct fifo_buffers,
+ * to be freed after a single syncronize_rcu(),
+ * outside a critical section. */
+ struct fifo_buffer *next;
unsigned int head_index;
unsigned int size;
int total; /* sum of all values */
int values[] __counted_by(size);
};
-extern struct fifo_buffer *fifo_alloc(unsigned int fifo_size);
+struct fifo_buffer *fifo_alloc(unsigned int fifo_size);
/* flag bits per connection */
-enum {
- NET_CONGESTED, /* The data socket is congested */
- RESOLVE_CONFLICTS, /* Set on one node, cleared on the peer! */
- SEND_PING,
- GOT_PING_ACK, /* set when we receive a ping_ack packet, ping_wait gets woken */
- CONN_WD_ST_CHG_REQ, /* A cluster wide state change on the connection is active */
- CONN_WD_ST_CHG_OKAY,
- CONN_WD_ST_CHG_FAIL,
+enum connection_flag {
+ PING_PENDING, /* cleared upon receiveing a ping_ack packet, wakes state_wait */
+ TWOPC_PREPARED,
+ TWOPC_YES,
+ TWOPC_NO,
+ TWOPC_RETRY,
CONN_DRY_RUN, /* Expect disconnect after resync handshake. */
- CREATE_BARRIER, /* next P_DATA is preceded by a P_BARRIER */
- STATE_SENT, /* Do not change state/UUIDs while this is set */
+ CONN_DRY_RUN_LOCAL, /* This node asked for the dry run. */
+ DISCONNECT_EXPECTED,
+ CORKED,
+ DATA_CORKED = CORKED, /* used as computed value CORKED + DATA_STREAM */
+ CONTROL_CORKED, /* used as computed value CORKED + CONTROL_STREAM */
+ C_UNREGISTERED,
+ RECONNECT,
+ CONN_DISCARD_MY_DATA,
+ SEND_STATE_AFTER_AHEAD_C,
+ NOTIFY_PEERS_LOST_PRIMARY,
+ CHECKING_PEER, /* used by make_new_urrent_uuid() to check liveliness */
+ CONN_CONGESTED,
+ CONN_HANDSHAKE_DISCONNECT,
+ CONN_HANDSHAKE_RETRY,
+ CONN_HANDSHAKE_READY,
+ CONN_HANDSHAKE_DEMOTE, /* call-pri-lost-after-sb: give up Primary */
+ RECEIVED_DAGTAG, /* Whether we received any write or dagtag since connecting. */
+ DAGTAG_STREAM_GONE, /* This peer's write stream is being torn down: the
+ * requests waiting for a position in it have been,
+ * or are about to be, resolved. Set before the
+ * teardown walks the wait lists, so a waiter that
+ * tests it under the list's own lock never queues
+ * behind that walk.
+ */
+ PING_TIMEOUT_ACTIVE,
+ CONN_PRE_110, /* agreed on a protocol below 110: the only connection */
+};
+
+/* flag bits per resource */
+enum resource_flag {
+ EXPLICIT_PRIMARY,
CALLBACK_PENDING, /* Whether we have a call_usermodehelper(, UMH_WAIT_PROC)
* pending, from drbd worker context.
*/
- DISCONNECT_SENT,
+ TWOPC_ABORT_LOCAL,
+ TWOPC_WORK_PENDING, /* Set while work for sending reply is scheduled */
+ TWOPC_EXECUTED, /* Commited or aborted */
+ TWOPC_STATE_CHANGE_PENDING, /* set between sending commit and changing local state */
+ TWOPC_PREPARING, /* participant, prepare not yet forwarded: no reply may go out */
+
+ TRY_BECOME_UP_TO_DATE_PENDING,
DEVICE_WORK_PENDING, /* tell worker that some device has pending work */
- C_UNREGISTERED, /* removed from the resource, may still be referenced */
+ PEER_DEVICE_WORK_PENDING,/* tell worker that some peer_device has pending work */
+
+ /* to be used in drbd_post_work() */
+ R_UNREGISTERED,
+ DOWN_IN_PROGRESS,
+ CHECKING_PEERS,
+ WRONG_MDF_EXISTS, /* Warned about MDF_EXISTS flag on all peer slots */
+ TWOPC_RECV_SIZES_ERR, /* Error processing sizes packet during 2PC connect */
};
enum which_state { NOW, OLD = NOW, NEW };
-/* flag bits per resource */
-enum {
- R_UNREGISTERED, /* removed from the resources list, may still be referenced */
+enum twopc_type {
+ TWOPC_STATE_CHANGE,
+ TWOPC_RESIZE,
+};
+
+struct twopc_reply {
+ int vnr;
+ unsigned int tid; /* transaction identifier */
+ int initiator_node_id; /* initiator of the transaction */
+ int target_node_id; /* target of the transaction (or -1) */
+ u64 target_reachable_nodes; /* behind the target node */
+ u64 reachable_nodes; /* behind other nodes */
+ union {
+ struct { /* type == TWOPC_STATE_CHANGE */
+ u64 primary_nodes;
+ u64 weak_nodes;
+ };
+ struct { /* type == TWOPC_RESIZE */
+ u64 diskful_primary_nodes;
+ u64 max_possible_size;
+ /* reachable_nodes is the union over what the
+ * participants report, this the intersection.
+ */
+ u64 common_reachable_nodes;
+ };
+ };
+ unsigned int is_disconnect:1;
+ unsigned int is_connect:1;
+ unsigned int is_aborted:1;
+ /* Whether the state change on receiving the twopc failed. When this is
+ * a twopc for transitioning to C_CONNECTED, we cannot immediately
+ * reply with P_TWOPC_NO. The state handshake must complete first to
+ * decide the appropriate reply. */
+ unsigned int state_change_failed:1;
+};
+
+struct twopc_request {
+ u64 nodes_to_reach;
+ enum drbd_packet cmd;
+ unsigned int tid;
+ int initiator_node_id;
+ int target_node_id;
+ int vnr;
+ u32 flags;
+};
+
+struct drbd_thread_timing_details {
+ unsigned long start_jif;
+ void *cb_addr;
+ const char *caller_fn;
+ unsigned int line;
+ unsigned int cb_nr;
+};
+#define DRBD_THREAD_DETAILS_HIST 16
+
+struct drbd_send_buffer {
+ struct page *page; /* current buffer page for sending data */
+ char *unsent; /* start of unsent area != pos if corked... */
+ char *pos; /* position within that page */
+ int allocated_size; /* currently allocated space */
+ int additional_size; /* additional space to be added to next packet's size */
+ bool dead; /* a send failed: this stream carries a truncated packet */
+};
+
+struct drbd_mutable_buffer {
+ u8 *buffer;
+ unsigned int avail;
};
enum drbd_per_resource_ratelimit {
@@ -596,36 +1209,169 @@ struct drbd_resource {
struct dentry *debugfs_res_volumes;
struct dentry *debugfs_res_connections;
struct dentry *debugfs_res_in_flight_summary;
+ struct dentry *debugfs_res_state_twopc;
+ struct dentry *debugfs_res_worker_pid;
+ struct dentry *debugfs_res_members;
#endif
struct kref kref;
- struct idr devices; /* volume number to device mapping */
+
+ /* Volume number to device mapping. Updates protected by conf_update. */
+ struct idr devices;
struct ratelimit_state ratelimit[1];
- unsigned long flags;
+ /* RCU list. Updates protected by adm_mutex, conf_update and state_rwlock. */
struct list_head connections;
- struct list_head resources;
- struct res_opts res_opts;
- struct mutex conf_update; /* mutex for ready-copy-update of net_conf and disk_conf */
+
+ struct list_head resources; /* list entry in global resources list */
+ struct drbd_res_opts res_opts;
+ int max_node_id;
+#ifdef CONFIG_DRBD_COMPAT_84
+ /*
+ * The version 1 dialect's disk_conf carries "fencing", whose DRBD 9
+ * home is net_conf.fencing_policy: a connection-scoped setting. 8.4's
+ * normal order is attach, then connect, so a v1 attach or
+ * disk-options call can send this before any connection exists to
+ * apply it to. Stashed here in that case, protected by adm_mutex;
+ * the v1 connect handler applies it once the connection is created
+ * and clears the flag.
+ */
+ bool pending_fencing_policy_84_set;
+ enum drbd_fencing_policy pending_fencing_policy_84;
+#endif
+ /*
+ * For read-copy-update of net_conf and disk_conf and devices,
+ * connection, peer_devices and paths lists.
+ */
+ struct mutex conf_update;
struct mutex adm_mutex; /* mutex to serialize administrative requests */
- spinlock_t req_lock;
+ struct mutex open_release; /* serialize open/release */
+ struct {
+ char comm[TASK_COMM_LEN];
+ unsigned int minor;
+ pid_t pid;
+ ktime_t opened;
+ } auto_promoted_by;
+
+ rwlock_t state_rwlock; /* serialize state changes */
+ u64 dagtag_sector; /* Protected by tl_update_lock.
+ * See also dagtag_sector in
+ * &drbd_request */
+ u64 dagtag_from_backing_dev;
+ u64 dagtag_before_attach;
+ u64 members; /* mask of online nodes */
+ unsigned long flags;
+
+ /* Protects updates to the transfer log and related counters. */
+ spinlock_t tl_update_lock;
+ struct list_head transfer_log; /* all requests not yet fully processed */
+ struct drbd_request *tl_previous_write;
+
+ spinlock_t peer_ack_lock;
+ struct list_head peer_ack_req_list; /* requests to send peer acks for */
+ struct list_head peer_ack_list; /* peer acks to send */
+ struct drbd_work peer_ack_work;
+ u64 last_peer_acked_dagtag; /* dagtag of last PEER_ACK'ed request */
+ struct drbd_request *peer_ack_req; /* last request not yet PEER_ACK'ed */
+
+ spinlock_t send_oos_lock; /* Protects send_oos list */
+ /*
+ * Peer writes for which we need to send some P_OUT_OF_SYNC, linked via
+ * drbd_peer_request.recv_order. The sender of each peer in
+ * send_oos_pending sends them. Each entry holds a kref on the
+ * connection it was received on, which keeps its peer_device alive.
+ */
+ struct list_head send_oos;
- unsigned susp:1; /* IO suspended by user */
- unsigned susp_nod:1; /* IO suspended because no data */
- unsigned susp_fen:1; /* IO suspended because fence peer handler runs */
+ /* Protects current_flush_sequence and pending_flush_mask (connection) */
+ spinlock_t initiator_flush_lock;
+ u64 current_flush_sequence;
+ /* A prepared connect of a Primary, which may need a new flush round */
+ u64 flush_connect_nodes;
+ u64 flush_connect_sequence;
+ bool flush_connect_own_missing;
+
+ struct semaphore state_sem;
+ wait_queue_head_t state_wait; /* upon each state change. */
+ unsigned int state_change_seq; /* bumped on each committed state change */
+ enum chg_state_flags state_change_flags;
+ const char **state_change_err_str;
+ bool remote_state_change; /* remote state change in progress */
+ enum drbd_packet twopc_prepare_reply_cmd; /* this node's answer to the prepare phase or 0 */
+ u64 twopc_parent_nodes;
+ struct twopc_reply twopc_reply;
+ struct timer_list twopc_timer;
+ struct work_struct twopc_work;
+ wait_queue_head_t twopc_wait;
+ struct {
+ enum twopc_type type;
+ union {
+ struct twopc_resize {
+ int dds_flags; /* from prepare phase */
+ sector_t user_size; /* from prepare phase */
+ u64 diskful_primary_nodes; /* added in commit phase */
+ u64 new_size; /* added in commit phase */
+ } resize;
+ struct twopc_state_change {
+ union drbd_state mask; /* from prepare phase */
+ union drbd_state val; /* from prepare phase */
+ u64 primary_nodes; /* added in commit phase */
+ u64 reachable_nodes; /* added in commit phase */
+ } state_change;
+ };
+ } twopc;
+ enum drbd_role role[2];
+ bool susp_user[2]; /* IO suspended by user */
+ bool susp_nod[2]; /* IO suspended because no data */
+ bool susp_quorum[2]; /* IO suspended because no quorum */
+ bool susp_uuid[2]; /* IO suspended because waiting new current UUID */
+ bool fail_io[2]; /* Fail all IO requests because forced a demote */
+ bool resume_held_for_outdate[2];/* A Primary regained data access, but a
+ * far-away member must be outdated before
+ * I/O resumes; keeps susp_nod set until the
+ * primary-resume 2PC has done that.
+ */
+ bool cached_susp; /* cached result of looking at all different suspend bits */
+ bool cached_all_devices_have_quorum;
enum write_ordering_e write_ordering;
+ /* Protects the current transfer log (tle) fields. */
+ spinlock_t current_tle_lock;
+ atomic_t current_tle_nr; /* transfer log epoch number */
+ unsigned current_tle_writes; /* writes seen within this tl epoch */
+
+ unsigned cached_min_aggreed_protocol_version;
+
cpumask_var_t cpu_mask;
-};
-struct drbd_thread_timing_details
-{
- unsigned long start_jif;
- void *cb_addr;
- const char *caller_fn;
- unsigned int line;
- unsigned int cb_nr;
+ struct drbd_work_queue work;
+ struct drbd_thread worker;
+
+ struct list_head listeners;
+ spinlock_t listeners_lock;
+
+ struct timer_list peer_ack_timer; /* send a P_PEER_ACK after last completion */
+
+ unsigned int w_cb_nr; /* keeps counting up */
+ struct drbd_thread_timing_details w_timing_details[DRBD_THREAD_DETAILS_HIST];
+ wait_queue_head_t barrier_wait; /* upon each state change. */
+ struct rcu_head rcu;
+
+ struct list_head suspended_reqs;
+ /*
+ * The side effects of an empty state change two-phase commit are:
+ *
+ * * A local consistent disk can upgrade to up-to-date when no primary is reachable
+ * (or become outdated if the prepare packets reach a primary).
+ *
+ * * resource->members are updates
+ *
+ * * Faraway nodes might outdate themselves if they learn about the existence of a primary
+ * (with access to data) node.
+ */
+ struct work_struct empty_twopc;
+ struct work_struct resume_twopc;
};
enum drbd_per_connection_ratelimit {
@@ -640,75 +1386,195 @@ struct drbd_connection {
struct dentry *debugfs_conn;
struct dentry *debugfs_conn_callback_history;
struct dentry *debugfs_conn_oldest_requests;
+ struct dentry *debugfs_conn_transport;
+ struct dentry *debugfs_conn_debug;
+ struct dentry *debugfs_conn_receiver_pid;
+ struct dentry *debugfs_conn_sender_pid;
#endif
struct kref kref;
struct idr peer_devices; /* volume number to peer device mapping */
+ enum drbd_conn_state cstate[2];
+ enum drbd_role peer_role[2];
+ bool susp_fen[2]; /* IO suspended because fence peer handler runs */
struct ratelimit_state ratelimit[1];
- enum drbd_conns cstate; /* Only C_STANDALONE to C_WF_REPORT_PARAMS */
- struct mutex cstate_mutex; /* Protects graceful disconnects */
- unsigned int connect_cnt; /* Inc each time a connection is established */
-
unsigned long flags;
- struct net_conf *net_conf; /* content protected by rcu */
- wait_queue_head_t ping_wait; /* Woken upon reception of a ping, and a state change */
-
- struct sockaddr_storage my_addr;
- int my_addr_len;
- struct sockaddr_storage peer_addr;
- int peer_addr_len;
+ enum drbd_fencing_policy fencing_policy;
+
+ struct drbd_send_buffer send_buffer[2];
+ struct mutex mutex[2]; /* Protect assembling of new packet until sending it (in send_buffer) */
+ /* scratch buffers for use while "owning" the DATA_STREAM send_buffer,
+ * to avoid larger on-stack temporary variables,
+ * introduced for holding digests in drbd_send_dblock() */
+ union {
+ /* MAX_DIGEST_SIZE in the linux kernel at this point is 64 byte, afaik */
+ struct {
+ char before[64];
+ char after[64];
+ } d;
+ } scratch_buffer;
- struct drbd_socket data; /* data/barrier/cstate/parameter packets */
- struct drbd_socket meta; /* ping/ack (metadata) packets */
int agreed_pro_version; /* actually used protocol version */
u32 agreed_features;
- unsigned long last_received; /* in jiffies, either socket */
- unsigned int ko_count;
+ atomic_t ap_in_flight; /* App sectors in flight (waiting for ack) */
+ atomic_t rs_in_flight; /* Resync sectors in flight */
+ int last_acked_epoch_nr; /* of the last P_BARRIER_ACK, set in tl_release() */
- struct list_head transfer_log; /* all requests not yet fully processed */
+ struct drbd_work connect_timer_work;
+ struct timer_list connect_timer;
+ /* Connect two-phase commits attempted in this transport session. */
+ unsigned int connect_tries;
struct crypto_shash *cram_hmac_tfm;
- struct crypto_shash *integrity_tfm; /* checksums we compute, updates protected by connection->data->mutex */
+ struct crypto_shash *integrity_tfm; /* checksums we compute, updates protected by connection->mutex[DATA_STREAM] */
struct crypto_shash *peer_integrity_tfm; /* checksums we verify, only accessed from receiver thread */
struct crypto_shash *csums_tfm;
struct crypto_shash *verify_tfm;
+ char peer_verify_alg[SHARED_SECRET_MAX]; /* peer's verify-alg; protected by conf_update */
+
void *int_dig_in;
void *int_dig_vv;
/* receiver side */
+ /* Payload copies to satisfy the backing device's alignment;
+ * receiver thread only.
+ */
+ unsigned int rx_misaligned_copies;
struct drbd_epoch *current_epoch;
spinlock_t epoch_lock;
unsigned int epochs;
- atomic_t current_tle_nr; /* transfer log epoch number */
- unsigned current_tle_writes; /* writes seen within this tl epoch */
unsigned long last_reconnect_jif;
/* empty member on older kernels without blk_start_plug() */
struct blk_plug receiver_plug;
struct drbd_thread receiver;
- struct drbd_thread worker;
- struct drbd_thread ack_receiver;
+ struct drbd_thread sender;
struct workqueue_struct *ack_sender;
+ struct work_struct peer_ack_work;
+
+ /* Work for sending P_OUT_OF_SYNC for the writes on resource->send_oos */
+ struct drbd_work send_oos_work;
+
+ atomic64_t last_dagtag_sector;
+ /* Record of last peer ack to determine whether we can ack flush */
+ u64 last_peer_ack_dagtag_seen;
+
+ /* Mask of nodes from which we are waiting for a flush ack corresponding to this Primary */
+ u64 pending_flush_mask;
+
+ /* Protects the flush members below for this connection */
+ spinlock_t primary_flush_lock;
+ /* For handling P_FLUSH_REQUESTS from this peer */
+ u64 flush_requests_dagtag;
+ u64 flush_sequence;
+ u64 flush_forward_sent_mask;
+
+ /* For handling forwarded flushes. On connection to initiator node. */
+ spinlock_t flush_ack_lock;
+ struct drbd_work flush_ack_work;
+ /* For forwarded flushes. On connection to initiator node. Indexed by primary node ID */
+ u64 flush_ack_sequence[DRBD_PEERS_MAX];
+
+ atomic_t active_ee_cnt; /* Peer write requests waiting for activity log or backing disk. */
+ atomic_t backing_ee_cnt; /* Other peer requests waiting for conflicts or backing disk. */
+ atomic_t done_ee_cnt;
+ spinlock_t peer_reqs_lock;
+
+ /* Lists using drbd_peer_request.recv_order (see also drbd_peer_device.resync_requests) */
+ struct list_head peer_requests; /* All peer writes in the order we received them */
+ struct list_head peer_reads; /* All reads in the order we received them */
+
+ /* Lists using drbd_peer_request.w.list */
+ struct list_head done_ee; /* Need to send P_WRITE_ACK/P_RS_WRITE_ACK */
+ struct list_head dagtag_wait_ee; /* Resync read waiting for dagtag to be reached */
+ /* Peer writes done, waiting for the sync source to have them as well */
+ struct list_head source_wait_ee;
+
+ /* Dagtag wait requests of sync targets, waiting for this connection's
+ * write stream to reach the position they name. Protected by
+ * peer_reqs_lock.
+ */
+ struct list_head dagtag_wait_reqs;
+
+ struct work_struct send_acks_work;
+ struct work_struct send_ping_ack_work;
+ struct work_struct send_ping_work;
+ wait_queue_head_t ee_wait;
+
+ atomic_t pp_in_use; /* allocated from page pool */
+ atomic_t pp_in_use_by_net; /* sendpage()d, still referenced by transport */
+ /* sender side */
+ struct drbd_work_queue sender_work;
+
+ struct drbd_work send_dagtag_work;
+ u64 send_dagtag;
+
+ struct sender_todo {
+ struct list_head work_list;
+
+ /* If upper layers trigger an unplug on this side, we want to
+ * send and unplug hint over to the peer. Sending it too
+ * early, or missing it completely, causes a potential latency
+ * penalty (requests idling too long in the remote queue).
+ * There is no harm done if we occasionally send one too many
+ * such unplug hints.
+ *
+ * We have two slots, which are used in an alternating fashion:
+ * If a new unplug event happens while the current pending one
+ * has not even been processed yet, we overwrite the next
+ * pending slot: there is not much point in unplugging on the
+ * remote side, if we have a full request queue to be send on
+ * this side still, and not even reached the position in the
+ * change stream when the previous local unplug happened.
+ */
+ u64 unplug_dagtag_sector[2];
+ unsigned int unplug_slot; /* 0 or 1 */
+
+ /* the currently (or last) processed request,
+ * see process_sender_todo() */
+ struct drbd_request *req;
+
+ /* Points to the next request on the resource->transfer_log,
+ * which is RQ_NET_QUEUED for this connection, and so can
+ * safely be used as next starting point for the list walk
+ * in tl_next_request_for_connection().
+ *
+ * If it is NULL (we walked off the tail last time), it will be
+ * set by __req_mod( QUEUE_FOR.* ), so fast connections don't
+ * need to walk the full transfer_log list every time, even if
+ * the list is kept long by some slow connections.
+ *
+ * req_next is only accessed by drbd_sender thread, in
+ * case of a resend from some worker, but then regular IO
+ * is suspended.
+ */
+ struct drbd_request *req_next;
+ } todo;
/* cached pointers,
* so we can look up the oldest pending requests more quickly.
- * protected by resource->req_lock */
- struct drbd_request *req_next; /* DRBD 9: todo.req_next */
+ * TODO: RCU */
struct drbd_request *req_ack_pending;
+ /* The oldest request that is or was queued for this peer, but is not
+ * done towards it. */
struct drbd_request *req_not_net_done;
+ /* The oldest request that is queued for this peer and ready to be
+ * processed by the sender. Lets the sender pass requests that are queued
+ * but not yet ready, see tl_next_request_for_connection().
+ */
+ struct drbd_request *req_next_ready;
+ /* Protects the caching pointers from being advanced concurrently. */
+ spinlock_t advance_cache_ptr_lock;
- /* sender side */
- struct drbd_work_queue sender_work;
-
-#define DRBD_THREAD_DETAILS_HIST 16
- unsigned int w_cb_nr; /* keeps counting up */
+ unsigned int s_cb_nr; /* keeps counting up */
unsigned int r_cb_nr; /* keeps counting up */
- struct drbd_thread_timing_details w_timing_details[DRBD_THREAD_DETAILS_HIST];
+ struct drbd_thread_timing_details s_timing_details[DRBD_THREAD_DETAILS_HIST];
struct drbd_thread_timing_details r_timing_details[DRBD_THREAD_DETAILS_HIST];
struct {
unsigned long last_sent_barrier_jif;
+ int last_sent_epoch_nr;
/* whether this sender thread
* has processed a single write yet. */
@@ -721,37 +1587,65 @@ struct drbd_connection {
* with req->epoch == current_epoch_nr.
* If none, no P_BARRIER will be sent. */
unsigned current_epoch_writes;
- } send;
-};
-static inline bool has_net_conf(struct drbd_connection *connection)
-{
- bool has_net_conf;
-
- rcu_read_lock();
- has_net_conf = rcu_dereference(connection->net_conf);
- rcu_read_unlock();
+ /* Position in change stream of last write sent. */
+ u64 current_dagtag_sector;
- return has_net_conf;
-}
+ /* Position in change stream of last ready request seen. */
+ u64 seen_dagtag_sector;
+ } send;
-void __update_timing_details(
- struct drbd_thread_timing_details *tdp,
- unsigned int *cb_nr,
- void *cb,
- const char *fn, const unsigned int line);
+ struct {
+ u64 dagtag_sector;
+ int lost_node_id;
+ } after_reconciliation;
+
+ /*
+ * Filled in during the connect handshake (DRBD_FF_RECONCILE_RECONNECT)
+ * from a P_PEER_DAGTAG the peer sends before its state: the peer's
+ * position in a common lost primary's change stream. drbd_uuid_compare()
+ * compares it against our own last_dagtag_sector toward that node to roll
+ * an equal-UUID both-dirty reconcile forward. lost_node_id == -1 if none.
+ * sent_lost_node records the mirror direction: we named a common lost
+ * primary to the peer in this handshake (conn_connect2).
+ */
+ struct {
+ u64 dagtag_sector;
+ int lost_node_id;
+ bool sent_lost_node;
+ } reconcile_handshake;
-#define update_worker_timing_details(c, cb) \
- __update_timing_details(c->w_timing_details, &c->w_cb_nr, cb, __func__ , __LINE__ )
-#define update_receiver_timing_details(c, cb) \
- __update_timing_details(c->r_timing_details, &c->r_cb_nr, cb, __func__ , __LINE__ )
+ unsigned int peer_node_id;
-struct submit_worker {
- struct workqueue_struct *wq;
- struct work_struct worker;
+ struct drbd_mutable_buffer reassemble_buffer;
+ union {
+ u8 bytes[8];
+ struct p_block_ack block_ack;
+ struct p_barrier_ack barrier_ack;
+ struct p_confirm_stable confirm_stable;
+ struct p_peer_ack peer_ack;
+ struct p_peer_block_desc peer_block_desc;
+ struct p_twopc_reply twopc_reply;
+ } reassemble_buffer_bytes;
+
+ /* Used when a network namespace is removed to track all connections
+ * that need disconnecting. */
+ struct list_head remove_net_list;
+
+ struct rcu_head rcu;
+
+ unsigned int ctl_packets;
+ unsigned int ctl_bytes;
+
+ struct drbd_transport transport; /* The transport needs to be the last member. The acutal
+ implementation might have more members than the
+ abstract one. */
+};
- /* protected by ..->resource->req_lock */
- struct list_head writes;
+/* used to get the next lower or next higher peer_device depending on device node-id */
+enum drbd_neighbor {
+ NEXT_LOWER,
+ NEXT_HIGHER
};
enum drbd_per_peer_device_ratelimit {
@@ -763,13 +1657,195 @@ struct drbd_peer_device {
struct list_head peer_devices;
struct drbd_device *device;
struct drbd_connection *connection;
- struct work_struct send_acks_work;
+ struct drbd_peer_device_conf __rcu *conf; /* RCU, for updates: resource->conf_update */
+ enum drbd_disk_state disk_state[2];
+ enum drbd_repl_state repl_state[2];
+ bool resync_susp_user[2];
+ bool resync_susp_peer[2];
+ bool resync_susp_dependency[2];
+ bool resync_susp_other_c[2];
+ bool resync_susp_max_parallel[2];
+ bool resync_active[2];
+ bool replication[2]; /* Only while peer is Inconsistent: Is replication enabled? */
+ bool peer_replication[2]; /* Whether we have instructed peer to replicate to us */
+ enum drbd_repl_state negotiation_result; /* To find disk state after attach */
+ unsigned int send_cnt;
+ unsigned int recv_cnt;
+ atomic_t packet_seq;
+ unsigned int peer_seq;
+ spinlock_t peer_seq_lock;
+ uint64_t d_size; /* size of disk */
+ uint64_t u_size; /* user requested size */
+ uint64_t c_size; /* current exported size */
+ uint64_t max_size;
+ int bitmap_index;
+ int node_id;
struct ratelimit_state ratelimit[1];
+ DECLARE_BITMAP(flags, PEER_DEVICE_FLAG_COUNT);
+
+ enum drbd_repl_state start_resync_side;
+ enum drbd_repl_state last_repl_state; /* What we received from the peer */
+ struct timer_list start_resync_timer;
+ struct drbd_work resync_work;
+ struct timer_list resync_timer;
+ struct drbd_work propagate_uuids_work;
+
+ enum drbd_disk_state resync_finished_pdsk; /* Finished while starting resync */
+ int resync_again; /* decided to resync again while resync running */
+ sector_t last_in_sync_end; /* sector after end of last completed resync request */
+ unsigned long resync_next_bit; /* bitmap bit to search from for next resync request */
+ unsigned long last_resync_pass_bits; /* bitmap weight at end of previous pass */
+
+ atomic_t ap_pending_cnt; /* AP data packets on the wire, ack expected (RQ_NET_PENDING set) */
+ atomic_t unacked_cnt; /* Need to send replies for */
+ atomic_t rs_pending_cnt; /* RS request/data packets on the wire */
+
+ /* Protected by connection->peer_reqs_lock */
+ struct list_head resync_requests; /* Resync requests in the order we sent them */
+ /*
+ * If not NULL, all requests in resync_requests until this one have
+ * been received. Discards are only counted as "received" once merging
+ * is complete.
+ */
+ struct drbd_peer_request *received_last;
+ /*
+ * If not NULL, all requests in resync_requests after received_last
+ * until this one are discards.
+ */
+ struct drbd_peer_request *discard_last;
+
+ /* use checksums for *this* resync */
+ bool use_csums;
+ /* blocks to resync in this run [unit BM_BLOCK_SIZE] */
+ unsigned long rs_total;
+ /* number of resync blocks that failed in this run */
+ unsigned long rs_failed;
+ /* Syncer's start time [unit jiffies] */
+ unsigned long rs_start;
+ /* cumulated time in PausedSyncX state [unit jiffies] */
+ unsigned long rs_paused;
+ /* skipped because csum was equal [unit BM_BLOCK_SIZE] */
+ unsigned long rs_same_csum;
+ unsigned long rs_last_progress_report_ts;
+#define DRBD_SYNC_MARKS 8
+#define DRBD_SYNC_MARK_STEP (3*HZ)
+ /* block not up-to-date at mark [unit BM_BLOCK_SIZE] */
+ unsigned long rs_mark_left[DRBD_SYNC_MARKS];
+ /* marks's time [unit jiffies] */
+ unsigned long rs_mark_time[DRBD_SYNC_MARKS];
+ /* current index into rs_mark_{left,time} */
+ int rs_last_mark;
+ unsigned long rs_last_writeout;
+
+ /* where does the admin want us to start? (sector) */
+ sector_t ov_start_sector;
+ sector_t ov_stop_sector;
+ /* where are we now? (sector) */
+ sector_t ov_position;
+ /* Start sector of out of sync range (to merge printk reporting). */
+ sector_t ov_last_oos_start;
+ /* size of out-of-sync range in sectors. */
+ sector_t ov_last_oos_size;
+ /* Start sector of skipped range (to merge printk reporting). */
+ sector_t ov_last_skipped_start;
+ /* size of skipped range in sectors. */
+ sector_t ov_last_skipped_size;
+ int c_sync_rate; /* current resync rate after syncer throttle magic */
+ struct fifo_buffer __rcu *rs_plan_s; /* correction values of resync planer (RCU, connection->conn_update) */
+ atomic_t rs_sect_in; /* for incoming resync data rate, SyncTarget */
+ /* Resync sectors received and parked at the backing-device depth
+ * bound: no longer in flight, not yet submitted.
+ */
+ atomic_t rs_sect_parked;
+ int rs_last_events; /* counter of read or write "events" (unit sectors)
+ * on the lower level device when we last looked. */
+ int rs_in_flight; /* resync sectors in flight (to proxy, in proxy and from proxy) */
+ ktime_t rs_last_mk_req_kt;
+ /* fixed resync-rate pacing: KiB*ns granted but not yet spent */
+ u64 rs_pacing_credit;
+ unsigned int rs_mxb_limited; /* consecutive rounds clamped by max-buffers */
+ atomic64_t ov_left; /* in bits */
+ unsigned long ov_skipped; /* in bits */
+ u64 rs_start_uuid;
+
+ u64 current_uuid;
+ u64 bitmap_uuids[DRBD_PEERS_MAX];
+ u64 history_uuids[HISTORY_UUIDS];
+ u64 dirty_bits;
+ u64 uuid_flags;
+ u64 uuid_node_mask; /* might be authoritative_nodes or weak_nodes */
+
+ unsigned long comm_bm_set; /* communicated number of set bits. */
+ u64 comm_current_uuid; /* communicated current UUID */
+ u64 comm_uuid_flags; /* communicated UUID flags */
+ /* Held across clearing SEND_UUIDS_PEER_ATTACH and the sends it calls for */
+ struct mutex peer_attach_send_mutex;
+ u64 comm_bitmap_uuid;
+ int comm_day0_node_id; /* communicated day0 slot, or -1 */
+ union drbd_state comm_state;
+
#ifdef CONFIG_DEBUG_FS
struct dentry *debugfs_peer_dev;
+ struct dentry *debugfs_peer_dev_proc_drbd;
#endif
+ ktime_t pre_send_kt;
+ ktime_t acked_kt;
+ ktime_t net_done_kt;
+
+ struct {/* sender todo per peer_device */
+ bool was_sending_out_of_sync;
+ } todo;
+ union drbd_state connect_state;
+ struct {
+ unsigned int physical_block_size;
+ unsigned int logical_block_size;
+ unsigned int alignment_offset;
+ unsigned int io_min;
+ unsigned int io_opt;
+ unsigned int max_bio_size;
+ } q_limits;
+ /* communicated as part of o_qlim, if agreed on DRBD_FF_BM_BLOCK_SHIFT */
+ unsigned int bm_block_shift;
+};
+
+/* Resync work this node has asked for and is not done with: still on the wire,
+ * plus what arrived and is parked at the depth bound. rs_in_flight drops when
+ * the data arrives, which for parked work is before the backing device has
+ * seen any of it, so the request loop has to add the parked share back or it
+ * plans against work it has already been credited for.
+ */
+static inline int drbd_rs_outstanding(struct drbd_peer_device *peer_device)
+{
+ return peer_device->rs_in_flight + atomic_read(&peer_device->rs_sect_parked);
+}
+
+struct conflict_worker {
+ struct workqueue_struct *wq;
+ struct work_struct worker;
+
+ spinlock_t lock;
+ struct list_head resync_writes;
+ struct list_head resync_reads;
+ struct list_head writes;
+ struct list_head peer_writes;
+};
+
+struct submit_worker {
+ struct workqueue_struct *wq;
+ struct work_struct worker;
+
+ spinlock_t lock;
+ struct list_head writes;
+ struct list_head peer_writes;
+};
+
+struct opener {
+ struct list_head list;
+ char comm[TASK_COMM_LEN];
+ pid_t pid;
+ ktime_t opened;
};
enum drbd_per_device_ratelimit {
@@ -782,7 +1858,11 @@ enum drbd_per_device_ratelimit {
struct drbd_device {
struct drbd_resource *resource;
+
+ /* RCU list. Updates protected by adm_mutex, conf_update and state_rwlock. */
struct list_head peer_devices;
+
+ spinlock_t pending_bmio_lock;
struct list_head pending_bitmap_io;
unsigned long flush_jif;
@@ -791,13 +1871,22 @@ struct drbd_device {
struct dentry *debugfs_vol;
struct dentry *debugfs_vol_oldest_requests;
struct dentry *debugfs_vol_act_log_extents;
- struct dentry *debugfs_vol_resync_extents;
+ struct dentry *debugfs_vol_act_log_histogram;
struct dentry *debugfs_vol_data_gen_id;
+ struct dentry *debugfs_vol_io_frozen;
struct dentry *debugfs_vol_ed_gen_id;
+ struct dentry *debugfs_vol_openers;
+ struct dentry *debugfs_vol_md_io;
+ struct dentry *debugfs_vol_interval_tree;
+ struct dentry *debugfs_vol_al_updates;
+ struct dentry *debugfs_vol_multi_bio_cnt;
+#ifdef CONFIG_DRBD_TIMING_STATS
+ struct dentry *debugfs_vol_req_timing;
+#endif
#endif
struct ratelimit_state ratelimit[__D_RL_D_N];
- unsigned int vnr; /* volume number within the connection */
+ unsigned int vnr; /* volume number within the resource */
unsigned int minor; /* device minor number */
struct kref kref;
@@ -805,184 +1894,248 @@ struct drbd_device {
/* things that are stored as / read from meta data on disk */
unsigned long flags;
+#ifdef CONFIG_DRBD_COMPAT_84
+ /*
+ * The version 1 dialect's disk_conf carries six resync-tuning
+ * fields whose DRBD 9 home is this device's single peer device's
+ * struct drbd_peer_device_conf. 8.4's normal order is attach, then
+ * connect, so a v1 attach or disk-options call can send these
+ * before any peer device exists to apply them to. Stashed here in
+ * that case, protected by resource->adm_mutex, one "was sent" flag
+ * per field so an unsent field does not overwrite live
+ * configuration with a stale or zero stash value; the v1 connect
+ * handler applies them once the peer device is created and clears
+ * the flags.
+ */
+ struct {
+ bool has_resync_rate;
+ u32 resync_rate;
+ bool has_c_plan_ahead;
+ u32 c_plan_ahead;
+ bool has_c_delay_target;
+ u32 c_delay_target;
+ bool has_c_fill_target;
+ u32 c_fill_target;
+ bool has_c_max_rate;
+ u32 c_max_rate;
+ bool has_c_min_rate;
+ u32 c_min_rate;
+ } pending_peer_device_conf_84;
+
+ /*
+ * The fused SIB_STATE_CHANGE DRBD_EVENT (drbd_nl_84.c) this device owes
+ * for the state change being notified: 8.4-packed state words before
+ * and after it, collected across that change's resource, connection,
+ * device and peer-device notifications, plus the connection's cstate
+ * before and after it (-1 if the connection did not change), for the
+ * peer-device notification to fall back to when it is not
+ * replicating. Serialized by notification_mutex.
+ */
+ bool bcast_pending_84;
+ u32 bcast_prev_84;
+ u32 bcast_new_84;
+ int bcast_cstate_prev_84;
+ int bcast_cstate_new_84;
+#endif
+
/* configured by drbdsetup */
- struct drbd_backing_dev *ldev;
+ struct drbd_backing_dev *ldev; /* enclose accessing code in get_ldev() / put_ldev() */
+
+ /* Used to close backing devices and destroy related structures. */
+ struct work_struct ldev_destroy_work;
+
+ /* Runs drbd_try_to_get_resynced() outside of the worker thread. */
+ struct work_struct try_get_resynced_work;
- sector_t p_size; /* partner's disk size */
struct request_queue *rq_queue;
struct gendisk *vdisk;
unsigned long last_reattach_jif;
- struct drbd_work resync_work;
- struct drbd_work unplug_work;
- struct timer_list resync_timer;
struct timer_list md_sync_timer;
- struct timer_list start_resync_timer;
struct timer_list request_timer;
- /* Used after attach while negotiating new disk state. */
- union drbd_state new_state_tmp;
-
- union drbd_dev_state state;
+ enum drbd_disk_state disk_state[2];
wait_queue_head_t misc_wait;
- wait_queue_head_t state_wait; /* upon each state change. */
- unsigned int send_cnt;
- unsigned int recv_cnt;
unsigned int read_cnt;
unsigned int writ_cnt;
unsigned int al_writ_cnt;
unsigned int bm_writ_cnt;
- atomic_t ap_bio_cnt; /* Requests we need to complete */
- atomic_t ap_actlog_cnt; /* Requests waiting for activity log */
- atomic_t ap_pending_cnt; /* AP data packets on the wire, ack expected */
- atomic_t rs_pending_cnt; /* RS request/data packets on the wire */
- atomic_t unacked_cnt; /* Need to send replies for */
+ unsigned int multi_bio_cnt; /* peer_reqs that needed multiple bios */
+ atomic_t ap_bio_cnt[2]; /* Requests we need to complete. [READ] and [WRITE] */
atomic_t local_cnt; /* Waiting for local completion */
- atomic_t suspend_cnt;
+ unsigned int cbt_flush_writ_cnt; /* writ_cnt at the last cbt-flush */
+ atomic_t ap_actlog_cnt; /* Requests waiting for activity log */
+ atomic_t wait_for_actlog; /* Peer requests waiting for activity log */
+ /* worst case extent count needed to satisfy both requests and peer requests
+ * currently waiting for the activity log */
+ atomic_t wait_for_actlog_ecnt;
- /* Interval tree of pending local requests */
- struct rb_root read_requests;
- struct rb_root write_requests;
+ atomic_t suspend_cnt; /* recursive suspend counter, if non-zero, IO will be blocked. */
+
+ /* Interval trees of pending requests */
+ spinlock_t interval_lock;
+ struct rb_root read_requests; /* Local reads */
+ struct rb_root requests; /* Local and peer writes, resync operations etc. */
/* for statistics and timeouts */
/* [0] read, [1] write */
+ spinlock_t pending_completion_lock;
struct list_head pending_master_completion[2];
struct list_head pending_completion[2];
- /* use checksums for *this* resync */
- bool use_csums;
- /* blocks to resync in this run [unit BM_BLOCK_SIZE] */
- unsigned long rs_total;
- /* number of resync blocks that failed in this run */
- unsigned long rs_failed;
- /* Syncer's start time [unit jiffies] */
- unsigned long rs_start;
- /* cumulated time in PausedSyncX state [unit jiffies] */
- unsigned long rs_paused;
- /* skipped because csum was equal [unit BM_BLOCK_SIZE] */
- unsigned long rs_same_csum;
-#define DRBD_SYNC_MARKS 8
-#define DRBD_SYNC_MARK_STEP (3*HZ)
- /* block not up-to-date at mark [unit BM_BLOCK_SIZE] */
- unsigned long rs_mark_left[DRBD_SYNC_MARKS];
- /* marks's time [unit jiffies] */
- unsigned long rs_mark_time[DRBD_SYNC_MARKS];
- /* current index into rs_mark_{left,time} */
- int rs_last_mark;
- unsigned long rs_last_bcast; /* [unit jiffies] */
-
- /* where does the admin want us to start? (sector) */
- sector_t ov_start_sector;
- sector_t ov_stop_sector;
- /* where are we now? (sector) */
- sector_t ov_position;
- /* Start sector of out of sync range (to merge printk reporting). */
- sector_t ov_last_oos_start;
- /* size of out-of-sync range in sectors. */
- sector_t ov_last_oos_size;
- unsigned long ov_left; /* in bits */
-
- struct drbd_bitmap *bitmap;
- unsigned long bm_resync_fo; /* bit offset for drbd_bm_find_next */
-
- /* Used to track operations of resync... */
- struct lru_cache *resync;
- /* Number of locked elements in resync LRU */
- unsigned int resync_locked;
- /* resync extent number waiting for application requests */
- unsigned int resync_wenr;
+ struct drbd_bitmap *bitmap; /* enclose accessing code in get_ldev() / put_ldev() */
+ /* We may want to report on resync progress
+ * even after we detached again (bitmap == NULL).
+ * Cache the last bitmap block size here.
+ */
+ unsigned int last_bm_block_shift;
int open_cnt;
- u64 *p_uuid;
-
- struct list_head active_ee; /* IO in progress (P_DATA gets written to disk) */
- struct list_head sync_ee; /* IO in progress (P_RS_DATA_REPLY gets written to disk) */
- struct list_head done_ee; /* need to send P_WRITE_ACK */
- struct list_head read_ee; /* [RS]P_DATA_REQUEST being read */
+ bool writable;
+ /* FIXME clean comments, restructure so it is more obvious which
+ * members are protected by what */
- struct list_head resync_reads;
- atomic_t pp_in_use; /* allocated from page pool */
- atomic_t pp_in_use_by_net; /* sendpage()d, still referenced by tcp */
- wait_queue_head_t ee_wait;
struct drbd_md_io md_io;
spinlock_t al_lock;
wait_queue_head_t al_wait;
struct lru_cache *act_log; /* activity log */
+ unsigned al_histogram[AL_UPDATES_PER_TRANSACTION+1];
unsigned int al_tr_number;
int al_tr_cycle;
wait_queue_head_t seq_wait;
- atomic_t packet_seq;
- unsigned int peer_seq;
- spinlock_t peer_seq_lock;
- unsigned long comm_bm_set; /* communicated number of set bits. */
- struct bm_io_work bm_io_work;
- u64 ed_uuid; /* UUID of the exposed data */
- struct mutex own_state_mutex;
- struct mutex *state_mutex; /* either own_state_mutex or first_peer_device(device)->connection->cstate_mutex */
- char congestion_reason; /* Why we where congested... */
- atomic_t rs_sect_in; /* for incoming resync data rate, SyncTarget */
- atomic_t rs_sect_ev; /* for submitted resync data rate, both */
- int rs_last_sect_ev; /* counter to compare with */
- int rs_last_events; /* counter of read or write "events" (unit sectors)
- * on the lower level device when we last looked. */
- int c_sync_rate; /* current resync rate after syncer throttle magic */
- struct fifo_buffer *rs_plan_s; /* correction values of resync planer (RCU, connection->conn_update) */
- int rs_in_flight; /* resync sectors in flight (to proxy, in proxy and from proxy) */
- atomic_t ap_in_flight; /* App sectors in flight (waiting for ack) */
- unsigned int peer_max_bio_size;
- unsigned int local_max_bio_size;
-
- /* any requests that would block in drbd_make_request()
- * are deferred to this single-threaded work queue */
+ u64 exposed_data_uuid; /* UUID of the exposed data */
+ u64 next_exposed_data_uuid;
+ /* When a diskless primary starts a new data generation it keeps the
+ * generation it bumped away from (the predecessor) and the write epoch
+ * the new generation began in. This lets a peer that reconnects on the
+ * predecessor generation be recognised and brought forward by replaying
+ * the transfer log (re-injecting the new current UUID at exposed_gen_epoch),
+ * rather than rejected -- as long as the writes bridging the two are still
+ * resendable. Protected by uuid_sem.
+ */
+ u64 exposed_data_uuid_predecessor;
+ unsigned int exposed_gen_epoch;
+ struct rw_semaphore uuid_sem;
+ /* Resync sectors completed on the backing device, counted where
+ * part_stat sectors[] counts them: the busy detector subtracts these.
+ */
+ atomic_t rs_sect_done;
+ /* Resync sectors submitted to the backing device and not yet
+ * completed: the queue a competing application request waits behind.
+ */
+ atomic_t rs_sect_queued;
+ /* Resync sectors the backing device has completed, and how fast it
+ * completed them while it had resync IO to work on. Updated unlocked:
+ * a torn 32-bit ktime_t read can skew one measurement.
+ */
+ atomic_t rs_drain_sect;
+ unsigned int rs_drain_mark; /* rs_drain_sect when this interval began */
+ ktime_t rs_drain_since; /* when the device last became busy */
+ s64 rs_drain_busy_ns; /* busy time in this interval, idle excluded */
+ unsigned int rs_drain_rate; /* sectors per second, 0 until measured */
+ unsigned int rs_drain_sample_sect; /* rs_drain_sample_sect(), at the last change */
+ /* The backing device's service time for resync IO, timed on one probe
+ * request submitted to an empty queue; see drbd_rs_depth_completed().
+ */
+ u64 rs_lat_ns; /* 0 until probed */
+ ktime_t rs_lat_probe_kt; /* when the probe request was submitted */
+ int rs_lat_probe; /* enum rs_lat_probe */
+ unsigned long rs_lat_due; /* jiffies: the next periodic probe */
+ unsigned long rs_lat_wend; /* jiffies: end of the watch window */
+ unsigned long rs_lat_wmin; /* min latency in the window + 1, jiffies */
+ struct pending_bitmap_work_s {
+ atomic_t n; /* inc when queued here, */
+ spinlock_t q_lock; /* dec only once finished. */
+ struct list_head q; /* n > 0 even if q already empty */
+ } pending_bitmap_work;
+ struct drbd_device_conf device_conf;
+
+ /* any requests that were blocked due to conflicts with other requests
+ * or resync are submitted on this ordered work queue */
+ struct conflict_worker submit_conflict;
+ /* any requests that would block due to the activity log
+ * are deferred to this ordered work queue */
struct submit_worker submit;
+ u64 read_nodes; /* used for balancing read requests among peers */
+ bool have_quorum[2]; /* no quorum -> suspend IO or error IO */
+ bool quorum[2]; /* real quorum, tracked even when quorum is
+ * disabled (then not enforced); used to decide
+ * reconciliation direction
+ */
+ bool cached_state_unstable; /* updates with each state change */
+ bool cached_err_io; /* complete all IOs with error */
+ sector_t auto_grow_asked; /* the backing-device maximum a cluster-wide
+ * size change already answered for; see
+ * drbd_auto_grow(). No lock: a lost update
+ * costs one transaction, or one more arming
+ * edge to start it.
+ */
+ u32 gen_obligation; /* state, auxiliary bits and reason set of the
+ * data-generation obligation; see enum
+ * drbd_gen_obl_state. Read with the accessors
+ * below, changed only by a transition holding
+ * gen_obligation_lock.
+ */
+ spinlock_t gen_obligation_lock;
+
+#ifdef CONFIG_DRBD_TIMING_STATS
+ spinlock_t timing_lock;
+ unsigned long reqs;
+ ktime_t in_actlog_kt;
+ ktime_t pre_submit_kt; /* sum over over all reqs */
+
+ ktime_t before_queue_kt; /* sum over all al_misses */
+ ktime_t before_al_begin_io_kt;
+
+ ktime_t al_before_bm_write_hinted_kt; /* sum over all al_writ_cnt */
+ ktime_t al_mid_kt;
+ ktime_t al_after_sync_page_kt;
+#endif
+ struct list_head openers;
+ spinlock_t openers_lock;
+ spinlock_t peer_req_bio_completion_lock;
+
+ struct rcu_head rcu;
+ struct work_struct finalize_work;
};
+/* Per-peer CURRENT_UUID_UNCONFIRMED clear; defined in drbd_req.c. */
+extern void drbd_peer_maybe_confirm_rotated_gen(struct drbd_peer_device *peer_device,
+ unsigned int acked_epoch);
+
+/* Device-level decision for the per-peer signals above; defined in
+ * drbd_receiver.c. Returns true (discharging the obligation) when the rotated
+ * generation just became confirmed across a quorate set of survivors.
+ */
+extern bool drbd_maybe_release_rotated_gen(struct drbd_device *device);
+extern void drbd_reconcile_settled_try_up_to_date(struct drbd_resource *resource);
+
struct drbd_bm_aio_ctx {
struct drbd_device *device;
- struct list_head list; /* on device->pending_bitmap_io */;
+ struct list_head list; /* on device->pending_bitmap_io */
unsigned long start_jif;
+ struct blk_plug bm_aio_plug;
atomic_t in_flight;
unsigned int done;
unsigned flags;
#define BM_AIO_COPY_PAGES 1
#define BM_AIO_WRITE_HINTED 2
#define BM_AIO_WRITE_ALL_PAGES 4
-#define BM_AIO_READ 8
+#define BM_AIO_READ 8
+#define BM_AIO_WRITE_LAZY 16
+ /* suppress the "bitmap WRITE of N pages took M ms" summary */
+#define BM_AIO_NO_STATS 32
int error;
struct kref kref;
};
-struct drbd_config_context {
- /* assigned from drbd_genlmsghdr */
- unsigned int minor;
- /* assigned from request attributes, if present */
- unsigned int volume;
-#define VOLUME_UNSPECIFIED (-1U)
- /* pointer into the request skb,
- * limited lifetime! */
- char *resource_name;
- struct nlattr *my_addr;
- struct nlattr *peer_addr;
-
- /* reply buffer */
- struct sk_buff *reply_skb;
- /* pointer into reply buffer */
- struct drbd_genlmsghdr *reply_dh;
- /* resolved from attributes, if possible */
- struct drbd_device *device;
- struct drbd_resource *resource;
- struct drbd_connection *connection;
-};
+#include "drbd_nl.h"
static inline struct drbd_device *minor_to_device(unsigned int minor)
{
return (struct drbd_device *)idr_find(&drbd_devices, minor);
}
-static inline struct drbd_peer_device *first_peer_device(struct drbd_device *device)
-{
- return list_first_entry_or_null(&device->peer_devices, struct drbd_peer_device, peer_devices);
-}
static inline struct drbd_peer_device *
conn_peer_device(struct drbd_connection *connection, int volume_number)
@@ -996,18 +2149,19 @@ conn_peer_device(struct drbd_connection *connection, int volume_number)
#define for_each_resource_rcu(resource, _resources) \
list_for_each_entry_rcu(resource, _resources, resources)
-#define for_each_resource_safe(resource, tmp, _resources) \
- list_for_each_entry_safe(resource, tmp, _resources, resources)
-
+/* see drbd_resource.connections for locking requirements */
#define for_each_connection(connection, resource) \
list_for_each_entry(connection, &resource->connections, connections)
#define for_each_connection_rcu(connection, resource) \
list_for_each_entry_rcu(connection, &resource->connections, connections)
-#define for_each_connection_safe(connection, tmp, resource) \
- list_for_each_entry_safe(connection, tmp, &resource->connections, connections)
+#define for_each_connection_ref(connection, m, resource) \
+ for (connection = __drbd_next_connection_ref(&m, NULL, resource); \
+ connection; \
+ connection = __drbd_next_connection_ref(&m, connection, resource))
+/* see drbd_device.peer_devices for locking requirements */
#define for_each_peer_device(peer_device, device) \
list_for_each_entry(peer_device, &device->peer_devices, peer_devices)
@@ -1017,10 +2171,10 @@ conn_peer_device(struct drbd_connection *connection, int volume_number)
#define for_each_peer_device_safe(peer_device, tmp, device) \
list_for_each_entry_safe(peer_device, tmp, &device->peer_devices, peer_devices)
-static inline unsigned int device_to_minor(struct drbd_device *device)
-{
- return device->minor;
-}
+#define for_each_peer_device_ref(peer_device, m, device) \
+ for (peer_device = __drbd_next_peer_device_ref(&m, NULL, device); \
+ peer_device; \
+ peer_device = __drbd_next_peer_device_ref(&m, peer_device, device))
/*
* function declarations
@@ -1029,97 +2183,172 @@ static inline unsigned int device_to_minor(struct drbd_device *device)
/* drbd_main.c */
enum dds_flags {
- DDSF_FORCED = 1,
+ /* This enum is part of the wire protocol!
+ * See P_SIZES, struct p_sizes; */
+ DDSF_ASSUME_UNCONNECTED_PEER_HAS_SPACE = 1,
DDSF_NO_RESYNC = 2, /* Do not run a resync for the new space */
+ DDSF_IGNORE_PEER_CONSTRAINTS = 4, /* no longer used */
+ DDSF_2PC = 8, /* local only, not on the wire */
};
+struct meta_data_on_disk_9;
-extern void drbd_init_set_defaults(struct drbd_device *device);
-extern int drbd_thread_start(struct drbd_thread *thi);
-extern void _drbd_thread_stop(struct drbd_thread *thi, int restart, int wait);
+int drbd_thread_start(struct drbd_thread *thi);
+void _drbd_thread_stop(struct drbd_thread *thi, int restart, int wait);
+void drbd_thread_consume_restart(struct drbd_thread *thi);
#ifdef CONFIG_SMP
-extern void drbd_thread_current_set_cpu(struct drbd_thread *thi);
+void drbd_thread_current_set_cpu(struct drbd_thread *thi);
#else
#define drbd_thread_current_set_cpu(A) ({})
#endif
-extern void tl_release(struct drbd_connection *, unsigned int barrier_nr,
- unsigned int set_size);
-extern void tl_clear(struct drbd_connection *);
-extern void drbd_free_sock(struct drbd_connection *connection);
-extern int drbd_send(struct drbd_connection *connection, struct socket *sock,
- void *buf, size_t size, unsigned msg_flags);
-extern int drbd_send_all(struct drbd_connection *, struct socket *, void *, size_t,
- unsigned);
-
-extern int __drbd_send_protocol(struct drbd_connection *connection, enum drbd_packet cmd);
-extern int drbd_send_protocol(struct drbd_connection *connection);
-extern int drbd_send_uuids(struct drbd_peer_device *);
-extern int drbd_send_uuids_skip_initial_sync(struct drbd_peer_device *);
-extern void drbd_gen_and_send_sync_uuid(struct drbd_peer_device *);
-extern int drbd_send_sizes(struct drbd_peer_device *, int trigger_reply, enum dds_flags flags);
-extern int drbd_send_state(struct drbd_peer_device *, union drbd_state s);
-extern int drbd_send_current_state(struct drbd_peer_device *);
-extern int drbd_send_sync_param(struct drbd_peer_device *);
-extern void drbd_send_b_ack(struct drbd_connection *connection, u32 barrier_nr,
- u32 set_size);
-extern int drbd_send_ack(struct drbd_peer_device *, enum drbd_packet,
- struct drbd_peer_request *);
-extern void drbd_send_ack_rp(struct drbd_peer_device *, enum drbd_packet,
- struct p_block_req *rp);
-extern void drbd_send_ack_dp(struct drbd_peer_device *, enum drbd_packet,
- struct p_data *dp, int data_size);
-extern int drbd_send_ack_ex(struct drbd_peer_device *, enum drbd_packet,
- sector_t sector, int blksize, u64 block_id);
-extern int drbd_send_out_of_sync(struct drbd_peer_device *, struct drbd_request *);
-extern int drbd_send_block(struct drbd_peer_device *, enum drbd_packet,
- struct drbd_peer_request *);
-extern int drbd_send_dblock(struct drbd_peer_device *, struct drbd_request *req);
-extern int drbd_send_drequest(struct drbd_peer_device *, int cmd,
- sector_t sector, int size, u64 block_id);
-extern int drbd_send_drequest_csum(struct drbd_peer_device *, sector_t sector,
- int size, void *digest, int digest_size,
- enum drbd_packet cmd);
-extern int drbd_send_ov_request(struct drbd_peer_device *, sector_t sector, int size);
-
-extern int drbd_send_bitmap(struct drbd_device *device, struct drbd_peer_device *peer_device);
-extern void drbd_send_sr_reply(struct drbd_peer_device *, enum drbd_state_rv retcode);
-extern void conn_send_sr_reply(struct drbd_connection *connection, enum drbd_state_rv retcode);
-extern int drbd_send_rs_deallocated(struct drbd_peer_device *, struct drbd_peer_request *);
-extern void drbd_backing_dev_free(struct drbd_device *device, struct drbd_backing_dev *ldev);
-extern void drbd_device_cleanup(struct drbd_device *device);
-extern void drbd_print_uuids(struct drbd_device *device, const char *text);
-extern void drbd_queue_unplug(struct drbd_device *device);
-
-extern void conn_md_sync(struct drbd_connection *connection);
-extern void drbd_md_write(struct drbd_device *device, void *buffer);
-extern void drbd_md_sync(struct drbd_device *device);
-extern int drbd_md_read(struct drbd_device *device, struct drbd_backing_dev *bdev);
-extern void drbd_uuid_set(struct drbd_device *device, int idx, u64 val) __must_hold(local);
-extern void _drbd_uuid_set(struct drbd_device *device, int idx, u64 val) __must_hold(local);
-extern void drbd_uuid_new_current(struct drbd_device *device) __must_hold(local);
-extern void drbd_uuid_set_bm(struct drbd_device *device, u64 val) __must_hold(local);
-extern void drbd_uuid_move_history(struct drbd_device *device) __must_hold(local);
-extern void __drbd_uuid_set(struct drbd_device *device, int idx, u64 val) __must_hold(local);
-extern void drbd_md_set_flag(struct drbd_device *device, int flags) __must_hold(local);
-extern void drbd_md_clear_flag(struct drbd_device *device, int flags)__must_hold(local);
-extern int drbd_md_test_flag(struct drbd_backing_dev *, int);
-extern void drbd_md_mark_dirty(struct drbd_device *device);
-extern void drbd_queue_bitmap_io(struct drbd_device *device,
- int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
- void (*done)(struct drbd_device *, int),
- char *why, enum bm_flag flags,
- struct drbd_peer_device *peer_device);
-extern int drbd_bitmap_io(struct drbd_device *device,
- int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
- char *why, enum bm_flag flags,
- struct drbd_peer_device *peer_device);
-extern int drbd_bitmap_io_from_worker(struct drbd_device *device,
- int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
- char *why, enum bm_flag flags,
- struct drbd_peer_device *peer_device);
-extern int drbd_bmio_set_n_write(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local);
-extern int drbd_bmio_clear_n_write(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local);
+int tl_release(struct drbd_connection *connection, uint64_t o_block_id,
+ uint64_t y_block_id, unsigned int barrier_nr,
+ unsigned int set_size);
+int wait_for_barrier_acks_interruptible(struct drbd_resource *resource);
+
+int __drbd_send_protocol(struct drbd_connection *connection,
+ enum drbd_packet cmd);
+u64 drbd_collect_local_uuid_flags(struct drbd_peer_device *peer_device,
+ u64 *authoritative_mask);
+u64 drbd_resolved_uuid(struct drbd_peer_device *peer_device_base,
+ u64 *uuid_flags);
+int drbd_send_uuids(struct drbd_peer_device *peer_device, u64 uuid_flags,
+ u64 node_mask);
+void drbd_gen_and_send_sync_uuid(struct drbd_peer_device *peer_device);
+int drbd_send_sizes(struct drbd_peer_device *peer_device,
+ uint64_t u_size_diskless, enum dds_flags flags);
+int conn_send_state(struct drbd_connection *connection,
+ union drbd_state state);
+int drbd_send_state(struct drbd_peer_device *peer_device,
+ union drbd_state state);
+int drbd_send_current_state(struct drbd_peer_device *peer_device);
+void drbd_send_uuids_peer_attach(struct drbd_peer_device *peer_device);
+int drbd_send_sync_param(struct drbd_peer_device *peer_device);
+int drbd_send_out_of_sync(struct drbd_peer_device *peer_device,
+ sector_t sector, unsigned int size);
+int drbd_send_block(struct drbd_peer_device *peer_device,
+ enum drbd_packet cmd, struct drbd_peer_request *peer_req);
+int drbd_send_dblock(struct drbd_peer_device *peer_device,
+ struct drbd_request *req);
+int drbd_send_drequest(struct drbd_peer_device *peer_device, sector_t sector,
+ int size, u64 block_id);
+int drbd_send_rs_request(struct drbd_peer_device *peer_device,
+ enum drbd_packet cmd, sector_t sector, int size,
+ u64 block_id, unsigned int dagtag_node_id,
+ u64 dagtag);
+void *drbd_prepare_drequest_csum(struct drbd_peer_request *peer_req,
+ enum drbd_packet cmd, int digest_size,
+ unsigned int dagtag_node_id, u64 dagtag);
+
+int drbd_send_bitmap(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+int drbd_send_dagtag(struct drbd_connection *connection, u64 dagtag);
+void drbd_send_sr_reply(struct drbd_connection *connection, int vnr,
+ enum drbd_state_rv retcode);
+int drbd_send_rs_deallocated(struct drbd_peer_device *peer_device,
+ struct drbd_peer_request *peer_req);
+void drbd_send_twopc_reply(struct drbd_connection *connection,
+ enum drbd_packet cmd, struct twopc_reply *reply);
+void drbd_send_peers_in_sync(struct drbd_peer_device *peer_device, u64 mask,
+ sector_t sector, int size);
+int drbd_send_peer_dagtag(struct drbd_connection *connection,
+ struct drbd_connection *lost_peer);
+int drbd_send_flush_requests(struct drbd_connection *connection,
+ u64 flush_sequence);
+int drbd_send_flush_forward(struct drbd_connection *connection,
+ u64 flush_sequence, int initiator_node_id);
+int drbd_send_flush_requests_ack(struct drbd_connection *connection,
+ u64 flush_sequence, int primary_node_id);
+int drbd_send_enable_replication_next(struct drbd_peer_device *peer_device);
+int drbd_send_enable_replication(struct drbd_peer_device *peer_device, bool enable);
+int drbd_send_current_uuid(struct drbd_peer_device *peer_device,
+ u64 current_uuid, u64 weak_nodes);
+void drbd_backing_dev_free(struct drbd_device *device,
+ struct drbd_backing_dev *ldev);
+void drbd_print_uuids(struct drbd_peer_device *peer_device, const char *text);
+void drbd_queue_unplug(struct drbd_device *device);
+
+u64 drbd_capacity_to_on_disk_bm_sect(u64 capacity_sect, const struct drbd_md *md);
+void drbd_md_set_sector_offsets(struct drbd_backing_dev *bdev);
+int drbd_md_write(struct drbd_device *device,
+ struct meta_data_on_disk_9 *buffer);
+int drbd_md_sync(struct drbd_device *device);
+int drbd_md_sync_if_dirty(struct drbd_device *device);
+void drbd_uuid_received_new_current(struct drbd_peer_device *from_pd, u64 val,
+ u64 weak_nodes);
+void drbd_uuid_set_bitmap(struct drbd_peer_device *peer_device, u64 uuid);
+void __drbd_uuid_set_bitmap(struct drbd_peer_device *peer_device, u64 val);
+void _drbd_uuid_set_bitmap(struct drbd_peer_device *peer_device, u64 val);
+void _drbd_uuid_set_current(struct drbd_device *device, u64 val);
+enum drbd_mint_outcome drbd_uuid_new_current(struct drbd_device *device, bool forced);
+void drbd_uuid_new_current_by_user(struct drbd_device *device);
+void _drbd_uuid_push_history(struct drbd_device *device, u64 val);
+u64 _drbd_uuid_pull_history(struct drbd_peer_device *peer_device);
+void drbd_set_peer_bitmap_uuid(struct drbd_peer_md *peer_md, u64 bitmap_uuid, u64 dagtag);
+int drbd_day0_node_id(struct drbd_md *md, int exclude);
+void drbd_uuid_resync_starting(struct drbd_peer_device *peer_device);
+void drbd_uuid_resync_starting_source(struct drbd_peer_device *peer_device);
+u64 drbd_uuid_resync_finished(struct drbd_peer_device *peer_device);
+void drbd_uuid_detect_finished_resyncs(struct drbd_peer_device *peer_device);
+bool drbd_uuid_set_exposed(struct drbd_device *device, u64 val, bool log);
+u64 drbd_weak_nodes_device(struct drbd_device *device);
+bool drbd_uuid_is_day0(struct drbd_device *device);
+int drbd_md_test_flag(struct drbd_backing_dev *bdev, enum mdf_flag flag);
+void drbd_md_set_peer_flag(struct drbd_peer_device *peer_device,
+ enum mdf_peer_flag_bit flag_bit);
+void drbd_md_clear_peer_flag(struct drbd_peer_device *peer_device,
+ enum mdf_peer_flag_bit flag_bit);
+bool drbd_md_test_peer_flag(struct drbd_peer_device *peer_device,
+ enum mdf_peer_flag_bit flag_bit);
+void drbd_md_slot_emptied(struct drbd_device *device, int bitmap_index);
+void drbd_md_set_bitmaps_authoritative(struct drbd_device *device);
+void drbd_md_mark_dirty(struct drbd_device *device);
+void drbd_queue_bitmap_io(struct drbd_device *device,
+ int (*io_fn)(struct drbd_device *device,
+ struct drbd_peer_device *peer_device),
+ void (*done)(struct drbd_device *device,
+ struct drbd_peer_device *peer_device,
+ int rv),
+ char *why, enum bm_flag flags,
+ struct drbd_peer_device *peer_device);
+int drbd_bitmap_io(struct drbd_device *device,
+ int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
+ char *why, enum bm_flag flags,
+ struct drbd_peer_device *peer_device);
+int drbd_bitmap_io_from_worker(struct drbd_device *device,
+ int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
+ char *why, enum bm_flag flags,
+ struct drbd_peer_device *peer_device);
+int drbd_bmio_set_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+int drbd_bmio_clear_all_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+int drbd_bmio_set_all_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+int drbd_bmio_set_allocated_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+int drbd_bmio_clear_one_peer(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+bool drbd_device_stable(struct drbd_device *device, u64 *authoritative_ptr);
+void drbd_flush_peer_acks(struct drbd_resource *resource);
+void drbd_cork(struct drbd_connection *connection, enum drbd_stream stream);
+int drbd_uncork(struct drbd_connection *connection, enum drbd_stream stream);
+void drbd_open_counts(struct drbd_resource *resource, int *rw_count_ptr,
+ int *ro_count_ptr);
+
+struct drbd_connection *
+__drbd_next_connection_ref(u64 *visited, struct drbd_connection *connection,
+ struct drbd_resource *resource);
+
+struct drbd_peer_device *
+__drbd_next_peer_device_ref(u64 *visited,
+ struct drbd_peer_device *peer_device,
+ struct drbd_device *device);
+
+void tl_abort_disk_io(struct drbd_device *device);
+
+sector_t drbd_get_max_capacity(struct drbd_device *device,
+ struct drbd_backing_dev *bdev, bool warn);
+sector_t drbd_partition_data_capacity(struct drbd_device *device);
/* Meta data layout
*
@@ -1151,59 +2380,10 @@ extern int drbd_bmio_clear_n_write(struct drbd_device *device,
* but is about to become configurable.
*/
-/* Our old fixed size meta data layout
- * allows up to about 3.8TB, so if you want more,
- * you need to use the "flexible" meta data format. */
-#define MD_128MB_SECT (128LLU << 11) /* 128 MB, unit sectors */
-#define MD_4kB_SECT 8
-#define MD_32kB_SECT 64
-
/* One activity log extent represents 4M of storage */
#define AL_EXTENT_SHIFT 22
#define AL_EXTENT_SIZE (1<<AL_EXTENT_SHIFT)
-/* We could make these currently hardcoded constants configurable
- * variables at create-md time (or even re-configurable at runtime?).
- * Which will require some more changes to the DRBD "super block"
- * and attach code.
- *
- * updates per transaction:
- * This many changes to the active set can be logged with one transaction.
- * This number is arbitrary.
- * context per transaction:
- * This many context extent numbers are logged with each transaction.
- * This number is resulting from the transaction block size (4k), the layout
- * of the transaction header, and the number of updates per transaction.
- * See drbd_actlog.c:struct al_transaction_on_disk
- * */
-#define AL_UPDATES_PER_TRANSACTION 64 // arbitrary
-#define AL_CONTEXT_PER_TRANSACTION 919 // (4096 - 36 - 6*64)/4
-
-#if BITS_PER_LONG == 32
-#define LN2_BPL 5
-#define cpu_to_lel(A) cpu_to_le32(A)
-#define lel_to_cpu(A) le32_to_cpu(A)
-#elif BITS_PER_LONG == 64
-#define LN2_BPL 6
-#define cpu_to_lel(A) cpu_to_le64(A)
-#define lel_to_cpu(A) le64_to_cpu(A)
-#else
-#error "LN2 of BITS_PER_LONG unknown!"
-#endif
-
-/* resync bitmap */
-/* 16MB sized 'bitmap extent' to track syncer usage */
-struct bm_extent {
- int rs_left; /* number of bits set (out of sync) in this extent. */
- int rs_failed; /* number of failed resync requests in this extent. */
- unsigned long flags;
- struct lc_element lce;
-};
-
-#define BME_NO_WRITES 0 /* bm_extent.flags: no more requests on this one! */
-#define BME_LOCKED 1 /* bm_extent.flags: syncer active on this one. */
-#define BME_PRIORITY 2 /* finish resync IO on this extent ASAP! App IO waiting! */
-
/* drbd_bitmap.c */
/*
* We need to store one bit for a block.
@@ -1212,94 +2392,119 @@ struct bm_extent {
* Bit 1 ==> local node thinks this block needs to be synced.
*/
-#define SLEEP_TIME (HZ/10)
-
-/* We do bitmap IO in units of 4k blocks.
- * We also still have a hardcoded 4k per bit relation. */
-#define BM_BLOCK_SHIFT 12 /* 4k per bit */
-#define BM_BLOCK_SIZE (1<<BM_BLOCK_SHIFT)
-/* mostly arbitrarily set the represented size of one bitmap extent,
- * aka resync extent, to 16 MiB (which is also 512 Byte worth of bitmap
- * at 4k per bit resolution) */
-#define BM_EXT_SHIFT 24 /* 16 MiB per resync extent */
-#define BM_EXT_SIZE (1<<BM_EXT_SHIFT)
-
-#if (BM_EXT_SHIFT != 24) || (BM_BLOCK_SHIFT != 12)
-#error "HAVE YOU FIXED drbdmeta AS WELL??"
-#endif
-
-/* thus many _storage_ sectors are described by one bit */
-#define BM_SECT_TO_BIT(x) ((x)>>(BM_BLOCK_SHIFT-9))
-#define BM_BIT_TO_SECT(x) ((sector_t)(x)<<(BM_BLOCK_SHIFT-9))
-#define BM_SECT_PER_BIT BM_BIT_TO_SECT(1)
-
-/* bit to represented kilo byte conversion */
-#define Bit2KB(bits) ((bits)<<(BM_BLOCK_SHIFT-10))
-
-/* in which _bitmap_ extent (resp. sector) the bit for a certain
- * _storage_ sector is located in */
-#define BM_SECT_TO_EXT(x) ((x)>>(BM_EXT_SHIFT-9))
-#define BM_BIT_TO_EXT(x) ((x) >> (BM_EXT_SHIFT - BM_BLOCK_SHIFT))
+#define RS_MAKE_REQS_INTV (HZ/10)
+/* Consecutive make_resync_request clamped by max-buffers before we say so:
+ * enough that a ramp-up or a brief stall stays quiet.
+ */
+#define RS_MXB_LIMITED_ROUNDS 100
-/* first storage sector a bitmap extent corresponds to */
-#define BM_EXT_TO_SECT(x) ((sector_t)(x) << (BM_EXT_SHIFT-9))
-/* how much _storage_ sectors we have per bitmap extent */
-#define BM_SECT_PER_EXT BM_EXT_TO_SECT(1)
-/* how many bits are covered by one bitmap extent (resync extent) */
-#define BM_BITS_PER_EXT (1UL << (BM_EXT_SHIFT - BM_BLOCK_SHIFT))
+/* The queueing delay a resync may add at the backing device: it keeps at most
+ * as much work queued there as the device works off in "this long".
+ * Arbitrary "best guess" value for now,
+ * slightly more than a default "jiffy" at HZ 250.
+ */
+#define RS_DEPTH_TARGET_MS 5
+/* ... and never less than eight 4 KiB requests, so the device stays fed. */
+#define RS_DEPTH_MIN_SECT 64
+/* Busy time the drain rate is measured over. Of the same order as the budget
+ * it sizes, a resync starting up should leave RS_DEPTH_MIN_SECT quickly.
+ */
+#define RS_DRAIN_SAMPLE_NS NSEC_PER_MSEC
+/* How often the backing device's service time is probed again, and the window
+ * over which resync completion latency is watched.
+ */
+#define RS_LAT_PROBE_INTV (10 * HZ)
+#define RS_LAT_WINDOW HZ
+/* A longer service time is a stalled device, not normal operation. */
+#define RS_LAT_MAX_NS (60ULL * NSEC_PER_SEC)
+/* No backing device works off resync IO faster (64 GiB/s, in sectors). */
+#define RS_DRAIN_RATE_MAX (1U << 27)
+
+enum rs_lat_probe {
+ RS_LAT_IDLE, /* watching completion latency */
+ RS_LAT_DRAIN, /* resync IO only into an empty queue; the first is the probe */
+ RS_LAT_TIMING, /* the probe is at the device */
+};
+#define RS_MAKE_REQS_INTV_NS (NSEC_PER_SEC/10)
-#define BM_BLOCKS_PER_BM_EXT_MASK (BM_BITS_PER_EXT - 1)
+#define LEGACY_BM_EXT_SHIFT 27 /* 128 MiB per resync extent */
+#define LEGACY_BM_EXT_SECT_MASK ((1UL << (LEGACY_BM_EXT_SHIFT - SECTOR_SHIFT)) - 1)
+static inline unsigned int bm_block_size(const struct drbd_bitmap *bm)
+{
+ return 1 << bm->bm_block_shift;
+}
+static inline sector_t bm_bit_to_kb(const struct drbd_bitmap *bm, unsigned long bit)
+{
+ return (sector_t)bit << (bm->bm_block_shift - 10);
+}
+static inline unsigned long bm_sect_to_bit(const struct drbd_bitmap *bm, sector_t s)
+{
+ return s >> (bm->bm_block_shift - 9);
+}
+static inline sector_t bm_bit_to_sect(const struct drbd_bitmap *bm, unsigned long bit)
+{
+ return (sector_t)bit << (bm->bm_block_shift - 9);
+}
+static inline sector_t bm_sect_per_bit(const struct drbd_bitmap *bm)
+{
+ return (sector_t)1 << (bm->bm_block_shift - 9);
+}
-/* in one sector of the bitmap, we have this many activity_log extents. */
-#define AL_EXT_PER_BM_SECT (1 << (BM_EXT_SHIFT - AL_EXTENT_SHIFT))
+static inline sector_t bit_to_kb(unsigned long bit, unsigned int bm_block_shift)
+{
+ return (sector_t)bit << (bm_block_shift - 10);
+}
+static inline unsigned long sect_to_bit(sector_t s, unsigned int bm_block_shift)
+{
+ return s >> (bm_block_shift - 9);
+}
+static inline sector_t bit_to_sect(unsigned long bit, unsigned int bm_block_shift)
+{
+ return (sector_t)bit << (bm_block_shift - 9);
+}
+static inline sector_t sect_per_bit(unsigned int bm_block_shift)
+{
+ return (sector_t)1 << (bm_block_shift - 9);
+}
-/* the extent in "PER_EXTENT" below is an activity log extent
- * we need that many (long words/bytes) to store the bitmap
- * of one AL_EXTENT_SIZE chunk of storage.
- * we can store the bitmap for that many AL_EXTENTS within
- * one sector of the _on_disk_ bitmap:
- * bit 0 bit 37 bit 38 bit (512*8)-1
- * ...|........|........|.. // ..|........|
- * sect. 0 `296 `304 ^(512*8*8)-1
- *
-#define BM_WORDS_PER_EXT ( (AL_EXT_SIZE/BM_BLOCK_SIZE) / BITS_PER_LONG )
-#define BM_BYTES_PER_EXT ( (AL_EXT_SIZE/BM_BLOCK_SIZE) / 8 ) // 128
-#define BM_EXT_PER_SECT ( 512 / BM_BYTES_PER_EXTENT ) // 4
+/* We may have just lost our backing device, and with it ->ldev and ->bitmap.
+ * But we can still report sync progress and similar based on our last known
+ * bitmap block size.
*/
+static inline sector_t device_bit_to_kb(struct drbd_device *device, unsigned long bit)
+{
+ return bit_to_kb(bit, device->last_bm_block_shift);
+}
+
+/* Send P_PEERS_IN_SYNC in steps defined by this shift. Set to the activity log
+ * extent shift since the P_PEERS_IN_SYNC intervals are broken up based on
+ * activity log extents anyway. */
+#define PEERS_IN_SYNC_STEP_SHIFT AL_EXTENT_SHIFT
+#define PEERS_IN_SYNC_STEP_SECT (1UL << (PEERS_IN_SYNC_STEP_SHIFT - SECTOR_SHIFT))
+#define PEERS_IN_SYNC_STEP_SECT_MASK (PEERS_IN_SYNC_STEP_SECT - 1)
-#define DRBD_MAX_SECTORS_32 (0xffffffffLU)
-/* we have a certain meta data variant that has a fixed on-disk size of 128
- * MiB, of which 4k are our "superblock", and 32k are the fixed size activity
+/* Indexed external meta data has a fixed on-disk size of 128MiB, of which
+ * 4KiB are our "superblock", and 32KiB are the fixed size activity
* log, leaving this many sectors for the bitmap.
*/
+#define DRBD_BM_SECTORS_INDEXED \
+ (((128 << 20) - (32 << 10) - (4 << 10)) >> SECTOR_SHIFT)
-#define DRBD_MAX_SECTORS_FIXED_BM \
- ((MD_128MB_SECT - MD_32kB_SECT - MD_4kB_SECT) * (1LL<<(BM_EXT_SHIFT-9)))
-#define DRBD_MAX_SECTORS DRBD_MAX_SECTORS_FIXED_BM
-/* 16 TB in units of sectors */
#if BITS_PER_LONG == 32
-/* adjust by one page worth of bitmap,
- * so we won't wrap around in drbd_bm_find_next_bit.
- * you should use 64bit OS for that much storage, anyways. */
-#define DRBD_MAX_SECTORS_FLEX BM_BIT_TO_SECT(0xffff7fff)
+#if !defined(CONFIG_LBDAF) && !defined(CONFIG_LBD)
+#define DRBD_MAX_SECTORS (0xffffffffLU)
#else
-/* we allow up to 1 PiB now on 64bit architecture with "flexible" meta data */
-#define DRBD_MAX_SECTORS_FLEX (1UL << 51)
-/* corresponds to (1UL << 38) bits right now. */
+/* With large block device support, the size is limited by the fact that we
+ * want to be able to address bitmap bits with a long. Additionally adjust by
+ * one page worth of bitmap, so we don't wrap around when iterating. */
+#define DRBD_MAX_SECTORS BM_BIT_TO_SECT(0xffff7fff)
#endif
-
-/* Estimate max bio size as 256 * PAGE_SIZE,
- * so for typical PAGE_SIZE of 4k, that is (1<<20) Byte.
- * Since we may live in a mixed-platform cluster,
- * we limit us to a platform agnostic constant here for now.
- * A followup commit may allow even bigger BIO sizes,
- * once we thought that through. */
-#define DRBD_MAX_BIO_SIZE (1U << 20)
-#if DRBD_MAX_BIO_SIZE > (BIO_MAX_VECS << PAGE_SHIFT)
-#error Architecture not supported: DRBD_MAX_BIO_SIZE > BIO_MAX_SIZE
+#else
+/* We allow up to 1 PiB on 64 bit architectures as long as our meta data
+ * is large enough. */
+#define DRBD_MAX_SECTORS (1UL << (50 - SECTOR_SHIFT))
#endif
-#define DRBD_MAX_BIO_SIZE_SAFE (1U << 12) /* Works always = 4k */
#define DRBD_MAX_SIZE_H80_PACKET (1U << 15) /* Header 80 only allows packets up to 32KiB data */
#define DRBD_MAX_BIO_SIZE_P95 (1U << 17) /* Protocol 95 to 99 allows bios up to 128KiB */
@@ -1310,61 +2515,98 @@ struct bm_extent {
#define DRBD_MAX_BATCH_BIO_SIZE (AL_UPDATES_PER_TRANSACTION/2*AL_EXTENT_SIZE)
#define DRBD_MAX_BBIO_SECTORS (DRBD_MAX_BATCH_BIO_SIZE >> 9)
-extern int drbd_bm_init(struct drbd_device *device);
-extern int drbd_bm_resize(struct drbd_device *device, sector_t sectors, int set_new_bits);
-extern void drbd_bm_cleanup(struct drbd_device *device);
-extern void drbd_bm_set_all(struct drbd_device *device);
-extern void drbd_bm_clear_all(struct drbd_device *device);
+/* This gets ignored if the backing device has a larger discard granularity */
+#define DRBD_MAX_RS_DISCARD_SIZE (1U << 27) /* 128MiB; arbitrary */
+
+/* how many activity log extents are touched by this interval? */
+static inline int interval_to_al_extents(struct drbd_interval *i)
+{
+ unsigned int first = i->sector >> (AL_EXTENT_SHIFT-9);
+ unsigned int last = i->size == 0 ? first : (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
+ return 1 + last - first; /* worst case: all touched extends are cold. */
+}
+
+struct drbd_bitmap *drbd_bm_alloc(unsigned int max_peers, unsigned int bm_block_shift);
+int drbd_bm_resize(struct drbd_device *device, struct drbd_bitmap *bitmap,
+ sector_t capacity, bool set_new_bits);
+void drbd_bm_free(struct drbd_device *device);
+void drbd_bm_set_all(struct drbd_device *device);
+void drbd_bm_clear_all(struct drbd_device *device);
/* set/clear/test only a few bits at a time */
-extern int drbd_bm_set_bits(
- struct drbd_device *device, unsigned long s, unsigned long e);
-extern int drbd_bm_clear_bits(
- struct drbd_device *device, unsigned long s, unsigned long e);
-extern int drbd_bm_count_bits(
- struct drbd_device *device, const unsigned long s, const unsigned long e);
+unsigned int drbd_bm_set_bits(struct drbd_device *device,
+ unsigned int bitmap_index, unsigned long start,
+ unsigned long end);
+unsigned int drbd_bm_clear_bits(struct drbd_device *device,
+ unsigned int bitmap_index,
+ unsigned long start, unsigned long end);
+int drbd_bm_count_bits(struct drbd_device *device, unsigned int bitmap_index,
+ unsigned long s, unsigned long e);
/* bm_set_bits variant for use while holding drbd_bm_lock,
* may process the whole bitmap in one go */
-extern void _drbd_bm_set_bits(struct drbd_device *device,
- const unsigned long s, const unsigned long e);
-extern int drbd_bm_test_bit(struct drbd_device *device, unsigned long bitnr);
-extern int drbd_bm_e_weight(struct drbd_device *device, unsigned long enr);
-extern int drbd_bm_read(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local);
-extern void drbd_bm_mark_for_writeout(struct drbd_device *device, int page_nr);
-extern int drbd_bm_write(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local);
-extern void drbd_bm_reset_al_hints(struct drbd_device *device) __must_hold(local);
-extern int drbd_bm_write_hinted(struct drbd_device *device) __must_hold(local);
-extern int drbd_bm_write_lazy(struct drbd_device *device, unsigned upper_idx) __must_hold(local);
-extern int drbd_bm_write_all(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local);
-extern int drbd_bm_write_copy_pages(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local);
-extern size_t drbd_bm_words(struct drbd_device *device);
-extern unsigned long drbd_bm_bits(struct drbd_device *device);
-extern sector_t drbd_bm_capacity(struct drbd_device *device);
+void drbd_bm_set_many_bits(struct drbd_peer_device *peer_device,
+ unsigned long start, unsigned long end);
+void drbd_bm_clear_many_bits(struct drbd_peer_device *peer_device,
+ unsigned long start, unsigned long end);
+void _drbd_bm_clear_many_bits(struct drbd_device *device, int bitmap_index,
+ unsigned long start, unsigned long end);
+void _drbd_bm_set_many_bits(struct drbd_device *device, int bitmap_index,
+ unsigned long start, unsigned long end);
+int drbd_bm_read(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+void drbd_bm_reset_al_hints(struct drbd_device *device);
+void drbd_bm_mark_range_for_writeout(struct drbd_device *device,
+ unsigned long start, unsigned long end);
+int drbd_bm_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+int drbd_bm_write_hinted(struct drbd_device *device);
+int drbd_bm_write_lazy(struct drbd_device *device, unsigned int upper_idx);
+int drbd_bm_write_all(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+int drbd_bm_write_copy_pages(struct drbd_device *device,
+ struct drbd_peer_device *peer_device);
+size_t drbd_bm_words(struct drbd_device *device);
+unsigned long drbd_bm_bits(struct drbd_device *device);
+unsigned long drbd_bm_bits_4k(struct drbd_device *device);
+sector_t drbd_bm_capacity(struct drbd_device *device);
#define DRBD_END_OF_BITMAP (~(unsigned long)0)
-extern unsigned long drbd_bm_find_next(struct drbd_device *device, unsigned long bm_fo);
+unsigned long drbd_bm_find_next(struct drbd_peer_device *peer_device,
+ unsigned long start);
/* bm_find_next variants for use while you hold drbd_bm_lock() */
-extern unsigned long _drbd_bm_find_next(struct drbd_device *device, unsigned long bm_fo);
-extern unsigned long _drbd_bm_find_next_zero(struct drbd_device *device, unsigned long bm_fo);
-extern unsigned long _drbd_bm_total_weight(struct drbd_device *device);
-extern unsigned long drbd_bm_total_weight(struct drbd_device *device);
+unsigned long _drbd_bm_find_next(struct drbd_peer_device *peer_device,
+ unsigned long start);
+unsigned long _drbd_bm_find_next_zero(struct drbd_peer_device *peer_device,
+ unsigned long start);
+unsigned long _drbd_bm_total_weight(struct drbd_device *device,
+ int bitmap_index);
+unsigned long drbd_bm_total_weight(struct drbd_peer_device *peer_device);
/* for receive_bitmap */
-extern void drbd_bm_merge_lel(struct drbd_device *device, size_t offset,
- size_t number, unsigned long *buffer);
+void drbd_bm_merge_lel(struct drbd_peer_device *peer_device, size_t offset,
+ size_t number, unsigned long *buffer);
/* for _drbd_send_bitmap */
-extern void drbd_bm_get_lel(struct drbd_device *device, size_t offset,
- size_t number, unsigned long *buffer);
-
-extern void drbd_bm_lock(struct drbd_device *device, char *why, enum bm_flag flags);
-extern void drbd_bm_unlock(struct drbd_device *device);
+void drbd_bm_get_lel(struct drbd_peer_device *peer_device, size_t offset,
+ size_t number, unsigned long *buffer);
+
+void drbd_bm_lock(struct drbd_device *device, const char *why,
+ enum bm_flag flags);
+void drbd_bm_unlock(struct drbd_device *device);
+/* For a bitmap that is not published in device->bitmap yet. */
+void _drbd_bm_lock(struct drbd_device *device, struct drbd_bitmap *b,
+ struct drbd_peer_device *peer_device, const char *why,
+ enum bm_flag flags);
+void _drbd_bm_unlock(struct drbd_device *device, struct drbd_bitmap *b);
+void drbd_bm_slot_lock(struct drbd_peer_device *peer_device, char *why,
+ enum bm_flag flags);
+void drbd_bm_slot_unlock(struct drbd_peer_device *peer_device);
+void drbd_bm_copy_slot(struct drbd_device *device, unsigned int from_index,
+ unsigned int to_index);
+void drbd_bm_or_slot(struct drbd_device *device, unsigned int from_index,
+ unsigned int to_index);
/* drbd_main.c */
+extern struct workqueue_struct *ping_ack_sender;
extern struct kmem_cache *drbd_request_cache;
extern struct kmem_cache *drbd_ee_cache; /* peer requests */
-extern struct kmem_cache *drbd_bm_ext_cache; /* bitmap extents */
extern struct kmem_cache *drbd_al_ext_cache; /* activity log extents */
extern mempool_t drbd_request_mempool;
extern mempool_t drbd_ee_mempool;
@@ -1385,38 +2627,87 @@ extern struct bio_set drbd_md_io_bio_set;
/* And a bio_set for cloning */
extern struct bio_set drbd_io_bio_set;
-extern struct mutex resources_mutex;
-
-extern enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsigned int minor);
-extern void drbd_destroy_device(struct kref *kref);
-extern void drbd_delete_device(struct drbd_device *device);
-
-extern struct drbd_resource *drbd_create_resource(const char *name);
-extern void drbd_free_resource(struct drbd_resource *resource);
-
-extern int set_resource_options(struct drbd_resource *resource, struct res_opts *res_opts);
-extern struct drbd_connection *conn_create(const char *name, struct res_opts *res_opts);
-extern void drbd_destroy_connection(struct kref *kref);
-extern struct drbd_connection *conn_get_by_addrs(void *my_addr, int my_addr_len,
- void *peer_addr, int peer_addr_len);
-extern struct drbd_resource *drbd_find_resource(const char *name);
-extern void drbd_destroy_resource(struct kref *kref);
-extern void conn_free_crypto(struct drbd_connection *connection);
+/* Peer request bios are allocated from drbd_peer_bio_set with this front pad. The
+ * block layer consumes bi_iter during I/O; drbd_submit_peer_request() saves
+ * what drbd_peer_request_endio() needs to restore it here.
+ */
+struct drbd_peer_bio {
+ unsigned int size; /* bi_iter.bi_size at submit */
+ unsigned int sector_offset; /* bi_sector - peer_req->i.sector at submit */
+ struct bio bio;
+};
+#define to_drbd_peer_bio(b) container_of(b, struct drbd_peer_bio, bio)
+extern struct bio_set drbd_peer_bio_set;
+
+struct drbd_peer_device *create_peer_device(struct drbd_device *device,
+ struct drbd_connection *connection);
+enum drbd_ret_code drbd_create_device(struct drbd_adm_ctx *adm_ctx,
+ unsigned int minor,
+ struct drbd_device_conf *device_conf,
+ struct drbd_device **p_device);
+void drbd_unregister_device(struct drbd_device *device);
+void drbd_reclaim_device(struct rcu_head *rp);
+void drbd_unregister_connection(struct drbd_connection *connection);
+void drbd_reclaim_connection(struct rcu_head *rp);
+void drbd_reclaim_path(struct rcu_head *rp);
+void del_connect_timer(struct drbd_connection *connection);
+
+struct drbd_resource *drbd_create_resource(const char *name,
+ struct drbd_res_opts *res_opts);
+void drbd_reclaim_resource(struct rcu_head *rp);
+struct drbd_resource *drbd_find_resource(const char *name);
+void drbd_destroy_resource(struct kref *kref);
+
+void drbd_destroy_device(struct kref *kref);
+
+int set_resource_options(struct drbd_resource *resource,
+ struct drbd_res_opts *res_opts, const char *tag);
+struct drbd_connection *drbd_create_connection(struct drbd_resource *resource,
+ struct drbd_transport_class *tc);
+void drbd_transport_shutdown(struct drbd_connection *connection,
+ enum drbd_tr_free_op op);
+void drbd_destroy_connection(struct kref *kref);
+void conn_free_crypto(struct drbd_connection *connection);
/* drbd_req */
-extern void do_submit(struct work_struct *ws);
-extern void __drbd_make_request(struct drbd_device *, struct bio *);
+void drbd_do_submit_conflict(struct work_struct *ws);
+void do_submit(struct work_struct *ws);
+#ifndef CONFIG_DRBD_TIMING_STATS
+#define __drbd_make_request(d, b, k, j) __drbd_make_request(d, b, j)
+#endif
+void __drbd_make_request(struct drbd_device *device, struct bio *bio,
+ ktime_t start_kt, unsigned long start_jif);
void drbd_submit_bio(struct bio *bio);
-/* drbd_nl.c */
-
-extern struct mutex notification_mutex;
+enum drbd_force_detach_flags {
+ DRBD_READ_ERROR,
+ DRBD_WRITE_ERROR,
+ DRBD_META_IO_ERROR,
+ DRBD_FORCE_DETACH,
+};
+#define drbd_handle_io_error(m, f) drbd_handle_io_error_(m, f, __func__)
+void drbd_handle_io_error_(struct drbd_device *device,
+ enum drbd_force_detach_flags df, const char *where);
-extern void drbd_suspend_io(struct drbd_device *device);
-extern void drbd_resume_io(struct drbd_device *device);
-extern char *ppsize(char *buf, unsigned long long size);
-extern sector_t drbd_new_dev_size(struct drbd_device *, struct drbd_backing_dev *, sector_t, int);
+/* drbd_nl.c */
+enum suspend_scope {
+ READ_AND_WRITE,
+ WRITE_ONLY
+};
+void __drbd_suspend_io(struct drbd_device *device);
+void drbd_suspend_io(struct drbd_device *device, enum suspend_scope ss);
+bool drbd_wait_io_drained(struct drbd_device *device, enum suspend_scope ss, long timeout);
+/* in drbd_state.c */
+void twopc_suspend_io(struct drbd_device *device);
+void twopc_drain_io(struct drbd_device *device, enum suspend_scope ss);
+int drbd_suspend_io_interruptible(struct drbd_device *device, enum suspend_scope ss);
+void drbd_resume_io(struct drbd_device *device);
+char *ppsize(char *buf, unsigned long long size);
+sector_t drbd_new_dev_size(struct drbd_device *device, sector_t agreed_max_size,
+ sector_t user_capped_size, enum dds_flags flags);
enum determine_dev_size {
+ DS_2PC_ERR = -5,
+ DS_2PC_NOT_SUPPORTED = -4,
DS_ERROR_SHRINK = -3,
DS_ERROR_SPACE_MD = -2,
DS_ERROR = -1,
@@ -1425,96 +2716,248 @@ enum determine_dev_size {
DS_GREW = 2,
DS_GREW_FROM_ZERO = 3,
};
-extern enum determine_dev_size
-drbd_determine_dev_size(struct drbd_device *, enum dds_flags, struct resize_parms *) __must_hold(local);
-extern void resync_after_online_grow(struct drbd_device *);
-extern void drbd_reconsider_queue_parameters(struct drbd_device *device,
- struct drbd_backing_dev *bdev, struct o_qlim *o);
-extern enum drbd_state_rv drbd_set_role(struct drbd_device *device,
- enum drbd_role new_role,
- int force);
-extern bool conn_try_outdate_peer(struct drbd_connection *connection);
-extern void conn_try_outdate_peer_async(struct drbd_connection *connection);
-extern enum drbd_peer_state conn_khelper(struct drbd_connection *connection, char *cmd);
-extern int drbd_khelper(struct drbd_device *device, char *cmd);
-
-/* drbd_worker.c */
-/* bi_end_io handlers */
-extern void drbd_md_endio(struct bio *bio);
-extern void drbd_peer_request_endio(struct bio *bio);
-extern void drbd_request_endio(struct bio *bio);
-extern int drbd_worker(struct drbd_thread *thi);
+enum determine_dev_size
+drbd_determine_dev_size(struct drbd_device *device,
+ sector_t peer_current_size, enum dds_flags flags,
+ struct drbd_resize_parms *rs);
+void resync_after_online_grow(struct drbd_peer_device *peer_device);
+void drbd_reconsider_queue_parameters(struct drbd_device *device,
+ struct drbd_backing_dev *bdev);
+bool barrier_pending(struct drbd_resource *resource);
+enum drbd_state_rv
+drbd_set_role(struct drbd_resource *resource, enum drbd_role role,
+ enum chg_state_flags flags, const char *tag, struct drbd_adm_ctx *ctx);
+void conn_try_outdate_peer_async(struct drbd_connection *connection);
+int drbd_maybe_khelper(struct drbd_device *device,
+ struct drbd_connection *connection, char *cmd);
+int drbd_create_peer_device_default_config(struct drbd_peer_device *peer_device);
+int drbd_unallocated_index(struct drbd_backing_dev *bdev);
+void youngest_and_oldest_opener_to_str(struct drbd_device *device, char *buf,
+ size_t len);
+int param_set_drbd_strict_names(const char *val,
+ const struct kernel_param *kp);
+/* drbd_sender.c */
+int drbd_sender(struct drbd_thread *thi);
+int drbd_worker(struct drbd_thread *thi);
enum drbd_ret_code drbd_resync_after_valid(struct drbd_device *device, int o_minor);
void drbd_resync_after_changed(struct drbd_device *device);
-extern void drbd_start_resync(struct drbd_device *device, enum drbd_conns side);
-extern void resume_next_sg(struct drbd_device *device);
-extern void suspend_other_sg(struct drbd_device *device);
-extern int drbd_resync_finished(struct drbd_peer_device *peer_device);
+bool drbd_stable_sync_source_present(struct drbd_peer_device *except_peer_device,
+ enum which_state which);
+void drbd_start_resync(struct drbd_peer_device *peer_device,
+ enum drbd_repl_state side, const char *tag);
+void resume_next_sg(struct drbd_device *device);
+void suspend_other_sg(struct drbd_device *device);
+void drbd_apply_resync_max_parallel(void);
+void drbd_resync_finished(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state new_peer_disk_state);
+void verify_progress(struct drbd_peer_device *peer_device,
+ const sector_t sector, const unsigned int size);
/* maybe rather drbd_main.c ? */
-extern void *drbd_md_get_buffer(struct drbd_device *device, const char *intent);
-extern void drbd_md_put_buffer(struct drbd_device *device);
-extern int drbd_md_sync_page_io(struct drbd_device *device,
- struct drbd_backing_dev *bdev, sector_t sector, enum req_op op);
-extern void drbd_ov_out_of_sync_found(struct drbd_peer_device *peer_device,
- sector_t sector, int size);
-extern void wait_until_done_or_force_detached(struct drbd_device *device,
- struct drbd_backing_dev *bdev, unsigned int *done);
-extern void drbd_rs_controller_reset(struct drbd_peer_device *peer_device);
+void *drbd_md_get_buffer(struct drbd_device *device, const char *intent);
+void drbd_md_put_buffer(struct drbd_device *device);
+int drbd_md_sync_page_io(struct drbd_device *device,
+ struct drbd_backing_dev *bdev, sector_t sector,
+ enum req_op op);
+bool drbd_al_active(struct drbd_device *device, sector_t sector,
+ unsigned int size);
+void drbd_ov_out_of_sync_found(struct drbd_peer_device *peer_device,
+ sector_t sector, int size);
+long drbd_disk_timeout(struct drbd_backing_dev *bdev);
+void wait_until_done_or_force_detached(struct drbd_device *device,
+ struct drbd_backing_dev *bdev,
+ unsigned int *done);
+void drbd_rs_controller_reset(struct drbd_peer_device *peer_device);
+void drbd_rs_all_in_flight_came_back(struct drbd_peer_device *peer_device,
+ int rs_sect_in);
+void drbd_check_peers(struct drbd_resource *resource);
+enum drbd_mint_outcome drbd_check_peers_new_current_uuid(struct drbd_device *device);
+void drbd_conflict_send_resync_request(struct drbd_peer_request *peer_req);
+void drbd_ping_peer(struct drbd_connection *connection);
+struct drbd_peer_device *peer_device_by_node_id(struct drbd_device *device,
+ int node_id);
+void drbd_update_mdf_al_disabled(struct drbd_device *device,
+ enum which_state which);
static inline void ov_out_of_sync_print(struct drbd_peer_device *peer_device)
{
- struct drbd_device *device = peer_device->device;
+ if (peer_device->ov_last_oos_size) {
+ drbd_err(peer_device, "Out of sync: start=%llu, size=%lu (sectors)\n",
+ (unsigned long long)peer_device->ov_last_oos_start,
+ (unsigned long)peer_device->ov_last_oos_size);
+ }
+ peer_device->ov_last_oos_size = 0;
+}
- if (device->ov_last_oos_size) {
- drbd_err(peer_device, "Out of sync: start=%llu, size=%lu (sectors)\n",
- (unsigned long long)device->ov_last_oos_start,
- (unsigned long)device->ov_last_oos_size);
+static inline void ov_skipped_print(struct drbd_peer_device *peer_device)
+{
+ if (peer_device->ov_last_skipped_size) {
+ drbd_info(peer_device, "Skipped verify, too busy: start=%llu, size=%lu (sectors)\n",
+ (unsigned long long)peer_device->ov_last_skipped_start,
+ (unsigned long)peer_device->ov_last_skipped_size);
}
- device->ov_last_oos_size = 0;
+ peer_device->ov_last_skipped_size = 0;
}
+void drbd_csum_bios(struct crypto_shash *tfm, struct bio_list *bios, void *digest);
+void drbd_csum_bio(struct crypto_shash *tfm, struct bio *bio, void *digest);
+void drbd_resync_read_req_mod(struct drbd_peer_request *peer_req,
+ enum drbd_interval_flags bit_to_set);
-extern void drbd_csum_bio(struct crypto_shash *, struct bio *, void *);
-extern void drbd_csum_ee(struct crypto_shash *, struct drbd_peer_request *,
- void *);
/* worker callbacks */
-extern int w_e_end_data_req(struct drbd_work *, int);
-extern int w_e_end_rsdata_req(struct drbd_work *, int);
-extern int w_e_end_csum_rs_req(struct drbd_work *, int);
-extern int w_e_end_ov_reply(struct drbd_work *, int);
-extern int w_e_end_ov_req(struct drbd_work *, int);
-extern int w_ov_finished(struct drbd_work *, int);
-extern int w_resync_timer(struct drbd_work *, int);
-extern int w_send_write_hint(struct drbd_work *, int);
-extern int w_send_dblock(struct drbd_work *, int);
-extern int w_send_read_req(struct drbd_work *, int);
-extern int w_restart_disk_io(struct drbd_work *, int);
-extern int w_send_out_of_sync(struct drbd_work *, int);
-
-extern void resync_timer_fn(struct timer_list *t);
-extern void start_resync_timer_fn(struct timer_list *t);
-
-extern void drbd_endio_write_sec_final(struct drbd_peer_request *peer_req);
+int w_e_end_data_req(struct drbd_work *w, int cancel);
+int w_e_end_rsdata_req(struct drbd_work *w, int cancel);
+int w_e_end_ov_reply(struct drbd_work *w, int cancel);
+int w_e_end_ov_req(struct drbd_work *w, int cancel);
+int w_resync_timer(struct drbd_work *w, int cancel);
+int w_e_reissue(struct drbd_work *w, int cancel);
+int w_send_dagtag(struct drbd_work *w, int cancel);
+int w_send_uuids(struct drbd_work *w, int cancel);
+
+bool drbd_any_flush_pending(struct drbd_resource *resource);
+bool drbd_flush_connect_needs_restart(struct drbd_resource *resource);
+void drbd_flush_ack_for_lost_peer(struct drbd_connection *lost_connection);
+void resync_timer_fn(struct timer_list *t);
+void start_resync_timer_fn(struct timer_list *t);
+
+int drbd_unmerge_discard(struct drbd_peer_request *peer_req_main,
+ struct list_head *list);
+void drbd_endio_write_sec_final(struct drbd_peer_request *peer_req);
+
+/* bi_end_io handlers */
+void drbd_md_endio(struct bio *bio);
+void drbd_peer_request_endio(struct bio *bio);
+void drbd_request_endio(struct bio *bio);
+
+void __update_timing_details(
+ struct drbd_thread_timing_details *tdp,
+ unsigned int *cb_nr,
+ void *cb,
+ const char *fn, const unsigned int line);
+
+#define update_sender_timing_details(c, cb) \
+ __update_timing_details(c->s_timing_details, &c->s_cb_nr, cb, __func__, __LINE__)
+#define update_receiver_timing_details(c, cb) \
+ __update_timing_details(c->r_timing_details, &c->r_cb_nr, cb, __func__, __LINE__)
+#define update_worker_timing_details(r, cb) \
+ __update_timing_details(r->w_timing_details, &r->w_cb_nr, cb, __func__, __LINE__)
/* drbd_receiver.c */
-extern int drbd_issue_discard_or_zero_out(struct drbd_device *device,
- sector_t start, unsigned int nr_sectors, int flags);
-extern int drbd_receiver(struct drbd_thread *thi);
-extern int drbd_ack_receiver(struct drbd_thread *thi);
-extern void drbd_send_acks_wf(struct work_struct *ws);
-extern bool drbd_rs_c_min_rate_throttle(struct drbd_device *device);
-extern bool drbd_rs_should_slow_down(struct drbd_peer_device *peer_device, sector_t sector,
- bool throttle_if_app_is_waiting);
-extern int drbd_submit_peer_request(struct drbd_peer_request *peer_req);
-extern int drbd_free_peer_reqs(struct drbd_device *, struct list_head *);
-extern struct drbd_peer_request *drbd_alloc_peer_req(struct drbd_peer_device *, u64,
- sector_t, unsigned int,
- unsigned int,
- gfp_t) __must_hold(local);
-extern void drbd_free_peer_req(struct drbd_device *device, struct drbd_peer_request *req);
-extern struct page *drbd_alloc_pages(struct drbd_peer_device *, unsigned int, bool);
-extern void _drbd_clear_done_ee(struct drbd_device *device, struct list_head *to_be_freed);
-extern int drbd_connected(struct drbd_peer_device *);
+struct packet_info {
+ enum drbd_packet cmd;
+ unsigned int size;
+ int vnr;
+ void *data;
+};
+
+/* packet_info->data is just a pointer into some temporary buffer
+ * owned by the transport. As soon as we call into the transport for
+ * any further receive operation, the data it points to is undefined.
+ * The buffer may be freed/recycled/re-used already.
+ * Convert and store the relevant information for any incoming data
+ * in drbd_peer_request_detail.
+ */
+
+struct drbd_peer_request_details {
+ uint64_t sector; /* be64_to_cpu(p_data.sector) */
+ uint64_t block_id; /* unmodified p_data.block_id */
+ uint32_t peer_seq; /* be32_to_cpu(p_data.seq_num) */
+ uint32_t dp_flags; /* be32_to_cpu(p_data.dp_flags) */
+ uint32_t length; /* endian converted p_head*.length */
+ uint32_t bi_size; /* resulting bio size */
+ /* for non-discards: bi_size = length - digest_size */
+ uint32_t digest_size;
+};
+
+
+void drbd_queue_update_peers(struct drbd_peer_device *peer_device,
+ sector_t sector_start, sector_t sector_end);
+int drbd_issue_discard_or_zero_out(struct drbd_device *device, sector_t start,
+ unsigned int nr_sectors, int flags);
+int drbd_send_ack_be(struct drbd_peer_device *peer_device,
+ enum drbd_packet cmd, sector_t sector, int size,
+ u64 block_id);
+int drbd_send_ack(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
+ struct drbd_peer_request *peer_req);
+int drbd_send_ov_result(struct drbd_peer_device *peer_device, sector_t sector,
+ int blksize, u64 block_id, enum ov_result result);
+int drbd_receiver(struct drbd_thread *thi);
+void drbd_unsuccessful_resync_request(struct drbd_peer_request *peer_req,
+ bool failed);
+int drbd_send_out_of_sync_wf(struct drbd_work *w, int cancel);
+void drbd_free_send_oos_leftovers(struct drbd_resource *resource);
+int drbd_flush_ack_wf(struct drbd_work *w, int unused);
+void drbd_send_ping_wf(struct work_struct *ws);
+void drbd_queue_ping(struct drbd_connection *connection);
+void drbd_queue_ping_ack(struct drbd_connection *connection);
+void drbd_send_acks_wf(struct work_struct *ws);
+void drbd_send_peer_ack_wf(struct work_struct *ws);
+bool drbd_rs_c_min_rate_throttle(struct drbd_peer_device *peer_device);
+void drbd_predict_peer_upgrade(struct drbd_peer_device *peer_device);
+void drbd_verify_skipped_block(struct drbd_peer_device *peer_device,
+ const sector_t sector, const unsigned int size);
+void drbd_conflict_submit_resync_request(struct drbd_peer_request *peer_req);
+void drbd_rs_depth_release(struct drbd_device *device);
+void drbd_rs_depth_queued(struct drbd_device *device, unsigned int sect);
+void drbd_rs_depth_completed(struct drbd_device *device, unsigned int sect,
+ unsigned long submit_jif);
+void drbd_conflict_submit_peer_read(struct drbd_peer_request *peer_req);
+bool drbd_rs_depth_exceeded(struct drbd_peer_device *peer_device);
+void drbd_rs_depth_defer(struct drbd_peer_device *peer_device);
+void drbd_rs_depth_exempt(struct drbd_device *device, struct drbd_interval *interval);
+void drbd_conflict_submit_peer_write(struct drbd_peer_request *peer_req);
+int drbd_submit_peer_request(struct drbd_peer_request *peer_req);
+void drbd_cleanup_after_failed_submit_peer_write(struct drbd_peer_request *peer_req);
+void drbd_cleanup_peer_requests_wfa(struct drbd_device *device,
+ struct list_head *cleanup);
+void drbd_remove_peer_req_interval(struct drbd_peer_request *peer_req);
+int drbd_free_peer_reqs(struct drbd_connection *connection,
+ struct list_head *list);
+struct drbd_peer_request *drbd_alloc_peer_req(struct drbd_peer_device *peer_device, gfp_t gfp_mask,
+ size_t size, blk_opf_t opf);
+void drbd_free_peer_req(struct drbd_peer_request *peer_req);
+void drbd_peer_req_strip_bio(struct drbd_peer_request *peer_req);
+int drbd_connected(struct drbd_peer_device *peer_device);
+
+/* drbd_transport.c */
+void drbd_connection_update_rx_alignment(struct drbd_resource *resource,
+ struct drbd_connection *connection);
+void drbd_resource_update_rx_alignment(struct drbd_resource *resource);
+void conn_connect2(struct drbd_connection *connection);
+void wait_initial_states_received(struct drbd_connection *connection);
+void abort_connect(struct drbd_connection *connection);
+void drbd_print_cluster_wide_state_change(struct drbd_resource *resource,
+ const char *message,
+ unsigned int tid,
+ unsigned int initiator_node_id,
+ int target_node_id,
+ union drbd_state mask,
+ union drbd_state val);
+void apply_unacked_peer_requests(struct drbd_connection *connection);
+void drbd_refuse_unsecured_writes(struct drbd_peer_device *source);
+void drbd_dagtag_wait_reqs_source_gone(struct drbd_peer_device *requester);
+struct drbd_connection *drbd_connection_by_node_id(struct drbd_resource *resource,
+ int node_id);
+struct drbd_connection *drbd_get_connection_by_node_id(struct drbd_resource *resource,
+ int node_id);
+bool drbd_have_local_disk(struct drbd_resource *resource);
+enum drbd_state_rv drbd_support_2pc_resize(struct drbd_resource *resource);
+enum determine_dev_size
+drbd_commit_size_change(struct drbd_device *device, struct drbd_resize_parms *rs,
+ u64 nodes_to_reach);
+void drbd_try_get_resynced_work_fn(struct work_struct *ws);
+bool diskless_primary_can_replay_to(struct drbd_peer_device *peer_device);
+u64 diskless_primary_present_current_uuid(struct drbd_peer_device *peer_device);
+bool peer_data_is_ancestor_of_mine(struct drbd_peer_device *peer_device);
+void drbd_process_rs_discards(struct drbd_peer_device *peer_device,
+ bool submit_all);
+void drbd_last_resync_request(struct drbd_peer_device *peer_device,
+ bool submit_all);
+void drbd_init_connect_state(struct drbd_connection *connection);
+
+static inline sector_t drbd_get_capacity(struct block_device *bdev)
+{
+ return bdev ? bdev_nr_sectors(bdev) : 0;
+}
/* sets the number of 512 byte sectors of our virtual device */
void drbd_set_my_capacity(struct drbd_device *device, sector_t size);
@@ -1525,208 +2968,127 @@ void drbd_set_my_capacity(struct drbd_device *device, sector_t size);
static inline void drbd_submit_bio_noacct(struct drbd_device *device,
int fault_type, struct bio *bio)
{
- __release(local);
- if (!bio->bi_bdev) {
- drbd_err(device, "drbd_submit_bio_noacct: bio->bi_bdev == NULL\n");
+ if (drbd_insert_fault(device, fault_type)) {
bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
- return;
- }
-
- if (drbd_insert_fault(device, fault_type))
- bio_io_error(bio);
- else
+ } else {
submit_bio_noacct(bio);
+ }
}
void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backing_dev *bdev,
enum write_ordering_e wo);
+void twopc_timer_fn(struct timer_list *t);
+void connect_timer_fn(struct timer_list *t);
+
/* drbd_proc.c */
extern struct proc_dir_entry *drbd_proc;
int drbd_seq_show(struct seq_file *seq, void *v);
/* drbd_actlog.c */
-extern bool drbd_al_begin_io_prepare(struct drbd_device *device, struct drbd_interval *i);
-extern int drbd_al_begin_io_nonblock(struct drbd_device *device, struct drbd_interval *i);
-extern void drbd_al_begin_io_commit(struct drbd_device *device);
-extern bool drbd_al_begin_io_fastpath(struct drbd_device *device, struct drbd_interval *i);
-extern void drbd_al_begin_io(struct drbd_device *device, struct drbd_interval *i);
-extern void drbd_al_complete_io(struct drbd_device *device, struct drbd_interval *i);
-extern void drbd_rs_complete_io(struct drbd_device *device, sector_t sector);
-extern int drbd_rs_begin_io(struct drbd_device *device, sector_t sector);
-extern int drbd_try_rs_begin_io(struct drbd_peer_device *peer_device, sector_t sector);
-extern void drbd_rs_cancel_all(struct drbd_device *device);
-extern int drbd_rs_del_all(struct drbd_device *device);
-extern void drbd_rs_failed_io(struct drbd_peer_device *peer_device,
- sector_t sector, int size);
-extern void drbd_advance_rs_marks(struct drbd_peer_device *peer_device, unsigned long still_to_go);
+bool drbd_al_try_lock(struct drbd_device *device);
+bool drbd_al_try_lock_for_transaction(struct drbd_device *device);
+int drbd_al_begin_io_nonblock(struct drbd_device *device,
+ struct drbd_interval *i);
+void drbd_al_begin_io_commit(struct drbd_device *device);
+bool drbd_al_begin_io_fastpath(struct drbd_device *device,
+ struct drbd_interval *i);
+bool drbd_al_complete_io(struct drbd_device *device, struct drbd_interval *i);
+void drbd_advance_rs_marks(struct drbd_peer_device *peer_device,
+ unsigned long still_to_go);
+bool drbd_lazy_bitmap_update_due(struct drbd_peer_device *peer_device);
+bool drbd_bitmap_slot_decides(struct drbd_peer_device *peer_device);
+unsigned long drbd_set_all_out_of_sync(struct drbd_device *device, int src_node_id,
+ sector_t sector, int size);
+unsigned long drbd_set_sync(struct drbd_device *device, sector_t sector, int size,
+ unsigned long bits, unsigned long mask);
+
+/* Lockless, drbd_cbt_mark() runs it for every write. cbt-start publishes
+ * bitmap_index with a release store of flags; cbt-stop leaves bitmap_index
+ * valid, so a stale IN_USE at worst sets a bit in the slot it just freed.
+ */
+static inline int drbd_cbt_bitmap_index(struct drbd_backing_dev *ldev)
+{
+ /* pairs with smp_store_release() in drbd_adm_cbt_start() */
+ u32 flags = smp_load_acquire(&ldev->md.cbt.flags);
+ return flags & CBT_BOOKMARK_IN_USE ? READ_ONCE(ldev->md.cbt.bitmap_index) : -1;
+}
+
+void drbd_cbt_mark(struct drbd_device *device, sector_t sector, sector_t nr_sectors);
+void drbd_cbt_merge_resync_bits(struct drbd_peer_device *peer_device);
+void drbd_cbt_clear_flushed(struct drbd_device *device);
enum update_sync_bits_mode { RECORD_RS_FAILED, SET_OUT_OF_SYNC, SET_IN_SYNC };
-extern int __drbd_change_sync(struct drbd_peer_device *peer_device, sector_t sector, int size,
- enum update_sync_bits_mode mode);
+int __drbd_change_sync(struct drbd_peer_device *peer_device, sector_t sector,
+ int size, enum update_sync_bits_mode mode);
#define drbd_set_in_sync(peer_device, sector, size) \
__drbd_change_sync(peer_device, sector, size, SET_IN_SYNC)
#define drbd_set_out_of_sync(peer_device, sector, size) \
__drbd_change_sync(peer_device, sector, size, SET_OUT_OF_SYNC)
#define drbd_rs_failed_io(peer_device, sector, size) \
__drbd_change_sync(peer_device, sector, size, RECORD_RS_FAILED)
-extern void drbd_al_shrink(struct drbd_device *device);
-extern int drbd_al_initialize(struct drbd_device *, void *);
+void drbd_al_shrink(struct drbd_device *device);
+int drbd_al_initialize(struct drbd_device *device, void *buffer);
/* drbd_nl.c */
-/* state info broadcast */
-struct sib_info {
- enum drbd_state_info_bcast_reason sib_reason;
- union {
- struct {
- char *helper_name;
- unsigned helper_exit_code;
- };
- struct {
- union drbd_state os;
- union drbd_state ns;
- };
- };
-};
-void drbd_bcast_event(struct drbd_device *device, const struct sib_info *sib);
-
-extern int notify_resource_state(struct sk_buff *,
- unsigned int,
- struct drbd_resource *,
- struct resource_info *,
- enum drbd_notification_type);
-extern int notify_device_state(struct sk_buff *,
- unsigned int,
- struct drbd_device *,
- struct device_info *,
- enum drbd_notification_type);
-extern int notify_connection_state(struct sk_buff *,
- unsigned int,
- struct drbd_connection *,
- struct connection_info *,
- enum drbd_notification_type);
-extern int notify_peer_device_state(struct sk_buff *,
- unsigned int,
- struct drbd_peer_device *,
- struct peer_device_info *,
- enum drbd_notification_type);
-extern void notify_helper(enum drbd_notification_type, struct drbd_device *,
- struct drbd_connection *, const char *, int);
+extern struct mutex notification_mutex;
+extern atomic_t drbd_genl_seq;
+
+int notify_resource_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *resource_info,
+ struct drbd_rename_resource_info *rename_resource_info,
+ enum drbd_notification_type type);
+int notify_device_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_device *device,
+ struct drbd_device_info *device_info,
+ enum drbd_notification_type type);
+int notify_connection_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_connection *connection,
+ struct drbd_connection_info *connection_info,
+ enum drbd_notification_type type);
+int notify_peer_device_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_info *peer_device_info,
+ enum drbd_notification_type type);
+void notify_helper(enum drbd_notification_type type,
+ struct drbd_device *device,
+ struct drbd_connection *connection, const char *name,
+ int status);
+int notify_path(struct drbd_connection *connection, struct drbd_path *path,
+ enum drbd_notification_type type);
+void drbd_broadcast_peer_device_state(struct drbd_peer_device *peer_device);
+
+sector_t drbd_local_max_size(struct drbd_device *device);
+void drbd_auto_grow(struct drbd_device *device);
+int drbd_open_ro_count(struct drbd_resource *resource);
+
+void device_to_info(struct drbd_device_info *info, struct drbd_device *device);
+void device_state_change_to_info(struct drbd_device_info *info,
+ struct drbd_device_state_change *state_change);
+void peer_device_state_change_to_info(struct drbd_peer_device_info *info,
+ struct drbd_peer_device_state_change *state_change);
/*
* inline helper functions
*************************/
-/* see also page_chain_add and friends in drbd_receiver.c */
-static inline struct page *page_chain_next(struct page *page)
-{
- return (struct page *)page_private(page);
-}
-#define page_chain_for_each(page) \
- for (; page && ({ prefetch(page_chain_next(page)); 1; }); \
- page = page_chain_next(page))
-#define page_chain_for_each_safe(page, n) \
- for (; page && ({ n = page_chain_next(page); 1; }); page = n)
-
-
-static inline union drbd_state drbd_read_state(struct drbd_device *device)
-{
- struct drbd_resource *resource = device->resource;
- union drbd_state rv;
-
- rv.i = device->state.i;
- rv.susp = resource->susp;
- rv.susp_nod = resource->susp_nod;
- rv.susp_fen = resource->susp_fen;
-
- return rv;
-}
-
-enum drbd_force_detach_flags {
- DRBD_READ_ERROR,
- DRBD_WRITE_ERROR,
- DRBD_META_IO_ERROR,
- DRBD_FORCE_DETACH,
-};
-
-#define __drbd_chk_io_error(m,f) __drbd_chk_io_error_(m,f, __func__)
-static inline void __drbd_chk_io_error_(struct drbd_device *device,
- enum drbd_force_detach_flags df,
- const char *where)
+/*
+ * When a device has a replication state above L_OFF, it must be
+ * connected. Otherwise, we report the connection state, which has values up
+ * to C_CONNECTED == L_OFF.
+ */
+static inline int combined_conn_state(struct drbd_peer_device *peer_device, enum which_state which)
{
- enum drbd_io_error_p ep;
-
- rcu_read_lock();
- ep = rcu_dereference(device->ldev->disk_conf)->on_io_error;
- rcu_read_unlock();
- switch (ep) {
- case EP_PASS_ON: /* FIXME would this be better named "Ignore"? */
- if (df == DRBD_READ_ERROR || df == DRBD_WRITE_ERROR) {
- if (drbd_device_ratelimit(device, BACKEND))
- drbd_err(device, "Local IO failed in %s.\n", where);
- if (device->state.disk > D_INCONSISTENT)
- _drbd_set_state(_NS(device, disk, D_INCONSISTENT), CS_HARD, NULL);
- break;
- }
- fallthrough; /* for DRBD_META_IO_ERROR or DRBD_FORCE_DETACH */
- case EP_DETACH:
- case EP_CALL_HELPER:
- /* Remember whether we saw a READ or WRITE error.
- *
- * Recovery of the affected area for WRITE failure is covered
- * by the activity log.
- * READ errors may fall outside that area though. Certain READ
- * errors can be "healed" by writing good data to the affected
- * blocks, which triggers block re-allocation in lower layers.
- *
- * If we can not write the bitmap after a READ error,
- * we may need to trigger a full sync (see w_go_diskless()).
- *
- * Force-detach is not really an IO error, but rather a
- * desperate measure to try to deal with a completely
- * unresponsive lower level IO stack.
- * Still it should be treated as a WRITE error.
- *
- * Meta IO error is always WRITE error:
- * we read meta data only once during attach,
- * which will fail in case of errors.
- */
- set_bit(WAS_IO_ERROR, &device->flags);
- if (df == DRBD_READ_ERROR)
- set_bit(WAS_READ_ERROR, &device->flags);
- if (df == DRBD_FORCE_DETACH)
- set_bit(FORCE_DETACH, &device->flags);
- if (device->state.disk > D_FAILED) {
- _drbd_set_state(_NS(device, disk, D_FAILED), CS_HARD, NULL);
- drbd_err(device,
- "Local IO failed in %s. Detaching...\n", where);
- }
- break;
- }
-}
+ enum drbd_repl_state repl_state = peer_device->repl_state[which];
-/**
- * drbd_chk_io_error: Handle the on_io_error setting, should be called from all io completion handlers
- * @device: DRBD device.
- * @error: Error code passed to the IO completion callback
- * @forcedetach: Force detach. I.e. the error happened while accessing the meta data
- *
- * See also drbd_main.c:after_state_ch() if (os.disk > D_FAILED && ns.disk == D_FAILED)
- */
-#define drbd_chk_io_error(m,e,f) drbd_chk_io_error_(m,e,f, __func__)
-static inline void drbd_chk_io_error_(struct drbd_device *device,
- int error, enum drbd_force_detach_flags forcedetach, const char *where)
-{
- if (error) {
- unsigned long flags;
- spin_lock_irqsave(&device->resource->req_lock, flags);
- __drbd_chk_io_error_(device, forcedetach, where);
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
- }
+ if (repl_state > L_OFF)
+ return repl_state;
+ else
+ return peer_device->connection->cstate[which];
}
-
/**
* drbd_md_first_sector() - Returns the first sector number of the meta data area
* @bdev: Meta data block device.
@@ -1755,52 +3117,11 @@ static inline sector_t drbd_md_last_sector(struct drbd_backing_dev *bdev)
switch (bdev->md.meta_dev_idx) {
case DRBD_MD_INDEX_INTERNAL:
case DRBD_MD_INDEX_FLEX_INT:
- return bdev->md.md_offset + MD_4kB_SECT -1;
- case DRBD_MD_INDEX_FLEX_EXT:
- default:
- return bdev->md.md_offset + bdev->md.md_size_sect -1;
- }
-}
-
-/* Returns the number of 512 byte sectors of the device */
-static inline sector_t drbd_get_capacity(struct block_device *bdev)
-{
- return bdev ? bdev_nr_sectors(bdev) : 0;
-}
-
-/**
- * drbd_get_max_capacity() - Returns the capacity we announce to out peer
- * @bdev: Meta data block device.
- *
- * returns the capacity we announce to out peer. we clip ourselves at the
- * various MAX_SECTORS, because if we don't, current implementation will
- * oops sooner or later
- */
-static inline sector_t drbd_get_max_capacity(struct drbd_backing_dev *bdev)
-{
- sector_t s;
-
- switch (bdev->md.meta_dev_idx) {
- case DRBD_MD_INDEX_INTERNAL:
- case DRBD_MD_INDEX_FLEX_INT:
- s = drbd_get_capacity(bdev->backing_bdev)
- ? min_t(sector_t, DRBD_MAX_SECTORS_FLEX,
- drbd_md_first_sector(bdev))
- : 0;
- break;
+ return bdev->md.md_offset + (4096 >> 9) - 1;
case DRBD_MD_INDEX_FLEX_EXT:
- s = min_t(sector_t, DRBD_MAX_SECTORS_FLEX,
- drbd_get_capacity(bdev->backing_bdev));
- /* clip at maximum size the meta device can support */
- s = min_t(sector_t, s,
- BM_EXT_TO_SECT(bdev->md.md_size_sect
- - bdev->md.bm_offset));
- break;
default:
- s = min_t(sector_t, DRBD_MAX_SECTORS,
- drbd_get_capacity(bdev->backing_bdev));
+ return bdev->md.md_offset + bdev->md.md_size_sect - 1;
}
- return s;
}
/**
@@ -1821,18 +3142,10 @@ static inline sector_t drbd_md_ss(struct drbd_backing_dev *bdev)
return (drbd_get_capacity(bdev->backing_bdev) & ~7ULL) - 8;
/* external, some index; this is the old fixed size layout */
- return MD_128MB_SECT * bdev->md.meta_dev_idx;
+ return (128 << 20 >> 9) * bdev->md.meta_dev_idx;
}
-static inline void
-drbd_queue_work(struct drbd_work_queue *q, struct drbd_work *w)
-{
- unsigned long flags;
- spin_lock_irqsave(&q->q_lock, flags);
- list_add_tail(&w->list, &q->q);
- spin_unlock_irqrestore(&q->q_lock, flags);
- wake_up(&q->q_wait);
-}
+void drbd_queue_work(struct drbd_work_queue *, struct drbd_work *);
static inline void
drbd_queue_work_if_unqueued(struct drbd_work_queue *q, struct drbd_work *w)
@@ -1849,46 +3162,73 @@ static inline void
drbd_device_post_work(struct drbd_device *device, int work_bit)
{
if (!test_and_set_bit(work_bit, &device->flags)) {
- struct drbd_connection *connection =
- first_peer_device(device)->connection;
- struct drbd_work_queue *q = &connection->sender_work;
- if (!test_and_set_bit(DEVICE_WORK_PENDING, &connection->flags))
+ struct drbd_resource *resource = device->resource;
+ struct drbd_work_queue *q = &resource->work;
+ if (!test_and_set_bit(DEVICE_WORK_PENDING, &resource->flags))
wake_up(&q->q_wait);
}
}
-extern void drbd_flush_workqueue(struct drbd_work_queue *work_queue);
-
-/* To get the ack_receiver out of the blocking network stack,
- * so it can change its sk_rcvtimeo from idle- to ping-timeout,
- * and send a ping, we need to send a signal.
- * Which signal we send is irrelevant. */
-static inline void wake_ack_receiver(struct drbd_connection *connection)
+static inline void
+drbd_peer_device_post_work(struct drbd_peer_device *peer_device, int work_bit)
{
- struct task_struct *task = connection->ack_receiver.task;
- if (task && get_t_state(&connection->ack_receiver) == RUNNING)
- send_sig(SIGXCPU, task, 1);
+ if (!test_and_set_bit(work_bit, peer_device->flags)) {
+ struct drbd_resource *resource = peer_device->device->resource;
+ struct drbd_work_queue *q = &resource->work;
+ if (!test_and_set_bit(PEER_DEVICE_WORK_PENDING, &resource->flags))
+ wake_up(&q->q_wait);
+ }
}
-static inline void request_ping(struct drbd_connection *connection)
+void drbd_flush_workqueue(struct drbd_work_queue *work_queue);
+void drbd_flush_workqueue_interruptible(struct drbd_device *device);
+
+void *__conn_prepare_command(struct drbd_connection *connection, int size,
+ enum drbd_stream drbd_stream);
+void *conn_prepare_command(struct drbd_connection *connection, int size,
+ enum drbd_stream drbd_stream);
+void *drbd_prepare_command(struct drbd_peer_device *peer_device, int size,
+ enum drbd_stream drbd_stream);
+int __send_command(struct drbd_connection *connection, int vnr,
+ enum drbd_packet cmd, int stream_and_flags);
+int send_command(struct drbd_connection *connection, int vnr,
+ enum drbd_packet cmd, int stream_and_flags);
+int drbd_send_command(struct drbd_peer_device *peer_device,
+ enum drbd_packet cmd, enum drbd_stream drbd_stream);
+
+int drbd_send_ping(struct drbd_connection *connection);
+int conn_send_state_req(struct drbd_connection *connection, int vnr,
+ enum drbd_packet cmd, union drbd_state mask,
+ union drbd_state val);
+int conn_send_twopc_request(struct drbd_connection *connection,
+ struct twopc_request *request);
+int drbd_send_peer_ack(struct drbd_connection *connection, u64 mask,
+ u64 dagtag_sector);
+
+/* DRBD 9 inserted D_DETACHING into enum drbd_disk_state; every state from
+ * D_FAILED up is one higher than in DRBD 8.4, which used D_FAILED for both.
+ * The v1 netlink dialect puts every disk or pdsk value through this too.
+ */
+static inline enum drbd_disk_state drbd_disk_state_84(enum drbd_disk_state state)
{
- set_bit(SEND_PING, &connection->flags);
- wake_ack_receiver(connection);
+ if (state > D_DETACHING)
+ state--;
+ return state;
}
-extern void *conn_prepare_command(struct drbd_connection *, struct drbd_socket *);
-extern void *drbd_prepare_command(struct drbd_peer_device *, struct drbd_socket *);
-extern int conn_send_command(struct drbd_connection *, struct drbd_socket *,
- enum drbd_packet, unsigned int, void *,
- unsigned int);
-extern int drbd_send_command(struct drbd_peer_device *, struct drbd_socket *,
- enum drbd_packet, unsigned int, void *,
- unsigned int);
+static inline void drbd_disk_states_to_84(union drbd_state *s)
+{
+ s->disk = drbd_disk_state_84(s->disk);
+ s->pdsk = drbd_disk_state_84(s->pdsk);
+}
-extern int drbd_send_ping(struct drbd_connection *connection);
-extern int drbd_send_ping_ack(struct drbd_connection *connection);
-extern int drbd_send_state_req(struct drbd_peer_device *, union drbd_state, union drbd_state);
-extern int conn_send_state_req(struct drbd_connection *, union drbd_state, union drbd_state);
+static inline void drbd_disk_states_from_84(union drbd_state *s)
+{
+ if (s->disk >= D_DETACHING)
+ s->disk++;
+ if (s->pdsk >= D_DETACHING)
+ s->pdsk++;
+}
static inline void drbd_thread_stop(struct drbd_thread *thi)
{
@@ -1905,59 +3245,41 @@ static inline void drbd_thread_restart_nowait(struct drbd_thread *thi)
_drbd_thread_stop(thi, true, false);
}
-/* counts how many answer packets packets we expect from our peer,
- * for either explicit application requests,
- * or implicit barrier packets as necessary.
- * increased:
- * w_send_barrier
- * _req_mod(req, QUEUE_FOR_NET_WRITE or QUEUE_FOR_NET_READ);
- * it is much easier and equally valid to count what we queue for the
- * worker, even before it actually was queued or send.
- * (drbd_make_request_common; recovery path on read io-error)
- * decreased:
- * got_BarrierAck (respective tl_clear, tl_clear_barrier)
- * _req_mod(req, DATA_RECEIVED)
- * [from receive_DataReply]
- * _req_mod(req, WRITE_ACKED_BY_PEER or RECV_ACKED_BY_PEER or NEG_ACKED)
- * [from got_BlockAck (P_WRITE_ACK, P_RECV_ACK)]
- * for some reason it is NOT decreased in got_NegAck,
- * but in the resulting cleanup code from report_params.
- * we should try to remember the reason for that...
- * _req_mod(req, SEND_FAILED or SEND_CANCELED)
- * _req_mod(req, CONNECTION_LOST_WHILE_PENDING)
- * [from tl_clear_barrier]
- */
-static inline void inc_ap_pending(struct drbd_device *device)
+static inline void inc_ap_pending(struct drbd_peer_device *peer_device)
{
- atomic_inc(&device->ap_pending_cnt);
+ atomic_inc(&peer_device->ap_pending_cnt);
}
-#define dec_ap_pending(device) ((void)expect((device), __dec_ap_pending(device) >= 0))
-static inline int __dec_ap_pending(struct drbd_device *device)
+#define dec_ap_pending(peer_device) \
+ ((void)expect((peer_device), __dec_ap_pending(peer_device) >= 0))
+static inline int __dec_ap_pending(struct drbd_peer_device *peer_device)
{
- int ap_pending_cnt = atomic_dec_return(&device->ap_pending_cnt);
-
+ int ap_pending_cnt = atomic_dec_return(&peer_device->ap_pending_cnt);
if (ap_pending_cnt == 0)
- wake_up(&device->misc_wait);
+ wake_up(&peer_device->device->misc_wait);
return ap_pending_cnt;
}
/* counts how many resync-related answers we still expect from the peer
* increase decrease
- * C_SYNC_TARGET sends P_RS_DATA_REQUEST (and expects P_RS_DATA_REPLY)
- * C_SYNC_SOURCE sends P_RS_DATA_REPLY (and expects P_WRITE_ACK with ID_SYNCER)
+ * L_SYNC_TARGET sends P_RS_DATA_REQUEST (and expects P_RS_DATA_REPLY)
+ * L_SYNC_SOURCE sends P_RS_DATA_REPLY (and expects P_WRITE_ACK with ID_SYNCER)
* (or P_NEG_ACK with ID_SYNCER)
*/
static inline void inc_rs_pending(struct drbd_peer_device *peer_device)
{
- atomic_inc(&peer_device->device->rs_pending_cnt);
+ atomic_inc(&peer_device->rs_pending_cnt);
}
#define dec_rs_pending(peer_device) \
((void)expect((peer_device), __dec_rs_pending(peer_device) >= 0))
static inline int __dec_rs_pending(struct drbd_peer_device *peer_device)
{
- return atomic_dec_return(&peer_device->device->rs_pending_cnt);
+ int rs_pending_cnt = atomic_dec_return(&peer_device->rs_pending_cnt);
+
+ if (rs_pending_cnt == 0)
+ wake_up(&peer_device->device->misc_wait);
+ return rs_pending_cnt;
}
/* counts how many answers we still need to send to the peer.
@@ -1966,42 +3288,91 @@ static inline int __dec_rs_pending(struct drbd_peer_device *peer_device)
* we need to send a P_RECV_ACK (proto B)
* or P_WRITE_ACK (proto C)
* receive_RSDataReply (recv_resync_read) we need to send a P_WRITE_ACK
- * receive_DataRequest (receive_RSDataRequest) we need to send back P_DATA
+ * receive_data_request etc we need to send back P_DATA
* receive_Barrier_* we need to send a P_BARRIER_ACK
*/
-static inline void inc_unacked(struct drbd_device *device)
+static inline void inc_unacked(struct drbd_peer_device *peer_device)
+{
+ atomic_inc(&peer_device->unacked_cnt);
+}
+
+#define dec_unacked(peer_device) \
+ ((void)expect(peer_device, __dec_unacked(peer_device) >= 0))
+static inline int __dec_unacked(struct drbd_peer_device *peer_device)
+{
+ return atomic_dec_return(&peer_device->unacked_cnt);
+}
+
+static inline bool repl_is_sync_target(enum drbd_repl_state repl_state)
+{
+ return repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T;
+}
+
+static inline bool repl_is_sync_source(enum drbd_repl_state repl_state)
+{
+ return repl_state == L_SYNC_SOURCE || repl_state == L_PAUSED_SYNC_S;
+}
+
+static inline bool repl_is_sync(enum drbd_repl_state repl_state)
+{
+ return repl_is_sync_source(repl_state) ||
+ repl_is_sync_target(repl_state);
+}
+
+/* Only the side that sends the requests takes a max_parallel_resyncs slot.
+ * A SyncSource answers what its target asks for, and the target's node
+ * counts and pauses it.
+ */
+static inline bool repl_holds_resync_slot(enum drbd_repl_state repl_state)
{
- atomic_inc(&device->unacked_cnt);
+ return repl_state == L_SYNC_TARGET || repl_state == L_VERIFY_S;
}
-#define dec_unacked(device) ((void)expect(device, __dec_unacked(device) >= 0))
-static inline int __dec_unacked(struct drbd_device *device)
+static inline bool is_sync_target_state(struct drbd_peer_device *peer_device,
+ enum which_state which)
{
- return atomic_dec_return(&device->unacked_cnt);
+ return repl_is_sync_target(peer_device->repl_state[which]);
}
-#define sub_unacked(device, n) ((void)expect(device, __sub_unacked(device) >= 0))
-static inline int __sub_unacked(struct drbd_device *device, int n)
+static inline bool is_sync_source_state(struct drbd_peer_device *peer_device,
+ enum which_state which)
{
- return atomic_sub_return(n, &device->unacked_cnt);
+ return repl_is_sync_source(peer_device->repl_state[which]);
}
-static inline bool is_sync_target_state(enum drbd_conns connection_state)
+static inline bool is_sync_state(struct drbd_peer_device *peer_device,
+ enum which_state which)
{
- return connection_state == C_SYNC_TARGET ||
- connection_state == C_PAUSED_SYNC_T;
+ return repl_is_sync(peer_device->repl_state[which]);
}
-static inline bool is_sync_source_state(enum drbd_conns connection_state)
+static inline bool is_verify_state(struct drbd_peer_device *peer_device,
+ enum which_state which)
{
- return connection_state == C_SYNC_SOURCE ||
- connection_state == C_PAUSED_SYNC_S;
+ enum drbd_repl_state repl_state = peer_device->repl_state[which];
+ return repl_state == L_VERIFY_S || repl_state == L_VERIFY_T;
}
-static inline bool is_sync_state(enum drbd_conns connection_state)
+static inline bool resync_susp_comb_dep(struct drbd_peer_device *peer_device, enum which_state which)
{
- return is_sync_source_state(connection_state) ||
- is_sync_target_state(connection_state);
+ struct drbd_device *device = peer_device->device;
+
+ return peer_device->resync_susp_dependency[which] || peer_device->resync_susp_other_c[which] ||
+ (is_sync_source_state(peer_device, which) && device->disk_state[which] <= D_INCONSISTENT);
+}
+
+static inline int
+drbd_insert_fault_conn(struct drbd_connection *connection, unsigned int type) {
+#ifdef CONFIG_DRBD_FAULT_INJECTION
+ int id = 0;
+ struct drbd_device *device = idr_get_next(&connection->resource->devices, &id);
+
+ return device && drbd_fault_rate &&
+ (drbd_enable_faults & (1<<type)) &&
+ _drbd_insert_fault(device, type);
+#else
+ return 0;
+#endif
}
/**
@@ -2011,14 +3382,11 @@ static inline bool is_sync_state(enum drbd_conns connection_state)
*
* You have to call put_ldev() when finished working with device->ldev.
*/
-#define get_ldev_if_state(_device, _min_state) \
- (_get_ldev_if_state((_device), (_min_state)) ? \
- ({ __acquire(x); true; }) : false)
#define get_ldev(_device) get_ldev_if_state(_device, D_INCONSISTENT)
static inline void put_ldev(struct drbd_device *device)
{
- enum drbd_disk_state disk_state = device->state.disk;
+ enum drbd_disk_state disk_state = device->disk_state[NOW];
/* We must check the state *before* the atomic_dec becomes visible,
* or we have a theoretical race where someone hitting zero,
* while state still D_FAILED, will then see D_DISKLESS in the
@@ -2028,13 +3396,14 @@ static inline void put_ldev(struct drbd_device *device)
/* This may be called from some endio handler,
* so we must not sleep here. */
- __release(local);
D_ASSERT(device, i >= 0);
if (i == 0) {
- if (disk_state == D_DISKLESS)
+ if (disk_state == D_DISKLESS) {
/* even internal references gone, safe to destroy */
- drbd_device_post_work(device, DESTROY_DISK);
- if (disk_state == D_FAILED)
+ kref_get(&device->kref);
+ schedule_work(&device->ldev_destroy_work);
+ }
+ if (disk_state == D_FAILED || disk_state == D_DETACHING)
/* all application IO references gone. */
if (!test_and_set_bit(GOING_DISKLESS, &device->flags))
drbd_device_post_work(device, GO_DISKLESS);
@@ -2042,122 +3411,64 @@ static inline void put_ldev(struct drbd_device *device)
}
}
-#ifndef __CHECKER__
-static inline int _get_ldev_if_state(struct drbd_device *device, enum drbd_disk_state mins)
+static inline int get_ldev_if_state(struct drbd_device *device, enum drbd_disk_state mins)
{
int io_allowed;
/* never get a reference while D_DISKLESS */
- if (device->state.disk == D_DISKLESS)
+ if (device->disk_state[NOW] == D_DISKLESS)
return 0;
atomic_inc(&device->local_cnt);
- io_allowed = (device->state.disk >= mins);
+ io_allowed = (device->disk_state[NOW] >= mins);
if (!io_allowed)
put_ldev(device);
return io_allowed;
}
-#else
-extern int _get_ldev_if_state(struct drbd_device *device, enum drbd_disk_state mins);
-#endif
-
-/* this throttles on-the-fly application requests
- * according to max_buffers settings;
- * maybe re-implement using semaphores? */
-static inline int drbd_get_max_buffers(struct drbd_device *device)
-{
- struct net_conf *nc;
- int mxb;
- rcu_read_lock();
- nc = rcu_dereference(first_peer_device(device)->connection->net_conf);
- mxb = nc ? nc->max_buffers : 1000000; /* arbitrary limit on open requests */
- rcu_read_unlock();
+void drbd_queue_pending_bitmap_work(struct drbd_device *device);
- return mxb;
-}
-
-static inline int drbd_state_is_stable(struct drbd_device *device)
-{
- union drbd_dev_state s = device->state;
-
- /* DO NOT add a default clause, we want the compiler to warn us
- * for any newly introduced state we may have forgotten to add here */
-
- switch ((enum drbd_conns)s.conn) {
- /* new io only accepted when there is no connection, ... */
- case C_STANDALONE:
- case C_WF_CONNECTION:
- /* ... or there is a well established connection. */
- case C_CONNECTED:
- case C_SYNC_SOURCE:
- case C_SYNC_TARGET:
- case C_VERIFY_S:
- case C_VERIFY_T:
- case C_PAUSED_SYNC_S:
- case C_PAUSED_SYNC_T:
- case C_AHEAD:
- case C_BEHIND:
- /* transitional states, IO allowed */
- case C_DISCONNECTING:
- case C_UNCONNECTED:
- case C_TIMEOUT:
- case C_BROKEN_PIPE:
- case C_NETWORK_FAILURE:
- case C_PROTOCOL_ERROR:
- case C_TEAR_DOWN:
- case C_WF_REPORT_PARAMS:
- case C_STARTING_SYNC_S:
- case C_STARTING_SYNC_T:
- break;
+/* rw = READ or WRITE (0 or 1); nothing else. */
+static inline void dec_ap_bio(struct drbd_device *device, int rw)
+{
+ unsigned int nr_requests = device->resource->res_opts.nr_requests;
+ int ap_bio = atomic_dec_return(&device->ap_bio_cnt[rw]);
- /* Allow IO in BM exchange states with new protocols */
- case C_WF_BITMAP_S:
- if (first_peer_device(device)->connection->agreed_pro_version < 96)
- return 0;
- break;
+ D_ASSERT(device, ap_bio >= 0);
- /* no new io accepted in these states */
- case C_WF_BITMAP_T:
- case C_WF_SYNC_UUID:
- case C_MASK:
- /* not "stable" */
- return 0;
- }
+ /* Check for list_empty outside the lock is ok. Worst case it queues
+ * nothing because someone else just now did. During list_add, a
+ * refcount on ap_bio_cnt[WRITE] is held, so the bitmap work will be
+ * queued when that is released if we miss it here.
+ * Checking pending_bitmap_work.n is not correct,
+ * it has a different lifetime. */
+ if (ap_bio == 0 && rw == WRITE && !list_empty(&device->pending_bitmap_work.q))
+ drbd_queue_pending_bitmap_work(device);
- switch ((enum drbd_disk_state)s.disk) {
- case D_DISKLESS:
- case D_INCONSISTENT:
- case D_OUTDATED:
- case D_CONSISTENT:
- case D_UP_TO_DATE:
- case D_FAILED:
- /* disk state is stable as well. */
- break;
+ if (ap_bio == 0 || ap_bio == nr_requests-1)
+ wake_up(&device->misc_wait);
+}
- /* no new io accepted during transitional states */
- case D_ATTACHING:
- case D_NEGOTIATING:
- case D_UNKNOWN:
- case D_MASK:
- /* not "stable" */
- return 0;
- }
+static inline unsigned long get_work_bits(const unsigned long mask, unsigned long *flags)
+{
+ unsigned long old, new;
- return 1;
+ do {
+ old = *flags;
+ new = old & ~mask;
+ } while (cmpxchg(flags, old, new) != old);
+ return old & mask;
}
-static inline int drbd_suspended(struct drbd_device *device)
+static inline bool drbd_suspended(struct drbd_device *device)
{
- struct drbd_resource *resource = device->resource;
-
- return resource->susp || resource->susp_fen || resource->susp_nod;
+ return device->resource->cached_susp;
}
static inline bool may_inc_ap_bio(struct drbd_device *device)
{
- int mxb = drbd_get_max_buffers(device);
-
+ if (device->cached_err_io)
+ return true;
if (drbd_suspended(device))
return false;
if (atomic_read(&device->suspend_cnt))
@@ -2168,76 +3479,112 @@ static inline bool may_inc_ap_bio(struct drbd_device *device)
* to start during "stable" states. */
/* no new io accepted when attaching or detaching the disk */
- if (!drbd_state_is_stable(device))
+ if (device->cached_state_unstable)
return false;
- /* since some older kernels don't have atomic_add_unless,
- * and we are within the spinlock anyways, we have this workaround. */
- if (atomic_read(&device->ap_bio_cnt) > mxb)
- return false;
- if (test_bit(BITMAP_IO, &device->flags))
+ if (atomic_read(&device->pending_bitmap_work.n))
return false;
return true;
}
-static inline bool inc_ap_bio_cond(struct drbd_device *device)
+extern bool drbd_gen_obligation_transition(struct drbd_device *device,
+ unsigned int from_states,
+ enum drbd_gen_obl_state to,
+ u16 reasons, u32 aux);
+extern void drbd_gen_obligation_str(u32 obligation, char *buf, size_t size);
+extern void drbd_gen_obligation_arm(struct drbd_device *device, u16 reasons);
+extern bool drbd_gen_obligation_materialize(struct drbd_device *device);
+extern bool drbd_gen_obligation_void(struct drbd_device *device);
+extern bool drbd_gen_obligation_discharge_by_adoption(struct drbd_device *device);
+extern bool drbd_gen_obligation_mint_start(struct drbd_device *device);
+extern void drbd_gen_obligation_mint_run(struct drbd_device *device);
+extern void drbd_gen_obligation_mint_done(struct drbd_device *device,
+ enum drbd_mint_outcome outcome);
+extern void drbd_gen_obligation_mint_before_resume(struct drbd_device *device, u64 only_nodes);
+
+static inline enum drbd_gen_obl_state drbd_gen_obligation_state(struct drbd_device *device)
{
- bool rv = false;
+ return READ_ONCE(device->gen_obligation) & GEN_OBL_STATE_MASK;
+}
- spin_lock_irq(&device->resource->req_lock);
- rv = may_inc_ap_bio(device);
- if (rv)
- atomic_inc(&device->ap_bio_cnt);
- spin_unlock_irq(&device->resource->req_lock);
+/* True while this volume owes a new data generation and a write must wait for
+ * it. A parked obligation is owed as well, but writes do not reach this test
+ * while it is parked; see GEN_OBL_PARKED. Readers take no lock.
+ * A transition can become visible slightly ahead of the state change that
+ * causes it, so a writer may find the obligation outstanding a moment before
+ * the state change is published -- that only holds back a write the state
+ * change is about to hold back anyway. The other direction, the new state
+ * visible while the obligation it created is not, cannot happen.
+ */
+static inline bool drbd_gen_obligation_outstanding(struct drbd_device *device)
+{
+ enum drbd_gen_obl_state state = drbd_gen_obligation_state(device);
- return rv;
+ return state == GEN_OBL_ARMED || state == GEN_OBL_MINTING;
}
-static inline void inc_ap_bio(struct drbd_device *device)
+/* True while this volume decided to complete writes that a replica it lost
+ * never saw. Nothing undoes such a completion, so the obligation is not
+ * voidable and its mint is mandatory.
+ */
+static inline bool drbd_gen_obligation_materialized(struct drbd_device *device)
{
- /* we wait here
- * as long as the device is suspended
- * until the bitmap is no longer on the fly during connection
- * handshake as long as we would exceed the max_buffer limit.
- *
- * to avoid races with the reconnect code,
- * we need to atomic_inc within the spinlock. */
+ return READ_ONCE(device->gen_obligation) & GEN_OBL_MATERIALIZED;
+}
- wait_event(device->misc_wait, inc_ap_bio_cond(device));
+static inline u64 drbd_current_uuid(struct drbd_device *device)
+{
+ if (!device->ldev)
+ return 0;
+ return device->ldev->md.current_uuid;
}
-static inline void dec_ap_bio(struct drbd_device *device)
+static inline bool verify_can_do_stop_sector(struct drbd_peer_device *peer_device)
{
- int mxb = drbd_get_max_buffers(device);
- int ap_bio = atomic_dec_return(&device->ap_bio_cnt);
+ return peer_device->connection->agreed_pro_version >= 97 &&
+ peer_device->connection->agreed_pro_version != 100;
+}
- D_ASSERT(device, ap_bio >= 0);
+static inline u64 drbd_bitmap_uuid(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md;
- if (ap_bio == 0 && test_bit(BITMAP_IO, &device->flags)) {
- if (!test_and_set_bit(BITMAP_IO_QUEUED, &device->flags))
- drbd_queue_work(&first_peer_device(device)->
- connection->sender_work,
- &device->bm_io_work.w);
- }
+ if (!device->ldev)
+ return 0;
- /* this currently does wake_up for every dec_ap_bio!
- * maybe rather introduce some type of hysteresis?
- * e.g. (ap_bio == mxb/2 || ap_bio == 0) ? */
- if (ap_bio < mxb)
- wake_up(&device->misc_wait);
+ peer_md = &device->ldev->md.peers[peer_device->node_id];
+ return peer_md->bitmap_uuid;
+}
+
+/* A slot with no bitmap of its own (MDF_HAVE_BITMAP clear) is implicitly a
+ * divergence bitmap. A day0 slot is such a slot: its bitmap is associated with
+ * it indirectly rather than owned directly. Only a directly-owned bitmap is
+ * cleared by a resync, and thereby turned into a convergence bitmap.
+ */
+static inline bool is_divergence_bitmap(struct drbd_peer_md *peer_md)
+{
+ return !test_bit(__MDF_HAVE_BITMAP, &peer_md->flags) ||
+ test_bit(__MDF_PEER_DIVERGENCE_BITMAP, &peer_md->flags);
}
-static inline bool verify_can_do_stop_sector(struct drbd_device *device)
+/* A day0 slot: a totally unused peer slot, without a bitmap of its own and
+ * without flags.
+ */
+static inline bool drbd_is_day0_slot(struct drbd_md *md, int node_id)
{
- return first_peer_device(device)->connection->agreed_pro_version >= 97 &&
- first_peer_device(device)->connection->agreed_pro_version != 100;
+ struct drbd_peer_md *peer_md = &md->peers[node_id];
+
+ return node_id != md->node_id && peer_md->bitmap_index == -1 &&
+ !peer_md->flags;
}
-static inline int drbd_set_ed_uuid(struct drbd_device *device, u64 val)
+static inline u64 drbd_history_uuid(struct drbd_device *device, int i)
{
- int changed = device->ed_uuid != val;
- device->ed_uuid = val;
- return changed;
+ if (!device->ldev || i >= ARRAY_SIZE(device->ldev->md.history_uuids))
+ return 0;
+
+ return device->ldev->md.history_uuids[i];
}
static inline int drbd_queue_order_type(struct drbd_device *device)
@@ -2256,4 +3603,217 @@ static inline struct drbd_connection *first_connection(struct drbd_resource *res
struct drbd_connection, connections);
}
+static inline struct net *drbd_net_assigned_to_connection(struct drbd_connection *connection)
+{
+ struct drbd_path *path;
+ struct net *net;
+
+ rcu_read_lock();
+ path = list_first_or_null_rcu(&connection->transport.paths, struct drbd_path, list);
+ net = path ? path->net : NULL;
+ rcu_read_unlock();
+
+ return net;
+}
+
+#define NODE_MASK(id) ((u64)1 << (id))
+
+static inline void drbd_list_del_resync_request(struct drbd_peer_request *peer_req)
+{
+ peer_req->flags &= ~EE_ON_RECV_ORDER;
+ list_del(&peer_req->recv_order);
+
+ if (peer_req == peer_req->peer_device->received_last)
+ peer_req->peer_device->received_last = NULL;
+
+ if (peer_req == peer_req->peer_device->discard_last)
+ peer_req->peer_device->discard_last = NULL;
+}
+
+/*
+ * drbd_interval_same_peer - determine whether "interval" is for the same peer as "i"
+ *
+ * "i" must be an interval corresponding to a drbd_peer_request.
+ */
+static inline bool drbd_interval_same_peer(struct drbd_interval *interval, struct drbd_interval *i)
+{
+ struct drbd_peer_request *interval_peer_req, *i_peer_req;
+
+ /* Ensure we only call "container_of" if it is actually a peer request. */
+ if (interval->type == INTERVAL_LOCAL_WRITE ||
+ interval->type == INTERVAL_LOCAL_READ ||
+ interval->type == INTERVAL_PEERS_IN_SYNC_LOCK)
+ return false;
+
+ interval_peer_req = container_of(interval, struct drbd_peer_request, i);
+ i_peer_req = container_of(i, struct drbd_peer_request, i);
+ return interval_peer_req->peer_device == i_peer_req->peer_device;
+}
+
+/*
+ * drbd_should_defer_to_resync - determine whether "interval" should defer to
+ * "i" in order to ensure that resync makes progress
+ */
+static inline bool drbd_should_defer_to_resync(struct drbd_interval *interval, struct drbd_interval *i)
+{
+ if (!drbd_interval_is_resync(i))
+ return false;
+
+ /* Always defer to resync requests once the reply has been received.
+ * These just need to wait for conflicting local I/O to complete. This
+ * is necessary to ensure that resync replies received before
+ * application writes are submitted first, so that the resync writes do
+ * not overwrite newer data. */
+ if (test_bit(INTERVAL_RECEIVED, &i->flags))
+ return true;
+
+ /* If we are still waiting for a reply from the peer, only defer to the
+ * request if it is towards a different peer. The exclusivity between
+ * resync requests and application writes from another peer is
+ * necessary to avoid overwriting newer data with older in the resync.
+ * When the data in both cases is coming from the same peer, this is
+ * not necessary. The peer ensures that the data stream is correctly
+ * ordered. */
+ return !drbd_interval_same_peer(interval, i);
+}
+
+/*
+ * drbd_should_defer_to_interval - determine whether "interval" should defer to "i"
+ */
+static inline bool drbd_should_defer_to_interval(struct drbd_interval *interval,
+ struct drbd_interval *i, bool defer_to_resync)
+{
+ if (test_bit(INTERVAL_SUBMITTED, &i->flags))
+ return true;
+
+ if (defer_to_resync && drbd_should_defer_to_resync(interval, i))
+ return true;
+
+ /*
+ * We do not send conflicting resync requests because that causes
+ * difficulties associating the replies to the requests.
+ */
+ if (interval->type == INTERVAL_RESYNC_WRITE &&
+ i->type == INTERVAL_RESYNC_WRITE &&
+ test_bit(INTERVAL_READY_TO_SEND, &i->flags))
+ return true;
+
+ return false;
+}
+
+/* Find conflicts at application level instead of at disk level. */
+#define CONFLICT_FLAG_APPLICATION_ONLY (1 << 0)
+
+/*
+ * Ignore peer writes from the peer that this request relates to. This is only
+ * used for determining whether to send a request. It must not be used for
+ * determining whether to submit a request, because that would allow concurrent
+ * writes to the backing disk.
+ */
+#define CONFLICT_FLAG_IGNORE_SAME_PEER (1 << 1)
+
+/*
+ * drbd_find_conflict - find conflicting interval, if any
+ */
+static inline struct drbd_interval *drbd_find_conflict(struct drbd_device *device,
+ struct drbd_interval *interval, unsigned long flags)
+{
+ struct drbd_interval *i;
+ sector_t sector = interval->sector;
+ int size = interval->size;
+ bool application_only = flags & CONFLICT_FLAG_APPLICATION_ONLY;
+ bool defer_to_resync =
+ (interval->type == INTERVAL_LOCAL_WRITE || interval->type == INTERVAL_PEER_WRITE) &&
+ !application_only;
+ bool exclusive_until_completed = interval->type == INTERVAL_LOCAL_WRITE || application_only;
+ bool ignore_same_peer = flags & CONFLICT_FLAG_IGNORE_SAME_PEER;
+
+ lockdep_assert_held(&device->interval_lock);
+
+ drbd_for_each_overlap(i, &device->requests, sector, size) {
+ /* Ignore the interval itself. */
+ if (i == interval)
+ continue;
+
+ if (exclusive_until_completed) {
+ /* Ignore, if already completed to upper layers. */
+ if (test_bit(INTERVAL_COMPLETED, &i->flags))
+ continue;
+ } else {
+ /* Ignore, if already completed by the backing disk. */
+ if (test_bit(INTERVAL_BACKING_COMPLETED, &i->flags))
+ continue;
+ }
+
+ /* Ignore, if there is no need to defer to it. */
+ if (!drbd_should_defer_to_interval(interval, i, defer_to_resync))
+ continue;
+
+ /*
+ * Ignore peer writes from the peer that this request relates
+ * to, if requested.
+ */
+ if (ignore_same_peer && i->type == INTERVAL_PEER_WRITE && drbd_interval_same_peer(interval, i))
+ continue;
+
+ if (unlikely(application_only)) {
+ /* Ignore, if not an application request. */
+ if (!drbd_interval_is_application(i))
+ continue;
+ }
+
+ if (drbd_interval_is_write(interval)) {
+ /*
+ * Mark verify requests as conflicting rather than
+ * treating them as conflicts for us.
+ */
+ if (drbd_interval_is_verify(i)) {
+ set_bit(INTERVAL_CONFLICT, &i->flags);
+ continue;
+ }
+ } else {
+ /* Ignore other resync reads. */
+ if (i->type == INTERVAL_RESYNC_READ)
+ continue;
+
+ /* Ignore verify requests, since they are always reads. */
+ if (drbd_interval_is_verify(i))
+ continue;
+
+ /* Ignore peers-in-sync intervals, since they are always reads. */
+ if (i->type == INTERVAL_PEERS_IN_SYNC_LOCK)
+ continue;
+ }
+
+ dynamic_drbd_dbg(device,
+ "%s at %llus+%u conflicts with %s at %llus+%u\n",
+ drbd_interval_type_str(interval),
+ (unsigned long long) sector, size,
+ drbd_interval_type_str(i),
+ (unsigned long long) i->sector, i->size);
+
+ break;
+ }
+
+ return i;
+}
+
+#ifdef CONFIG_DRBD_TIMING_STATS
+#define ktime_aggregate_delta(D, ST, M) (D->M = ktime_add(D->M, ktime_sub(ktime_get(), ST)))
+#define ktime_aggregate(D, R, M) (D->M = ktime_add(D->M, ktime_sub(R->M, R->start_kt)))
+#define ktime_aggregate_pd(P, N, R, M) (P->M = ktime_add(P->M, ktime_sub(R->M[N], R->start_kt)))
+#define ktime_get_accounting(V) (V = ktime_get())
+#define ktime_get_accounting_assign(V, T) (V = T)
+#define ktime_var_for_accounting(V) ktime_t V = ktime_get()
+#define ktime_var_assign_for_accounting(V, T) ktime_t V = T
+#else
+#define ktime_aggregate_delta(D, ST, M)
+#define ktime_aggregate(D, R, M)
+#define ktime_aggregate_pd(P, N, R, M)
+#define ktime_get_accounting(V)
+#define ktime_get_accounting_assign(V, T)
+#define ktime_var_for_accounting(V)
+#define ktime_var_assign_for_accounting(V, T)
+#endif
+
#endif
diff --git a/drivers/block/drbd/drbd_interval.h b/drivers/block/drbd/drbd_interval.h
index 3749377ec172..667c91223b62 100644
--- a/drivers/block/drbd/drbd_interval.h
+++ b/drivers/block/drbd/drbd_interval.h
@@ -9,30 +9,115 @@
#include <linux/types.h>
#include <linux/rbtree.h>
-/*
- * Interval types stored directly in drbd_interval so that we can handle
+/* Interval types stored directly in drbd_interval so that we can handle
* conflicts without having to inspect the containing object. The value 0 is
- * reserved for uninitialized intervals.
- */
+ * reserved for uninitialized intervals. */
enum drbd_interval_type {
INTERVAL_LOCAL_WRITE = 1,
INTERVAL_PEER_WRITE,
INTERVAL_LOCAL_READ,
INTERVAL_PEER_READ,
- INTERVAL_RESYNC_WRITE, /* C_SYNC_TARGET */
- INTERVAL_RESYNC_READ, /* C_SYNC_SOURCE */
- INTERVAL_OV_READ_SOURCE, /* C_VERIFY_S */
- INTERVAL_OV_READ_TARGET, /* C_VERIFY_T */
+ INTERVAL_RESYNC_WRITE, /* L_SYNC_TARGET */
+ INTERVAL_RESYNC_READ, /* L_SYNC_SOURCE */
+ INTERVAL_OV_READ_SOURCE, /* L_VERIFY_S */
+ INTERVAL_OV_READ_TARGET, /* L_VERIFY_T */
+ INTERVAL_PEERS_IN_SYNC_LOCK,
};
+#define INTERVAL_TYPE_MASK(type) (1 << (type))
+
enum drbd_interval_flags {
- /* Someone is waiting on device->misc_wait for this to make progress. */
- INTERVAL_WAITING,
+ /* Whether this peer request may be sent. */
+ INTERVAL_READY_TO_SEND,
+
+ /*
+ * Used for resync reads. This flag is set after sending and is used to
+ * manage the lifetime of the request. When INTERVAL_SENT is not set,
+ * the sending path still has a reference to the request.
+ */
+ INTERVAL_SENT,
+
+ /*
+ * Whether this peer request has been received yet.
+ *
+ * For resync reads, this flag is set when the corresponding ack has
+ * been received and is used to manage the lifetime of the request.
+ * When INTERVAL_RECEIVED is not set, the receiving path has a
+ * reference to the request. This reference counting is protected by
+ * peer_reqs_lock.
+ */
+ INTERVAL_RECEIVED,
+
+ /* Whether this has been queued after conflict. */
+ INTERVAL_SUBMIT_CONFLICT_QUEUED,
+
+ /* Whether this has been submitted already. */
+ INTERVAL_SUBMITTED,
+
+ /* Whether the local backing device bio is complete. */
+ INTERVAL_BACKING_COMPLETED,
/* This has been completed already; ignore for conflict detection. */
INTERVAL_COMPLETED,
+
+ /* For verify requests: whether this has conflicts. */
+ INTERVAL_CONFLICT,
+
+ /* For resync requests: whether this was canceled while waiting for conflict resolution. */
+ INTERVAL_CANCELED,
+
+ /*
+ * For local requests: whether this is done.
+ *
+ * Included here instead of in local_rq_state to allow access with
+ * atomic bit operations instead of taking rq_lock.
+ */
+ INTERVAL_DONE,
+
+ /*
+ * For local requests: when we put the AL extent for this request, it
+ * was the last in that extent.
+ *
+ * Included here instead of in local_rq_state to allow access with
+ * atomic bit operations instead of taking rq_lock.
+ */
+ INTERVAL_AL_EXTENT_LAST,
+
+ /* For resync requests: an application write waits for this one,
+ * so the resync depth bound does not hold it back.
+ */
+ INTERVAL_APP_WAITS,
};
+/* Intervals used to manage conflicts between application requests and various
+ * internal requests, so that the disk content is deterministic.
+ *
+ * The requests progress through states indicated by successively setting the
+ * flags "INTERVAL_SUBMITTED", "INTERVAL_BACKING_COMPLETED" and
+ * "INTERVAL_COMPLETED".
+ *
+ * Application and resync requests wait to be submitted until any conflicts
+ * that are "INTERVAL_SUBMITTED" have reached "INTERVAL_BACKING_COMPLETED"
+ * state. Application requests also wait for conflicting application requests
+ * to ensure consistency between the replicated copies. In addition,
+ * application requests wait for resync requests that have not yet been
+ * submitted. Resync takes priority over application writes in this way because
+ * a resync locks each block at most once, so it will finish at some point,
+ * whereas the application may repeatedly write the same blocks, which would
+ * potentially lock out resync indefinitely.
+ *
+ * Resync read requests do not conflict with each other, but they are
+ * nevertheless mutually exclusive with writes, so that the bitmap can be
+ * updated reliably.
+ *
+ * Verify requests do not wait for other requests. If there are conflicts, they
+ * are simply cancelled. Futhermore, they do not lock out other requests;
+ * instead they are simply marked as having conflicts and ignored.
+ *
+ * Application write request intervals are retained even when they are
+ * "INTERVAL_COMPLETED", so that they can be used to look up remote replies
+ * that are still pending.
+ */
struct drbd_interval {
struct rb_node rb;
sector_t sector; /* start sector of the interval */
@@ -45,6 +130,28 @@ struct drbd_interval {
unsigned int partially_in_al_next_enr;
};
+static inline bool drbd_interval_is_application(struct drbd_interval *i)
+{
+ return i->type == INTERVAL_LOCAL_WRITE || i->type == INTERVAL_PEER_WRITE ||
+ i->type == INTERVAL_LOCAL_READ || i->type == INTERVAL_PEER_READ;
+}
+
+static inline bool drbd_interval_is_write(struct drbd_interval *i)
+{
+ return i->type == INTERVAL_LOCAL_WRITE || i->type == INTERVAL_PEER_WRITE ||
+ i->type == INTERVAL_RESYNC_WRITE;
+}
+
+static inline bool drbd_interval_is_resync(struct drbd_interval *i)
+{
+ return i->type == INTERVAL_RESYNC_WRITE || i->type == INTERVAL_RESYNC_READ;
+}
+
+static inline bool drbd_interval_is_verify(struct drbd_interval *i)
+{
+ return i->type == INTERVAL_OV_READ_SOURCE || i->type == INTERVAL_OV_READ_TARGET;
+}
+
static inline bool drbd_interval_is_local(struct drbd_interval *i)
{
return i->type == INTERVAL_LOCAL_READ || i->type == INTERVAL_LOCAL_WRITE;
@@ -60,6 +167,7 @@ static inline bool drbd_interval_empty(struct drbd_interval *i)
return RB_EMPTY_NODE(&i->rb);
}
+const char *drbd_interval_type_str(struct drbd_interval *i);
bool drbd_insert_interval(struct rb_root *root, struct drbd_interval *this);
bool drbd_contains_interval(struct rb_root *root, sector_t sector,
struct drbd_interval *interval);
@@ -68,6 +176,8 @@ struct drbd_interval *drbd_find_overlap(struct rb_root *root, sector_t sector,
unsigned int size);
struct drbd_interval *drbd_next_overlap(struct drbd_interval *i,
sector_t sector, unsigned int size);
+void drbd_update_interval_size(struct drbd_interval *this,
+ unsigned int new_size);
#define drbd_for_each_overlap(i, root, sector, size) \
for (i = drbd_find_overlap(root, sector, size); \
diff --git a/drivers/block/drbd/drbd_polymorph_printk.h b/drivers/block/drbd/drbd_polymorph_printk.h
index 71dcdd1f45b9..7b0873d2980e 100644
--- a/drivers/block/drbd/drbd_polymorph_printk.h
+++ b/drivers/block/drbd/drbd_polymorph_printk.h
@@ -42,24 +42,29 @@
__unregistered, __r->name, ## args); \
})
-/*
- * As long as the connection is still "registered", the resource
- * can not yet be "unregistered", no need to test R_UNREGISTERED
- */
+// As long as the connection is still "registered", the resource
+// can not yet be "unregistered", no need to test R_UNREGISTERED
#define ___drbd_printk_peer_device(prmacro, rlt, peer_device, lvl_or_desc, fmt, args...)\
({ \
- const struct drbd_peer_device *__pd = \
- (const struct drbd_peer_device *)(peer_device); \
- const struct drbd_device *__d = __pd->device; \
- const struct drbd_connection *__c = __pd->connection; \
- const struct drbd_resource *__r = __d->resource; \
+ const struct drbd_peer_device *__pd; \
+ const struct drbd_device *__d; \
+ const struct drbd_connection *__c; \
+ const struct drbd_resource *__r; \
+ const char *__cn; \
const char *__unregistered = ""; \
+ rcu_read_lock(); \
+ __pd = (const struct drbd_peer_device *)(peer_device); \
+ __d = __pd->device; \
+ __c = __pd->connection; \
+ __r = __d->resource; \
+ __cn = rcu_dereference(__c->transport.net_conf)->name; \
if (test_bit(C_UNREGISTERED, &__c->flags)) \
__unregistered = "/unregistered/"; \
if (drbd_peer_device_ratelimit(__pd, rlt)) \
- prmacro(lvl_or_desc, "drbd %s%s/%u drbd%u: " fmt, \
- __unregistered, __r->name, __d->vnr, __d->minor,\
- ## args); \
+ prmacro(lvl_or_desc, "drbd %s%s/%u drbd%u %s: " fmt, \
+ __unregistered, __r->name, __d->vnr, __d->minor, __cn, \
+ ## args); \
+ rcu_read_unlock(); \
})
#define ___drbd_printk_connection(prmacro, rlt, connection, lvl_or_desc, fmt, args...) \
@@ -67,12 +72,16 @@
const struct drbd_connection *__c = \
(const struct drbd_connection *)(connection); \
const struct drbd_resource *__r = __c->resource; \
+ const char *__cn; \
const char *__unregistered = ""; \
+ rcu_read_lock(); \
+ __cn = rcu_dereference(__c->transport.net_conf)->name; \
if (test_bit(C_UNREGISTERED, &__c->flags)) \
__unregistered = "/unregistered/"; \
if (drbd_connection_ratelimit(__c, rlt)) \
- prmacro(lvl_or_desc, "drbd %s%s: " fmt, \
- __unregistered, __r->name, ## args); \
+ prmacro(lvl_or_desc, "drbd %s%s %s: " fmt, \
+ __unregistered, __r->name, __cn, ## args); \
+ rcu_read_unlock(); \
})
#define __drbd_printk_device(rlt, device, level, fmt, args...)\
diff --git a/drivers/block/drbd/drbd_req.h b/drivers/block/drbd/drbd_req.h
index 51b12a960e45..4e2aa70ec630 100644
--- a/drivers/block/drbd/drbd_req.h
+++ b/drivers/block/drbd/drbd_req.h
@@ -60,37 +60,33 @@
*/
enum drbd_req_event {
- CREATED,
- TO_BE_SENT,
TO_BE_SUBMITTED,
- /* XXX yes, now I am inconsistent...
- * these are not "events" but "actions"
- * oh, well... */
- QUEUE_FOR_NET_WRITE,
- QUEUE_FOR_NET_READ,
- QUEUE_FOR_SEND_OOS,
+ NEW_NET_READ,
+ NEW_NET_WRITE,
+ NEW_NET_OOS,
+ READY_FOR_NET,
+ SKIP_OOS,
- /* An empty flush is queued as P_BARRIER,
- * which will cause it to complete "successfully",
- * even if the local disk flush failed.
+ /* For an empty flush, mark that a corresponding barrier has been sent
+ * to this peer. This causes it to complete "successfully", even if the
+ * local disk flush failed.
*
* Just like "real" requests, empty flushes (blkdev_issue_flush()) will
* only see an error if neither local nor remote data is reachable. */
- QUEUE_AS_DRBD_BARRIER,
+ BARRIER_SENT,
SEND_CANCELED,
SEND_FAILED,
HANDED_OVER_TO_NETWORK,
OOS_HANDED_TO_NETWORK,
- CONNECTION_LOST_WHILE_PENDING,
- READ_RETRY_REMOTE_CANCELED,
+ CONNECTION_LOST,
+ CONNECTION_LOST_WHILE_SUSPENDED,
RECV_ACKED_BY_PEER,
WRITE_ACKED_BY_PEER,
WRITE_ACKED_BY_PEER_AND_SIS, /* and set_in_sync */
- CONFLICT_RESOLVED,
- POSTPONE_WRITE,
NEG_ACKED,
+ POSTPONED_BY_PEER,
BARRIER_ACKED, /* in protocol A and B */
DATA_RECEIVED, /* (remote read) */
@@ -103,82 +99,94 @@ enum drbd_req_event {
ABORT_DISK_IO,
RESEND,
- FAIL_FROZEN_DISK_IO,
- RESTART_FROZEN_DISK_IO,
+ CANCEL_SUSPENDED_IO,
+ COMPLETION_RESUMED,
+ NEW_UUID_CONFIRMED,
NOTHING,
};
-/* encoding of request states for now. we don't actually need that many bits.
- * we don't need to do atomic bit operations either, since most of the time we
- * need to look at the connection state and/or manipulate some lists at the
- * same time, so we should hold the request lock anyways.
+/*
+ * Encoding of request states. Modifications are protected by rq_lock. We don't
+ * do atomic bit operations.
*/
enum drbd_req_state_bits {
- /* 3210
- * 0000: no local possible
- * 0001: to be submitted
- * UNUSED, we could map: 011: submitted, completion still pending
- * 0110: completed ok
- * 0010: completed with error
- * 1001: Aborted (before completion)
- * 1x10: Aborted and completed -> free
- */
- __RQ_LOCAL_PENDING,
- __RQ_LOCAL_COMPLETED,
- __RQ_LOCAL_OK,
- __RQ_LOCAL_ABORTED,
-
- /* 87654
- * 00000: no network possible
- * 00001: to be send
- * 00011: to be send, on worker queue
- * 00101: sent, expecting recv_ack (B) or write_ack (C)
- * 11101: sent,
- * recv_ack (B) or implicit "ack" (A),
- * still waiting for the barrier ack.
- * master_bio may already be completed and invalidated.
- * 11100: write acked (C),
- * data received (for remote read, any protocol)
- * or finally the barrier ack has arrived (B,A)...
- * request can be freed
- * 01100: neg-acked (write, protocol C)
- * or neg-d-acked (read, any protocol)
- * or killed from the transfer log
- * during cleanup after connection loss
- * request can be freed
- * 01000: canceled or send failed...
- * request can be freed
+ /*
+ * Here are the possible combinations of the core net flags pending, pending-oos,
+ * queued, ready, sent, done, ok.
+ *
+ * <none>:
+ * No network required, or not yet processed.
+ * pending,queued:
+ * To be sent, must not be processed yet.
+ * pending,queued,ready:
+ * To be sent, processing allowed.
+ * pending,ready,sent:
+ * Sent, expecting P_RECV_ACK (B) or P_WRITE_ACK (C).
+ * queued,ready,ok:
+ * P_RECV_ACK (B) or P_WRITE_ACK (C) received before request marked
+ * as having been sent.
+ * ready,sent,ok:
+ * Sent, implicit "ack" (A), P_RECV_ACK (B) or P_WRITE_ACK (C) received.
+ * Still waiting for the barrier ack.
+ * master_bio may already be completed and invalidated.
+ * pending:
+ * Intended for this peer, but connection lost before processing
+ * allowed.
+ * pending,ready:
+ * Intended for this peer, but connection lost. If
+ * IO is suspended, it will stay in this state until the connection
+ * is restored or IO is resumed.
+ * ready,sent,done,ok:
+ * Data received (for remote read, any protocol),
+ * or finally the barrier ack has arrived.
+ * ready,sent,done:
+ * Received P_NEG_ACK for write (protocol C, or we are SyncSource),
+ * or P_NEG_DREPLY for read (any protocol).
+ * Or cleaned up after connection loss after send.
+ * pending-oos,queued,done:
+ * P_OUT_OF_SYNC to be sent, must not be processed yet.
+ * pending-oos,queued,ready,done:
+ * P_OUT_OF_SYNC to be sent, processing allowed.
+ * queued,ready,done:
+ * P_OUT_OF_SYNC was intended, but skipped.
+ * done:
+ * P_OUT_OF_SYNC was intended, but connection lost before processing
+ * allowed.
+ * ready,done:
+ * P_OUT_OF_SYNC sent.
+ * Or cleaned up after connection loss, either before send or when
+ * only P_OUT_OF_SYNC was intended.
*/
- /* if "SENT" is not set, yet, this can still fail or be canceled.
- * if "SENT" is set already, we still wait for an Ack packet.
- * when cleared, the master_bio may be completed.
- * in (B,A) the request object may still linger on the transaction log
- * until the corresponding barrier ack comes in */
+ /* Pending some network interaction towards the peer apart from
+ * barriers or P_OUT_OF_SYNC.
+ * If "sent" is not yet set, this can still fail or be canceled.
+ * While set, the master_bio may not be completed. */
__RQ_NET_PENDING,
- /* If it is QUEUED, and it is a WRITE, it is also registered in the
- * transfer log. Currently we need this flag to avoid conflicts between
- * worker canceling the request and tl_clear_barrier killing it from
- * transfer log. We should restructure the code so this conflict does
- * no longer occur. */
+ /* Pending send of P_OUT_OF_SYNC */
+ __RQ_NET_PENDING_OOS,
+
+ /* The sender might store pointers to it */
__RQ_NET_QUEUED,
- /* well, actually only "handed over to the network stack".
- *
- * TODO can potentially be dropped because of the similar meaning
- * of RQ_NET_SENT and ~RQ_NET_QUEUED.
- * however it is not exactly the same. before we drop it
- * we must ensure that we can tell a request with network part
- * from a request without, regardless of what happens to it. */
+ /* Ready for processing by the sender */
+ __RQ_NET_READY,
+
+ /* Well, actually only "handed over to the network stack". */
__RQ_NET_SENT,
- /* when set, the request may be freed (if RQ_NET_QUEUED is clear).
- * basically this means the corresponding P_BARRIER_ACK was received */
+ /* When set, the data stage is done, as far as interaction with this
+ * peer is concerned. Basically this means the corresponding
+ * P_BARRIER_ACK was received. */
__RQ_NET_DONE,
- /* whether or not we know (C) or pretend (B,A) that the write
- * was successfully written on the peer.
+ /* Set when the request was successful. That is, the corresponding
+ * condition is fulfilled:
+ * - The write was sent (A)
+ * - Receipt of the write was acknowledged (B)
+ * - The write was successfully written on the peer (C)
+ * - Read data was received
*/
__RQ_NET_OK,
@@ -188,6 +196,29 @@ enum drbd_req_state_bits {
/* keep this last, its for the RQ_NET_MASK */
__RQ_NET_MAX,
+ /* We expect a receive ACK (wire proto B) */
+ __RQ_EXP_RECEIVE_ACK,
+
+ /* We expect a write ACK (wite proto C) */
+ __RQ_EXP_WRITE_ACK,
+
+ /* waiting for a barrier ack, did an extra kref_get */
+ __RQ_EXP_BARR_ACK,
+
+ /* 4321
+ * 0000: no local possible
+ * 0001: to be submitted
+ * UNUSED, we could map: 011: submitted, completion still pending
+ * 0110: completed ok
+ * 0010: completed with error
+ * 1001: Aborted (before completion)
+ * 1x10: Aborted and completed -> free
+ */
+ __RQ_LOCAL_PENDING,
+ __RQ_LOCAL_COMPLETED,
+ __RQ_LOCAL_OK,
+ __RQ_LOCAL_ABORTED,
+
/* Set when this is a write, clear for a read */
__RQ_WRITE,
__RQ_WSAME,
@@ -197,6 +228,12 @@ enum drbd_req_state_bits {
/* Should call drbd_al_complete_io() for this request... */
__RQ_IN_ACT_LOG,
+ /* Accounted in device->wait_for_actlog_ecnt, respectively in
+ * device->ap_actlog_cnt. AL_SUSPENDED may flip at any time.
+ */
+ __RQ_WAIT_FOR_AL_ECNT,
+ __RQ_AP_ACTLOG_CNT,
+
/* This was the most recent request during some blk_finish_plug()
* or its implicit from-schedule equivalent.
* We may use it as hint to send a P_UNPLUG_REMOTE */
@@ -209,25 +246,25 @@ enum drbd_req_state_bits {
* but was not, because of drbd_suspended() */
__RQ_COMPLETION_SUSP,
- /* We expect a receive ACK (wire proto B) */
- __RQ_EXP_RECEIVE_ACK,
-
- /* We expect a write ACK (wite proto C) */
- __RQ_EXP_WRITE_ACK,
+ /* Completion held back because this write was acknowledged in a data
+ * generation that a diskless primary rotated to optimistically, before
+ * any quorate peer confirmed persisting the new current UUID
+ * (confirm-before-complete). The hold is implemented via
+ * RQ_COMPLETION_SUSP; this flag marks it as a generation hold so that
+ * NEW_UUID_CONFIRMED releases exactly these requests once an in-order
+ * barrier ack proves the generation durable.
+ */
+ __RQ_UNCONF_GEN,
- /* waiting for a barrier ack, did an extra kref_get */
- __RQ_EXP_BARR_ACK,
+ /* Completed to the application with success. Read at a connection
+ * loss, to tell an acknowledged write from one that failed.
+ */
+ __RQ_COMPLETED_OK,
};
-
-#define RQ_LOCAL_PENDING (1UL << __RQ_LOCAL_PENDING)
-#define RQ_LOCAL_COMPLETED (1UL << __RQ_LOCAL_COMPLETED)
-#define RQ_LOCAL_OK (1UL << __RQ_LOCAL_OK)
-#define RQ_LOCAL_ABORTED (1UL << __RQ_LOCAL_ABORTED)
-
-#define RQ_LOCAL_MASK ((RQ_LOCAL_ABORTED << 1)-1)
-
#define RQ_NET_PENDING (1UL << __RQ_NET_PENDING)
+#define RQ_NET_PENDING_OOS (1UL << __RQ_NET_PENDING_OOS)
#define RQ_NET_QUEUED (1UL << __RQ_NET_QUEUED)
+#define RQ_NET_READY (1UL << __RQ_NET_READY)
#define RQ_NET_SENT (1UL << __RQ_NET_SENT)
#define RQ_NET_DONE (1UL << __RQ_NET_DONE)
#define RQ_NET_OK (1UL << __RQ_NET_OK)
@@ -235,22 +272,53 @@ enum drbd_req_state_bits {
#define RQ_NET_MASK (((1UL << __RQ_NET_MAX)-1) & ~RQ_LOCAL_MASK)
+#define RQ_EXP_RECEIVE_ACK (1UL << __RQ_EXP_RECEIVE_ACK)
+#define RQ_EXP_WRITE_ACK (1UL << __RQ_EXP_WRITE_ACK)
+#define RQ_EXP_BARR_ACK (1UL << __RQ_EXP_BARR_ACK)
+
+#define RQ_LOCAL_PENDING (1UL << __RQ_LOCAL_PENDING)
+#define RQ_LOCAL_COMPLETED (1UL << __RQ_LOCAL_COMPLETED)
+#define RQ_LOCAL_OK (1UL << __RQ_LOCAL_OK)
+#define RQ_LOCAL_ABORTED (1UL << __RQ_LOCAL_ABORTED)
+
+#define RQ_LOCAL_MASK \
+ (RQ_LOCAL_ABORTED | RQ_LOCAL_OK | RQ_LOCAL_COMPLETED | RQ_LOCAL_PENDING)
+
#define RQ_WRITE (1UL << __RQ_WRITE)
#define RQ_WSAME (1UL << __RQ_WSAME)
#define RQ_UNMAP (1UL << __RQ_UNMAP)
#define RQ_ZEROES (1UL << __RQ_ZEROES)
#define RQ_IN_ACT_LOG (1UL << __RQ_IN_ACT_LOG)
+#define RQ_WAIT_FOR_AL_ECNT (1UL << __RQ_WAIT_FOR_AL_ECNT)
+#define RQ_AP_ACTLOG_CNT (1UL << __RQ_AP_ACTLOG_CNT)
#define RQ_UNPLUG (1UL << __RQ_UNPLUG)
#define RQ_POSTPONED (1UL << __RQ_POSTPONED)
#define RQ_COMPLETION_SUSP (1UL << __RQ_COMPLETION_SUSP)
-#define RQ_EXP_RECEIVE_ACK (1UL << __RQ_EXP_RECEIVE_ACK)
-#define RQ_EXP_WRITE_ACK (1UL << __RQ_EXP_WRITE_ACK)
-#define RQ_EXP_BARR_ACK (1UL << __RQ_EXP_BARR_ACK)
-
-/* For waking up the frozen transfer log mod_req() has to return if the request
- should be counted in the epoch object*/
-#define MR_WRITE 1
-#define MR_READ 2
+#define RQ_UNCONF_GEN (1UL << __RQ_UNCONF_GEN)
+#define RQ_COMPLETED_OK (1UL << __RQ_COMPLETED_OK)
+
+
+/* these flags go into local_rq_state,
+ * orhter flags go into their respective net_rq_state[idx] */
+#define RQ_STATE_0_MASK \
+ (RQ_LOCAL_MASK |\
+ RQ_WRITE |\
+ RQ_WSAME |\
+ RQ_UNMAP |\
+ RQ_ZEROES |\
+ RQ_IN_ACT_LOG |\
+ RQ_WAIT_FOR_AL_ECNT |\
+ RQ_AP_ACTLOG_CNT |\
+ RQ_UNPLUG |\
+ RQ_POSTPONED |\
+ RQ_COMPLETION_SUSP |\
+ RQ_UNCONF_GEN |\
+ RQ_COMPLETED_OK)
+
+static inline bool drbd_req_is_write(struct drbd_request *req)
+{
+ return req->local_rq_state & RQ_WRITE;
+}
/* Short lived temporary struct on the stack.
* We could squirrel the error to be returned into
@@ -260,61 +328,63 @@ struct bio_and_error {
int error;
};
-extern void start_new_tl_epoch(struct drbd_connection *connection);
-extern void drbd_req_destroy(struct kref *kref);
-extern int __req_mod(struct drbd_request *req, enum drbd_req_event what,
- struct drbd_peer_device *peer_device,
- struct bio_and_error *m);
-extern void complete_master_bio(struct drbd_device *device,
- struct bio_and_error *m);
-extern void request_timer_fn(struct timer_list *t);
-extern void tl_restart(struct drbd_connection *connection, enum drbd_req_event what);
-extern void _tl_restart(struct drbd_connection *connection, enum drbd_req_event what);
-extern void tl_abort_disk_io(struct drbd_device *device);
+bool start_new_tl_epoch(struct drbd_resource *resource);
+void drbd_req_destroy(struct kref *kref);
+void __req_mod(struct drbd_request *req, enum drbd_req_event what,
+ struct drbd_peer_device *peer_device, struct bio_and_error *m);
+void complete_master_bio(struct drbd_device *device, struct bio_and_error *m);
+void drbd_release_conflicts(struct drbd_device *device,
+ struct drbd_interval *release_interval);
+void drbd_put_ref_tl_walk(struct drbd_request *req, int done_put, int oos_send_put);
+void drbd_set_pending_out_of_sync(struct drbd_peer_device *peer_device);
+void request_timer_fn(struct timer_list *t);
+void tl_walk(struct drbd_connection *connection,
+ struct drbd_request **from_req, enum drbd_req_event what);
+void __tl_walk(struct drbd_resource * const resource,
+ struct drbd_connection * const connection,
+ struct drbd_request **from_req, const enum drbd_req_event what);
+void drbd_destroy_peer_ack_if_done(struct drbd_peer_ack *peer_ack);
+int w_queue_peer_ack(struct drbd_work *w, int cancel);
+void drbd_queue_peer_ack(struct drbd_resource *resource,
+ struct drbd_request *req);
+bool drbd_should_do_remote(struct drbd_peer_device *peer_device,
+ enum which_state which);
+void drbd_reclaim_req(struct rcu_head *rp);
/* this is in drbd_main.c */
-extern void drbd_restart_request(struct drbd_request *req);
+void drbd_restart_request(struct drbd_request *req);
+void drbd_restart_suspended_reqs(struct drbd_resource *resource);
/* use this if you don't want to deal with calling complete_master_bio()
* outside the spinlock, e.g. when walking some list on cleanup. */
-static inline int _req_mod(struct drbd_request *req, enum drbd_req_event what,
+static inline void _req_mod(struct drbd_request *req, enum drbd_req_event what,
struct drbd_peer_device *peer_device)
{
struct drbd_device *device = req->device;
struct bio_and_error m;
- int rv;
/* __req_mod possibly frees req, do not touch req after that! */
- rv = __req_mod(req, what, peer_device, &m);
+ __req_mod(req, what, peer_device, &m);
if (m.bio)
complete_master_bio(device, &m);
-
- return rv;
}
-/* completion of master bio is outside of our spinlock.
- * We still may or may not be inside some irqs disabled section
- * of the lower level driver completion callback, so we need to
- * spin_lock_irqsave here. */
-static inline int req_mod(struct drbd_request *req,
+/* completion of master bio is outside of spinlock.
+ * If you need it irqsave, do it your self!
+ * Which means: don't use from bio endio callback. */
+static inline void req_mod(struct drbd_request *req,
enum drbd_req_event what,
struct drbd_peer_device *peer_device)
{
- unsigned long flags;
struct drbd_device *device = req->device;
struct bio_and_error m;
- int rv;
- spin_lock_irqsave(&device->resource->req_lock, flags);
- rv = __req_mod(req, what, peer_device, &m);
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
+ read_lock_irq(&device->resource->state_rwlock);
+ __req_mod(req, what, peer_device, &m);
+ read_unlock_irq(&device->resource->state_rwlock);
if (m.bio)
complete_master_bio(device, &m);
-
- return rv;
}
-extern bool drbd_should_do_remote(union drbd_dev_state);
-
#endif
diff --git a/drivers/block/drbd/drbd_state.h b/drivers/block/drbd/drbd_state.h
index cbaeb8018dbf..95e6beebe47d 100644
--- a/drivers/block/drbd/drbd_state.h
+++ b/drivers/block/drbd/drbd_state.h
@@ -2,26 +2,19 @@
#ifndef DRBD_STATE_H
#define DRBD_STATE_H
+#include "drbd_protocol.h"
+
+struct drbd_resource;
struct drbd_device;
struct drbd_connection;
+struct drbd_peer_device;
+struct drbd_work;
+struct twopc_request;
/**
* DOC: DRBD State macros
*
* These macros are used to express state changes in easily readable form.
- *
- * The NS macros expand to a mask and a value, that can be bit ored onto the
- * current state as soon as the spinlock (req_lock) was taken.
- *
- * The _NS macros are used for state functions that get called with the
- * spinlock. These macros expand directly to the new state value.
- *
- * Besides the basic forms NS() and _NS() additional _?NS[23] are defined
- * to express state changes that affect more than one aspect of the state.
- *
- * E.g. NS2(conn, C_CONNECTED, peer, R_SECONDARY)
- * Means that the network connection was established and that the peer
- * is in secondary role.
*/
#define role_MASK R_MASK
#define peer_MASK R_MASK
@@ -34,141 +27,208 @@ struct drbd_connection;
#define susp_nod_MASK 1
#define susp_fen_MASK 1
-#define NS(T, S) \
- ({ union drbd_state mask; mask.i = 0; mask.T = T##_MASK; mask; }), \
- ({ union drbd_state val; val.i = 0; val.T = (S); val; })
-#define NS2(T1, S1, T2, S2) \
- ({ union drbd_state mask; mask.i = 0; mask.T1 = T1##_MASK; \
- mask.T2 = T2##_MASK; mask; }), \
- ({ union drbd_state val; val.i = 0; val.T1 = (S1); \
- val.T2 = (S2); val; })
-#define NS3(T1, S1, T2, S2, T3, S3) \
- ({ union drbd_state mask; mask.i = 0; mask.T1 = T1##_MASK; \
- mask.T2 = T2##_MASK; mask.T3 = T3##_MASK; mask; }), \
- ({ union drbd_state val; val.i = 0; val.T1 = (S1); \
- val.T2 = (S2); val.T3 = (S3); val; })
-
-#define _NS(D, T, S) \
- D, ({ union drbd_state __ns; __ns = drbd_read_state(D); __ns.T = (S); __ns; })
-#define _NS2(D, T1, S1, T2, S2) \
- D, ({ union drbd_state __ns; __ns = drbd_read_state(D); __ns.T1 = (S1); \
- __ns.T2 = (S2); __ns; })
-#define _NS3(D, T1, S1, T2, S2, T3, S3) \
- D, ({ union drbd_state __ns; __ns = drbd_read_state(D); __ns.T1 = (S1); \
- __ns.T2 = (S2); __ns.T3 = (S3); __ns; })
-
enum chg_state_flags {
- CS_HARD = 1 << 0,
+ CS_HARD = 1 << 0, /* Forced state change, such as a connection loss */
CS_VERBOSE = 1 << 1,
CS_WAIT_COMPLETE = 1 << 2,
CS_SERIALIZE = 1 << 3,
- CS_ORDERED = CS_WAIT_COMPLETE + CS_SERIALIZE,
- CS_LOCAL_ONLY = 1 << 4, /* Do not consider a device pair wide state change */
- CS_DC_ROLE = 1 << 5, /* DC = display as connection state change */
- CS_DC_PEER = 1 << 6,
- CS_DC_CONN = 1 << 7,
- CS_DC_DISK = 1 << 8,
- CS_DC_PDSK = 1 << 9,
- CS_DC_SUSP = 1 << 10,
- CS_DC_MASK = CS_DC_ROLE + CS_DC_PEER + CS_DC_CONN + CS_DC_DISK + CS_DC_PDSK,
- CS_IGN_OUTD_FAIL = 1 << 11,
-
- /* Make sure no meta data IO is in flight, by calling
- * drbd_md_get_buffer(). Used for graceful detach. */
- CS_INHIBIT_MD_IO = 1 << 12,
+ CS_ALREADY_SERIALIZED = 1 << 4, /* resource->state_sem already taken */
+ CS_LOCAL_ONLY = 1 << 5, /* Do not consider a cluster-wide state change */
+ CS_PREPARE = 1 << 6,
+ CS_PREPARED = 1 << 7,
+ CS_ABORT = 1 << 8,
+ CS_TWOPC = 1 << 9,
+ CS_IGN_OUTD_FAIL = 1 << 10,
+ CS_DONT_RETRY = 1 << 11, /* Disable internal retry. Caller has a retry loop */
+ CS_FORCE_RECALC = 1 << 13, /* Force re-evaluation of state logic */
+ CS_CLUSTER_WIDE = 1 << 14, /* Make this a cluster wide state change! */
+ CS_FP_LOCAL_UP_TO_DATE = 1 << 15, /* force promotion by making local disk state up_to_date */
+ CS_FP_OUTDATE_PEERS = 1 << 16, /* force promotion by marking unknown peers as outdated */
+ CS_FS_IGN_OPENERS = 1 << 17, /* force demote, ignore openers */
+ CS_FORCE_ROLE = 1 << 18, /* drbd_set_role(): the user forces the role change */
+ CS_NO_SIGNALS = 1 << 19, /* implicit change: ignore the caller's signals */
};
-/* drbd_dev_state and drbd_state are different types. This is to stress the
- small difference. There is no suspended flag (.susp), and no suspended
- while fence handler runs flas (susp_fen). */
-union drbd_dev_state {
- struct {
-#if defined(__LITTLE_ENDIAN_BITFIELD)
- unsigned role:2 ; /* 3/4 primary/secondary/unknown */
- unsigned peer:2 ; /* 3/4 primary/secondary/unknown */
- unsigned conn:5 ; /* 17/32 cstates */
- unsigned disk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned pdsk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned _unused:1 ;
- unsigned aftr_isp:1 ; /* isp .. imposed sync pause */
- unsigned peer_isp:1 ;
- unsigned user_isp:1 ;
- unsigned _pad:11; /* 0 unused */
-#elif defined(__BIG_ENDIAN_BITFIELD)
- unsigned _pad:11;
- unsigned user_isp:1 ;
- unsigned peer_isp:1 ;
- unsigned aftr_isp:1 ; /* isp .. imposed sync pause */
- unsigned _unused:1 ;
- unsigned pdsk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned disk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned conn:5 ; /* 17/32 cstates */
- unsigned peer:2 ; /* 3/4 primary/secondary/unknown */
- unsigned role:2 ; /* 3/4 primary/secondary/unknown */
-#else
-# error "this endianess is not supported"
-#endif
- };
- unsigned int i;
-};
+void drbd_resume_al(struct drbd_device *device);
+void drbd_flush_connect_prepared(struct drbd_resource *resource, int node_a, int node_b);
+void drbd_flush_connect_committed(struct drbd_resource *resource);
-extern enum drbd_state_rv drbd_change_state(struct drbd_device *device,
- enum chg_state_flags f,
- union drbd_state mask,
- union drbd_state val);
-extern void drbd_force_state(struct drbd_device *, union drbd_state,
- union drbd_state);
-extern enum drbd_state_rv _drbd_request_state(struct drbd_device *,
- union drbd_state,
- union drbd_state,
- enum chg_state_flags);
-
-extern enum drbd_state_rv
-_drbd_request_state_holding_state_mutex(struct drbd_device *, union drbd_state,
- union drbd_state, enum chg_state_flags);
-
-extern enum drbd_state_rv _drbd_set_state(struct drbd_device *, union drbd_state,
- enum chg_state_flags,
- struct completion *done);
-extern void print_st_err(struct drbd_device *, union drbd_state,
- union drbd_state, enum drbd_state_rv);
-
-enum drbd_state_rv
-_conn_request_state(struct drbd_connection *connection, union drbd_state mask, union drbd_state val,
- enum chg_state_flags flags);
-
-enum drbd_state_rv
-conn_request_state(struct drbd_connection *connection, union drbd_state mask, union drbd_state val,
- enum chg_state_flags flags);
-
-extern void drbd_resume_al(struct drbd_device *device);
-extern bool conn_all_vols_unconf(struct drbd_connection *connection);
+enum drbd_disk_state conn_highest_disk(struct drbd_connection *connection);
+enum drbd_disk_state conn_highest_pdsk(struct drbd_connection *connection);
-/**
- * drbd_request_state() - Request a state change
- * @device: DRBD device.
- * @mask: mask of state bits to change.
- * @val: value of new state bits.
+void state_change_lock(struct drbd_resource *resource,
+ unsigned long *irq_flags, enum chg_state_flags flags);
+void state_change_unlock(struct drbd_resource *resource,
+ unsigned long *irq_flags);
+
+void begin_state_change(struct drbd_resource *resource,
+ unsigned long *irq_flags, enum chg_state_flags flags);
+enum drbd_state_rv end_state_change(struct drbd_resource *resource,
+ unsigned long *irq_flags, const char *tag);
+void abort_state_change(struct drbd_resource *resource,
+ unsigned long *irq_flags);
+void abort_state_change_locked(struct drbd_resource *resource);
+
+void begin_state_change_locked(struct drbd_resource *resource,
+ enum chg_state_flags flags);
+enum drbd_state_rv end_state_change_locked(struct drbd_resource *resource,
+ const char *tag);
+
+void clear_remote_state_change(struct drbd_resource *resource);
+void __clear_remote_state_change(struct drbd_resource *resource);
+
+
+enum which_state;
+bool drbd_all_peer_replication(struct drbd_device *device, enum which_state which);
+union drbd_state drbd_get_device_state(struct drbd_device *device,
+ enum which_state which);
+union drbd_state drbd_get_peer_device_state(struct drbd_peer_device *peer_device,
+ enum which_state which);
+
+/* Retry the state change until it is no longer refused as transient.
*
- * This is the most graceful way of requesting a state change. It is verbose
- * quite verbose in case the state change is not possible, and all those
- * state changes are globally serialized.
+ * A transient state is left by further state changes on the resource, and a
+ * connect handshake takes more than one two-phase commit, so there is no useful
+ * bound on how long a legitimate wait lasts. Wait for as long as the resource
+ * keeps committing state changes; give up once none was committed for two
+ * two-phase-commit timeouts, and return SS_TIMEOUT. Two, not one, so that a
+ * commit landing just after the deadline does not end the wait. The callers
+ * that matter hold resource->state_sem across this, so waiting with no bound at
+ * all also blocks this node's own connect attempts.
*/
-static inline int drbd_request_state(struct drbd_device *device,
- union drbd_state mask,
- union drbd_state val)
+#define __stable_state_change(resource, change_state, wait_timeout) ({ \
+ struct drbd_resource *__resource = (resource); \
+ enum drbd_state_rv __rv; \
+ \
+ for (;;) { \
+ long __to = 2 * twopc_timeout(__resource); \
+ unsigned int __seq = READ_ONCE(__resource->state_change_seq); \
+ long __t = wait_timeout(__resource->state_wait, \
+ (__rv = (change_state)) != SS_IN_TRANSIENT_STATE, __to); \
+ if (__t < 0) { \
+ __rv = -SS_UNKNOWN_ERROR; \
+ break; \
+ } \
+ if (__t > 0) \
+ break; \
+ if (__seq == READ_ONCE(__resource->state_change_seq)) { \
+ drbd_warn(__resource, \
+ "State change refused as transient, no state change in %ums\n", \
+ jiffies_to_msecs(__to)); \
+ __rv = SS_TIMEOUT; \
+ break; \
+ } \
+ } \
+ (int)__rv; \
+})
+
+#define stable_state_change(resource, change_state) \
+ __stable_state_change(resource, change_state, wait_event_interruptible_timeout)
+
+/* For a change made with CS_NO_SIGNALS. */
+#define stable_state_change_no_signals(resource, change_state) \
+ __stable_state_change(resource, change_state, wait_event_timeout)
+
+void nested_twopc_work(struct work_struct *work);
+void drbd_maybe_cluster_wide_reply(struct drbd_resource *resource);
+enum drbd_state_rv nested_twopc_request(struct drbd_resource *resource,
+ struct twopc_request *request);
+bool drbd_twopc_between_peer_and_me(struct drbd_connection *connection);
+bool cluster_wide_reply_ready(struct drbd_resource *resource);
+
+enum drbd_state_rv change_role(struct drbd_resource *resource,
+ enum drbd_role role,
+ enum chg_state_flags flags, const char *tag,
+ const char **err_str);
+
+void __change_io_susp_user(struct drbd_resource *resource, bool value);
+enum drbd_state_rv change_io_susp_user(struct drbd_resource *resource,
+ bool value, enum chg_state_flags flags);
+void __change_io_susp_no_data(struct drbd_resource *resource, bool value);
+void __change_io_susp_fencing(struct drbd_connection *connection, bool value);
+void __change_io_susp_quorum(struct drbd_resource *resource, bool value);
+
+void __change_disk_state(struct drbd_device *device,
+ enum drbd_disk_state disk_state);
+void __downgrade_disk_states(struct drbd_resource *resource,
+ enum drbd_disk_state disk_state);
+enum drbd_state_rv change_disk_state(struct drbd_device *device,
+ enum drbd_disk_state disk_state,
+ enum chg_state_flags flags,
+ const char *tag, const char **err_str);
+
+void __change_cstate(struct drbd_connection *connection,
+ enum drbd_conn_state cstate);
+enum drbd_state_rv change_cstate_tag(struct drbd_connection *connection,
+ enum drbd_conn_state cstate,
+ enum chg_state_flags flags,
+ const char *tag, const char **err_str);
+static inline enum drbd_state_rv change_cstate(struct drbd_connection *connection,
+ enum drbd_conn_state cstate,
+ enum chg_state_flags flags)
{
- return _drbd_request_state(device, mask, val, CS_VERBOSE + CS_ORDERED);
+ return change_cstate_tag(connection, cstate, flags, NULL, NULL);
}
-/* for use in adm_detach() (drbd_adm_detach(), drbd_adm_down()) */
-int drbd_request_detach_interruptible(struct drbd_device *device);
-
-enum drbd_role conn_highest_role(struct drbd_connection *connection);
-enum drbd_role conn_highest_peer(struct drbd_connection *connection);
-enum drbd_disk_state conn_highest_disk(struct drbd_connection *connection);
-enum drbd_disk_state conn_lowest_disk(struct drbd_connection *connection);
-enum drbd_disk_state conn_highest_pdsk(struct drbd_connection *connection);
-enum drbd_conns conn_lowest_conn(struct drbd_connection *connection);
-
+void __change_peer_role(struct drbd_connection *connection,
+ enum drbd_role peer_role);
+
+void __change_repl_state(struct drbd_peer_device *peer_device,
+ enum drbd_repl_state repl_state);
+enum drbd_state_rv change_repl_state(struct drbd_peer_device *peer_device,
+ enum drbd_repl_state new_repl_state,
+ enum chg_state_flags flags,
+ const char *tag);
+enum drbd_state_rv stable_change_repl_state(struct drbd_peer_device *peer_device,
+ enum drbd_repl_state repl_state,
+ enum chg_state_flags flags,
+ const char *tag);
+
+void __change_peer_disk_state(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state disk_state);
+void __downgrade_peer_disk_states(struct drbd_connection *connection,
+ enum drbd_disk_state disk_state);
+void __outdate_myself(struct drbd_resource *resource);
+enum drbd_state_rv change_peer_disk_state(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state disk_state,
+ enum chg_state_flags flags,
+ const char *tag);
+
+void __change_resync_susp_user(struct drbd_peer_device *peer_device,
+ bool value);
+enum drbd_state_rv change_resync_susp_user(struct drbd_peer_device *peer_device,
+ bool value,
+ enum chg_state_flags flags);
+void __change_resync_susp_peer(struct drbd_peer_device *peer_device,
+ bool value);
+void __change_resync_susp_dependency(struct drbd_peer_device *peer_device,
+ bool value);
+void __change_resync_susp_max_parallel(struct drbd_peer_device *peer_device,
+ bool value);
+void apply_connect(struct drbd_connection *connection, bool commit);
+
+struct drbd_work;
+
+bool resource_is_suspended(struct drbd_resource *resource,
+ enum which_state which);
+bool is_suspended_fen(struct drbd_resource *resource, enum which_state which);
+
+enum dds_flags;
+enum determine_dev_size;
+struct drbd_resize_parms;
+
+enum determine_dev_size
+change_cluster_wide_device_size(struct drbd_device *device,
+ sector_t local_max_size,
+ uint64_t new_user_size,
+ enum dds_flags dds_flags,
+ bool automatic,
+ struct drbd_resize_parms *rs);
+
+bool drbd_data_accessible(struct drbd_device *device, enum which_state which);
+bool drbd_res_data_accessible(struct drbd_resource *resource);
+
+
+void drbd_empty_twopc_work_fn(struct work_struct *work);
+void drbd_resume_twopc_work_fn(struct work_struct *work);
#endif
diff --git a/drivers/block/drbd/drbd_state_change.h b/drivers/block/drbd/drbd_state_change.h
index a56a57d67686..73deabf59bbd 100644
--- a/drivers/block/drbd/drbd_state_change.h
+++ b/drivers/block/drbd/drbd_state_change.h
@@ -7,58 +7,94 @@ struct drbd_resource_state_change {
enum drbd_role role[2];
bool susp[2];
bool susp_nod[2];
- bool susp_fen[2];
+ bool susp_uuid[2];
+ bool fail_io[2];
+ bool resume_held_for_outdate[2];
};
struct drbd_device_state_change {
struct drbd_device *device;
enum drbd_disk_state disk_state[2];
+ bool have_quorum[2];
};
struct drbd_connection_state_change {
struct drbd_connection *connection;
- enum drbd_conns cstate[2]; /* drbd9: enum drbd_conn_state */
+ enum drbd_conn_state cstate[2];
enum drbd_role peer_role[2];
+ bool susp_fen[2];
+};
+
+/* exception: stores state, not change.
+ * for get_initial_state. */
+struct drbd_path_state {
+ struct drbd_connection *connection;
+ struct drbd_path *path;
+ /* not an array,
+ * because it's not an array in struct drbd_path either */
+ bool path_established;
};
struct drbd_peer_device_state_change {
struct drbd_peer_device *peer_device;
enum drbd_disk_state disk_state[2];
- enum drbd_conns repl_state[2]; /* drbd9: enum drbd_repl_state */
+ enum drbd_repl_state repl_state[2];
bool resync_susp_user[2];
bool resync_susp_peer[2];
bool resync_susp_dependency[2];
+ bool resync_susp_other_c[2];
+ bool resync_susp_max_parallel[2];
+ bool resync_active[2];
+ bool replication[2];
+ bool peer_replication[2];
+};
+
+struct drbd_state_change_object_count {
+ unsigned int n_devices;
+ unsigned int n_connections;
+ unsigned int n_paths;
};
struct drbd_state_change {
struct list_head list;
unsigned int n_devices;
unsigned int n_connections;
+ unsigned int n_paths;
struct drbd_resource_state_change resource[1];
struct drbd_device_state_change *devices;
struct drbd_connection_state_change *connections;
struct drbd_peer_device_state_change *peer_devices;
+ struct drbd_path_state *paths;
};
-extern struct drbd_state_change *remember_old_state(struct drbd_resource *, gfp_t);
-extern void copy_old_to_new_state_change(struct drbd_state_change *);
-extern void forget_state_change(struct drbd_state_change *);
+struct drbd_state_change *remember_state_change(struct drbd_resource *resource,
+ gfp_t gfp);
+void copy_old_to_new_state_change(struct drbd_state_change *state_change);
+void forget_state_change(struct drbd_state_change *state_change);
+
+struct sk_buff;
+struct drbd_nl_dialect;
-extern int notify_resource_state_change(struct sk_buff *,
- unsigned int,
- void *,
- enum drbd_notification_type type);
-extern int notify_connection_state_change(struct sk_buff *,
- unsigned int,
- void *,
- enum drbd_notification_type type);
-extern int notify_device_state_change(struct sk_buff *,
- unsigned int,
- void *,
- enum drbd_notification_type type);
-extern int notify_peer_device_state_change(struct sk_buff *,
- unsigned int,
- void *,
- enum drbd_notification_type type);
+/*
+ * Turn a remembered state change into the matching notification. With a
+ * dialect given this is the initial state replay into that dialect's
+ * dump skb; without one the event is announced to all dialects.
+ */
+int notify_resource_state_change(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type);
+int notify_connection_state_change(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type);
+int notify_device_state_change(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type);
+int notify_peer_device_state_change(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type);
#endif /* DRBD_STATE_CHANGE_H */
diff --git a/drivers/block/drbd/drbd_strings.h b/drivers/block/drbd/drbd_strings.h
index a66d9f5fe4d5..92ad15d1115a 100644
--- a/drivers/block/drbd/drbd_strings.h
+++ b/drivers/block/drbd/drbd_strings.h
@@ -12,6 +12,7 @@ struct state_names {
};
extern struct state_names drbd_conn_state_names;
+extern struct state_names drbd_repl_state_names;
extern struct state_names drbd_role_state_names;
extern struct state_names drbd_disk_state_names;
extern struct state_names drbd_error_messages;
@@ -19,7 +20,8 @@ extern struct state_names drbd_packet_names;
enum drbd_packet;
-const char *drbd_conn_str(enum drbd_conns s);
+const char *drbd_repl_str(enum drbd_repl_state s);
+const char *drbd_conn_str(enum drbd_conn_state s);
const char *drbd_role_str(enum drbd_role s);
const char *drbd_disk_str(enum drbd_disk_state s);
const char *drbd_set_st_err_str(enum drbd_state_rv err);
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 11/20] drbd: rewrite state machine for DRBD 9 multi-peer clusters
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (9 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 10/20] drbd: replace per-device state model with multi-peer data structures Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 12/20] drbd: rework activity log and bitmap for multi-peer replication Christoph Böhmwalder
` (8 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Andreas Gruenbacher,
Joel Colledge
Replace the monolithic DRBD 8.4 state machine with an architecture
suited for clusters with more than one peer.
The central concept is a transactional model: state is held in
per-object arrays indexed by [NOW] and [NEW], and every change is
bracketed by begin/end calls that validate the proposed transition
resource-wide before atomically committing it or rolling it back.
Replace the single spinlock that serialized everything with
finer-grained locking: a read-write lock for state access, separate
locks for peer requests and interval trees.
Cluster-wide state changes (role changes, connect/disconnect, resize)
use a two-phase commit protocol.
The initiating node sends a prepare message to all reachable peers,
collects replies with timeout and exponential backoff, then commits
or aborts.
Not-fully-connected topologies are handled by forwarding nested 2PC
rounds through intermediate nodes.
Add a quorum mechanism with tiebreaker support for even-sized clusters.
This can suspend or fail I/O when the cluster loses more than half of
its peers.
Unify post-state-change processing into a single resource-wide work
item that handles UUID propagation, resync startup, I/O suspension,
metadata persistence, and netlink notifications for all objects in one
pass, replacing the separate per-device and per-connection callbacks
from 8.4.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_state.c | 9203 ++++++++++++++++++++++++-------
1 file changed, 7298 insertions(+), 1905 deletions(-)
diff --git a/drivers/block/drbd/drbd_state.c b/drivers/block/drbd/drbd_state.c
index f1a5d899fd54..03cc0723973b 100644
--- a/drivers/block/drbd/drbd_state.c
+++ b/drivers/block/drbd/drbd_state.c
@@ -6,2385 +6,7778 @@
* Copyright (C) 2008, LINBIT HA-Solutions GmbH.
*/
-#include <linux/drbd_limits.h>
+#include <linux/random.h>
+#include <linux/jiffies.h>
#include "drbd_int.h"
#include "drbd_protocol.h"
#include "drbd_req.h"
#include "drbd_state_change.h"
-struct after_state_chg_work {
+
+struct after_state_change_work {
struct drbd_work w;
- struct drbd_device *device;
- union drbd_state os;
- union drbd_state ns;
- enum chg_state_flags flags;
- struct completion *done;
struct drbd_state_change *state_change;
+ struct completion *done;
};
-enum sanitize_state_warnings {
- NO_WARNING,
- ABORTED_ONLINE_VERIFY,
- ABORTED_RESYNC,
- CONNECTION_LOST_NEGOTIATING,
- IMPLICITLY_UPGRADED_DISK,
- IMPLICITLY_UPGRADED_PDSK,
+struct quorum_info {
+ int up_to_date;
+ int present;
+ int voters;
+ int quorum_at;
+ int min_redundancy_at;
};
-static void count_objects(struct drbd_resource *resource,
- unsigned int *n_devices,
- unsigned int *n_connections)
-{
- struct drbd_device *device;
- struct drbd_connection *connection;
- int vnr;
-
- *n_devices = 0;
- *n_connections = 0;
-
- idr_for_each_entry(&resource->devices, device, vnr)
- (*n_devices)++;
- for_each_connection(connection, resource)
- (*n_connections)++;
-}
-
-static struct drbd_state_change *alloc_state_change(unsigned int n_devices, unsigned int n_connections, gfp_t gfp)
-{
- struct drbd_state_change *state_change;
- unsigned int size, n;
-
- size = sizeof(struct drbd_state_change) +
- n_devices * sizeof(struct drbd_device_state_change) +
- n_connections * sizeof(struct drbd_connection_state_change) +
- n_devices * n_connections * sizeof(struct drbd_peer_device_state_change);
- state_change = kmalloc(size, gfp);
- if (!state_change)
- return NULL;
- state_change->n_devices = n_devices;
- state_change->n_connections = n_connections;
- state_change->devices = (void *)(state_change + 1);
- state_change->connections = (void *)&state_change->devices[n_devices];
- state_change->peer_devices = (void *)&state_change->connections[n_connections];
- state_change->resource->resource = NULL;
- for (n = 0; n < n_devices; n++)
- state_change->devices[n].device = NULL;
- for (n = 0; n < n_connections; n++)
- state_change->connections[n].connection = NULL;
- return state_change;
-}
+struct quorum_detail {
+ int up_to_date;
+ int present;
+ int outdated;
+ int diskless;
+ int missing_diskless;
+ int quorumless;
+ int unknown;
+ int quorate_peers;
+};
-struct drbd_state_change *remember_old_state(struct drbd_resource *resource, gfp_t gfp)
-{
- struct drbd_state_change *state_change;
- struct drbd_device *device;
- unsigned int n_devices;
- struct drbd_connection *connection;
- unsigned int n_connections;
+struct change_context {
+ struct drbd_resource *resource;
int vnr;
+ union drbd_state mask;
+ union drbd_state val;
+ int target_node_id;
+ enum chg_state_flags flags;
+ bool change_local_state_last;
+ const char **err_str;
+};
- struct drbd_device_state_change *device_state_change;
- struct drbd_peer_device_state_change *peer_device_state_change;
- struct drbd_connection_state_change *connection_state_change;
-
- /* Caller holds req_lock spinlock.
- * No state, no device IDR, no connections lists can change. */
- count_objects(resource, &n_devices, &n_connections);
- state_change = alloc_state_change(n_devices, n_connections, gfp);
- if (!state_change)
- return NULL;
-
- kref_get(&resource->kref);
- state_change->resource->resource = resource;
- state_change->resource->role[OLD] =
- conn_highest_role(first_connection(resource));
- state_change->resource->susp[OLD] = resource->susp;
- state_change->resource->susp_nod[OLD] = resource->susp_nod;
- state_change->resource->susp_fen[OLD] = resource->susp_fen;
-
- connection_state_change = state_change->connections;
- for_each_connection(connection, resource) {
- kref_get(&connection->kref);
- connection_state_change->connection = connection;
- connection_state_change->cstate[OLD] =
- connection->cstate;
- connection_state_change->peer_role[OLD] =
- conn_highest_peer(connection);
- connection_state_change++;
- }
-
- device_state_change = state_change->devices;
- peer_device_state_change = state_change->peer_devices;
- idr_for_each_entry(&resource->devices, device, vnr) {
- kref_get(&device->kref);
- device_state_change->device = device;
- device_state_change->disk_state[OLD] = device->state.disk;
+enum change_phase {
+ PH_LOCAL_COMMIT,
+ PH_PREPARE,
+ PH_84_COMMIT,
+ PH_COMMIT,
+};
- /* The peer_devices for each device have to be enumerated in
- the order of the connections. We may not use for_each_peer_device() here. */
- for_each_connection(connection, resource) {
- struct drbd_peer_device *peer_device;
+struct change_disk_state_context {
+ struct change_context context;
+ struct drbd_device *device;
+};
- peer_device = conn_peer_device(connection, device->vnr);
- peer_device_state_change->peer_device = peer_device;
- peer_device_state_change->disk_state[OLD] =
- device->state.pdsk;
- peer_device_state_change->repl_state[OLD] =
- max_t(enum drbd_conns,
- C_WF_REPORT_PARAMS, device->state.conn);
- peer_device_state_change->resync_susp_user[OLD] =
- device->state.user_isp;
- peer_device_state_change->resync_susp_peer[OLD] =
- device->state.peer_isp;
- peer_device_state_change->resync_susp_dependency[OLD] =
- device->state.aftr_isp;
- peer_device_state_change++;
- }
- device_state_change++;
- }
+static bool lost_contact_to_peer_data(enum drbd_disk_state *peer_disk_state);
+static bool peer_returns_diskless(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state os, enum drbd_disk_state ns);
+static void print_state_change(struct drbd_resource *resource, const char *prefix, const char *tag);
+static void finish_state_change(struct drbd_resource *, const char *tag);
+static int w_after_state_change(struct drbd_work *w, int unused);
+static enum drbd_state_rv is_valid_soft_transition(struct drbd_resource *);
+static enum drbd_state_rv is_valid_transition(struct drbd_resource *resource);
+static void sanitize_state(struct drbd_resource *resource);
+static void ensure_exposed_data_uuid(struct drbd_device *device);
+static enum drbd_state_rv change_peer_state(struct drbd_connection *, int, union drbd_state,
+ union drbd_state, unsigned long *);
+static void check_wrongly_set_mdf_exists(struct drbd_device *);
+static void update_members(struct drbd_resource *resource);
+static bool calc_data_accessible(struct drbd_state_change *state_change, int n_device,
+ enum which_state which);
+
+static const char * const gen_obl_state_names[] = {
+ [GEN_OBL_NONE] = "NONE",
+ [GEN_OBL_ARMED] = "ARMED",
+ [GEN_OBL_MINTING] = "MINTING",
+ [GEN_OBL_UNCONFIRMED] = "UNCONFIRMED",
+ [GEN_OBL_DISCHARGED] = "DISCHARGED",
+ [GEN_OBL_PARKED] = "PARKED",
+};
- return state_change;
-}
+static const char * const gen_obl_reason_names[GEN_OBL_REASON_COUNT] = {
+ "peer-data-lost",
+ "peer-returned-diskless",
+ "peer-disk-failed",
+ "own-disk-failed",
+ "promoted",
+ "ahead",
+ "pre-110",
+ "completion-decided",
+};
-static void remember_new_state(struct drbd_state_change *state_change)
+/* Render an obligation word as "ARMED(peer-data-lost,promoted)". The reason
+ * set of a met obligation lingers in the word; it describes an obligation, so
+ * show it only while there is one.
+ */
+void drbd_gen_obligation_str(u32 obligation, char *buf, size_t size)
{
- struct drbd_resource_state_change *resource_state_change;
- struct drbd_resource *resource;
- unsigned int n;
-
- if (!state_change)
+ enum drbd_gen_obl_state state = obligation & GEN_OBL_STATE_MASK;
+ u16 reasons = obligation >> GEN_OBL_REASON_SHIFT;
+ const char *sep = "(";
+ size_t len = 0;
+ int i;
+
+ len += scnprintf(buf + len, size - len, "%s", gen_obl_state_names[state]);
+ if (state == GEN_OBL_NONE)
return;
- resource_state_change = &state_change->resource[0];
- resource = resource_state_change->resource;
-
- resource_state_change->role[NEW] =
- conn_highest_role(first_connection(resource));
- resource_state_change->susp[NEW] = resource->susp;
- resource_state_change->susp_nod[NEW] = resource->susp_nod;
- resource_state_change->susp_fen[NEW] = resource->susp_fen;
-
- for (n = 0; n < state_change->n_devices; n++) {
- struct drbd_device_state_change *device_state_change =
- &state_change->devices[n];
- struct drbd_device *device = device_state_change->device;
-
- device_state_change->disk_state[NEW] = device->state.disk;
+ for (i = 0; i < GEN_OBL_REASON_COUNT; i++) {
+ if (!(reasons & (1 << i)))
+ continue;
+ len += scnprintf(buf + len, size - len, "%s%s", sep, gen_obl_reason_names[i]);
+ sep = ",";
}
-
- for (n = 0; n < state_change->n_connections; n++) {
- struct drbd_connection_state_change *connection_state_change =
- &state_change->connections[n];
- struct drbd_connection *connection =
- connection_state_change->connection;
-
- connection_state_change->cstate[NEW] = connection->cstate;
- connection_state_change->peer_role[NEW] =
- conn_highest_peer(connection);
+ if (obligation & GEN_OBL_MATERIALIZED) {
+ len += scnprintf(buf + len, size - len, "%smaterialized", sep);
+ sep = ",";
}
-
- for (n = 0; n < state_change->n_devices * state_change->n_connections; n++) {
- struct drbd_peer_device_state_change *peer_device_state_change =
- &state_change->peer_devices[n];
- struct drbd_device *device =
- peer_device_state_change->peer_device->device;
- union drbd_dev_state state = device->state;
-
- peer_device_state_change->disk_state[NEW] = state.pdsk;
- peer_device_state_change->repl_state[NEW] =
- max_t(enum drbd_conns, C_WF_REPORT_PARAMS, state.conn);
- peer_device_state_change->resync_susp_user[NEW] =
- state.user_isp;
- peer_device_state_change->resync_susp_peer[NEW] =
- state.peer_isp;
- peer_device_state_change->resync_susp_dependency[NEW] =
- state.aftr_isp;
+ if (obligation & GEN_OBL_REARM_PENDING) {
+ len += scnprintf(buf + len, size - len, "%srearm-pending", sep);
+ sep = ",";
}
+ if (sep[0] == ',')
+ scnprintf(buf + len, size - len, ")");
}
-void copy_old_to_new_state_change(struct drbd_state_change *state_change)
+/* The single place that changes device->gen_obligation.
+ *
+ * Move to state @to if the current state is one of @from_states, and merge
+ * @reasons into the reason set. Reasons given while there is no obligation
+ * replace the lingering set of the previous one; an empty @reasons keeps what
+ * is recorded, which is what re-arming an obligation whose mint could not run
+ * wants. @aux are auxiliary bits to set, and only apply when the state moves.
+ * Returns whether the state moved.
+ *
+ * A divergence-start event that arrives while the mint of an earlier obligation
+ * runs, or while its generation waits to be confirmed, is a second obligation:
+ * the running generation cannot cover it, because it may already be exposed to
+ * the peers. Record it as GEN_OBL_REARM_PENDING and let the discharge of the
+ * running one land in ARMED instead, so it is minted next. Such an arm is the
+ * caller whose @from_states include NONE.
+ *
+ * GEN_OBL_MATERIALIZED describes one obligation, the way the reason set does,
+ * and is cleared where that set is: when a fresh obligation arms out of a met
+ * state, and on the re-arm path above, which arms a fresh one without passing
+ * through a met state. It survives everywhere else, so that an obligation
+ * whose mint failed keeps it. Unlike a reason bit it is decisive -- it makes
+ * the mint mandatory -- so it must never be inherited by the next obligation.
+ *
+ * It is also the one @aux bit that applies without the state moving, and only
+ * out of MINTING: a completion decision taken while the mint executor runs is
+ * a fact about that obligation, and the executor's own exit is what reads the
+ * attribute. Moving the state there instead would release the obligation
+ * under a running executor, whose exit then no longer matches -- which is why
+ * MINTING is not in GEN_OBL_MATERIALIZE_FROM.
+ *
+ * Data generations start rarely, so log every change at info level.
+ */
+bool drbd_gen_obligation_transition(struct drbd_device *device, unsigned int from_states,
+ enum drbd_gen_obl_state to, u16 reasons, u32 aux)
{
- struct drbd_resource_state_change *resource_state_change = &state_change->resource[0];
- unsigned int n_device, n_connection, n_peer_device, n_peer_devices;
-
-#define OLD_TO_NEW(x) \
- (x[NEW] = x[OLD])
-
- OLD_TO_NEW(resource_state_change->role);
- OLD_TO_NEW(resource_state_change->susp);
- OLD_TO_NEW(resource_state_change->susp_nod);
- OLD_TO_NEW(resource_state_change->susp_fen);
-
- for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
- struct drbd_connection_state_change *connection_state_change =
- &state_change->connections[n_connection];
-
- OLD_TO_NEW(connection_state_change->peer_role);
- OLD_TO_NEW(connection_state_change->cstate);
+ char from_str[GEN_OBL_STR_MAX], to_str[GEN_OBL_STR_MAX];
+ enum drbd_gen_obl_state state;
+ unsigned long irq_flags;
+ u32 old, new;
+ bool moved;
+
+ spin_lock_irqsave(&device->gen_obligation_lock, irq_flags);
+ old = device->gen_obligation;
+ state = old & GEN_OBL_STATE_MASK;
+ moved = GEN_OBL_IN(state) & from_states;
+
+ new = old;
+ if (reasons) {
+ if (state == GEN_OBL_NONE || state == GEN_OBL_DISCHARGED)
+ new &= ~(GEN_OBL_REASON_MASK | GEN_OBL_MATERIALIZED);
+ new |= (u32)reasons << GEN_OBL_REASON_SHIFT;
+ }
+ if (!moved && (from_states & GEN_OBL_IN(GEN_OBL_NONE)) &&
+ (state == GEN_OBL_MINTING || state == GEN_OBL_UNCONFIRMED))
+ new |= GEN_OBL_REARM_PENDING;
+ if ((aux & GEN_OBL_MATERIALIZED) && state == GEN_OBL_MINTING)
+ new |= GEN_OBL_MATERIALIZED;
+ if (moved && to == GEN_OBL_DISCHARGED && (new & GEN_OBL_REARM_PENDING)) {
+ new &= ~(GEN_OBL_REARM_PENDING | GEN_OBL_MATERIALIZED);
+ to = GEN_OBL_ARMED;
+ }
+ if (moved)
+ new = (new & ~GEN_OBL_STATE_MASK) | to | aux;
+ if (new != old) {
+ drbd_gen_obligation_str(old, from_str, sizeof(from_str));
+ drbd_gen_obligation_str(new, to_str, sizeof(to_str));
+ WRITE_ONCE(device->gen_obligation, new);
}
+ spin_unlock_irqrestore(&device->gen_obligation_lock, irq_flags);
- for (n_device = 0; n_device < state_change->n_devices; n_device++) {
- struct drbd_device_state_change *device_state_change =
- &state_change->devices[n_device];
+ if (new != old)
+ drbd_info(device, "gen-obligation: %s -> %s\n", from_str, to_str);
- OLD_TO_NEW(device_state_change->disk_state);
- }
+ return moved;
+}
- n_peer_devices = state_change->n_devices * state_change->n_connections;
- for (n_peer_device = 0; n_peer_device < n_peer_devices; n_peer_device++) {
- struct drbd_peer_device_state_change *p =
- &state_change->peer_devices[n_peer_device];
+/* A divergence-start event obliges this volume to start a new data
+ * generation before it admits further writes. Arming while an earlier
+ * obligation is being minted or waits for its confirmation records
+ * GEN_OBL_REARM_PENDING, see drbd_gen_obligation_transition().
+ *
+ * PARKED is deliberately not in the from-set: while writers fail fast there is
+ * nothing to separate from anything, so a further event only merges its reason
+ * and the volume stays parked until write admission returns.
+ */
+void drbd_gen_obligation_arm(struct drbd_device *device, u16 reasons)
+{
+ drbd_gen_obligation_transition(device,
+ GEN_OBL_IN(GEN_OBL_NONE) | GEN_OBL_IN(GEN_OBL_ARMED) |
+ GEN_OBL_IN(GEN_OBL_DISCHARGED),
+ GEN_OBL_ARMED, reasons, 0);
+}
- OLD_TO_NEW(p->disk_state);
- OLD_TO_NEW(p->repl_state);
- OLD_TO_NEW(p->resync_susp_user);
- OLD_TO_NEW(p->resync_susp_peer);
- OLD_TO_NEW(p->resync_susp_dependency);
- }
+/* Where an obligation lands that is kept rather than met: parked while writers
+ * fail fast (io-error policy), because the data set does not change and no
+ * generation is owed for that span; armed otherwise. A materialized
+ * obligation is about writes that already completed to the application, and no
+ * policy on later writes undoes those, so it stays armed.
+ */
+static enum drbd_gen_obl_state gen_obligation_retained_state(struct drbd_device *device)
+{
+ if (device->cached_err_io && !drbd_gen_obligation_materialized(device))
+ return GEN_OBL_PARKED;
-#undef OLD_TO_NEW
+ return GEN_OBL_ARMED;
}
-void forget_state_change(struct drbd_state_change *state_change)
+/* The io-error policy took effect for this volume: park the obligation. */
+static void gen_obligation_park(struct drbd_device *device)
{
- unsigned int n;
+ drbd_gen_obligation_transition(device, GEN_OBL_IN(GEN_OBL_ARMED),
+ gen_obligation_retained_state(device), 0, 0);
+}
- if (!state_change)
- return;
+/* Write admission returned, so the data set can change again and the parked
+ * obligation is owed. Dispatch its mint here rather than leave it to the
+ * first write: the generation separates the writes to come from every peer
+ * that may be missing some.
+ */
+static void gen_obligation_unpark(struct drbd_device *device)
+{
+ if (drbd_gen_obligation_transition(device, GEN_OBL_IN(GEN_OBL_PARKED),
+ GEN_OBL_ARMED, 0, 0) &&
+ drbd_gen_obligation_mint_start(device))
+ drbd_device_post_work(device, MAKE_NEW_CUR_UUID);
+}
- if (state_change->resource->resource)
- kref_put(&state_change->resource->resource->kref, drbd_destroy_resource);
- for (n = 0; n < state_change->n_devices; n++) {
- struct drbd_device *device = state_change->devices[n].device;
+/* This node decides to complete writes to the application although acks of a
+ * replica it lost are missing. The divergence is a fact
+ * from that decision on: the obligation is not voidable any more, its mint is
+ * mandatory, and no io-error policy on future writes drops it. A parked
+ * obligation materializes as well and leaves the park: the policy governs the
+ * writes to come, not the ones that already completed. One whose mint is
+ * running takes the attribute where it stands, see
+ * drbd_gen_obligation_transition(). Returns whether the state moved.
+ */
+bool drbd_gen_obligation_materialize(struct drbd_device *device)
+{
+ return drbd_gen_obligation_transition(device, GEN_OBL_MATERIALIZE_FROM,
+ GEN_OBL_ARMED, GEN_OBL_COMPLETION_DECIDED,
+ GEN_OBL_MATERIALIZED);
+}
- if (device)
- kref_put(&device->kref, drbd_destroy_device);
- }
- for (n = 0; n < state_change->n_connections; n++) {
- struct drbd_connection *connection =
- state_change->connections[n].connection;
+/* Void the obligation: the caller must hold the proof that no generation is
+ * owed. Refused while the mint executor runs -- it owns the obligation until
+ * it reports the outcome. A parked obligation is latent by construction, so
+ * a proof about a latent one covers it as well; without this its callers
+ * would leave it behind to be minted at the unpark, for writes that never
+ * happened.
+ */
+bool drbd_gen_obligation_void(struct drbd_device *device)
+{
+ return drbd_gen_obligation_transition(device,
+ GEN_OBL_IN(GEN_OBL_ARMED) |
+ GEN_OBL_IN(GEN_OBL_PARKED),
+ GEN_OBL_NONE, 0, 0);
+}
- if (connection)
- kref_put(&connection->kref, drbd_destroy_connection);
- }
- kfree(state_change);
+/* This volume adopts the peer's current UUID. That adopted current is the new
+ * data generation, and the peer it comes from confirms it, so the obligation is
+ * met rather than dropped. Allowed out of MINTING as well: an executor racing
+ * the adoption finds the state moved on and its own exit no longer matches, so
+ * its outcome changes nothing. A divergence-start event recorded during that
+ * mint re-arms here, which is right -- the adoption confirms the generation
+ * being minted, not the later event.
+ */
+bool drbd_gen_obligation_discharge_by_adoption(struct drbd_device *device)
+{
+ return drbd_gen_obligation_transition(device,
+ GEN_OBL_IN(GEN_OBL_ARMED) |
+ GEN_OBL_IN(GEN_OBL_MINTING),
+ GEN_OBL_DISCHARGED, 0, 0);
}
-static int w_after_state_ch(struct drbd_work *w, int unused);
-static void after_state_ch(struct drbd_device *device, union drbd_state os,
- union drbd_state ns, enum chg_state_flags flags,
- struct drbd_state_change *);
-static enum drbd_state_rv is_valid_state(struct drbd_device *, union drbd_state);
-static enum drbd_state_rv is_valid_soft_transition(union drbd_state, union drbd_state, struct drbd_connection *);
-static enum drbd_state_rv is_valid_transition(union drbd_state os, union drbd_state ns);
-static union drbd_state sanitize_state(struct drbd_device *device, union drbd_state os,
- union drbd_state ns, enum sanitize_state_warnings *warn);
+/* Dispatch guard of the mint executor: only one caller enters MINTING, and
+ * writes stay blocked while it runs.
+ */
+bool drbd_gen_obligation_mint_start(struct drbd_device *device)
+{
+ return drbd_gen_obligation_transition(device, GEN_OBL_IN(GEN_OBL_ARMED),
+ GEN_OBL_MINTING, 0, 0);
+}
-static inline bool is_susp(union drbd_state s)
+/* The single exit of the mint executor. A new generation, or the proof that
+ * none is owed, meets the obligation; anything else leaves it outstanding, so
+ * that the next write, or a later trigger, tries again -- parked while writers
+ * fail fast, see gen_obligation_retained_state().
+ *
+ * MINT_EXPOSED is the exception that does not move: the mint left the state at
+ * UNCONFIRMED, where it stays until a peer confirms the generation. Writes are
+ * admitted meanwhile, with their completion held.
+ */
+void drbd_gen_obligation_mint_done(struct drbd_device *device, enum drbd_mint_outcome outcome)
{
- return s.susp || s.susp_nod || s.susp_fen;
+ enum drbd_gen_obl_state to;
+
+ if (!drbd_mint_still_owed(outcome))
+ to = GEN_OBL_DISCHARGED;
+ else
+ to = gen_obligation_retained_state(device);
+
+ drbd_gen_obligation_transition(device, GEN_OBL_IN(GEN_OBL_MINTING), to, 0, 0);
}
-bool conn_all_vols_unconf(struct drbd_connection *connection)
+/* True if the transfer log still holds a write of this volume towards a peer
+ * that is about to be given up: one of only_nodes, or -- with no node named --
+ * any peer whose connection is not established. Lifting the last IO suspension
+ * runs CANCEL_SUSPENDED_IO and then COMPLETION_RESUMED over exactly these
+ * requests, so the test uses the guard of the CANCEL_SUSPENDED_IO arm itself
+ * and cannot drift from what that walk finalizes.
+ */
+static bool writes_held_towards_lost_peers(struct drbd_device *device, u64 only_nodes)
{
struct drbd_peer_device *peer_device;
- bool rv = true;
- int vnr;
+ struct drbd_request *req;
+ u64 lost_nodes = only_nodes;
+ bool found = false;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- if (device->state.disk != D_DISKLESS ||
- device->state.conn != C_STANDALONE ||
- device->state.role != R_SECONDARY) {
- rv = false;
+ if (!lost_nodes) {
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->connection->cstate[NOW] < C_CONNECTED)
+ lost_nodes |= NODE_MASK(peer_device->node_id);
+ }
+ }
+
+ list_for_each_entry_rcu(req, &device->resource->transfer_log, tl_requests) {
+ int node_id;
+
+ if (req->device != device)
+ continue;
+ if (!(req->local_rq_state & RQ_WRITE))
+ continue;
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ unsigned long s;
+
+ if (!(lost_nodes & NODE_MASK(node_id)))
+ continue;
+ s = req->net_rq_state[node_id];
+ if (!(s & RQ_NET_MASK) || s & RQ_NET_DONE)
+ continue;
+ found = true;
break;
}
+ if (found)
+ break;
}
rcu_read_unlock();
-
- return rv;
+ return found;
}
-/* Unfortunately the states where not correctly ordered, when
- they where defined. therefore can not use max_t() here. */
-static enum drbd_role max_role(enum drbd_role role1, enum drbd_role role2)
+/* Called at a resume exit, before the state change that can lift the last IO
+ * suspension. If that state change leaves peers behind, finish_state_change()
+ * gives up the writes still held towards them (CANCEL_SUSPENDED_IO) and then
+ * completes them to the application (COMPLETION_RESUMED). That is a
+ * completion decision on behalf of a replica which never saw the data, so the
+ * data generation labelling it must exist before those completions: with such
+ * writes in the log, materialize the obligation and mint here, synchronously
+ * and ungated -- the decision carries the authority the mint would otherwise
+ * ask for.
+ *
+ * With none, the mint is opportunistic and goes to the worker. Where the
+ * caller cannot tell whether its state change is the one lifting the last
+ * suspension, minting anyway is correct: those writes are stranded towards a
+ * peer that is gone either way.
+ */
+void drbd_gen_obligation_mint_before_resume(struct drbd_device *device, u64 only_nodes)
{
- if (role1 == R_PRIMARY || role2 == R_PRIMARY)
- return R_PRIMARY;
- if (role1 == R_SECONDARY || role2 == R_SECONDARY)
- return R_SECONDARY;
- return R_UNKNOWN;
-}
+ if (drbd_gen_obligation_state(device) != GEN_OBL_ARMED)
+ return;
-static enum drbd_role min_role(enum drbd_role role1, enum drbd_role role2)
-{
- if (role1 == R_UNKNOWN || role2 == R_UNKNOWN)
- return R_UNKNOWN;
- if (role1 == R_SECONDARY || role2 == R_SECONDARY)
- return R_SECONDARY;
- return R_PRIMARY;
+ if (writes_held_towards_lost_peers(device, only_nodes)) {
+ drbd_gen_obligation_materialize(device);
+ if (drbd_gen_obligation_mint_start(device)) {
+ drbd_gen_obligation_mint_done(device,
+ drbd_uuid_new_current(device, false));
+ wake_up(&device->misc_wait);
+ }
+ } else if (drbd_gen_obligation_mint_start(device)) {
+ drbd_device_post_work(device, MAKE_NEW_CUR_UUID);
+ }
}
-enum drbd_role conn_highest_role(struct drbd_connection *connection)
+/* A D_CONSISTENT survivor owes a post-loss reconcile before regaining
+ * D_UP_TO_DATE, gap-free across
+ * NOTIFY_PEERS_LOST_PRIMARY -> RECONCILE_PENDING -> RECONCILIATION_RESYNC.
+ * Upgrade-only: outdating (< D_CONSISTENT) stays allowed, so a survivor of an
+ * unreachable primary still outdates via the lost-peer twopc.
+ */
+static bool reconcile_hold_active(struct drbd_device *device)
{
- enum drbd_role role = R_SECONDARY;
struct drbd_peer_device *peer_device;
- int vnr;
+ bool rv = false;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- role = max_role(role, device->state.role);
+ for_each_peer_device_rcu(peer_device, device) {
+ if (test_bit(NOTIFY_PEERS_LOST_PRIMARY, &peer_device->connection->flags) ||
+ test_bit(RECONCILE_PENDING, peer_device->flags) ||
+ test_bit(RECONCILIATION_RESYNC, peer_device->flags)) {
+ rv = true;
+ break;
+ }
}
rcu_read_unlock();
- return role;
+ return rv;
}
-enum drbd_role conn_highest_peer(struct drbd_connection *connection)
+/* We need to stay consistent if we are neighbor of a diskless primary with
+ * different UUID. This function should be used if the device was D_UP_TO_DATE
+ * before.
+ */
+static bool may_return_to_up_to_date(struct drbd_device *device, enum which_state which)
{
- enum drbd_role peer = R_UNKNOWN;
struct drbd_peer_device *peer_device;
- int vnr;
+ bool rv = true;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- peer = max_role(peer, device->state.peer);
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->disk_state[which] == D_DISKLESS &&
+ peer_device->connection->peer_role[which] == R_PRIMARY &&
+ peer_device->current_uuid != drbd_current_uuid(device)) {
+ rv = false;
+ break;
+ }
}
rcu_read_unlock();
- return peer;
+ return rv;
}
-enum drbd_disk_state conn_highest_disk(struct drbd_connection *connection)
+/* Losing a primary whose own disk was at best D_OUTDATED cannot leave this node
+ * behind data that survives nowhere else -- such a primary holds no
+ * authoritative copy of its own. This node only owes a post-loss reconcile if
+ * some *surviving* diskful peer might hold a write it missed. If this node is
+ * the sync source for every surviving diskful peer, it is the sole
+ * authoritative copy and must stay D_UP_TO_DATE instead of dropping to
+ * D_CONSISTENT to await a reconcile that no one owes it.
+ */
+static bool sole_authority_after_incapable_primary_loss(struct drbd_device *device)
{
- enum drbd_disk_state disk_state = D_DISKLESS;
struct drbd_peer_device *peer_device;
- int vnr;
+ bool lost_incapable_primary = false;
+ bool peer_might_be_ahead = false;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- disk_state = max_t(enum drbd_disk_state, disk_state, device->state.disk);
+ for_each_peer_device_rcu(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
+
+ if (connection->peer_role[OLD] == R_PRIMARY &&
+ connection->cstate[NEW] < C_CONNECTED) {
+ if (peer_device->disk_state[OLD] <= D_OUTDATED)
+ lost_incapable_primary = true;
+ else
+ peer_might_be_ahead = true;
+
+ /* A surviving diskful peer that we are not syncing to could hold
+ * a write we are missing (an outdated/inconsistent peer cannot).
+ */
+ } else if (peer_device->disk_state[NEW] >= D_CONSISTENT)
+ peer_might_be_ahead = true;
}
rcu_read_unlock();
- return disk_state;
+ return lost_incapable_primary && !peer_might_be_ahead;
}
-enum drbd_disk_state conn_lowest_disk(struct drbd_connection *connection)
+/**
+ * may_be_up_to_date() - check if transition from D_CONSISTENT to D_UP_TO_DATE is allowed
+ * @device: DRBD device.
+ * @which: OLD or NEW
+ *
+ * When fencing is enabled, it may only transition from D_CONSISTENT to D_UP_TO_DATE
+ * when ether all peers are connected, or outdated.
+ */
+static bool may_be_up_to_date(struct drbd_device *device, enum which_state which)
{
- enum drbd_disk_state disk_state = D_MASK;
- struct drbd_peer_device *peer_device;
- int vnr;
+ bool all_peers_outdated = true;
+ int node_id;
+
+ if (!may_return_to_up_to_date(device, which) || reconcile_hold_active(device))
+ return false;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- disk_state = min_t(enum drbd_disk_state, disk_state, device->state.disk);
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
+ struct drbd_peer_device *peer_device;
+ enum drbd_disk_state peer_disk_state;
+ bool want_bitmap = true;
+
+ if (node_id == device->ldev->md.node_id)
+ continue;
+
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md->flags) &&
+ !test_bit(__MDF_NODE_EXISTS, &peer_md->flags))
+ continue;
+
+ if (!test_bit(__MDF_PEER_FENCING, &peer_md->flags))
+ continue;
+ peer_device = peer_device_by_node_id(device, node_id);
+ if (peer_device) {
+ struct drbd_peer_device_conf *pdc = rcu_dereference(peer_device->conf);
+
+ want_bitmap = pdc->bitmap;
+ peer_disk_state = peer_device->disk_state[NEW];
+ } else {
+ peer_disk_state = D_UNKNOWN;
+ }
+
+ switch (peer_disk_state) {
+ case D_DISKLESS:
+ if (!test_bit(__MDF_PEER_DEVICE_SEEN, &peer_md->flags))
+ continue;
+ fallthrough;
+ case D_ATTACHING:
+ case D_DETACHING:
+ case D_FAILED:
+ case D_NEGOTIATING:
+ case D_UNKNOWN:
+ if (!want_bitmap)
+ continue;
+ if (test_bit(__MDF_PEER_OUTDATED, &peer_md->flags))
+ continue;
+ break;
+ case D_INCONSISTENT:
+ case D_OUTDATED:
+ continue;
+ case D_CONSISTENT:
+ case D_UP_TO_DATE:
+ /* These states imply that there is a connection. If there is
+ a connection we do not need to insist that the peer was
+ outdated. */
+ continue;
+ case D_MASK:
+ break;
+ }
+
+ all_peers_outdated = false;
}
rcu_read_unlock();
-
- return disk_state;
+ return all_peers_outdated;
}
-enum drbd_disk_state conn_highest_pdsk(struct drbd_connection *connection)
+static bool stable_up_to_date_neighbor(struct drbd_device *device)
{
- enum drbd_disk_state disk_state = D_DISKLESS;
struct drbd_peer_device *peer_device;
- int vnr;
+ bool rv = false;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- disk_state = max_t(enum drbd_disk_state, disk_state, device->state.pdsk);
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->disk_state[NEW] == D_UP_TO_DATE &&
+ peer_device->uuid_flags & UUID_FLAG_STABLE && /* primary is also stable */
+ peer_device->current_uuid == drbd_current_uuid(device)) {
+ rv = true;
+ break;
+ }
}
rcu_read_unlock();
- return disk_state;
+ return rv;
}
-enum drbd_conns conn_lowest_conn(struct drbd_connection *connection)
-{
- enum drbd_conns conn = C_MASK;
- struct drbd_peer_device *peer_device;
- int vnr;
+/**
+ * disk_state_from_md() - determine initial disk state
+ * @device: DRBD device.
+ *
+ * When a disk is attached to a device, we set the disk state to D_NEGOTIATING.
+ * We then wait for all connected peers to send the peer disk state. Once that
+ * has happened, we can determine the actual disk state based on the peer disk
+ * states and the state of the disk itself.
+ *
+ * The initial disk state becomes D_UP_TO_DATE without fencing or when we know
+ * that all peers have been outdated, and D_CONSISTENT otherwise.
+ *
+ * The caller either needs to have a get_ldev() reference, or need to call
+ * this function only if disk_state[NOW] >= D_NEGOTIATING and holding the
+ * state_rwlock.
+ */
+enum drbd_disk_state disk_state_from_md(struct drbd_device *device)
+{
+ enum drbd_disk_state disk_state;
+
+ if (!drbd_md_test_flag(device->ldev, MDF_CONSISTENT))
+ disk_state = D_INCONSISTENT;
+ else if (!drbd_md_test_flag(device->ldev, MDF_WAS_UP_TO_DATE))
+ disk_state = D_OUTDATED;
+ else
+ disk_state = may_be_up_to_date(device, NOW) ? D_UP_TO_DATE : D_CONSISTENT;
+
+ return disk_state;
+}
+
+bool is_suspended_fen(struct drbd_resource *resource, enum which_state which)
+{
+ struct drbd_connection *connection;
+ bool rv = false;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- conn = min_t(enum drbd_conns, conn, device->state.conn);
+ for_each_connection_rcu(connection, resource) {
+ if (connection->susp_fen[which]) {
+ rv = true;
+ break;
+ }
}
rcu_read_unlock();
- return conn;
+ return rv;
}
-static bool no_peer_wf_report_params(struct drbd_connection *connection)
+bool resource_is_suspended(struct drbd_resource *resource, enum which_state which)
{
- struct drbd_peer_device *peer_device;
+ bool rv = resource->susp_user[which] || resource->susp_nod[which] ||
+ resource->susp_quorum[which] || resource->susp_uuid[which];
+
+ if (rv)
+ return rv;
+
+ return is_suspended_fen(resource, which);
+}
+
+static void count_objects(struct drbd_resource *resource,
+ struct drbd_state_change_object_count *ocnt)
+{
+ struct drbd_path *path;
+ struct drbd_device *device;
+ struct drbd_connection *connection;
int vnr;
- bool rv = true;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
- if (peer_device->device->state.conn == C_WF_REPORT_PARAMS) {
- rv = false;
- break;
+ lockdep_assert_held(&resource->state_rwlock);
+
+ ocnt->n_devices = 0;
+ ocnt->n_connections = 0;
+ ocnt->n_paths = 0;
+
+ idr_for_each_entry(&resource->devices, device, vnr)
+ ocnt->n_devices++;
+ for_each_connection(connection, resource) {
+ ocnt->n_connections++;
+ list_for_each_entry(path, &connection->transport.paths, list) {
+ ocnt->n_paths++;
}
- rcu_read_unlock();
+ }
+}
+
+static struct drbd_state_change *alloc_state_change(struct drbd_state_change_object_count *ocnt, gfp_t flags)
+{
+ struct drbd_state_change *state_change;
+ unsigned int size;
+
+ size = sizeof(struct drbd_state_change) +
+ ocnt->n_devices * sizeof(struct drbd_device_state_change) +
+ ocnt->n_connections * sizeof(struct drbd_connection_state_change) +
+ ocnt->n_devices * ocnt->n_connections * sizeof(struct drbd_peer_device_state_change) +
+ ocnt->n_paths * sizeof(struct drbd_path_state);
+ state_change = kzalloc(size, flags);
+ if (!state_change)
+ return NULL;
+ state_change->n_connections = ocnt->n_connections;
+ state_change->n_devices = ocnt->n_devices;
+ state_change->n_paths = ocnt->n_paths;
+ state_change->devices = (void *)(state_change + 1);
+ state_change->connections = (void *)&state_change->devices[ocnt->n_devices];
+ state_change->peer_devices = (void *)&state_change->connections[ocnt->n_connections];
+ state_change->paths = (void *)&state_change->peer_devices[ocnt->n_devices*ocnt->n_connections];
+ return state_change;
+}
+
+struct drbd_state_change *remember_state_change(struct drbd_resource *resource, gfp_t gfp)
+{
+ struct drbd_state_change *state_change;
+ struct drbd_device *device;
+ struct drbd_connection *connection;
+ struct drbd_state_change_object_count ocnt;
+ int vnr;
+
+ struct drbd_device_state_change *device_state_change;
+ struct drbd_peer_device_state_change *peer_device_state_change;
+ struct drbd_connection_state_change *connection_state_change;
+ struct drbd_path_state *path_state; /* yes, not a _change :-( */
+
+ lockdep_assert_held(&resource->state_rwlock);
+
+ count_objects(resource, &ocnt);
+ state_change = alloc_state_change(&ocnt, gfp);
+ if (!state_change)
+ return NULL;
+
+ kref_get(&resource->kref);
+ state_change->resource->resource = resource;
+ memcpy(state_change->resource->role,
+ resource->role, sizeof(resource->role));
+ memcpy(state_change->resource->susp,
+ resource->susp_user, sizeof(resource->susp_user));
+ memcpy(state_change->resource->susp_nod,
+ resource->susp_nod, sizeof(resource->susp_nod));
+ memcpy(state_change->resource->susp_uuid,
+ resource->susp_uuid, sizeof(resource->susp_uuid));
+ memcpy(state_change->resource->fail_io,
+ resource->fail_io, sizeof(resource->fail_io));
+ memcpy(state_change->resource->resume_held_for_outdate,
+ resource->resume_held_for_outdate,
+ sizeof(resource->resume_held_for_outdate));
+
+ device_state_change = state_change->devices;
+ peer_device_state_change = state_change->peer_devices;
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+
+ kref_get(&device->kref);
+ device_state_change->device = device;
+ memcpy(device_state_change->disk_state,
+ device->disk_state, sizeof(device->disk_state));
+ memcpy(device_state_change->have_quorum,
+ device->have_quorum, sizeof(device->have_quorum));
+
+ /* The peer_devices for each device have to be enumerated in
+ the order of the connections. We may not use for_each_peer_device() here. */
+ for_each_connection(connection, resource) {
+ peer_device = conn_peer_device(connection, device->vnr);
+
+ peer_device_state_change->peer_device = peer_device;
+ memcpy(peer_device_state_change->disk_state,
+ peer_device->disk_state, sizeof(peer_device->disk_state));
+ memcpy(peer_device_state_change->repl_state,
+ peer_device->repl_state, sizeof(peer_device->repl_state));
+ memcpy(peer_device_state_change->resync_susp_user,
+ peer_device->resync_susp_user,
+ sizeof(peer_device->resync_susp_user));
+ memcpy(peer_device_state_change->resync_susp_peer,
+ peer_device->resync_susp_peer,
+ sizeof(peer_device->resync_susp_peer));
+ memcpy(peer_device_state_change->resync_susp_dependency,
+ peer_device->resync_susp_dependency,
+ sizeof(peer_device->resync_susp_dependency));
+ memcpy(peer_device_state_change->resync_susp_other_c,
+ peer_device->resync_susp_other_c,
+ sizeof(peer_device->resync_susp_other_c));
+ memcpy(peer_device_state_change->resync_susp_max_parallel,
+ peer_device->resync_susp_max_parallel,
+ sizeof(peer_device->resync_susp_max_parallel));
+ memcpy(peer_device_state_change->resync_active,
+ peer_device->resync_active,
+ sizeof(peer_device->resync_active));
+ memcpy(peer_device_state_change->replication,
+ peer_device->replication,
+ sizeof(peer_device->replication));
+ memcpy(peer_device_state_change->peer_replication,
+ peer_device->peer_replication,
+ sizeof(peer_device->peer_replication));
+ peer_device_state_change++;
+ }
+ device_state_change++;
+ }
+
+ connection_state_change = state_change->connections;
+ path_state = state_change->paths;
+ for_each_connection(connection, resource) {
+ struct drbd_path *path;
+
+ kref_get(&connection->kref);
+ connection_state_change->connection = connection;
+ memcpy(connection_state_change->cstate,
+ connection->cstate, sizeof(connection->cstate));
+ memcpy(connection_state_change->peer_role,
+ connection->peer_role, sizeof(connection->peer_role));
+ memcpy(connection_state_change->susp_fen,
+ connection->susp_fen, sizeof(connection->susp_fen));
+
+ list_for_each_entry(path, &connection->transport.paths, list) {
+ /* Share the connection kref with above.
+ * Could also share the pointer, but would then need to
+ * remember an additional n_paths per connection
+ * count/offset (connection_state_change->n_paths++)
+ * to be able to associate the paths with its connection.
+ * So why not directly store the pointer here again. */
+ path_state->connection = connection;
+ kref_get(&path->kref);
+ path_state->path = path;
+ path_state->path_established = test_bit(TR_ESTABLISHED, &path->flags);
+
+ path_state++;
+ }
+
+ connection_state_change++;
+ }
+
+ return state_change;
+}
+
+void copy_old_to_new_state_change(struct drbd_state_change *state_change)
+{
+ struct drbd_resource_state_change *resource_state_change = &state_change->resource[0];
+ unsigned int n_device, n_connection, n_peer_device, n_peer_devices;
+
+#define OLD_TO_NEW(x) \
+ (x[NEW] = x[OLD])
+
+ OLD_TO_NEW(resource_state_change->role);
+ OLD_TO_NEW(resource_state_change->susp);
+ OLD_TO_NEW(resource_state_change->susp_nod);
+ OLD_TO_NEW(resource_state_change->susp_uuid);
+ OLD_TO_NEW(resource_state_change->fail_io);
+ OLD_TO_NEW(resource_state_change->resume_held_for_outdate);
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change =
+ &state_change->connections[n_connection];
+
+ OLD_TO_NEW(connection_state_change->peer_role);
+ OLD_TO_NEW(connection_state_change->cstate);
+ OLD_TO_NEW(connection_state_change->susp_fen);
+ }
+
+ for (n_device = 0; n_device < state_change->n_devices; n_device++) {
+ struct drbd_device_state_change *device_state_change =
+ &state_change->devices[n_device];
+
+ OLD_TO_NEW(device_state_change->disk_state);
+ OLD_TO_NEW(device_state_change->have_quorum);
+ }
+
+ n_peer_devices = state_change->n_devices * state_change->n_connections;
+ for (n_peer_device = 0; n_peer_device < n_peer_devices; n_peer_device++) {
+ struct drbd_peer_device_state_change *p =
+ &state_change->peer_devices[n_peer_device];
+
+ OLD_TO_NEW(p->disk_state);
+ OLD_TO_NEW(p->repl_state);
+ OLD_TO_NEW(p->resync_susp_user);
+ OLD_TO_NEW(p->resync_susp_peer);
+ OLD_TO_NEW(p->resync_susp_dependency);
+ OLD_TO_NEW(p->resync_susp_other_c);
+ OLD_TO_NEW(p->resync_susp_max_parallel);
+ OLD_TO_NEW(p->resync_active);
+ OLD_TO_NEW(p->replication);
+ OLD_TO_NEW(p->peer_replication);
+ }
+
+#undef OLD_TO_NEW
+}
+
+void forget_state_change(struct drbd_state_change *state_change)
+{
+ unsigned int n;
+
+ if (!state_change)
+ return;
+
+ if (state_change->resource->resource) {
+ kref_put(&state_change->resource->resource->kref, drbd_destroy_resource);
+ }
+ for (n = 0; n < state_change->n_devices; n++) {
+ struct drbd_device *device = state_change->devices[n].device;
+
+ if (device) {
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+ }
+ for (n = 0; n < state_change->n_connections; n++) {
+ struct drbd_connection *connection =
+ state_change->connections[n].connection;
+
+ if (connection) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+ }
+ for (n = 0; n < state_change->n_paths; n++) {
+ struct drbd_path *path = state_change->paths[n].path;
+
+ if (path)
+ kref_put(&path->kref, drbd_destroy_path);
+ }
+ kfree(state_change);
+}
+
+static bool state_has_changed(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ int vnr;
+
+ if (resource->state_change_flags & CS_FORCE_RECALC)
+ return true;
+
+ if (resource->role[OLD] != resource->role[NEW] ||
+ resource->susp_user[OLD] != resource->susp_user[NEW] ||
+ resource->susp_nod[OLD] != resource->susp_nod[NEW] ||
+ resource->susp_quorum[OLD] != resource->susp_quorum[NEW] ||
+ resource->susp_uuid[OLD] != resource->susp_uuid[NEW] ||
+ resource->fail_io[OLD] != resource->fail_io[NEW])
+ return true;
+
+ for_each_connection(connection, resource) {
+ if (connection->cstate[OLD] != connection->cstate[NEW] ||
+ connection->peer_role[OLD] != connection->peer_role[NEW] ||
+ connection->susp_fen[OLD] != connection->susp_fen[NEW])
+ return true;
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+
+ if (device->disk_state[OLD] != device->disk_state[NEW] ||
+ device->have_quorum[OLD] != device->have_quorum[NEW])
+ return true;
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->disk_state[OLD] != peer_device->disk_state[NEW] ||
+ peer_device->repl_state[OLD] != peer_device->repl_state[NEW] ||
+ peer_device->resync_susp_user[OLD] !=
+ peer_device->resync_susp_user[NEW] ||
+ peer_device->resync_susp_peer[OLD] !=
+ peer_device->resync_susp_peer[NEW] ||
+ peer_device->resync_susp_dependency[OLD] !=
+ peer_device->resync_susp_dependency[NEW] ||
+ peer_device->resync_susp_other_c[OLD] !=
+ peer_device->resync_susp_other_c[NEW] ||
+ peer_device->resync_susp_max_parallel[OLD] !=
+ peer_device->resync_susp_max_parallel[NEW] ||
+ peer_device->resync_active[OLD] !=
+ peer_device->resync_active[NEW] ||
+ peer_device->replication[OLD] !=
+ peer_device->replication[NEW] ||
+ peer_device->peer_replication[OLD] !=
+ peer_device->peer_replication[NEW] ||
+ peer_device->uuid_flags & UUID_FLAG_GOT_STABLE)
+ return true;
+ }
+ }
+ return false;
+}
+
+static void ___begin_state_change(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ int vnr;
+
+ resource->role[NEW] = resource->role[NOW];
+ resource->susp_user[NEW] = resource->susp_user[NOW];
+ resource->susp_nod[NEW] = resource->susp_nod[NOW];
+ resource->susp_quorum[NEW] = resource->susp_quorum[NOW];
+ resource->susp_uuid[NEW] = resource->susp_uuid[NOW];
+ resource->fail_io[NEW] = resource->fail_io[NOW];
+ resource->resume_held_for_outdate[NEW] = resource->resume_held_for_outdate[NOW];
+
+ for_each_connection_rcu(connection, resource) {
+ connection->cstate[NEW] = connection->cstate[NOW];
+ connection->peer_role[NEW] = connection->peer_role[NOW];
+ connection->susp_fen[NEW] = connection->susp_fen[NOW];
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+
+ device->disk_state[NEW] = device->disk_state[NOW];
+ device->have_quorum[NEW] = device->have_quorum[NOW];
+ device->quorum[NEW] = device->quorum[NOW];
+
+ for_each_peer_device_rcu(peer_device, device) {
+ peer_device->disk_state[NEW] = peer_device->disk_state[NOW];
+ peer_device->repl_state[NEW] = peer_device->repl_state[NOW];
+ peer_device->resync_susp_user[NEW] =
+ peer_device->resync_susp_user[NOW];
+ peer_device->resync_susp_peer[NEW] =
+ peer_device->resync_susp_peer[NOW];
+ peer_device->resync_susp_dependency[NEW] =
+ peer_device->resync_susp_dependency[NOW];
+ peer_device->resync_susp_other_c[NEW] =
+ peer_device->resync_susp_other_c[NOW];
+ peer_device->resync_susp_max_parallel[NEW] =
+ peer_device->resync_susp_max_parallel[NOW];
+ peer_device->resync_active[NEW] =
+ peer_device->resync_active[NOW];
+ peer_device->replication[NEW] =
+ peer_device->replication[NOW];
+ peer_device->peer_replication[NEW] =
+ peer_device->peer_replication[NOW];
+ }
+ }
+}
+
+static void __begin_state_change(struct drbd_resource *resource)
+{
+ rcu_read_lock();
+ ___begin_state_change(resource);
+}
+
+static enum drbd_state_rv try_state_change(struct drbd_resource *resource)
+{
+ enum drbd_state_rv rv;
+ if (!state_has_changed(resource))
+ return SS_NOTHING_TO_DO;
+ sanitize_state(resource);
+ rv = is_valid_transition(resource);
+ if (rv >= SS_SUCCESS && !(resource->state_change_flags & CS_HARD))
+ rv = is_valid_soft_transition(resource);
return rv;
}
-static void wake_up_all_devices(struct drbd_connection *connection)
+static void apply_update_to_exposed_data_uuid(struct drbd_resource *resource)
{
- struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+ int vnr;
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ u64 nedu = device->next_exposed_data_uuid;
+ int changed = 0;
+
+ if (!nedu)
+ continue;
+ if (device->disk_state[NOW] < D_INCONSISTENT)
+ changed = drbd_uuid_set_exposed(device, nedu, false);
+
+ device->next_exposed_data_uuid = 0;
+ if (changed)
+ drbd_info(device, "Executing delayed exposed data uuid update: %016llX\n",
+ (unsigned long long)device->exposed_data_uuid);
+ else
+ drbd_info(device, "Canceling delayed exposed data uuid update\n");
+ }
+}
+
+/* Discard the replies along with the transaction they answered, so the next
+ * one cannot read a reply to this one as its own.
+ */
+static void drbd_clear_twopc_replies(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ clear_bit(TWOPC_PREPARED, &connection->flags);
+ clear_bit(TWOPC_YES, &connection->flags);
+ clear_bit(TWOPC_NO, &connection->flags);
+ clear_bit(TWOPC_RETRY, &connection->flags);
+ }
+ rcu_read_unlock();
+}
+
+/* A participant of a resize keeps its application I/O suspended until the
+ * transaction ends here, so that no write is in flight on any link when the
+ * COMMIT arrives and every node writes its bitmap in parallel. A participant
+ * takes it in the state_rwlock section that makes it one, the initiator once
+ * it knows that it commits; either way under the lock that orders it against
+ * the end of the transaction.
+ */
+void twopc_suspend_io(struct drbd_device *device)
+{
+ lockdep_assert_held(&device->resource->state_rwlock);
+
+ if (!test_and_set_bit(TWOPC_IO_SUSPENDED, &device->flags))
+ __drbd_suspend_io(device);
+}
+
+/* The peers' 2PC timers start when they receive the prepare. The initiator's
+ * drain before the COMMIT adds to a peer's drain before its reply, and
+ * forwarded prepares chain the drains, so their sum has to stay well below
+ * the 2PC timeout. A drain that does not finish in time falls back to
+ * serialized bitmap writes; a 2PC timeout on a peer would leave the cluster
+ * with diverging sizes.
+ */
+static long twopc_drain_timeout(struct drbd_resource *resource)
+{
+ return twopc_timeout(resource) / 8;
+}
+
+void twopc_drain_io(struct drbd_device *device, enum suspend_scope ss)
+{
+ long timeout = twopc_drain_timeout(device->resource);
+
+ if (!drbd_wait_io_drained(device, ss, timeout))
+ drbd_info(device, "Application I/O did not drain within %ums, continuing size change\n",
+ jiffies_to_msecs(timeout));
+}
+
+void __clear_remote_state_change(struct drbd_resource *resource)
+{
+ bool is_connect = resource->twopc_reply.is_connect;
+ int initiator_node_id = resource->twopc_reply.initiator_node_id;
+ struct drbd_connection *connection, *connect_connection = NULL;
+ struct drbd_device *device;
int vnr;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
- wake_up(&peer_device->device->state_wait);
- rcu_read_unlock();
+ lockdep_assert_held(&resource->state_rwlock);
+
+ if (is_connect) {
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (drbd_twopc_between_peer_and_me(connection))
+ connect_connection = connection;
+ }
+ rcu_read_unlock();
+ }
+ spin_lock(&resource->initiator_flush_lock);
+ resource->flush_connect_nodes = 0;
+ spin_unlock(&resource->initiator_flush_lock);
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (test_and_clear_bit(TWOPC_IO_SUSPENDED, &device->flags))
+ drbd_resume_io(device);
+ }
+
+ resource->remote_state_change = false;
+ resource->twopc_reply.initiator_node_id = -1;
+ resource->twopc_reply.tid = 0;
+ /* A reply this transaction scheduled but did not get to send answers a
+ * tid that no longer exists. Leaving the flag set also blocks
+ * when_done_lock() until the work runs.
+ */
+ clear_bit(TWOPC_WORK_PENDING, &resource->flags);
+ clear_bit(TWOPC_PREPARING, &resource->flags);
+ drbd_clear_twopc_replies(resource);
+
+ if (is_connect && resource->twopc_prepare_reply_cmd == 0) {
+ rcu_read_lock();
+ connection = drbd_connection_by_node_id(resource, initiator_node_id);
+ if (connection)
+ abort_connect(connection);
+ rcu_read_unlock();
+ }
+
+ /* A connect that did not happen: answer what we held back for it */
+ if (connect_connection && connect_connection->cstate[NOW] != C_CONNECTED)
+ drbd_flush_ack_for_lost_peer(connect_connection);
+
+ wake_up_all(&resource->twopc_wait);
+
+ /* Do things that where postponed to after two-phase commits finished */
+ apply_update_to_exposed_data_uuid(resource);
+}
+
+static bool state_is_stable(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ bool stable = true;
+
+ /* DO NOT add a default clause, we want the compiler to warn us
+ * for any newly introduced state we may have forgotten to add here */
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ switch (peer_device->repl_state[NOW]) {
+ /* New io is only accepted when the peer device is unknown or there is
+ * a well-established connection. */
+ case L_OFF:
+ case L_ESTABLISHED:
+ case L_SYNC_SOURCE:
+ case L_SYNC_TARGET:
+ case L_VERIFY_S:
+ case L_VERIFY_T:
+ case L_PAUSED_SYNC_S:
+ case L_PAUSED_SYNC_T:
+ case L_AHEAD:
+ case L_BEHIND:
+ case L_STARTING_SYNC_S:
+ case L_STARTING_SYNC_T:
+ break;
+
+ /* Allow IO in BM exchange states with new protocols */
+ case L_WF_BITMAP_S:
+ if (peer_device->connection->agreed_pro_version < 96)
+ stable = false;
+ break;
+
+ /* no new io accepted in these states */
+ case L_WF_BITMAP_T:
+ case L_WF_SYNC_UUID:
+ stable = false;
+ break;
+ }
+ if (!stable)
+ break;
+ }
+ rcu_read_unlock();
+
+ switch (device->disk_state[NOW]) {
+ case D_DISKLESS:
+ case D_INCONSISTENT:
+ case D_OUTDATED:
+ case D_CONSISTENT:
+ case D_UP_TO_DATE:
+ case D_FAILED:
+ case D_DETACHING:
+ /* disk state is stable as well. */
+ break;
+
+ /* no new io accepted during transitional states */
+ case D_ATTACHING:
+ case D_NEGOTIATING:
+ case D_UNKNOWN:
+ case D_MASK:
+ stable = false;
+ }
+
+ return stable;
+}
+
+static bool drbd_state_change_is_connect(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+
+ for_each_connection(connection, resource) {
+ if (connection->cstate[NOW] == C_CONNECTING &&
+ connection->cstate[NEW] == C_CONNECTED)
+ return true;
+ }
+
+ return false;
+}
+
+static struct after_state_change_work *alloc_after_state_change_work(struct drbd_resource *resource)
+{
+ struct after_state_change_work *work;
+
+ lockdep_assert_held(&resource->state_rwlock);
+
+ /* If the resource is already "unregistered", the worker thread
+ * is gone, there is no-one to consume the work item and release
+ * the associated refcounts. Just don't even create it.
+ */
+ if (test_bit(R_UNREGISTERED, &resource->flags))
+ return NULL;
+
+ work = kmalloc_obj(*work, GFP_ATOMIC);
+ if (work) {
+ work->state_change = remember_state_change(resource, GFP_ATOMIC);
+ if (!work->state_change) {
+ kfree(work);
+ work = NULL;
+ }
+ }
+ if (!work)
+ drbd_err(resource, "Could not allocate after state change work\n");
+
+ return work;
+}
+
+static void queue_after_state_change_work(struct drbd_resource *resource,
+ struct completion *done,
+ struct after_state_change_work *work)
+{
+ if (work) {
+ work->w.cb = w_after_state_change;
+ work->done = done;
+ drbd_queue_work(&resource->work, &work->w);
+ } else if (done) {
+ complete(done);
+ }
+}
+
+static enum drbd_state_rv ___end_state_change(struct drbd_resource *resource, struct completion *done,
+ enum drbd_state_rv rv, const char *tag)
+{
+ enum chg_state_flags flags = resource->state_change_flags;
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ bool is_connect;
+ unsigned int pro_ver;
+ int vnr;
+ bool all_devs_have_quorum = true;
+ struct after_state_change_work *work;
+
+ if (flags & CS_ABORT)
+ goto out;
+ if (rv >= SS_SUCCESS)
+ rv = try_state_change(resource);
+ if (rv < SS_SUCCESS) {
+ if (flags & CS_VERBOSE) {
+ drbd_err(resource, "State change failed: %s (%d)\n",
+ drbd_set_st_err_str(rv), rv);
+ print_state_change(resource, "Failed: ", tag);
+ }
+ goto out;
+ }
+ if (flags & CS_PREPARE)
+ goto out;
+
+ update_members(resource);
+ finish_state_change(resource, tag);
+
+ /* Check whether we are establishing a connection before applying the change. */
+ is_connect = drbd_state_change_is_connect(resource);
+
+ /* This remembers the state change, so call before applying the change. */
+ work = alloc_after_state_change_work(resource);
+
+ /* changes to local_cnt and device flags should be visible before
+ * changes to state, which again should be visible before anything else
+ * depending on that change happens. */
+ smp_wmb();
+ resource->role[NOW] = resource->role[NEW];
+ resource->susp_user[NOW] = resource->susp_user[NEW];
+ resource->susp_nod[NOW] = resource->susp_nod[NEW];
+ resource->susp_quorum[NOW] = resource->susp_quorum[NEW];
+ resource->susp_uuid[NOW] = resource->susp_uuid[NEW];
+ resource->fail_io[NOW] = resource->fail_io[NEW];
+ resource->resume_held_for_outdate[NOW] = resource->resume_held_for_outdate[NEW];
+ resource->cached_susp = resource_is_suspended(resource, NEW);
+
+ pro_ver = PRO_VERSION_MAX;
+ for_each_connection(connection, resource) {
+ bool was_down = connection->cstate[NOW] < C_CONNECTED;
+
+ connection->cstate[NOW] = connection->cstate[NEW];
+ connection->peer_role[NOW] = connection->peer_role[NEW];
+ connection->susp_fen[NOW] = connection->susp_fen[NEW];
+
+ pro_ver = min_t(unsigned int, pro_ver,
+ connection->agreed_pro_version);
+
+ wake_up(&connection->ee_wait);
+
+ /* Wake the sender here and not in finish_state_change(): only
+ * now does it read a cstate that lets it pick up the transfer
+ * log again.
+ */
+ if (was_down && connection->cstate[NOW] == C_CONNECTED)
+ wake_up(&connection->sender_work.q_wait);
+ }
+ resource->cached_min_aggreed_protocol_version = pro_ver;
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ bool err_io_before = device->cached_err_io;
+ struct drbd_res_opts *o = &resource->res_opts;
+ struct drbd_peer_device *peer_device;
+ bool err_io;
+
+ device->disk_state[NOW] = device->disk_state[NEW];
+ device->have_quorum[NOW] = device->have_quorum[NEW];
+ device->quorum[NOW] = device->quorum[NEW];
+
+ if (!device->have_quorum[NOW])
+ all_devs_have_quorum = false;
+
+ for_each_peer_device(peer_device, device) {
+ peer_device->disk_state[NOW] = peer_device->disk_state[NEW];
+ peer_device->repl_state[NOW] = peer_device->repl_state[NEW];
+ peer_device->resync_susp_user[NOW] =
+ peer_device->resync_susp_user[NEW];
+ peer_device->resync_susp_peer[NOW] =
+ peer_device->resync_susp_peer[NEW];
+ peer_device->resync_susp_dependency[NOW] =
+ peer_device->resync_susp_dependency[NEW];
+ peer_device->resync_susp_other_c[NOW] =
+ peer_device->resync_susp_other_c[NEW];
+ peer_device->resync_susp_max_parallel[NOW] =
+ peer_device->resync_susp_max_parallel[NEW];
+ peer_device->resync_active[NOW] =
+ peer_device->resync_active[NEW];
+ peer_device->replication[NOW] =
+ peer_device->replication[NEW];
+ peer_device->peer_replication[NOW] =
+ peer_device->peer_replication[NEW];
+ }
+ device->cached_state_unstable = !state_is_stable(device);
+ err_io = (o->on_no_quorum == ONQ_IO_ERROR && !device->have_quorum[NOW]) ||
+ (o->on_no_data == OND_IO_ERROR && !drbd_data_accessible(device, NOW)) ||
+ resource->fail_io[NEW];
+
+ /* Park and unpark the obligation at the edges of the io-error
+ * policy, so that whichever of the two holds writes back covers
+ * the moment the other changes: unpark before write admission
+ * returns, park once the errors already fail every write.
+ */
+ if (err_io_before && !err_io)
+ gen_obligation_unpark(device);
+ device->cached_err_io = err_io;
+ if (err_io && !err_io_before)
+ gen_obligation_park(device);
+ }
+ resource->cached_all_devices_have_quorum = all_devs_have_quorum;
+ smp_wmb(); /* Make the new state visible before the wake-ups below! */
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+
+ ensure_exposed_data_uuid(device);
+
+ wake_up(&device->al_wait);
+ wake_up(&device->misc_wait);
+
+ /* Due to the exclusivity of two-phase commits, there can only
+ * be one connection being established at once. Hence it is OK
+ * to release uuid_sem for all connections if the state change
+ * is establishing any connection. */
+ if (is_connect) {
+ for_each_peer_device(peer_device, device) {
+ if (test_and_clear_bit(HOLDING_UUID_READ_LOCK, peer_device->flags))
+ up_read_non_owner(&device->uuid_sem);
+ }
+ }
+ }
+
+ /* Progress for the bounded wait in stable_state_change(). The other
+ * wake-ups of state_wait are not state changes, and must not count.
+ */
+ if (rv != SS_NOTHING_TO_DO)
+ resource->state_change_seq++;
+ wake_up_all(&resource->state_wait);
+
+ /* Now that NOW carries this change, re-derive the outcome of a cluster-wide
+ * state change that is still waiting for replies: a prepared peer that just
+ * lost its connection cannot reply, so the wait ends in a retry instead of a
+ * twopc-timeout. The wake_up_all() above reaches a transaction this node
+ * initiated; this reaches one it only forwards, whose reply goes upstream
+ * from twopc_work and has nobody sleeping on state_wait. Not where
+ * this state change ends the transaction itself: the
+ * __clear_remote_state_change() below discards it and its replies.
+ */
+ if (!(flags & CS_TWOPC))
+ drbd_maybe_cluster_wide_reply(resource);
+
+ /* Informed confirmation of a rotated data generation. This state change
+ * may have gained quorum, brought a sync peer UpToDate, or lost a peer
+ * (discharging its durable-ack requirement) -- any of which can complete the
+ * "every surviving sync peer has it, and we are quorate" picture. Release
+ * the held completions of any generation that just became confirmed. Runs
+ * post-commit (NOW reflects the new state) under state_rwlock, like the
+ * COMPLETION_RESUMED walk in finish_state_change.
+ */
+ {
+ bool release_gen = false;
+
+ idr_for_each_entry(&resource->devices, device, vnr)
+ if (drbd_maybe_release_rotated_gen(device))
+ release_gen = true;
+ if (release_gen)
+ __tl_walk(resource, NULL, NULL, NEW_UUID_CONFIRMED);
+ }
+
+ /* Call this after applying the state change from NEW to NOW. */
+ queue_after_state_change_work(resource, done, work);
+out:
+ rcu_read_unlock();
+
+ if ((flags & CS_TWOPC) && !(flags & CS_PREPARE))
+ __clear_remote_state_change(resource);
+
+ resource->state_change_err_str = NULL;
+ return rv;
+}
+
+void state_change_lock(struct drbd_resource *resource, unsigned long *irq_flags, enum chg_state_flags flags)
+{
+ if ((flags & CS_SERIALIZE) && !(flags & (CS_ALREADY_SERIALIZED | CS_PREPARED))) {
+ WARN_ONCE(current == resource->worker.task,
+ "worker should not initiate state changes with CS_SERIALIZE\n");
+ down(&resource->state_sem);
+ }
+ write_lock_irqsave(&resource->state_rwlock, *irq_flags);
+ resource->state_change_flags = flags;
+}
+
+static void __state_change_unlock(struct drbd_resource *resource, unsigned long *irq_flags, struct completion *done)
+{
+ enum chg_state_flags flags = resource->state_change_flags;
+
+ resource->state_change_flags = 0;
+ write_unlock_irqrestore(&resource->state_rwlock, *irq_flags);
+ if (done && expect(resource, current != resource->worker.task))
+ wait_for_completion(done);
+ if ((flags & CS_SERIALIZE) && !(flags & (CS_ALREADY_SERIALIZED | CS_PREPARE)))
+ up(&resource->state_sem);
+}
+
+void state_change_unlock(struct drbd_resource *resource, unsigned long *irq_flags)
+{
+ __state_change_unlock(resource, irq_flags, NULL);
+}
+
+void begin_state_change_locked(struct drbd_resource *resource, enum chg_state_flags flags)
+{
+ BUG_ON(flags & (CS_SERIALIZE | CS_WAIT_COMPLETE | CS_PREPARE | CS_ABORT));
+ resource->state_change_flags = flags;
+ __begin_state_change(resource);
+}
+
+enum drbd_state_rv end_state_change_locked(struct drbd_resource *resource, const char *tag)
+{
+ return ___end_state_change(resource, NULL, SS_SUCCESS, tag);
+}
+
+void begin_state_change(struct drbd_resource *resource, unsigned long *irq_flags, enum chg_state_flags flags)
+{
+ state_change_lock(resource, irq_flags, flags);
+ __begin_state_change(resource);
+}
+
+static enum drbd_state_rv __end_state_change(struct drbd_resource *resource,
+ unsigned long *irq_flags,
+ enum drbd_state_rv rv,
+ const char *tag)
+{
+ enum chg_state_flags flags = resource->state_change_flags;
+ struct completion __done, *done = NULL;
+
+ if ((flags & CS_WAIT_COMPLETE) && !(flags & (CS_PREPARE | CS_ABORT))) {
+ done = &__done;
+ init_completion(done);
+ }
+ rv = ___end_state_change(resource, done, rv, tag);
+ __state_change_unlock(resource, irq_flags, rv >= SS_SUCCESS ? done : NULL);
+ return rv;
+}
+
+enum drbd_state_rv end_state_change(struct drbd_resource *resource, unsigned long *irq_flags,
+ const char *tag)
+{
+ return __end_state_change(resource, irq_flags, SS_SUCCESS, tag);
+}
+
+void abort_state_change(struct drbd_resource *resource, unsigned long *irq_flags)
+{
+ resource->state_change_flags &= ~CS_VERBOSE;
+ __end_state_change(resource, irq_flags, SS_UNKNOWN_ERROR, NULL);
+}
+
+void abort_state_change_locked(struct drbd_resource *resource)
+{
+ resource->state_change_flags &= ~CS_VERBOSE;
+ ___end_state_change(resource, NULL, SS_UNKNOWN_ERROR, NULL);
+}
+
+static void begin_remote_state_change(struct drbd_resource *resource, unsigned long *irq_flags)
+{
+ rcu_read_unlock();
+ write_unlock_irqrestore(&resource->state_rwlock, *irq_flags);
+}
+
+static void __end_remote_state_change(struct drbd_resource *resource, enum chg_state_flags flags)
+{
+ rcu_read_lock();
+ resource->state_change_flags = flags;
+ ___begin_state_change(resource);
+}
+
+static void end_remote_state_change(struct drbd_resource *resource, unsigned long *irq_flags, enum chg_state_flags flags)
+{
+ write_lock_irqsave(&resource->state_rwlock, *irq_flags);
+ __end_remote_state_change(resource, flags);
+}
+
+void clear_remote_state_change(struct drbd_resource *resource)
+{
+ unsigned long irq_flags;
+
+ write_lock_irqsave(&resource->state_rwlock, irq_flags);
+ __clear_remote_state_change(resource);
+ write_unlock_irqrestore(&resource->state_rwlock, irq_flags);
+}
+
+static union drbd_state drbd_get_resource_state(struct drbd_resource *resource, enum which_state which)
+{
+ union drbd_state rv = { {
+ .conn = C_STANDALONE, /* really: undefined */
+ /* (user_isp, peer_isp, and aftr_isp are undefined as well.) */
+ .disk = D_UNKNOWN, /* really: undefined */
+ .role = resource->role[which],
+ .peer = R_UNKNOWN, /* really: undefined */
+ .susp = resource->susp_user[which] || resource->susp_quorum[which] || resource->susp_uuid[which],
+ .susp_nod = resource->susp_nod[which],
+ .susp_fen = is_suspended_fen(resource, which),
+ .pdsk = D_UNKNOWN, /* really: undefined */
+ } };
+
+ return rv;
+}
+
+union drbd_state drbd_get_device_state(struct drbd_device *device, enum which_state which)
+{
+ union drbd_state rv = drbd_get_resource_state(device->resource, which);
+
+ rv.disk = device->disk_state[which];
+ rv.quorum = device->have_quorum[which];
+
+ return rv;
+}
+
+union drbd_state drbd_get_peer_device_state(struct drbd_peer_device *peer_device, enum which_state which)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ union drbd_state rv;
+
+ rv = drbd_get_device_state(peer_device->device, which);
+ rv.user_isp = peer_device->resync_susp_user[which];
+ rv.peer_isp = peer_device->resync_susp_peer[which];
+ rv.aftr_isp = resync_susp_comb_dep(peer_device, which);
+ rv.conn = combined_conn_state(peer_device, which);
+ rv.peer = connection->peer_role[which];
+ rv.pdsk = peer_device->disk_state[which];
+
+ return rv;
+}
+
+enum drbd_disk_state conn_highest_disk(struct drbd_connection *connection)
+{
+ enum drbd_disk_state disk_state = D_DISKLESS;
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ disk_state = max_t(enum drbd_disk_state, disk_state, device->disk_state[NOW]);
+ }
+ rcu_read_unlock();
+
+ return disk_state;
+}
+
+enum drbd_disk_state conn_highest_pdsk(struct drbd_connection *connection)
+{
+ enum drbd_disk_state disk_state = D_DISKLESS;
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ disk_state = max_t(enum drbd_disk_state, disk_state, peer_device->disk_state[NOW]);
+ rcu_read_unlock();
+
+ return disk_state;
+}
+
+static bool suspend_reason_changed(struct drbd_resource *resource)
+{
+ return resource->susp_user[OLD] != resource->susp_user[NEW] ||
+ resource->susp_nod[OLD] != resource->susp_nod[NEW] ||
+ resource->susp_quorum[OLD] != resource->susp_quorum[NEW] ||
+ resource->susp_uuid[OLD] != resource->susp_uuid[NEW] ||
+ is_suspended_fen(resource, OLD) != is_suspended_fen(resource, NEW);
+}
+
+static bool resync_suspended(struct drbd_peer_device *peer_device, enum which_state which)
+{
+ return peer_device->resync_susp_user[which] ||
+ peer_device->resync_susp_peer[which] ||
+ peer_device->resync_susp_max_parallel[which] ||
+ resync_susp_comb_dep(peer_device, which);
+}
+
+static int scnprintf_resync_suspend_flags(char *buffer, size_t size,
+ struct drbd_peer_device *peer_device,
+ enum which_state which)
+{
+ struct drbd_device *device = peer_device->device;
+ char *b = buffer, *end = buffer + size;
+
+ if (!resync_suspended(peer_device, which))
+ return scnprintf(buffer, size, "no");
+
+ if (peer_device->resync_susp_user[which])
+ b += scnprintf(b, end - b, "user,");
+ if (peer_device->resync_susp_peer[which])
+ b += scnprintf(b, end - b, "peer,");
+ if (peer_device->resync_susp_dependency[which])
+ b += scnprintf(b, end - b, "after dependency,");
+ if (peer_device->resync_susp_other_c[which])
+ b += scnprintf(b, end - b, "connection dependency,");
+ if (peer_device->resync_susp_max_parallel[which])
+ b += scnprintf(b, end - b, "max-parallel,");
+ if (is_sync_source_state(peer_device, which) && device->disk_state[which] <= D_INCONSISTENT)
+ b += scnprintf(b, end - b, "disk inconsistent,");
+
+ *(--b) = 0;
+
+ return b - buffer;
+}
+
+static int scnprintf_io_suspend_flags(char *buffer, size_t size,
+ struct drbd_resource *resource,
+ enum which_state which)
+{
+ char *b = buffer, *end = buffer + size;
+
+ if (!resource_is_suspended(resource, which))
+ return scnprintf(buffer, size, "no");
+
+ if (resource->susp_user[which])
+ b += scnprintf(b, end - b, "user,");
+ if (resource->susp_nod[which])
+ b += scnprintf(b, end - b, "no-disk,");
+ if (is_suspended_fen(resource, which))
+ b += scnprintf(b, end - b, "fencing,");
+ if (resource->susp_quorum[which])
+ b += scnprintf(b, end - b, "quorum,");
+ if (resource->susp_uuid[which])
+ b += scnprintf(b, end - b, "uuid,");
+ *(--b) = 0;
+
+ return b - buffer;
+}
+
+static void print_state_change(struct drbd_resource *resource, const char *prefix, const char *tag)
+{
+ char buffer[150], *b, *end = buffer + sizeof(buffer);
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ enum drbd_role *role = resource->role;
+ bool *fail_io = resource->fail_io;
+ int vnr;
+
+ b = buffer;
+ if (role[OLD] != role[NEW])
+ b += scnprintf(b, end - b, "role( %s -> %s ) ",
+ drbd_role_str(role[OLD]),
+ drbd_role_str(role[NEW]));
+ if (suspend_reason_changed(resource)) {
+ b += scnprintf(b, end - b, "susp-io( ");
+ b += scnprintf_io_suspend_flags(b, end - b, resource, OLD);
+ b += scnprintf(b, end - b, " -> ");
+ b += scnprintf_io_suspend_flags(b, end - b, resource, NEW);
+ b += scnprintf(b, end - b, " ) ");
+ }
+ if (fail_io[OLD] != fail_io[NEW])
+ b += scnprintf(b, end - b, "force-io-failures( %s -> %s ) ",
+ fail_io[OLD] ? "yes" : "no",
+ fail_io[NEW] ? "yes" : "no");
+ if (b != buffer) {
+ *(b-1) = 0;
+ drbd_info(resource, "%s%s%s%s%s\n", prefix, buffer,
+ tag ? " [" : "", tag ?: "", tag ? "]" : "");
+ }
+
+ for_each_connection(connection, resource) {
+ enum drbd_conn_state *cstate = connection->cstate;
+ enum drbd_role *peer_role = connection->peer_role;
+
+ b = buffer;
+ if (cstate[OLD] != cstate[NEW])
+ b += scnprintf(b, end - b, "conn( %s -> %s ) ",
+ drbd_conn_str(cstate[OLD]),
+ drbd_conn_str(cstate[NEW]));
+ if (peer_role[OLD] != peer_role[NEW])
+ b += scnprintf(b, end - b, "peer( %s -> %s ) ",
+ drbd_role_str(peer_role[OLD]),
+ drbd_role_str(peer_role[NEW]));
+
+ if (b != buffer) {
+ *(b-1) = 0;
+ drbd_info(connection, "%s%s%s%s%s\n", prefix, buffer,
+ tag ? " [" : "", tag ?: "", tag ? "]" : "");
+ }
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+ enum drbd_disk_state *disk_state = device->disk_state;
+ bool *have_quorum = device->have_quorum;
+
+ b = buffer;
+ if (disk_state[OLD] != disk_state[NEW])
+ b += scnprintf(b, end - b, "disk( %s -> %s ) ",
+ drbd_disk_str(disk_state[OLD]),
+ drbd_disk_str(disk_state[NEW]));
+ if (have_quorum[OLD] != have_quorum[NEW])
+ b += scnprintf(b, end - b, "quorum( %s -> %s ) ",
+ have_quorum[OLD] ? "yes" : "no",
+ have_quorum[NEW] ? "yes" : "no");
+ if (b != buffer) {
+ *(b-1) = 0;
+ drbd_info(device, "%s%s%s%s%s\n", prefix, buffer,
+ tag ? " [" : "", tag ?: "", tag ? "]" : "");
+ }
+
+ for_each_peer_device(peer_device, device) {
+ enum drbd_disk_state *peer_disk_state = peer_device->disk_state;
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+ bool *replication = peer_device->replication;
+ bool *peer_replication = peer_device->peer_replication;
+
+ b = buffer;
+ if (peer_disk_state[OLD] != peer_disk_state[NEW])
+ b += scnprintf(b, end - b, "pdsk( %s -> %s ) ",
+ drbd_disk_str(peer_disk_state[OLD]),
+ drbd_disk_str(peer_disk_state[NEW]));
+ if (repl_state[OLD] != repl_state[NEW])
+ b += scnprintf(b, end - b, "repl( %s -> %s ) ",
+ drbd_repl_str(repl_state[OLD]),
+ drbd_repl_str(repl_state[NEW]));
+
+ if (resync_suspended(peer_device, OLD) !=
+ resync_suspended(peer_device, NEW)) {
+ b += scnprintf(b, end - b, "resync-susp( ");
+ b += scnprintf_resync_suspend_flags(b, end - b, peer_device, OLD);
+ b += scnprintf(b, end - b, " -> ");
+ b += scnprintf_resync_suspend_flags(b, end - b, peer_device, NEW);
+ b += scnprintf(b, end - b, " ) ");
+ }
+
+ if (replication[OLD] != replication[NEW])
+ b += scnprintf(b, end - b, "replication( %s -> %s ) ",
+ replication[OLD] ? "yes" : "no",
+ replication[NEW] ? "yes" : "no");
+
+ if (peer_replication[OLD] != peer_replication[NEW])
+ b += scnprintf(b, end - b, "peer_replication( %s -> %s ) ",
+ peer_replication[OLD] ? "yes" : "no",
+ peer_replication[NEW] ? "yes" : "no");
+
+ if (b != buffer) {
+ *(b-1) = 0;
+ drbd_info(peer_device, "%s%s%s%s%s\n", prefix, buffer,
+ tag ? " [" : "", tag ?: "", tag ? "]" : "");
+ }
+ }
+ }
+}
+
+static bool local_disk_may_be_outdated(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+
+ if (device->resource->role[NEW] == R_PRIMARY) {
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->disk_state[NEW] == D_UP_TO_DATE &&
+ peer_device->repl_state[NEW] == L_WF_BITMAP_T)
+ return true;
+ }
+ return false;
+ }
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->connection->peer_role[NEW] == R_PRIMARY &&
+ peer_device->repl_state[NEW] > L_OFF)
+ goto have_primary_neighbor;
+ }
+
+ return true; /* No neighbor primary, I might be outdated*/
+
+have_primary_neighbor:
+ /* Allow self outdating while connecting to a diskless primary. */
+ if (peer_device->disk_state[NEW] == D_DISKLESS &&
+ peer_device->repl_state[OLD] == L_OFF && peer_device->repl_state[NEW] == L_ESTABLISHED)
+ return true;
+
+ for_each_peer_device(peer_device, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NEW];
+
+ switch (repl_state) {
+ case L_WF_BITMAP_S:
+ case L_STARTING_SYNC_S:
+ case L_SYNC_SOURCE:
+ case L_PAUSED_SYNC_S:
+ case L_AHEAD:
+ case L_ESTABLISHED:
+ case L_VERIFY_S:
+ case L_VERIFY_T:
+ case L_OFF:
+ continue;
+ case L_WF_SYNC_UUID:
+ case L_WF_BITMAP_T:
+ case L_STARTING_SYNC_T:
+ case L_SYNC_TARGET:
+ case L_PAUSED_SYNC_T:
+ case L_BEHIND:
+ return true;
+ }
+ }
+
+ return false;
+}
+
+static int calc_quorum_at(s32 setting, int voters)
+{
+ int quorum_at;
+
+ switch (setting) {
+ case QOU_MAJORITY:
+ quorum_at = voters / 2 + 1;
+ break;
+ case QOU_ALL:
+ quorum_at = voters;
+ break;
+ default:
+ quorum_at = setting;
+ }
+
+ return quorum_at;
+}
+
+static void __calc_quorum_with_disk(struct drbd_device *device, struct quorum_detail *qd)
+{
+ struct drbd_resource *resource = device->resource;
+ const u64 quorumless_nodes = device->have_quorum[NOW] ? ~resource->members : 0;
+ const int my_node_id = resource->res_opts.node_id;
+ int node_id;
+
+ check_wrongly_set_mdf_exists(device);
+
+ rcu_read_lock();
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
+ struct drbd_peer_device *peer_device;
+ enum drbd_disk_state disk_state;
+ enum drbd_repl_state repl_state;
+ bool is_intentional_diskless, is_tiebreaker;
+ struct drbd_net_conf *nc;
+
+ if (node_id == my_node_id) {
+ disk_state = device->disk_state[NEW];
+ if (disk_state > D_DISKLESS) {
+ if (disk_state == D_UP_TO_DATE)
+ qd->up_to_date++;
+ else
+ qd->present++;
+ }
+ continue;
+ }
+
+ /* Ignore non existing nodes.
+ Note: a fresh (before connected once), intentional diskless peer
+ gets ignored as well by this.
+ A fresh diskful peer counts! (since it has MDF_HAVE_BITMAP) */
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md->flags) &&
+ !test_bit(__MDF_NODE_EXISTS, &peer_md->flags) &&
+ !test_bit(__MDF_PEER_DEVICE_SEEN, &peer_md->flags))
+ continue;
+
+ peer_device = peer_device_by_node_id(device, node_id);
+
+ if (peer_device) {
+ is_intentional_diskless = !want_bitmap(peer_device);
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ is_tiebreaker = rcu_dereference(peer_device->conf)->peer_tiebreaker;
+ if (nc && !nc->allow_remote_read) {
+ dynamic_drbd_dbg(peer_device,
+ "Excluding from quorum calculation because allow-remote-read = no\n");
+ continue;
+ }
+ } else {
+ is_intentional_diskless =
+ !test_bit(__MDF_PEER_DEVICE_SEEN, &peer_md->flags);
+ is_tiebreaker = true;
+ }
+
+ if (is_intentional_diskless && !is_tiebreaker)
+ continue;
+
+ repl_state = peer_device ? peer_device->repl_state[NEW] : L_OFF;
+ disk_state = peer_device ? peer_device->disk_state[NEW] : D_UNKNOWN;
+
+ if (repl_state == L_OFF) {
+ if (is_intentional_diskless)
+ /* device should be diskless but is absent */
+ qd->missing_diskless++;
+ else if (disk_state <= D_OUTDATED ||
+ test_bit(__MDF_PEER_OUTDATED, &peer_md->flags))
+ qd->outdated++;
+ else if (NODE_MASK(node_id) & quorumless_nodes)
+ qd->quorumless++;
+ else
+ qd->unknown++;
+ } else {
+ if (disk_state == D_DISKLESS && is_intentional_diskless)
+ qd->diskless++;
+ else if (disk_state == D_UP_TO_DATE)
+ qd->up_to_date++;
+ else
+ qd->present++;
+ }
+ }
+ rcu_read_unlock();
+}
+
+static void __calc_quorum_no_disk(struct drbd_device *device, struct quorum_detail *qd)
+{
+ struct drbd_resource *resource = device->resource;
+ const u64 quorumless_nodes = device->have_quorum[NOW] ? ~resource->members : 0;
+ struct drbd_peer_device *peer_device;
+ bool is_intentional_diskless;
+
+ if (device->disk_state[NEW] == D_DISKLESS) {
+ /* We only want to consider ourselves as a diskless node when
+ * we actually intended to be diskless in the config. Otherwise,
+ * we shouldn't get a vote in the quorum process, so count
+ * ourselves as unknown. */
+ if (device->device_conf.intentional_diskless)
+ qd->diskless++;
+ else
+ qd->unknown++;
+ }
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_disk_state disk_state;
+ enum drbd_repl_state repl_state;
+ struct drbd_net_conf *nc;
+ bool is_tiebreaker;
+
+ repl_state = peer_device->repl_state[NEW];
+ disk_state = peer_device->disk_state[NEW];
+
+ is_intentional_diskless = !want_bitmap(peer_device);
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ is_tiebreaker = rcu_dereference(peer_device->conf)->peer_tiebreaker;
+ if (nc && !nc->allow_remote_read) {
+ dynamic_drbd_dbg(peer_device,
+ "Excluding from quorum calculation because allow-remote-read = no\n");
+ continue;
+ }
+ if (is_intentional_diskless && !is_tiebreaker)
+ continue;
+
+ if (repl_state == L_OFF) {
+ if (is_intentional_diskless)
+ /* device should be diskless but is absent */
+ qd->missing_diskless++;
+ else if (disk_state <= D_OUTDATED)
+ qd->outdated++;
+ else if (NODE_MASK(peer_device->node_id) & quorumless_nodes)
+ qd->quorumless++;
+ else
+ qd->unknown++;
+ } else {
+ if (disk_state == D_DISKLESS)
+ qd->diskless++;
+ else if (disk_state == D_UP_TO_DATE)
+ qd->up_to_date++;
+ else
+ qd->present++;
+ }
+
+ if (disk_state == D_UP_TO_DATE && test_bit(PEER_QUORATE, peer_device->flags))
+ qd->quorate_peers++;
+
+
+ }
+ rcu_read_unlock();
+}
+
+/*
+ * Computes the real quorum, including the diskless tiebreaker. The tiebreaker
+ * is sticky: it consults the previous result via device->quorum[NOW] (the
+ * tracked real quorum, NOT the enforced have_quorum[NOW], which is forced true
+ * when quorum is disabled). have_quorum is derived from this afterwards.
+ */
+static bool calc_quorum(struct drbd_device *device, struct quorum_info *qi)
+{
+ struct drbd_resource *resource = device->resource;
+ int voters, quorum_at, diskless_majority_at, min_redundancy_at;
+ struct quorum_detail qd = {};
+ bool have_quorum;
+
+ if (device->disk_state[NEW] > D_ATTACHING && get_ldev_if_state(device, D_ATTACHING)) {
+ __calc_quorum_with_disk(device, &qd);
+ put_ldev(device);
+ } else {
+ __calc_quorum_no_disk(device, &qd);
+ }
+
+ /* Check if a partition containing all missing nodes might have quorum */
+ voters = qd.outdated + qd.quorumless + qd.unknown + qd.up_to_date + qd.present;
+ quorum_at = calc_quorum_at(resource->res_opts.quorum, voters);
+ if (qd.outdated + qd.quorumless + qd.unknown >= quorum_at) {
+ /* when the missing nodes have the quorum, give up the quorumless */
+ qd.unknown += qd.quorumless;
+ qd.quorumless = 0;
+ }
+
+ /* When all the absent nodes are D_OUTDATED (no one D_UNKNOWN), we can be
+ sure that the other partition is not able to promote. ->
+ We remove them from the voters. -> We have quorum */
+ if (qd.unknown)
+ voters = qd.outdated + qd.quorumless + qd.unknown + qd.up_to_date + qd.present;
+ else
+ voters = qd.up_to_date + qd.present;
+
+ quorum_at = calc_quorum_at(resource->res_opts.quorum, voters);
+ diskless_majority_at = calc_quorum_at(QOU_MAJORITY, qd.diskless + qd.missing_diskless);
+ min_redundancy_at = calc_quorum_at(resource->res_opts.quorum_min_redundancy, voters);
+
+ if (qi) {
+ qi->voters = voters;
+ qi->up_to_date = qd.up_to_date;
+ qi->present = qd.present;
+ qi->quorum_at = quorum_at;
+ qi->min_redundancy_at = min_redundancy_at;
+ }
+
+ have_quorum = qd.quorate_peers ||
+ ((qd.up_to_date + qd.present) >= quorum_at && qd.up_to_date >= min_redundancy_at);
+
+ if (!have_quorum && voters != 0 && voters % 2 == 0 && qd.up_to_date + qd.present == quorum_at - 1 &&
+ /* It is an even number of nodes (think 2) and we failed by one vote.
+ Check if we have majority of the diskless nodes connected.
+ Using the diskless nodes a tie-breaker! */
+ qd.diskless >= diskless_majority_at && device->quorum[NOW]) {
+ have_quorum = true;
+ if (!test_bit(TIEBREAKER_QUORUM, &device->flags)) {
+ set_bit(TIEBREAKER_QUORUM, &device->flags);
+ drbd_info(device, "Would lose quorum, but using tiebreaker logic to keep\n");
+ }
+ } else {
+ clear_bit(TIEBREAKER_QUORUM, &device->flags);
+ }
+
+ return have_quorum;
+}
+
+static __printf(2, 3) void _drbd_state_err(struct change_context *context, const char *fmt, ...)
+{
+ struct drbd_resource *resource = context->resource;
+ const char *err_str;
+ va_list args;
+
+ va_start(args, fmt);
+ err_str = kvasprintf(GFP_ATOMIC, fmt, args);
+ va_end(args);
+ if (!err_str)
+ return;
+ if (context->flags & CS_VERBOSE)
+ drbd_err(resource, "%s\n", err_str);
+
+ if (context->err_str)
+ *context->err_str = err_str;
+ else
+ kfree(err_str);
+}
+
+static __printf(2, 3) void drbd_state_err(struct drbd_resource *resource, const char *fmt, ...)
+{
+ const char *err_str;
+ va_list args;
+
+ va_start(args, fmt);
+ err_str = kvasprintf(GFP_ATOMIC, fmt, args);
+ va_end(args);
+ if (!err_str)
+ return;
+ if (resource->state_change_flags & CS_VERBOSE)
+ drbd_err(resource, "%s\n", err_str);
+
+ if (resource->state_change_err_str)
+ *resource->state_change_err_str = err_str;
+ else
+ kfree(err_str);
+}
+
+/* Without a disk of our own, we know the data generation we expose. A peer
+ * that holds an ancestor of it lacks writes that reached other peers only.
+ * It must not become Primary: the downgrade in
+ * diskless_with_peers_different_current_uuids() changes only our view of it.
+ */
+static bool peer_behind_exposed_data(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ u64 exposed = device->exposed_data_uuid & ~UUID_PRIMARY;
+
+ return device->disk_state[NEW] == D_DISKLESS && exposed &&
+ peer_device->disk_state[NEW] > D_INCONSISTENT &&
+ (peer_device->current_uuid & ~UUID_PRIMARY) != exposed &&
+ peer_data_is_ancestor_of_mine(peer_device);
+}
+
+static enum drbd_state_rv __is_valid_soft_transition(struct drbd_resource *resource)
+{
+ enum drbd_role *role = resource->role;
+ bool *fail_io = resource->fail_io;
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ bool in_handshake = false;
+ int vnr;
+
+ /* See drbd_state_sw_errors in drbd_strings.c */
+
+ if (role[OLD] != R_PRIMARY && role[NEW] == R_PRIMARY) {
+ for_each_connection_rcu(connection, resource) {
+ struct drbd_net_conf *nc;
+
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (!nc || nc->two_primaries)
+ continue;
+ if (connection->peer_role[NEW] == R_PRIMARY)
+ return SS_TWO_PRIMARIES;
+ }
+ }
+
+ for_each_connection_rcu(connection, resource) {
+ struct drbd_peer_device *peer_device;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (test_bit(HOLDING_UUID_READ_LOCK, peer_device->flags) &&
+ peer_device->repl_state[NOW] == L_OFF) {
+ in_handshake = true;
+ goto handshake_found;
+ }
+ }
+ }
+handshake_found:
+
+ if (in_handshake && role[OLD] != R_PRIMARY && role[NEW] == R_PRIMARY)
+ return SS_IN_TRANSIENT_STATE;
+
+ if (role[OLD] == R_SECONDARY && role[NEW] == R_PRIMARY && fail_io[NEW])
+ return SS_DEVICE_IN_USE;
+
+ for_each_connection_rcu(connection, resource) {
+ enum drbd_conn_state *cstate = connection->cstate;
+ enum drbd_role *peer_role = connection->peer_role;
+ struct drbd_peer_device *peer_device;
+ struct drbd_net_conf *nc;
+ bool two_primaries;
+
+ if (cstate[NEW] == C_DISCONNECTING && cstate[OLD] == C_STANDALONE)
+ return SS_ALREADY_STANDALONE;
+
+ if (cstate[NEW] == C_CONNECTING && cstate[OLD] < C_UNCONNECTED)
+ return SS_NO_NET_CONFIG;
+
+ if (cstate[NEW] == C_DISCONNECTING && cstate[OLD] == C_UNCONNECTED)
+ return SS_IN_TRANSIENT_STATE;
+
+ nc = rcu_dereference(connection->transport.net_conf);
+ two_primaries = nc ? nc->two_primaries : false;
+ if (peer_role[NEW] == R_PRIMARY && peer_role[OLD] != R_PRIMARY && !two_primaries) {
+ if (role[NOW] == R_PRIMARY)
+ return SS_TWO_PRIMARIES;
+ if (!fail_io[NEW]) {
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (!device->writable && device->open_cnt)
+ return SS_PRIMARY_READER;
+ /*
+ * One might be tempted to add "|| open_rw_cont" here.
+ * That is wrong. The promotion of a rw opener will be
+ * handled in its own two-phase commit.
+ * Returning SS_PRIMARY_READER for a rw_opener might
+ * causes confusion for the caller, if that then waits
+ * for the read-only openers to go away.
+ */
+ }
+ }
+ }
+
+ if (peer_role[OLD] == R_SECONDARY && peer_role[NEW] == R_PRIMARY) {
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (peer_behind_exposed_data(peer_device)) {
+ u64 exposed = peer_device->device->exposed_data_uuid;
+
+ drbd_warn(peer_device,
+ "Refusing promotion, peer's data is older than %016llX\n",
+ (unsigned long long)exposed);
+ return SS_NO_UP_TO_DATE_DISK;
+ }
+ }
+ }
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ enum drbd_disk_state *disk_state = device->disk_state;
+ struct drbd_peer_device *peer_device;
+ bool any_disk_up_to_date[2];
+ enum which_state which;
+ int nr_negotiating = 0;
+
+ if (in_handshake &&
+ ((disk_state[OLD] < D_ATTACHING && disk_state[NEW] == D_ATTACHING) ||
+ (disk_state[OLD] > D_DETACHING && disk_state[NEW] == D_DETACHING)))
+ return SS_IN_TRANSIENT_STATE;
+
+ if (role[OLD] == R_PRIMARY && role[NEW] == R_SECONDARY && device->writable &&
+ !(resource->state_change_flags & CS_FS_IGN_OPENERS))
+ return SS_DEVICE_IN_USE;
+
+ if (disk_state[NEW] > D_ATTACHING && disk_state[OLD] == D_DISKLESS)
+ return SS_IS_DISKLESS;
+
+ if (disk_state[NEW] == D_OUTDATED && disk_state[OLD] < D_OUTDATED &&
+ disk_state[OLD] != D_ATTACHING && disk_state[OLD] != D_NEGOTIATING) {
+ /* Do not allow outdate of inconsistent or diskless.
+ But we have to allow Inconsistent -> Outdated if a resync
+ finishes over one connection, and is paused on other connections */
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+
+ if (repl_state[OLD] == L_SYNC_TARGET && repl_state[NEW] == L_ESTABLISHED)
+ goto allow;
+ }
+ return SS_LOWER_THAN_OUTDATED;
+ }
+allow:
+
+ for (which = OLD; which <= NEW; which++)
+ any_disk_up_to_date[which] = drbd_data_accessible(device, which);
+
+ /* Prevent becoming primary while there is not data accessible
+ and prevent detach or disconnect while primary */
+ if (!(role[OLD] == R_PRIMARY && !any_disk_up_to_date[OLD]) &&
+ (role[NEW] == R_PRIMARY && !any_disk_up_to_date[NEW]))
+ return SS_NO_UP_TO_DATE_DISK;
+
+ /* Prevent detach or disconnect while held open read only */
+ if (!device->writable && device->open_cnt &&
+ any_disk_up_to_date[OLD] && !any_disk_up_to_date[NEW])
+ return SS_NO_UP_TO_DATE_DISK;
+
+ if (disk_state[NEW] == D_NEGOTIATING)
+ nr_negotiating++;
+
+ /* Prevent promote when there is no quorum and
+ * prevent graceful disconnect/detach that would kill quorum
+ */
+ if ((role[OLD] == R_SECONDARY || device->have_quorum[OLD]) &&
+ role[NEW] == R_PRIMARY && !device->have_quorum[NEW]) {
+ struct quorum_info qi;
+
+ calc_quorum(device, &qi);
+
+ if (disk_state[NEW] <= D_ATTACHING)
+ drbd_state_err(resource, "no UpToDate peer with quorum");
+ else if (qi.up_to_date + qi.present < qi.quorum_at)
+ drbd_state_err(resource, "%d of %d nodes visible, need %d for quorum",
+ qi.up_to_date + qi.present, qi.voters, qi.quorum_at);
+ else if (qi.up_to_date < qi.min_redundancy_at)
+ drbd_state_err(resource, "%d of %d nodes up_to_date, need %d for "
+ "quorum-minimum-redundancy",
+ qi.up_to_date, qi.voters, qi.min_redundancy_at);
+ return SS_NO_QUORUM;
+ }
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_disk_state *peer_disk_state = peer_device->disk_state;
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+
+ if (peer_disk_state[NEW] == D_NEGOTIATING)
+ nr_negotiating++;
+
+ if (nr_negotiating > 1)
+ return SS_IN_TRANSIENT_STATE;
+
+ if (peer_device->connection->fencing_policy >= FP_RESOURCE &&
+ !(role[OLD] == R_PRIMARY && repl_state[OLD] < L_ESTABLISHED && !(peer_disk_state[OLD] <= D_OUTDATED)) &&
+ (role[NEW] == R_PRIMARY && repl_state[NEW] < L_ESTABLISHED && !(peer_disk_state[NEW] <= D_OUTDATED)))
+ return SS_PRIMARY_NOP;
+
+ if (!(repl_state[OLD] > L_ESTABLISHED && disk_state[OLD] < D_INCONSISTENT) &&
+ (repl_state[NEW] > L_ESTABLISHED && disk_state[NEW] < D_INCONSISTENT))
+ return SS_NO_LOCAL_DISK;
+
+ if (!(repl_state[OLD] > L_ESTABLISHED && peer_disk_state[OLD] < D_INCONSISTENT) &&
+ (repl_state[NEW] > L_ESTABLISHED && peer_disk_state[NEW] < D_INCONSISTENT))
+ return SS_NO_REMOTE_DISK;
+
+ if (disk_state[OLD] > D_OUTDATED && disk_state[NEW] == D_OUTDATED &&
+ !local_disk_may_be_outdated(device))
+ return SS_CONNECTED_OUTDATES;
+
+ if (!(repl_state[OLD] == L_VERIFY_S || repl_state[OLD] == L_VERIFY_T) &&
+ (repl_state[NEW] == L_VERIFY_S || repl_state[NEW] == L_VERIFY_T)) {
+ struct drbd_net_conf *nc;
+
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ if (!nc || nc->verify_alg[0] == 0)
+ return SS_NO_VERIFY_ALG;
+ }
+
+ if (!(repl_state[OLD] == L_VERIFY_S || repl_state[OLD] == L_VERIFY_T) &&
+ (repl_state[NEW] == L_VERIFY_S || repl_state[NEW] == L_VERIFY_T) &&
+ peer_device->connection->agreed_pro_version < 88)
+ return SS_NOT_SUPPORTED;
+
+ if (repl_is_sync_source(repl_state[OLD]) &&
+ repl_state[NEW] == L_WF_BITMAP_S)
+ return SS_RESYNC_RUNNING;
+
+ if (repl_is_sync_target(repl_state[OLD]) &&
+ repl_state[NEW] == L_WF_BITMAP_T)
+ return SS_RESYNC_RUNNING;
+
+ if (repl_state[NEW] != repl_state[OLD] &&
+ (repl_state[NEW] == L_STARTING_SYNC_T || repl_state[NEW] == L_STARTING_SYNC_S) &&
+ repl_state[OLD] > L_ESTABLISHED)
+ return SS_RESYNC_RUNNING;
+
+ if ((repl_state[NEW] == L_VERIFY_S || repl_state[NEW] == L_VERIFY_T) && repl_state[OLD] < L_ESTABLISHED)
+ return SS_NEED_CONNECTION;
+
+ if ((repl_state[NEW] == L_VERIFY_S || repl_state[NEW] == L_VERIFY_T) &&
+ repl_state[NEW] != repl_state[OLD] && repl_state[OLD] > L_ESTABLISHED)
+ return SS_RESYNC_RUNNING;
+
+ if ((repl_state[NEW] == L_STARTING_SYNC_S || repl_state[NEW] == L_STARTING_SYNC_T) &&
+ repl_state[OLD] < L_ESTABLISHED)
+ return SS_NEED_CONNECTION;
+
+ if ((repl_state[NEW] == L_SYNC_TARGET || repl_state[NEW] == L_SYNC_SOURCE)
+ && repl_state[OLD] < L_OFF)
+ return SS_NEED_CONNECTION; /* No NetworkFailure -> SyncTarget etc... */
+
+ if ((peer_disk_state[NEW] > D_DISKLESS && peer_disk_state[NEW] != D_UNKNOWN) &&
+ peer_disk_state[OLD] == D_DISKLESS && !want_bitmap(peer_device))
+ return SS_ATTACH_NO_BITMAP; /* peer with --bitmap=no wannts to attach ??? */
+ }
+ }
+
+ return SS_SUCCESS;
+}
+
+/**
+ * is_valid_soft_transition() - Returns an SS_ error code if state[NEW] is not valid
+ *
+ * "Soft" transitions are voluntary state changes which drbd may decline, such
+ * as a user request to promote a resource to primary. Opposed to that are
+ * involuntary or "hard" transitions like a network connection loss.
+ *
+ * When deciding if a "soft" transition should be allowed, "hard" transitions
+ * may already have forced the resource into a critical state. It may take
+ * several "soft" transitions to get the resource back to normal. To allow
+ * those, rather than checking if the desired new state is valid, we can only
+ * check if the desired new state is "at least as good" as the current state.
+ *
+ * @resource: DRBD resource
+ */
+static enum drbd_state_rv is_valid_soft_transition(struct drbd_resource *resource)
+{
+ enum drbd_state_rv rv;
+
+ rcu_read_lock();
+ rv = __is_valid_soft_transition(resource);
+ rcu_read_unlock();
+
+ return rv;
+}
+
+static enum drbd_state_rv
+is_valid_conn_transition(enum drbd_conn_state oc, enum drbd_conn_state nc)
+{
+ /* no change -> nothing to do, at least for the connection part */
+ if (oc == nc)
+ return SS_NOTHING_TO_DO;
+
+ /* disconnect of an unconfigured connection does not make sense */
+ if (oc == C_STANDALONE && nc == C_DISCONNECTING)
+ return SS_ALREADY_STANDALONE;
+
+ /* from C_STANDALONE, we start with C_UNCONNECTED */
+ if (oc == C_STANDALONE && nc != C_UNCONNECTED)
+ return SS_NEED_CONNECTION;
+
+ /* After a network error only C_UNCONNECTED or C_DISCONNECTING may follow. */
+ if (oc >= C_TIMEOUT && oc <= C_TEAR_DOWN && nc != C_UNCONNECTED && nc != C_DISCONNECTING)
+ return SS_IN_TRANSIENT_STATE;
+
+ /* After C_DISCONNECTING only C_STANDALONE may follow */
+ if (oc == C_DISCONNECTING && nc != C_STANDALONE)
+ return SS_IN_TRANSIENT_STATE;
+
+ return SS_SUCCESS;
+}
+
+
+/**
+ * is_valid_transition() - Returns an SS_ error code if the state transition is not possible
+ * This limits hard state transitions. Hard state transitions are facts there are
+ * imposed on DRBD by the environment. E.g. disk broke or network broke down.
+ * But those hard state transitions are still not allowed to do everything.
+ * @resource: DRBD resource.
+ */
+static enum drbd_state_rv is_valid_transition(struct drbd_resource *resource)
+{
+ enum drbd_state_rv rv;
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ int vnr;
+
+ for_each_connection(connection, resource) {
+ rv = is_valid_conn_transition(connection->cstate[OLD], connection->cstate[NEW]);
+ if (rv < SS_SUCCESS)
+ return rv;
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ /* we cannot fail (again) if we already detached */
+ if ((device->disk_state[NEW] == D_FAILED || device->disk_state[NEW] == D_DETACHING) &&
+ device->disk_state[OLD] == D_DISKLESS) {
+ return SS_IS_DISKLESS;
+ }
+ }
+
+ return SS_SUCCESS;
+}
+
+static bool is_sync_target_other_c(struct drbd_peer_device *ign_peer_device)
+{
+ struct drbd_device *device = ign_peer_device->device;
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device(peer_device, device) {
+ enum drbd_repl_state r;
+
+ if (peer_device == ign_peer_device)
+ continue;
+
+ r = peer_device->repl_state[NEW];
+ if (r == L_SYNC_TARGET || r == L_PAUSED_SYNC_T)
+ return true;
+ }
+
+ return false;
+}
+
+static void drbd_start_other_targets_paused(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_device *p;
+
+ for_each_peer_device(p, device) {
+ if (p == peer_device)
+ continue;
+
+ if (p->disk_state[NEW] >= D_INCONSISTENT && p->repl_state[NEW] == L_ESTABLISHED)
+ p->repl_state[NEW] = L_PAUSED_SYNC_T;
+ }
+}
+
+static bool drbd_is_sync_target_candidate(struct drbd_peer_device *peer_device)
+{
+ if (!repl_is_sync_target(peer_device->repl_state[NEW]))
+ return false;
+
+ if (peer_device->resync_susp_dependency[NEW] ||
+ peer_device->resync_susp_peer[NEW] ||
+ peer_device->resync_susp_user[NEW] ||
+ peer_device->resync_susp_max_parallel[NEW])
+ return false;
+
+ if (peer_device->disk_state[NEW] < D_OUTDATED)
+ return false;
+
+ return true;
+
+}
+
+static void drbd_select_sync_target(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_peer_device *target_current = NULL;
+ struct drbd_peer_device *target_active = NULL;
+ struct drbd_peer_device *target_desired = NULL;
+
+ /* Find current and active resync peers. */
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->repl_state[OLD] == L_SYNC_TARGET && drbd_is_sync_target_candidate(peer_device))
+ target_current = peer_device;
+
+ if (peer_device->resync_active[NEW])
+ target_active = peer_device;
+ }
+
+ /* Choose desired resync peer. */
+ for_each_peer_device_rcu(peer_device, device) {
+ if (!drbd_is_sync_target_candidate(peer_device))
+ continue;
+
+ if (target_desired && drbd_bm_total_weight(peer_device) > drbd_bm_total_weight(target_desired))
+ continue;
+
+ target_desired = peer_device;
+ }
+
+ /* Keep current resync target if the alternative has less than 1MiB
+ * storage (256 bits) less to resync. */
+ if (target_current && target_desired &&
+ drbd_bm_total_weight(target_current) < drbd_bm_total_weight(target_desired) + 256UL)
+ target_desired = target_current;
+
+ /* Do not activate/unpause a resync if some other is still active. */
+ if (target_desired && target_active && target_desired != target_active)
+ target_desired = NULL;
+
+ /* Activate resync (if not already active). */
+ if (target_desired)
+ target_desired->resync_active[NEW] = true;
+
+ /* Make sure that the targets are correctly paused/unpaused. */
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+
+ peer_device->resync_susp_other_c[NEW] = target_desired && peer_device != target_desired;
+
+ if (!repl_is_sync_target(repl_state[NEW]))
+ continue;
+
+ peer_device->repl_state[NEW] = peer_device == target_desired ? L_SYNC_TARGET : L_PAUSED_SYNC_T;
+ }
+}
+
+static bool drbd_change_to_inconsistent(enum drbd_disk_state *disk_state,
+ enum drbd_conn_state *cstate)
+{
+ return !(disk_state[OLD] == D_INCONSISTENT && cstate[OLD] == C_CONNECTED) &&
+ (disk_state[NEW] == D_INCONSISTENT && cstate[NEW] == C_CONNECTED);
+}
+
+static void sanitize_state(struct drbd_resource *resource)
+{
+ enum drbd_role *role = resource->role;
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ bool maybe_crashed_primary = false;
+ bool volume_lost_data_access = false;
+ bool volume_gained_data_access = false;
+ bool volumes_have_data_access = true;
+ bool resource_has_quorum = true;
+ int connected_primaries = 0;
+ int vnr;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ enum drbd_conn_state *cstate = connection->cstate;
+
+ if (cstate[NEW] < C_CONNECTED)
+ connection->peer_role[NEW] = R_UNKNOWN;
+
+ if (connection->peer_role[OLD] == R_PRIMARY && cstate[OLD] == C_CONNECTED &&
+ ((cstate[NEW] >= C_TIMEOUT && cstate[NEW] <= C_PROTOCOL_ERROR) ||
+ (cstate[NEW] == C_DISCONNECTING && resource->state_change_flags & CS_HARD)))
+ /* implies also C_BROKEN_PIPE and C_NETWORK_FAILURE */
+ maybe_crashed_primary = true;
+
+ if (connection->peer_role[NEW] == R_PRIMARY)
+ connected_primaries++;
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+ enum drbd_disk_state *disk_state = device->disk_state;
+ bool lost_connection = false;
+ bool have_good_peer = false;
+
+ if (disk_state[OLD] == D_DISKLESS && disk_state[NEW] == D_DETACHING)
+ disk_state[NEW] = D_DISKLESS;
+
+ if ((resource->state_change_flags & CS_IGN_OUTD_FAIL) &&
+ disk_state[OLD] < D_OUTDATED && disk_state[NEW] == D_OUTDATED)
+ disk_state[NEW] = disk_state[OLD];
+
+ if (disk_state[NEW] == D_NEGOTIATING) {
+ int all = 0, target = 0, no_result = 0;
+ bool up_to_date_neighbor = false;
+
+ if (disk_state[OLD] != D_NEGOTIATING) {
+ for_each_peer_device_rcu(peer_device, device)
+ peer_device->negotiation_result = L_NEGOTIATING;
+ }
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NEW];
+ enum drbd_repl_state nr = peer_device->negotiation_result;
+ enum drbd_disk_state pdsk = peer_device->disk_state[NEW];
+
+ if (pdsk < D_NEGOTIATING || repl_state == L_OFF)
+ continue;
+
+ if (pdsk == D_UP_TO_DATE)
+ up_to_date_neighbor = true;
+
+ all++;
+ if (nr == L_NEG_NO_RESULT)
+ no_result++;
+ else if (nr == L_NEGOTIATING)
+ goto stay_negotiating;
+ else if (nr == L_WF_BITMAP_T)
+ target++;
+ else if (nr != L_ESTABLISHED && nr != L_WF_BITMAP_S)
+ drbd_err(peer_device, "Unexpected nr = %s\n", drbd_repl_str(nr));
+ }
+
+ /* negotiation finished */
+ if (no_result > 0 && no_result == all)
+ disk_state[NEW] = D_DETACHING;
+ else if (target)
+ disk_state[NEW] = D_INCONSISTENT;
+ else
+ disk_state[NEW] = up_to_date_neighbor ? D_UP_TO_DATE :
+ /* ldev_safe: dstate */ disk_state_from_md(device);
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state nr = peer_device->negotiation_result;
+
+ if (peer_device->connection->cstate[NEW] < C_CONNECTED ||
+ nr == L_NEGOTIATING)
+ continue;
+
+ if (nr == L_NEG_NO_RESULT)
+ nr = L_ESTABLISHED;
+
+ if (nr == L_WF_BITMAP_S && disk_state[NEW] == D_INCONSISTENT) {
+ /* Should be sync source for one peer and sync
+ target for an other peer. Delay the sync source
+ role */
+ nr = L_PAUSED_SYNC_S;
+ peer_device->resync_susp_other_c[NEW] = true;
+ drbd_warn(peer_device, "Finish me\n");
+ }
+ peer_device->repl_state[NEW] = nr;
+ }
+ }
+stay_negotiating:
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+ enum drbd_disk_state *peer_disk_state = peer_device->disk_state;
+ struct drbd_connection *connection = peer_device->connection;
+ enum drbd_conn_state *cstate = connection->cstate;
+
+ if (peer_disk_state[NEW] == D_UP_TO_DATE &&
+ (device->exposed_data_uuid & ~UUID_PRIMARY) ==
+ (peer_device->current_uuid & ~UUID_PRIMARY))
+ have_good_peer = true;
+
+ if (repl_state[NEW] < L_ESTABLISHED) {
+ peer_device->resync_susp_peer[NEW] = false;
+ if (peer_disk_state[NEW] > D_UNKNOWN ||
+ peer_disk_state[NEW] < D_INCONSISTENT)
+ peer_disk_state[NEW] = D_UNKNOWN;
+ }
+ if (repl_state[OLD] >= L_ESTABLISHED && repl_state[NEW] < L_ESTABLISHED) {
+ lost_connection = true;
+ peer_device->resync_active[NEW] = false;
+ }
+
+ /* Clear the aftr_isp when becoming unconfigured */
+ if (cstate[NEW] == C_STANDALONE &&
+ disk_state[NEW] == D_DISKLESS &&
+ role[NEW] == R_SECONDARY)
+ peer_device->resync_susp_dependency[NEW] = false;
+
+ /* Abort resync if a disk fails/detaches */
+ if (repl_state[NEW] > L_ESTABLISHED &&
+ (disk_state[NEW] <= D_FAILED ||
+ peer_disk_state[NEW] <= D_FAILED)) {
+ repl_state[NEW] = L_ESTABLISHED;
+ clear_bit(RECONCILIATION_RESYNC, peer_device->flags);
+ peer_device->resync_active[NEW] = false;
+ }
+
+ /* Suspend IO while fence-peer handler runs (peer lost) */
+ if (connection->fencing_policy == FP_STONITH &&
+ (role[NEW] == R_PRIMARY &&
+ repl_state[NEW] < L_ESTABLISHED &&
+ peer_disk_state[NEW] == D_UNKNOWN) &&
+ (role[OLD] != R_PRIMARY ||
+ peer_disk_state[OLD] != D_UNKNOWN))
+ connection->susp_fen[NEW] = true;
+ }
+
+ drbd_select_sync_target(device);
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+ enum drbd_disk_state *peer_disk_state = peer_device->disk_state;
+ struct drbd_connection *connection = peer_device->connection;
+ enum drbd_disk_state min_disk_state, max_disk_state;
+ enum drbd_disk_state min_peer_disk_state, max_peer_disk_state;
+ enum drbd_role *peer_role = connection->peer_role;
+ bool uuids_match, cond;
+
+ /* Pause a SyncSource until it finishes resync as target on other connections */
+ if (repl_state[OLD] != L_SYNC_SOURCE && repl_state[NEW] == L_SYNC_SOURCE &&
+ is_sync_target_other_c(peer_device))
+ peer_device->resync_susp_other_c[NEW] = true;
+
+ if (resync_suspended(peer_device, NEW)) {
+ if (repl_state[NEW] == L_SYNC_SOURCE)
+ repl_state[NEW] = L_PAUSED_SYNC_S;
+ } else {
+ if (repl_state[NEW] == L_PAUSED_SYNC_S)
+ repl_state[NEW] = L_SYNC_SOURCE;
+ }
+
+ /* Implication of the repl state on other peer's repl state */
+ if (repl_state[OLD] != L_STARTING_SYNC_T && repl_state[NEW] == L_STARTING_SYNC_T)
+ drbd_start_other_targets_paused(peer_device);
+
+ /* D_CONSISTENT vanish when we get connected (pre 9.0) */
+ if (connection->agreed_pro_version < 110 &&
+ repl_state[NEW] >= L_ESTABLISHED && repl_state[NEW] < L_AHEAD) {
+ if (disk_state[NEW] == D_CONSISTENT)
+ disk_state[NEW] = D_UP_TO_DATE;
+ if (peer_disk_state[NEW] == D_CONSISTENT)
+ peer_disk_state[NEW] = D_UP_TO_DATE;
+ }
+
+ /* Implications of the repl state on the disk states */
+ min_disk_state = D_DISKLESS;
+ max_disk_state = D_UP_TO_DATE;
+ min_peer_disk_state = D_INCONSISTENT;
+ max_peer_disk_state = D_UNKNOWN;
+ switch (repl_state[NEW]) {
+ case L_OFF:
+ /* values from above */
+ break;
+ case L_WF_BITMAP_T:
+ case L_STARTING_SYNC_T:
+ case L_WF_SYNC_UUID:
+ case L_BEHIND:
+ min_disk_state = D_INCONSISTENT;
+ max_disk_state = D_OUTDATED;
+ min_peer_disk_state = D_INCONSISTENT;
+ max_peer_disk_state = D_UP_TO_DATE;
+ break;
+ case L_VERIFY_S:
+ case L_VERIFY_T:
+ min_disk_state = D_INCONSISTENT;
+ max_disk_state = D_UP_TO_DATE;
+ min_peer_disk_state = D_INCONSISTENT;
+ max_peer_disk_state = D_UP_TO_DATE;
+ break;
+ case L_ESTABLISHED:
+ min_disk_state = D_DISKLESS;
+ max_disk_state = D_UP_TO_DATE;
+ min_peer_disk_state = D_DISKLESS;
+ max_peer_disk_state = D_UP_TO_DATE;
+ break;
+ case L_WF_BITMAP_S:
+ case L_PAUSED_SYNC_S:
+ case L_STARTING_SYNC_S:
+ case L_AHEAD:
+ min_disk_state = D_INCONSISTENT;
+ max_disk_state = D_UP_TO_DATE;
+ min_peer_disk_state = D_INCONSISTENT;
+ max_peer_disk_state = D_CONSISTENT; /* D_OUTDATED would be nice. But explicit outdate necessary*/
+ break;
+ case L_PAUSED_SYNC_T:
+ case L_SYNC_TARGET:
+ min_disk_state = D_INCONSISTENT;
+ max_disk_state = D_INCONSISTENT;
+ min_peer_disk_state = D_INCONSISTENT;
+ max_peer_disk_state = D_UP_TO_DATE;
+ break;
+ case L_SYNC_SOURCE:
+ min_disk_state = D_INCONSISTENT;
+ max_disk_state = D_UP_TO_DATE;
+ min_peer_disk_state = D_INCONSISTENT;
+ max_peer_disk_state = D_INCONSISTENT;
+ break;
+ }
+
+ /* Implications of the repl state on the disk states */
+ if (disk_state[NEW] > max_disk_state)
+ disk_state[NEW] = max_disk_state;
+
+ if (disk_state[NEW] < min_disk_state)
+ disk_state[NEW] = min_disk_state;
+
+ if (peer_disk_state[NEW] > max_peer_disk_state)
+ peer_disk_state[NEW] = max_peer_disk_state;
+
+ if (peer_disk_state[NEW] < min_peer_disk_state)
+ peer_disk_state[NEW] = min_peer_disk_state;
+
+ /* A detach is a cluster wide transaction. The peer_disk_state updates
+ are coming in while we have it prepared. When the cluster wide
+ state change gets committed prevent D_DISKLESS -> D_FAILED */
+ if (peer_disk_state[OLD] == D_DISKLESS &&
+ (peer_disk_state[NEW] == D_FAILED || peer_disk_state[NEW] == D_DETACHING))
+ peer_disk_state[NEW] = D_DISKLESS;
+
+ /* Upgrade myself from D_OUTDATED if..
+ 1) We connect to stable D_UP_TO_DATE(or D_CONSISTENT) peer without resync
+ 2) The peer just became stable
+ 3) the peer was stable and just became D_UP_TO_DATE */
+ if (repl_state[NEW] == L_ESTABLISHED && disk_state[NEW] == D_OUTDATED &&
+ peer_disk_state[NEW] >= D_CONSISTENT &&
+ test_bit(UUIDS_RECEIVED, peer_device->flags) &&
+ peer_device->uuid_flags & UUID_FLAG_STABLE &&
+ (repl_state[OLD] < L_ESTABLISHED ||
+ peer_device->uuid_flags & UUID_FLAG_GOT_STABLE ||
+ peer_disk_state[OLD] == D_OUTDATED))
+ disk_state[NEW] = peer_disk_state[NEW];
+
+ /* The attempted resync made us D_OUTDATED, roll that back in case */
+ if (repl_state[OLD] == L_WF_BITMAP_T && repl_state[NEW] == L_OFF &&
+ disk_state[NEW] == D_OUTDATED && stable_up_to_date_neighbor(device) &&
+ /* ldev_safe: repl_state[OLD] */ may_be_up_to_date(device, NEW))
+ disk_state[NEW] = D_UP_TO_DATE;
+
+ /* clause intentional here, the D_CONSISTENT form above might trigger this */
+ if (repl_state[OLD] < L_ESTABLISHED && repl_state[NEW] >= L_ESTABLISHED &&
+ disk_state[NEW] == D_CONSISTENT &&
+ /* ldev_safe: repl_state[NEW] */ may_be_up_to_date(device, NEW))
+ disk_state[NEW] = D_UP_TO_DATE;
+
+ /* Follow a neighbor that goes from D_CONSISTENT TO D_UP_TO_DATE */
+ if (disk_state[NEW] == D_CONSISTENT &&
+ peer_disk_state[OLD] == D_CONSISTENT &&
+ peer_disk_state[NEW] == D_UP_TO_DATE &&
+ peer_device->uuid_flags & UUID_FLAG_STABLE)
+ disk_state[NEW] = D_UP_TO_DATE;
+
+ peer_device->uuid_flags &= ~UUID_FLAG_GOT_STABLE;
+
+ uuids_match =
+ (peer_device->current_uuid & ~UUID_PRIMARY) ==
+ (drbd_current_uuid(device) & ~UUID_PRIMARY);
+
+ if (peer_role[OLD] == R_UNKNOWN && peer_role[NEW] == R_PRIMARY &&
+ peer_disk_state[NEW] == D_DISKLESS && disk_state[NEW] >= D_NEGOTIATING) {
+ /* Got connected to a diskless primary */
+ if (uuids_match && !is_sync_target_other_c(peer_device)) {
+ if (device->disk_state[NOW] < D_UP_TO_DATE) {
+ drbd_info(peer_device, "Upgrading local disk to D_UP_TO_DATE since current UUID matches.\n");
+ disk_state[NEW] = D_UP_TO_DATE;
+ }
+ } else {
+ set_bit(TRY_TO_GET_RESYNC, &device->flags);
+ if (disk_state[NEW] == D_UP_TO_DATE) {
+ drbd_info(peer_device, "Downgrading local disk to D_CONSISTENT since current UUID differs.\n");
+ disk_state[NEW] = D_CONSISTENT;
+ /* This is a "safety net"; it can only happen if fencing and quorum
+ are both disabled. This alone would be racy, look for
+ "Do not trust this guy!" (see also may_return_to_up_to_date()) */
+ }
+ }
+ }
+
+ if (connection->agreed_features & DRBD_FF_RS_SKIP_UUID)
+ cond = (have_good_peer || device->exposed_data_uuid) &&
+ peer_disk_state[OLD] != D_UP_TO_DATE &&
+ (device->exposed_data_uuid & ~UUID_PRIMARY) !=
+ (peer_device->current_uuid & ~UUID_PRIMARY);
+ else
+ cond = peer_disk_state[OLD] == D_UNKNOWN &&
+ role[NEW] == R_PRIMARY && !uuids_match;
+
+ if (disk_state[NEW] == D_DISKLESS && peer_disk_state[NEW] == D_UP_TO_DATE &&
+ cond && !test_bit(RECONCILE_INJECT_CUR_UUID, peer_device->flags)) {
+ /* Do not trust this guy!
+ He wants to be D_UP_TO_DATE, but has a different current
+ UUID. Do not accept him as D_UP_TO_DATE but downgrade that to
+ D_CONSISTENT here.
+
+ Exception: a peer we have armed for predecessor-relabel
+ (RECONCILE_INJECT_CUR_UUID) legitimately presents the
+ predecessor UUID (!= our unconfirmed current). We asserted it
+ D_UP_TO_DATE deliberately -- it holds a complete generation and
+ diskless_primary_can_replay_to() gated the arming -- and will
+ relabel it to the current generation post-handshake. Keep it
+ UpToDate so it provides data access and the relabel can fire.
+ */
+ peer_disk_state[NEW] = D_CONSISTENT;
+ }
+ } /* Finish the state clamping before considering consequences. */
+
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+ enum drbd_disk_state *peer_disk_state = peer_device->disk_state;
+ enum drbd_conn_state *cstate = peer_device->connection->cstate;
+
+ /*
+ * Determine whether peer will disable replication due to this transition.
+ *
+ * This matches the condition on the peer below.
+ */
+ if (drbd_change_to_inconsistent(disk_state, cstate) ||
+ (!repl_is_sync_target(repl_state[OLD]) &&
+ repl_is_sync_target(repl_state[NEW])))
+ peer_device->peer_replication[NEW] =
+ test_bit(PEER_REPLICATION_NEXT, peer_device->flags);
+
+ /*
+ * Decide whether to disable replication when the peer
+ * transitions to Inconsistent. Only consider the disk
+ * state when we are Connected because we want to wait
+ * until we know whether replication should be enabled
+ * on the next transition to Inconsistent. This is
+ * communicated with the P_ENABLE_REPLICATION_NEXT
+ * packet.
+ *
+ * Also re-evaluate whether to disable replication when
+ * we become SyncSource, even when the peer's disk was
+ * already Inconsistent. This is relevant when
+ * switching between Ahead-Behind+Inconsistent and
+ * SyncSource-SyncTarget.
+ *
+ * This matches the condition on the peer above.
+ */
+ if (drbd_change_to_inconsistent(peer_disk_state, cstate) ||
+ (!repl_is_sync_source(repl_state[OLD]) &&
+ repl_is_sync_source(repl_state[NEW])))
+ peer_device->replication[NEW] =
+ test_bit(REPLICATION_NEXT, peer_device->flags);
+
+ /*
+ * Not strictly necessary, since "replication" is only
+ * considered when the peer disk is Inconsistent, but
+ * it makes the logs clearer.
+ */
+ if (peer_disk_state[OLD] == D_INCONSISTENT &&
+ peer_disk_state[NEW] != D_INCONSISTENT)
+ peer_device->replication[NEW] = true;
+ }
+
+ /* Always track the real quorum (so reconciliation can use it);
+ * enforce it via have_quorum unless quorum is disabled.
+ */
+ device->quorum[NEW] = calc_quorum(device, NULL);
+ if (resource->res_opts.quorum != QOU_OFF) {
+ device->have_quorum[NEW] = device->quorum[NEW];
+ } else {
+ device->have_quorum[NEW] = true;
+ if (device->quorum[OLD] && !device->quorum[NEW])
+ drbd_info(device, "Would have lost quorum (not enforced, quorum disabled)\n");
+ }
+
+ if (!device->have_quorum[NEW] && disk_state[NEW] == D_UP_TO_DATE &&
+ test_bit(RESTORE_QUORUM, &device->flags)) {
+ device->have_quorum[NEW] = true;
+ set_bit(RESTORING_QUORUM, &device->flags);
+ }
+
+ if (!device->have_quorum[NEW])
+ resource_has_quorum = false;
+
+ /* Suspend IO if we have no accessible data available.
+ * Policy may be extended later to be able to suspend
+ * if redundancy falls below a certain level. */
+ if (role[NEW] == R_PRIMARY && !drbd_data_accessible(device, NEW)) {
+ volumes_have_data_access = false;
+ if (role[OLD] != R_PRIMARY || drbd_data_accessible(device, OLD))
+ volume_lost_data_access = true;
+ }
+ if (role[NEW] == R_PRIMARY && !drbd_data_accessible(device, OLD) &&
+ drbd_data_accessible(device, NEW))
+ volume_gained_data_access = true;
+
+ if (lost_connection && disk_state[NEW] == D_NEGOTIATING)
+ disk_state[NEW] = /* ldev_safe: disk_state */ disk_state_from_md(device);
+
+ if (maybe_crashed_primary && !connected_primaries &&
+ disk_state[NEW] == D_UP_TO_DATE && role[NOW] == R_SECONDARY &&
+ !sole_authority_after_incapable_primary_loss(device))
+ disk_state[NEW] = D_CONSISTENT;
+ }
+ rcu_read_unlock();
+
+ if (volumes_have_data_access) {
+ /* A Primary regaining data access while a member exists that we
+ * cannot reach directly must outdate that far-away member before
+ * resuming I/O: otherwise a diskless Primary's queued writes reach
+ * only the close peer and silently diverge the far-away one. Hold
+ * the resume (keep susp_nod) until the primary-resume 2PC has
+ * outdated the far-away member(s) and released the hold.
+ */
+ if (volume_gained_data_access &&
+ resource->res_opts.on_no_data == OND_SUSPEND_IO &&
+ (resource->members & ~(directly_connected_nodes(resource, NEW) |
+ NODE_MASK(resource->res_opts.node_id))))
+ resource->resume_held_for_outdate[NEW] = true;
+
+ resource->susp_nod[NEW] = resource->resume_held_for_outdate[NEW];
+ }
+ if (volume_lost_data_access && resource->res_opts.on_no_data == OND_SUSPEND_IO)
+ resource->susp_nod[NEW] = true;
+
+ /* Clearing the susp-uuid bridge must not resume into inaccessibility:
+ * the bridged bump attempt may have been REFUSED (the
+ * unconfirmed-generation deferral), leaving no follow-up state
+ * change. The edge conditions above cannot catch that (access was
+ * lost in an earlier state change, and after a demote no volume is
+ * Primary here), so on the falling edge of susp_uuid re-evaluate
+ * data accessibility at the level.
+ */
+ if (resource->susp_uuid[OLD] && !resource->susp_uuid[NEW] &&
+ resource->res_opts.on_no_data == OND_SUSPEND_IO) {
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (!drbd_data_accessible(device, NEW)) {
+ resource->susp_nod[NEW] = true;
+ break;
+ }
+ }
+ }
+
+ resource->susp_quorum[NEW] =
+ resource->res_opts.on_no_quorum == ONQ_SUSPEND_IO ? !resource_has_quorum : false;
+
+ if (!resource->susp_uuid[OLD] &&
+ resource_is_suspended(resource, OLD) && !resource_is_suspended(resource, NEW)) {
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (drbd_gen_obligation_outstanding(device)) {
+ resource->susp_uuid[NEW] = true;
+ break;
+ }
+ }
+ }
+
+ if (role[OLD] == R_PRIMARY && role[NEW] == R_SECONDARY &&
+ (resource->state_change_flags & CS_FS_IGN_OPENERS)) {
+ int rw_count, ro_count;
+
+ drbd_open_counts(resource, &rw_count, &ro_count);
+ if (rw_count)
+ resource->fail_io[NEW] = true;
+ }
+}
+
+void drbd_resume_al(struct drbd_device *device)
+{
+ if (test_and_clear_bit(AL_SUSPENDED, &device->flags))
+ drbd_info(device, "Resumed AL updates\n");
+}
+
+static bool drbd_need_twopc_after_lost_peer(struct drbd_connection *connection)
+{
+ enum drbd_conn_state *cstate = connection->cstate;
+
+ /* Is the state change a disconnect? */
+ if (!(cstate[OLD] == C_CONNECTED && cstate[NEW] < C_CONNECTED))
+ return false;
+
+ /*
+ * The peer did not provide reachable_nodes when disconnecting, so
+ * trigger a twopc ourselves.
+ */
+ if (!(connection->agreed_features & DRBD_FF_2PC_V2))
+ return true;
+
+ /* Trigger a twopc if it was a non-graceful disconnect. */
+ return cstate[NEW] != C_TEAR_DOWN;
+}
+
+static void drbd_schedule_empty_twopc(struct drbd_resource *resource)
+{
+ kref_get(&resource->kref);
+ if (!schedule_work(&resource->empty_twopc)) {
+ kref_put(&resource->kref, drbd_destroy_resource);
+ }
+}
+
+static void drbd_schedule_resume_twopc(struct drbd_resource *resource)
+{
+ kref_get(&resource->kref);
+ if (!schedule_work(&resource->resume_twopc)) {
+ kref_put(&resource->kref, drbd_destroy_resource);
+ }
+}
+
+/* Re-arm the return to UpToDate once no reconcile is owed. The empty twopc
+ * re-checks the hold per device at commit, so it no-ops while still gated and
+ * the next clear re-arms.
+ */
+void drbd_reconcile_settled_try_up_to_date(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ bool any_consistent = false;
+ bool still_pending = false;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (device->disk_state[NOW] == D_CONSISTENT)
+ any_consistent = true;
+ if (reconcile_hold_active(device)) {
+ still_pending = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ if (any_consistent && !still_pending) {
+ set_bit(TRY_BECOME_UP_TO_DATE_PENDING, &resource->flags);
+ drbd_schedule_empty_twopc(resource);
+ }
+}
+
+/*
+ * We cache a node mask of the online members of the cluster. It might
+ * be off because a node is still marked as online immediately after
+ * it crashes. That means it might have an online mark for an already
+ * offline node. On the other hand, we guarantee that it never has
+ * a zero for an online node.
+ */
+static void update_members(struct drbd_resource *resource)
+{
+ enum chg_state_flags flags = resource->state_change_flags;
+ struct twopc_reply *reply = &resource->twopc_reply;
+ const int my_node_id = resource->res_opts.node_id;
+ struct drbd_connection *connection;
+
+ /* in case we initiated 2PC we know the reachable nodes */
+ if (flags & CS_TWOPC && reply->initiator_node_id == my_node_id) {
+ resource->members = reply->reachable_nodes;
+ return;
+ }
+
+ /* In case I am 2PC target of a connect or non-graceful disconnect */
+ for_each_connection(connection, resource) {
+ enum drbd_conn_state *cstate = connection->cstate;
+ const int peer_node_mask = NODE_MASK(connection->peer_node_id);
+
+ /* add a fresh connection to the members */
+ if (cstate[OLD] < C_CONNECTED && cstate[NEW] == C_CONNECTED)
+ resource->members |= peer_node_mask;
+
+ /* Connection to peer lost. Check if we should remove it from the members */
+ if (drbd_need_twopc_after_lost_peer(connection) &&
+ resource->members & peer_node_mask)
+ drbd_schedule_empty_twopc(resource);
+ }
+}
+
+static bool drbd_any_peer_device_up_to_date(struct drbd_connection *connection,
+ enum which_state which)
+{
+ int vnr;
+ struct drbd_peer_device *peer_device;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (peer_device->disk_state[which] == D_UP_TO_DATE)
+ return true;
+ }
+
+ return false;
+}
+
+/* Whether replication is enabled on all peers for this device */
+bool drbd_all_peer_replication(struct drbd_device *device, enum which_state which)
+{
+ struct drbd_peer_device *peer_device;
+ bool all_peer_replication = true;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (!peer_device->peer_replication[which])
+ all_peer_replication = false;
+ }
+ rcu_read_unlock();
+
+ return all_peer_replication;
+}
+
+/* As drbd_all_peer_replication() but takes a state change object */
+static bool drbd_all_peer_replication_change(struct drbd_state_change *state_change, int n_device,
+ enum which_state which)
+{
+ int n_connection;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[
+ n_device * state_change->n_connections + n_connection];
+
+ if (!peer_device_state_change->peer_replication[which])
+ return false;
+ }
+
+ return true;
+}
+
+/* Mark which acks the current flush round waits for. */
+static void arm_flush_masks(struct drbd_resource *resource, enum which_state which)
+{
+ struct drbd_connection *flush_connection;
+ struct drbd_connection *up_to_date_connection;
+
+ for_each_connection(flush_connection, resource) {
+ flush_connection->pending_flush_mask = 0;
+
+ if (flush_connection->cstate[which] != C_CONNECTED)
+ continue;
+
+ if (flush_connection->agreed_pro_version < 123)
+ continue;
+
+ /*
+ * A Secondary owes only its own answer. It shows that the
+ * peer saw the P_ENABLE_REPLICATION sent ahead of the flush,
+ * so a later promotion replicates to us. Its other peers may
+ * not be connected to it, so their answers may never come.
+ */
+ if (flush_connection->peer_role[which] != R_PRIMARY) {
+ flush_connection->pending_flush_mask =
+ NODE_MASK(flush_connection->peer_node_id);
+ continue;
+ }
+
+ /*
+ * A Primary's own answer, diskless or not, shows it processed
+ * our state: none of its later writes misses us.
+ */
+ flush_connection->pending_flush_mask =
+ NODE_MASK(flush_connection->peer_node_id);
+
+ for_each_connection(up_to_date_connection, resource) {
+ u64 up_to_date_mask = NODE_MASK(up_to_date_connection->peer_node_id);
+
+ if (!drbd_any_peer_device_up_to_date(up_to_date_connection, which))
+ continue;
+
+ if (up_to_date_connection->agreed_pro_version < 123)
+ continue;
+
+ flush_connection->pending_flush_mask |= up_to_date_mask;
+ }
+ }
+}
+
+static void drbd_determine_flush_pending(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ struct drbd_connection *flush_connection;
+ struct drbd_connection *up_to_date_connection;
+ int vnr;
+ bool send_flush_requests = false;
+
+ /* Clear any bits if we no longer expect or require a flush ack */
+ spin_lock(&resource->initiator_flush_lock);
+ for_each_connection(flush_connection, resource) {
+ u64 *pending_flush_mask = &flush_connection->pending_flush_mask;
+
+ /*
+ * Clear bits if we no longer expect or require a flush ack due
+ * to loss of connection to the flushed peer.
+ */
+ if (flush_connection->cstate[NEW] != C_CONNECTED) {
+ if (*pending_flush_mask)
+ *pending_flush_mask = 0;
+ continue;
+ }
+
+ /*
+ * Clear bits if we no longer expect or require a flush ack
+ * because the peer that was UpToDate is no longer UpToDate.
+ * For instance, if we lose the connection to that peer.
+ */
+ for_each_connection(up_to_date_connection, resource) {
+ u64 up_to_date_mask = NODE_MASK(up_to_date_connection->peer_node_id);
+
+ if (up_to_date_connection == flush_connection ||
+ drbd_any_peer_device_up_to_date(up_to_date_connection, NEW))
+ continue;
+
+ if (*pending_flush_mask & up_to_date_mask)
+ *pending_flush_mask &= ~up_to_date_mask;
+ }
+ }
+ spin_unlock(&resource->initiator_flush_lock);
+
+ /* Check if we need a new flush */
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device(peer_device, device) {
+ if (!(is_sync_target_state(peer_device, NOW) &&
+ drbd_all_peer_replication(device, NOW)) &&
+ is_sync_target_state(peer_device, NEW) &&
+ drbd_all_peer_replication(device, NEW))
+ send_flush_requests = true;
+ }
+ }
+
+ if (!send_flush_requests)
+ return;
+
+ /* We need a new flush. Mark which acks we are waiting for. */
+ spin_lock(&resource->initiator_flush_lock);
+ resource->current_flush_sequence++;
+ arm_flush_masks(resource, NEW);
+ spin_unlock(&resource->initiator_flush_lock);
+}
+
+/*
+ * A connect between a Primary and another peer committed before that
+ * Primary answered our flush round for itself. Writes it sent the peer on
+ * the new connection may predate its flush point, and an answer the peer
+ * gave on its behalf does not cover them: start a new round.
+ */
+static void drbd_restart_flush_round(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ struct drbd_device *device;
+ bool sync_target = false;
+ u64 flush_sequence, im;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device_rcu(peer_device, device)
+ sync_target |= is_sync_target_state(peer_device, NOW);
+ }
+ rcu_read_unlock();
+ if (!sync_target)
+ return;
+
+ read_lock_irq(&resource->state_rwlock);
+ spin_lock(&resource->initiator_flush_lock);
+ flush_sequence = ++resource->current_flush_sequence;
+ arm_flush_masks(resource, NOW);
+ spin_unlock(&resource->initiator_flush_lock);
+ read_unlock_irq(&resource->state_rwlock);
+
+ drbd_info(resource, "Restarting flush round %llu after a connect of a Primary\n",
+ (unsigned long long)flush_sequence);
+ for_each_connection_ref(connection, im, resource) {
+ if (connection->cstate[NOW] == C_CONNECTED &&
+ connection->agreed_pro_version >= 123)
+ drbd_send_flush_requests(connection, flush_sequence);
+ }
+}
+
+void drbd_flush_connect_prepared(struct drbd_resource *resource, int node_a, int node_b)
+{
+ int node_ids[] = { node_a, node_b };
+ int i;
+
+ spin_lock_irq(&resource->initiator_flush_lock);
+ resource->flush_connect_nodes = 0;
+ resource->flush_connect_sequence = resource->current_flush_sequence;
+ resource->flush_connect_own_missing = false;
+ rcu_read_lock();
+ for (i = 0; i < ARRAY_SIZE(node_ids); i++) {
+ struct drbd_connection *connection;
+
+ connection = drbd_connection_by_node_id(resource, node_ids[i]);
+ if (!connection || connection->agreed_pro_version < 123 ||
+ connection->peer_role[NOW] != R_PRIMARY)
+ continue;
+ resource->flush_connect_nodes |= NODE_MASK(node_ids[i]);
+ if (connection->pending_flush_mask & NODE_MASK(node_ids[i]))
+ resource->flush_connect_own_missing = true;
+ }
+ rcu_read_unlock();
+ spin_unlock_irq(&resource->initiator_flush_lock);
+}
+
+static bool flush_connect_clear(struct drbd_resource *resource)
+{
+ bool needs_restart;
+
+ spin_lock_irq(&resource->initiator_flush_lock);
+ needs_restart = drbd_flush_connect_needs_restart(resource);
+ resource->flush_connect_nodes = 0;
+ spin_unlock_irq(&resource->initiator_flush_lock);
+
+ return needs_restart;
+}
+
+void drbd_flush_connect_committed(struct drbd_resource *resource)
+{
+ if (flush_connect_clear(resource))
+ drbd_restart_flush_round(resource);
+}
+
+static void set_ov_position(struct drbd_peer_device *peer_device,
+ enum drbd_repl_state repl_state)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_bitmap *bm = device->bitmap;
+
+ if (peer_device->connection->agreed_pro_version < 90)
+ peer_device->ov_start_sector = 0;
+ peer_device->rs_total = drbd_bm_bits(device);
+ peer_device->ov_position = 0;
+ if (repl_state == L_VERIFY_T) {
+ /* starting online verify from an arbitrary position
+ * does not fit well into the existing protocol.
+ * on L_VERIFY_T, we initialize ov_left and friends
+ * implicitly in receive_common_data_request once the
+ * first P_OV_REQUEST is received */
+ peer_device->ov_start_sector = ~(sector_t)0;
+ } else {
+ unsigned long bit = bm_sect_to_bit(bm, peer_device->ov_start_sector);
+
+ if (bit >= peer_device->rs_total) {
+ peer_device->ov_start_sector =
+ bm_bit_to_sect(bm, peer_device->rs_total - 1);
+ peer_device->rs_total = 1;
+ } else
+ peer_device->rs_total -= bit;
+ peer_device->ov_position = peer_device->ov_start_sector;
+ }
+ atomic64_set(&peer_device->ov_left, peer_device->rs_total);
+ peer_device->ov_skipped = 0;
+}
+
+static void initialize_resync_progress_marks(struct drbd_peer_device *peer_device)
+{
+ unsigned long tw = drbd_bm_total_weight(peer_device);
+ unsigned long now = jiffies;
+ int i;
+
+ peer_device->rs_last_progress_report_ts = now;
+ for (i = 0; i < DRBD_SYNC_MARKS; i++) {
+ peer_device->rs_mark_left[i] = tw;
+ peer_device->rs_mark_time[i] = now;
+ }
+}
+
+static void initialize_resync(struct drbd_peer_device *peer_device)
+{
+ unsigned long tw = drbd_bm_total_weight(peer_device);
+ unsigned long now = jiffies;
+
+ peer_device->last_in_sync_end = 0;
+ peer_device->resync_next_bit = 0;
+ peer_device->last_resync_pass_bits = tw;
+ peer_device->rs_failed = 0;
+ peer_device->rs_paused = 0;
+ peer_device->rs_same_csum = 0;
+ peer_device->rs_total = tw;
+ peer_device->rs_start = now;
+ peer_device->rs_last_writeout = now;
+ initialize_resync_progress_marks(peer_device);
+ drbd_rs_controller_reset(peer_device);
+}
+
+/* Is there a primary with access to up to date data known */
+static bool primary_and_data_present(struct drbd_device *device)
+{
+ bool up_to_date_data = device->disk_state[NEW] == D_UP_TO_DATE;
+ struct drbd_resource *resource = device->resource;
+ bool primary = resource->role[NEW] == R_PRIMARY;
+ bool readable_peer_data = false, peer_data = false;
+ struct drbd_peer_device *peer_device;
+ bool remote_primary = false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_net_conf *nc;
+
+ /* Do not consider the peer if we are disconnecting. */
+ if (resource->remote_state_change &&
+ drbd_twopc_between_peer_and_me(connection) &&
+ resource->twopc_reply.is_disconnect)
+ continue;
+
+ if (connection->peer_role[NEW] == R_PRIMARY)
+ primary = remote_primary = true;
+
+ if (peer_device->disk_state[NEW] != D_UP_TO_DATE)
+ continue;
+
+ peer_data = true;
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (!nc || nc->allow_remote_read)
+ readable_peer_data = true;
+ }
+ rcu_read_unlock();
+
+ /*
+ * A remote primary writes on data it can reach itself. If this node is
+ * the only primary, only peer data it may read keeps the cluster going.
+ */
+ up_to_date_data |= remote_primary ? peer_data : readable_peer_data;
+
+ return primary && up_to_date_data;
+}
+
+static bool extra_ldev_ref_for_after_state_chg(enum drbd_disk_state *disk_state)
+{
+ return (disk_state[OLD] != D_FAILED && disk_state[NEW] == D_FAILED) ||
+ (disk_state[OLD] != D_DETACHING && disk_state[NEW] == D_DETACHING) ||
+ (disk_state[OLD] != D_DISKLESS && disk_state[NEW] == D_DISKLESS);
+}
+
+static bool has_starting_resyncs(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (peer_device->repl_state[NEW] > L_ESTABLISHED)
+ return true;
+ }
+ return false;
+}
+
+static bool should_try_become_up_to_date(struct drbd_device *device, enum drbd_disk_state *disk_state,
+ enum which_state which)
+{
+ return disk_state[OLD] == D_UP_TO_DATE && disk_state[NEW] == D_CONSISTENT &&
+ may_return_to_up_to_date(device, which);
+}
+
+/* Set or clear a bit atomically; return true if it changed. */
+static bool test_and_assign_bit(int nr, unsigned long *addr, bool value)
+{
+ return value ? !test_and_set_bit(nr, addr) : test_and_clear_bit(nr, addr);
+}
+
+/* Bring the peer's meta-data flags in line with the new state. */
+static void update_peer_md_flags(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ unsigned long *mdf = &device->ldev->md.peers[peer_device->node_id].flags;
+ enum drbd_disk_state pdsk = peer_device->disk_state[NEW];
+ bool changed;
+
+ changed = test_and_assign_bit(__MDF_PEER_CONNECTED, mdf,
+ peer_device->repl_state[NEW] > L_OFF);
+ changed |= test_and_assign_bit(__MDF_PEER_OUTDATED, mdf,
+ pdsk >= D_INCONSISTENT && pdsk <= D_OUTDATED);
+ changed |= test_and_assign_bit(__MDF_PEER_FENCING, mdf,
+ peer_device->connection->fencing_policy != FP_DONT_CARE);
+
+ /* Do NOT clear MDF_PEER_DEVICE_SEEN whenever the peer disk is gone. We
+ * want to be able to refuse a resize beyond "last agreed" size, even if
+ * the peer is currently detached.
+ */
+ if (pdsk >= D_INCONSISTENT && pdsk != D_UNKNOWN)
+ changed |= !test_and_set_bit(__MDF_PEER_DEVICE_SEEN, mdf);
+ else if (pdsk == D_DISKLESS && !want_bitmap(peer_device))
+ changed |= test_and_clear_bit(__MDF_PEER_DEVICE_SEEN, mdf);
+
+ if (changed)
+ drbd_md_mark_dirty(device);
+}
+
+/**
+ * finish_state_change - carry out actions triggered by a state change
+ * @resource: DBRD resource.
+ * @tag: State change tag to print in status messages.
+ */
+static void finish_state_change(struct drbd_resource *resource, const char *tag)
+{
+ enum drbd_role *role = resource->role;
+ struct drbd_device *device;
+ struct drbd_connection *connection;
+ bool starting_resync = false;
+ bool reconciliation_resync_done = false;
+ bool start_new_epoch = false;
+ bool lost_a_primary_peer = false;
+ bool some_peer_is_primary = false;
+ bool some_peer_request_in_flight = false;
+ bool resource_suspended[2];
+ bool unfreeze_io = false;
+ int vnr;
+
+ print_state_change(resource, "", tag);
+
+ resource_suspended[OLD] = resource_is_suspended(resource, OLD);
+ resource_suspended[NEW] = resource_is_suspended(resource, NEW);
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ bool *have_quorum = device->have_quorum;
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
+ bool did, should;
+
+ did = drbd_should_do_remote(peer_device, NOW);
+ should = drbd_should_do_remote(peer_device, NEW);
+
+ if (!did && should) {
+ /* Since "did" is false, the request with this
+ * dagtag and prior requests were not be marked
+ * to be sent to this peer. Hence this will not
+ * send a dagtag packet before the
+ * corresponding data packet.
+ *
+ * It is possible that this peer does not
+ * actually have the data corresponding to this
+ * dagtag. However in that case, the disk state
+ * of that peer will not be D_UP_TO_DATE, so it
+ * not be relevant what dagtag we have sent it. */
+ connection->send_dagtag = resource->dagtag_sector;
+ drbd_queue_work_if_unqueued(
+ &connection->sender_work,
+ &connection->send_dagtag_work);
+ }
+
+ if (did != should)
+ start_new_epoch = true;
+
+ if (peer_device->repl_state[OLD] != L_WF_BITMAP_S &&
+ peer_device->repl_state[NEW] == L_WF_BITMAP_S)
+ clear_bit(B_RS_H_DONE, peer_device->flags);
+
+ if (peer_device->repl_state[OLD] != L_WF_BITMAP_T &&
+ peer_device->repl_state[NEW] == L_WF_BITMAP_T)
+ clear_bit(B_RS_H_DONE, peer_device->flags);
+
+ if (!is_sync_state(peer_device, NOW) &&
+ is_sync_state(peer_device, NEW)) {
+ clear_bit(RS_DONE, peer_device->flags);
+ clear_bit(B_RS_H_DONE, peer_device->flags);
+ clear_bit(SYNC_TARGET_TO_BEHIND, peer_device->flags);
+ }
+ }
+
+ if (role[NEW] == R_PRIMARY && !have_quorum[NEW])
+ set_bit(PRIMARY_LOST_QUORUM, &device->flags);
+ }
+ if (start_new_epoch)
+ start_new_tl_epoch(resource);
+
+ spin_lock(&resource->peer_ack_lock);
+ if (role[OLD] == R_PRIMARY && role[NEW] == R_SECONDARY && resource->peer_ack_req) {
+ resource->last_peer_acked_dagtag = resource->peer_ack_req->dagtag_sector;
+ drbd_queue_peer_ack(resource, resource->peer_ack_req);
+ resource->peer_ack_req = NULL;
+ }
+ spin_unlock(&resource->peer_ack_lock);
+
+ drbd_determine_flush_pending(resource);
+
+ if (!resource->fail_io[OLD] && resource->fail_io[NEW])
+ drbd_warn(resource, "Failing IOs\n");
+
+ for_each_connection(connection, resource) {
+ enum drbd_role *peer_role = connection->peer_role;
+ enum drbd_conn_state *cstate = connection->cstate;
+
+ if (peer_role[NEW] == R_PRIMARY)
+ some_peer_is_primary = true;
+
+ switch (cstate[NEW]) {
+ case C_CONNECTED:
+ if (atomic_read(&connection->active_ee_cnt)
+ || atomic_read(&connection->done_ee_cnt))
+ some_peer_request_in_flight = true;
+ break;
+ case C_STANDALONE:
+ case C_UNCONNECTED:
+ case C_CONNECTING:
+ /* maybe others are safe as well? which ones? */
+ break;
+ default:
+ /* if we just disconnected, there may still be some request in flight. */
+ some_peer_request_in_flight = true;
+ }
+
+ if (some_peer_is_primary && some_peer_request_in_flight)
+ break;
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+ enum drbd_disk_state *disk_state = device->disk_state;
+ u16 gen_obl_reasons = 0;
+
+ if (test_bit(RESTORING_QUORUM, &device->flags) &&
+ !device->have_quorum[OLD] && device->have_quorum[NEW]) {
+ clear_bit(RESTORING_QUORUM, &device->flags);
+ drbd_info(resource, "Restored quorum from before reboot\n");
+ }
+
+ if (test_bit(RESTORE_QUORUM, &device->flags) &&
+ (device->have_quorum[NEW] || disk_state[NEW] < D_UP_TO_DATE))
+ clear_bit(RESTORE_QUORUM, &device->flags);
+
+ /* if we are going -> D_FAILED or D_DISKLESS, grab one extra reference
+ * on the ldev here, to be sure the transition -> D_DISKLESS resp.
+ * drbd_ldev_destroy() won't happen before our corresponding
+ * w_after_state_change works run, where we put_ldev again. */
+ if (extra_ldev_ref_for_after_state_chg(disk_state))
+ atomic_inc(&device->local_cnt);
+
+ if (disk_state[OLD] != D_DISKLESS && disk_state[NEW] == D_DISKLESS) {
+ /* who knows if we are ever going to be attached again,
+ * and whether that will be the same device, or a newly
+ * initialized one. */
+ for_each_peer_device(peer_device, device)
+ peer_device->bitmap_index = -1;
+ }
+
+ /* ldev_safe: transitioning from D_ATTACHING, ldev just established */
+ if (disk_state[OLD] == D_ATTACHING && disk_state[NEW] >= D_NEGOTIATING)
+ drbd_info(device, "attached to current UUID: %016llX\n", device->ldev->md.current_uuid);
+
+ for_each_peer_device(peer_device, device) {
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+ enum drbd_disk_state *peer_disk_state = peer_device->disk_state;
+ struct drbd_connection *connection = peer_device->connection;
+ enum drbd_role *peer_role = connection->peer_role;
+
+ if (repl_state[OLD] <= L_ESTABLISHED && repl_state[NEW] == L_WF_BITMAP_S)
+ starting_resync = true;
+
+ if (peer_disk_state[OLD] < D_NEGOTIATING &&
+ peer_disk_state[NEW] == D_NEGOTIATING)
+ set_bit(SEND_UUIDS_PEER_ATTACH, peer_device->flags);
+
+ if ((disk_state[OLD] != D_UP_TO_DATE || peer_disk_state[OLD] != D_UP_TO_DATE) &&
+ (disk_state[NEW] == D_UP_TO_DATE && peer_disk_state[NEW] == D_UP_TO_DATE)) {
+ clear_bit(CRASHED_PRIMARY, &device->flags);
+ if (test_bit(UUIDS_RECEIVED, peer_device->flags))
+ peer_device->uuid_flags &= ~((u64)UUID_FLAG_CRASHED_PRIMARY);
+ }
+
+ /* Aborted verify run, or we reached the stop sector.
+ * Log the last position, unless end-of-device. */
+ if ((repl_state[OLD] == L_VERIFY_S || repl_state[OLD] == L_VERIFY_T) &&
+ repl_state[NEW] <= L_ESTABLISHED) {
+ /* ldev_safe: repl_state[OLD] */
+ struct drbd_bitmap *bm = device->bitmap;
+ unsigned long ov_left = atomic64_read(&peer_device->ov_left);
+
+ /* ldev_safe: repl_state[OLD] */
+ peer_device->ov_start_sector =
+ bm_bit_to_sect(bm, drbd_bm_bits(device) - ov_left);
+ if (ov_left)
+ drbd_info(peer_device, "Online Verify reached sector %llu\n",
+ (unsigned long long)peer_device->ov_start_sector);
+ }
+
+ if ((repl_state[OLD] == L_PAUSED_SYNC_T || repl_state[OLD] == L_PAUSED_SYNC_S) &&
+ (repl_state[NEW] == L_SYNC_TARGET || repl_state[NEW] == L_SYNC_SOURCE)) {
+ drbd_info(peer_device, "Syncer continues.\n");
+ peer_device->rs_paused += (long)jiffies
+ -(long)peer_device->rs_mark_time[peer_device->rs_last_mark];
+ initialize_resync_progress_marks(peer_device);
+ peer_device->resync_next_bit = 0;
+ }
+
+ if ((repl_state[OLD] == L_SYNC_TARGET || repl_state[OLD] == L_SYNC_SOURCE) &&
+ (repl_state[NEW] == L_PAUSED_SYNC_T || repl_state[NEW] == L_PAUSED_SYNC_S)) {
+ drbd_info(peer_device, "Resync suspended\n");
+ peer_device->rs_mark_time[peer_device->rs_last_mark] = jiffies;
+ }
+
+
+ if (repl_state[OLD] > L_ESTABLISHED && repl_state[NEW] <= L_ESTABLISHED) {
+ if (test_and_clear_bit(RECONCILIATION_RESYNC, peer_device->flags))
+ reconciliation_resync_done = true;
+ /* dagtag sender as reconcile target: it must receive the
+ * data first, so discharge on resync end (as source it
+ * discharged at SyncSource onset, below).
+ */
+ if (test_and_clear_bit(RECONCILE_PENDING, peer_device->flags))
+ reconciliation_resync_done = true;
+ }
+
+ /* dagtag sender as reconcile source: it is the holder, so
+ * discharge as soon as it starts sourcing -- no need to wait
+ * for the peer to finish catching up.
+ */
+ if (!repl_is_sync_source(repl_state[OLD]) &&
+ repl_is_sync_source(repl_state[NEW]) &&
+ test_and_clear_bit(RECONCILE_PENDING, peer_device->flags))
+ reconciliation_resync_done = true;
+
+ if (repl_state[OLD] >= L_ESTABLISHED && repl_state[NEW] < L_ESTABLISHED)
+ clear_bit(AHEAD_TO_SYNC_SOURCE, peer_device->flags);
+
+ if (repl_state[OLD] == L_ESTABLISHED &&
+ (repl_state[NEW] == L_VERIFY_S || repl_state[NEW] == L_VERIFY_T)) {
+ unsigned long now = jiffies;
+ int i;
+
+ /* ldev_safe: repl_state[NEW] */
+ set_ov_position(peer_device, repl_state[NEW]);
+ peer_device->rs_start = now;
+ peer_device->ov_last_oos_size = 0;
+ peer_device->ov_last_oos_start = 0;
+ peer_device->ov_last_skipped_size = 0;
+ peer_device->ov_last_skipped_start = 0;
+ peer_device->rs_last_writeout = now;
+ peer_device->rs_last_progress_report_ts = now;
+ for (i = 0; i < DRBD_SYNC_MARKS; i++) {
+ peer_device->rs_mark_left[i] = peer_device->rs_total;
+ peer_device->rs_mark_time[i] = now;
+ }
+
+ drbd_rs_controller_reset(peer_device);
+ } else if (!(repl_state[OLD] >= L_SYNC_SOURCE && repl_state[OLD] <= L_PAUSED_SYNC_T) &&
+ (repl_state[NEW] >= L_SYNC_SOURCE && repl_state[NEW] <= L_PAUSED_SYNC_T)) {
+ initialize_resync(peer_device);
+ }
+
+ if (disk_state[NEW] != D_NEGOTIATING && get_ldev(device)) {
+ if (peer_device->bitmap_index != -1)
+ update_peer_md_flags(peer_device);
+
+ /* Peer was forced D_UP_TO_DATE & R_PRIMARY, consider to resync */
+ if (disk_state[OLD] == D_INCONSISTENT &&
+ peer_disk_state[OLD] == D_INCONSISTENT && peer_disk_state[NEW] == D_UP_TO_DATE &&
+ peer_role[OLD] == R_SECONDARY && peer_role[NEW] == R_PRIMARY)
+ set_bit(CONSIDER_RESYNC, peer_device->flags);
+
+ /* Resume AL writing if we get a connection */
+ if (repl_state[OLD] < L_ESTABLISHED && repl_state[NEW] >= L_ESTABLISHED)
+ drbd_resume_al(device);
+ put_ldev(device);
+ }
+
+ if (repl_state[OLD] == L_AHEAD && repl_state[NEW] == L_SYNC_SOURCE) {
+ set_bit(SEND_STATE_AFTER_AHEAD, peer_device->flags);
+ set_bit(SEND_STATE_AFTER_AHEAD_C, &connection->flags);
+
+ clear_bit(CONN_CONGESTED, &connection->flags);
+ wake_up(&connection->sender_work.q_wait);
+ }
+
+ /* We start writing locally without replicating the changes,
+ * better start a new data generation */
+ if (repl_state[OLD] != L_AHEAD && repl_state[NEW] == L_AHEAD)
+ gen_obl_reasons |= GEN_OBL_AHEAD;
+
+ if (lost_contact_to_peer_data(peer_disk_state)) {
+ if (role[NEW] == R_PRIMARY &&
+ !test_bit(UNREGISTERED, &device->flags) &&
+ (drbd_data_accessible(device, OLD) ||
+ drbd_data_accessible(device, NEW)))
+ gen_obl_reasons |= GEN_OBL_PEER_DATA_LOST;
+
+ if (connection->agreed_pro_version < 110 &&
+ peer_role[NEW] == R_PRIMARY &&
+ disk_state[NEW] >= D_UP_TO_DATE)
+ gen_obl_reasons |= GEN_OBL_PRE_110;
+ }
+ if (peer_returns_diskless(peer_device, peer_disk_state[OLD], peer_disk_state[NEW])) {
+ if (role[NEW] == R_PRIMARY && !test_bit(UNREGISTERED, &device->flags) &&
+ disk_state[NEW] == D_UP_TO_DATE)
+ gen_obl_reasons |= GEN_OBL_PEER_RETURNED_DISKLESS;
+ }
+
+ if (disk_state[OLD] > D_FAILED && disk_state[NEW] == D_FAILED &&
+ role[NEW] == R_PRIMARY && drbd_data_accessible(device, NEW))
+ gen_obl_reasons |= GEN_OBL_OWN_DISK_FAILED;
+
+ if (peer_disk_state[NEW] < D_UP_TO_DATE &&
+ test_bit(GOT_NEG_ACK, peer_device->flags))
+ clear_bit(GOT_NEG_ACK, peer_device->flags);
+
+ if (repl_state[OLD] > L_ESTABLISHED && repl_state[NEW] <= L_ESTABLISHED)
+ clear_bit(SYNC_SRC_CRASHED_PRI, peer_device->flags);
+
+ /* disk_state: CRASHED_PRIMARY changes with it */
+ if (peer_role[OLD] != peer_role[NEW] || role[OLD] != role[NEW] ||
+ peer_disk_state[OLD] != peer_disk_state[NEW] ||
+ disk_state[OLD] != disk_state[NEW])
+ drbd_update_mdf_al_disabled(device, NEW);
+ }
+
+ if (disk_state[OLD] >= D_INCONSISTENT && disk_state[NEW] < D_INCONSISTENT &&
+ role[NEW] == R_PRIMARY && drbd_data_accessible(device, NEW))
+ gen_obl_reasons |= GEN_OBL_OWN_DISK_FAILED;
+
+ if (role[OLD] == R_SECONDARY && role[NEW] == R_PRIMARY)
+ gen_obl_reasons |= GEN_OBL_PROMOTED;
+
+ if (gen_obl_reasons)
+ drbd_gen_obligation_arm(device, gen_obl_reasons);
+
+ if (disk_state[NEW] != D_NEGOTIATING && get_ldev_if_state(device, D_DETACHING)) {
+ u32 mdf = device->ldev->md.flags;
+ bool graceful_detach = disk_state[NEW] == D_DETACHING && !test_bit(FORCE_DETACH, &device->flags);
+
+ /* For now, always require a drbdmeta apply-al run,
+ * even if that ends up only re-initializing the AL */
+ mdf &= ~MDF_AL_CLEAN;
+ /* reset some flags to what we know now */
+ mdf &= ~MDF_CRASHED_PRIMARY;
+ if (test_bit(CRASHED_PRIMARY, &device->flags) ||
+ (role[NEW] == R_PRIMARY && !graceful_detach))
+ mdf |= MDF_CRASHED_PRIMARY;
+ mdf &= ~MDF_PRIMARY_LOST_QUORUM;
+ if (test_bit(PRIMARY_LOST_QUORUM, &device->flags))
+ mdf |= MDF_PRIMARY_LOST_QUORUM;
+ /* Do not touch MDF_CONSISTENT if we are D_FAILED */
+ if (disk_state[NEW] >= D_INCONSISTENT) {
+ mdf &= ~(MDF_CONSISTENT | MDF_WAS_UP_TO_DATE);
+
+ if (disk_state[NEW] > D_INCONSISTENT)
+ mdf |= MDF_CONSISTENT;
+ if (disk_state[NEW] > D_OUTDATED)
+ mdf |= MDF_WAS_UP_TO_DATE;
+ } else if ((disk_state[NEW] == D_FAILED || disk_state[NEW] == D_DETACHING) &&
+ mdf & MDF_WAS_UP_TO_DATE &&
+ primary_and_data_present(device)) {
+ /* There are cases when we still can update meta-data even if disk
+ state is failed.... Clear MDF_WAS_UP_TO_DATE if appropriate */
+ mdf &= ~MDF_WAS_UP_TO_DATE;
+ }
+
+/*
+ * MDF_PRIMARY_IND IS set: apply activity log after crash
+ * MDF_PRIMARY_IND NOT set: do not apply, forget and re-initialize activity log after crash.
+ * We want the MDF_PRIMARY_IND set *always* before our backend could possibly
+ * be target of write requests, whether we are Secondary or Primary ourselves.
+ *
+ * We want to avoid to clear that flag just because we lost the connection to a
+ * detached Primary, but before all in-flight IO was drained, because we may
+ * have some dirty bits not yet persisted.
+ *
+ * We want it cleared only once we are *certain* that we no longer see any Primary,
+ * are not Primary ourselves, AND all previously received WRITE (peer-) requests
+ * have been processed, NOTHING is in flight against our backend anymore,
+ * AND we have successfully written out any dirty bitmap pages.
+ *
+ *
+ * MDF_PEER_DEVICE_SEEN ... The peer had a backing device at some point
+ * MDF_NODE_EXISTS ... We have seen evidence that this node exists in the cluster.
+ * Note: This bit does **not** get set when a new peer/connection is created with
+ * `drbdsetup new-peer ...`. The bit gets set when we establish a connection
+ * successfully for the first time or we learn via other nodes about the
+ * existence.
+ */
+
+ /* set, if someone is/becomes primary */
+ if (role[NEW] == R_PRIMARY || some_peer_is_primary)
+ mdf |= MDF_PRIMARY_IND;
+ /* clear, if */
+ else if (/* NO peer requests in flight, AND */
+ !some_peer_request_in_flight &&
+ (graceful_detach ||
+ /* or everyone secondary ... */
+ (role[NEW] == R_SECONDARY && !some_peer_is_primary &&
+ /* ... and not detaching because of IO error. */
+ disk_state[NEW] >= D_INCONSISTENT)))
+ mdf &= ~MDF_PRIMARY_IND;
+
+ if (device->have_quorum[NEW])
+ mdf |= MDF_HAVE_QUORUM;
+ else
+ mdf &= ~MDF_HAVE_QUORUM;
+ /* apply changed flags to md.flags,
+ * and "schedule" for write-out */
+ if (mdf != device->ldev->md.flags ||
+ device->ldev->md.members != resource->members) {
+ device->ldev->md.flags = mdf;
+ device->ldev->md.members = resource->members;
+ drbd_md_mark_dirty(device);
+ }
+ if (disk_state[OLD] < D_CONSISTENT && disk_state[NEW] >= D_CONSISTENT)
+ drbd_uuid_set_exposed(device, device->ldev->md.current_uuid, true);
+ put_ldev(device);
+ }
+
+ /* remember last attach time so request_timer_fn() won't
+ * kill newly established sessions while we are still trying to thaw
+ * previously frozen IO */
+ if ((disk_state[OLD] == D_ATTACHING || disk_state[OLD] == D_NEGOTIATING) &&
+ disk_state[NEW] > D_NEGOTIATING)
+ device->last_reattach_jif = jiffies;
+
+ if (!device->have_quorum[OLD] && device->have_quorum[NEW])
+ clear_bit(PRIMARY_LOST_QUORUM, &device->flags);
+
+ if (resource_suspended[NEW] &&
+ !(role[OLD] == R_PRIMARY && !drbd_data_accessible(device, OLD)) &&
+ (role[NEW] == R_PRIMARY && !drbd_data_accessible(device, NEW)) &&
+ resource->res_opts.on_no_data == OND_IO_ERROR)
+ unfreeze_io = true;
+
+ if (!resource->fail_io[OLD] && resource->fail_io[NEW])
+ unfreeze_io = true;
+
+ /* An idle promote/demote round trip creates no data generation,
+ * however many peers were lost in between, so a latent
+ * obligation is void here. A materialized one is about writes
+ * that already completed and only reaches a demote if its mint
+ * failed: keep it and dispatch a retry. This runs inside the
+ * state-commit critical section, so it cannot mint itself.
+ */
+ if (role[OLD] == R_PRIMARY && role[NEW] == R_SECONDARY) {
+ if (!drbd_gen_obligation_materialized(device))
+ drbd_gen_obligation_void(device);
+ else if (drbd_gen_obligation_mint_start(device))
+ drbd_device_post_work(device, MAKE_NEW_CUR_UUID);
+ }
+
+ if (should_try_become_up_to_date(device, disk_state, NEW))
+ set_bit(TRY_BECOME_UP_TO_DATE_PENDING, &resource->flags);
+
+ if (disk_state[OLD] != D_DISKLESS && disk_state[NEW] == D_DISKLESS)
+ clear_bit(FORCE_DETACH, &device->flags);
+ }
+
+ for_each_connection(connection, resource) {
+ enum drbd_conn_state *cstate = connection->cstate;
+ enum drbd_role *peer_role = connection->peer_role;
+
+ /*
+ * If we lose connection to a Primary node then we need to
+ * inform our peers so that we can potentially do a
+ * reconciliation resync. The function conn_disconnect()
+ * informs the peers. So we must set the flag before stopping
+ * the receiver.
+ */
+ if (cstate[OLD] == C_CONNECTED && cstate[NEW] < C_CONNECTED &&
+ peer_role[OLD] == R_PRIMARY)
+ set_bit(NOTIFY_PEERS_LOST_PRIMARY, &connection->flags);
+
+ /* Receiver should clean up itself */
+ if (cstate[OLD] != C_DISCONNECTING && cstate[NEW] == C_DISCONNECTING)
+ drbd_thread_stop_nowait(&connection->receiver);
+
+ /* Now the receiver finished cleaning up itself, it should die */
+ if (cstate[OLD] != C_STANDALONE && cstate[NEW] == C_STANDALONE)
+ drbd_thread_stop_nowait(&connection->receiver);
+
+ /* Upon network failure, we need to restart the receiver. */
+ if (cstate[OLD] >= C_CONNECTING &&
+ cstate[NEW] <= C_TEAR_DOWN && cstate[NEW] >= C_TIMEOUT)
+ drbd_thread_restart_nowait(&connection->receiver);
+
+ /* remember last connect time so request_timer_fn() won't
+ * kill newly established sessions while we are still trying to thaw
+ * previously frozen IO */
+ if (cstate[OLD] < C_CONNECTED && cstate[NEW] == C_CONNECTED)
+ connection->last_reconnect_jif = jiffies;
+
+ /* A dry run that got this far had no resync handshake to
+ * report on (a diskless side or no volumes); the connection is
+ * a real one now. With protocol < 118 the handshake follows
+ * C_CONNECTED, so the dry run is still pending there.
+ */
+ if (cstate[OLD] < C_CONNECTED && cstate[NEW] == C_CONNECTED &&
+ connection->agreed_pro_version >= 118) {
+ clear_bit(CONN_DRY_RUN, &connection->flags);
+ clear_bit(CONN_DRY_RUN_LOCAL, &connection->flags);
+ }
+
+ if (resource_suspended[OLD]) {
+ enum drbd_req_event walk_event = -1;
+
+ /* If we resume IO without this connection, then we
+ * need to cancel suspended requests. */
+ if ((!resource_suspended[NEW] || unfreeze_io) && cstate[NEW] < C_CONNECTED)
+ walk_event = CANCEL_SUSPENDED_IO;
+ /* On reconnection when we have been suspended we need
+ * to process suspended requests. If there are resyncs,
+ * that means that it was not a simple disconnect and
+ * reconnect, so we cannot resend. We must cancel
+ * instead. */
+ else if (cstate[OLD] < C_CONNECTED && cstate[NEW] == C_CONNECTED)
+ walk_event = has_starting_resyncs(connection) ? CANCEL_SUSPENDED_IO : RESEND;
+
+ if (walk_event != -1)
+ __tl_walk(resource, connection, &connection->req_not_net_done, walk_event);
+
+ /* The sender finds the requests to resend when it
+ * rescans the transfer log. ___end_state_change() wakes
+ * it for that, once cstate[NOW] says it may send.
+ */
+ }
+
+ if (cstate[OLD] == C_CONNECTED && cstate[NEW] < C_CONNECTED)
+ set_bit(RECONNECT, &connection->flags);
+
+ if (starting_resync && peer_role[NEW] == R_PRIMARY)
+ apply_unacked_peer_requests(connection);
+
+ if (peer_role[OLD] == R_PRIMARY && peer_role[NEW] == R_UNKNOWN)
+ lost_a_primary_peer = true;
+ }
+
+ if (lost_a_primary_peer) {
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device(peer_device, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NEW];
+
+ if (!test_bit(UNSTABLE_RESYNC, peer_device->flags) &&
+ (repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T) &&
+ !(peer_device->uuid_flags & UUID_FLAG_STABLE) &&
+ !drbd_stable_sync_source_present(peer_device, NEW))
+ set_bit(UNSTABLE_RESYNC, peer_device->flags);
+ }
+ }
+ }
+
+ /* The completion-resumed walk may drop a postponed request's last
+ * completion reference; drbd_suspended() still reads the old,
+ * suspended state here, so that request parks on suspended_reqs.
+ * Restart the suspended requests after the walk, or it is missed.
+ */
+ if ((resource_suspended[OLD] && !resource_suspended[NEW]) || unfreeze_io)
+ __tl_walk(resource, NULL, NULL, COMPLETION_RESUMED);
+
+ if (resource_suspended[OLD] && !resource_suspended[NEW])
+ drbd_restart_suspended_reqs(resource);
+
+ /* reconcile settled: a held-Consistent survivor may return to UpToDate */
+ if (reconciliation_resync_done)
+ drbd_reconcile_settled_try_up_to_date(resource);
+}
+
+static void abw_start_sync(struct drbd_device *device,
+ struct drbd_peer_device *peer_device, int rv)
+{
+ struct drbd_peer_device *pd;
+ enum drbd_state_rv srv;
+
+ if (rv) {
+ drbd_err(device, "Writing the bitmap failed not starting resync.\n");
+ srv = stable_change_repl_state(peer_device, L_ESTABLISHED, CS_VERBOSE,
+ "start-sync");
+ if (srv < SS_SUCCESS)
+ drbd_err(peer_device,
+ "Leaving %s failed (%s); disconnect to recover.\n",
+ drbd_repl_str(peer_device->repl_state[NOW]),
+ drbd_set_st_err_str(srv));
+ return;
+ }
+
+ switch (peer_device->repl_state[NOW]) {
+ case L_STARTING_SYNC_T:
+ /* Since the number of set bits changed and the other peer_devices are
+ lready in L_PAUSED_SYNC_T state, we need to set rs_total here */
+ rcu_read_lock();
+ for_each_peer_device_rcu(pd, device)
+ initialize_resync(pd);
+ rcu_read_unlock();
+
+ if (peer_device->connection->agreed_pro_version < 110) {
+ srv = stable_change_repl_state(peer_device, L_WF_SYNC_UUID,
+ CS_VERBOSE, "start-sync");
+ if (srv < SS_SUCCESS)
+ drbd_err(peer_device, "Not starting resync (%s)\n",
+ drbd_set_st_err_str(srv));
+ } else {
+ drbd_start_resync(peer_device, L_SYNC_TARGET, "start-sync");
+ }
+ break;
+ case L_STARTING_SYNC_S:
+ drbd_start_resync(peer_device, L_SYNC_SOURCE, "start-sync");
+ break;
+ default:
+ break;
+ }
+}
+
+int drbd_bitmap_io_from_worker(struct drbd_device *device,
+ int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
+ char *why, enum bm_flag flags,
+ struct drbd_peer_device *peer_device)
+{
+ int rv;
+
+ D_ASSERT(device, current == device->resource->worker.task);
+
+ if (!device->bitmap)
+ return 0;
+
+ /* open coded non-blocking drbd_suspend_io(device); */
+ atomic_inc(&device->suspend_cnt);
+
+ if (flags & BM_LOCK_SINGLE_SLOT)
+ drbd_bm_slot_lock(peer_device, why, flags);
+ else
+ drbd_bm_lock(device, why, flags);
+ rv = io_fn(device, peer_device);
+ if (flags & BM_LOCK_SINGLE_SLOT)
+ drbd_bm_slot_unlock(peer_device);
+ else
+ drbd_bm_unlock(device);
+
+ drbd_resume_io(device);
+
+ return rv;
+}
+
+static bool state_change_is_susp_fen(struct drbd_state_change *state_change,
+ enum which_state which)
+{
+ int n_connection;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change =
+ &state_change->connections[n_connection];
+
+ if (connection_state_change->susp_fen[which])
+ return true;
+ }
+
+ return false;
+}
+
+static bool state_change_is_susp_quorum(struct drbd_state_change *state_change,
+ enum which_state which)
+{
+ struct drbd_resource *resource = state_change->resource[0].resource;
+ int n_device;
+
+ if (resource->res_opts.on_no_quorum != ONQ_SUSPEND_IO)
+ return false;
+
+ for (n_device = 0; n_device < state_change->n_devices; n_device++) {
+ struct drbd_device_state_change *device_state_change =
+ &state_change->devices[n_device];
+
+ if (!device_state_change->have_quorum[which])
+ return true;
+ }
+
+ return false;
+}
+
+static bool resync_susp_comb_dep_sc(struct drbd_state_change *state_change,
+ unsigned int n_device, int n_connection,
+ enum which_state which)
+{
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[n_device * state_change->n_connections + n_connection];
+ struct drbd_device_state_change *device_state_change = &state_change->devices[n_device];
+ bool resync_susp_dependency = peer_device_state_change->resync_susp_dependency[which];
+ bool resync_susp_other_c = peer_device_state_change->resync_susp_other_c[which];
+ enum drbd_repl_state repl_state = peer_device_state_change->repl_state[which];
+ enum drbd_disk_state disk_state = device_state_change->disk_state[which];
+
+ return resync_susp_dependency || resync_susp_other_c ||
+ ((repl_state == L_SYNC_SOURCE || repl_state == L_PAUSED_SYNC_S)
+ && disk_state <= D_INCONSISTENT);
+}
+
+static union drbd_state state_change_word(struct drbd_state_change *state_change,
+ unsigned int n_device, int n_connection,
+ enum which_state which)
+{
+ struct drbd_resource_state_change *resource_state_change =
+ &state_change->resource[0];
+ struct drbd_device_state_change *device_state_change =
+ &state_change->devices[n_device];
+ union drbd_state state = { {
+ .role = R_UNKNOWN,
+ .peer = R_UNKNOWN,
+ .conn = C_STANDALONE,
+ .disk = D_UNKNOWN,
+ .pdsk = D_UNKNOWN,
+ } };
+
+ state.role = resource_state_change->role[which];
+ state.susp = resource_state_change->susp[which] || state_change_is_susp_quorum(state_change, which) ||
+ resource_state_change->susp_uuid[which];
+ state.susp_nod = resource_state_change->susp_nod[which];
+ state.susp_fen = state_change_is_susp_fen(state_change, which);
+ state.quorum = device_state_change->have_quorum[which];
+ state.disk = device_state_change->disk_state[which];
+ if (n_connection != -1) {
+ struct drbd_connection_state_change *connection_state_change =
+ &state_change->connections[n_connection];
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[n_device * state_change->n_connections + n_connection];
+
+ state.peer = connection_state_change->peer_role[which];
+ state.conn = peer_device_state_change->repl_state[which];
+ if (state.conn <= L_OFF)
+ state.conn = connection_state_change->cstate[which];
+ state.pdsk = peer_device_state_change->disk_state[which];
+ state.aftr_isp = resync_susp_comb_dep_sc(state_change, n_device, n_connection, which);
+ state.peer_isp = peer_device_state_change->resync_susp_peer[which];
+ state.user_isp = peer_device_state_change->resync_susp_user[which];
+ }
+ return state;
+}
+
+int notify_resource_state_change(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type)
+{
+ struct drbd_resource_state_change *resource_state_change =
+ ((struct drbd_state_change *)state_change)->resource;
+ struct drbd_resource *resource = resource_state_change->resource;
+ struct drbd_resource_info resource_info = {
+ .res_role = resource_state_change->role[NEW],
+ .res_susp = resource_state_change->susp[NEW],
+ .res_susp_nod = resource_state_change->susp_nod[NEW],
+ .res_susp_fen = state_change_is_susp_fen(state_change, NEW),
+ .res_susp_quorum = state_change_is_susp_quorum(state_change, NEW) ||
+ resource_state_change->susp_uuid[NEW],
+ .res_fail_io = resource_state_change->fail_io[NEW],
+#ifdef CONFIG_DRBD_COMPAT_84
+ /*
+ * old_res_susp mirrors drbd_get_resource_state()'s own .susp
+ * formula (susp_user || susp_quorum || susp_uuid), evaluated
+ * at OLD instead of NOW/NEW, so the v1 dialect's prev_state
+ * combines the same three reasons into its one wire bit that
+ * new_state's live drbd_pack_state_84() does.
+ */
+ .old_res_role = resource_state_change->role[OLD],
+ .old_res_susp = resource_state_change->susp[OLD] ||
+ state_change_is_susp_quorum(state_change, OLD) ||
+ resource_state_change->susp_uuid[OLD],
+ .old_res_susp_nod = resource_state_change->susp_nod[OLD],
+ .old_res_susp_fen = state_change_is_susp_fen(state_change, OLD),
+#endif
+ };
+
+ return drbd_notify_resource_state(skb, seq, dialect, resource, &resource_info,
+ NULL, type);
+}
+
+int notify_connection_state_change(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type)
+{
+ struct drbd_connection_state_change *connection_state_change = state_change;
+ struct drbd_connection *connection = connection_state_change->connection;
+ struct drbd_connection_info connection_info = {
+ .conn_connection_state = connection_state_change->cstate[NEW],
+ .conn_role = connection_state_change->peer_role[NEW],
+#ifdef CONFIG_DRBD_COMPAT_84
+ .old_conn_connection_state = connection_state_change->cstate[OLD],
+ .old_conn_role = connection_state_change->peer_role[OLD],
+#endif
+ };
+
+ return drbd_notify_connection_state(skb, seq, dialect, connection,
+ &connection_info, type);
+}
+
+int notify_device_state_change(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type)
+{
+ struct drbd_device_state_change *device_state_change = state_change;
+ struct drbd_device *device = device_state_change->device;
+ struct drbd_device_info device_info;
+
+ device_state_change_to_info(&device_info, device_state_change);
+
+ return drbd_notify_device_state(skb, seq, dialect, device, &device_info, type);
+}
+
+int notify_peer_device_state_change(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ void *state_change,
+ enum drbd_notification_type type)
+{
+ struct drbd_peer_device_state_change *peer_device_state_change = state_change;
+ struct drbd_peer_device *peer_device = peer_device_state_change->peer_device;
+ struct drbd_peer_device_info peer_device_info;
+
+ peer_device_state_change_to_info(&peer_device_info, state_change);
+
+ return drbd_notify_peer_device_state(skb, seq, dialect, peer_device,
+ &peer_device_info, type);
+}
+
+static void notify_state_change(struct drbd_state_change *state_change)
+{
+ struct drbd_resource_state_change *resource_state_change = &state_change->resource[0];
+ bool resource_state_has_changed;
+ unsigned int n_device, n_connection, n_peer_device, n_peer_devices;
+ int (*last_func)(struct sk_buff *, unsigned int, const struct drbd_nl_dialect *,
+ void *, enum drbd_notification_type) = NULL;
+ void *last_arg = NULL;
+
+#define HAS_CHANGED(state) ((state)[OLD] != (state)[NEW])
+#define FINAL_STATE_CHANGE(type) \
+ ({ if (last_func) \
+ last_func(NULL, 0, NULL, last_arg, type); \
+ })
+#define REMEMBER_STATE_CHANGE(func, arg, type) \
+ ({ FINAL_STATE_CHANGE(type | NOTIFY_CONTINUES); \
+ last_func = (typeof(last_func))func; \
+ last_arg = arg; \
+ })
+
+ mutex_lock(¬ification_mutex);
+
+ resource_state_has_changed =
+ HAS_CHANGED(resource_state_change->role) ||
+ HAS_CHANGED(resource_state_change->susp) ||
+ HAS_CHANGED(resource_state_change->susp_nod) ||
+ HAS_CHANGED(resource_state_change->susp_uuid) ||
+ state_change_is_susp_fen(state_change, OLD) !=
+ state_change_is_susp_fen(state_change, NEW) ||
+ state_change_is_susp_quorum(state_change, OLD) !=
+ state_change_is_susp_quorum(state_change, NEW) ||
+ HAS_CHANGED(resource_state_change->fail_io);
+
+ if (resource_state_has_changed)
+ REMEMBER_STATE_CHANGE(notify_resource_state_change,
+ state_change, NOTIFY_CHANGE);
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change =
+ &state_change->connections[n_connection];
+
+ if (HAS_CHANGED(connection_state_change->peer_role) ||
+ HAS_CHANGED(connection_state_change->cstate))
+ REMEMBER_STATE_CHANGE(notify_connection_state_change,
+ connection_state_change, NOTIFY_CHANGE);
+ }
+
+ for (n_device = 0; n_device < state_change->n_devices; n_device++) {
+ struct drbd_device_state_change *device_state_change =
+ &state_change->devices[n_device];
+
+ if (HAS_CHANGED(device_state_change->disk_state) ||
+ HAS_CHANGED(device_state_change->have_quorum))
+ REMEMBER_STATE_CHANGE(notify_device_state_change,
+ device_state_change, NOTIFY_CHANGE);
+ }
+
+ n_peer_devices = state_change->n_devices * state_change->n_connections;
+ for (n_peer_device = 0; n_peer_device < n_peer_devices; n_peer_device++) {
+ struct drbd_peer_device_state_change *p =
+ &state_change->peer_devices[n_peer_device];
+
+ if (HAS_CHANGED(p->disk_state) ||
+ HAS_CHANGED(p->repl_state) ||
+ HAS_CHANGED(p->resync_susp_user) ||
+ HAS_CHANGED(p->resync_susp_peer) ||
+ HAS_CHANGED(p->resync_susp_dependency) ||
+ HAS_CHANGED(p->resync_susp_other_c) ||
+ HAS_CHANGED(p->resync_susp_max_parallel))
+ REMEMBER_STATE_CHANGE(notify_peer_device_state_change,
+ p, NOTIFY_CHANGE);
+ }
+
+ FINAL_STATE_CHANGE(NOTIFY_CHANGE);
+ mutex_unlock(¬ification_mutex);
+
+#undef HAS_CHANGED
+#undef FINAL_STATE_CHANGE
+#undef REMEMBER_STATE_CHANGE
+}
+
+static void send_role_to_all_peers(struct drbd_state_change *state_change)
+{
+ unsigned int n_connection;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change =
+ &state_change->connections[n_connection];
+ struct drbd_connection *connection = connection_state_change->connection;
+ enum drbd_conn_state new_cstate = connection_state_change->cstate[NEW];
+
+ if (new_cstate < C_CONNECTED)
+ continue;
+
+ if (connection->agreed_pro_version < 110) {
+ unsigned int n_device;
+
+ /* Before DRBD 9, the role is a device attribute
+ * instead of a resource attribute. */
+ for (n_device = 0; n_device < state_change->n_devices; n_device++) {
+ struct drbd_peer_device *peer_device =
+ state_change->peer_devices[n_connection].peer_device;
+ union drbd_state state =
+ state_change_word(state_change, n_device, n_connection, NEW);
+
+ drbd_send_state(peer_device, state);
+ }
+ } else {
+ union drbd_state state = { {
+ .role = state_change->resource[0].role[NEW],
+ } };
+
+ conn_send_state(connection, state);
+ }
+ }
+}
+
+static void send_new_state_to_all_peer_devices(struct drbd_state_change *state_change, int n_device)
+{
+ unsigned int n_connection;
+
+ BUG_ON(state_change->n_devices <= n_device);
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[n_device * state_change->n_connections + n_connection];
+ struct drbd_peer_device *peer_device = peer_device_state_change->peer_device;
+ union drbd_state new_state = state_change_word(state_change, n_device, n_connection, NEW);
+
+ if (new_state.conn >= C_CONNECTED)
+ drbd_send_state(peer_device, new_state);
+ }
+}
+
+/* Skip peers that did not finish the initial handshake yet. */
+static void send_uuids_to_all_peer_devices(struct drbd_state_change *state_change, int n_device)
+{
+ int n_connection;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[
+ n_device * state_change->n_connections + n_connection];
+
+ if (peer_device_state_change->disk_state[NEW] != D_UNKNOWN &&
+ peer_device_state_change->repl_state[NEW] > L_OFF)
+ drbd_send_uuids(peer_device_state_change->peer_device, 0, 0);
+ }
+}
+
+/* This function is supposed to have the same semantics as drbd_device_stable() in drbd_main.c
+ A primary is stable since it is authoritative.
+ Unstable are neighbors of a primary and resync target nodes.
+ Nodes further away from a primary are stable! Do no confuse with "weak".*/
+static bool calc_device_stable(struct drbd_state_change *state_change, int n_device, enum which_state which)
+{
+ int n_connection;
+
+ if (state_change->resource->role[which] == R_PRIMARY)
+ return true;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change =
+ &state_change->connections[n_connection];
+ enum drbd_role *peer_role = connection_state_change->peer_role;
+
+ if (peer_role[which] == R_PRIMARY)
+ return false;
+ }
+
+ return true;
+}
+
+static bool calc_resync_target(struct drbd_state_change *state_change, int n_device, enum which_state which)
+{
+ int n_connection;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[n_device * state_change->n_connections + n_connection];
+ enum drbd_repl_state *repl_state = peer_device_state_change->repl_state;
+
+ switch (repl_state[which]) {
+ case L_WF_BITMAP_T:
+ case L_SYNC_TARGET:
+ case L_PAUSED_SYNC_T:
+ return true;
+ default:
+ continue;
+ }
+ }
+
+ return false;
+}
+
+/* takes old and new peer disk state */
+static bool lost_contact_to_peer_data(enum drbd_disk_state *peer_disk_state)
+{
+ enum drbd_disk_state os = peer_disk_state[OLD];
+ enum drbd_disk_state ns = peer_disk_state[NEW];
+
+ return (os >= D_INCONSISTENT && os != D_UNKNOWN && os != D_OUTDATED)
+ && (ns < D_INCONSISTENT || ns == D_UNKNOWN || ns == D_OUTDATED);
+}
+
+static bool peer_returns_diskless(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state os, enum drbd_disk_state ns)
+{
+ struct drbd_device *device = peer_device->device;
+ bool rv = false;
+
+ /* Scenario, starting with normal operation
+ * Connected Primary/Secondary UpToDate/UpToDate
+ * NetworkFailure Primary/Unknown UpToDate/DUnknown (frozen)
+ * ...
+ * Connected Primary/Secondary UpToDate/Diskless (resumed; needs to bump uuid!)
+ */
+
+ if (get_ldev(device)) {
+ if (os == D_UNKNOWN && (ns == D_DISKLESS || ns == D_FAILED || ns == D_OUTDATED) &&
+ drbd_bitmap_uuid(peer_device) == 0)
+ rv = true;
+ put_ldev(device);
+ }
+ return rv;
+}
+
+static void check_may_resume_io_after_fencing(struct drbd_state_change *state_change, int n_connection)
+{
+ struct drbd_connection_state_change *connection_state_change = &state_change->connections[n_connection];
+ struct drbd_resource_state_change *resource_state_change = &state_change->resource[0];
+ struct drbd_connection *connection = connection_state_change->connection;
+ struct drbd_resource *resource = resource_state_change->resource;
+ bool all_peer_disks_outdated = true;
+ bool all_peer_disks_connected = true;
+ struct drbd_peer_device *peer_device;
+ unsigned long irq_flags;
+ int vnr, n_device;
+
+ for (n_device = 0; n_device < state_change->n_devices; n_device++) {
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[n_device * state_change->n_connections + n_connection];
+ enum drbd_repl_state *repl_state = peer_device_state_change->repl_state;
+ enum drbd_disk_state *peer_disk_state = peer_device_state_change->disk_state;
+
+ if (peer_disk_state[NEW] > D_OUTDATED)
+ all_peer_disks_outdated = false;
+ if (repl_state[NEW] < L_ESTABLISHED)
+ all_peer_disks_connected = false;
+ }
+
+ /* case1: The outdate peer handler is successful: */
+ if (all_peer_disks_outdated) {
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+ u64 fenced_node = NODE_MASK(peer_device->node_id);
+
+ kref_get(&device->kref);
+ rcu_read_unlock();
+ /* gen-rotate reason: DEGRADE (conn lost, peers fenced).
+ * This node resumes as the authority; end_state_change()
+ * below runs the walk that finalizes writes held towards
+ * the fenced peers.
+ */
+ drbd_gen_obligation_mint_before_resume(device, fenced_node);
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ __change_io_susp_fencing(connection, false);
+ end_state_change(resource, &irq_flags, "after-fencing");
+ }
+ /* case2: The connection was established again: */
+ if (all_peer_disks_connected) {
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ /* Every peer is back, so a resend replicates the writes that
+ * were held: the data ends identical to the last generation
+ * boundary and no generation is owed. That proof holds only
+ * while the obligation is latent -- once a completion decision
+ * made the divergence a fact, no resend heals it, and the
+ * generation is still owed.
+ */
+ if (!drbd_gen_obligation_materialized(device))
+ drbd_gen_obligation_void(device);
+ else if (drbd_gen_obligation_mint_start(device))
+ drbd_device_post_work(device, MAKE_NEW_CUR_UUID);
+ }
+ rcu_read_unlock();
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ __change_io_susp_fencing(connection, false);
+ end_state_change(resource, &irq_flags, "after-fencing");
+ }
+}
+
+static bool drbd_should_unfence(struct drbd_state_change *state_change, int n_connection)
+{
+ bool some_peer_was_not_up_to_date = false;
+ int n_device;
+
+ for (n_device = 0; n_device < state_change->n_devices; n_device++) {
+ struct drbd_device_state_change *device_state_change =
+ &state_change->devices[n_device];
+ enum drbd_disk_state *disk_state = device_state_change->disk_state;
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[
+ n_device * state_change->n_connections + n_connection];
+ enum drbd_disk_state *peer_disk_state = peer_device_state_change->disk_state;
+
+ /* Do not unfence if some volume is not yet up-to-date. */
+ if (disk_state[NEW] != D_UP_TO_DATE || peer_disk_state[NEW] != D_UP_TO_DATE)
+ return false;
+
+ /* Only unfence when the final volume becomes up-to-date. */
+ if (peer_disk_state[OLD] != D_UP_TO_DATE)
+ some_peer_was_not_up_to_date = true;
+ }
+
+ return some_peer_was_not_up_to_date;
+}
+
+static bool use_checksum_based_resync(struct drbd_peer_device *peer_device)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ bool csums_after_crash_only;
+
+ rcu_read_lock();
+ csums_after_crash_only = rcu_dereference(connection->transport.net_conf)->csums_after_crash_only;
+ rcu_read_unlock();
+ /* A reconciliation resync runs over a pessimistic, mostly-matching bitmap
+ * between two full copies after a failure -- exactly where csums pay off,
+ * so use them there too (both the connected dagtag reconcile and the
+ * equal-UUID handshake reconcile set RECONCILIATION_RESYNC).
+ */
+ return connection->agreed_pro_version >= 89 && /* supported? */
+ connection->csums_tfm && /* configured? */
+ (csums_after_crash_only == false /* use for each resync? */
+ || test_bit(CRASHED_PRIMARY, &device->flags) /* or only after Primary crash? */
+ || test_bit(RECONCILIATION_RESYNC, peer_device->flags));
+}
+
+static void drbd_run_resync(struct drbd_peer_device *peer_device, enum drbd_repl_state repl_state)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_bitmap *bm = device->bitmap;
+ struct drbd_connection *connection = peer_device->connection;
+ enum drbd_repl_state side = repl_is_sync_target(repl_state) ? L_SYNC_TARGET : L_SYNC_SOURCE;
+
+ drbd_info(peer_device, "Began resync as %s (will sync %llu KB [%lu bits set]).\n",
+ drbd_repl_str(repl_state),
+ bm_bit_to_kb(bm, peer_device->rs_total),
+ (unsigned long) peer_device->rs_total);
+
+ if (side == L_SYNC_TARGET)
+ drbd_uuid_set_exposed(device, peer_device->current_uuid, false);
+
+ peer_device->use_csums = side == L_SYNC_TARGET ?
+ use_checksum_based_resync(peer_device) : false;
+
+ if (side == L_SYNC_TARGET &&
+ !(peer_device->uuid_flags & UUID_FLAG_STABLE) &&
+ !drbd_stable_sync_source_present(peer_device, NOW))
+ set_bit(UNSTABLE_RESYNC, peer_device->flags);
+
+ /* Since protocol 96, we must serialize drbd_gen_and_send_sync_uuid
+ * with w_send_oos, or the sync target will get confused as to
+ * how much bits to resync. We cannot do that always, because for an
+ * empty resync and protocol < 95, we need to do it here, as we call
+ * drbd_resync_finished from here in that case.
+ * We drbd_gen_and_send_sync_uuid here for protocol < 96,
+ * and from after_state_ch otherwise. */
+ if (side == L_SYNC_SOURCE && connection->agreed_pro_version < 96)
+ drbd_gen_and_send_sync_uuid(peer_device);
+
+ if (connection->agreed_pro_version < 95 && peer_device->rs_total == 0) {
+ /* This still has a race (about when exactly the peers
+ * detect connection loss) that can lead to a full sync
+ * on next handshake. In 8.3.9 we fixed this with explicit
+ * resync-finished notifications, but the fix
+ * introduces a protocol change. Sleeping for some
+ * time longer than the ping interval + timeout on the
+ * SyncSource, to give the SyncTarget the chance to
+ * detect connection loss, then waiting for a ping
+ * response (implicit in drbd_resync_finished) reduces
+ * the race considerably, but does not solve it. */
+ if (side == L_SYNC_SOURCE) {
+ struct drbd_net_conf *nc;
+ int timeo;
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ timeo = nc->ping_int * HZ + nc->ping_timeo * HZ / 9;
+ rcu_read_unlock();
+ schedule_timeout_interruptible(timeo);
+ }
+ drbd_resync_finished(peer_device, D_MASK);
+ }
+
+ /* ns.conn may already be != peer_device->repl_state[NOW],
+ * we may have been paused in between, or become paused until
+ * the timer triggers.
+ * No matter, that is handled in resync_timer_fn() */
+ if (side == L_SYNC_TARGET)
+ drbd_uuid_resync_starting(peer_device);
+ else
+ drbd_uuid_resync_starting_source(peer_device);
+
+ drbd_md_sync_if_dirty(device);
+}
+
+
+/* A resync-free connect concluded data identity from equal current UUIDs; bits
+ * set for blocks the peer lacks contradict that, so keep them for the next
+ * connect to repair (maybe_reconcile_equal_uuid_bitmap()).
+ */
+static void no_resync_connect_bm(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ unsigned long set = drbd_bm_total_weight(peer_device);
+ bool keep = set && drbd_bitmap_slot_decides(peer_device);
+
+ /* equal currents are no generation boundary; a bitmap UUID claiming one
+ * reads as a split brain
+ */
+ down_write(&device->uuid_sem);
+ drbd_uuid_set_bitmap(peer_device, 0);
+ up_write(&device->uuid_sem);
+
+ if (keep) {
+ drbd_warn(peer_device,
+ "resync-free connect at equal current UUIDs, %lu bits stay out of sync\n",
+ set);
+ drbd_print_uuids(peer_device, "cleared bm UUID, kept bitmap");
+ return;
+ }
+
+ drbd_print_uuids(peer_device, "cleared bm UUID and bitmap");
+ drbd_bitmap_io_from_worker(device, &drbd_bmio_clear_one_peer,
+ "clearing bm one peer",
+ BM_LOCK_CLEAR | BM_LOCK_BULK | BM_LOCK_SINGLE_SLOT,
+ peer_device);
+}
+
+/*
+ * Perform after state change actions that may sleep.
+ */
+static int w_after_state_change(struct drbd_work *w, int unused)
+{
+ struct after_state_change_work *work =
+ container_of(w, struct after_state_change_work, w);
+ struct drbd_state_change *state_change = work->state_change;
+ struct drbd_resource_state_change *resource_state_change = &state_change->resource[0];
+ struct drbd_resource *resource = resource_state_change->resource;
+ enum drbd_role *role = resource_state_change->role;
+ bool *susp_uuid = resource_state_change->susp_uuid;
+ bool *resume_held_for_outdate = resource_state_change->resume_held_for_outdate;
+ struct drbd_peer_device *send_state_others = NULL;
+ int n_device, n_connection;
+ bool still_connected = false;
+ bool try_become_up_to_date = false;
+ bool healed_primary = false;
+ bool send_flush_requests = false;
+
+ notify_state_change(state_change);
+
+ for (n_device = 0; n_device < state_change->n_devices; n_device++) {
+ struct drbd_device_state_change *device_state_change = &state_change->devices[n_device];
+ struct drbd_device *device = device_state_change->device;
+ enum drbd_disk_state *disk_state = device_state_change->disk_state;
+ bool have_ldev = extra_ldev_ref_for_after_state_chg(disk_state);
+ bool *have_quorum = device_state_change->have_quorum;
+ bool effective_disk_size_determined = false;
+ bool device_stable[2], resync_target[2];
+ bool data_accessible[2];
+ bool all_peer_replication[2];
+ bool resync_finished = false;
+ bool some_peer_demoted = false;
+ enum which_state which;
+
+ for (which = OLD; which <= NEW; which++) {
+ device_stable[which] = calc_device_stable(state_change, n_device, which);
+ resync_target[which] = calc_resync_target(state_change, n_device, which);
+ data_accessible[which] =
+ calc_data_accessible(state_change, n_device, which);
+ all_peer_replication[which] =
+ drbd_all_peer_replication_change(state_change, n_device, which);
+
+ }
+
+ if (disk_state[NEW] == D_UP_TO_DATE)
+ effective_disk_size_determined = true;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ enum drbd_repl_state *repl_state = state_change->peer_devices[
+ n_device * state_change->n_connections + n_connection].repl_state;
+
+ if ((repl_state[OLD] == L_SYNC_TARGET ||
+ repl_state[OLD] == L_PAUSED_SYNC_T) &&
+ repl_state[NEW] == L_ESTABLISHED)
+ resync_finished = true;
+ }
+
+ /*
+ * In case we finished a resync as resync-target update all neighbors
+ * about having a bitmap_uuid of 0 towards the previous sync-source.
+ * That needs to go out before sending the new disk state
+ * to avoid a race where the other node might downgrade our disk
+ * state due to old UUID values. The loop below sends that state to
+ * all peers as soon as it reaches the former sync source.
+ */
+ if (resync_finished)
+ send_uuids_to_all_peer_devices(state_change, n_device);
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[
+ n_device * state_change->n_connections + n_connection];
+ struct drbd_peer_device *peer_device = peer_device_state_change->peer_device;
+ enum drbd_disk_state *peer_disk_state = peer_device_state_change->disk_state;
+ enum drbd_repl_state *repl_state = peer_device_state_change->repl_state;
+
+ /* Writes withheld from acknowledgment waited for this
+ * resync; no answer settles them once it is gone.
+ */
+ if (repl_is_sync_target(repl_state[OLD]) &&
+ !repl_is_sync_target(repl_state[NEW]))
+ drbd_refuse_unsecured_writes(peer_device);
+
+ /* The dagtag wait requests this peer asked as our sync
+ * target are void once we stopped being its source.
+ */
+ if (repl_is_sync_source(repl_state[OLD]) &&
+ !repl_is_sync_source(repl_state[NEW]))
+ drbd_dagtag_wait_reqs_source_gone(peer_device);
+
+ if (disk_state[OLD] == D_INCONSISTENT && disk_state[NEW] == D_UP_TO_DATE &&
+ peer_disk_state[OLD] == D_INCONSISTENT && peer_disk_state[NEW] == D_UP_TO_DATE)
+ send_state_others = peer_device;
+
+ /* connect without resync or remote attach without resync */
+ if (disk_state[NOW] >= D_OUTDATED && repl_state[NEW] == L_ESTABLISHED &&
+ ((repl_state[OLD] == L_OFF && peer_disk_state[NEW] >= D_OUTDATED) ||
+ (peer_disk_state[OLD] == D_DISKLESS && peer_disk_state[NEW] >= D_OUTDATED))) {
+ u64 peer_current_uuid = peer_device->current_uuid & ~UUID_PRIMARY;
+ u64 my_current_uuid = drbd_current_uuid(device) & ~UUID_PRIMARY;
+
+ if (peer_current_uuid == my_current_uuid && get_ldev(device)) {
+ no_resync_connect_bm(peer_device);
+ put_ldev(device);
+ }
+ }
+ }
+
+ if (role[NEW] == R_PRIMARY && !data_accessible[OLD] && data_accessible[NEW])
+ healed_primary = true;
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change = &state_change->connections[n_connection];
+ struct drbd_connection *connection = connection_state_change->connection;
+ enum drbd_conn_state *cstate = connection_state_change->cstate;
+ enum drbd_role *peer_role = connection_state_change->peer_role;
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[
+ n_device * state_change->n_connections + n_connection];
+ struct drbd_peer_device *peer_device = peer_device_state_change->peer_device;
+ enum drbd_repl_state *repl_state = peer_device_state_change->repl_state;
+ enum drbd_disk_state *peer_disk_state = peer_device_state_change->disk_state;
+ bool *resync_susp_user = peer_device_state_change->resync_susp_user;
+ bool *resync_susp_peer = peer_device_state_change->resync_susp_peer;
+ bool *resync_susp_dependency = peer_device_state_change->resync_susp_dependency;
+ bool *resync_susp_max_parallel =
+ peer_device_state_change->resync_susp_max_parallel;
+ union drbd_state new_state =
+ state_change_word(state_change, n_device, n_connection, NEW);
+ bool send_uuids = false, send_state = false;
+
+ /* Send UUIDs again if they changed while establishing the connection */
+ if (repl_state[OLD] == L_OFF && repl_state[NEW] > L_OFF &&
+ peer_device->comm_current_uuid != drbd_resolved_uuid(peer_device, NULL))
+ send_uuids = true;
+
+ if (repl_state[NEW] > L_OFF && device_stable[OLD] != device_stable[NEW])
+ send_uuids = true;
+
+ if (send_uuids)
+ drbd_send_uuids(peer_device, 0, 0);
+
+ if (peer_disk_state[NEW] == D_UP_TO_DATE)
+ effective_disk_size_determined = true;
+
+ if (!(role[OLD] == R_PRIMARY && !data_accessible[OLD]) &&
+ (role[NEW] == R_PRIMARY && !data_accessible[NEW]) &&
+ !test_bit(UNREGISTERED, &device->flags))
+ drbd_maybe_khelper(device, connection, "pri-on-incon-degr");
+
+ /* Became sync source. With protocol >= 96, we still need to send out
+ * the sync uuid now. Need to do that before any drbd_send_state, or
+ * the other side may go "paused sync" before receiving the sync uuids,
+ * which is unexpected. */
+ if (!(repl_state[OLD] == L_SYNC_SOURCE || repl_state[OLD] == L_PAUSED_SYNC_S) &&
+ (repl_state[NEW] == L_SYNC_SOURCE || repl_state[NEW] == L_PAUSED_SYNC_S) &&
+ connection->agreed_pro_version >= 96 && connection->agreed_pro_version < 110 &&
+ get_ldev(device)) {
+ drbd_gen_and_send_sync_uuid(peer_device);
+ put_ldev(device);
+ }
+
+ /* Do not change the order of the if above and the two below... */
+ if (peer_disk_state[OLD] < D_NEGOTIATING &&
+ peer_disk_state[NEW] == D_NEGOTIATING) { /* attach on the peer */
+ /* we probably will start a resync soon.
+ * make sure those things are properly reset. */
+ peer_device->rs_total = 0;
+ peer_device->rs_failed = 0;
+
+ drbd_send_uuids_peer_attach(peer_device);
+ }
+ /* No point in queuing send_bitmap if we don't have a connection
+ * anymore, so check also the _current_ state, not only the new state
+ * at the time this work was queued. */
+ if (repl_state[OLD] != L_WF_BITMAP_S && repl_state[NEW] == L_WF_BITMAP_S &&
+ peer_device->repl_state[NOW] == L_WF_BITMAP_S) {
+ /* Now that the connection is L_WF_BITMAP_S,
+ * new requests will be sent to the peer as
+ * P_OUT_OF_SYNC packets. However, active
+ * requests may not have been communicated to
+ * the peer and may not yet be marked in the
+ * local bitmap. Mark these requests in the
+ * bitmap before reading and sending that
+ * bitmap. This may set bits unnecessarily, but
+ * it does no harm to resync a small amount of
+ * additional data. */
+ drbd_set_pending_out_of_sync(peer_device);
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
+ drbd_queue_bitmap_io(device, &drbd_send_bitmap, NULL,
+ "send_bitmap (WFBitMapS)",
+ BM_LOCK_SET | BM_LOCK_CLEAR | BM_LOCK_BULK | BM_LOCK_SINGLE_SLOT,
+ peer_device);
+ }
+
+ if (peer_role[OLD] == R_PRIMARY && peer_role[NEW] == R_SECONDARY)
+ some_peer_demoted = true;
+
+ /* Last part of the attaching process ... */
+ if (cstate[NEW] == C_CONNECTED && /* repl_state[NEW] might still be L_OFF */
+ disk_state[OLD] == D_ATTACHING && disk_state[NEW] >= D_NEGOTIATING) {
+ drbd_send_sizes(peer_device, 0, 0); /* to start sync... */
+ drbd_send_uuids(peer_device, 0, 0);
+ drbd_send_state(peer_device, new_state);
+ }
+
+ /* Started resync, tell peer if drbd9 */
+ if (repl_state[NEW] >= L_SYNC_SOURCE && repl_state[NEW] <= L_PAUSED_SYNC_T &&
+ (repl_state[OLD] < L_SYNC_SOURCE || repl_state[OLD] > L_PAUSED_SYNC_T))
+ send_state = true;
+
+ /* We want to pause/continue resync, tell peer. */
+ if (repl_state[NEW] >= L_ESTABLISHED &&
+ ((resync_susp_comb_dep_sc(state_change, n_device, n_connection, OLD) !=
+ resync_susp_comb_dep_sc(state_change, n_device, n_connection, NEW)) ||
+ (resync_susp_user[OLD] != resync_susp_user[NEW]) ||
+ (resync_susp_max_parallel[OLD] != resync_susp_max_parallel[NEW])))
+ send_state = true;
+
+ /* finished resync, tell sync source */
+ if ((repl_state[OLD] == L_SYNC_TARGET || repl_state[OLD] == L_PAUSED_SYNC_T) &&
+ repl_state[NEW] == L_ESTABLISHED)
+ send_state = true;
+
+ /* In case one of the isp bits got set, suspend other devices. */
+ if (!(resync_susp_dependency[OLD] || resync_susp_peer[OLD] || resync_susp_user[OLD]) &&
+ (resync_susp_dependency[NEW] || resync_susp_peer[NEW] || resync_susp_user[NEW]))
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
+ suspend_other_sg(device);
+
+ /* Make sure the peer gets informed about eventual state
+ changes (ISP bits) while we were in L_OFF. */
+ if (repl_state[OLD] == L_OFF && repl_state[NEW] >= L_ESTABLISHED)
+ send_state = true;
+
+ if (repl_state[OLD] != L_AHEAD && repl_state[NEW] == L_AHEAD)
+ send_state = true;
+
+ /* We are in the progress to start a full sync. SyncTarget sets all slots. */
+ if (repl_state[OLD] != L_STARTING_SYNC_T && repl_state[NEW] == L_STARTING_SYNC_T)
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
+ drbd_queue_bitmap_io(device,
+ &drbd_bmio_set_all_n_write, &abw_start_sync,
+ "set_n_write from StartingSync",
+ BM_LOCK_CLEAR | BM_LOCK_BULK,
+ peer_device);
+
+ /* We are in the progress to start a full sync. SyncSource one slot. */
+ if (repl_state[OLD] != L_STARTING_SYNC_S && repl_state[NEW] == L_STARTING_SYNC_S)
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
+ drbd_queue_bitmap_io(device,
+ &drbd_bmio_set_n_write, &abw_start_sync,
+ "set_n_write from StartingSync",
+ BM_LOCK_CLEAR | BM_LOCK_BULK | BM_LOCK_SINGLE_SLOT,
+ peer_device);
+
+ /* A backing device that grew under DRBD reaches the
+ * cluster only through a size transaction; see
+ * drbd_auto_grow(). A peer that was not here may
+ * bring space with it, so what the cluster answered
+ * before says nothing about what it answers now.
+ */
+ if (repl_state[OLD] == L_OFF && repl_state[NEW] >= L_ESTABLISHED)
+ device->auto_grow_asked = 0;
+
+ /* Ask where the answer can be a new one: a connection
+ * that has settled, and the end of a resync, which a
+ * transaction has to wait for anyway. An online
+ * verify and a pull-ahead phase both start from
+ * L_ESTABLISHED, where this was asked already, so
+ * coming back from them is no edge.
+ */
+ if (repl_state[NEW] == L_ESTABLISHED &&
+ repl_state[OLD] != L_ESTABLISHED &&
+ repl_state[OLD] != L_VERIFY_S &&
+ repl_state[OLD] != L_VERIFY_T &&
+ repl_state[OLD] != L_AHEAD &&
+ repl_state[OLD] != L_BEHIND)
+ drbd_device_post_work(device, AUTO_GROW);
+
+ /* Disks got bigger while they were detached */
+ if (disk_state[NEW] > D_NEGOTIATING && peer_disk_state[NEW] > D_NEGOTIATING &&
+ test_and_clear_bit(RESYNC_AFTER_NEG, peer_device->flags)) {
+ if (repl_state[NEW] == L_ESTABLISHED)
+ resync_after_online_grow(peer_device);
+ }
+
+ /* A resync finished or aborted, wake paused devices... */
+ if ((repl_state[OLD] > L_ESTABLISHED && repl_state[NEW] <= L_ESTABLISHED) ||
+ (resync_susp_peer[OLD] && !resync_susp_peer[NEW]) ||
+ (resync_susp_user[OLD] && !resync_susp_user[NEW])) {
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
+ resume_next_sg(device);
+ drbd_apply_resync_max_parallel();
+ } else if (repl_holds_resync_slot(repl_state[OLD]) &&
+ !repl_holds_resync_slot(repl_state[NEW])) {
+ drbd_apply_resync_max_parallel();
+ }
+
+ /* sync target done with resync. Explicitly notify all peers. Our sync
+ source should even know by himself, but the others need that info. */
+ if (disk_state[OLD] < D_UP_TO_DATE && repl_state[OLD] >= L_SYNC_SOURCE && repl_state[NEW] == L_ESTABLISHED)
+ send_new_state_to_all_peer_devices(state_change, n_device);
+
+ /* Outdated myself, or became D_UP_TO_DATE tell peers
+ * Do not do it, when the local node was forced from R_SECONDARY to R_PRIMARY,
+ * because that is part of the 2-phase-commit and that is necessary to trigger
+ * the initial resync. */
+ if ((disk_state[NEW] >= D_INCONSISTENT && disk_state[NEW] != disk_state[OLD] &&
+ repl_state[OLD] >= L_ESTABLISHED && repl_state[NEW] >= L_ESTABLISHED) &&
+ !(role[OLD] == R_SECONDARY && role[NEW] == R_PRIMARY))
+ send_state = true;
+
+ /* diskless peers need to be informed about quorum changes, since they consider
+ the quorum state of the diskfull nodes. */
+ if (have_quorum[OLD] != have_quorum[NEW] && disk_state[NEW] >= D_INCONSISTENT)
+ send_state = true;
+
+ /* Skipped resync with peer_device, tell others... */
+ if (send_state_others && send_state_others != peer_device)
+ send_state = true;
+
+ /* This triggers bitmap writeout of potentially still unwritten pages
+ * if the resync finished cleanly, or aborted because of peer disk
+ * failure, or on transition from resync back to AHEAD/BEHIND.
+ *
+ * Connection loss is handled in conn_disconnect() by the receiver.
+ *
+ * For resync aborted because of local disk failure, we cannot do
+ * any bitmap writeout anymore.
+ *
+ * No harm done if some bits change during this phase.
+ */
+ if ((repl_state[OLD] > L_ESTABLISHED && repl_state[OLD] < L_AHEAD) &&
+ (repl_state[NEW] == L_ESTABLISHED || repl_state[NEW] >= L_AHEAD) &&
+ get_ldev(device)) {
+ drbd_queue_bitmap_io(device, &drbd_bm_write_copy_pages, NULL,
+ "write from resync_finished", BM_LOCK_BULK,
+ NULL);
+ put_ldev(device);
+ }
+
+ /* Verify finished, or reached stop sector. Peer did not know about
+ * the stop sector, and we may even have changed the stop sector during
+ * verify to interrupt/stop early. Send the new state. */
+ if (repl_state[OLD] == L_VERIFY_S && repl_state[NEW] == L_ESTABLISHED
+ && verify_can_do_stop_sector(peer_device))
+ send_new_state_to_all_peer_devices(state_change, n_device);
+
+ if (disk_state[NEW] == D_DISKLESS &&
+ cstate[NEW] == C_STANDALONE &&
+ role[NEW] == R_SECONDARY) {
+ if (resync_susp_dependency[OLD] != resync_susp_dependency[NEW])
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg */
+ resume_next_sg(device);
+ }
+
+ if (device_stable[OLD] && !device_stable[NEW] &&
+ repl_state[NEW] >= L_ESTABLISHED && get_ldev(device)) {
+ /* Inform peers about being unstable...
+ Maybe it would be a better idea to have the stable bit as
+ part of the state (and being sent with the state) */
+ drbd_send_uuids(peer_device, 0, 0);
+ put_ldev(device);
+ }
+
+ if (send_state && cstate[NEW] == C_CONNECTED)
+ drbd_send_state(peer_device, new_state);
+
+ if (((!device_stable[OLD] && device_stable[NEW]) ||
+ (resync_target[OLD] && !resync_target[NEW] && device_stable[NEW])) &&
+ !(repl_state[OLD] == L_SYNC_TARGET || repl_state[OLD] == L_PAUSED_SYNC_T) &&
+ !(peer_role[OLD] == R_PRIMARY) && disk_state[NEW] >= D_OUTDATED &&
+ repl_state[NEW] >= L_ESTABLISHED &&
+ get_ldev(device)) {
+ /* Offer all peers a resync, with the exception of ...
+ ... the node that made me up-to-date (with a resync)
+ ... I was primary
+ ... the peer that transitioned from primary to secondary
+ */
+ drbd_send_uuids(peer_device, UUID_FLAG_GOT_STABLE, 0);
+ if (peer_disk_state[NEW] == D_OUTDATED &&
+ peer_device->uuid_flags & UUID_FLAG_STABLE)
+ drbd_predict_peer_upgrade(peer_device);
+ put_ldev(device);
+ }
+
+ if (disk_state[OLD] == D_OUTDATED && disk_state[NEW] >= D_CONSISTENT &&
+ peer_disk_state[NEW] == D_OUTDATED &&
+ repl_state[NEW] == L_ESTABLISHED &&
+ role[NEW] == R_SECONDARY && device_stable[NEW])
+ drbd_predict_peer_upgrade(peer_device);
+
+ if (peer_disk_state[OLD] == D_UP_TO_DATE &&
+ (peer_disk_state[NEW] == D_FAILED || peer_disk_state[NEW] == D_INCONSISTENT) &&
+ drbd_gen_obligation_mint_start(device))
+ /* When a peer disk goes from D_UP_TO_DATE to D_FAILED
+ * or D_INCONSISTENT we know that a write failed on that
+ * node. Mint right now, not at the next write. Refused,
+ * an earlier obligation is already being minted.
+ */
+ drbd_device_post_work(device, MAKE_NEW_CUR_UUID);
+
+ /* A diskless-primary reconcile peer we asserted UpToDate on its
+ * predecessor generation has now settled UpToDate -- the connection
+ * handshake is complete and the peer knows our role. Only now is it
+ * safe to relabel it forward to our (real) current generation: hand the
+ * work to the sender so the P_CURRENT_UUID is ordered after the replayed
+ * writes, and the peer takes the adopt path rather than outdating. See
+ * diskless_with_peers_different_current_uuids().
+ */
+ if (peer_disk_state[NEW] == D_UP_TO_DATE &&
+ peer_disk_state[OLD] != D_UP_TO_DATE &&
+ test_bit(RECONCILE_INJECT_CUR_UUID, peer_device->flags))
+ drbd_peer_device_post_work(peer_device, SEND_RECONCILE_UUID);
+
+ if (disk_state[OLD] > D_FAILED && disk_state[NEW] == D_FAILED &&
+ role[NEW] == R_PRIMARY && drbd_gen_obligation_mint_start(device))
+ /* Our own disk failed while we are Primary: same as above. */
+ drbd_device_post_work(device, MAKE_NEW_CUR_UUID);
+
+ if (repl_state[OLD] != L_VERIFY_S && repl_state[NEW] == L_VERIFY_S) {
+ drbd_info(peer_device, "Starting Online Verify from sector %llu\n",
+ (unsigned long long)peer_device->ov_position);
+ drbd_queue_work_if_unqueued(
+ &peer_device->connection->sender_work,
+ &peer_device->resync_work);
+ }
+
+ if (!repl_is_sync(repl_state[OLD]) && repl_is_sync(repl_state[NEW]))
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
+ drbd_run_resync(peer_device, repl_state[NEW]);
+
+ if (repl_is_sync(repl_state[OLD]) && !repl_is_sync(repl_state[NEW]))
+ drbd_last_resync_request(peer_device, false);
+
+ if (peer_device_state_change->repl_state[OLD] != L_SYNC_TARGET &&
+ peer_device_state_change->repl_state[NEW] == L_SYNC_TARGET)
+ drbd_queue_work_if_unqueued(
+ &peer_device->connection->sender_work,
+ &peer_device->resync_work);
+
+ if (!(repl_is_sync_target(repl_state[OLD]) &&
+ all_peer_replication[OLD]) &&
+ repl_is_sync_target(repl_state[NEW]) &&
+ all_peer_replication[NEW])
+ send_flush_requests = true;
+
+ if (!peer_device_state_change->peer_replication[OLD] &&
+ peer_device_state_change->peer_replication[NEW])
+ drbd_send_enable_replication(peer_device, true);
+ }
+
+ if (((role[OLD] == R_PRIMARY && role[NEW] == R_SECONDARY) || some_peer_demoted) &&
+ get_ldev(device)) {
+ /* The some_peer_demoted case is superseded by
+ * handle_neighbor_demotion(). We keep this call for
+ * compatibility until support for protocol version 121
+ * is removed.
+ *
+ * No changes to the bitmap expected after this point, so write out any
+ * changes up to now to ensure that the metadata disk has the full
+ * bitmap content. Even if the bitmap changes (e.g. it was dual primary)
+ * no harm was done if it did change. */
+ drbd_bitmap_io_from_worker(device, &drbd_bm_write,
+ "demote", BM_LOCK_SET | BM_LOCK_CLEAR | BM_LOCK_BULK,
+ NULL);
+ put_ldev(device);
+ }
+
+ /* Make sure the effective disk size is stored in the metadata
+ * if a local disk is attached and either the local disk state
+ * or a peer disk state is D_UP_TO_DATE. */
+ if (effective_disk_size_determined && get_ldev(device)) {
+ sector_t size = get_capacity(device->vdisk);
+
+ if (device->ldev->md.effective_size != size) {
+ char ppb[10];
+
+ drbd_info(device, "persisting effective size = %s (%llu KB)\n",
+ ppsize(ppb, size >> 1),
+ (unsigned long long)size >> 1);
+ device->ldev->md.effective_size = size;
+ drbd_md_mark_dirty(device);
+ }
+ put_ldev(device);
+ }
+
+ /* first half of local IO error, failure to attach,
+ * or administrative detach */
+ if ((disk_state[OLD] != D_FAILED && disk_state[NEW] == D_FAILED) ||
+ (disk_state[OLD] != D_DETACHING && disk_state[NEW] == D_DETACHING)) {
+ enum drbd_io_error_p eh = EP_PASS_ON;
+ int was_io_error = 0;
+
+ /* Our cleanup here with the transition to D_DISKLESS.
+ * It is still not safe to dereference ldev here, since
+ * we might come from an failed Attach before ldev was set. */
+ /* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
+ if (have_ldev && device->ldev) {
+ rcu_read_lock();
+ eh = rcu_dereference(device->ldev->disk_conf)->on_io_error;
+ rcu_read_unlock();
+
+ was_io_error = disk_state[NEW] == D_FAILED;
+
+ /* Intentionally call this handler first, before drbd_send_state().
+ * See: 2932204 drbd: call local-io-error handler early
+ * People may chose to hard-reset the box from this handler.
+ * It is useful if this looks like a "regular node crash". */
+ if (was_io_error && eh == EP_CALL_HELPER)
+ drbd_maybe_khelper(device, NULL, "local-io-error");
+
+ /* Immediately allow completion of all application IO,
+ * that waits for completion from the local disk,
+ * if this was a force-detach due to disk_timeout
+ * or administrator request (drbdsetup detach --force).
+ * Do NOT abort otherwise.
+ * Aborting local requests may cause serious problems,
+ * if requests are completed to upper layers already,
+ * and then later the already submitted local bio completes.
+ * This can cause DMA into former bio pages that meanwhile
+ * have been re-used for other things.
+ * So aborting local requests may cause crashes,
+ * or even worse, silent data corruption.
+ */
+ if (test_bit(FORCE_DETACH, &device->flags))
+ tl_abort_disk_io(device);
+
+ send_new_state_to_all_peer_devices(state_change, n_device);
+
+ /* In case we want to get something to stable storage still,
+ * this may be the last chance.
+ * Following put_ldev may transition to D_DISKLESS. */
+ drbd_bitmap_io_from_worker(device, &drbd_bm_write,
+ "detach", BM_LOCK_SET | BM_LOCK_CLEAR | BM_LOCK_BULK,
+ NULL);
+ drbd_md_sync_if_dirty(device);
+ }
+ }
+
+ /* second half of local IO error, failure to attach,
+ * or administrative detach,
+ * after local_cnt references have reached zero again */
+ if (disk_state[OLD] != D_DISKLESS && disk_state[NEW] == D_DISKLESS) {
+ /* We must still be diskless,
+ * re-attach has to be serialized with this! */
+ if (device->disk_state[NOW] != D_DISKLESS)
+ drbd_err(device,
+ "ASSERT FAILED: disk is %s while going diskless\n",
+ drbd_disk_str(device->disk_state[NOW]));
+
+ /* we may need to cancel the md_sync timer */
+ timer_delete_sync(&device->md_sync_timer);
+
+ if (have_ldev)
+ send_new_state_to_all_peer_devices(state_change, n_device);
+ }
+
+ if (have_ldev)
+ put_ldev(device);
+
+ /* Notify peers that I had a local IO error and did not detach. */
+ if (disk_state[OLD] == D_UP_TO_DATE && disk_state[NEW] == D_INCONSISTENT)
+ send_new_state_to_all_peer_devices(state_change, n_device);
+
+ /* Testing EMPTY_TWOPC_PENDING would cause more queuing than necessary */
+ if (should_try_become_up_to_date(device, disk_state, NOW))
+ try_become_up_to_date = true;
+
+ if (test_and_clear_bit(TRY_TO_GET_RESYNC, &device->flags)) {
+ /* Got connected to a diskless primary */
+ kref_get(&device->kref);
+ if (!schedule_work(&device->try_get_resynced_work))
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+
+ drbd_md_sync_if_dirty(device);
+
+ if (role[NEW] == R_PRIMARY && have_quorum[OLD] && !have_quorum[NEW])
+ drbd_maybe_khelper(device, NULL, "quorum-lost");
+
+ /* gen-rotate reason: DEGRADE (lost quorum/data then regained).
+ * The susp_uuid clear further below is the edge that lifts the last
+ * suspension, so mint before it: a volume that stays armed keeps
+ * blocking its own first write, but writes the walk gives up there
+ * complete to the application, and need the generation first.
+ */
+ if (!susp_uuid[OLD] && susp_uuid[NEW])
+ drbd_gen_obligation_mint_before_resume(device, 0);
+
+ if (disk_state[OLD] > D_DISKLESS && disk_state[NEW] == D_DISKLESS)
+ drbd_reconsider_queue_parameters(device, NULL);
+ }
+
+ if (role[OLD] == R_PRIMARY && role[NEW] == R_SECONDARY)
+ send_role_to_all_peers(state_change);
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change = &state_change->connections[n_connection];
+ struct drbd_connection *connection = connection_state_change->connection;
+ enum drbd_conn_state *cstate = connection_state_change->cstate;
+ bool *susp_fen = connection_state_change->susp_fen;
+ enum drbd_fencing_policy fencing_policy;
+
+ if (cstate[NEW] == C_CONNECTED && send_flush_requests &&
+ connection->agreed_pro_version >= 123) {
+ u64 current_flush_sequence;
+
+ spin_lock_irq(&resource->initiator_flush_lock);
+ /* Requirement: At least the value from the corresponding state change */
+ current_flush_sequence = resource->current_flush_sequence;
+ spin_unlock_irq(&resource->initiator_flush_lock);
+
+ drbd_send_flush_requests(connection, current_flush_sequence);
+ }
+
+ /*
+ * A flush round that reached us before this connect committed
+ * was not forwarded on it. Forward it now, nothing else may.
+ */
+ if (cstate[OLD] != C_CONNECTED && cstate[NEW] == C_CONNECTED) {
+ read_lock_irq(&resource->state_rwlock);
+ if (connection->cstate[NOW] == C_CONNECTED)
+ queue_work(connection->ack_sender, &connection->peer_ack_work);
+ read_unlock_irq(&resource->state_rwlock);
+ }
+
+ /* Upon network configuration, we need to start the receiver */
+ if (cstate[OLD] == C_STANDALONE && cstate[NEW] == C_UNCONNECTED)
+ drbd_thread_start(&connection->receiver);
+
+ if (susp_fen[NEW])
+ check_may_resume_io_after_fencing(state_change, n_connection);
+
+ rcu_read_lock();
+ fencing_policy = connection->fencing_policy;
+ rcu_read_unlock();
+ if (fencing_policy != FP_DONT_CARE &&
+ drbd_should_unfence(state_change, n_connection))
+ drbd_maybe_khelper(NULL, connection, "unfence-peer");
+ }
+
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_connection_state_change *connection_state_change = &state_change->connections[n_connection];
+ enum drbd_conn_state *cstate = connection_state_change->cstate;
+
+ if (cstate[NEW] == C_CONNECTED || cstate[NEW] == C_CONNECTING)
+ still_connected = true;
+ }
+
+ if (susp_uuid[NEW]) {
+ unsigned long irq_flags;
+
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ resource->susp_uuid[NEW] = false;
+ end_state_change(resource, &irq_flags, "susp-uuid");
+ }
+
+ if (try_become_up_to_date || healed_primary)
+ drbd_schedule_empty_twopc(resource);
+
+ /* The Primary regained data access but sanitize_state held its resume
+ * because a far-away member must be outdated first. Drive that from
+ * here.
+ */
+ if (!resume_held_for_outdate[OLD] && resume_held_for_outdate[NEW])
+ drbd_schedule_resume_twopc(resource);
+
+ if (!still_connected)
+ mod_timer_pending(&resource->twopc_timer, jiffies);
-}
+ if (work->done)
+ complete(work->done);
+ forget_state_change(state_change);
+ kfree(work);
+ return 0;
+}
-/**
- * cl_wide_st_chg() - true if the state change is a cluster wide one
- * @device: DRBD device.
- * @os: old (current) state.
- * @ns: new (wanted) state.
- */
-static int cl_wide_st_chg(struct drbd_device *device,
- union drbd_state os, union drbd_state ns)
+static bool local_state_change(enum chg_state_flags flags)
{
- return (os.conn >= C_CONNECTED && ns.conn >= C_CONNECTED &&
- ((os.role != R_PRIMARY && ns.role == R_PRIMARY) ||
- (os.conn != C_STARTING_SYNC_T && ns.conn == C_STARTING_SYNC_T) ||
- (os.conn != C_STARTING_SYNC_S && ns.conn == C_STARTING_SYNC_S) ||
- (os.disk != D_FAILED && ns.disk == D_FAILED))) ||
- (os.conn >= C_CONNECTED && ns.conn == C_DISCONNECTING) ||
- (os.conn == C_CONNECTED && ns.conn == C_VERIFY_S) ||
- (os.conn == C_CONNECTED && ns.conn == C_WF_REPORT_PARAMS);
+ return flags & (CS_HARD | CS_LOCAL_ONLY);
}
-static union drbd_state
-apply_mask_val(union drbd_state os, union drbd_state mask, union drbd_state val)
+static enum drbd_state_rv
+__peer_request(struct drbd_connection *connection, int vnr,
+ union drbd_state mask, union drbd_state val)
{
- union drbd_state ns;
- ns.i = (os.i & ~mask.i) | val.i;
- return ns;
+ enum drbd_state_rv rv = SS_SUCCESS;
+
+ if (connection->cstate[NOW] == C_CONNECTED) {
+ enum drbd_packet cmd = (vnr == -1) ? P_CONN_ST_CHG_REQ : P_STATE_CHG_REQ;
+
+ if (!conn_send_state_req(connection, vnr, cmd, mask, val)) {
+ set_bit(TWOPC_PREPARED, &connection->flags);
+ rv = SS_CW_SUCCESS;
+ }
+ }
+ return rv;
}
-enum drbd_state_rv
-drbd_change_state(struct drbd_device *device, enum chg_state_flags f,
- union drbd_state mask, union drbd_state val)
+static enum drbd_state_rv __peer_reply(struct drbd_connection *connection)
{
- unsigned long flags;
- union drbd_state ns;
- enum drbd_state_rv rv;
+ if (test_and_clear_bit(TWOPC_NO, &connection->flags))
+ return SS_CW_FAILED_BY_PEER;
+ if (test_and_clear_bit(TWOPC_YES, &connection->flags) ||
+ !test_bit(TWOPC_PREPARED, &connection->flags))
+ return SS_CW_SUCCESS;
- spin_lock_irqsave(&device->resource->req_lock, flags);
- ns = apply_mask_val(drbd_read_state(device), mask, val);
- rv = _drbd_set_state(device, ns, f, NULL);
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
+ /* This is DRBD 9.x <-> 8.4 compat code.
+ * Consistent with __peer_request() above:
+ * No more connection: fake success. */
+ if (connection->cstate[NOW] != C_CONNECTED)
+ return SS_SUCCESS;
+ return SS_UNKNOWN_ERROR;
+}
- return rv;
+static bool when_done_lock(struct drbd_resource *resource,
+ unsigned long *irq_flags)
+{
+ write_lock_irqsave(&resource->state_rwlock, *irq_flags);
+ if (!resource->remote_state_change && !test_bit(TWOPC_WORK_PENDING, &resource->flags))
+ return true;
+ write_unlock_irqrestore(&resource->state_rwlock, *irq_flags);
+ return false;
}
/**
- * drbd_force_state() - Impose a change which happens outside our control on our state
- * @device: DRBD device.
- * @mask: mask of state bits to change.
- * @val: value of new state bits.
+ * complete_remote_state_change - Wait for other remote state changes to complete
+ * @resource: DRBD resource.
+ * @irq_flags: IRQ flags from begin_state_change.
*/
-void drbd_force_state(struct drbd_device *device,
- union drbd_state mask, union drbd_state val)
+static void complete_remote_state_change(struct drbd_resource *resource,
+ unsigned long *irq_flags)
{
- drbd_change_state(device, CS_HARD, mask, val);
+ if (resource->remote_state_change) {
+ enum chg_state_flags flags = resource->state_change_flags;
+
+ begin_remote_state_change(resource, irq_flags);
+ for (;;) {
+ long t = twopc_timeout(resource);
+
+ t = wait_event_timeout(resource->twopc_wait,
+ when_done_lock(resource, irq_flags), t);
+ if (t)
+ break;
+ if (when_done_lock(resource, irq_flags)) {
+ drbd_info(resource, "Two-phase commit: "
+ "not woken up in time\n");
+ break;
+ }
+ }
+ __end_remote_state_change(resource, flags);
+ }
}
static enum drbd_state_rv
-_req_st_cond(struct drbd_device *device, union drbd_state mask,
- union drbd_state val)
+change_peer_state(struct drbd_connection *connection, int vnr,
+ union drbd_state mask, union drbd_state val, unsigned long *irq_flags)
{
- union drbd_state os, ns;
- unsigned long flags;
+ struct drbd_resource *resource = connection->resource;
+ enum chg_state_flags flags = resource->state_change_flags | CS_TWOPC;
enum drbd_state_rv rv;
- if (test_and_clear_bit(CL_ST_CHG_SUCCESS, &device->flags))
- return SS_CW_SUCCESS;
-
- if (test_and_clear_bit(CL_ST_CHG_FAIL, &device->flags))
+ if (!expect(resource, flags & CS_SERIALIZE))
return SS_CW_FAILED_BY_PEER;
- spin_lock_irqsave(&device->resource->req_lock, flags);
- os = drbd_read_state(device);
- ns = sanitize_state(device, os, apply_mask_val(os, mask, val), NULL);
- rv = is_valid_transition(os, ns);
- if (rv >= SS_SUCCESS)
- rv = SS_UNKNOWN_ERROR; /* cont waiting, otherwise fail. */
+ complete_remote_state_change(resource, irq_flags);
- if (!cl_wide_st_chg(device, os, ns))
- rv = SS_CW_NO_NEED;
- if (rv == SS_UNKNOWN_ERROR) {
- rv = is_valid_state(device, ns);
- if (rv >= SS_SUCCESS) {
- rv = is_valid_soft_transition(os, ns, first_peer_device(device)->connection);
- if (rv >= SS_SUCCESS)
- rv = SS_UNKNOWN_ERROR; /* cont waiting, otherwise fail. */
- }
+ resource->remote_state_change = true;
+ resource->twopc_reply.initiator_node_id = resource->res_opts.node_id;
+ resource->twopc_reply.tid = 0;
+ /* A reply to the request of an earlier round can still be set, having
+ * arrived after __peer_reply() stopped waiting for it. Discard it before
+ * asking again, so that the answer we read is to this request.
+ */
+ clear_bit(TWOPC_YES, &connection->flags);
+ clear_bit(TWOPC_NO, &connection->flags);
+ begin_remote_state_change(resource, irq_flags);
+ rv = __peer_request(connection, vnr, mask, val);
+ if (rv == SS_CW_SUCCESS) {
+ wait_event(resource->state_wait,
+ ((rv = __peer_reply(connection)) != SS_UNKNOWN_ERROR));
+ clear_bit(TWOPC_PREPARED, &connection->flags);
}
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
-
+ end_remote_state_change(resource, irq_flags, flags);
return rv;
}
-/**
- * drbd_req_state() - Perform an eventually cluster wide state change
- * @device: DRBD device.
- * @mask: mask of state bits to change.
- * @val: value of new state bits.
- * @f: flags
- *
- * Should not be called directly, use drbd_request_state() or
- * _drbd_request_state().
- */
static enum drbd_state_rv
-drbd_req_state(struct drbd_device *device, union drbd_state mask,
- union drbd_state val, enum chg_state_flags f)
+__cluster_wide_request(struct drbd_resource *resource, struct twopc_request *request,
+ u64 reach_immediately)
{
- struct completion done;
- unsigned long flags;
- union drbd_state os, ns;
- enum drbd_state_rv rv;
- void *buffer = NULL;
+ enum drbd_packet cmd = request->cmd;
+ struct drbd_connection *connection;
+ enum drbd_state_rv rv = SS_SUCCESS;
+ u64 im;
- init_completion(&done);
+ for_each_connection_ref(connection, im, resource) {
+ u64 mask;
+ int err;
- if (f & CS_SERIALIZE)
- mutex_lock(device->state_mutex);
- if (f & CS_INHIBIT_MD_IO)
- buffer = drbd_md_get_buffer(device, __func__);
+ /* The prepared set and the replies are cleared where the transaction
+ * ends, so a connection this attempt skips carries none from an
+ * earlier one.
+ */
+ clear_bit(TWOPC_PREPARED, &connection->flags);
- spin_lock_irqsave(&device->resource->req_lock, flags);
- os = drbd_read_state(device);
- ns = sanitize_state(device, os, apply_mask_val(os, mask, val), NULL);
- rv = is_valid_transition(os, ns);
- if (rv < SS_SUCCESS) {
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
- goto abort;
+ if (connection->agreed_pro_version < 110)
+ continue;
+ mask = NODE_MASK(connection->peer_node_id);
+ if (reach_immediately & mask)
+ set_bit(TWOPC_PREPARED, &connection->flags);
+ else
+ continue;
+
+ err = conn_send_twopc_request(connection, request);
+ if (err) {
+ clear_bit(TWOPC_PREPARED, &connection->flags);
+ wake_up(&resource->work.q_wait);
+ /* The send failed, so this connection is done, but
+ * nothing has told the state machine yet: a send error
+ * changes no state on its way out. Say it here, where it
+ * was discovered, as the ping and ping-ack senders do.
+ */
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ continue;
+ }
+ if (cmd == P_TWOPC_PREPARE || cmd == P_TWOPC_PREP_RSZ)
+ drbd_queue_ping(connection);
+ rv = SS_CW_SUCCESS;
}
+ return rv;
+}
- if (cl_wide_st_chg(device, os, ns)) {
- rv = is_valid_state(device, ns);
- if (rv == SS_SUCCESS)
- rv = is_valid_soft_transition(os, ns, first_peer_device(device)->connection);
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
+bool drbd_twopc_between_peer_and_me(struct drbd_connection *connection)
+{
+ const int my_node_id = connection->resource->res_opts.node_id;
+ struct twopc_reply *o = &connection->resource->twopc_reply;
- if (rv < SS_SUCCESS) {
- if (f & CS_VERBOSE)
- print_st_err(device, os, ns, rv);
- goto abort;
- }
+ return ((o->target_node_id == my_node_id || o->target_node_id == -1) &&
+ o->initiator_node_id == connection->peer_node_id) ||
+ ((o->target_node_id == connection->peer_node_id || o->target_node_id == -1) &&
+ o->initiator_node_id == my_node_id);
+}
- if (drbd_send_state_req(first_peer_device(device), mask, val)) {
- rv = SS_CW_FAILED_BY_PEER;
- if (f & CS_VERBOSE)
- print_st_err(device, os, ns, rv);
- goto abort;
- }
+bool cluster_wide_reply_ready(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ bool connect_ready = true;
+ bool have_no = resource->twopc_reply.state_change_failed;
+ bool have_retry = false;
+ bool all_yes = true;
- wait_event(device->state_wait,
- (rv = _req_st_cond(device, mask, val)));
+ if (test_bit(TWOPC_ABORT_LOCAL, &resource->flags))
+ return true;
- if (rv < SS_SUCCESS) {
- if (f & CS_VERBOSE)
- print_st_err(device, os, ns, rv);
- goto abort;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->agreed_pro_version >= 118 &&
+ !idr_is_empty(&resource->devices) &&
+ resource->twopc_reply.is_connect &&
+ drbd_twopc_between_peer_and_me(connection) &&
+ !test_bit(CONN_HANDSHAKE_READY, &connection->flags))
+ connect_ready = false;
+
+ if (!test_bit(TWOPC_PREPARED, &connection->flags))
+ continue;
+ /* A prepared peer that is no longer connected cannot reply; count
+ * that as a retry. A connect/disconnect target is transitional.
+ */
+ if (connection->cstate[NOW] < C_CONNECTED &&
+ !((resource->twopc_reply.is_connect || resource->twopc_reply.is_disconnect) &&
+ connection->peer_node_id == resource->twopc_reply.target_node_id)) {
+ have_retry = true;
+ continue;
}
- spin_lock_irqsave(&device->resource->req_lock, flags);
- ns = apply_mask_val(drbd_read_state(device), mask, val);
- rv = _drbd_set_state(device, ns, f, &done);
- } else {
- rv = _drbd_set_state(device, ns, f, &done);
+ if (test_bit(TWOPC_NO, &connection->flags))
+ have_no = true;
+ if (test_bit(TWOPC_RETRY, &connection->flags))
+ have_retry = true;
+ if (!test_bit(TWOPC_YES, &connection->flags))
+ all_yes = false;
}
+ rcu_read_unlock();
+
+ return have_retry || (connect_ready && (have_no || all_yes));
+}
+
+static enum drbd_state_rv get_cluster_wide_reply(struct drbd_resource *resource,
+ struct change_context *context)
+{
+ struct drbd_connection *connection, *failed_by = NULL;
+ bool handshake_disconnect = false;
+ bool handshake_retry = false;
+ bool have_no = resource->twopc_reply.state_change_failed;
+ bool have_retry = false;
+ enum drbd_state_rv rv = SS_CW_SUCCESS;
+
+ if (test_bit(TWOPC_ABORT_LOCAL, &resource->flags))
+ return SS_CONCURRENT_ST_CHG;
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (resource->twopc_reply.is_connect &&
+ drbd_twopc_between_peer_and_me(connection)) {
+ if (test_bit(CONN_HANDSHAKE_DISCONNECT, &connection->flags))
+ handshake_disconnect = true;
+ if (test_bit(CONN_HANDSHAKE_RETRY, &connection->flags))
+ handshake_retry = true;
+ }
+
+ if (!test_bit(TWOPC_PREPARED, &connection->flags))
+ continue;
+ /* As in cluster_wide_reply_ready(). */
+ if (connection->cstate[NOW] < C_CONNECTED &&
+ !((resource->twopc_reply.is_connect || resource->twopc_reply.is_disconnect) &&
+ connection->peer_node_id == resource->twopc_reply.target_node_id)) {
+ have_retry = true;
+ continue;
+ }
+ if (test_bit(TWOPC_NO, &connection->flags)) {
+ failed_by = connection;
+ have_no = true;
+ }
+ if (test_bit(TWOPC_RETRY, &connection->flags))
+ have_retry = true;
+ }
- if (f & CS_WAIT_COMPLETE && rv == SS_SUCCESS) {
- D_ASSERT(device, current != first_peer_device(device)->connection->worker.task);
- wait_for_completion(&done);
+ if (have_retry)
+ rv = SS_CONCURRENT_ST_CHG;
+ else if (handshake_retry)
+ rv = SS_HANDSHAKE_RETRY;
+ else if (handshake_disconnect)
+ rv = SS_HANDSHAKE_DISCONNECT;
+ else if (have_no) {
+ if (context && failed_by)
+ _drbd_state_err(context, "Declined by peer %s (id: %d), see the kernel log there",
+ rcu_dereference(failed_by->transport.net_conf)->name,
+ failed_by->peer_node_id);
+ rv = SS_CW_FAILED_BY_PEER;
}
+ rcu_read_unlock();
-abort:
- if (buffer)
- drbd_md_put_buffer(device);
- if (f & CS_SERIALIZE)
- mutex_unlock(device->state_mutex);
+ if (rv == SS_CW_SUCCESS && test_bit(TWOPC_RECV_SIZES_ERR, &resource->flags))
+ rv = SS_HANDSHAKE_DISCONNECT;
return rv;
}
-/**
- * _drbd_request_state() - Request a state change (with flags)
- * @device: DRBD device.
- * @mask: mask of state bits to change.
- * @val: value of new state bits.
- * @f: flags
- *
- * Cousin of drbd_request_state(), useful with the CS_WAIT_COMPLETE
- * flag, or when logging of failed state change requests is not desired.
- */
-enum drbd_state_rv
-_drbd_request_state(struct drbd_device *device, union drbd_state mask,
- union drbd_state val, enum chg_state_flags f)
+static bool supports_two_phase_commit(struct drbd_resource *resource)
{
- enum drbd_state_rv rv;
+ struct drbd_connection *connection;
+ bool supported = true;
- wait_event(device->state_wait,
- (rv = drbd_req_state(device, mask, val, f)) != SS_IN_TRANSIENT_STATE);
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->cstate[NOW] != C_CONNECTED)
+ continue;
+ if (connection->agreed_pro_version < 110) {
+ supported = false;
+ break;
+ }
+ }
+ rcu_read_unlock();
- return rv;
+ return supported;
}
-/*
- * We grab drbd_md_get_buffer(), because we don't want to "fail" the disk while
- * there is IO in-flight: the transition into D_FAILED for detach purposes
- * may get misinterpreted as actual IO error in a confused endio function.
- *
- * We wrap it all into wait_event(), to retry in case the drbd_req_state()
- * returns SS_IN_TRANSIENT_STATE.
- *
- * To avoid potential deadlock with e.g. the receiver thread trying to grab
- * drbd_md_get_buffer() while trying to get out of the "transient state", we
- * need to grab and release the meta data buffer inside of that wait_event loop.
- */
-static enum drbd_state_rv
-request_detach(struct drbd_device *device)
+static struct drbd_connection *get_first_connection(struct drbd_resource *resource)
{
- return drbd_req_state(device, NS(disk, D_FAILED),
- CS_VERBOSE | CS_ORDERED | CS_INHIBIT_MD_IO);
+ struct drbd_connection *connection = NULL;
+
+ rcu_read_lock();
+ if (!list_empty(&resource->connections)) {
+ connection = first_connection(resource);
+ kref_get(&connection->kref);
+ }
+ rcu_read_unlock();
+ return connection;
}
-int drbd_request_detach_interruptible(struct drbd_device *device)
+/* That two_primaries is a connection option is one of those things of
+ the past, that should be cleaned up!! it should be a resource config!
+ Here is a inaccurate heuristic */
+static bool multiple_primaries_allowed(struct drbd_resource *resource)
{
- int ret, rv;
-
- drbd_suspend_io(device); /* so no-one is stuck in drbd_al_begin_io */
- wait_event_interruptible(device->state_wait,
- (rv = request_detach(device)) != SS_IN_TRANSIENT_STATE);
- drbd_resume_io(device);
-
- ret = wait_event_interruptible(device->misc_wait,
- device->state.disk != D_FAILED);
+ struct drbd_connection *connection;
+ bool allowed = false;
+ struct drbd_net_conf *nc;
- if (rv == SS_IS_DISKLESS)
- rv = SS_NOTHING_TO_DO;
- if (ret)
- rv = ERR_INTR;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (nc && nc->two_primaries) {
+ allowed = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
- return rv;
+ return allowed;
}
-enum drbd_state_rv
-_drbd_request_state_holding_state_mutex(struct drbd_device *device, union drbd_state mask,
- union drbd_state val, enum chg_state_flags f)
+static enum drbd_state_rv
+check_primaries_distances(struct drbd_resource *resource)
{
- enum drbd_state_rv rv;
+ struct twopc_reply *reply = &resource->twopc_reply;
+ int nr_primaries = hweight64(reply->primary_nodes);
+ u64 common_server;
+
+ if (nr_primaries <= 1)
+ return SS_SUCCESS;
+ if (nr_primaries > 1 && !multiple_primaries_allowed(resource))
+ return SS_TWO_PRIMARIES;
+ /* All primaries directly connected. Good */
+ if (!(reply->primary_nodes & reply->weak_nodes))
+ return SS_SUCCESS;
+
+ /* For virtualization setups with diskless hypervisors (R_PRIMARY) and one
+ or multiple storage servers (R_SECONDARY) allow live-migration between the
+ hypervisors. */
+ common_server = ~reply->weak_nodes;
+ if (common_server) {
+ int node_id;
+ /* Only allow if the new primary is diskless. See also far_away_change()
+ in drbd_receiver.c for the diskless check on the other primary */
+ if ((reply->primary_nodes & NODE_MASK(resource->res_opts.node_id)) &&
+ drbd_have_local_disk(resource))
+ return SS_WEAKLY_CONNECTED;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_connection *connection;
+ struct drbd_net_conf *nc;
+ bool two_primaries;
+
+ if (!(common_server & NODE_MASK(node_id)))
+ continue;
+ connection = drbd_connection_by_node_id(resource, node_id);
+ if (!connection)
+ continue;
- BUG_ON(f & CS_SERIALIZE);
-
- wait_event_cmd(device->state_wait,
- (rv = drbd_req_state(device, mask, val, f)) != SS_IN_TRANSIENT_STATE,
- mutex_unlock(device->state_mutex),
- mutex_lock(device->state_mutex));
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ two_primaries = nc ? nc->two_primaries : false;
+ rcu_read_unlock();
- return rv;
-}
+ if (!two_primaries)
+ return SS_TWO_PRIMARIES;
+ }
-static void print_st(struct drbd_device *device, const char *name, union drbd_state ns)
-{
- drbd_err(device, " %s = { cs:%s ro:%s/%s ds:%s/%s %c%c%c%c%c%c }\n",
- name,
- drbd_conn_str(ns.conn),
- drbd_role_str(ns.role),
- drbd_role_str(ns.peer),
- drbd_disk_str(ns.disk),
- drbd_disk_str(ns.pdsk),
- is_susp(ns) ? 's' : 'r',
- ns.aftr_isp ? 'a' : '-',
- ns.peer_isp ? 'p' : '-',
- ns.user_isp ? 'u' : '-',
- ns.susp_fen ? 'F' : '-',
- ns.susp_nod ? 'N' : '-'
- );
+ return SS_SUCCESS;
+ }
+ return SS_WEAKLY_CONNECTED;
}
-void print_st_err(struct drbd_device *device, union drbd_state os,
- union drbd_state ns, enum drbd_state_rv err)
+static enum drbd_state_rv
+check_ro_cnt_and_primary(struct drbd_resource *resource)
{
- if (err == SS_IN_TRANSIENT_STATE)
- return;
- drbd_err(device, "State change failed: %s\n", drbd_set_st_err_str(err));
- print_st(device, " state", os);
- print_st(device, "wanted", ns);
-}
+ struct twopc_reply *reply = &resource->twopc_reply;
+ struct drbd_connection *connection;
+ enum drbd_state_rv rv = SS_SUCCESS;
+ struct drbd_net_conf *nc;
-static long print_state_change(char *pb, union drbd_state os, union drbd_state ns,
- enum chg_state_flags flags)
-{
- char *pbp;
- pbp = pb;
- *pbp = 0;
+ if (drbd_open_ro_count(resource) == 0)
+ return rv;
- if (ns.role != os.role && flags & CS_DC_ROLE)
- pbp += sprintf(pbp, "role( %s -> %s ) ",
- drbd_role_str(os.role),
- drbd_role_str(ns.role));
- if (ns.peer != os.peer && flags & CS_DC_PEER)
- pbp += sprintf(pbp, "peer( %s -> %s ) ",
- drbd_role_str(os.peer),
- drbd_role_str(ns.peer));
- if (ns.conn != os.conn && flags & CS_DC_CONN)
- pbp += sprintf(pbp, "conn( %s -> %s ) ",
- drbd_conn_str(os.conn),
- drbd_conn_str(ns.conn));
- if (ns.disk != os.disk && flags & CS_DC_DISK)
- pbp += sprintf(pbp, "disk( %s -> %s ) ",
- drbd_disk_str(os.disk),
- drbd_disk_str(ns.disk));
- if (ns.pdsk != os.pdsk && flags & CS_DC_PDSK)
- pbp += sprintf(pbp, "pdsk( %s -> %s ) ",
- drbd_disk_str(os.pdsk),
- drbd_disk_str(ns.pdsk));
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (!nc->two_primaries &&
+ NODE_MASK(connection->peer_node_id) & reply->primary_nodes) {
+ rv = SS_PRIMARY_READER;
+ break;
+ }
+ }
+ rcu_read_unlock();
- return pbp - pb;
+ return rv;
}
-static void drbd_pr_state_change(struct drbd_device *device, union drbd_state os, union drbd_state ns,
- enum chg_state_flags flags)
+long twopc_retry_timeout(struct drbd_resource *resource, int retries)
{
- char pb[300];
- char *pbp = pb;
-
- pbp += print_state_change(pbp, os, ns, flags ^ CS_DC_MASK);
+ struct drbd_connection *connection;
+ int connections = 0;
+ long timeout = 0;
- if (ns.aftr_isp != os.aftr_isp)
- pbp += sprintf(pbp, "aftr_isp( %d -> %d ) ",
- os.aftr_isp,
- ns.aftr_isp);
- if (ns.peer_isp != os.peer_isp)
- pbp += sprintf(pbp, "peer_isp( %d -> %d ) ",
- os.peer_isp,
- ns.peer_isp);
- if (ns.user_isp != os.user_isp)
- pbp += sprintf(pbp, "user_isp( %d -> %d ) ",
- os.user_isp,
- ns.user_isp);
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->cstate[NOW] < C_CONNECTING)
+ continue;
+ connections++;
+ }
+ rcu_read_unlock();
- if (pbp != pb)
- drbd_info(device, "%s\n", pb);
+ if (connections > 0) {
+ if (retries > 5)
+ retries = 5;
+ timeout = resource->res_opts.twopc_retry_timeout *
+ HZ / 10 * connections * (1 << retries);
+ timeout = get_random_u32_below(timeout);
+ }
+ return timeout;
}
-static void conn_pr_state_change(struct drbd_connection *connection, union drbd_state os, union drbd_state ns,
- enum chg_state_flags flags)
+void abort_connect(struct drbd_connection *connection)
{
- char pb[300];
- char *pbp = pb;
-
- pbp += print_state_change(pbp, os, ns, flags);
-
- if (is_susp(ns) != is_susp(os) && flags & CS_DC_SUSP)
- pbp += sprintf(pbp, "susp( %d -> %d ) ",
- is_susp(os),
- is_susp(ns));
+ struct drbd_peer_device *peer_device;
+ int vnr;
- if (pbp != pb)
- drbd_info(connection, "%s\n", pb);
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (test_and_clear_bit(HOLDING_UUID_READ_LOCK, peer_device->flags))
+ up_read_non_owner(&peer_device->device->uuid_sem);
+ clear_bit(INITIAL_STATE_SENT, peer_device->flags);
+ clear_bit(INITIAL_STATE_RECEIVED, peer_device->flags);
+ clear_bit(UUIDS_RECEIVED, peer_device->flags);
+ clear_bit(CURRENT_UUID_RECEIVED, peer_device->flags);
+ }
+ rcu_read_unlock();
}
-
-/**
- * is_valid_state() - Returns an SS_ error code if ns is not valid
- * @device: DRBD device.
- * @ns: State to consider.
- */
-static enum drbd_state_rv
-is_valid_state(struct drbd_device *device, union drbd_state ns)
+static void twopc_phase2(struct drbd_resource *resource,
+ struct twopc_request *request,
+ u64 reach_immediately)
{
- /* See drbd_state_sw_errors in drbd_strings.c */
+ struct drbd_connection *connection;
+ u64 im;
- enum drbd_fencing_p fp;
- enum drbd_state_rv rv = SS_SUCCESS;
- struct net_conf *nc;
+ for_each_connection_ref(connection, im, resource) {
+ u64 mask = NODE_MASK(connection->peer_node_id);
- rcu_read_lock();
- fp = FP_DONT_CARE;
- if (get_ldev(device)) {
- fp = rcu_dereference(device->ldev->disk_conf)->fencing;
- put_ldev(device);
- }
+ if (!(reach_immediately & mask))
+ continue;
- nc = rcu_dereference(first_peer_device(device)->connection->net_conf);
- if (nc) {
- if (!nc->two_primaries && ns.role == R_PRIMARY) {
- if (ns.peer == R_PRIMARY)
- rv = SS_TWO_PRIMARIES;
- else if (conn_highest_peer(first_peer_device(device)->connection) == R_PRIMARY)
- rv = SS_O_VOL_PEER_PRI;
- }
+ /* The peer will not learn the verdict, and nothing else has
+ * told the state machine that this connection is gone.
+ */
+ if (conn_send_twopc_request(connection, request))
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
}
+}
- if (rv <= 0)
- goto out; /* already found a reason to abort */
- else if (ns.role == R_SECONDARY && device->open_cnt)
- rv = SS_DEVICE_IN_USE;
+void drbd_print_cluster_wide_state_change(struct drbd_resource *resource, const char *message,
+ unsigned int tid, unsigned int initiator_node_id, int target_node_id,
+ union drbd_state mask, union drbd_state val)
+{
+ char buffer[150], *b, *end = buffer + sizeof(buffer);
- else if (ns.role == R_PRIMARY && ns.conn < C_CONNECTED && ns.disk < D_UP_TO_DATE)
- rv = SS_NO_UP_TO_DATE_DISK;
+ b = buffer;
+ b += scnprintf(b, end - b, "%u->", initiator_node_id);
+ if (target_node_id == -1)
+ b += scnprintf(b, end - b, "all");
+ else
+ b += scnprintf(b, end - b, "%d", target_node_id);
- else if (fp >= FP_RESOURCE &&
- ns.role == R_PRIMARY && ns.conn < C_CONNECTED && ns.pdsk >= D_UNKNOWN)
- rv = SS_PRIMARY_NOP;
+ if (mask.role)
+ b += scnprintf(b, end - b, " role( %s )", drbd_role_str(val.role));
- else if (ns.role == R_PRIMARY && ns.disk <= D_INCONSISTENT && ns.pdsk <= D_INCONSISTENT)
- rv = SS_NO_UP_TO_DATE_DISK;
+ if (mask.peer)
+ b += scnprintf(b, end - b, " peer( %s )", drbd_role_str(val.peer));
- else if (ns.conn > C_CONNECTED && ns.disk < D_INCONSISTENT)
- rv = SS_NO_LOCAL_DISK;
+ if (mask.conn) {
+ if (val.conn > C_CONNECTED)
+ b += scnprintf(b, end - b, " repl( %s )", drbd_repl_str(val.conn));
+ else
+ b += scnprintf(b, end - b, " conn( %s )", drbd_conn_str(val.conn));
+ }
- else if (ns.conn > C_CONNECTED && ns.pdsk < D_INCONSISTENT)
- rv = SS_NO_REMOTE_DISK;
+ if (mask.disk)
+ b += scnprintf(b, end - b, " disk( %s )", drbd_disk_str(val.disk));
- else if (ns.conn > C_CONNECTED && ns.disk < D_UP_TO_DATE && ns.pdsk < D_UP_TO_DATE)
- rv = SS_NO_UP_TO_DATE_DISK;
+ if (mask.pdsk)
+ b += scnprintf(b, end - b, " pdsk( %s )", drbd_disk_str(val.pdsk));
- else if ((ns.conn == C_CONNECTED ||
- ns.conn == C_WF_BITMAP_S ||
- ns.conn == C_SYNC_SOURCE ||
- ns.conn == C_PAUSED_SYNC_S) &&
- ns.disk == D_OUTDATED)
- rv = SS_CONNECTED_OUTDATES;
+ // Any of "susp-io( user )", "susp-io( quorum )" or "susp-io( uuid )"
+ if (mask.susp)
+ b += scnprintf(b, end - b, " %ssusp-io", val.susp ? "+" : "-");
- else if (nc && (ns.conn == C_VERIFY_S || ns.conn == C_VERIFY_T) &&
- (nc->verify_alg[0] == 0))
- rv = SS_NO_VERIFY_ALG;
+ if (mask.susp_nod)
+ b += scnprintf(b, end - b, " susp-io( %sno-disk )", val.susp_nod ? "+" : "-");
- else if ((ns.conn == C_VERIFY_S || ns.conn == C_VERIFY_T) &&
- first_peer_device(device)->connection->agreed_pro_version < 88)
- rv = SS_NOT_SUPPORTED;
+ if (mask.susp_fen)
+ b += scnprintf(b, end - b, " susp-io( %sfencing )", val.susp_fen ? "+" : "-");
- else if (ns.role == R_PRIMARY && ns.disk < D_UP_TO_DATE && ns.pdsk < D_UP_TO_DATE)
- rv = SS_NO_UP_TO_DATE_DISK;
+ if (mask.user_isp)
+ b += scnprintf(b, end - b, " resync-susp( %suser )", val.user_isp ? "+" : "-");
- else if ((ns.conn == C_STARTING_SYNC_S || ns.conn == C_STARTING_SYNC_T) &&
- ns.pdsk == D_UNKNOWN)
- rv = SS_NEED_CONNECTION;
+ if (mask.peer_isp)
+ b += scnprintf(b, end - b, " resync-susp( %speer )", val.peer_isp ? "+" : "-");
- else if (ns.conn >= C_CONNECTED && ns.pdsk == D_UNKNOWN)
- rv = SS_CONNECTED_OUTDATES;
+ if (mask.aftr_isp)
+ b += scnprintf(b, end - b, " resync-susp( %safter dependency )",
+ val.aftr_isp ? "+" : "-");
-out:
- rcu_read_unlock();
+ if (!mask.i)
+ b += scnprintf(b, end - b, " empty");
- return rv;
+ drbd_info(resource, "%s %u: %s\n", message, tid, buffer);
}
/**
- * is_valid_soft_transition() - Returns an SS_ error code if the state transition is not possible
- * This function limits state transitions that may be declined by DRBD. I.e.
- * user requests (aka soft transitions).
- * @os: old state.
- * @ns: new state.
- * @connection: DRBD connection.
+ * change_cluster_wide_state - Cluster-wide two-phase commit
+ * @change: The callback function that does the actual state change.
+ * @context: State change context.
+ * @tag: State change tag to print in status messages.
+ *
+ * Perform a two-phase commit transaction among all (reachable) nodes in the
+ * cluster. In our transaction model, the initiator of a transaction is also
+ * the coordinator.
+ *
+ * In phase one of the transaction, the coordinator sends all nodes in the
+ * cluster a P_TWOPC_PREPARE packet. Each node replies with either P_TWOPC_YES
+ * if it consents or with P_TWOPC_NO if it denies the transaction. Once all
+ * replies have been received, the coordinator sends all nodes in the cluster a
+ * P_TWOPC_COMMIT or P_TWOPC_ABORT packet to finish the transaction.
+ *
+ * When a node in the cluster is busy with another transaction, it replies with
+ * P_TWOPC_NO. The coordinator is then responsible for retrying the
+ * transaction.
+ *
+ * Since a cluster is not guaranteed to always be fully connected, some nodes
+ * will not be directly reachable from other nodes. In order to still reach
+ * all nodes in the cluster, participants will forward requests to nodes which
+ * haven't received the request yet:
+ *
+ * The nodes_to_reach field in requests indicates which nodes have received the
+ * request already. Before forwarding a request to a peer, a node removes
+ * itself from nodes_to_reach; it then sends the request to all directly
+ * connected nodes in nodes_to_reach.
+ *
+ * If there are redundant paths in the cluster, requests will reach some nodes
+ * more than once. Nodes remember when they are taking part in a transaction;
+ * they detect duplicate requests and reply to them with P_TWOPC_YES packets.
+ * (Transactions are identified by the node id of the initiator and a random,
+ * unique-enough transaction identifier.)
+ *
+ * A configurable timeout determines how long a coordinator or participant will
+ * wait for a transaction to finish. A transaction that times out is assumed
+ * to have aborted.
*/
static enum drbd_state_rv
-is_valid_soft_transition(union drbd_state os, union drbd_state ns, struct drbd_connection *connection)
+change_cluster_wide_state(bool (*change)(struct change_context *, enum change_phase),
+ struct change_context *context, const char *tag)
{
- enum drbd_state_rv rv = SS_SUCCESS;
+ struct drbd_resource *resource = context->resource;
+ unsigned long irq_flags;
+ struct twopc_request request;
+ struct twopc_reply *reply = &resource->twopc_reply;
+ struct drbd_connection *connection, *target_connection = NULL;
+ enum drbd_state_rv rv;
+ u64 reach_immediately;
+ int retries = 1;
+ unsigned long start_time;
+ bool have_peers, prepared_peers;
+
+ begin_state_change(resource, &irq_flags, context->flags | CS_LOCAL_ONLY);
+ resource->state_change_err_str = context->err_str;
+
+ if (local_state_change(context->flags)) {
+ /* Not a cluster-wide state change. */
+ change(context, PH_LOCAL_COMMIT);
+ return end_state_change(resource, &irq_flags, tag);
+ } else {
+ if (!change(context, PH_PREPARE)) {
+ /* Not a cluster-wide state change. */
+ return end_state_change(resource, &irq_flags, tag);
+ }
+ rv = try_state_change(resource);
+ if (rv != SS_SUCCESS) {
+ /* Failure or nothing to do. */
+ /* abort_state_change(resource, &irq_flags); */
+ if (rv == SS_NOTHING_TO_DO)
+ resource->state_change_flags &= ~CS_VERBOSE;
+ return __end_state_change(resource, &irq_flags, rv, tag);
+ }
+ /* Really a cluster-wide state change. */
+ }
+
+ if (!supports_two_phase_commit(resource)) {
+ connection = get_first_connection(resource);
+ rv = SS_SUCCESS;
+ if (connection) {
+ rv = change_peer_state(connection, context->vnr, context->mask, context->val, &irq_flags);
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+ if (rv >= SS_SUCCESS)
+ change(context, PH_84_COMMIT);
+ return __end_state_change(resource, &irq_flags, rv, tag);
+ }
+
+ if (!expect(resource, context->flags & CS_SERIALIZE || context->mask.i == 0)) {
+ rv = SS_CW_FAILED_BY_PEER;
+ return __end_state_change(resource, &irq_flags, rv, tag);
+ }
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (!expect(connection, current != connection->receiver.task)) {
+ rcu_read_unlock();
+ BUG();
+ }
+ }
+ rcu_read_unlock();
- if ((ns.conn == C_STARTING_SYNC_T || ns.conn == C_STARTING_SYNC_S) &&
- os.conn > C_CONNECTED)
- rv = SS_RESYNC_RUNNING;
+retry:
+ if (current == resource->worker.task && resource->remote_state_change)
+ return __end_state_change(resource, &irq_flags, SS_CONCURRENT_ST_CHG, tag);
- if (ns.conn == C_DISCONNECTING && os.conn == C_STANDALONE)
- rv = SS_ALREADY_STANDALONE;
+ complete_remote_state_change(resource, &irq_flags);
+ start_time = jiffies;
+ resource->state_change_err_str = context->err_str;
- if (ns.disk > D_ATTACHING && os.disk == D_DISKLESS)
- rv = SS_IS_DISKLESS;
+ *reply = (struct twopc_reply) { 0 };
- if (ns.conn == C_WF_CONNECTION && os.conn < C_UNCONNECTED)
- rv = SS_NO_NET_CONFIG;
+ reach_immediately = directly_connected_nodes(resource, NOW);
+ if (context->target_node_id != -1) {
+ struct drbd_connection *connection;
- if (ns.disk == D_OUTDATED && os.disk < D_OUTDATED && os.disk != D_ATTACHING)
- rv = SS_LOWER_THAN_OUTDATED;
+ /* Fail if the target node is no longer directly reachable. */
+ connection = drbd_get_connection_by_node_id(resource, context->target_node_id);
+ if (!connection) {
+ rv = SS_NEED_CONNECTION;
+ return __end_state_change(resource, &irq_flags, rv, tag);
+ }
- if (ns.conn == C_DISCONNECTING && os.conn == C_UNCONNECTED)
- rv = SS_IN_TRANSIENT_STATE;
+ if (!(connection->cstate[NOW] == C_CONNECTED ||
+ (connection->cstate[NOW] == C_CONNECTING &&
+ context->mask.conn == conn_MASK &&
+ context->val.conn == C_CONNECTED))) {
+ rv = SS_NEED_CONNECTION;
+ kref_put(&connection->kref, drbd_destroy_connection);
+ return __end_state_change(resource, &irq_flags, rv, tag);
+ }
+ target_connection = connection;
- /* While establishing a connection only allow cstate to change.
- Delay/refuse role changes, detach attach etc... (they do not touch cstate) */
- if (test_bit(STATE_SENT, &connection->flags) &&
- !((ns.conn == C_WF_REPORT_PARAMS && os.conn == C_WF_CONNECTION) ||
- (ns.conn >= C_CONNECTED && os.conn == C_WF_REPORT_PARAMS)))
- rv = SS_IN_TRANSIENT_STATE;
+ /* For connect transactions, add the target node id. */
+ reach_immediately |= NODE_MASK(context->target_node_id);
+ }
- /* Do not promote during resync handshake triggered by "force primary".
- * This is a hack. It should really be rejected by the peer during the
- * cluster wide state change request. */
- if (os.role != R_PRIMARY && ns.role == R_PRIMARY
- && ns.pdsk == D_UP_TO_DATE
- && ns.disk != D_UP_TO_DATE && ns.disk != D_DISKLESS
- && (ns.conn <= C_WF_SYNC_UUID || ns.conn != os.conn))
- rv = SS_IN_TRANSIENT_STATE;
+ do
+ reply->tid = get_random_u32();
+ while (!reply->tid);
+
+ clear_bit(TWOPC_RECV_SIZES_ERR, &resource->flags);
+ request.tid = reply->tid;
+ request.initiator_node_id = resource->res_opts.node_id;
+ request.target_node_id = context->target_node_id;
+ request.nodes_to_reach = ~(reach_immediately | NODE_MASK(resource->res_opts.node_id));
+ request.vnr = context->vnr;
+ request.cmd = P_TWOPC_PREPARE;
+ request.flags = TWOPC_HAS_REACHABLE;
+
+ resource->twopc.type = TWOPC_STATE_CHANGE;
+ resource->twopc.state_change.mask = context->mask;
+ resource->twopc.state_change.val = context->val;
+ resource->twopc.state_change.primary_nodes = 0;
+ resource->twopc.state_change.reachable_nodes = 0;
+ resource->twopc_parent_nodes = 0;
+ resource->remote_state_change = true;
+
+ drbd_print_cluster_wide_state_change(resource, "Preparing cluster-wide state change",
+ request.tid, resource->res_opts.node_id, context->target_node_id,
+ context->mask, context->val);
+
+ reply->initiator_node_id = resource->res_opts.node_id;
+ reply->target_node_id = context->target_node_id;
+
+ reply->reachable_nodes = directly_connected_nodes(resource, NOW) |
+ NODE_MASK(resource->res_opts.node_id);
+ if (context->mask.conn == conn_MASK && context->val.conn == C_CONNECTED) {
+ reply->reachable_nodes |= NODE_MASK(context->target_node_id);
+ reply->target_reachable_nodes = reply->reachable_nodes;
+ reply->is_connect = 1;
+ drbd_init_connect_state(target_connection);
+ } else if (context->mask.conn == conn_MASK && context->val.conn == C_DISCONNECTING) {
+ reply->target_reachable_nodes = NODE_MASK(context->target_node_id);
+ reply->reachable_nodes &= ~reply->target_reachable_nodes;
+ reply->is_disconnect = 1;
+ } else {
+ reply->target_reachable_nodes = reply->reachable_nodes;
+ }
- if ((ns.conn == C_VERIFY_S || ns.conn == C_VERIFY_T) && os.conn < C_CONNECTED)
- rv = SS_NEED_CONNECTION;
+ D_ASSERT(resource, !test_bit(TWOPC_WORK_PENDING, &resource->flags));
+ begin_remote_state_change(resource, &irq_flags);
+ rv = __cluster_wide_request(resource, &request, reach_immediately);
- if ((ns.conn == C_VERIFY_S || ns.conn == C_VERIFY_T) &&
- ns.conn != os.conn && os.conn > C_CONNECTED)
- rv = SS_RESYNC_RUNNING;
+ /* Some peers may have accepted the P_TWOPC_PREPARE even if the change
+ * as a whole fails below; remember so that we abort it on them.
+ */
+ prepared_peers = rv == SS_CW_SUCCESS;
- if ((ns.conn == C_STARTING_SYNC_S || ns.conn == C_STARTING_SYNC_T) &&
- os.conn < C_CONNECTED)
+ /* If we are changing state attached to a particular connection then we
+ * expect that connection to remain connected. A failure to send
+ * P_TWOPC_PREPARE on that connection is a failure for the whole
+ * cluster-wide state change. */
+ if (target_connection && !test_bit(TWOPC_PREPARED, &target_connection->flags))
rv = SS_NEED_CONNECTION;
- if ((ns.conn == C_SYNC_TARGET || ns.conn == C_SYNC_SOURCE)
- && os.conn < C_WF_REPORT_PARAMS)
- rv = SS_NEED_CONNECTION; /* No NetworkFailure -> SyncTarget etc... */
+ have_peers = rv == SS_CW_SUCCESS;
+ if (have_peers) {
+ long t;
+
+ if (context->mask.conn == conn_MASK && context->val.conn == C_CONNECTED &&
+ target_connection->agreed_pro_version >= 118)
+ conn_connect2(target_connection);
+
+ if (context->flags & CS_NO_SIGNALS)
+ t = wait_event_timeout(resource->state_wait,
+ cluster_wide_reply_ready(resource),
+ twopc_timeout(resource));
+ else
+ t = wait_event_interruptible_timeout(resource->state_wait,
+ cluster_wide_reply_ready(resource),
+ twopc_timeout(resource));
+ if (t > 0)
+ rv = get_cluster_wide_reply(resource, context);
+ else
+ rv = t == 0 ? SS_TIMEOUT : SS_INTERRUPTED;
+
+ /* while waiting for the replies, reach_immediately might have changed. */
+ reach_immediately = directly_connected_nodes(resource, NOW);
+ if (target_connection && target_connection->cstate[NOW] == C_CONNECTING)
+ reach_immediately |= NODE_MASK(context->target_node_id);
+
+ request.nodes_to_reach =
+ ~(reach_immediately | NODE_MASK(resource->res_opts.node_id));
+
+ if (rv == SS_CW_SUCCESS) {
+ u64 directly_reachable = reach_immediately |
+ NODE_MASK(resource->res_opts.node_id);
+
+ if (context->mask.conn == conn_MASK && context->val.conn == C_DISCONNECTING)
+ directly_reachable &= ~NODE_MASK(context->target_node_id);
+
+ if ((context->mask.role == role_MASK && context->val.role == R_PRIMARY) ||
+ (context->mask.role != role_MASK && resource->role[NOW] == R_PRIMARY)) {
+ reply->primary_nodes |= NODE_MASK(resource->res_opts.node_id);
+ if (drbd_res_data_accessible(resource))
+ reply->weak_nodes |= ~directly_reachable;
+ }
- if (ns.conn == C_DISCONNECTING && ns.pdsk == D_OUTDATED &&
- os.conn < C_CONNECTED && os.pdsk > D_OUTDATED)
- rv = SS_OUTDATE_WO_CONN;
+ /*
+ * When a node is Primary and has access to UpToDate data, it sets
+ * weak_nodes to the mask of those it is not connected to. This includes the
+ * bits for nodes which are not configured, so will always have some set
+ * bits. Thus if there is a Primary node and no bits are set in weak_nodes,
+ * the Primary cannot have access to UpToDate data.
+ */
+ if (reply->primary_nodes && !reply->weak_nodes)
+ request.flags |= TWOPC_PRI_INCAPABLE;
+
+ drbd_info(resource, "State change %u: primary_nodes=%lX, weak_nodes=%lX\n",
+ reply->tid, (unsigned long)reply->primary_nodes,
+ (unsigned long)reply->weak_nodes);
+
+ if ((context->mask.role == role_MASK && context->val.role == R_PRIMARY) ||
+ (context->mask.conn == conn_MASK && context->val.conn == C_CONNECTED))
+ rv = check_primaries_distances(resource);
+
+ if (rv >= SS_SUCCESS &&
+ context->mask.conn == conn_MASK && context->val.conn == C_CONNECTED)
+ rv = check_ro_cnt_and_primary(resource);
+
+ if (!(context->mask.conn == conn_MASK && context->val.conn == C_DISCONNECTING) ||
+ (reply->reachable_nodes & reply->target_reachable_nodes)) {
+ /* The cluster is still connected after this
+ * transaction: either this transaction does
+ * not disconnect a connection, or there are
+ * redundant connections. */
+
+ u64 m;
+
+ m = reply->reachable_nodes | reply->target_reachable_nodes;
+ reply->reachable_nodes = m;
+ reply->target_reachable_nodes = m;
+ } else {
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ int node_id = connection->peer_node_id;
+
+ if (node_id == context->target_node_id) {
+ drbd_info(connection, "Cluster is now split\n");
+ break;
+ }
+ }
+ rcu_read_unlock();
+ }
- return rv;
-}
+ resource->twopc.state_change.primary_nodes = reply->primary_nodes;
+ resource->twopc.state_change.reachable_nodes =
+ reply->target_reachable_nodes;
+ }
-static enum drbd_state_rv
-is_valid_conn_transition(enum drbd_conns oc, enum drbd_conns nc)
-{
- /* no change -> nothing to do, at least for the connection part */
- if (oc == nc)
- return SS_NOTHING_TO_DO;
+ /* On the state return codes below the peer never gave a verdict, so
+ * no initial state is on its way and waiting for it is a stall.
+ */
+ if (context->mask.conn == conn_MASK && context->val.conn == C_CONNECTED &&
+ target_connection->agreed_pro_version >= 118 &&
+ rv != SS_TIMEOUT && rv != SS_INTERRUPTED && rv != SS_CONCURRENT_ST_CHG) {
+ wait_initial_states_received(target_connection);
- /* disconnect of an unconfigured connection does not make sense */
- if (oc == C_STANDALONE && nc == C_DISCONNECTING)
- return SS_ALREADY_STANDALONE;
+ if (rv >= SS_SUCCESS && test_bit(TWOPC_RECV_SIZES_ERR, &resource->flags))
+ rv = SS_HANDSHAKE_DISCONNECT;
+ }
+ }
- /* from C_STANDALONE, we start with C_UNCONNECTED */
- if (oc == C_STANDALONE && nc != C_UNCONNECTED)
- return SS_NEED_CONNECTION;
+ request.cmd = rv >= SS_SUCCESS ? P_TWOPC_COMMIT : P_TWOPC_ABORT;
+ if (rv < SS_SUCCESS && target_connection)
+ abort_connect(target_connection);
+
+ if ((rv == SS_TIMEOUT || rv == SS_CONCURRENT_ST_CHG) &&
+ !(context->flags & CS_DONT_RETRY)) {
+ long timeout = twopc_retry_timeout(resource, retries++);
+
+ drbd_info(resource, "Retrying cluster-wide state change after %ums\n",
+ jiffies_to_msecs(timeout));
+ if (have_peers)
+ twopc_phase2(resource, &request, reach_immediately);
+ if (target_connection) {
+ kref_put(&target_connection->kref, drbd_destroy_connection);
+ target_connection = NULL;
+ }
+ clear_remote_state_change(resource);
+ schedule_timeout_interruptible(timeout);
+ end_remote_state_change(resource, &irq_flags, context->flags | CS_TWOPC);
+ goto retry;
+ }
- /* When establishing a connection we need to go through WF_REPORT_PARAMS!
- Necessary to do the right thing upon invalidate-remote on a disconnected resource */
- if (oc < C_WF_REPORT_PARAMS && nc >= C_CONNECTED)
- return SS_NEED_CONNECTION;
+ if (rv >= SS_SUCCESS)
+ drbd_info(resource, "Committing cluster-wide state change %u (%ums)\n",
+ request.tid,
+ jiffies_to_msecs(jiffies - start_time));
+ else
+ drbd_info(resource, "Aborting cluster-wide state change %u (%ums) rv = %d\n",
+ request.tid,
+ jiffies_to_msecs(jiffies - start_time),
+ rv);
+
+ if (have_peers && context->change_local_state_last) {
+ set_bit(TWOPC_STATE_CHANGE_PENDING, &resource->flags);
+ twopc_phase2(resource, &request, reach_immediately);
+ }
- /* After a network error only C_UNCONNECTED or C_DISCONNECTING may follow. */
- if (oc >= C_TIMEOUT && oc <= C_TEAR_DOWN && nc != C_UNCONNECTED && nc != C_DISCONNECTING)
- return SS_IN_TRANSIENT_STATE;
+ end_remote_state_change(resource, &irq_flags, context->flags | CS_TWOPC);
+ clear_bit(TWOPC_STATE_CHANGE_PENDING, &resource->flags);
+ if (rv >= SS_SUCCESS) {
+ change(context, PH_COMMIT);
+ rv = end_state_change(resource, &irq_flags, tag);
+ if (rv < SS_SUCCESS)
+ drbd_err(resource, "FATAL: Local commit of already committed %u failed!\n",
+ request.tid);
+ } else {
+ abort_state_change(resource, &irq_flags);
+ }
- /* After C_DISCONNECTING only C_STANDALONE may follow */
- if (oc == C_DISCONNECTING && nc != C_STANDALONE)
- return SS_IN_TRANSIENT_STATE;
+ if (have_peers && !context->change_local_state_last)
+ twopc_phase2(resource, &request, reach_immediately);
- return SS_SUCCESS;
+ /* If peers accepted the prepare but the connection the change is
+ * attached to did not (rv == SS_NEED_CONNECTION), have_peers is false
+ * and the verdict was not sent above. request.cmd is P_TWOPC_ABORT in
+ * that case; tell the peers that did prepare to abort now. Otherwise
+ * they hold the transaction until their twopc-timeout and reject every
+ * other cluster-wide state change in the meantime.
+ */
+ if (!have_peers && prepared_peers)
+ twopc_phase2(resource, &request, reach_immediately);
+
+ if (target_connection) {
+ kref_put(&target_connection->kref, drbd_destroy_connection);
+ }
+ return rv;
}
+/* Which participants of a size transaction are expected to have a disk, as far
+ * as this node can tell: itself, and every configured peer this node keeps a
+ * bitmap for, connected or not. A peer configured with "bitmap no" is a client
+ * by intent. Anything less certain counts: a peer with a bitmap that is
+ * detached or not connected, and a participant this node has no peer device
+ * for, since nothing here says it is a client.
+ */
+static u64 diskful_participants(struct drbd_device *device, u64 reachable_nodes)
+{
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
+ u64 known = NODE_MASK(resource->res_opts.node_id);
+ u64 diskful = known;
-/**
- * is_valid_transition() - Returns an SS_ error code if the state transition is not possible
- * This limits hard state transitions. Hard state transitions are facts there are
- * imposed on DRBD by the environment. E.g. disk broke or network broke down.
- * But those hard state transitions are still not allowed to do everything.
- * @ns: new state.
- * @os: old state.
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ known |= NODE_MASK(peer_device->node_id);
+ if (want_bitmap(peer_device))
+ diskful |= NODE_MASK(peer_device->node_id);
+ }
+ rcu_read_unlock();
+
+ return diskful | (reachable_nodes & ~known);
+}
+
+/* A size change nobody asked for reports its progress as debug detail: on a
+ * node with thousands of volumes, one prepare and one abort per volume is what
+ * a connect storm would otherwise leave in the log for changing nothing. What
+ * it does change is logged either way.
*/
-static enum drbd_state_rv
-is_valid_transition(union drbd_state os, union drbd_state ns)
+#define size_change_info(automatic, device, fmt, args...) do { \
+ if (automatic) \
+ dynamic_drbd_dbg(device, fmt, ## args); \
+ else \
+ drbd_info(device, fmt, ## args); \
+} while (0)
+
+/* automatic: this is drbd_auto_grow(), not an administrative resize. Commit
+ * only what every participant applies, keep the log quiet, and leave a
+ * transaction that could not be run to the next arming edge.
+ */
+enum determine_dev_size
+change_cluster_wide_device_size(struct drbd_device *device,
+ sector_t local_max_size,
+ uint64_t new_user_size,
+ enum dds_flags dds_flags,
+ bool automatic,
+ struct drbd_resize_parms *rs)
{
+ struct drbd_resource *resource = device->resource;
+ struct twopc_reply *reply = &resource->twopc_reply;
+ struct twopc_request request;
+ unsigned long start_time;
+ unsigned long irq_flags;
enum drbd_state_rv rv;
+ enum determine_dev_size dd;
+ u64 reach_immediately;
+ bool have_peers, commit_it, implicit_shrink = false;
+ sector_t new_size = 0;
+ int retries = 1;
+
+retry:
+ rv = drbd_support_2pc_resize(resource);
+ if (rv < SS_SUCCESS)
+ return DS_2PC_NOT_SUPPORTED;
- rv = is_valid_conn_transition(os.conn, ns.conn);
+ state_change_lock(resource, &irq_flags, CS_VERBOSE | CS_LOCAL_ONLY);
+ rcu_read_lock();
+ complete_remote_state_change(resource, &irq_flags);
+ start_time = jiffies;
+ reach_immediately = directly_connected_nodes(resource, NOW);
+
+ *reply = (struct twopc_reply) { 0 };
+ reply->vnr = device->vnr;
+
+ do
+ reply->tid = get_random_u32();
+ while (!reply->tid);
+
+ clear_bit(TWOPC_RECV_SIZES_ERR, &resource->flags);
+ request.tid = reply->tid;
+ request.initiator_node_id = resource->res_opts.node_id;
+ request.target_node_id = -1;
+ request.nodes_to_reach = ~(reach_immediately | NODE_MASK(resource->res_opts.node_id));
+ request.vnr = device->vnr;
+ request.cmd = P_TWOPC_PREP_RSZ;
+ request.flags = 0;
+ resource->twopc.type = TWOPC_RESIZE;
+ resource->twopc.resize.dds_flags = dds_flags;
+ resource->twopc.resize.user_size = new_user_size;
+ resource->twopc.resize.diskful_primary_nodes = 0;
+ resource->twopc.resize.new_size = 0;
+ resource->twopc_parent_nodes = 0;
+ resource->remote_state_change = true;
+
+ reply->initiator_node_id = resource->res_opts.node_id;
+ reply->target_node_id = -1;
+ reply->max_possible_size = local_max_size;
+ reply->reachable_nodes = reach_immediately | NODE_MASK(resource->res_opts.node_id);
+ reply->target_reachable_nodes = reply->reachable_nodes;
+ reply->common_reachable_nodes = reply->reachable_nodes;
+ if (resource->role[NOW] == R_PRIMARY)
+ reply->diskful_primary_nodes = NODE_MASK(resource->res_opts.node_id);
+ rcu_read_unlock();
+ state_change_unlock(resource, &irq_flags);
- /* we cannot fail (again) if we already detached */
- if (ns.disk == D_FAILED && os.disk == D_DISKLESS)
- rv = SS_IS_DISKLESS;
+ size_change_info(automatic, device, "Preparing cluster-wide size change %u "
+ "(local_max_size = %llu KB, user_cap = %llu KB)\n",
+ request.tid,
+ (unsigned long long)local_max_size >> 1,
+ (unsigned long long)new_user_size >> 1);
- return rv;
-}
+ rv = __cluster_wide_request(resource, &request, reach_immediately);
-static void print_sanitize_warnings(struct drbd_device *device, enum sanitize_state_warnings warn)
-{
- static const char *msg_table[] = {
- [NO_WARNING] = "",
- [ABORTED_ONLINE_VERIFY] = "Online-verify aborted.",
- [ABORTED_RESYNC] = "Resync aborted.",
- [CONNECTION_LOST_NEGOTIATING] = "Connection lost while negotiating, no data!",
- [IMPLICITLY_UPGRADED_DISK] = "Implicitly upgraded disk",
- [IMPLICITLY_UPGRADED_PDSK] = "Implicitly upgraded pdsk",
- };
+ have_peers = rv == SS_CW_SUCCESS;
+ if (have_peers) {
+ if (wait_event_timeout(resource->state_wait,
+ cluster_wide_reply_ready(resource),
+ twopc_timeout(resource)))
+ rv = get_cluster_wide_reply(resource, NULL);
+ else
+ rv = SS_TIMEOUT;
- if (warn != NO_WARNING)
- drbd_warn(device, "%s\n", msg_table[warn]);
-}
+ if (!automatic && (rv == SS_TIMEOUT || rv == SS_CONCURRENT_ST_CHG)) {
+ long timeout = twopc_retry_timeout(resource, retries++);
-/**
- * sanitize_state() - Resolves implicitly necessary additional changes to a state transition
- * @device: DRBD device.
- * @os: old state.
- * @ns: new state.
- * @warn: placeholder for returned state warning.
- *
- * When we loose connection, we have to set the state of the peers disk (pdsk)
- * to D_UNKNOWN. This rule and many more along those lines are in this function.
- */
-static union drbd_state sanitize_state(struct drbd_device *device, union drbd_state os,
- union drbd_state ns, enum sanitize_state_warnings *warn)
-{
- enum drbd_fencing_p fp;
- enum drbd_disk_state disk_min, disk_max, pdsk_min, pdsk_max;
+ drbd_info(device, "Retrying cluster-wide size change after %ums\n",
+ jiffies_to_msecs(timeout));
- if (warn)
- *warn = NO_WARNING;
+ request.cmd = P_TWOPC_ABORT;
+ twopc_phase2(resource, &request, reach_immediately);
- fp = FP_DONT_CARE;
- if (get_ldev(device)) {
- rcu_read_lock();
- fp = rcu_dereference(device->ldev->disk_conf)->fencing;
- rcu_read_unlock();
- put_ldev(device);
+ clear_remote_state_change(resource);
+ schedule_timeout_interruptible(timeout);
+ goto retry;
+ }
}
- /* Implications from connection to peer and peer_isp */
- if (ns.conn < C_CONNECTED) {
- ns.peer_isp = 0;
- ns.peer = R_UNKNOWN;
- if (ns.pdsk > D_UNKNOWN || ns.pdsk < D_INCONSISTENT)
- ns.pdsk = D_UNKNOWN;
- }
+ if (rv >= SS_SUCCESS) {
+ new_size = drbd_new_dev_size(device, reply->max_possible_size,
+ new_user_size, dds_flags | DDSF_2PC);
+ commit_it = new_size != get_capacity(device->vdisk);
+
+ if (commit_it && new_size < get_capacity(device->vdisk) &&
+ new_size != new_user_size) {
+ /* Nobody asked for a volume this small: some participant
+ * has less space than the cluster agreed on, and taking
+ * that as the new size would truncate a volume that may
+ * be in use. drbd_new_dev_size() returns a requested
+ * size when it fits, so this also covers a --size the
+ * cluster can not serve. Shrinking stays an explicit --size.
+ */
+ drbd_err(device, "Not shrinking to %llu sectors: a participant lost space and no --size asked for that\n",
+ (unsigned long long)new_size);
+ commit_it = false;
+ implicit_shrink = true;
+ }
- /* Clear the aftr_isp when becoming unconfigured */
- if (ns.conn == C_STANDALONE && ns.disk == D_DISKLESS && ns.role == R_SECONDARY)
- ns.aftr_isp = 0;
+ if (commit_it && automatic) {
+ u64 need = diskful_participants(device, reply->reachable_nodes);
- /* An implication of the disk states onto the connection state */
- /* Abort resync if a disk fails/detaches */
- if (ns.conn > C_CONNECTED && (ns.disk <= D_FAILED || ns.pdsk <= D_FAILED)) {
- if (warn)
- *warn = ns.conn == C_VERIFY_S || ns.conn == C_VERIFY_T ?
- ABORTED_ONLINE_VERIFY : ABORTED_RESYNC;
- ns.conn = C_CONNECTED;
- }
+ /* A node that can not see the whole cluster derives
+ * the size again on apply, and one from before that
+ * took DDSF_2PC there keeps the size it had. A diskless
+ * one takes the agreed size as it is, on every version,
+ * so what a client sees decides no size.
+ */
+ if ((reply->common_reachable_nodes & need) != need) {
+ dynamic_drbd_dbg(device, "Not growing: of 0x%llx only 0x%llx are seen by every node\n",
+ (unsigned long long)need,
+ (unsigned long long)reply->common_reachable_nodes);
+ commit_it = false;
+ /* This is about the cluster's connectivity, not
+ * about its space: it can heal between nodes
+ * this one sees nothing of, so let the next
+ * arming edge ask again.
+ */
+ device->auto_grow_asked = 0;
+ }
+ }
- /* Connection breaks down before we finished "Negotiating" */
- if (ns.conn < C_CONNECTED && ns.disk == D_NEGOTIATING &&
- get_ldev_if_state(device, D_NEGOTIATING)) {
- if (device->ed_uuid == device->ldev->md.uuid[UI_CURRENT]) {
- ns.disk = device->new_state_tmp.disk;
- ns.pdsk = device->new_state_tmp.pdsk;
+ if (commit_it) {
+ resource->twopc.resize.new_size = new_size;
+ resource->twopc.resize.diskful_primary_nodes = reply->diskful_primary_nodes;
+ drbd_info(device, "Committing cluster-wide size change %u (%ums)\n",
+ request.tid,
+ jiffies_to_msecs(jiffies - start_time));
} else {
- if (warn)
- *warn = CONNECTION_LOST_NEGOTIATING;
- ns.disk = D_DISKLESS;
- ns.pdsk = D_UNKNOWN;
+ size_change_info(automatic, device, "Aborting cluster-wide size change %u (%ums) size unchanged\n",
+ request.tid,
+ jiffies_to_msecs(jiffies - start_time));
}
- put_ldev(device);
- }
-
- /* D_CONSISTENT and D_OUTDATED vanish when we get connected */
- if (ns.conn >= C_CONNECTED && ns.conn < C_AHEAD) {
- if (ns.disk == D_CONSISTENT || ns.disk == D_OUTDATED)
- ns.disk = D_UP_TO_DATE;
- if (ns.pdsk == D_CONSISTENT || ns.pdsk == D_OUTDATED)
- ns.pdsk = D_UP_TO_DATE;
- }
-
- /* Implications of the connection state on the disk states */
- disk_min = D_DISKLESS;
- disk_max = D_UP_TO_DATE;
- pdsk_min = D_INCONSISTENT;
- pdsk_max = D_UNKNOWN;
- switch ((enum drbd_conns)ns.conn) {
- case C_WF_BITMAP_T:
- case C_PAUSED_SYNC_T:
- case C_STARTING_SYNC_T:
- case C_WF_SYNC_UUID:
- case C_BEHIND:
- disk_min = D_INCONSISTENT;
- disk_max = D_OUTDATED;
- pdsk_min = D_UP_TO_DATE;
- pdsk_max = D_UP_TO_DATE;
- break;
- case C_VERIFY_S:
- case C_VERIFY_T:
- disk_min = D_UP_TO_DATE;
- disk_max = D_UP_TO_DATE;
- pdsk_min = D_UP_TO_DATE;
- pdsk_max = D_UP_TO_DATE;
- break;
- case C_CONNECTED:
- disk_min = D_DISKLESS;
- disk_max = D_UP_TO_DATE;
- pdsk_min = D_DISKLESS;
- pdsk_max = D_UP_TO_DATE;
- break;
- case C_WF_BITMAP_S:
- case C_PAUSED_SYNC_S:
- case C_STARTING_SYNC_S:
- case C_AHEAD:
- disk_min = D_UP_TO_DATE;
- disk_max = D_UP_TO_DATE;
- pdsk_min = D_INCONSISTENT;
- pdsk_max = D_CONSISTENT; /* D_OUTDATED would be nice. But explicit outdate necessary*/
- break;
- case C_SYNC_TARGET:
- disk_min = D_INCONSISTENT;
- disk_max = D_INCONSISTENT;
- pdsk_min = D_UP_TO_DATE;
- pdsk_max = D_UP_TO_DATE;
- break;
- case C_SYNC_SOURCE:
- disk_min = D_UP_TO_DATE;
- disk_max = D_UP_TO_DATE;
- pdsk_min = D_INCONSISTENT;
- pdsk_max = D_INCONSISTENT;
- break;
- case C_STANDALONE:
- case C_DISCONNECTING:
- case C_UNCONNECTED:
- case C_TIMEOUT:
- case C_BROKEN_PIPE:
- case C_NETWORK_FAILURE:
- case C_PROTOCOL_ERROR:
- case C_TEAR_DOWN:
- case C_WF_CONNECTION:
- case C_WF_REPORT_PARAMS:
- case C_MASK:
- break;
- }
- if (ns.disk > disk_max)
- ns.disk = disk_max;
-
- if (ns.disk < disk_min) {
- if (warn)
- *warn = IMPLICITLY_UPGRADED_DISK;
- ns.disk = disk_min;
+ } else {
+ commit_it = false;
+ size_change_info(automatic, device, "Aborting cluster-wide size change %u (%ums) rv = %d\n",
+ request.tid,
+ jiffies_to_msecs(jiffies - start_time),
+ rv);
}
- if (ns.pdsk > pdsk_max)
- ns.pdsk = pdsk_max;
- if (ns.pdsk < pdsk_min) {
- if (warn)
- *warn = IMPLICITLY_UPGRADED_PDSK;
- ns.pdsk = pdsk_min;
+ request.cmd = commit_it ? P_TWOPC_COMMIT : P_TWOPC_ABORT;
+ if (have_peers) {
+ /* A write queued behind the COMMIT on the data socket reaches
+ * the peer only after its bitmap write, and our own drain in
+ * drbd_determine_dev_size() would wait for that. Reads too:
+ * this is not a receiver, and a read in flight to a peer is
+ * answered only after the peer's bitmap write as well.
+ * Not earlier: a request sent during the prepare phase sits
+ * behind the prepare on the peer's data socket and is answered
+ * once that receiver has replied, so this drain covers it, and
+ * a transaction that does not commit freezes nothing.
+ */
+ if (commit_it) {
+ write_lock_irq(&resource->state_rwlock);
+ twopc_suspend_io(device);
+ write_unlock_irq(&resource->state_rwlock);
+ twopc_drain_io(device, READ_AND_WRITE);
+ }
+ twopc_phase2(resource, &request, reach_immediately);
}
- if (fp == FP_STONITH &&
- (ns.role == R_PRIMARY && ns.conn < C_CONNECTED && ns.pdsk > D_OUTDATED) &&
- !(os.role == R_PRIMARY && os.conn < C_CONNECTED && os.pdsk > D_OUTDATED))
- ns.susp_fen = 1; /* Suspend IO while fence-peer handler runs (peer lost) */
+ if (commit_it) {
+ struct twopc_resize *tr = &resource->twopc.resize;
- if (device->resource->res_opts.on_no_data == OND_SUSPEND_IO &&
- (ns.role == R_PRIMARY && ns.disk < D_UP_TO_DATE && ns.pdsk < D_UP_TO_DATE) &&
- !(os.role == R_PRIMARY && os.disk < D_UP_TO_DATE && os.pdsk < D_UP_TO_DATE))
- ns.susp_nod = 1; /* Suspend IO while no data available (no accessible data available) */
+ tr->diskful_primary_nodes = reply->diskful_primary_nodes;
+ tr->new_size = new_size;
+ tr->dds_flags = dds_flags;
+ tr->user_size = new_user_size;
- if (ns.aftr_isp || ns.peer_isp || ns.user_isp) {
- if (ns.conn == C_SYNC_SOURCE)
- ns.conn = C_PAUSED_SYNC_S;
- if (ns.conn == C_SYNC_TARGET)
- ns.conn = C_PAUSED_SYNC_T;
+ dd = drbd_commit_size_change(device, rs, reach_immediately);
} else {
- if (ns.conn == C_PAUSED_SYNC_S)
- ns.conn = C_SYNC_SOURCE;
- if (ns.conn == C_PAUSED_SYNC_T)
- ns.conn = C_SYNC_TARGET;
+ if (rv == SS_CW_FAILED_BY_PEER)
+ dd = DS_2PC_NOT_SUPPORTED;
+ else if (implicit_shrink)
+ dd = DS_ERROR_SHRINK;
+ else if (rv >= SS_SUCCESS)
+ dd = DS_UNCHANGED;
+ else
+ dd = DS_2PC_ERR;
}
- return ns;
-}
-
-void drbd_resume_al(struct drbd_device *device)
-{
- if (test_and_clear_bit(AL_SUSPENDED, &device->flags))
- drbd_info(device, "Resumed AL updates\n");
+ clear_remote_state_change(resource);
+ return dd;
}
+#undef size_change_info
-/* helper for _drbd_set_state */
-static void set_ov_position(struct drbd_peer_device *peer_device, enum drbd_conns cs)
+static enum drbd_packet reply_cmd_from(enum drbd_state_rv rv)
{
- struct drbd_device *device = peer_device->device;
-
- if (peer_device->connection->agreed_pro_version < 90)
- device->ov_start_sector = 0;
- device->rs_total = drbd_bm_bits(device);
- device->ov_position = 0;
- if (cs == C_VERIFY_T) {
- /* starting online verify from an arbitrary position
- * does not fit well into the existing protocol.
- * on C_VERIFY_T, we initialize ov_left and friends
- * implicitly in receive_DataRequest once the
- * first P_OV_REQUEST is received */
- device->ov_start_sector = ~(sector_t)0;
- } else {
- unsigned long bit = BM_SECT_TO_BIT(device->ov_start_sector);
- if (bit >= device->rs_total) {
- device->ov_start_sector =
- BM_BIT_TO_SECT(device->rs_total - 1);
- device->rs_total = 1;
- } else
- device->rs_total -= bit;
- device->ov_position = device->ov_start_sector;
- }
- device->ov_left = device->rs_total;
+ if (rv >= SS_SUCCESS)
+ return P_TWOPC_YES;
+ if (rv == SS_CONCURRENT_ST_CHG || rv == SS_HANDSHAKE_RETRY)
+ return P_TWOPC_RETRY;
+ return P_TWOPC_NO;
}
-/**
- * _drbd_set_state() - Set a new DRBD state
- * @device: DRBD device.
- * @ns: new state.
- * @flags: Flags
- * @done: Optional completion, that will get completed after the after_state_ch() finished
- *
- * Caller needs to hold req_lock. Do not call directly.
+/* The reply answers one transaction: from_work, the one that set
+ * TWOPC_WORK_PENDING; otherwise tid. A cmd of 0 derives the verdict here.
*/
-enum drbd_state_rv
-_drbd_set_state(struct drbd_device *device, union drbd_state ns,
- enum chg_state_flags flags, struct completion *done)
+static void twopc_end_nested(struct drbd_resource *resource, unsigned int tid,
+ enum drbd_packet cmd, bool from_work)
{
- struct drbd_peer_device *peer_device = first_peer_device(device);
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- union drbd_state os;
- enum drbd_state_rv rv = SS_SUCCESS;
- enum sanitize_state_warnings ssw;
- struct after_state_chg_work *ascw;
- struct drbd_state_change *state_change;
-
- os = drbd_read_state(device);
-
- ns = sanitize_state(device, os, ns, &ssw);
- if (ns.i == os.i)
- return SS_NOTHING_TO_DO;
+ struct drbd_connection *twopc_parent;
+ u64 im;
+ struct twopc_reply twopc_reply;
+ u64 twopc_parent_nodes = 0;
+
+ write_lock_irq(&resource->state_rwlock);
+ if (from_work) {
+ if (!test_and_clear_bit(TWOPC_WORK_PENDING, &resource->flags))
+ goto out_unlock;
+ } else if (resource->twopc_reply.tid != tid) {
+ goto out_unlock;
+ }
+ if (cmd == 0)
+ cmd = reply_cmd_from(get_cluster_wide_reply(resource, NULL));
+ twopc_reply = resource->twopc_reply;
+ /* Only send replies if we are in a twopc and have not yet sent replies. */
+ if (twopc_reply.tid && resource->twopc_prepare_reply_cmd == 0) {
+ resource->twopc_prepare_reply_cmd = cmd;
+ twopc_parent_nodes = resource->twopc_parent_nodes;
+ }
+ write_unlock_irq(&resource->state_rwlock);
- rv = is_valid_transition(os, ns);
- if (rv < SS_SUCCESS)
- return rv;
+ if (!twopc_reply.tid)
+ return;
- if (!(flags & CS_HARD)) {
- /* pre-state-change checks ; only look at ns */
- /* See drbd_state_sw_errors in drbd_strings.c */
+ for_each_connection_ref(twopc_parent, im, resource) {
+ if (!(twopc_parent_nodes & NODE_MASK(twopc_parent->peer_node_id)))
+ continue;
- rv = is_valid_state(device, ns);
- if (rv < SS_SUCCESS) {
- /* If the old state was illegal as well, then let
- this happen...*/
+ if (twopc_reply.is_disconnect)
+ set_bit(DISCONNECT_EXPECTED, &twopc_parent->flags);
- if (is_valid_state(device, os) == rv)
- rv = is_valid_soft_transition(os, ns, connection);
- } else
- rv = is_valid_soft_transition(os, ns, connection);
- }
+ dynamic_drbd_dbg(twopc_parent, "Nested state change %u result: %s\n",
+ twopc_reply.tid, drbd_packet_name(cmd));
- if (rv < SS_SUCCESS) {
- if (flags & CS_VERBOSE)
- print_st_err(device, os, ns, rv);
- return rv;
+ drbd_send_twopc_reply(twopc_parent, cmd, &twopc_reply);
}
+ wake_up_all(&resource->twopc_wait);
+ return;
- print_sanitize_warnings(device, ssw);
+out_unlock:
+ write_unlock_irq(&resource->state_rwlock);
+}
- drbd_pr_state_change(device, os, ns, flags);
+void nested_twopc_work(struct work_struct *work)
+{
+ struct drbd_resource *resource =
+ container_of(work, struct drbd_resource, twopc_work);
- /* Display changes to the susp* flags that where caused by the call to
- sanitize_state(). Only display it here if we where not called from
- _conn_request_state() */
- if (!(flags & CS_DC_SUSP))
- conn_pr_state_change(connection, os, ns,
- (flags & ~CS_DC_MASK) | CS_DC_SUSP);
+ twopc_end_nested(resource, 0, 0, true);
+ kref_put(&resource->kref, drbd_destroy_resource);
+}
- /* if we are going -> D_FAILED or D_DISKLESS, grab one extra reference
- * on the ldev here, to be sure the transition -> D_DISKLESS resp.
- * drbd_ldev_destroy() won't happen before our corresponding
- * after_state_ch works run, where we put_ldev again. */
- if ((os.disk != D_FAILED && ns.disk == D_FAILED) ||
- (os.disk != D_DISKLESS && ns.disk == D_DISKLESS))
- atomic_inc(&device->local_cnt);
+void drbd_maybe_cluster_wide_reply(struct drbd_resource *resource)
+{
+ lockdep_assert_held(&resource->state_rwlock);
- if (!is_sync_state(os.conn) && is_sync_state(ns.conn))
- clear_bit(RS_DONE, &device->flags);
+ /* Until this node forwarded the prepare, none of its children is
+ * prepared, and cluster_wide_reply_ready() would vacuously answer YES
+ * for a transaction this node has not even evaluated yet.
+ * nested_twopc_request() re-derives the reply once it is clear.
+ */
+ if (!resource->remote_state_change || test_bit(TWOPC_PREPARING, &resource->flags) ||
+ !cluster_wide_reply_ready(resource))
+ return;
- /* FIXME: Have any flags been set earlier in this function already? */
- state_change = remember_old_state(device->resource, GFP_ATOMIC);
+ if (resource->twopc_reply.initiator_node_id == resource->res_opts.node_id) {
+ wake_up_all(&resource->state_wait);
+ return;
+ }
- /* changes to local_cnt and device flags should be visible before
- * changes to state, which again should be visible before anything else
- * depending on that change happens. */
- smp_wmb();
- device->state.i = ns.i;
- device->resource->susp = ns.susp;
- device->resource->susp_nod = ns.susp_nod;
- device->resource->susp_fen = ns.susp_fen;
- smp_wmb();
+ /* The reply to this transaction went out already. */
+ if (resource->twopc_prepare_reply_cmd)
+ return;
- remember_new_state(state_change);
+ if (test_and_set_bit(TWOPC_WORK_PENDING, &resource->flags))
+ return;
- /* put replicated vs not-replicated requests in seperate epochs */
- if (drbd_should_do_remote((union drbd_dev_state)os.i) !=
- drbd_should_do_remote((union drbd_dev_state)ns.i))
- start_new_tl_epoch(connection);
+ kref_get(&resource->kref);
+ if (!schedule_work(&resource->twopc_work)) {
+ /* Still queued from a transaction that ended before it ran; that
+ * instance's reference covers this reply as well.
+ */
+ kref_put(&resource->kref, drbd_destroy_resource);
+ }
+}
- if (os.disk == D_ATTACHING && ns.disk >= D_NEGOTIATING)
- drbd_print_uuids(device, "attached to UUIDs");
+enum drbd_state_rv
+nested_twopc_request(struct drbd_resource *resource, struct twopc_request *request)
+{
+ u64 nodes_to_reach, reach_immediately;
+ enum drbd_packet cmd = request->cmd;
+ enum drbd_state_rv rv;
+ bool have_peers;
+
+ write_lock_irq(&resource->state_rwlock);
+ nodes_to_reach = request->nodes_to_reach;
+ reach_immediately = directly_connected_nodes(resource, NOW) & nodes_to_reach;
+ nodes_to_reach &= ~(reach_immediately | NODE_MASK(resource->res_opts.node_id));
+ request->nodes_to_reach = nodes_to_reach;
+ write_unlock_irq(&resource->state_rwlock);
+
+ rv = __cluster_wide_request(resource, request, reach_immediately);
+ have_peers = rv == SS_CW_SUCCESS;
+ if (cmd == P_TWOPC_PREPARE || cmd == P_TWOPC_PREP_RSZ) {
+ write_lock_irq(&resource->state_rwlock);
+ /* The flag may belong to a later transaction already */
+ if (resource->twopc_reply.tid == request->tid)
+ clear_bit(TWOPC_PREPARING, &resource->flags);
+ write_unlock_irq(&resource->state_rwlock);
- /* Wake up role changes, that were delayed because of connection establishing */
- if (os.conn == C_WF_REPORT_PARAMS && ns.conn != C_WF_REPORT_PARAMS &&
- no_peer_wf_report_params(connection)) {
- clear_bit(STATE_SENT, &connection->flags);
- wake_up_all_devices(connection);
+ if (rv < SS_SUCCESS) {
+ twopc_end_nested(resource, request->tid, P_TWOPC_NO, false);
+ } else if (!have_peers && cluster_wide_reply_ready(resource)) {
+ /* no nested nodes */
+ twopc_end_nested(resource, request->tid, 0, false);
+ } else if (have_peers) {
+ /* A peer that dropped between the reachability check above and
+ * its prepare produces no further event to re-derive on.
+ */
+ write_lock_irq(&resource->state_rwlock);
+ drbd_maybe_cluster_wide_reply(resource);
+ write_unlock_irq(&resource->state_rwlock);
+ }
}
+ return rv;
+}
- wake_up(&device->misc_wait);
- wake_up(&device->state_wait);
- wake_up(&connection->ping_wait);
+static bool has_up_to_date_peer_disks(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
- /* Aborted verify run, or we reached the stop sector.
- * Log the last position, unless end-of-device. */
- if ((os.conn == C_VERIFY_S || os.conn == C_VERIFY_T) &&
- ns.conn <= C_CONNECTED) {
- device->ov_start_sector =
- BM_BIT_TO_SECT(drbd_bm_bits(device) - device->ov_left);
- if (device->ov_left)
- drbd_info(device, "Online Verify reached sector %llu\n",
- (unsigned long long)device->ov_start_sector);
- }
+ for_each_peer_device(peer_device, device)
+ if (peer_device->disk_state[NEW] == D_UP_TO_DATE)
+ return true;
+ return false;
+}
- if ((os.conn == C_PAUSED_SYNC_T || os.conn == C_PAUSED_SYNC_S) &&
- (ns.conn == C_SYNC_TARGET || ns.conn == C_SYNC_SOURCE)) {
- drbd_info(device, "Syncer continues.\n");
- device->rs_paused += (long)jiffies
- -(long)device->rs_mark_time[device->rs_last_mark];
- if (ns.conn == C_SYNC_TARGET)
- mod_timer(&device->resync_timer, jiffies);
- }
+static void disconnect_where_resync_target(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
- if ((os.conn == C_SYNC_TARGET || os.conn == C_SYNC_SOURCE) &&
- (ns.conn == C_PAUSED_SYNC_T || ns.conn == C_PAUSED_SYNC_S)) {
- drbd_info(device, "Resync suspended\n");
- device->rs_mark_time[device->rs_last_mark] = jiffies;
- }
+ for_each_peer_device(peer_device, device)
+ if (is_sync_target_state(peer_device, NEW))
+ __change_cstate(peer_device->connection, C_TEAR_DOWN);
+}
- if (os.conn == C_CONNECTED &&
- (ns.conn == C_VERIFY_S || ns.conn == C_VERIFY_T)) {
- unsigned long now = jiffies;
- int i;
+static bool do_change_role(struct change_context *context, enum change_phase phase)
+{
+ struct drbd_resource *resource = context->resource;
+ enum drbd_role role = context->val.role;
+ int flags = context->flags;
+ struct drbd_device *device;
+ int vnr;
- set_ov_position(peer_device, ns.conn);
- device->rs_start = now;
- device->rs_last_sect_ev = 0;
- device->ov_last_oos_size = 0;
- device->ov_last_oos_start = 0;
+ resource->role[NEW] = role;
- for (i = 0; i < DRBD_SYNC_MARKS; i++) {
- device->rs_mark_left[i] = device->ov_left;
- device->rs_mark_time[i] = now;
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (role == R_PRIMARY && (flags & CS_FP_LOCAL_UP_TO_DATE)) {
+ if (device->disk_state[NEW] < D_UP_TO_DATE &&
+ device->disk_state[NEW] >= D_INCONSISTENT &&
+ !has_up_to_date_peer_disks(device)) {
+ device->disk_state[NEW] = D_UP_TO_DATE;
+ /* adding it to the context so that it gets sent to the peers */
+ context->mask.disk |= disk_MASK;
+ context->val.disk |= D_UP_TO_DATE;
+ disconnect_where_resync_target(device);
+ }
}
- drbd_rs_controller_reset(peer_device);
+ if (role == R_PRIMARY && (flags & CS_FP_OUTDATE_PEERS)) {
+ struct drbd_peer_device *peer_device;
- if (ns.conn == C_VERIFY_S) {
- drbd_info(device, "Starting Online Verify from sector %llu\n",
- (unsigned long long)device->ov_position);
- mod_timer(&device->resync_timer, jiffies);
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->disk_state[NEW] == D_UNKNOWN)
+ __change_peer_disk_state(peer_device, D_OUTDATED);
+ }
}
- }
- if (get_ldev(device)) {
- u32 mdf = device->ldev->md.flags & ~(MDF_CONSISTENT|MDF_PRIMARY_IND|
- MDF_CONNECTED_IND|MDF_WAS_UP_TO_DATE|
- MDF_PEER_OUT_DATED|MDF_CRASHED_PRIMARY);
-
- mdf &= ~MDF_AL_CLEAN;
- if (test_bit(CRASHED_PRIMARY, &device->flags))
- mdf |= MDF_CRASHED_PRIMARY;
- if (device->state.role == R_PRIMARY ||
- (device->state.pdsk < D_INCONSISTENT && device->state.peer == R_PRIMARY))
- mdf |= MDF_PRIMARY_IND;
- if (device->state.conn > C_WF_REPORT_PARAMS)
- mdf |= MDF_CONNECTED_IND;
- if (device->state.disk > D_INCONSISTENT)
- mdf |= MDF_CONSISTENT;
- if (device->state.disk > D_OUTDATED)
- mdf |= MDF_WAS_UP_TO_DATE;
- if (device->state.pdsk <= D_OUTDATED && device->state.pdsk >= D_INCONSISTENT)
- mdf |= MDF_PEER_OUT_DATED;
- if (mdf != device->ldev->md.flags) {
- device->ldev->md.flags = mdf;
- drbd_md_mark_dirty(device);
- }
- if (os.disk < D_CONSISTENT && ns.disk >= D_CONSISTENT)
- drbd_set_ed_uuid(device, device->ldev->md.uuid[UI_CURRENT]);
- put_ldev(device);
- }
+ if (role == R_PRIMARY && phase == PH_COMMIT) {
+ u64 reachable_nodes = resource->twopc_reply.reachable_nodes;
+ struct drbd_peer_device *peer_device;
- /* Peer was forced D_UP_TO_DATE & R_PRIMARY, consider to resync */
- if (os.disk == D_INCONSISTENT && os.pdsk == D_INCONSISTENT &&
- os.peer == R_SECONDARY && ns.peer == R_PRIMARY)
- set_bit(CONSIDER_RESYNC, &device->flags);
-
- /* Receiver should clean up itself */
- if (os.conn != C_DISCONNECTING && ns.conn == C_DISCONNECTING)
- drbd_thread_stop_nowait(&connection->receiver);
-
- /* Now the receiver finished cleaning up itself, it should die */
- if (os.conn != C_STANDALONE && ns.conn == C_STANDALONE)
- drbd_thread_stop_nowait(&connection->receiver);
-
- /* Upon network failure, we need to restart the receiver. */
- if (os.conn > C_WF_CONNECTION &&
- ns.conn <= C_TEAR_DOWN && ns.conn >= C_TIMEOUT)
- drbd_thread_restart_nowait(&connection->receiver);
-
- /* Resume AL writing if we get a connection */
- if (os.conn < C_CONNECTED && ns.conn >= C_CONNECTED) {
- drbd_resume_al(device);
- connection->connect_cnt++;
- }
-
- /* remember last attach time so request_timer_fn() won't
- * kill newly established sessions while we are still trying to thaw
- * previously frozen IO */
- if ((os.disk == D_ATTACHING || os.disk == D_NEGOTIATING) &&
- ns.disk > D_NEGOTIATING)
- device->last_reattach_jif = jiffies;
-
- ascw = kmalloc_obj(*ascw, GFP_ATOMIC);
- if (ascw) {
- ascw->os = os;
- ascw->ns = ns;
- ascw->flags = flags;
- ascw->w.cb = w_after_state_ch;
- ascw->device = device;
- ascw->done = done;
- ascw->state_change = state_change;
- drbd_queue_work(&connection->sender_work,
- &ascw->w);
- } else {
- drbd_err(device, "Could not kmalloc an ascw\n");
+ for_each_peer_device_rcu(peer_device, device) {
+ if (NODE_MASK(peer_device->node_id) & reachable_nodes &&
+ peer_device->disk_state[NEW] == D_UNKNOWN &&
+ want_bitmap(peer_device))
+ __change_peer_disk_state(peer_device, D_OUTDATED);
+ }
+ }
}
+ rcu_read_unlock();
- return rv;
+ return phase != PH_PREPARE ||
+ context->resource->role[NOW] != context->val.role;
}
-static int w_after_state_ch(struct drbd_work *w, int unused)
+enum drbd_state_rv change_role(struct drbd_resource *resource,
+ enum drbd_role role,
+ enum chg_state_flags flags,
+ const char *tag,
+ const char **err_str)
{
- struct after_state_chg_work *ascw =
- container_of(w, struct after_state_chg_work, w);
- struct drbd_device *device = ascw->device;
-
- after_state_ch(device, ascw->os, ascw->ns, ascw->flags, ascw->state_change);
- forget_state_change(ascw->state_change);
- if (ascw->flags & CS_WAIT_COMPLETE)
- complete(ascw->done);
- kfree(ascw);
+ struct change_context role_context = {
+ .resource = resource,
+ .vnr = -1,
+ .mask = { { .role = role_MASK } },
+ .val = { { .role = role } },
+ .target_node_id = -1,
+ .flags = flags | CS_SERIALIZE,
+ .err_str = err_str,
+ };
+ enum drbd_state_rv rv;
+ bool got_state_sem = false;
- return 0;
+ if (role == R_SECONDARY) {
+ if (!(flags & CS_ALREADY_SERIALIZED)) {
+ down(&resource->state_sem);
+ got_state_sem = true;
+ role_context.flags |= CS_ALREADY_SERIALIZED;
+ }
+ role_context.change_local_state_last = true;
+ }
+ rv = change_cluster_wide_state(do_change_role, &role_context, tag);
+ if (got_state_sem)
+ up(&resource->state_sem);
+ return rv;
}
-static void abw_start_sync(struct drbd_device *device, int rv)
+void __change_io_susp_user(struct drbd_resource *resource, bool value)
{
- if (rv) {
- drbd_err(device, "Writing the bitmap failed not starting resync.\n");
- _drbd_request_state(device, NS(conn, C_CONNECTED), CS_VERBOSE);
- return;
- }
-
- switch (device->state.conn) {
- case C_STARTING_SYNC_T:
- _drbd_request_state(device, NS(conn, C_WF_SYNC_UUID), CS_VERBOSE);
- break;
- case C_STARTING_SYNC_S:
- drbd_start_resync(device, C_SYNC_SOURCE);
- break;
- }
+ resource->susp_user[NEW] = value;
}
-int drbd_bitmap_io_from_worker(struct drbd_device *device,
- int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
- char *why, enum bm_flag flags,
- struct drbd_peer_device *peer_device)
+enum drbd_state_rv change_io_susp_user(struct drbd_resource *resource,
+ bool value,
+ enum chg_state_flags flags)
{
- int rv;
-
- D_ASSERT(device, current == first_peer_device(device)->connection->worker.task);
-
- /* open coded non-blocking drbd_suspend_io(device); */
- atomic_inc(&device->suspend_cnt);
-
- drbd_bm_lock(device, why, flags);
- rv = io_fn(device, peer_device);
- drbd_bm_unlock(device);
-
- drbd_resume_io(device);
+ unsigned long irq_flags;
- return rv;
+ begin_state_change(resource, &irq_flags, flags);
+ __change_io_susp_user(resource, value);
+ return end_state_change(resource, &irq_flags, value ? "suspend-io" : "resume-io");
}
-int notify_resource_state_change(struct sk_buff *skb,
- unsigned int seq,
- void *state_change,
- enum drbd_notification_type type)
+void __change_io_susp_no_data(struct drbd_resource *resource, bool value)
{
- struct drbd_resource_state_change *resource_state_change = state_change;
- struct drbd_resource *resource = resource_state_change->resource;
- struct resource_info resource_info = {
- .res_role = resource_state_change->role[NEW],
- .res_susp = resource_state_change->susp[NEW],
- .res_susp_nod = resource_state_change->susp_nod[NEW],
- .res_susp_fen = resource_state_change->susp_fen[NEW],
- };
+ resource->susp_nod[NEW] = value;
+}
- return notify_resource_state(skb, seq, resource, &resource_info, type);
+void __change_io_susp_fencing(struct drbd_connection *connection, bool value)
+{
+ connection->susp_fen[NEW] = value;
}
-int notify_connection_state_change(struct sk_buff *skb,
- unsigned int seq,
- void *state_change,
- enum drbd_notification_type type)
+void __change_io_susp_quorum(struct drbd_resource *resource, bool value)
{
- struct drbd_connection_state_change *p = state_change;
- struct drbd_connection *connection = p->connection;
- struct connection_info connection_info = {
- .conn_connection_state = p->cstate[NEW],
- .conn_role = p->peer_role[NEW],
- };
+ resource->susp_quorum[NEW] = value;
+}
- return notify_connection_state(skb, seq, connection, &connection_info, type);
+void __change_disk_state(struct drbd_device *device, enum drbd_disk_state disk_state)
+{
+ device->disk_state[NEW] = disk_state;
}
-int notify_device_state_change(struct sk_buff *skb,
- unsigned int seq,
- void *state_change,
- enum drbd_notification_type type)
+void __downgrade_disk_states(struct drbd_resource *resource, enum drbd_disk_state disk_state)
{
- struct drbd_device_state_change *device_state_change = state_change;
- struct drbd_device *device = device_state_change->device;
- struct device_info device_info = {
- .dev_disk_state = device_state_change->disk_state[NEW],
- };
+ struct drbd_device *device;
+ int vnr;
- return notify_device_state(skb, seq, device, &device_info, type);
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (device->disk_state[NEW] > disk_state)
+ __change_disk_state(device, disk_state);
+ }
+ rcu_read_unlock();
}
-int notify_peer_device_state_change(struct sk_buff *skb,
- unsigned int seq,
- void *state_change,
- enum drbd_notification_type type)
+void __outdate_myself(struct drbd_resource *resource)
{
- struct drbd_peer_device_state_change *p = state_change;
- struct drbd_peer_device *peer_device = p->peer_device;
- struct peer_device_info peer_device_info = {
- .peer_repl_state = p->repl_state[NEW],
- .peer_disk_state = p->disk_state[NEW],
- .peer_resync_susp_user = p->resync_susp_user[NEW],
- .peer_resync_susp_peer = p->resync_susp_peer[NEW],
- .peer_resync_susp_dependency = p->resync_susp_dependency[NEW],
- };
+ struct drbd_device *device;
+ int vnr;
- return notify_peer_device_state(skb, seq, peer_device, &peer_device_info, type);
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (device->disk_state[NOW] > D_OUTDATED)
+ __change_disk_state(device, D_OUTDATED);
+ }
}
-static void broadcast_state_change(struct drbd_state_change *state_change)
+static bool device_has_connected_peer_devices(struct drbd_device *device)
{
- struct drbd_resource_state_change *resource_state_change = &state_change->resource[0];
- bool resource_state_has_changed;
- unsigned int n_device, n_connection, n_peer_device, n_peer_devices;
- int (*last_func)(struct sk_buff *, unsigned int,
- void *, enum drbd_notification_type) = NULL;
- void *last_arg = NULL;
-
-#define HAS_CHANGED(state) ((state)[OLD] != (state)[NEW])
-#define FINAL_STATE_CHANGE(type) \
- ({ if (last_func) \
- last_func(NULL, 0, last_arg, type); \
- })
-#define REMEMBER_STATE_CHANGE(func, arg, type) \
- ({ FINAL_STATE_CHANGE(type | NOTIFY_CONTINUES); \
- last_func = func; \
- last_arg = arg; \
- })
+ struct drbd_peer_device *peer_device;
- mutex_lock(¬ification_mutex);
+ for_each_peer_device(peer_device, device)
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED)
+ return true;
+ return false;
+}
- resource_state_has_changed =
- HAS_CHANGED(resource_state_change->role) ||
- HAS_CHANGED(resource_state_change->susp) ||
- HAS_CHANGED(resource_state_change->susp_nod) ||
- HAS_CHANGED(resource_state_change->susp_fen);
+static bool device_has_peer_devices_with_disk(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ bool rv = false;
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->connection->cstate[NOW] == C_CONNECTED) {
+ /* We expect to receive up-to-date UUIDs soon.
+ To avoid a race in receive_state, "clear" uuids while
+ holding state_rwlock. I.e. atomic with the state change */
+ clear_bit(UUIDS_RECEIVED, peer_device->flags);
+ if (peer_device->disk_state[NOW] > D_DISKLESS)
+ rv = true;
+ }
+ }
- if (resource_state_has_changed)
- REMEMBER_STATE_CHANGE(notify_resource_state_change,
- resource_state_change, NOTIFY_CHANGE);
+ return rv;
+}
- for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
- struct drbd_connection_state_change *connection_state_change =
- &state_change->connections[n_connection];
+static void restore_outdated_in_pdsk(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
- if (HAS_CHANGED(connection_state_change->peer_role) ||
- HAS_CHANGED(connection_state_change->cstate))
- REMEMBER_STATE_CHANGE(notify_connection_state_change,
- connection_state_change, NOTIFY_CHANGE);
- }
+ if (!get_ldev_if_state(device, D_ATTACHING))
+ return;
- for (n_device = 0; n_device < state_change->n_devices; n_device++) {
- struct drbd_device_state_change *device_state_change =
- &state_change->devices[n_device];
+ for_each_peer_device(peer_device, device) {
+ int node_id = peer_device->connection->peer_node_id;
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
- if (HAS_CHANGED(device_state_change->disk_state))
- REMEMBER_STATE_CHANGE(notify_device_state_change,
- device_state_change, NOTIFY_CHANGE);
+ if (test_bit(__MDF_PEER_OUTDATED, &peer_md->flags) &&
+ peer_device->disk_state[NEW] == D_UNKNOWN)
+ __change_peer_disk_state(peer_device, D_OUTDATED);
}
- n_peer_devices = state_change->n_devices * state_change->n_connections;
- for (n_peer_device = 0; n_peer_device < n_peer_devices; n_peer_device++) {
- struct drbd_peer_device_state_change *p =
- &state_change->peer_devices[n_peer_device];
+ put_ldev(device);
+}
- if (HAS_CHANGED(p->disk_state) ||
- HAS_CHANGED(p->repl_state) ||
- HAS_CHANGED(p->resync_susp_user) ||
- HAS_CHANGED(p->resync_susp_peer) ||
- HAS_CHANGED(p->resync_susp_dependency))
- REMEMBER_STATE_CHANGE(notify_peer_device_state_change,
- p, NOTIFY_CHANGE);
+static bool do_twopc_after_lost_peer(struct change_context *context, enum change_phase phase)
+{
+ struct drbd_resource *resource = context->resource;
+ struct twopc_reply *reply = &resource->twopc_reply;
+ u64 directly_reachable = directly_connected_nodes(resource, NEW) |
+ NODE_MASK(resource->res_opts.node_id);
+ bool pri_incapable = reply->primary_nodes && !reply->weak_nodes; /* TWOPC_PRI_INCAPABLE */
+
+ if (phase == PH_COMMIT && (reply->primary_nodes & ~directly_reachable && !pri_incapable)) {
+ __outdate_myself(resource);
+ } else {
+ struct drbd_device *device;
+ int vnr;
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (device->disk_state[NOW] != D_CONSISTENT ||
+ !may_return_to_up_to_date(device, NOW))
+ continue;
+ /* Stage the optimistic UpToDate at PH_PREPARE even while the
+ * reconcile hold is active: it keeps the twopc non-empty so it
+ * reaches PH_COMMIT, where primary_nodes is known and the outdate
+ * branch above can fire. At PH_COMMIT the hold is honored, so a
+ * held survivor commits no change and stays D_CONSISTENT.
+ */
+ if (phase == PH_PREPARE || !reconcile_hold_active(device))
+ __change_disk_state(device, D_UP_TO_DATE);
+ }
}
- FINAL_STATE_CHANGE(NOTIFY_CHANGE);
- mutex_unlock(¬ification_mutex);
+ return phase != PH_PREPARE || reply->reachable_nodes != NODE_MASK(resource->res_opts.node_id);
+}
-#undef HAS_CHANGED
-#undef FINAL_STATE_CHANGE
-#undef REMEMBER_STATE_CHANGE
+static enum drbd_state_rv twopc_after_lost_peer(struct drbd_resource *resource,
+ enum chg_state_flags flags)
+{
+ struct change_context context = {
+ .resource = resource,
+ .vnr = -1,
+ .mask = { },
+ .val = { },
+ .target_node_id = -1,
+ .flags = flags | (resource->res_opts.quorum != QOU_OFF ? CS_FORCE_RECALC : 0),
+ .change_local_state_last = false,
+ };
+
+ /* The other nodes get the request for an empty state change. I.e. they
+ will agree to this change request. At commit time we know where to
+ go from the D_CONSISTENT, since we got the primary mask. */
+ return change_cluster_wide_state(do_twopc_after_lost_peer, &context, "lost-peer");
}
-/* takes old and new peer disk state */
-static bool lost_contact_to_peer_data(enum drbd_disk_state os, enum drbd_disk_state ns)
+void drbd_empty_twopc_work_fn(struct work_struct *work)
{
- if ((os >= D_INCONSISTENT && os != D_UNKNOWN && os != D_OUTDATED)
- && (ns < D_INCONSISTENT || ns == D_UNKNOWN || ns == D_OUTDATED))
- return true;
+ struct drbd_resource *resource = container_of(work, struct drbd_resource, empty_twopc);
- /* Scenario, starting with normal operation
- * Connected Primary/Secondary UpToDate/UpToDate
- * NetworkFailure Primary/Unknown UpToDate/DUnknown (frozen)
- * ...
- * Connected Primary/Secondary UpToDate/Diskless (resumed; needs to bump uuid!)
- */
- if (os == D_UNKNOWN
- && (ns == D_DISKLESS || ns == D_FAILED || ns == D_OUTDATED))
- return true;
+ twopc_after_lost_peer(resource, CS_VERBOSE);
- return false;
+ clear_bit(TRY_BECOME_UP_TO_DATE_PENDING, &resource->flags);
+ wake_up_all(&resource->state_wait);
+ kref_put(&resource->kref, drbd_destroy_resource);
}
-/**
- * after_state_ch() - Perform after state change actions that may sleep
- * @device: DRBD device.
- * @os: old state.
- * @ns: new state.
- * @flags: Flags
- * @state_change: state change to broadcast
+/* A resumed Primary drives an empty 2PC so that far-away members it cannot
+ * reach directly outdate themselves (do_twopc_after_lost_peer at PH_COMMIT ->
+ * far_away_change on those nodes) before we resume.
*/
-static void after_state_ch(struct drbd_device *device, union drbd_state os,
- union drbd_state ns, enum chg_state_flags flags,
- struct drbd_state_change *state_change)
+static enum drbd_state_rv twopc_primary_resume(struct drbd_resource *resource,
+ enum chg_state_flags flags)
{
- struct drbd_resource *resource = device->resource;
- struct drbd_peer_device *peer_device = first_peer_device(device);
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- struct sib_info sib;
+ struct change_context context = {
+ .resource = resource,
+ .vnr = -1,
+ .mask = { },
+ .val = { },
+ .target_node_id = -1,
+ .flags = flags | CS_FORCE_RECALC,
+ .change_local_state_last = true,
+ };
- broadcast_state_change(state_change);
+ return change_cluster_wide_state(do_twopc_after_lost_peer, &context, "primary-resume");
+}
- sib.sib_reason = SIB_STATE_CHANGE;
- sib.os = os;
- sib.ns = ns;
+void drbd_resume_twopc_work_fn(struct work_struct *work)
+{
+ struct drbd_resource *resource = container_of(work, struct drbd_resource, resume_twopc);
+ unsigned long irq_flags;
- if ((os.disk != D_UP_TO_DATE || os.pdsk != D_UP_TO_DATE)
- && (ns.disk == D_UP_TO_DATE && ns.pdsk == D_UP_TO_DATE)) {
- clear_bit(CRASHED_PRIMARY, &device->flags);
- if (device->p_uuid)
- device->p_uuid[UI_FLAGS] &= ~((u64)2);
- }
+ /* First outdate the far-away member(s) behind this Primary. */
+ twopc_primary_resume(resource, CS_VERBOSE);
- /* Inform userspace about the change... */
- drbd_bcast_event(device, &sib);
+ /* Then release the hold, which resumes the held I/O. */
+ begin_state_change(resource, &irq_flags, CS_VERBOSE | CS_FORCE_RECALC);
+ resource->resume_held_for_outdate[NEW] = false;
+ end_state_change(resource, &irq_flags, "primary-resumed");
+ kref_put(&resource->kref, drbd_destroy_resource);
+}
- if (!(os.role == R_PRIMARY && os.disk < D_UP_TO_DATE && os.pdsk < D_UP_TO_DATE) &&
- (ns.role == R_PRIMARY && ns.disk < D_UP_TO_DATE && ns.pdsk < D_UP_TO_DATE))
- drbd_khelper(device, "pri-on-incon-degr");
+static bool do_change_disk_state(struct change_context *context, enum change_phase phase)
+{
+ struct drbd_device *device =
+ container_of(context, struct change_disk_state_context, context)->device;
+ bool cluster_wide_state_change = false;
+
+ if (device->disk_state[NOW] == D_ATTACHING &&
+ context->val.disk == D_NEGOTIATING) {
+ if (device_has_peer_devices_with_disk(device)) {
+ cluster_wide_state_change =
+ supports_two_phase_commit(device->resource);
+ } else {
+ /* very last part of attach */
+ /* ldev_safe: D_ATTACHING->D_NEGOTIATING, state_rwlock held, ldev exists */
+ context->val.disk = disk_state_from_md(device);
+ restore_outdated_in_pdsk(device);
+ }
+ } else if (device->disk_state[NOW] != D_DETACHING &&
+ context->val.disk == D_DETACHING &&
+ device_has_connected_peer_devices(device)) {
+ cluster_wide_state_change = true;
+ }
+ __change_disk_state(device, context->val.disk);
+ return phase != PH_PREPARE || cluster_wide_state_change;
+}
- /* Here we have the actions that are performed after a
- state change. This function might sleep */
+enum drbd_state_rv change_disk_state(struct drbd_device *device,
+ enum drbd_disk_state disk_state,
+ enum chg_state_flags flags,
+ const char *tag,
+ const char **err_str)
+{
+ struct change_disk_state_context disk_state_context = {
+ .context = {
+ .resource = device->resource,
+ .vnr = device->vnr,
+ .mask = { { .disk = disk_MASK } },
+ .val = { { .disk = disk_state } },
+ .target_node_id = -1,
+ .flags = flags,
+ .change_local_state_last = true,
+ .err_str = err_str,
+ },
+ .device = device,
+ };
- if (ns.susp_nod) {
- enum drbd_req_event what = NOTHING;
+ return change_cluster_wide_state(do_change_disk_state,
+ &disk_state_context.context, tag);
+}
- spin_lock_irq(&device->resource->req_lock);
- if (os.conn < C_CONNECTED && conn_lowest_conn(connection) >= C_CONNECTED)
- what = RESEND;
+void __change_cstate(struct drbd_connection *connection, enum drbd_conn_state cstate)
+{
+ if (cstate == C_DISCONNECTING)
+ set_bit(DISCONNECT_EXPECTED, &connection->flags);
- if ((os.disk == D_ATTACHING || os.disk == D_NEGOTIATING) &&
- conn_lowest_disk(connection) == D_UP_TO_DATE)
- what = RESTART_FROZEN_DISK_IO;
+ connection->cstate[NEW] = cstate;
+ if (cstate < C_CONNECTED) {
+ struct drbd_peer_device *peer_device;
+ int vnr;
- if (resource->susp_nod && what != NOTHING) {
- _tl_restart(connection, what);
- _conn_request_state(connection,
- (union drbd_state) { { .susp_nod = 1 } },
- (union drbd_state) { { .susp_nod = 0 } },
- CS_VERBOSE);
- }
- spin_unlock_irq(&device->resource->req_lock);
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ __change_repl_state(peer_device, L_OFF);
+ rcu_read_unlock();
}
+}
- if (ns.susp_fen) {
- spin_lock_irq(&device->resource->req_lock);
- if (resource->susp_fen && conn_lowest_conn(connection) >= C_CONNECTED) {
- /* case2: The connection was established again: */
- struct drbd_peer_device *peer_device;
- int vnr;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
- clear_bit(NEW_CUR_UUID, &peer_device->device->flags);
- rcu_read_unlock();
+enum outdate_what { OUTDATE_NOTHING, OUTDATE_DISKS, OUTDATE_PEER_DISKS };
- /* We should actively create a new uuid, _before_
- * we resume/resent, if the peer is diskless
- * (recovery from a multiple error scenario).
- * Currently, this happens with a slight delay
- * below when checking lost_contact_to_peer_data() ...
- */
- _tl_restart(connection, RESEND);
- _conn_request_state(connection,
- (union drbd_state) { { .susp_fen = 1 } },
- (union drbd_state) { { .susp_fen = 0 } },
- CS_VERBOSE);
- }
- spin_unlock_irq(&device->resource->req_lock);
- }
-
- /* Became sync source. With protocol >= 96, we still need to send out
- * the sync uuid now. Need to do that before any drbd_send_state, or
- * the other side may go "paused sync" before receiving the sync uuids,
- * which is unexpected. */
- if ((os.conn != C_SYNC_SOURCE && os.conn != C_PAUSED_SYNC_S) &&
- (ns.conn == C_SYNC_SOURCE || ns.conn == C_PAUSED_SYNC_S) &&
- connection->agreed_pro_version >= 96 && get_ldev(device)) {
- drbd_gen_and_send_sync_uuid(peer_device);
- put_ldev(device);
+static enum outdate_what outdate_on_disconnect(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+
+ if (connection->cstate[NOW] == C_CONNECTED &&
+ (connection->fencing_policy >= FP_RESOURCE ||
+ connection->resource->res_opts.quorum != QOU_OFF) &&
+ resource->role[NOW] != connection->peer_role[NOW]) {
+ /* primary politely disconnects from secondary,
+ * tells peer to please outdate itself */
+ if (resource->role[NOW] == R_PRIMARY)
+ return OUTDATE_PEER_DISKS;
+
+ /* secondary politely disconnect from primary,
+ * proposes to outdate itself. */
+ if (connection->peer_role[NOW] == R_PRIMARY)
+ return OUTDATE_DISKS;
}
+ return OUTDATE_NOTHING;
+}
- /* Do not change the order of the if above and the two below... */
- if (os.pdsk == D_DISKLESS &&
- ns.pdsk > D_DISKLESS && ns.pdsk != D_UNKNOWN) { /* attach on the peer */
- /* we probably will start a resync soon.
- * make sure those things are properly reset. */
- device->rs_total = 0;
- device->rs_failed = 0;
- atomic_set(&device->rs_pending_cnt, 0);
- drbd_rs_cancel_all(device);
-
- drbd_send_uuids(peer_device);
- drbd_send_state(peer_device, ns);
- }
- /* No point in queuing send_bitmap if we don't have a connection
- * anymore, so check also the _current_ state, not only the new state
- * at the time this work was queued. */
- if (os.conn != C_WF_BITMAP_S && ns.conn == C_WF_BITMAP_S &&
- device->state.conn == C_WF_BITMAP_S)
- drbd_queue_bitmap_io(device, &drbd_send_bitmap, NULL,
- "send_bitmap (WFBitMapS)",
- BM_LOCKED_TEST_ALLOWED, peer_device);
-
- /* Lost contact to peer's copy of the data */
- if (lost_contact_to_peer_data(os.pdsk, ns.pdsk)) {
- if (get_ldev(device)) {
- if ((ns.role == R_PRIMARY || ns.peer == R_PRIMARY) &&
- device->ldev->md.uuid[UI_BITMAP] == 0 && ns.disk >= D_UP_TO_DATE) {
- if (drbd_suspended(device)) {
- set_bit(NEW_CUR_UUID, &device->flags);
- } else {
- drbd_uuid_new_current(device);
- drbd_send_uuids(peer_device);
- }
- }
- put_ldev(device);
- }
+static void __change_cstate_and_outdate(struct drbd_connection *connection,
+ enum drbd_conn_state cstate,
+ enum outdate_what outdate_what)
+{
+ __change_cstate(connection, cstate);
+ switch (outdate_what) {
+ case OUTDATE_DISKS:
+ __downgrade_disk_states(connection->resource, D_OUTDATED);
+ break;
+ case OUTDATE_PEER_DISKS:
+ __downgrade_peer_disk_states(connection, D_OUTDATED);
+ break;
+ case OUTDATE_NOTHING:
+ break;
}
+}
- if (ns.pdsk < D_INCONSISTENT && get_ldev(device)) {
- if (os.peer != R_PRIMARY && ns.peer == R_PRIMARY &&
- device->ldev->md.uuid[UI_BITMAP] == 0 && ns.disk >= D_UP_TO_DATE) {
- drbd_uuid_new_current(device);
- drbd_send_uuids(peer_device);
- }
- /* D_DISKLESS Peer becomes secondary */
- if (os.peer == R_PRIMARY && ns.peer == R_SECONDARY)
- /* We may still be Primary ourselves.
- * No harm done if the bitmap still changes,
- * redirtied pages will follow later. */
- drbd_bitmap_io_from_worker(device, &drbd_bm_write,
- "demote diskless peer", BM_LOCKED_SET_ALLOWED, peer_device);
- put_ldev(device);
- }
+void apply_connect(struct drbd_connection *connection, bool commit)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
- /* Write out all changed bits on demote.
- * Though, no need to da that just yet
- * if there is a resync going on still */
- if (os.role == R_PRIMARY && ns.role == R_SECONDARY &&
- device->state.conn <= C_CONNECTED && get_ldev(device)) {
- /* No changes to the bitmap expected this time, so assert that,
- * even though no harm was done if it did change. */
- drbd_bitmap_io_from_worker(device, &drbd_bm_write,
- "demote", BM_LOCKED_TEST_ALLOWED, peer_device);
- put_ldev(device);
- }
+ if (!commit || connection->cstate[NEW] != C_CONNECTED)
+ return;
- /* Last part of the attaching process ... */
- if (ns.conn >= C_CONNECTED &&
- os.disk == D_ATTACHING && ns.disk == D_NEGOTIATING) {
- drbd_send_sizes(peer_device, 0, 0); /* to start sync... */
- drbd_send_uuids(peer_device);
- drbd_send_state(peer_device, ns);
- }
-
- /* We want to pause/continue resync, tell peer. */
- if (ns.conn >= C_CONNECTED &&
- ((os.aftr_isp != ns.aftr_isp) ||
- (os.user_isp != ns.user_isp)))
- drbd_send_state(peer_device, ns);
-
- /* In case one of the isp bits got set, suspend other devices. */
- if ((!os.aftr_isp && !os.peer_isp && !os.user_isp) &&
- (ns.aftr_isp || ns.peer_isp || ns.user_isp))
- suspend_other_sg(device);
-
- /* Make sure the peer gets informed about eventual state
- changes (ISP bits) while we were in WFReportParams. */
- if (os.conn == C_WF_REPORT_PARAMS && ns.conn >= C_CONNECTED)
- drbd_send_state(peer_device, ns);
-
- if (os.conn != C_AHEAD && ns.conn == C_AHEAD)
- drbd_send_state(peer_device, ns);
-
- /* We are in the progress to start a full sync... */
- if ((os.conn != C_STARTING_SYNC_T && ns.conn == C_STARTING_SYNC_T) ||
- (os.conn != C_STARTING_SYNC_S && ns.conn == C_STARTING_SYNC_S))
- /* no other bitmap changes expected during this phase */
- drbd_queue_bitmap_io(device,
- &drbd_bmio_set_n_write, &abw_start_sync,
- "set_n_write from StartingSync", BM_LOCKED_TEST_ALLOWED,
- peer_device);
-
- /* first half of local IO error, failure to attach,
- * or administrative detach */
- if (os.disk != D_FAILED && ns.disk == D_FAILED) {
- enum drbd_io_error_p eh = EP_PASS_ON;
- int was_io_error = 0;
- /* corresponding get_ldev was in _drbd_set_state, to serialize
- * our cleanup here with the transition to D_DISKLESS.
- * But is is still not save to dreference ldev here, since
- * we might come from an failed Attach before ldev was set. */
- if (device->ldev) {
- rcu_read_lock();
- eh = rcu_dereference(device->ldev->disk_conf)->on_io_error;
- rcu_read_unlock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+ union drbd_state s = peer_device->connect_state;
- was_io_error = test_and_clear_bit(WAS_IO_ERROR, &device->flags);
-
- /* Intentionally call this handler first, before drbd_send_state().
- * See: 2932204 drbd: call local-io-error handler early
- * People may chose to hard-reset the box from this handler.
- * It is useful if this looks like a "regular node crash". */
- if (was_io_error && eh == EP_CALL_HELPER)
- drbd_khelper(device, "local-io-error");
-
- /* Immediately allow completion of all application IO,
- * that waits for completion from the local disk,
- * if this was a force-detach due to disk_timeout
- * or administrator request (drbdsetup detach --force).
- * Do NOT abort otherwise.
- * Aborting local requests may cause serious problems,
- * if requests are completed to upper layers already,
- * and then later the already submitted local bio completes.
- * This can cause DMA into former bio pages that meanwhile
- * have been re-used for other things.
- * So aborting local requests may cause crashes,
- * or even worse, silent data corruption.
- */
- if (test_and_clear_bit(FORCE_DETACH, &device->flags))
- tl_abort_disk_io(device);
+ if (s.disk != D_MASK)
+ __change_disk_state(device, s.disk);
+ if (device->disk_state[NOW] != D_NEGOTIATING)
+ __change_repl_state(peer_device, s.conn);
+ __change_peer_disk_state(peer_device, s.pdsk);
+ __change_resync_susp_peer(peer_device, s.peer_isp);
- /* current state still has to be D_FAILED,
- * there is only one way out: to D_DISKLESS,
- * and that may only happen after our put_ldev below. */
- if (device->state.disk != D_FAILED)
- drbd_err(device,
- "ASSERT FAILED: disk is %s during detach\n",
- drbd_disk_str(device->state.disk));
+ if (s.conn == L_OFF)
+ __change_cstate(connection, C_DISCONNECTING);
- if (ns.conn >= C_CONNECTED)
- drbd_send_state(peer_device, ns);
+ if (commit)
+ clear_bit(DISCARD_MY_DATA, peer_device->flags);
+ }
+}
- drbd_rs_cancel_all(device);
+struct change_cstate_context {
+ struct change_context context;
+ struct drbd_connection *connection;
+ enum outdate_what outdate_what;
+};
- /* In case we want to get something to stable storage still,
- * this may be the last chance.
- * Following put_ldev may transition to D_DISKLESS. */
- drbd_md_sync(device);
+static bool do_change_cstate(struct change_context *context, enum change_phase phase)
+{
+ struct change_cstate_context *cstate_context =
+ container_of(context, struct change_cstate_context, context);
+ struct drbd_connection *connection = cstate_context->connection;
+ struct drbd_resource *resource = context->resource;
+ struct twopc_reply *reply = &resource->twopc_reply;
+
+ if (phase == PH_PREPARE) {
+ cstate_context->outdate_what = OUTDATE_NOTHING;
+ if (context->val.conn == C_DISCONNECTING && !(context->flags & CS_HARD)) {
+ cstate_context->outdate_what =
+ outdate_on_disconnect(connection);
+ switch (cstate_context->outdate_what) {
+ case OUTDATE_DISKS:
+ context->mask.disk = disk_MASK;
+ context->val.disk = D_OUTDATED;
+ break;
+ case OUTDATE_PEER_DISKS:
+ context->mask.pdsk = pdsk_MASK;
+ context->val.pdsk = D_OUTDATED;
+ break;
+ case OUTDATE_NOTHING:
+ break;
+ }
}
- put_ldev(device);
}
-
- /* second half of local IO error, failure to attach,
- * or administrative detach,
- * after local_cnt references have reached zero again */
- if (os.disk != D_DISKLESS && ns.disk == D_DISKLESS) {
- /* We must still be diskless,
- * re-attach has to be serialized with this! */
- if (device->state.disk != D_DISKLESS)
- drbd_err(device,
- "ASSERT FAILED: disk is %s while going diskless\n",
- drbd_disk_str(device->state.disk));
-
- if (ns.conn >= C_CONNECTED)
- drbd_send_state(peer_device, ns);
- /* corresponding get_ldev in __drbd_set_state
- * this may finally trigger drbd_ldev_destroy. */
- put_ldev(device);
+ if ((context->val.conn == C_CONNECTED && connection->cstate[NEW] == C_CONNECTING) ||
+ context->val.conn != C_CONNECTED)
+ __change_cstate_and_outdate(connection,
+ context->val.conn,
+ cstate_context->outdate_what);
+
+ if (context->val.conn == C_CONNECTED &&
+ connection->agreed_pro_version >= 117)
+ apply_connect(connection, phase == PH_COMMIT);
+
+ if (phase == PH_COMMIT) {
+ u64 directly_reachable = directly_connected_nodes(resource, NEW) |
+ NODE_MASK(resource->res_opts.node_id);
+
+ if (reply->primary_nodes & ~directly_reachable)
+ __outdate_myself(resource);
}
- /* Notify peer that I had a local IO error, and did not detached.. */
- if (os.disk == D_UP_TO_DATE && ns.disk == D_INCONSISTENT && ns.conn >= C_CONNECTED)
- drbd_send_state(peer_device, ns);
-
- /* Disks got bigger while they were detached */
- if (ns.disk > D_NEGOTIATING && ns.pdsk > D_NEGOTIATING &&
- test_and_clear_bit(RESYNC_AFTER_NEG, &device->flags)) {
- if (ns.conn == C_CONNECTED)
- resync_after_online_grow(device);
- }
-
- /* A resync finished or aborted, wake paused devices... */
- if ((os.conn > C_CONNECTED && ns.conn <= C_CONNECTED) ||
- (os.peer_isp && !ns.peer_isp) ||
- (os.user_isp && !ns.user_isp))
- resume_next_sg(device);
-
- /* sync target done with resync. Explicitly notify peer, even though
- * it should (at least for non-empty resyncs) already know itself. */
- if (os.disk < D_UP_TO_DATE && os.conn >= C_SYNC_SOURCE && ns.conn == C_CONNECTED)
- drbd_send_state(peer_device, ns);
-
- /* Verify finished, or reached stop sector. Peer did not know about
- * the stop sector, and we may even have changed the stop sector during
- * verify to interrupt/stop early. Send the new state. */
- if (os.conn == C_VERIFY_S && ns.conn == C_CONNECTED
- && verify_can_do_stop_sector(device))
- drbd_send_state(peer_device, ns);
-
- /* This triggers bitmap writeout of potentially still unwritten pages
- * if the resync finished cleanly, or aborted because of peer disk
- * failure, or on transition from resync back to AHEAD/BEHIND.
- *
- * Connection loss is handled in drbd_disconnected() by the receiver.
- *
- * For resync aborted because of local disk failure, we cannot do
- * any bitmap writeout anymore.
- *
- * No harm done if some bits change during this phase.
- */
- if ((os.conn > C_CONNECTED && os.conn < C_AHEAD) &&
- (ns.conn == C_CONNECTED || ns.conn >= C_AHEAD) && get_ldev(device)) {
- drbd_queue_bitmap_io(device, &drbd_bm_write_copy_pages, NULL,
- "write from resync_finished", BM_LOCKED_CHANGE_ALLOWED,
- peer_device);
- put_ldev(device);
+ if (context->val.conn == C_CONNECTED && connection->peer_role[NOW] == R_UNKNOWN) {
+ enum drbd_role target_role =
+ (reply->primary_nodes & NODE_MASK(context->target_node_id)) ?
+ R_PRIMARY : R_SECONDARY;
+
+ __change_peer_role(connection, target_role);
}
- if (ns.disk == D_DISKLESS &&
- ns.conn == C_STANDALONE &&
- ns.role == R_SECONDARY) {
- if (os.aftr_isp != ns.aftr_isp)
- resume_next_sg(device);
+ return phase != PH_PREPARE ||
+ context->val.conn == C_CONNECTED ||
+ (context->val.conn == C_DISCONNECTING &&
+ connection->cstate[NOW] == C_CONNECTED);
+}
+
+/**
+ * change_cstate_tag() - change the connection state of a connection
+ * @connection: DRBD connection.
+ * @cstate: The connection state to change to.
+ * @flags: State change flags.
+ * @tag: State change tag to print in status messages.
+ * @err_str: Pointer to save the error string to.
+ *
+ * When disconnecting from a peer, we may also need to outdate the local or
+ * peer disks depending on the fencing policy. This cannot easily be split
+ * into two state changes.
+ */
+enum drbd_state_rv change_cstate_tag(struct drbd_connection *connection,
+ enum drbd_conn_state cstate,
+ enum chg_state_flags flags,
+ const char *tag,
+ const char **err_str)
+{
+ struct change_cstate_context cstate_context = {
+ .context = {
+ .resource = connection->resource,
+ .vnr = -1,
+ .mask = { { .conn = conn_MASK } },
+ .val = { { .conn = cstate } },
+ .target_node_id = connection->peer_node_id,
+ .flags = flags,
+ .change_local_state_last = true,
+ .err_str = err_str,
+ },
+ .connection = connection,
+ };
+
+ if (cstate == C_CONNECTED) {
+ cstate_context.context.mask.role = role_MASK;
+ cstate_context.context.val.role = connection->resource->role[NOW];
}
- drbd_md_sync(device);
+ /*
+ * Hard connection state changes like a protocol error or forced
+ * disconnect may occur while we are holding resource->state_sem. In
+ * that case, omit CS_SERIALIZE so that we don't deadlock trying to
+ * grab that mutex again.
+ */
+ if (!(flags & CS_HARD))
+ cstate_context.context.flags |= CS_SERIALIZE;
+
+ return change_cluster_wide_state(do_change_cstate, &cstate_context.context, tag);
}
-struct after_conn_state_chg_work {
- struct drbd_work w;
- enum drbd_conns oc;
- union drbd_state ns_min;
- union drbd_state ns_max; /* new, max state, over all devices */
- enum chg_state_flags flags;
- struct drbd_connection *connection;
- struct drbd_state_change *state_change;
+void __change_peer_role(struct drbd_connection *connection, enum drbd_role peer_role)
+{
+ connection->peer_role[NEW] = peer_role;
+}
+
+void __change_repl_state(struct drbd_peer_device *peer_device, enum drbd_repl_state repl_state)
+{
+ peer_device->repl_state[NEW] = repl_state;
+ if (repl_state > L_OFF)
+ peer_device->connection->cstate[NEW] = C_CONNECTED;
+}
+
+struct change_repl_context {
+ struct change_context context;
+ struct drbd_peer_device *peer_device;
};
-static int w_after_conn_state_ch(struct drbd_work *w, int unused)
+static bool do_change_repl_state(struct change_context *context, enum change_phase phase)
+{
+ struct change_repl_context *repl_context =
+ container_of(context, struct change_repl_context, context);
+ struct drbd_peer_device *peer_device = repl_context->peer_device;
+ enum drbd_repl_state *repl_state = peer_device->repl_state;
+ enum drbd_repl_state new_repl_state = context->val.conn;
+ bool cluster_wide = context->flags & CS_CLUSTER_WIDE;
+
+ __change_repl_state(peer_device, new_repl_state);
+
+ return phase != PH_PREPARE ||
+ ((repl_state[NOW] >= L_ESTABLISHED &&
+ (new_repl_state == L_STARTING_SYNC_S || new_repl_state == L_STARTING_SYNC_T)) ||
+ (repl_state[NOW] == L_ESTABLISHED &&
+ (new_repl_state == L_VERIFY_S || new_repl_state == L_OFF)) ||
+ (repl_state[NOW] == L_ESTABLISHED && cluster_wide &&
+ (new_repl_state == L_WF_BITMAP_S || new_repl_state == L_WF_BITMAP_T)));
+}
+
+enum drbd_state_rv change_repl_state(struct drbd_peer_device *peer_device,
+ enum drbd_repl_state new_repl_state,
+ enum chg_state_flags flags,
+ const char *tag)
+{
+ struct change_repl_context repl_context = {
+ .context = {
+ .resource = peer_device->device->resource,
+ .vnr = peer_device->device->vnr,
+ .mask = { { .conn = conn_MASK } },
+ .val = { { .conn = new_repl_state } },
+ .target_node_id = peer_device->node_id,
+ .flags = flags
+ },
+ .peer_device = peer_device
+ };
+
+ if (new_repl_state == L_WF_BITMAP_S || new_repl_state == L_VERIFY_S)
+ repl_context.context.change_local_state_last = true;
+
+ return change_cluster_wide_state(do_change_repl_state, &repl_context.context, tag);
+}
+
+enum drbd_state_rv stable_change_repl_state(struct drbd_peer_device *peer_device,
+ enum drbd_repl_state repl_state,
+ enum chg_state_flags flags,
+ const char *tag)
+{
+ return stable_state_change(peer_device->device->resource,
+ change_repl_state(peer_device, repl_state, flags, tag));
+}
+
+void __change_peer_disk_state(struct drbd_peer_device *peer_device, enum drbd_disk_state disk_state)
+{
+ peer_device->disk_state[NEW] = disk_state;
+}
+
+void __downgrade_peer_disk_states(struct drbd_connection *connection, enum drbd_disk_state disk_state)
{
- struct after_conn_state_chg_work *acscw =
- container_of(w, struct after_conn_state_chg_work, w);
- struct drbd_connection *connection = acscw->connection;
- enum drbd_conns oc = acscw->oc;
- union drbd_state ns_max = acscw->ns_max;
struct drbd_peer_device *peer_device;
int vnr;
- broadcast_state_change(acscw->state_change);
- forget_state_change(acscw->state_change);
- kfree(acscw);
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (peer_device->disk_state[NEW] > disk_state)
+ __change_peer_disk_state(peer_device, disk_state);
+ }
+ rcu_read_unlock();
+}
+
+enum drbd_state_rv change_peer_disk_state(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state disk_state,
+ enum chg_state_flags flags,
+ const char *tag)
+{
+ struct drbd_resource *resource = peer_device->device->resource;
+ unsigned long irq_flags;
- /* Upon network configuration, we need to start the receiver */
- if (oc == C_STANDALONE && ns_max.conn == C_UNCONNECTED)
- drbd_thread_start(&connection->receiver);
+ begin_state_change(resource, &irq_flags, flags);
+ __change_peer_disk_state(peer_device, disk_state);
+ return end_state_change(resource, &irq_flags, tag);
+}
- if (oc == C_DISCONNECTING && ns_max.conn == C_STANDALONE) {
- struct net_conf *old_conf;
+void __change_resync_susp_user(struct drbd_peer_device *peer_device,
+ bool value)
+{
+ peer_device->resync_susp_user[NEW] = value;
+}
- mutex_lock(¬ification_mutex);
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
- notify_peer_device_state(NULL, 0, peer_device, NULL,
- NOTIFY_DESTROY | NOTIFY_CONTINUES);
- notify_connection_state(NULL, 0, connection, NULL, NOTIFY_DESTROY);
- mutex_unlock(¬ification_mutex);
+enum drbd_state_rv change_resync_susp_user(struct drbd_peer_device *peer_device,
+ bool value,
+ enum chg_state_flags flags)
+{
+ struct drbd_resource *resource = peer_device->device->resource;
+ unsigned long irq_flags;
- mutex_lock(&connection->resource->conf_update);
- old_conf = connection->net_conf;
- connection->my_addr_len = 0;
- connection->peer_addr_len = 0;
- RCU_INIT_POINTER(connection->net_conf, NULL);
- conn_free_crypto(connection);
- mutex_unlock(&connection->resource->conf_update);
+ begin_state_change(resource, &irq_flags, flags);
+ __change_resync_susp_user(peer_device, value);
+ return end_state_change(resource, &irq_flags, value ? "pause-sync" : "resume-sync");
+}
- kvfree_rcu_mightsleep(old_conf);
- }
+void __change_resync_susp_peer(struct drbd_peer_device *peer_device,
+ bool value)
+{
+ peer_device->resync_susp_peer[NEW] = value;
+}
- if (ns_max.susp_fen) {
- /* case1: The outdate peer handler is successful: */
- if (ns_max.pdsk <= D_OUTDATED) {
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- if (test_bit(NEW_CUR_UUID, &device->flags)) {
- drbd_uuid_new_current(device);
- clear_bit(NEW_CUR_UUID, &device->flags);
- }
- }
- rcu_read_unlock();
- spin_lock_irq(&connection->resource->req_lock);
- _tl_restart(connection, CONNECTION_LOST_WHILE_PENDING);
- _conn_request_state(connection,
- (union drbd_state) { { .susp_fen = 1 } },
- (union drbd_state) { { .susp_fen = 0 } },
- CS_VERBOSE);
- spin_unlock_irq(&connection->resource->req_lock);
- }
- }
- conn_md_sync(connection);
- kref_put(&connection->kref, drbd_destroy_connection);
+void __change_resync_susp_dependency(struct drbd_peer_device *peer_device,
+ bool value)
+{
+ peer_device->resync_susp_dependency[NEW] = value;
+}
- return 0;
+void __change_resync_susp_max_parallel(struct drbd_peer_device *peer_device,
+ bool value)
+{
+ peer_device->resync_susp_max_parallel[NEW] = value;
}
-static void conn_old_common_state(struct drbd_connection *connection, union drbd_state *pcs, enum chg_state_flags *pf)
+static void log_current_uuids(struct drbd_device *device)
{
- enum chg_state_flags flags = ~0;
struct drbd_peer_device *peer_device;
- int vnr, first_vol = 1;
- union drbd_dev_state os, cs = {
- { .role = R_SECONDARY,
- .peer = R_UNKNOWN,
- .conn = connection->cstate,
- .disk = D_DISKLESS,
- .pdsk = D_UNKNOWN,
- } };
+ struct drbd_connection *connection;
+ char msg[120];
+ int ret, pos = 0;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- os = device->state;
-
- if (first_vol) {
- cs = os;
- first_vol = 0;
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->disk_state[NOW] != D_UP_TO_DATE)
continue;
+ connection = peer_device->connection;
+ ret = snprintf(msg + pos, 120 - pos, "%s: %016llX ",
+ rcu_dereference(connection->transport.net_conf)->name,
+ peer_device->current_uuid);
+ if (ret > 0)
+ pos += ret;
+ if (pos >= 120)
+ break;
+ }
+ rcu_read_unlock();
+ drbd_warn(device, "%s", msg);
+}
+
+bool drbd_res_data_accessible(struct drbd_resource *resource)
+{
+ bool data_accessible = false;
+ struct drbd_device *device;
+ int vnr;
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (drbd_data_accessible(device, NOW)) {
+ data_accessible = true;
+ break;
}
+ }
- if (cs.role != os.role)
- flags &= ~CS_DC_ROLE;
+ return data_accessible;
+}
- if (cs.peer != os.peer)
- flags &= ~CS_DC_PEER;
+/**
+ * calc_data_accessible() - returns if up-to-data data is reachable
+ *
+ * @state_change: where to get the state information from
+ * @n_device: index into the devices array
+ * @which: OLD or NEW
+ *
+ * calc_data_accessible() returns true if either the local disk is up-to-date
+ * or of the peers. The related drbd_data_accessible() computes the same
+ * result from different inputs.
+ */
+static bool calc_data_accessible(struct drbd_state_change *state_change, int n_device,
+ enum which_state which)
+{
+ struct drbd_device_state_change *device_state_change = &state_change->devices[n_device];
+ enum drbd_disk_state *disk_state = device_state_change->disk_state;
+ int n_connection;
- if (cs.conn != os.conn)
- flags &= ~CS_DC_CONN;
+ if (disk_state[which] == D_UP_TO_DATE)
+ return true;
- if (cs.disk != os.disk)
- flags &= ~CS_DC_DISK;
+ for (n_connection = 0; n_connection < state_change->n_connections; n_connection++) {
+ struct drbd_peer_device_state_change *peer_device_state_change =
+ &state_change->peer_devices[
+ n_device * state_change->n_connections + n_connection];
+ struct drbd_peer_device *peer_device = peer_device_state_change->peer_device;
+ enum drbd_disk_state *peer_disk_state = peer_device_state_change->disk_state;
+ struct drbd_net_conf *nc;
- if (cs.pdsk != os.pdsk)
- flags &= ~CS_DC_PDSK;
+ rcu_read_lock();
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ if (nc && !nc->allow_remote_read) {
+ rcu_read_unlock();
+ continue;
+ }
+ rcu_read_unlock();
+ if (peer_disk_state[which] == D_UP_TO_DATE)
+ return true;
}
- rcu_read_unlock();
- *pf |= CS_DC_MASK;
- *pf &= flags;
- (*pcs).i = cs.i;
+ return false;
}
-static enum drbd_state_rv
-conn_is_valid_transition(struct drbd_connection *connection, union drbd_state mask, union drbd_state val,
- enum chg_state_flags flags)
+/**
+ * drbd_data_accessible() - returns if up-to-data data is reachable
+ *
+ * @device: the device, the question is about
+ * @which: OLD, NEW, or NOW (Only use OLD within a state change!)
+ *
+ * drbd_data_accessible() returns true if either the local disk is up-to-date
+ * or of the peers. The related calc_data_accessible() computes the same
+ * result from different inputs.
+ */
+bool drbd_data_accessible(struct drbd_device *device, enum which_state which)
{
- enum drbd_state_rv rv = SS_SUCCESS;
- union drbd_state ns, os;
struct drbd_peer_device *peer_device;
- int vnr;
+ bool data_accessible = false;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- os = drbd_read_state(device);
- ns = sanitize_state(device, os, apply_mask_val(os, mask, val), NULL);
+ if (device->disk_state[which] == D_UP_TO_DATE)
+ return true;
- if (flags & CS_IGN_OUTD_FAIL && ns.disk == D_OUTDATED && os.disk < D_OUTDATED)
- ns.disk = os.disk;
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ struct drbd_net_conf *nc;
- if (ns.i == os.i)
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ if (nc && !nc->allow_remote_read)
continue;
-
- rv = is_valid_transition(os, ns);
-
- if (rv >= SS_SUCCESS && !(flags & CS_HARD)) {
- rv = is_valid_state(device, ns);
- if (rv < SS_SUCCESS) {
- if (is_valid_state(device, os) == rv)
- rv = is_valid_soft_transition(os, ns, connection);
- } else
- rv = is_valid_soft_transition(os, ns, connection);
- }
-
- if (rv < SS_SUCCESS) {
- if (flags & CS_VERBOSE)
- print_st_err(device, os, ns, rv);
+ if (peer_device->disk_state[which] == D_UP_TO_DATE) {
+ data_accessible = true;
break;
}
}
rcu_read_unlock();
- return rv;
+ return data_accessible;
}
-
-static void
-conn_set_state(struct drbd_connection *connection, union drbd_state mask, union drbd_state val,
- union drbd_state *pns_min, union drbd_state *pns_max, enum chg_state_flags flags)
+/* drbd_data_accessible() and exposable_data_uuid() have the same structure. By intention. */
+static u64 exposable_data_uuid(struct drbd_device *device)
{
- union drbd_state ns, os, ns_max = { };
- union drbd_state ns_min = {
- { .role = R_MASK,
- .peer = R_MASK,
- .conn = val.conn,
- .disk = D_MASK,
- .pdsk = D_MASK
- } };
struct drbd_peer_device *peer_device;
- enum drbd_state_rv rv;
- int vnr, number_of_volumes = 0;
-
- if (mask.conn == C_MASK) {
- /* remember last connect time so request_timer_fn() won't
- * kill newly established sessions while we are still trying to thaw
- * previously frozen IO */
- if (connection->cstate != C_WF_REPORT_PARAMS && val.conn == C_WF_REPORT_PARAMS)
- connection->last_reconnect_jif = jiffies;
+ u64 uuid = 0;
- connection->cstate = val.conn;
+ if (get_ldev_if_state(device, D_UP_TO_DATE)) {
+ uuid = device->ldev->md.current_uuid;
+ put_ldev(device);
+ return uuid;
}
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- number_of_volumes++;
- os = drbd_read_state(device);
- ns = apply_mask_val(os, mask, val);
- ns = sanitize_state(device, os, ns, NULL);
-
- if (flags & CS_IGN_OUTD_FAIL && ns.disk == D_OUTDATED && os.disk < D_OUTDATED)
- ns.disk = os.disk;
-
- rv = _drbd_set_state(device, ns, flags, NULL);
- BUG_ON(rv < SS_SUCCESS);
- ns.i = device->state.i;
- ns_max.role = max_role(ns.role, ns_max.role);
- ns_max.peer = max_role(ns.peer, ns_max.peer);
- ns_max.conn = max_t(enum drbd_conns, ns.conn, ns_max.conn);
- ns_max.disk = max_t(enum drbd_disk_state, ns.disk, ns_max.disk);
- ns_max.pdsk = max_t(enum drbd_disk_state, ns.pdsk, ns_max.pdsk);
-
- ns_min.role = min_role(ns.role, ns_min.role);
- ns_min.peer = min_role(ns.peer, ns_min.peer);
- ns_min.conn = min_t(enum drbd_conns, ns.conn, ns_min.conn);
- ns_min.disk = min_t(enum drbd_disk_state, ns.disk, ns_min.disk);
- ns_min.pdsk = min_t(enum drbd_disk_state, ns.pdsk, ns_min.pdsk);
- }
- rcu_read_unlock();
+ for_each_peer_device_rcu(peer_device, device) {
+ struct drbd_net_conf *nc;
- if (number_of_volumes == 0) {
- ns_min = ns_max = (union drbd_state) { {
- .role = R_SECONDARY,
- .peer = R_UNKNOWN,
- .conn = val.conn,
- .disk = D_DISKLESS,
- .pdsk = D_UNKNOWN
- } };
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ if (nc && !nc->allow_remote_read)
+ continue;
+ if (peer_device->disk_state[NOW] == D_UP_TO_DATE &&
+ (uuid & ~UUID_PRIMARY) != (peer_device->current_uuid & ~UUID_PRIMARY)) {
+ if (!uuid) {
+ uuid = peer_device->current_uuid;
+ continue;
+ }
+ drbd_err(device, "Multiple UpToDate peers have different current UUIDs\n");
+ log_current_uuids(device);
+ }
}
+ rcu_read_unlock();
- ns_min.susp = ns_max.susp = connection->resource->susp;
- ns_min.susp_nod = ns_max.susp_nod = connection->resource->susp_nod;
- ns_min.susp_fen = ns_max.susp_fen = connection->resource->susp_fen;
-
- *pns_min = ns_min;
- *pns_max = ns_max;
+ return uuid;
}
-static enum drbd_state_rv
-_conn_rq_cond(struct drbd_connection *connection, union drbd_state mask, union drbd_state val)
+static void ensure_exposed_data_uuid(struct drbd_device *device)
{
- enum drbd_state_rv err, rv = SS_UNKNOWN_ERROR; /* continue waiting */;
-
- if (test_and_clear_bit(CONN_WD_ST_CHG_OKAY, &connection->flags))
- rv = SS_CW_SUCCESS;
-
- if (test_and_clear_bit(CONN_WD_ST_CHG_FAIL, &connection->flags))
- rv = SS_CW_FAILED_BY_PEER;
+ u64 uuid = exposable_data_uuid(device);
- err = conn_is_valid_transition(connection, mask, val, 0);
- if (err == SS_SUCCESS && connection->cstate == C_WF_REPORT_PARAMS)
- return rv;
+ if (uuid)
+ drbd_uuid_set_exposed(device, uuid, true);
- return err;
}
-enum drbd_state_rv
-_conn_request_state(struct drbd_connection *connection, union drbd_state mask, union drbd_state val,
- enum chg_state_flags flags)
+/* Between 9.1.7 and 9.1.12 drbd was setting MDF_NODE_EXISTS for all peers.
+ * With that the flag got useless. It is a meta-data flag that persists.
+ * Clear it for all not configured nodes if we find it in every peer slot.
+ */
+static void check_wrongly_set_mdf_exists(struct drbd_device *device)
{
- enum drbd_state_rv rv = SS_SUCCESS;
- struct after_conn_state_chg_work *acscw;
- enum drbd_conns oc = connection->cstate;
- union drbd_state ns_max, ns_min, os;
- bool have_mutex = false;
- struct drbd_state_change *state_change;
+ struct drbd_resource *resource = device->resource;
+ const int my_node_id = resource->res_opts.node_id;
+ bool wrong = true;
+ int node_id;
- if (mask.conn) {
- rv = is_valid_conn_transition(oc, val.conn);
- if (rv < SS_SUCCESS)
- goto abort;
- }
+ if (!get_ldev(device))
+ return;
- rv = conn_is_valid_transition(connection, mask, val, flags);
- if (rv < SS_SUCCESS)
- goto abort;
-
- if (oc == C_WF_REPORT_PARAMS && val.conn == C_DISCONNECTING &&
- !(flags & (CS_LOCAL_ONLY | CS_HARD))) {
-
- /* This will be a cluster-wide state change.
- * Need to give up the spinlock, grab the mutex,
- * then send the state change request, ... */
- spin_unlock_irq(&connection->resource->req_lock);
- mutex_lock(&connection->cstate_mutex);
- have_mutex = true;
-
- set_bit(CONN_WD_ST_CHG_REQ, &connection->flags);
- if (conn_send_state_req(connection, mask, val)) {
- /* sending failed. */
- clear_bit(CONN_WD_ST_CHG_REQ, &connection->flags);
- rv = SS_CW_FAILED_BY_PEER;
- /* need to re-aquire the spin lock, though */
- goto abort_unlocked;
- }
-
- if (val.conn == C_DISCONNECTING)
- set_bit(DISCONNECT_SENT, &connection->flags);
-
- /* ... and re-aquire the spinlock.
- * If _conn_rq_cond() returned >= SS_SUCCESS, we must call
- * conn_set_state() within the same spinlock. */
- spin_lock_irq(&connection->resource->req_lock);
- wait_event_lock_irq(connection->ping_wait,
- (rv = _conn_rq_cond(connection, mask, val)),
- connection->resource->req_lock);
- clear_bit(CONN_WD_ST_CHG_REQ, &connection->flags);
- if (rv < SS_SUCCESS)
- goto abort;
- }
-
- state_change = remember_old_state(connection->resource, GFP_ATOMIC);
- conn_old_common_state(connection, &os, &flags);
- flags |= CS_DC_SUSP;
- conn_set_state(connection, mask, val, &ns_min, &ns_max, flags);
- conn_pr_state_change(connection, os, ns_max, flags);
- remember_new_state(state_change);
-
- acscw = kmalloc_obj(*acscw, GFP_ATOMIC);
- if (acscw) {
- acscw->oc = os.conn;
- acscw->ns_min = ns_min;
- acscw->ns_max = ns_max;
- acscw->flags = flags;
- acscw->w.cb = w_after_conn_state_ch;
- kref_get(&connection->kref);
- acscw->connection = connection;
- acscw->state_change = state_change;
- drbd_queue_work(&connection->sender_work, &acscw->w);
- } else {
- drbd_err(connection, "Could not kmalloc an acscw\n");
- }
+ rcu_read_lock();
- abort:
- if (have_mutex) {
- /* mutex_unlock() "... must not be used in interrupt context.",
- * so give up the spinlock, then re-aquire it */
- spin_unlock_irq(&connection->resource->req_lock);
- abort_unlocked:
- mutex_unlock(&connection->cstate_mutex);
- spin_lock_irq(&connection->resource->req_lock);
- }
- if (rv < SS_SUCCESS && flags & CS_VERBOSE) {
- drbd_err(connection, "State change failed: %s\n", drbd_set_st_err_str(rv));
- drbd_err(connection, " mask = 0x%x val = 0x%x\n", mask.i, val.i);
- drbd_err(connection, " old_conn:%s wanted_conn:%s\n", drbd_conn_str(oc), drbd_conn_str(val.conn));
- }
- return rv;
-}
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_device *peer_device = peer_device_by_node_id(device, node_id);
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
-enum drbd_state_rv
-conn_request_state(struct drbd_connection *connection, union drbd_state mask, union drbd_state val,
- enum chg_state_flags flags)
-{
- enum drbd_state_rv rv;
+ if (!test_bit(__MDF_NODE_EXISTS, &peer_md->flags) &&
+ !peer_device && node_id != my_node_id) {
+ wrong = false;
+ break;
+ }
+ }
- spin_lock_irq(&connection->resource->req_lock);
- rv = _conn_request_state(connection, mask, val, flags);
- spin_unlock_irq(&connection->resource->req_lock);
+ if (wrong) {
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_device *peer_device = peer_device_by_node_id(device, node_id);
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
- return rv;
+ if (!peer_device)
+ clear_bit(__MDF_NODE_EXISTS, &peer_md->flags);
+ }
+ if (!test_bit(WRONG_MDF_EXISTS, &resource->flags)) {
+ set_bit(WRONG_MDF_EXISTS, &resource->flags);
+ drbd_warn(resource, "Clearing excess MDF_NODE_EXISTS flags\n");
+ }
+ }
+ rcu_read_unlock();
+ put_ldev(device);
}
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 12/20] drbd: rework activity log and bitmap for multi-peer replication
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (10 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 11/20] drbd: rewrite state machine for DRBD 9 multi-peer clusters Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 13/20] drbd: rework request processing for DRBD 9 multi-peer IO Christoph Böhmwalder
` (7 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Andreas Gruenbacher,
Joel Colledge, Roland Kammerer
Adapt the activity log and on-disk bitmap from the single-peer
DRBD 8.4 model to the multi-peer DRBD 9 architecture.
Restructure the bitmap from a single flat layout to an interleaved
per-peer format: consecutive words on disk cycle through all configured
peers, so that all peers' bits for the same disk region share a page.
This enables atomic cross-peer operations and is a prerequisite for
coordinated multi-peer resync.
Consolidate all bitmap operations into a single function instead of
many separate ones.
Make the bitmap block size adjustable at runtime rather than being a
compile-time constant.
Introduce a per-peer-slot lock variant so that concurrent operations
on different peer slots no longer need to serialize.
On the activity log side, the resync extent LRU cache and its
associated write-blocking protocol are removed.
In DRBD 9, resync-to-application-write conflict detection is handled
by the sender's interval tree, making the old extent-lock layer
redundant.
Resync progress tracking moves from the device to the per-peer-device
object, enabling independent progress reporting and rate control per
peer.
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_actlog.c | 1373 ++++++++++----------
drivers/block/drbd/drbd_bitmap.c | 2075 +++++++++++++++++-------------
2 files changed, 1832 insertions(+), 1616 deletions(-)
diff --git a/drivers/block/drbd/drbd_actlog.c b/drivers/block/drbd/drbd_actlog.c
index 43345b82daa2..0158fe90c019 100644
--- a/drivers/block/drbd/drbd_actlog.c
+++ b/drivers/block/drbd/drbd_actlog.c
@@ -9,81 +9,41 @@
#include <linux/slab.h>
#include <linux/crc32c.h>
#include <linux/drbd.h>
-#include <linux/drbd_limits.h>
+#include <linux/dynamic_debug.h>
#include "drbd_int.h"
-
-
-enum al_transaction_types {
- AL_TR_UPDATE = 0,
- AL_TR_INITIALIZED = 0xffff
-};
-/* all fields on disc in big endian */
-struct __packed al_transaction_on_disk {
- /* don't we all like magic */
- __be32 magic;
-
- /* to identify the most recent transaction block
- * in the on disk ring buffer */
- __be32 tr_number;
-
- /* checksum on the full 4k block, with this field set to 0. */
- __be32 crc32c;
-
- /* type of transaction, special transaction types like:
- * purge-all, set-all-idle, set-all-active, ... to-be-defined
- * see also enum al_transaction_types */
- __be16 transaction_type;
-
- /* we currently allow only a few thousand extents,
- * so 16bit will be enough for the slot number. */
-
- /* how many updates in this transaction */
- __be16 n_updates;
-
- /* maximum slot number, "al-extents" in drbd.conf speak.
- * Having this in each transaction should make reconfiguration
- * of that parameter easier. */
- __be16 context_size;
-
- /* slot number the context starts with */
- __be16 context_start_slot_nr;
-
- /* Some reserved bytes. Expected usage is a 64bit counter of
- * sectors-written since device creation, and other data generation tag
- * supporting usage */
- __be32 __reserved[4];
-
- /* --- 36 byte used --- */
-
- /* Reserve space for up to AL_UPDATES_PER_TRANSACTION changes
- * in one transaction, then use the remaining byte in the 4k block for
- * context information. "Flexible" number of updates per transaction
- * does not help, as we have to account for the case when all update
- * slots are used anyways, so it would only complicate code without
- * additional benefit.
- */
- __be16 update_slot_nr[AL_UPDATES_PER_TRANSACTION];
-
- /* but the extent number is 32bit, which at an extent size of 4 MiB
- * allows to cover device sizes of up to 2**54 Byte (16 PiB) */
- __be32 update_extent_nr[AL_UPDATES_PER_TRANSACTION];
-
- /* --- 420 bytes used (36 + 64*6) --- */
-
- /* 4096 - 420 = 3676 = 919 * 4 */
- __be32 context[AL_CONTEXT_PER_TRANSACTION];
-};
+#include "drbd_meta_data.h"
+#include "drbd_dax_pmem.h"
void *drbd_md_get_buffer(struct drbd_device *device, const char *intent)
{
int r;
+ long t;
+ unsigned long t0 = jiffies;
+ unsigned int warn_s = 10;
+
+ for (;;) {
+ t = wait_event_timeout(device->misc_wait,
+ (r = atomic_cmpxchg(&device->md_io.in_use, 0, 1)) == 0 ||
+ device->disk_state[NOW] <= D_FAILED,
+ HZ * warn_s);
+
+ if (r == 0)
+ break;
- wait_event(device->misc_wait,
- (r = atomic_cmpxchg(&device->md_io.in_use, 0, 1)) == 0 ||
- device->state.disk <= D_FAILED);
+ if (t != 0) {
+ drbd_err(device, "Failed to get md_buffer for %s: disk state %s\n",
+ intent, drbd_disk_str(device->disk_state[NOW]));
+ return NULL;
+ }
- if (r)
- return NULL;
+ /* r != 0, t == 0: still in use, hit the timeout above.
+ * Warn, but keep trying.
+ */
+ drbd_err(device, "Waited %lds on md_buffer for %s; in use by %s; still trying...\n",
+ (jiffies - t0 + HZ-1)/HZ, intent, device->md_io.current_use);
+ /* reduce warn frequency */
+ warn_s = max(30U, warn_s + 10);
+ }
device->md_io.current_use = intent;
device->md_io.start_jif = jiffies;
@@ -97,8 +57,8 @@ void drbd_md_put_buffer(struct drbd_device *device)
wake_up(&device->misc_wait);
}
-void wait_until_done_or_force_detached(struct drbd_device *device, struct drbd_backing_dev *bdev,
- unsigned int *done)
+/* disk-timeout in jiffies, MAX_SCHEDULE_TIMEOUT if it is not set */
+long drbd_disk_timeout(struct drbd_backing_dev *bdev)
{
long dt;
@@ -109,11 +69,23 @@ void wait_until_done_or_force_detached(struct drbd_device *device, struct drbd_b
if (dt == 0)
dt = MAX_SCHEDULE_TIMEOUT;
+ return dt;
+}
+
+void wait_until_done_or_force_detached(struct drbd_device *device, struct drbd_backing_dev *bdev,
+ unsigned int *done)
+{
+ long dt = drbd_disk_timeout(bdev);
+
dt = wait_event_timeout(device->misc_wait,
- *done || test_bit(FORCE_DETACH, &device->flags), dt);
+ *done ||
+ test_bit(FORCE_DETACH, &device->flags) ||
+ test_bit(ABORT_MDIO, &device->flags),
+ dt);
+
if (dt == 0) {
drbd_err(device, "meta-data IO operation timed out\n");
- drbd_chk_io_error(device, 1, DRBD_FORCE_DETACH);
+ drbd_handle_io_error(device, DRBD_FORCE_DETACH);
}
}
@@ -127,15 +99,15 @@ static int _drbd_md_sync_page_io(struct drbd_device *device,
int err;
blk_opf_t op_flags = 0;
- device->md_io.done = 0;
- device->md_io.error = -ENODEV;
-
if ((op == REQ_OP_WRITE) && !test_bit(MD_NO_FUA, &device->flags))
op_flags |= REQ_FUA | REQ_PREFLUSH;
- op_flags |= REQ_SYNC;
+ op_flags |= REQ_META | REQ_SYNC;
+
+ device->md_io.done = 0;
+ device->md_io.error = -ENODEV;
- bio = bio_alloc_bioset(bdev->md_bdev, 1, op | op_flags, GFP_NOIO,
- &drbd_md_io_bio_set);
+ bio = bio_alloc_bioset(bdev->md_bdev, 1, op | op_flags,
+ GFP_NOIO, &drbd_md_io_bio_set);
bio->bi_iter.bi_sector = sector;
err = -EIO;
if (bio_add_page(bio, device->md_io.page, size, 0) != size)
@@ -143,8 +115,8 @@ static int _drbd_md_sync_page_io(struct drbd_device *device,
bio->bi_private = device;
bio->bi_end_io = drbd_md_endio;
- if (op != REQ_OP_WRITE && device->state.disk == D_DISKLESS && device->ldev == NULL)
- /* special case, drbd_md_read() during drbd_adm_attach(): no get_ldev */
+ if (op != REQ_OP_WRITE && device->disk_state[NOW] == D_DISKLESS && device->ldev == NULL)
+ /* special case, drbd_md_read() during drbd_nl_attach_doit(): no get_ldev */
;
else if (!get_ldev_if_state(device, D_ATTACHING)) {
/* Corresponding put_ldev in drbd_md_endio() */
@@ -156,14 +128,14 @@ static int _drbd_md_sync_page_io(struct drbd_device *device,
bio_get(bio); /* one bio_put() is in the completion handler */
atomic_inc(&device->md_io.in_use); /* drbd_md_put_buffer() is in the completion handler */
device->md_io.submit_jif = jiffies;
- if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD))
- bio_io_error(bio);
- else
+ if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) {
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ } else {
submit_bio(bio);
+ }
wait_until_done_or_force_detached(device, bdev, &device->md_io.done);
- if (!bio->bi_status)
- err = device->md_io.error;
-
+ err = device->md_io.error;
out:
bio_put(bio);
return err;
@@ -173,14 +145,18 @@ int drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bd
sector_t sector, enum req_op op)
{
int err;
+
D_ASSERT(device, atomic_read(&device->md_io.in_use) == 1);
- BUG_ON(!bdev->md_bdev);
+ if (!bdev->md_bdev) {
+ drbd_err_ratelimit(device, "bdev->md_bdev==NULL\n");
+ return -EIO;
+ }
dynamic_drbd_dbg(device, "meta_data io: %s [%d]:%s(,%llus,%s) %pS\n",
current->comm, current->pid, __func__,
(unsigned long long)sector, (op == REQ_OP_WRITE) ? "WRITE" : "READ",
- (void*)_RET_IP_ );
+ (void *)_RET_IP_);
if (sector < drbd_md_first_sector(bdev) ||
sector + 7 > drbd_md_last_sector(bdev))
@@ -198,96 +174,147 @@ int drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bd
return err;
}
-static struct bm_extent *find_active_resync_extent(struct drbd_device *device, unsigned int enr)
+bool drbd_al_active(struct drbd_device *device, sector_t sector, unsigned int size)
{
- struct lc_element *tmp;
- tmp = lc_find(device->resync, enr/AL_EXT_PER_BM_SECT);
- if (unlikely(tmp != NULL)) {
- struct bm_extent *bm_ext = lc_entry(tmp, struct bm_extent, lce);
- if (test_bit(BME_NO_WRITES, &bm_ext->flags))
- return bm_ext;
+ unsigned int first = sector >> (AL_EXTENT_SHIFT-9);
+ unsigned int last = size == 0 ? first : (sector + (size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
+ unsigned int enr;
+ bool active = false;
+
+ spin_lock_irq(&device->al_lock);
+ for (enr = first; enr <= last; enr++) {
+ struct lc_element *al_ext;
+
+ al_ext = lc_find(device->act_log, enr);
+ if (al_ext && al_ext->refcnt > 0) {
+ active = true;
+ break;
+ }
}
- return NULL;
+ spin_unlock_irq(&device->al_lock);
+
+ return active;
}
-static struct lc_element *_al_get(struct drbd_device *device, unsigned int enr, bool nonblock)
+static
+struct lc_element *_al_get_nonblock(struct drbd_device *device, unsigned int enr)
{
struct lc_element *al_ext;
- struct bm_extent *bm_ext;
- int wake;
spin_lock_irq(&device->al_lock);
- bm_ext = find_active_resync_extent(device, enr);
- if (bm_ext) {
- wake = !test_and_set_bit(BME_PRIORITY, &bm_ext->flags);
- spin_unlock_irq(&device->al_lock);
- if (wake)
- wake_up(&device->al_wait);
- return NULL;
- }
- if (nonblock)
- al_ext = lc_try_get(device->act_log, enr);
- else
- al_ext = lc_get(device->act_log, enr);
+ al_ext = lc_try_get(device->act_log, enr);
spin_unlock_irq(&device->al_lock);
+
return al_ext;
}
-bool drbd_al_begin_io_fastpath(struct drbd_device *device, struct drbd_interval *i)
+#if IS_ENABLED(CONFIG_DEV_DAX_PMEM)
+static
+struct lc_element *_al_get(struct drbd_device *device, unsigned int enr)
{
- /* for bios crossing activity log extent boundaries,
- * we may need to activate two extents in one go */
- unsigned first = i->sector >> (AL_EXTENT_SHIFT-9);
- unsigned last = i->size == 0 ? first : (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
+ struct lc_element *al_ext;
- D_ASSERT(device, first <= last);
- D_ASSERT(device, atomic_read(&device->local_cnt) > 0);
+ spin_lock_irq(&device->al_lock);
+ al_ext = lc_get(device->act_log, enr);
+ spin_unlock_irq(&device->al_lock);
- /* FIXME figure out a fast path for bios crossing AL extent boundaries */
- if (first != last)
- return false;
+ return al_ext;
+}
+
+static bool
+drbd_dax_begin_io_fp(struct drbd_device *device, unsigned int first, unsigned int last)
+{
+ struct lc_element *al_ext;
+ unsigned long flags;
+ unsigned int enr;
+ unsigned int abort_enr;
+ bool wake = 0;
- return _al_get(device, first, true);
+ for (enr = first; enr <= last; enr++) {
+ al_ext = _al_get(device, enr);
+ if (!al_ext)
+ goto abort;
+
+ if (al_ext->lc_number != enr) {
+ spin_lock_irqsave(&device->al_lock, flags);
+ drbd_dax_al_update(device, al_ext);
+ lc_committed(device->act_log);
+ spin_unlock_irqrestore(&device->al_lock, flags);
+ }
+ }
+ return true;
+abort:
+ abort_enr = enr;
+ for (enr = first; enr < abort_enr; enr++) {
+ spin_lock_irqsave(&device->al_lock, flags);
+ al_ext = lc_find(device->act_log, enr);
+ wake |= lc_put(device->act_log, al_ext) == 0;
+ spin_unlock_irqrestore(&device->al_lock, flags);
+ }
+ if (wake)
+ wake_up(&device->al_wait);
+ return false;
+}
+#else
+static bool
+drbd_dax_begin_io_fp(struct drbd_device *device, unsigned int first, unsigned int last)
+{
+ return false;
}
+#endif
-bool drbd_al_begin_io_prepare(struct drbd_device *device, struct drbd_interval *i)
+bool drbd_al_begin_io_fastpath(struct drbd_device *device, struct drbd_interval *i)
{
/* for bios crossing activity log extent boundaries,
* we may need to activate two extents in one go */
- unsigned first = i->sector >> (AL_EXTENT_SHIFT-9);
- unsigned last = i->size == 0 ? first : (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
- unsigned enr;
- bool need_transaction = false;
+ unsigned int first = i->sector >> (AL_EXTENT_SHIFT-9);
+ unsigned int last = i->size == 0 ? first :
+ (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
D_ASSERT(device, first <= last);
D_ASSERT(device, atomic_read(&device->local_cnt) > 0);
- for (enr = first; enr <= last; enr++) {
- struct lc_element *al_ext;
- wait_event(device->al_wait,
- (al_ext = _al_get(device, enr, false)) != NULL);
- if (al_ext->lc_number != enr)
- need_transaction = true;
- }
- return need_transaction;
+ if (drbd_md_dax_active(device->ldev))
+ return drbd_dax_begin_io_fp(device, first, last);
+
+ /* FIXME figure out a fast path for bios crossing AL extent boundaries */
+ if (first != last)
+ return false;
+
+ if (test_bit(AL_ENABLE_PENDING, &device->flags))
+ return false;
+
+ return _al_get_nonblock(device, first) != NULL;
}
-#if (PAGE_SHIFT + 3) < (AL_EXTENT_SHIFT - BM_BLOCK_SHIFT)
-/* Currently BM_BLOCK_SHIFT, BM_EXT_SHIFT and AL_EXTENT_SHIFT
+#if AL_EXTENT_SHIFT > 27
+/* Condition used to be:
+ * (PAGE_SHIFT + 3) < (AL_EXTENT_SHIFT - BM_BLOCK_SHIFT)
+ * """
+ * Currently BM_BLOCK_SHIFT and AL_EXTENT_SHIFT
* are still coupled, or assume too much about their relation.
* Code below will not work if this is violated.
- * Will be cleaned up with some followup patch.
+ * """
+ *
+ * We want configurable bitmap granularity now.
+ * We only allow bytes per bit >= 4k, though: BM_BLOCK_SHIFT >= 12,
+ * increasing it will only make the right side smaller,
+ * not changing the boolean result.
+ * PAGE_SHIFT is 12 or larger (may be 14,16,18 for some architectures).
+ * That will increase the left side, so won't change the boolean result.
+ *
+ * Unfortunately I don't remember the specifics of which simplifications
+ * below this is supposed to protect.
+ * But assuming it is still relevant,
+ * we keep AL_EXTENT_SHIFT at 22, and must not increase it above 27
+ * without proving the code below to still work.
*/
# error FIXME
#endif
-static unsigned int al_extent_to_bm_page(unsigned int al_enr)
+static unsigned long al_extent_to_bm_bit(const struct drbd_bitmap *bm, unsigned int al_enr)
{
- return al_enr >>
- /* bit to page */
- ((PAGE_SHIFT + 3) -
- /* al extent number to bit */
- (AL_EXTENT_SHIFT - BM_BLOCK_SHIFT));
+ return (unsigned long)al_enr << (AL_EXTENT_SHIFT - bm->bm_block_shift);
}
static sector_t al_tr_number_to_on_disk_sector(struct drbd_device *device)
@@ -310,12 +337,14 @@ static sector_t al_tr_number_to_on_disk_sector(struct drbd_device *device)
static int __al_write_transaction(struct drbd_device *device, struct al_transaction_on_disk *buffer)
{
+ struct drbd_bitmap *bm = device->bitmap;
struct lc_element *e;
sector_t sector;
int i, mx;
- unsigned extent_nr;
- unsigned crc = 0;
+ unsigned int extent_nr;
+ unsigned int crc = 0;
int err = 0;
+ ktime_var_for_accounting(start_kt);
memset(buffer, 0, sizeof(*buffer));
buffer->magic = cpu_to_be32(DRBD_AL_MAGIC);
@@ -337,9 +366,13 @@ static int __al_write_transaction(struct drbd_device *device, struct al_transact
}
buffer->update_slot_nr[i] = cpu_to_be16(e->lc_index);
buffer->update_extent_nr[i] = cpu_to_be32(e->lc_new_number);
- if (e->lc_number != LC_FREE)
- drbd_bm_mark_for_writeout(device,
- al_extent_to_bm_page(e->lc_number));
+ if (e->lc_number != LC_FREE) {
+ unsigned long start, end;
+
+ start = al_extent_to_bm_bit(bm, e->lc_number);
+ end = al_extent_to_bm_bit(bm, e->lc_number + 1) - 1;
+ drbd_bm_mark_range_for_writeout(device, start, end);
+ }
i++;
}
spin_unlock_irq(&device->al_lock);
@@ -357,7 +390,8 @@ static int __al_write_transaction(struct drbd_device *device, struct al_transact
mx = min_t(int, AL_CONTEXT_PER_TRANSACTION,
device->act_log->nr_elements - device->al_tr_cycle);
for (i = 0; i < mx; i++) {
- unsigned idx = device->al_tr_cycle + i;
+ unsigned int idx = device->al_tr_cycle + i;
+
extent_nr = lc_element_by_index(device->act_log, idx)->lc_number;
buffer->context[i] = cpu_to_be32(extent_nr);
}
@@ -373,22 +407,21 @@ static int __al_write_transaction(struct drbd_device *device, struct al_transact
crc = crc32c(0, buffer, 4096);
buffer->crc32c = cpu_to_be32(crc);
- if (drbd_bm_write_hinted(device))
- err = -EIO;
- else {
- bool write_al_updates;
- rcu_read_lock();
- write_al_updates = rcu_dereference(device->ldev->disk_conf)->al_updates;
- rcu_read_unlock();
- if (write_al_updates) {
- if (drbd_md_sync_page_io(device, device->ldev, sector, REQ_OP_WRITE)) {
- err = -EIO;
- drbd_chk_io_error(device, 1, DRBD_META_IO_ERROR);
- } else {
- device->al_tr_number++;
- device->al_writ_cnt++;
- }
+ ktime_aggregate_delta(device, start_kt, al_before_bm_write_hinted_kt);
+ err = drbd_bm_write_hinted(device);
+ if (!err) {
+ ktime_aggregate_delta(device, start_kt, al_mid_kt);
+ if (drbd_md_sync_page_io(device, device->ldev, sector, REQ_OP_WRITE)) {
+ err = -EIO;
+ drbd_handle_io_error(device, DRBD_META_IO_ERROR);
+ } else {
+ device->al_tr_number++;
+ device->al_writ_cnt++;
+ device->al_histogram[min_t(unsigned int,
+ device->act_log->pending_changes,
+ AL_UPDATES_PER_TRANSACTION)]++;
}
+ ktime_aggregate_delta(device, start_kt, al_after_sync_page_kt);
}
return err;
@@ -401,15 +434,15 @@ static int al_write_transaction(struct drbd_device *device)
if (!get_ldev(device)) {
drbd_err(device, "disk is %s, cannot start al transaction\n",
- drbd_disk_str(device->state.disk));
+ drbd_disk_str(device->disk_state[NOW]));
return -EIO;
}
/* The bitmap write may have failed, causing a state change. */
- if (device->state.disk < D_INCONSISTENT) {
+ if (device->disk_state[NOW] < D_INCONSISTENT) {
drbd_err(device,
"disk is %s, cannot write al transaction\n",
- drbd_disk_str(device->state.disk));
+ drbd_disk_str(device->disk_state[NOW]));
put_ldev(device);
return -EIO;
}
@@ -430,30 +463,90 @@ static int al_write_transaction(struct drbd_device *device)
return err;
}
+bool drbd_al_try_lock(struct drbd_device *device)
+{
+ bool locked;
+
+ spin_lock_irq(&device->al_lock);
+ locked = lc_try_lock(device->act_log);
+ spin_unlock_irq(&device->al_lock);
+
+ return locked;
+}
+
+bool drbd_al_try_lock_for_transaction(struct drbd_device *device)
+{
+ bool locked;
+
+ spin_lock_irq(&device->al_lock);
+ locked = lc_try_lock_for_transaction(device->act_log);
+ spin_unlock_irq(&device->al_lock);
+
+ return locked;
+}
+
+/* While AL updates were off, extents became active without a transaction, and
+ * evicted extents left their bitmap pages unwritten. Put both on disk.
+ */
+static bool al_enable_updates(struct drbd_device *device)
+{
+ struct lru_cache *al = device->act_log;
+ bool enable;
+ int i, n = 0;
+
+ for (i = 0; i < al->nr_elements; i++) {
+ if (lc_element_by_index(al, i)->lc_number != LC_FREE) {
+ n = DIV_ROUND_UP(al->nr_elements, AL_CONTEXT_PER_TRANSACTION);
+ break;
+ }
+ }
+
+ if (drbd_bm_write_copy_pages(device, NULL))
+ return false;
+ for (i = 0; i < n; i++) {
+ if (al_write_transaction(device))
+ return false;
+ }
+ /* The AL lock serializes against the other md.flags writers; the read
+ * lock against the write-back in finish_state_change().
+ */
+ read_lock_irq(&device->resource->state_rwlock);
+ enable = test_and_clear_bit(AL_ENABLE_PENDING, &device->flags);
+ if (enable)
+ device->ldev->md.flags &= ~MDF_AL_DISABLED;
+ read_unlock_irq(&device->resource->state_rwlock);
+ if (enable)
+ drbd_md_sync(device);
+ return n > 0;
+}
void drbd_al_begin_io_commit(struct drbd_device *device)
{
bool locked = false;
- /* Serialize multiple transactions.
- * This uses test_and_set_bit, memory barrier is implicit.
- */
+
+ if (drbd_md_dax_active(device->ldev)) {
+ drbd_dax_al_begin_io_commit(device);
+ return;
+ }
+
wait_event(device->al_wait,
- device->act_log->pending_changes == 0 ||
- (locked = lc_try_lock_for_transaction(device->act_log)));
+ (device->act_log->pending_changes == 0 &&
+ !test_bit(AL_ENABLE_PENDING, &device->flags)) ||
+ (locked = drbd_al_try_lock_for_transaction(device)));
if (locked) {
- /* Double check: it may have been committed by someone else,
- * while we have been waiting for the lock. */
- if (device->act_log->pending_changes) {
- bool write_al_updates;
+ /* A transaction written there also carries the pending changes. */
+ bool written = test_bit(AL_ENABLE_PENDING, &device->flags) &&
+ al_enable_updates(device);
- rcu_read_lock();
- write_al_updates = rcu_dereference(device->ldev->disk_conf)->al_updates;
- rcu_read_unlock();
+ /* Double check: it may have been committed by someone else
+ * while we were waiting for the lock. */
+ if (!written && device->act_log->pending_changes &&
+ !(device->ldev->md.flags & MDF_AL_DISABLED))
+ al_write_transaction(device);
- if (write_al_updates)
- al_write_transaction(device);
+ if (device->act_log->pending_changes) {
spin_lock_irq(&device->al_lock);
/* FIXME
if (err)
@@ -467,22 +560,42 @@ void drbd_al_begin_io_commit(struct drbd_device *device)
}
}
-/*
- * @delegate: delegate activity log I/O to the worker thread
- */
-void drbd_al_begin_io(struct drbd_device *device, struct drbd_interval *i)
+static bool put_actlog(struct drbd_device *device, unsigned int first, unsigned int last)
{
- if (drbd_al_begin_io_prepare(device, i))
- drbd_al_begin_io_commit(device);
+ struct lc_element *extent;
+ unsigned long flags;
+ unsigned int enr;
+ bool wake = false;
+
+ D_ASSERT(device, first <= last);
+ spin_lock_irqsave(&device->al_lock, flags);
+ for (enr = first; enr <= last; enr++) {
+ extent = lc_find(device->act_log, enr);
+ /* Yes, this masks a bug elsewhere. However, during normal
+ * operation this is harmless, so no need to crash the kernel
+ * by the BUG_ON(refcount == 0) in lc_put().
+ */
+ if (!extent || extent->refcnt == 0) {
+ drbd_err(device, "al_complete_io() called on inactive extent %u\n", enr);
+ continue;
+ }
+ if (lc_put(device->act_log, extent) == 0)
+ wake = true;
+ }
+ spin_unlock_irqrestore(&device->al_lock, flags);
+ if (wake)
+ wake_up(&device->al_wait);
+ return wake;
}
int drbd_al_begin_io_nonblock(struct drbd_device *device, struct drbd_interval *i)
{
/* for bios crossing activity log extent boundaries,
* we may need to activate two extents in one go */
- unsigned first = i->sector >> (AL_EXTENT_SHIFT-9);
- unsigned last = i->size == 0 ? first : (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
- unsigned enr;
+ unsigned int first = i->sector >> (AL_EXTENT_SHIFT-9);
+ unsigned int last = i->size == 0 ? first :
+ (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
+ unsigned int enr;
if (i->partially_in_al_next_enr) {
D_ASSERT(device, first < i->partially_in_al_next_enr);
@@ -492,23 +605,10 @@ int drbd_al_begin_io_nonblock(struct drbd_device *device, struct drbd_interval *
D_ASSERT(device, first <= last);
- /* Is resync active in this area? */
- for (enr = first; enr <= last; enr++) {
- struct lc_element *tmp;
- tmp = lc_find(device->resync, enr/AL_EXT_PER_BM_SECT);
- if (unlikely(tmp != NULL)) {
- struct bm_extent *bm_ext = lc_entry(tmp, struct bm_extent, lce);
- if (test_bit(BME_NO_WRITES, &bm_ext->flags)) {
- if (!test_and_set_bit(BME_PRIORITY, &bm_ext->flags))
- return -EBUSY;
- return -EWOULDBLOCK;
- }
- }
- }
-
/* Try to checkout the refcounts. */
for (enr = first; enr <= last; enr++) {
struct lc_element *al_ext;
+
al_ext = lc_get_cumulative(device->act_log, enr);
if (!al_ext) {
@@ -525,33 +625,19 @@ int drbd_al_begin_io_nonblock(struct drbd_device *device, struct drbd_interval *
return 0;
}
-void drbd_al_complete_io(struct drbd_device *device, struct drbd_interval *i)
+/* put activity log extent references corresponding to interval i, return true
+ * if at least one extent is now unreferenced. */
+bool drbd_al_complete_io(struct drbd_device *device, struct drbd_interval *i)
{
/* for bios crossing activity log extent boundaries,
* we may need to activate two extents in one go */
- unsigned first = i->sector >> (AL_EXTENT_SHIFT-9);
- unsigned last = i->size == 0 ? first : (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
- unsigned enr;
- struct lc_element *extent;
- unsigned long flags;
+ unsigned int first = i->sector >> (AL_EXTENT_SHIFT-9);
+ unsigned int last = i->size == 0 ? first :
+ (i->sector + (i->size >> 9) - 1) >> (AL_EXTENT_SHIFT-9);
D_ASSERT(device, first <= last);
- spin_lock_irqsave(&device->al_lock, flags);
- for (enr = first; enr <= last; enr++) {
- extent = lc_find(device->act_log, enr);
- /* Yes, this masks a bug elsewhere. However, during normal
- * operation this is harmless, so no need to crash the kernel
- * by the BUG_ON(refcount == 0) in lc_put().
- */
- if (!extent || extent->refcnt == 0) {
- drbd_err(device, "al_complete_io() called on inactive extent %u\n", enr);
- continue;
- }
- lc_put(device->act_log, extent);
- }
- spin_unlock_irqrestore(&device->al_lock, flags);
- wake_up(&device->al_wait);
+ return put_actlog(device, first, last);
}
static int _try_lc_del(struct drbd_device *device, struct lc_element *al_ext)
@@ -600,6 +686,9 @@ int drbd_al_initialize(struct drbd_device *device, void *buffer)
int al_size_4k = md->al_stripes * md->al_stripe_size_4k;
int i;
+ if (drbd_md_dax_active(device->ldev))
+ return drbd_dax_al_initialize(device);
+
__al_write_transaction(device, al);
/* There may or may not have been a pending transaction. */
spin_lock_irq(&device->al_lock);
@@ -611,225 +700,134 @@ int drbd_al_initialize(struct drbd_device *device, void *buffer)
* on-disk ring buffer. */
for (i = 1; i < al_size_4k; i++) {
int err = __al_write_transaction(device, al);
+
if (err)
return err;
}
return 0;
}
-static const char *drbd_change_sync_fname[] = {
- [RECORD_RS_FAILED] = "drbd_rs_failed_io",
- [SET_IN_SYNC] = "drbd_set_in_sync",
- [SET_OUT_OF_SYNC] = "drbd_set_out_of_sync"
-};
-
-/* ATTENTION. The AL's extents are 4MB each, while the extents in the
- * resync LRU-cache are 16MB each.
- * The caller of this function has to hold an get_ldev() reference.
- *
- * Adjusts the caching members ->rs_left (success) or ->rs_failed (!success),
- * potentially pulling in (and recounting the corresponding bits)
- * this resync extent into the resync extent lru cache.
- *
- * Returns whether all bits have been cleared for this resync extent,
- * precisely: (rs_left <= rs_failed)
- *
- * TODO will be obsoleted once we have a caching lru of the on disk bitmap
- */
-static bool update_rs_extent(struct drbd_device *device,
- unsigned int enr, int count,
- enum update_sync_bits_mode mode)
+void drbd_advance_rs_marks(struct drbd_peer_device *peer_device, unsigned long still_to_go)
{
- struct lc_element *e;
+ unsigned long now;
+ int next;
- D_ASSERT(device, atomic_read(&device->local_cnt));
-
- /* When setting out-of-sync bits,
- * we don't need it cached (lc_find).
- * But if it is present in the cache,
- * we should update the cached bit count.
- * Otherwise, that extent should be in the resync extent lru cache
- * already -- or we want to pull it in if necessary -- (lc_get),
- * then update and check rs_left and rs_failed. */
- if (mode == SET_OUT_OF_SYNC)
- e = lc_find(device->resync, enr);
- else
- e = lc_get(device->resync, enr);
- if (e) {
- struct bm_extent *ext = lc_entry(e, struct bm_extent, lce);
- if (ext->lce.lc_number == enr) {
- if (mode == SET_IN_SYNC)
- ext->rs_left -= count;
- else if (mode == SET_OUT_OF_SYNC)
- ext->rs_left += count;
- else
- ext->rs_failed += count;
- if (ext->rs_left < ext->rs_failed) {
- drbd_warn(device, "BAD! enr=%u rs_left=%d "
- "rs_failed=%d count=%d cstate=%s\n",
- ext->lce.lc_number, ext->rs_left,
- ext->rs_failed, count,
- drbd_conn_str(device->state.conn));
-
- /* We don't expect to be able to clear more bits
- * than have been set when we originally counted
- * the set bits to cache that value in ext->rs_left.
- * Whatever the reason (disconnect during resync,
- * delayed local completion of an application write),
- * try to fix it up by recounting here. */
- ext->rs_left = drbd_bm_e_weight(device, enr);
- }
- } else {
- /* Normally this element should be in the cache,
- * since drbd_rs_begin_io() pulled it already in.
- *
- * But maybe an application write finished, and we set
- * something outside the resync lru_cache in sync.
- */
- int rs_left = drbd_bm_e_weight(device, enr);
- if (ext->flags != 0) {
- drbd_warn(device, "changing resync lce: %d[%u;%02lx]"
- " -> %d[%u;00]\n",
- ext->lce.lc_number, ext->rs_left,
- ext->flags, enr, rs_left);
- ext->flags = 0;
- }
- if (ext->rs_failed) {
- drbd_warn(device, "Kicking resync_lru element enr=%u "
- "out with rs_failed=%d\n",
- ext->lce.lc_number, ext->rs_failed);
- }
- ext->rs_left = rs_left;
- ext->rs_failed = (mode == RECORD_RS_FAILED) ? count : 0;
- /* we don't keep a persistent log of the resync lru,
- * we can commit any change right away. */
- lc_committed(device->resync);
- }
- if (mode != SET_OUT_OF_SYNC)
- lc_put(device->resync, &ext->lce);
- /* no race, we are within the al_lock! */
+ /* report progress and advance marks only if we made progress */
+ if (peer_device->rs_mark_left[peer_device->rs_last_mark] == still_to_go)
+ return;
- if (ext->rs_left <= ext->rs_failed) {
- ext->rs_failed = 0;
- return true;
- }
- } else if (mode != SET_OUT_OF_SYNC) {
- /* be quiet if lc_find() did not find it. */
- drbd_err(device, "lc_get() failed! locked=%d/%d flags=%lu\n",
- device->resync_locked,
- device->resync->nr_elements,
- device->resync->flags);
- }
- return false;
-}
+ /* report progress and advance marks at most once every DRBD_SYNC_MARK_STEP (3 seconds) */
+ now = jiffies;
+ if (!time_after_eq(now, peer_device->rs_last_progress_report_ts + DRBD_SYNC_MARK_STEP))
+ return;
-void drbd_advance_rs_marks(struct drbd_peer_device *peer_device, unsigned long still_to_go)
-{
- struct drbd_device *device = peer_device->device;
- unsigned long now = jiffies;
- unsigned long last = device->rs_mark_time[device->rs_last_mark];
- int next = (device->rs_last_mark + 1) % DRBD_SYNC_MARKS;
- if (time_after_eq(now, last + DRBD_SYNC_MARK_STEP)) {
- if (device->rs_mark_left[device->rs_last_mark] != still_to_go &&
- device->state.conn != C_PAUSED_SYNC_T &&
- device->state.conn != C_PAUSED_SYNC_S) {
- device->rs_mark_time[next] = now;
- device->rs_mark_left[next] = still_to_go;
- device->rs_last_mark = next;
- }
+ /* Do not advance marks if we are "paused" */
+ if (peer_device->repl_state[NOW] != L_PAUSED_SYNC_T &&
+ peer_device->repl_state[NOW] != L_PAUSED_SYNC_S) {
+ next = (peer_device->rs_last_mark + 1) % DRBD_SYNC_MARKS;
+ peer_device->rs_mark_time[next] = now;
+ peer_device->rs_mark_left[next] = still_to_go;
+ peer_device->rs_last_mark = next;
}
-}
-/* It is called lazy update, so don't do write-out too often. */
-static bool lazy_bitmap_update_due(struct drbd_device *device)
-{
- return time_after(jiffies, device->rs_last_bcast + 2*HZ);
+ /* But still report progress even if paused. */
+ peer_device->rs_last_progress_report_ts = now;
+ drbd_peer_device_post_work(peer_device, RS_PROGRESS);
}
-static void maybe_schedule_on_disk_bitmap_update(struct drbd_device *device, bool rs_done)
+/* It is called lazy update, so don't do write-out too often. */
+bool drbd_lazy_bitmap_update_due(struct drbd_peer_device *peer_device)
{
- if (rs_done) {
- struct drbd_connection *connection = first_peer_device(device)->connection;
- if (connection->agreed_pro_version <= 95 ||
- is_sync_target_state(device->state.conn))
- set_bit(RS_DONE, &device->flags);
- /* and also set RS_PROGRESS below */
-
- /* Else: rather wait for explicit notification via receive_state,
- * to avoid uuids-rotated-too-fast causing full resync
- * in next handshake, in case the replication link breaks
- * at the most unfortunate time... */
- } else if (!lazy_bitmap_update_due(device))
- return;
-
- drbd_device_post_work(device, RS_PROGRESS);
+ return time_after(jiffies, peer_device->rs_last_writeout + 2*HZ);
}
-static int update_sync_bits(struct drbd_device *device,
+static int update_sync_bits(struct drbd_peer_device *peer_device,
unsigned long sbnr, unsigned long ebnr,
enum update_sync_bits_mode mode)
{
- /*
- * We keep a count of set bits per resync-extent in the ->rs_left
- * caching member, so we need to loop and work within the resync extent
- * alignment. Typically this loop will execute exactly once.
- */
- unsigned long flags;
+ struct drbd_device *device = peer_device->device;
unsigned long count = 0;
- unsigned int cleared = 0;
- while (sbnr <= ebnr) {
- /* set temporary boundary bit number to last bit number within
- * the resync extent of the current start bit number,
- * but cap at provided end bit number */
- unsigned long tbnr = min(ebnr, sbnr | BM_BLOCKS_PER_BM_EXT_MASK);
- unsigned long c;
-
- if (mode == RECORD_RS_FAILED)
- /* Only called from drbd_rs_failed_io(), bits
- * supposedly still set. Recount, maybe some
- * of the bits have been successfully cleared
- * by application IO meanwhile.
- */
- c = drbd_bm_count_bits(device, sbnr, tbnr);
- else if (mode == SET_IN_SYNC)
- c = drbd_bm_clear_bits(device, sbnr, tbnr);
- else /* if (mode == SET_OUT_OF_SYNC) */
- c = drbd_bm_set_bits(device, sbnr, tbnr);
+ int bmi = peer_device->bitmap_index;
+
+ if (mode == RECORD_RS_FAILED)
+ /* Only called from drbd_rs_failed_io(), bits
+ * supposedly still set. Recount, maybe some
+ * of the bits have been successfully cleared
+ * by application IO meanwhile.
+ */
+ count = drbd_bm_count_bits(device, bmi, sbnr, ebnr);
+ else if (mode == SET_IN_SYNC)
+ count = drbd_bm_clear_bits(device, bmi, sbnr, ebnr);
+ else /* if (mode == SET_OUT_OF_SYNC) */
+ count = drbd_bm_set_bits(device, bmi, sbnr, ebnr);
- if (c) {
- spin_lock_irqsave(&device->al_lock, flags);
- cleared += update_rs_extent(device, BM_BIT_TO_EXT(sbnr), c, mode);
- spin_unlock_irqrestore(&device->al_lock, flags);
- count += c;
- }
- sbnr = tbnr + 1;
- }
if (count) {
if (mode == SET_IN_SYNC) {
- unsigned long still_to_go = drbd_bm_total_weight(device);
- bool rs_is_done = (still_to_go <= device->rs_failed);
- drbd_advance_rs_marks(first_peer_device(device), still_to_go);
- if (cleared || rs_is_done)
- maybe_schedule_on_disk_bitmap_update(device, rs_is_done);
- } else if (mode == RECORD_RS_FAILED)
- device->rs_failed += count;
+ unsigned long still_to_go = drbd_bm_total_weight(peer_device);
+
+ drbd_advance_rs_marks(peer_device, still_to_go);
+
+ if (drbd_lazy_bitmap_update_due(peer_device))
+ drbd_peer_device_post_work(peer_device, RS_LAZY_BM_WRITE);
+
+ if (peer_device->connection->agreed_pro_version <= 95 &&
+ still_to_go <= peer_device->rs_failed &&
+ is_sync_source_state(peer_device, NOW))
+ drbd_peer_device_post_work(peer_device, RS_DONE);
+ } else if (mode == RECORD_RS_FAILED) {
+ peer_device->rs_failed += count;
+ } else /* if (mode == SET_OUT_OF_SYNC) */ {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+
+ if (repl_state >= L_SYNC_SOURCE && repl_state <= L_PAUSED_SYNC_T)
+ peer_device->rs_total += count;
+ }
wake_up(&device->al_wait);
}
return count;
}
-static bool plausible_request_size(int size)
+/* Record that the bits toward this slot were set for blocks the peer lacks.
+ * drbd_md_slot_emptied() clears this once the slot holds no bits.
+ */
+static void bitmap_slot_authoritative(struct drbd_device *device, int node_id)
+{
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
+
+ if (!test_and_set_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md->flags))
+ drbd_md_mark_dirty(device);
+}
+
+/* Record which node's resync set placeholder bits in this slot. They are
+ * retired by that node's P_PEERS_IN_SYNC; while it is reachable, the slot does
+ * not decide a reconcile.
+ */
+static void bitmap_slot_placeholder(struct drbd_device *device, int node_id, int src_node_id)
+{
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
+
+ if (src_node_id < 0 || src_node_id == node_id)
+ return;
+ peer_md->placeholder_src |= NODE_MASK(src_node_id);
+}
+
+static int node_id_by_bitmap_index(struct drbd_device *device, int bitmap_index)
{
- return size > 0
- && size <= DRBD_MAX_BATCH_BIO_SIZE
- && IS_ALIGNED(size, 512);
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int node_id;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++)
+ if (peer_md[node_id].bitmap_index == bitmap_index)
+ return node_id;
+ return -1;
}
-/* clear the bit corresponding to the piece of storage in question:
- * size byte of data starting from sector. Only clear bits of the affected
- * one or more _aligned_ BM_BLOCK_SIZE blocks.
+/* Change bits corresponding to the piece of storage in question:
+ * size byte of data starting from sector.
+ * Only clear bits for fully affected _aligned_ BM_BLOCK_SIZE blocks.
+ * Set bits even for partially affected blocks.
*
- * called by worker on C_SYNC_TARGET and receiver on SyncSource.
+ * called by worker on L_SYNC_TARGET and receiver on SyncSource.
*
*/
int __drbd_change_sync(struct drbd_peer_device *peer_device, sector_t sector, int size,
@@ -837,394 +835,325 @@ int __drbd_change_sync(struct drbd_peer_device *peer_device, sector_t sector, in
{
/* Is called from worker and receiver context _only_ */
struct drbd_device *device = peer_device->device;
+ struct drbd_bitmap *bm;
unsigned long sbnr, ebnr, lbnr;
unsigned long count = 0;
sector_t esector, nr_sectors;
- /* This would be an empty REQ_PREFLUSH, be silent. */
+ /* This would be an empty REQ_OP_FLUSH, be silent. */
if ((mode == SET_OUT_OF_SYNC) && size == 0)
return 0;
- if (!plausible_request_size(size)) {
- drbd_err(device, "%s: sector=%llus size=%d nonsense!\n",
- drbd_change_sync_fname[mode],
- (unsigned long long)sector, size);
+ if (peer_device->bitmap_index == -1) /* no bitmap... */
return 0;
- }
if (!get_ldev(device))
return 0; /* no disk, no metadata, no bitmap to manipulate bits in */
+ bm = device->bitmap;
+
nr_sectors = get_capacity(device->vdisk);
esector = sector + (size >> 9) - 1;
- if (!expect(device, sector < nr_sectors))
+ if (!expect(peer_device, sector < nr_sectors))
goto out;
- if (!expect(device, esector < nr_sectors))
+ if (!expect(peer_device, esector < nr_sectors))
esector = nr_sectors - 1;
- lbnr = BM_SECT_TO_BIT(nr_sectors-1);
+ lbnr = bm_sect_to_bit(bm, nr_sectors-1);
if (mode == SET_IN_SYNC) {
/* Round up start sector, round down end sector. We make sure
* we only clear full, aligned, BM_BLOCK_SIZE blocks. */
- if (unlikely(esector < BM_SECT_PER_BIT-1))
+ if (unlikely(esector < bm_sect_per_bit(bm)-1))
goto out;
if (unlikely(esector == (nr_sectors-1)))
ebnr = lbnr;
else
- ebnr = BM_SECT_TO_BIT(esector - (BM_SECT_PER_BIT-1));
- sbnr = BM_SECT_TO_BIT(sector + BM_SECT_PER_BIT-1);
+ ebnr = bm_sect_to_bit(bm, esector - (bm_sect_per_bit(bm)-1));
+ sbnr = bm_sect_to_bit(bm, sector + bm_sect_per_bit(bm)-1);
} else {
/* We set it out of sync, or record resync failure.
* Should not round anything here. */
- sbnr = BM_SECT_TO_BIT(sector);
- ebnr = BM_SECT_TO_BIT(esector);
+ sbnr = bm_sect_to_bit(bm, sector);
+ ebnr = bm_sect_to_bit(bm, esector);
}
- count = update_sync_bits(device, sbnr, ebnr, mode);
+ count = update_sync_bits(peer_device, sbnr, ebnr, mode);
+ if (count && mode == SET_OUT_OF_SYNC)
+ bitmap_slot_authoritative(device, peer_device->node_id);
out:
put_ldev(device);
return count;
}
-static
-struct bm_extent *_bme_get(struct drbd_device *device, unsigned int enr)
+
+/**
+ * __drbd_set_sync - Set a disk range in or out of sync
+ * @device: DRBD device
+ * @sector: start sector of disk range
+ * @size: size of disk range in bytes
+ * @bits: bit values to use by bitmap index
+ * @mask: bitmap indexes to modify (mask set)
+ * @authoritative: whether the caller knows that the peers it sets out of sync
+ * lack the blocks; if not, the bitmap slots record placeholders
+ * @src_node_id: for non-authoritative bits, the node whose resync data this
+ * is, or -1 if no node can retire the placeholders
+ *
+ * Return: a mask of the bitmap indexes which were modified.
+ */
+static unsigned long __drbd_set_sync(struct drbd_device *device, sector_t sector, int size,
+ unsigned long bits, unsigned long mask, bool authoritative,
+ int src_node_id)
{
- struct lc_element *e;
- struct bm_extent *bm_ext;
- int wakeup = 0;
- unsigned long rs_flags;
+ long set_start, set_end, clear_start, clear_end;
+ struct drbd_peer_device *peer_device;
+ struct drbd_bitmap *bm;
+ sector_t esector, nr_sectors;
+ unsigned long irq_flags;
+ unsigned long modified = 0;
+ int cbt_index;
- spin_lock_irq(&device->al_lock);
- if (device->resync_locked > device->resync->nr_elements/2) {
- spin_unlock_irq(&device->al_lock);
- return NULL;
- }
- e = lc_get(device->resync, enr);
- bm_ext = e ? lc_entry(e, struct bm_extent, lce) : NULL;
- if (bm_ext) {
- if (bm_ext->lce.lc_number != enr) {
- bm_ext->rs_left = drbd_bm_e_weight(device, enr);
- bm_ext->rs_failed = 0;
- lc_committed(device->resync);
- wakeup = 1;
- }
- if (bm_ext->lce.refcnt == 1)
- device->resync_locked++;
- set_bit(BME_NO_WRITES, &bm_ext->flags);
+ if (size <= 0 || !IS_ALIGNED(size, 512)) {
+ drbd_err(device, "%s sector: %llus, size: %d\n",
+ __func__, (unsigned long long)sector, size);
+ return 0;
}
- rs_flags = device->resync->flags;
- spin_unlock_irq(&device->al_lock);
- if (wakeup)
- wake_up(&device->al_wait);
- if (!bm_ext) {
- if (rs_flags & LC_STARVING)
- drbd_warn(device, "Have to wait for element"
- " (resync LRU too small?)\n");
- BUG_ON(rs_flags & LC_LOCKED);
- }
+ if (!get_ldev(device))
+ return 0; /* no disk, no metadata, no bitmap to set bits in */
- return bm_ext;
-}
+ bm = device->bitmap;
+ if (!bm) /* attached without a bitmap: nothing to track */
+ goto out;
+ mask &= (1 << bm->bm_max_peers) - 1;
-static int _is_in_al(struct drbd_device *device, unsigned int enr)
-{
- int rv;
+ cbt_index = drbd_cbt_bitmap_index(device->ldev);
+ if (cbt_index != -1)
+ mask &= ~(1UL << cbt_index); /* not clearing anything out of CBT */
- spin_lock_irq(&device->al_lock);
- rv = lc_is_used(device->act_log, enr);
- spin_unlock_irq(&device->al_lock);
+ nr_sectors = get_capacity(device->vdisk);
+ esector = sector + (size >> 9) - 1;
- return rv;
-}
+ if (!expect(device, sector < nr_sectors))
+ goto out;
+ if (!expect(device, esector < nr_sectors))
+ esector = nr_sectors - 1;
-/**
- * drbd_rs_begin_io() - Gets an extent in the resync LRU cache and sets it to BME_LOCKED
- * @device: DRBD device.
- * @sector: The sector number.
- *
- * This functions sleeps on al_wait.
- *
- * Returns: %0 on success, -EINTR if interrupted.
- */
-int drbd_rs_begin_io(struct drbd_device *device, sector_t sector)
-{
- unsigned int enr = BM_SECT_TO_EXT(sector);
- struct bm_extent *bm_ext;
- int i, sig;
- bool sa;
-
-retry:
- sig = wait_event_interruptible(device->al_wait,
- (bm_ext = _bme_get(device, enr)));
- if (sig)
- return -EINTR;
-
- if (test_bit(BME_LOCKED, &bm_ext->flags))
- return 0;
+ /* For marking sectors as out of sync, we need to round up. */
+ set_start = bm_sect_to_bit(bm, sector);
+ set_end = bm_sect_to_bit(bm, esector);
+
+ /* For marking sectors as in sync, we need to round down except when we
+ * reach the end of the device: The last bit in the bitmap does not
+ * account for sectors past the end of the device.
+ * CLEAR_END can become negative here. */
+ clear_start = bm_sect_to_bit(bm, sector + bm_sect_per_bit(bm) - 1);
+ if (esector == nr_sectors - 1)
+ clear_end = bm_sect_to_bit(bm, esector);
+ else
+ clear_end = bm_sect_to_bit(bm, esector + 1) - 1;
+
+ spin_lock_irqsave(&bm->bm_all_slots_lock, irq_flags);
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ int bitmap_index = peer_device->bitmap_index;
- /* step aside only while we are above c-min-rate; unless disabled. */
- sa = drbd_rs_c_min_rate_throttle(device);
+ if (bitmap_index == -1)
+ continue;
- for (i = 0; i < AL_EXT_PER_BM_SECT; i++) {
- sig = wait_event_interruptible(device->al_wait,
- !_is_in_al(device, enr * AL_EXT_PER_BM_SECT + i) ||
- (sa && test_bit(BME_PRIORITY, &bm_ext->flags)));
+ if (!test_and_clear_bit(bitmap_index, &mask))
+ continue;
- if (sig || (sa && test_bit(BME_PRIORITY, &bm_ext->flags))) {
- spin_lock_irq(&device->al_lock);
- if (lc_put(device->resync, &bm_ext->lce) == 0) {
- bm_ext->flags = 0; /* clears BME_NO_WRITES and eventually BME_PRIORITY */
- device->resync_locked--;
- wake_up(&device->al_wait);
+ if (test_bit(bitmap_index, &bits)) {
+ if (update_sync_bits(peer_device, set_start, set_end, SET_OUT_OF_SYNC)) {
+ __set_bit(bitmap_index, &modified);
+ if (authoritative)
+ bitmap_slot_authoritative(device, peer_device->node_id);
+ else
+ bitmap_slot_placeholder(device, peer_device->node_id,
+ src_node_id);
}
- spin_unlock_irq(&device->al_lock);
- if (sig)
- return -EINTR;
- if (schedule_timeout_interruptible(HZ/10))
- return -EINTR;
- goto retry;
+ } else if (clear_start <= clear_end) {
+ if (update_sync_bits(peer_device, clear_start, clear_end, SET_IN_SYNC))
+ __set_bit(bitmap_index, &modified);
}
}
- set_bit(BME_LOCKED, &bm_ext->flags);
- return 0;
+ rcu_read_unlock();
+ if (mask) {
+ int bitmap_index;
+
+ for_each_set_bit(bitmap_index, &mask, BITS_PER_LONG) {
+ if (test_bit(bitmap_index, &bits)) {
+ if (drbd_bm_set_bits(device, bitmap_index, set_start, set_end)) {
+ int node_id = node_id_by_bitmap_index(device, bitmap_index);
+
+ __set_bit(bitmap_index, &modified);
+ if (node_id != -1 && authoritative)
+ bitmap_slot_authoritative(device, node_id);
+ else if (node_id != -1)
+ bitmap_slot_placeholder(device, node_id,
+ src_node_id);
+ }
+ } else if (clear_start <= clear_end) {
+ if (drbd_bm_clear_bits(device, bitmap_index,
+ clear_start, clear_end))
+ __set_bit(bitmap_index, &modified);
+ }
+ }
+ }
+ spin_unlock_irqrestore(&bm->bm_all_slots_lock, irq_flags);
+out:
+ put_ldev(device);
+
+ return modified;
}
-/**
- * drbd_try_rs_begin_io() - Gets an extent in the resync LRU cache, does not sleep
- * @peer_device: DRBD device.
- * @sector: The sector number.
- *
- * Gets an extent in the resync LRU cache, sets it to BME_NO_WRITES, then
- * tries to set it to BME_LOCKED.
- *
- * Returns: %0 upon success, and -EAGAIN
- * if there is still application IO going on in this area.
+unsigned long drbd_set_sync(struct drbd_device *device, sector_t sector, int size,
+ unsigned long bits, unsigned long mask)
+{
+ return __drbd_set_sync(device, sector, size, bits, mask, true, -1);
+}
+
+/* Record a write to the backing device in the changed block tracking slot.
+ * Called before the write is submitted, with an ldev reference held.
*/
-int drbd_try_rs_begin_io(struct drbd_peer_device *peer_device, sector_t sector)
+void drbd_cbt_mark(struct drbd_device *device, sector_t sector, sector_t nr_sectors)
{
- struct drbd_device *device = peer_device->device;
- unsigned int enr = BM_SECT_TO_EXT(sector);
- const unsigned int al_enr = enr*AL_EXT_PER_BM_SECT;
- struct lc_element *e;
- struct bm_extent *bm_ext;
- int i;
- bool throttle = drbd_rs_should_slow_down(peer_device, sector, true);
+ struct drbd_bitmap *bm = device->bitmap;
+ int cbt_index = drbd_cbt_bitmap_index(device->ldev);
- /* If we need to throttle, a half-locked (only marked BME_NO_WRITES,
- * not yet BME_LOCKED) extent needs to be kicked out explicitly if we
- * need to throttle. There is at most one such half-locked extent,
- * which is remembered in resync_wenr. */
+ if (cbt_index != -1 && bm && nr_sectors)
+ drbd_bm_set_bits(device, cbt_index, bm_sect_to_bit(bm, sector),
+ bm_sect_to_bit(bm, sector + nr_sectors - 1));
+}
- if (throttle && device->resync_wenr != enr)
- return -EAGAIN;
+/* A resync rewrites the blocks set in the peer's slot, outside the activity
+ * log. Merge them into the tracking slot and write it out before the first
+ * resync write, so that a crash during the resync cannot lose them: the bits a
+ * crash leaves in the peer's slot get merged again when the resync restarts.
+ */
+void drbd_cbt_merge_resync_bits(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ int cbt_index;
- spin_lock_irq(&device->al_lock);
- if (device->resync_wenr != LC_FREE && device->resync_wenr != enr) {
- /* in case you have very heavy scattered io, it may
- * stall the syncer undefined if we give up the ref count
- * when we try again and requeue.
- *
- * if we don't give up the refcount, but the next time
- * we are scheduled this extent has been "synced" by new
- * application writes, we'd miss the lc_put on the
- * extent we keep the refcount on.
- * so we remembered which extent we had to try again, and
- * if the next requested one is something else, we do
- * the lc_put here...
- * we also have to wake_up
- */
- e = lc_find(device->resync, device->resync_wenr);
- bm_ext = e ? lc_entry(e, struct bm_extent, lce) : NULL;
- if (bm_ext) {
- D_ASSERT(device, !test_bit(BME_LOCKED, &bm_ext->flags));
- D_ASSERT(device, test_bit(BME_NO_WRITES, &bm_ext->flags));
- clear_bit(BME_NO_WRITES, &bm_ext->flags);
- device->resync_wenr = LC_FREE;
- if (lc_put(device->resync, &bm_ext->lce) == 0) {
- bm_ext->flags = 0;
- device->resync_locked--;
- }
- wake_up(&device->al_wait);
- } else {
- drbd_alert(device, "LOGIC BUG\n");
- }
- }
- /* TRY. */
- e = lc_try_get(device->resync, enr);
- bm_ext = e ? lc_entry(e, struct bm_extent, lce) : NULL;
- if (bm_ext) {
- if (test_bit(BME_LOCKED, &bm_ext->flags))
- goto proceed;
- if (!test_and_set_bit(BME_NO_WRITES, &bm_ext->flags)) {
- device->resync_locked++;
- } else {
- /* we did set the BME_NO_WRITES,
- * but then could not set BME_LOCKED,
- * so we tried again.
- * drop the extra reference. */
- bm_ext->lce.refcnt--;
- D_ASSERT(device, bm_ext->lce.refcnt > 0);
- }
- goto check_al;
- } else {
- /* do we rather want to try later? */
- if (device->resync_locked > device->resync->nr_elements-3)
- goto try_again;
- /* Do or do not. There is no try. -- Yoda */
- e = lc_get(device->resync, enr);
- bm_ext = e ? lc_entry(e, struct bm_extent, lce) : NULL;
- if (!bm_ext) {
- const unsigned long rs_flags = device->resync->flags;
- if (rs_flags & LC_STARVING)
- drbd_warn(device, "Have to wait for element"
- " (resync LRU too small?)\n");
- BUG_ON(rs_flags & LC_LOCKED);
- goto try_again;
- }
- if (bm_ext->lce.lc_number != enr) {
- bm_ext->rs_left = drbd_bm_e_weight(device, enr);
- bm_ext->rs_failed = 0;
- lc_committed(device->resync);
- wake_up(&device->al_wait);
- D_ASSERT(device, test_bit(BME_LOCKED, &bm_ext->flags) == 0);
- }
- set_bit(BME_NO_WRITES, &bm_ext->flags);
- D_ASSERT(device, bm_ext->lce.refcnt == 1);
- device->resync_locked++;
- goto check_al;
- }
-check_al:
- for (i = 0; i < AL_EXT_PER_BM_SECT; i++) {
- if (lc_is_used(device->act_log, al_enr+i))
- goto try_again;
- }
- set_bit(BME_LOCKED, &bm_ext->flags);
-proceed:
- device->resync_wenr = LC_FREE;
- spin_unlock_irq(&device->al_lock);
- return 0;
+ if (!get_ldev(device))
+ return;
-try_again:
- if (bm_ext) {
- if (throttle) {
- D_ASSERT(device, !test_bit(BME_LOCKED, &bm_ext->flags));
- D_ASSERT(device, test_bit(BME_NO_WRITES, &bm_ext->flags));
- clear_bit(BME_NO_WRITES, &bm_ext->flags);
- device->resync_wenr = LC_FREE;
- if (lc_put(device->resync, &bm_ext->lce) == 0) {
- bm_ext->flags = 0;
- device->resync_locked--;
- }
- wake_up(&device->al_wait);
- } else
- device->resync_wenr = enr;
+ cbt_index = drbd_cbt_bitmap_index(device->ldev);
+ if (cbt_index != -1 && device->bitmap) {
+ if (peer_device->bitmap_index != -1)
+ drbd_bm_or_slot(device, peer_device->bitmap_index, cbt_index);
+ else
+ _drbd_bm_set_many_bits(device, cbt_index, 0, -1UL);
+ drbd_bm_write_copy_pages(device, NULL);
}
- spin_unlock_irq(&device->al_lock);
- return -EAGAIN;
+ put_ldev(device);
}
-void drbd_rs_complete_io(struct drbd_device *device, sector_t sector)
+/* Writes resume: the bitmap on disk stops being complete. Persist that before
+ * the first of them, so that no later snapshot claims otherwise. resume-io
+ * calls this before it lifts the suspension, the only transition that does.
+ */
+void drbd_cbt_clear_flushed(struct drbd_device *device)
{
- unsigned int enr = BM_SECT_TO_EXT(sector);
- struct lc_element *e;
- struct bm_extent *bm_ext;
- unsigned long flags;
-
- spin_lock_irqsave(&device->al_lock, flags);
- e = lc_find(device->resync, enr);
- bm_ext = e ? lc_entry(e, struct bm_extent, lce) : NULL;
- if (!bm_ext) {
- spin_unlock_irqrestore(&device->al_lock, flags);
- drbd_err_ratelimit(device, "%s() called, but extent not found\n", __func__);
- return;
- }
+ clear_bit(CBT_BITMAPS_FLUSHED, &device->flags);
- if (bm_ext->lce.refcnt == 0) {
- spin_unlock_irqrestore(&device->al_lock, flags);
- drbd_err(device, "drbd_rs_complete_io(,%llu [=%u]) called, "
- "but refcnt is 0!?\n",
- (unsigned long long)sector, enr);
+ if (!get_ldev(device))
return;
+ if (device->ldev->md.cbt.flags & CBT_BOOKMARK_FLUSHED) {
+ WRITE_ONCE(device->ldev->md.cbt.flags,
+ device->ldev->md.cbt.flags & ~CBT_BOOKMARK_FLUSHED);
+ drbd_md_mark_dirty(device);
+ drbd_md_sync_if_dirty(device);
}
-
- if (lc_put(device->resync, &bm_ext->lce) == 0) {
- bm_ext->flags = 0; /* clear BME_LOCKED, BME_NO_WRITES and BME_PRIORITY */
- device->resync_locked--;
- wake_up(&device->al_wait);
- }
-
- spin_unlock_irqrestore(&device->al_lock, flags);
+ put_ldev(device);
}
-/**
- * drbd_rs_cancel_all() - Removes all extents from the resync LRU (even BME_LOCKED)
- * @device: DRBD device.
+/* A sync target sets every other peer out of sync for each block it receives,
+ * without knowing which of them lack it: these bits are not authoritative.
+ * src_node_id names the node whose resync data this is, so the slots know who
+ * can retire the placeholders; -1 where no node can (a local read error).
*/
-void drbd_rs_cancel_all(struct drbd_device *device)
+unsigned long drbd_set_all_out_of_sync(struct drbd_device *device, int src_node_id,
+ sector_t sector, int size)
{
- spin_lock_irq(&device->al_lock);
-
- if (get_ldev_if_state(device, D_FAILED)) { /* Makes sure ->resync is there. */
- lc_reset(device->resync);
- put_ldev(device);
- }
- device->resync_locked = 0;
- device->resync_wenr = LC_FREE;
- spin_unlock_irq(&device->al_lock);
- wake_up(&device->al_wait);
+ return __drbd_set_sync(device, sector, size, -1, -1, false, src_node_id);
}
-/**
- * drbd_rs_del_all() - Gracefully remove all extents from the resync LRU
- * @device: DRBD device.
+/* Do the bits standing toward this peer decide a reconcile at equal current
+ * UUIDs? They do when the setter had a reason to believe the peer lacks the
+ * blocks, and they do when they are placeholders that nothing can retire any
+ * more: their retirement is a P_PEERS_IN_SYNC from the node whose resync set
+ * them, and that node is unreachable.
*
- * Returns: %0 upon success, -EAGAIN if at least one reference count was
- * not zero.
+ * Bits that are neither decide nothing. A sync target marks its *other*
+ * peers, so a slot toward the only peer that has one never holds placeholders
+ * at all -- unless the record names a peer that was forgotten since, which is
+ * the source gone for good that this is about. Beyond that, while this node
+ * runs it knows which bits are which:
+ * drbd_md_slot_emptied() drops the record with the last bit, so a slot that
+ * filled up since then is described by its source set alone, and an empty one
+ * says an invalidate or a full sync set these bits.
+ *
+ * That record is not in the meta data, and a newly allocated slot inherits
+ * the day-0 tracking bits without one, so for those bits the slot asks the
+ * weaker question -- is there any other peer left that could still sort this
+ * out -- rather than dropping the record, which is the direction that costs
+ * the data.
*/
-int drbd_rs_del_all(struct drbd_device *device)
+bool drbd_bitmap_slot_decides(struct drbd_peer_device *peer_device)
{
- struct lc_element *e;
- struct bm_extent *bm_ext;
- int i;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_device *peer_device_i;
+ struct drbd_peer_md *peer_md;
+ bool complete, retirable = false;
+ int node_id, other_slots = 0;
+ u64 may_retire;
- spin_lock_irq(&device->al_lock);
+ if (!get_ldev(device))
+ return false;
- if (get_ldev_if_state(device, D_FAILED)) {
- /* ok, ->resync is there. */
- for (i = 0; i < device->resync->nr_elements; i++) {
- e = lc_element_by_index(device->resync, i);
- bm_ext = lc_entry(e, struct bm_extent, lce);
- if (bm_ext->lce.lc_number == LC_FREE)
- continue;
- if (bm_ext->lce.lc_number == device->resync_wenr) {
- drbd_info(device, "dropping %u in drbd_rs_del_all, apparently"
- " got 'synced' by application io\n",
- device->resync_wenr);
- D_ASSERT(device, !test_bit(BME_LOCKED, &bm_ext->flags));
- D_ASSERT(device, test_bit(BME_NO_WRITES, &bm_ext->flags));
- clear_bit(BME_NO_WRITES, &bm_ext->flags);
- device->resync_wenr = LC_FREE;
- lc_put(device->resync, &bm_ext->lce);
- }
- if (bm_ext->lce.refcnt != 0) {
- drbd_info(device, "Retrying drbd_rs_del_all() later. "
- "refcnt=%d\n", bm_ext->lce.refcnt);
- put_ldev(device);
- spin_unlock_irq(&device->al_lock);
- return -EAGAIN;
- }
- D_ASSERT(device, !test_bit(BME_LOCKED, &bm_ext->flags));
- D_ASSERT(device, !test_bit(BME_NO_WRITES, &bm_ext->flags));
- lc_del(device->resync, &bm_ext->lce);
- }
- D_ASSERT(device, device->resync->used == 0);
+ peer_md = &device->ldev->md.peers[peer_device->node_id];
+ if (test_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md->flags)) {
put_ldev(device);
+ return true;
}
- spin_unlock_irq(&device->al_lock);
- wake_up(&device->al_wait);
- return 0;
+ if (!drbd_bm_total_weight(peer_device)) {
+ put_ldev(device);
+ return false;
+ }
+
+ may_retire = peer_md->placeholder_src;
+ complete = peer_md->placeholder_src_complete;
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (node_id == peer_device->node_id)
+ continue;
+ if (test_bit(__MDF_HAVE_BITMAP, &device->ldev->md.peers[node_id].flags))
+ other_slots++;
+ }
+ put_ldev(device);
+
+ if (!other_slots && !may_retire)
+ return false;
+ if (complete && !may_retire)
+ return false;
+ /* bits of unrecorded origin: any peer may be the one to retire them */
+ if (!complete)
+ may_retire = 0;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device_i, device) {
+ if (peer_device_i == peer_device)
+ continue;
+ if (may_retire && !(may_retire & NODE_MASK(peer_device_i->node_id)))
+ continue;
+ if (peer_device_i->connection->cstate[NOW] == C_CONNECTED)
+ retirable = true;
+ }
+ rcu_read_unlock();
+
+ return !retirable;
}
diff --git a/drivers/block/drbd/drbd_bitmap.c b/drivers/block/drbd/drbd_bitmap.c
index 6892a99eb6f0..e4ca9672496c 100644
--- a/drivers/block/drbd/drbd_bitmap.c
+++ b/drivers/block/drbd/drbd_bitmap.c
@@ -8,15 +8,27 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/bitmap.h>
+#include <linux/bitops.h>
#include <linux/vmalloc.h>
#include <linux/string.h>
#include <linux/drbd.h>
#include <linux/slab.h>
-#include <linux/highmem.h>
+#include <linux/dynamic_debug.h>
+#include <linux/libnvdimm.h>
#include "drbd_int.h"
+#include "drbd_meta_data.h"
+#include "drbd_dax_pmem.h"
+#ifndef BITS_PER_PAGE
+#define BITS_PER_PAGE (1UL << (PAGE_SHIFT + 3))
+#else
+# if BITS_PER_PAGE != (1UL << (PAGE_SHIFT + 3))
+# error "ambiguous BITS_PER_PAGE"
+# endif
+#endif
+
+#define PAGES_TO_KIB(pages) (((unsigned long long) (pages)) * (PAGE_SIZE / 1024))
/* OPAQUE outside this file!
* interface defined in drbd_int.h
@@ -76,50 +88,64 @@
* so we need spin_lock_irqsave().
* And we need the kmap_atomic.
*/
-struct drbd_bitmap {
- struct page **bm_pages;
- spinlock_t bm_lock;
- /* exclusively to be used by __al_write_transaction(),
- * drbd_bm_mark_for_writeout() and
- * and drbd_bm_write_hinted() -> bm_rw() called from there.
- */
- unsigned int n_bitmap_hints;
- unsigned int al_bitmap_hints[AL_UPDATES_PER_TRANSACTION];
-
- /* see LIMITATIONS: above */
-
- unsigned long bm_set; /* nr of set bits; THINK maybe atomic_t? */
- unsigned long bm_bits;
- size_t bm_words;
- size_t bm_number_of_pages;
- sector_t bm_dev_capacity;
- struct mutex bm_change; /* serializes resize operations */
-
- wait_queue_head_t bm_io_wait; /* used to serialize IO of single pages */
-
- enum bm_flag bm_flags;
-
- /* debugging aid, in case we are still racy somewhere */
- char *bm_why;
- struct task_struct *bm_task;
+enum bitmap_operations {
+ BM_OP_CLEAR,
+ BM_OP_SET,
+ BM_OP_TEST,
+ BM_OP_COUNT,
+ BM_OP_MERGE,
+ BM_OP_EXTRACT,
+ BM_OP_FIND_BIT,
+ BM_OP_FIND_ZERO_BIT,
};
-#define bm_print_lock_info(m) __bm_print_lock_info(m, __func__)
-static void __bm_print_lock_info(struct drbd_device *device, const char *func)
-{
+static void
+bm_print_lock_info(struct drbd_device *device, unsigned int bitmap_index, enum bitmap_operations op)
+{
+ static const char * const op_names[] = {
+ [BM_OP_CLEAR] = "clear",
+ [BM_OP_SET] = "set",
+ [BM_OP_TEST] = "test",
+ [BM_OP_COUNT] = "count",
+ [BM_OP_MERGE] = "merge",
+ [BM_OP_EXTRACT] = "extract",
+ [BM_OP_FIND_BIT] = "find_bit",
+ [BM_OP_FIND_ZERO_BIT] = "find_zero_bit",
+ };
+
struct drbd_bitmap *b = device->bitmap;
+
if (!drbd_device_ratelimit(device, GENERIC))
return;
- drbd_err(device, "FIXME %s[%d] in %s, bitmap locked for '%s' by %s[%d]\n",
+ drbd_err(device, "FIXME %s[%d] op %s on slot %u, %s for '%s' by %s[%d]\n",
current->comm, task_pid_nr(current),
- func, b->bm_why ?: "?",
- b->bm_task->comm, task_pid_nr(b->bm_task));
+ op_names[op], bitmap_index,
+ b->bm_locked_slot_index >= 0 ? "slot locked" : "bitmap locked device-wide",
+ b->bm_why ?: "?",
+ b->bm_task_comm, b->bm_task_pid);
}
-void drbd_bm_lock(struct drbd_device *device, char *why, enum bm_flag flags)
+/* drbd_bm_lock() was introduced before drbd-9.0 to ensure that access to
+ bitmap is locked out by other means (states, etc..). If a needed lock was
+ not acquired or already taken a warning gets logged, and the critical
+ sections get serialized on a mutex.
+
+ Since drbd-9.0 actions on the bitmap could happen in parallel (e.g. "receive
+ bitmap").
+ The cheap solution taken right now, is to completely serialize bitmap
+ operations but do not warn if they operate on different bitmap slots.
+
+ The real solution is to make the locking more fine grained (one lock per
+ bitmap slot) and to allow those operations to happen parallel.
+ */
+void
+_drbd_bm_lock(struct drbd_device *device, struct drbd_bitmap *b,
+ struct drbd_peer_device *peer_device,
+ const char *why, enum bm_flag flags)
{
- struct drbd_bitmap *b = device->bitmap;
+ bool slot_lock = peer_device && peer_device->bitmap_index >= 0;
+ unsigned long irq_flags;
int trylock_failed;
if (!b) {
@@ -130,37 +156,99 @@ void drbd_bm_lock(struct drbd_device *device, char *why, enum bm_flag flags)
trylock_failed = !mutex_trylock(&b->bm_change);
if (trylock_failed) {
- drbd_warn(device, "%s[%d] going to '%s' but bitmap already locked for '%s' by %s[%d]\n",
- current->comm, task_pid_nr(current),
- why, b->bm_why ?: "?",
- b->bm_task->comm, task_pid_nr(b->bm_task));
+ enum bm_flag locked_flags;
+ int locked_slot_index;
+ pid_t locked_pid;
+ const char *locked_why;
+ char locked_comm[TASK_COMM_LEN];
+
+ /* Read the holder's fields under bm_lock so they cannot tear. */
+ spin_lock_irqsave(&b->bm_lock, irq_flags);
+ locked_flags = b->bm_flags;
+ locked_slot_index = b->bm_locked_slot_index;
+ locked_pid = b->bm_task_pid;
+ locked_why = b->bm_why;
+ strscpy(locked_comm, b->bm_task_comm, sizeof(locked_comm));
+ spin_unlock_irqrestore(&b->bm_lock, irq_flags);
+
+ /* The holder may have unlocked between the trylock and the
+ * read above. Suppress the warning in that case.
+ */
+ if ((locked_flags & BM_LOCK_ALL) &&
+ (!slot_lock || locked_slot_index == peer_device->bitmap_index)) {
+ if (locked_slot_index >= 0)
+ drbd_warn(device, "%s[%d] going to '%s' but bitmap already locked for '%s' on slot %d by %s[%d]\n",
+ current->comm, task_pid_nr(current),
+ why, locked_why ?: "?",
+ locked_slot_index,
+ locked_comm, locked_pid);
+ else
+ drbd_warn(device, "%s[%d] going to '%s' but bitmap already locked device-wide for '%s' by %s[%d]\n",
+ current->comm, task_pid_nr(current),
+ why, locked_why ?: "?",
+ locked_comm, locked_pid);
+ }
mutex_lock(&b->bm_change);
}
- if (BM_LOCKED_MASK & b->bm_flags)
+
+ spin_lock_irqsave(&b->bm_lock, irq_flags);
+ if (b->bm_flags & BM_LOCK_ALL)
drbd_err(device, "FIXME bitmap already locked in bm_lock\n");
- b->bm_flags |= flags & BM_LOCKED_MASK;
+ b->bm_flags |= flags & BM_LOCK_ALL;
b->bm_why = why;
- b->bm_task = current;
+ strscpy(b->bm_task_comm, current->comm);
+ b->bm_task_pid = task_pid_nr(current);
+ if (slot_lock)
+ b->bm_locked_slot_index = peer_device->bitmap_index;
+ else
+ b->bm_locked_slot_index = -1;
+ spin_unlock_irqrestore(&b->bm_lock, irq_flags);
}
-void drbd_bm_unlock(struct drbd_device *device)
+void drbd_bm_lock(struct drbd_device *device, const char *why, enum bm_flag flags)
{
- struct drbd_bitmap *b = device->bitmap;
+ _drbd_bm_lock(device, device->bitmap, NULL, why, flags);
+}
+
+void drbd_bm_slot_lock(struct drbd_peer_device *peer_device, char *why, enum bm_flag flags)
+{
+ _drbd_bm_lock(peer_device->device, peer_device->device->bitmap, peer_device, why, flags);
+}
+
+void _drbd_bm_unlock(struct drbd_device *device, struct drbd_bitmap *b)
+{
+ unsigned long irq_flags;
+
if (!b) {
drbd_err(device, "FIXME no bitmap in drbd_bm_unlock!?\n");
return;
}
- if (!(BM_LOCKED_MASK & device->bitmap->bm_flags))
+ spin_lock_irqsave(&b->bm_lock, irq_flags);
+ if (!(b->bm_flags & BM_LOCK_ALL))
drbd_err(device, "FIXME bitmap not locked in bm_unlock\n");
- b->bm_flags &= ~BM_LOCKED_MASK;
+ b->bm_flags &= ~BM_LOCK_ALL;
b->bm_why = NULL;
- b->bm_task = NULL;
+ b->bm_task_comm[0] = 0;
+ b->bm_task_pid = 0;
+ b->bm_locked_slot_index = -1;
+ spin_unlock_irqrestore(&b->bm_lock, irq_flags);
+
mutex_unlock(&b->bm_change);
}
+void drbd_bm_unlock(struct drbd_device *device)
+{
+ _drbd_bm_unlock(device, device->bitmap);
+}
+
+void drbd_bm_slot_unlock(struct drbd_peer_device *peer_device)
+{
+ drbd_bm_unlock(peer_device->device);
+}
+
/* we store some "meta" info about our pages in page->private */
/* at a granularity of 4k storage per bitmap bit:
* one peta byte storage: 1<<50 byte, 1<<38 * 4k storage blocks
@@ -208,6 +296,7 @@ static void bm_page_lock_io(struct drbd_device *device, int page_nr)
{
struct drbd_bitmap *b = device->bitmap;
void *addr = &page_private(b->bm_pages[page_nr]);
+
wait_event(b->bm_io_wait, !test_and_set_bit(BM_PAGE_IO_LOCK, addr));
}
@@ -215,8 +304,9 @@ static void bm_page_unlock_io(struct drbd_device *device, int page_nr)
{
struct drbd_bitmap *b = device->bitmap;
void *addr = &page_private(b->bm_pages[page_nr]);
+
clear_bit_unlock(BM_PAGE_IO_LOCK, addr);
- wake_up(&device->bitmap->bm_io_wait);
+ wake_up(&b->bm_io_wait);
}
/* set _before_ submit_io, so it may be reset due to being changed
@@ -228,9 +318,13 @@ static void bm_set_page_unchanged(struct page *page)
clear_bit(BM_PAGE_LAZY_WRITEOUT, &page_private(page));
}
-static void bm_set_page_need_writeout(struct page *page)
+static void bm_set_page_need_writeout(struct drbd_bitmap *bitmap, unsigned int page_nr)
{
- set_bit(BM_PAGE_NEED_WRITEOUT, &page_private(page));
+ if (!(bitmap->bm_flags & BM_ON_DAX_PMEM)) {
+ struct page *page = bitmap->bm_pages[page_nr];
+
+ set_bit(BM_PAGE_NEED_WRITEOUT, &page_private(page));
+ }
}
void drbd_bm_reset_al_hints(struct drbd_device *device)
@@ -238,33 +332,10 @@ void drbd_bm_reset_al_hints(struct drbd_device *device)
device->bitmap->n_bitmap_hints = 0;
}
-/**
- * drbd_bm_mark_for_writeout() - mark a page with a "hint" to be considered for writeout
- * @device: DRBD device.
- * @page_nr: the bitmap page to mark with the "hint" flag
- *
- * From within an activity log transaction, we mark a few pages with these
- * hints, then call drbd_bm_write_hinted(), which will only write out changed
- * pages which are flagged with this mark.
- */
-void drbd_bm_mark_for_writeout(struct drbd_device *device, int page_nr)
-{
- struct drbd_bitmap *b = device->bitmap;
- struct page *page;
- if (page_nr >= device->bitmap->bm_number_of_pages) {
- drbd_warn(device, "BAD: page_nr: %u, number_of_pages: %u\n",
- page_nr, (int)device->bitmap->bm_number_of_pages);
- return;
- }
- page = device->bitmap->bm_pages[page_nr];
- BUG_ON(b->n_bitmap_hints >= ARRAY_SIZE(b->al_bitmap_hints));
- if (!test_and_set_bit(BM_PAGE_HINT_WRITEOUT, &page_private(page)))
- b->al_bitmap_hints[b->n_bitmap_hints++] = page_nr;
-}
-
static int bm_test_page_unchanged(struct page *page)
{
volatile const unsigned long *addr = &page_private(page);
+
return (*addr & ((1UL<<BM_PAGE_NEED_WRITEOUT)|(1UL<<BM_PAGE_LAZY_WRITEOUT))) == 0;
}
@@ -278,9 +349,13 @@ static void bm_clear_page_io_err(struct page *page)
clear_bit(BM_PAGE_IO_ERROR, &page_private(page));
}
-static void bm_set_page_lazy_writeout(struct page *page)
+static void bm_set_page_lazy_writeout(struct drbd_bitmap *bitmap, unsigned int page_nr)
{
- set_bit(BM_PAGE_LAZY_WRITEOUT, &page_private(page));
+ if (!(bitmap->bm_flags & BM_ON_DAX_PMEM)) {
+ struct page *page = bitmap->bm_pages[page_nr];
+
+ set_bit(BM_PAGE_LAZY_WRITEOUT, &page_private(page));
+ }
}
static int bm_test_page_lazy_writeout(struct page *page)
@@ -288,57 +363,6 @@ static int bm_test_page_lazy_writeout(struct page *page)
return test_bit(BM_PAGE_LAZY_WRITEOUT, &page_private(page));
}
-/* on a 32bit box, this would allow for exactly (2<<38) bits. */
-static unsigned int bm_word_to_page_idx(struct drbd_bitmap *b, unsigned long long_nr)
-{
- /* page_nr = (word*sizeof(long)) >> PAGE_SHIFT; */
- unsigned int page_nr = long_nr >> (PAGE_SHIFT - LN2_BPL + 3);
- BUG_ON(page_nr >= b->bm_number_of_pages);
- return page_nr;
-}
-
-static unsigned int bm_bit_to_page_idx(struct drbd_bitmap *b, u64 bitnr)
-{
- /* page_nr = (bitnr/8) >> PAGE_SHIFT; */
- unsigned int page_nr = bitnr >> (PAGE_SHIFT + 3);
- BUG_ON(page_nr >= b->bm_number_of_pages);
- return page_nr;
-}
-
-static unsigned long *__bm_map_pidx(struct drbd_bitmap *b, unsigned int idx)
-{
- struct page *page = b->bm_pages[idx];
- return (unsigned long *) kmap_atomic(page);
-}
-
-static unsigned long *bm_map_pidx(struct drbd_bitmap *b, unsigned int idx)
-{
- return __bm_map_pidx(b, idx);
-}
-
-static void __bm_unmap(unsigned long *p_addr)
-{
- kunmap_atomic(p_addr);
-};
-
-static void bm_unmap(unsigned long *p_addr)
-{
- return __bm_unmap(p_addr);
-}
-
-/* long word offset of _bitmap_ sector */
-#define S2W(s) ((s)<<(BM_EXT_SHIFT-BM_BLOCK_SHIFT-LN2_BPL))
-/* word offset from start of bitmap to word number _in_page_
- * modulo longs per page
-#define MLPP(X) ((X) % (PAGE_SIZE/sizeof(long))
- hm, well, Philipp thinks gcc might not optimize the % into & (... - 1)
- so do it explicitly:
- */
-#define MLPP(X) ((X) & ((PAGE_SIZE/sizeof(long))-1))
-
-/* Long words per page */
-#define LWPP (PAGE_SIZE/sizeof(long))
-
/*
* actually most functions herein should take a struct drbd_bitmap*, not a
* struct drbd_device*, but for the debug macros I like to have the device around
@@ -349,6 +373,7 @@ static void bm_unmap(unsigned long *p_addr)
static void bm_free_pages(struct page **pages, unsigned long number)
{
unsigned long i;
+
if (!pages)
return;
@@ -363,15 +388,11 @@ static void bm_free_pages(struct page **pages, unsigned long number)
}
}
-static inline void bm_vk_free(void *ptr)
-{
- kvfree(ptr);
-}
-
/*
* "have" and "want" are NUMBER OF PAGES.
*/
-static struct page **bm_realloc_pages(struct drbd_bitmap *b, unsigned long want)
+static struct page **bm_realloc_pages(struct drbd_device *device, struct drbd_bitmap *b,
+ unsigned long want)
{
struct page **old_pages = b->bm_pages;
struct page **new_pages, *page;
@@ -384,15 +405,18 @@ static struct page **bm_realloc_pages(struct drbd_bitmap *b, unsigned long want)
if (have == want)
return old_pages;
- /* Trying kmalloc first, falling back to vmalloc.
+ /*
+ * Trying kmalloc first, falling back to vmalloc.
* GFP_NOIO, as this is called while drbd IO is "suspended",
* and during resize or attach on diskless Primary,
* we must not block on IO to ourselves.
- * Context is receiver thread or dmsetup. */
+ * Context is receiver thread or drbdsetup.
+ */
bytes = sizeof(struct page *)*want;
new_pages = kzalloc(bytes, GFP_NOIO | __GFP_NOWARN);
if (!new_pages) {
- new_pages = __vmalloc(bytes, GFP_NOIO | __GFP_ZERO);
+ new_pages = __vmalloc(bytes,
+ GFP_NOIO | __GFP_HIGHMEM | __GFP_ZERO);
if (!new_pages)
return NULL;
}
@@ -401,10 +425,14 @@ static struct page **bm_realloc_pages(struct drbd_bitmap *b, unsigned long want)
for (i = 0; i < have; i++)
new_pages[i] = old_pages[i];
for (; i < want; i++) {
- page = alloc_page(GFP_NOIO | __GFP_HIGHMEM);
+ page = alloc_page(GFP_NOIO | __GFP_HIGHMEM | __GFP_RETRY_MAYFAIL |
+ __GFP_NOWARN | __GFP_ZERO);
if (!page) {
bm_free_pages(new_pages + have, i - have);
- bm_vk_free(new_pages);
+ kvfree(new_pages);
+ drbd_err(device, "Failed to allocate bitmap; allocated %lu KiB / %lu KiB\n",
+ (unsigned long) i << (PAGE_SHIFT - 10),
+ want << (PAGE_SHIFT - 10));
return NULL;
}
/* we want to know which page it is
@@ -419,27 +447,33 @@ static struct page **bm_realloc_pages(struct drbd_bitmap *b, unsigned long want)
bm_free_pages(old_pages + want, have - want);
*/
}
-
return new_pages;
}
-/*
- * allocates the drbd_bitmap and stores it in device->bitmap.
- */
-int drbd_bm_init(struct drbd_device *device)
+struct drbd_bitmap *drbd_bm_alloc(unsigned int max_peers, unsigned int bm_block_shift)
{
- struct drbd_bitmap *b = device->bitmap;
- WARN_ON(b != NULL);
+ struct drbd_bitmap *b;
+
+ if (bm_block_shift < BM_BLOCK_SHIFT_MIN
+ || bm_block_shift > BM_BLOCK_SIZE_MAX)
+ return NULL;
+ if (max_peers < 1 || max_peers > DRBD_PEERS_MAX)
+ return NULL;
+
b = kzalloc_obj(struct drbd_bitmap);
if (!b)
- return -ENOMEM;
+ return NULL;
+
spin_lock_init(&b->bm_lock);
+ spin_lock_init(&b->bm_all_slots_lock);
mutex_init(&b->bm_change);
init_waitqueue_head(&b->bm_io_wait);
- device->bitmap = b;
+ b->bm_max_peers = max_peers;
+ b->bm_block_shift = bm_block_shift;
+ b->bm_locked_slot_index = -1;
- return 0;
+ return b;
}
sector_t drbd_bm_capacity(struct drbd_device *device)
@@ -449,170 +483,464 @@ sector_t drbd_bm_capacity(struct drbd_device *device)
return device->bitmap->bm_dev_capacity;
}
-/* called on driver unload. TODO: call when a device is destroyed.
- */
-void drbd_bm_cleanup(struct drbd_device *device)
+void drbd_bm_free(struct drbd_device *device)
{
- if (!expect(device, device->bitmap))
+ /* ldev_safe: atomically claiming the pointer, NULL check below */
+ struct drbd_bitmap *bitmap = xchg(&device->bitmap, NULL);
+
+ if (bitmap == NULL)
return;
- bm_free_pages(device->bitmap->bm_pages, device->bitmap->bm_number_of_pages);
- bm_vk_free(device->bitmap->bm_pages);
- kfree(device->bitmap);
- device->bitmap = NULL;
+
+ _drbd_bm_lock(device, bitmap, NULL, __func__, BM_LOCK_ALL);
+ /* ldev_safe: explicit NULL check above */
+ drbd_bm_resize(device, bitmap, 0, 0);
+ _drbd_bm_unlock(device, bitmap);
+
+ kfree(bitmap);
+}
+
+static inline unsigned long interleaved_word32(struct drbd_bitmap *bitmap,
+ unsigned int bitmap_index,
+ unsigned long bit)
+{
+ return (bit >> 5) * bitmap->bm_max_peers + bitmap_index;
+}
+
+static inline unsigned long word32_to_page(unsigned long word)
+{
+ return word >> (PAGE_SHIFT - 2);
+}
+
+static inline unsigned int word32_in_page(unsigned long word)
+{
+ return word & ((1 << (PAGE_SHIFT - 2)) - 1);
+}
+
+static inline unsigned long last_bit_on_page(struct drbd_bitmap *bitmap,
+ unsigned int bitmap_index,
+ unsigned long bit)
+{
+ unsigned long word = interleaved_word32(bitmap, bitmap_index, bit);
+
+ return (bit | 31) + ((word32_in_page(-(word + 1)) / bitmap->bm_max_peers) << 5);
+}
+
+static inline unsigned long bit_to_page_interleaved(struct drbd_bitmap *bitmap,
+ unsigned int bitmap_index,
+ unsigned long bit)
+{
+ return word32_to_page(interleaved_word32(bitmap, bitmap_index, bit));
+}
+
+static void *bm_map(struct drbd_bitmap *bitmap, unsigned int page)
+{
+ if (!(bitmap->bm_flags & BM_ON_DAX_PMEM))
+ return kmap_atomic(bitmap->bm_pages[page]);
+
+ return ((unsigned char *)bitmap->bm_on_pmem) + (unsigned long)page * PAGE_SIZE;
+}
+
+static void bm_unmap(struct drbd_bitmap *bitmap, void *addr)
+{
+ if (!(bitmap->bm_flags & BM_ON_DAX_PMEM))
+ kunmap_atomic(addr);
}
+
/*
- * since (b->bm_bits % BITS_PER_LONG) != 0,
- * this masks out the remaining bits.
- * Returns the number of bits cleared.
+ * find_next_bit() and find_next_zero_bit() expect an (unsigned long *),
+ * and will dereference it.
+ * When scanning our bitmap, we are interested in 32bit words of it.
+ * The "current 32 bit word pointer" may point to the last 32 bits in a page.
+ * For 64bit long, if the page after the current page is not mapped,
+ * this causes "page fault - not-present page".
+ * Duplicate the "fast path" of these functions,
+ * simplified for "size: 32, offset: 0".
+ * Little endian arch: le32_to_cpu is a no-op.
+ * Big endian arch: le32_to_cpu moves the least significant 32 bits around.
+ * __ffs / ffz do an implicit cast to (unsignd long). On 64bit, that fills up
+ * the most significant bits with 0; we are not interested in those anyways.
*/
-#ifndef BITS_PER_PAGE
-#define BITS_PER_PAGE (1UL << (PAGE_SHIFT + 3))
-#define BITS_PER_PAGE_MASK (BITS_PER_PAGE - 1)
-#else
-# if BITS_PER_PAGE != (1UL << (PAGE_SHIFT + 3))
-# error "ambiguous BITS_PER_PAGE"
-# endif
-#endif
-#define BITS_PER_LONG_MASK (BITS_PER_LONG - 1)
-static int bm_clear_surplus(struct drbd_bitmap *b)
-{
- unsigned long mask;
- unsigned long *p_addr, *bm;
- int tmp;
- int cleared = 0;
-
- /* number of bits modulo bits per page */
- tmp = (b->bm_bits & BITS_PER_PAGE_MASK);
- /* mask the used bits of the word containing the last bit */
- mask = (1UL << (tmp & BITS_PER_LONG_MASK)) -1;
- /* bitmap is always stored little endian,
- * on disk and in core memory alike */
- mask = cpu_to_lel(mask);
-
- p_addr = bm_map_pidx(b, b->bm_number_of_pages - 1);
- bm = p_addr + (tmp/BITS_PER_LONG);
- if (mask) {
- /* If mask != 0, we are not exactly aligned, so bm now points
- * to the long containing the last bit.
- * If mask == 0, bm already points to the word immediately
- * after the last (long word aligned) bit. */
- cleared = hweight_long(*bm & ~mask);
- *bm &= mask;
- bm++;
- }
+static inline unsigned long find_next_bit_le32(const __le32 *addr)
+{
+ uint32_t val = *addr;
+
+ return val ? __ffs(le32_to_cpu(val)) : 32;
+}
+
+static inline unsigned long find_next_zero_bit_le32(const __le32 *addr)
+{
+ uint32_t val = *addr;
+
+ return val == ~0U ? 32 : ffz(le32_to_cpu(val));
+}
+
+
+static __always_inline unsigned long
+____bm_op(struct drbd_device *device, struct drbd_bitmap *bitmap, unsigned int bitmap_index,
+ unsigned long start, unsigned long end,
+ enum bitmap_operations op, __le32 *buffer)
+{
+ unsigned int word32_skip = 32 * bitmap->bm_max_peers;
+ unsigned long total = 0;
+ unsigned long word;
+ unsigned int page, bit_in_page;
+
+ if (end >= bitmap->bm_bits)
+ end = bitmap->bm_bits - 1;
+
+ word = interleaved_word32(bitmap, bitmap_index, start);
+ page = word32_to_page(word);
+ bit_in_page = (word32_in_page(word) << 5) | (start & 31);
+
+ for (; start <= end; page++) {
+ unsigned int count = 0;
+ void *addr;
+
+ addr = bm_map(bitmap, page);
+ if (((start & 31) && (start | 31) <= end) || op == BM_OP_TEST) {
+ unsigned int last = bit_in_page | 31;
+
+ switch (op) {
+ default:
+ do {
+ switch (op) {
+ case BM_OP_CLEAR:
+ if (__test_and_clear_bit_le(bit_in_page, addr))
+ count++;
+ break;
+ case BM_OP_SET:
+ if (!__test_and_set_bit_le(bit_in_page, addr))
+ count++;
+ break;
+ case BM_OP_COUNT:
+ if (test_bit_le(bit_in_page, addr))
+ total++;
+ break;
+ case BM_OP_TEST:
+ total = !!test_bit_le(bit_in_page, addr);
+ bm_unmap(bitmap, addr);
+ return total;
+ default:
+ break;
+ }
+ bit_in_page++;
+ } while (bit_in_page <= last);
+ break;
+ case BM_OP_MERGE:
+ case BM_OP_EXTRACT:
+ BUG();
+ break;
+ case BM_OP_FIND_BIT:
+ count = find_next_bit_le(addr, last + 1, bit_in_page);
+ if (count < last + 1)
+ goto found;
+ bit_in_page = last + 1;
+ break;
+ case BM_OP_FIND_ZERO_BIT:
+ count = find_next_zero_bit_le(addr, last + 1, bit_in_page);
+ if (count < last + 1)
+ goto found;
+ bit_in_page = last + 1;
+ break;
+ }
+ start = (start | 31) + 1;
+ bit_in_page += word32_skip - 32;
+ if (bit_in_page >= BITS_PER_PAGE)
+ goto next_page;
+ }
+
+ while (start + 31 <= end) {
+ __le32 *p = (__le32 *)addr + (bit_in_page >> 5);
+
+ switch (op) {
+ case BM_OP_CLEAR:
+ count += hweight32(*p);
+ *p = 0;
+ break;
+ case BM_OP_SET:
+ count += hweight32(~*p);
+ *p = -1;
+ break;
+ case BM_OP_TEST:
+ BUG();
+ break;
+ case BM_OP_COUNT:
+ total += hweight32(*p);
+ break;
+ case BM_OP_MERGE:
+ count += hweight32(~*p & *buffer);
+ *p |= *buffer++;
+ break;
+ case BM_OP_EXTRACT:
+ *buffer++ = *p;
+ break;
+ case BM_OP_FIND_BIT:
+ count = find_next_bit_le32(p);
+ if (count < 32) {
+ count += bit_in_page;
+ goto found;
+ }
+ break;
+ case BM_OP_FIND_ZERO_BIT:
+ count = find_next_zero_bit_le32(p);
+ if (count < 32) {
+ count += bit_in_page;
+ goto found;
+ }
+ break;
+ }
+ start += 32;
+ bit_in_page += word32_skip;
+ if (bit_in_page >= BITS_PER_PAGE)
+ goto next_page;
+ }
+
+ /* don't overrun buffers with MERGE or EXTRACT,
+ * jump to the kunmap and then out... */
+ if (start > end)
+ goto next_page;
+
+ switch (op) {
+ default:
+ while (start <= end) {
+ switch (op) {
+ case BM_OP_CLEAR:
+ if (__test_and_clear_bit_le(bit_in_page, addr))
+ count++;
+ break;
+ case BM_OP_SET:
+ if (!__test_and_set_bit_le(bit_in_page, addr))
+ count++;
+ break;
+ case BM_OP_COUNT:
+ if (test_bit_le(bit_in_page, addr))
+ total++;
+ break;
+ default:
+ break;
+ }
+ start++;
+ bit_in_page++;
+ }
+ break;
+ case BM_OP_MERGE:
+ {
+ __le32 *p = (__le32 *)addr + (bit_in_page >> 5);
+ __le32 b = *buffer++ & cpu_to_le32((1 << (end - start + 1)) - 1);
+
+ count += hweight32(~*p & b);
+ *p |= b;
+
+ start = end + 1;
+ }
+ break;
+ case BM_OP_EXTRACT:
+ {
+ __le32 *p = (__le32 *)addr + (bit_in_page >> 5);
+
+ *buffer++ = *p & cpu_to_le32((1 << (end - start + 1)) - 1);
+ start = end + 1;
+ }
+ break;
+ case BM_OP_FIND_BIT:
+ {
+ unsigned int last = bit_in_page + (end - start);
+
+ count = find_next_bit_le(addr, last + 1, bit_in_page);
+ if (count < last + 1)
+ goto found;
+ start = end + 1;
+ }
+ break;
+ case BM_OP_FIND_ZERO_BIT:
+ {
+ unsigned int last = bit_in_page + (end - start);
+
+ count = find_next_zero_bit_le(addr, last + 1, bit_in_page);
+ if (count < last + 1)
+ goto found;
+ start = end + 1;
+ }
+ break;
+ }
- if (BITS_PER_LONG == 32 && ((bm - p_addr) & 1) == 1) {
- /* on a 32bit arch, we may need to zero out
- * a padding long to align with a 64bit remote */
- cleared += hweight_long(*bm);
- *bm = 0;
+ next_page:
+ bm_unmap(bitmap, addr);
+ bit_in_page -= BITS_PER_PAGE;
+ switch (op) {
+ case BM_OP_CLEAR:
+ if (count) {
+ bm_set_page_lazy_writeout(bitmap, page);
+ total += count;
+ }
+ break;
+ case BM_OP_SET:
+ case BM_OP_MERGE:
+ if (count) {
+ bm_set_page_need_writeout(bitmap, page);
+ total += count;
+ }
+ break;
+ default:
+ break;
+ }
+ continue;
+
+ found:
+ bm_unmap(bitmap, addr);
+ return start + count - bit_in_page;
}
- bm_unmap(p_addr);
- return cleared;
-}
-
-static void bm_set_surplus(struct drbd_bitmap *b)
-{
- unsigned long mask;
- unsigned long *p_addr, *bm;
- int tmp;
-
- /* number of bits modulo bits per page */
- tmp = (b->bm_bits & BITS_PER_PAGE_MASK);
- /* mask the used bits of the word containing the last bit */
- mask = (1UL << (tmp & BITS_PER_LONG_MASK)) -1;
- /* bitmap is always stored little endian,
- * on disk and in core memory alike */
- mask = cpu_to_lel(mask);
-
- p_addr = bm_map_pidx(b, b->bm_number_of_pages - 1);
- bm = p_addr + (tmp/BITS_PER_LONG);
- if (mask) {
- /* If mask != 0, we are not exactly aligned, so bm now points
- * to the long containing the last bit.
- * If mask == 0, bm already points to the word immediately
- * after the last (long word aligned) bit. */
- *bm |= ~mask;
- bm++;
+ switch (op) {
+ case BM_OP_CLEAR:
+ if (total) {
+ bitmap->bm_set[bitmap_index] -= total;
+ if (bitmap->bm_set[bitmap_index] == 0)
+ drbd_md_slot_emptied(device, bitmap_index);
+ }
+ break;
+ case BM_OP_SET:
+ case BM_OP_MERGE:
+ if (total)
+ bitmap->bm_set[bitmap_index] += total;
+ break;
+ case BM_OP_FIND_BIT:
+ case BM_OP_FIND_ZERO_BIT:
+ total = DRBD_END_OF_BITMAP;
+ break;
+ default:
+ break;
}
+ return total;
+}
- if (BITS_PER_LONG == 32 && ((bm - p_addr) & 1) == 1) {
- /* on a 32bit arch, we may need to zero out
- * a padding long to align with a 64bit remote */
- *bm = ~0UL;
+/* Returns the number of bits changed. */
+static __always_inline unsigned long
+__bm_op(struct drbd_device *device, struct drbd_bitmap *bitmap, unsigned int bitmap_index,
+ unsigned long start, unsigned long end,
+ enum bitmap_operations op, __le32 *buffer)
+{
+ if (!expect(device, bitmap))
+ return 1;
+ if (!expect(device, bitmap->bm_pages))
+ return 0;
+
+ if (!bitmap->bm_bits)
+ return 0;
+
+ /* Warn only when the op runs against the lock holder's scope: a
+ * different task, and either a device-wide lock or one on this slot.
+ */
+ if (bitmap->bm_task_pid != task_pid_nr(current) &&
+ (bitmap->bm_locked_slot_index < 0 ||
+ bitmap->bm_locked_slot_index == bitmap_index)) {
+ switch (op) {
+ case BM_OP_CLEAR:
+ if (bitmap->bm_flags & BM_LOCK_CLEAR)
+ bm_print_lock_info(device, bitmap_index, op);
+ break;
+ case BM_OP_SET:
+ case BM_OP_MERGE:
+ if (bitmap->bm_flags & BM_LOCK_SET)
+ bm_print_lock_info(device, bitmap_index, op);
+ break;
+ case BM_OP_TEST:
+ case BM_OP_COUNT:
+ case BM_OP_EXTRACT:
+ case BM_OP_FIND_BIT:
+ case BM_OP_FIND_ZERO_BIT:
+ if (bitmap->bm_flags & BM_LOCK_TEST)
+ bm_print_lock_info(device, bitmap_index, op);
+ break;
+ }
}
- bm_unmap(p_addr);
+ return ____bm_op(device, bitmap, bitmap_index, start, end, op, buffer);
+}
+
+static __always_inline unsigned long
+bm_op(struct drbd_device *device, struct drbd_bitmap *bitmap, unsigned int bitmap_index,
+ unsigned long start, unsigned long end,
+ enum bitmap_operations op, __le32 *buffer)
+{
+ unsigned long irq_flags;
+ unsigned long count;
+
+ spin_lock_irqsave(&bitmap->bm_lock, irq_flags);
+ count = __bm_op(device, bitmap, bitmap_index, start, end, op, buffer);
+ spin_unlock_irqrestore(&bitmap->bm_lock, irq_flags);
+ return count;
}
+#ifdef BITMAP_DEBUG
+#define bm_op(device, bitmap, bitmap_index, start, end, op, buffer) \
+ ({ unsigned long ret; \
+ drbd_info(device, "%s: bm_op(..., %u, %lu, %lu, %u, %p)\n", \
+ __func__, bitmap_index, start, end, op, buffer); \
+ ret = bm_op(device, bitmap, bitmap_index, start, end, op, buffer); \
+ drbd_info(device, "= %lu\n", ret); \
+ ret; })
+
+#define __bm_op(device, bitmap, bitmap_index, start, end, op, buffer) \
+ ({ unsigned long ret; \
+ drbd_info(device, "%s: __bm_op(..., %u, %lu, %lu, %u, %p)\n", \
+ __func__, bitmap_index, start, end, op, buffer); \
+ ret = __bm_op(device, bitmap, bitmap_index, start, end, op, buffer); \
+ drbd_info(device, "= %lu\n", ret); \
+ ret; })
+#endif
+
+#ifdef BITMAP_DEBUG
+#define ___bm_op(device, bitmap, bitmap_index, start, end, op, buffer) \
+ ({ unsigned long ret; \
+ pr_info("%s: ___bm_op(..., %u, %lu, %lu, %u, %p)\n", \
+ __func__, bitmap_index, start, end, op, buffer); \
+ ret = ____bm_op(device, bitmap, bitmap_index, start, end, op, buffer); \
+ pr_info("= %lu\n", ret); \
+ ret; })
+#else
+#define ___bm_op(device, bitmap, bitmap_index, start, end, op, buffer) \
+ ____bm_op(device, bitmap, bitmap_index, start, end, op, buffer)
+#endif
+
/* you better not modify the bitmap while this is running,
* or its results will be stale */
-static unsigned long bm_count_bits(struct drbd_bitmap *b)
-{
- unsigned long *p_addr;
- unsigned long bits = 0;
- unsigned long mask = (1UL << (b->bm_bits & BITS_PER_LONG_MASK)) -1;
- int idx, last_word;
-
- /* all but last page */
- for (idx = 0; idx < b->bm_number_of_pages - 1; idx++) {
- p_addr = __bm_map_pidx(b, idx);
- bits += bitmap_weight(p_addr, BITS_PER_PAGE);
- __bm_unmap(p_addr);
- cond_resched();
- }
- /* last (or only) page */
- last_word = ((b->bm_bits - 1) & BITS_PER_PAGE_MASK) >> LN2_BPL;
- p_addr = __bm_map_pidx(b, idx);
- bits += bitmap_weight(p_addr, last_word * BITS_PER_LONG);
- p_addr[last_word] &= cpu_to_lel(mask);
- bits += hweight_long(p_addr[last_word]);
- /* 32bit arch, may have an unused padding long */
- if (BITS_PER_LONG == 32 && (last_word & 1) == 0)
- p_addr[last_word+1] = 0;
- __bm_unmap(p_addr);
- return bits;
-}
-
-/* offset and len in long words.*/
-static void bm_memset(struct drbd_bitmap *b, size_t offset, int c, size_t len)
-{
- unsigned long *p_addr, *bm;
- unsigned int idx;
- size_t do_now, end;
-
- end = offset + len;
-
- if (end > b->bm_words) {
- pr_alert("bm_memset end > bm_words\n");
- return;
- }
+static void bm_count_bits(struct drbd_device *device, struct drbd_bitmap *bitmap)
+{
+ unsigned int bitmap_index;
+
+ for (bitmap_index = 0; bitmap_index < bitmap->bm_max_peers; bitmap_index++) {
+ unsigned long bit = 0, bits_set = 0;
- while (offset < end) {
- do_now = min_t(size_t, ALIGN(offset + 1, LWPP), end) - offset;
- idx = bm_word_to_page_idx(b, offset);
- p_addr = bm_map_pidx(b, idx);
- bm = p_addr + MLPP(offset);
- if (bm+do_now > p_addr + LWPP) {
- pr_alert("BUG BUG BUG! p_addr:%p bm:%p do_now:%d\n",
- p_addr, bm, (int)do_now);
- } else
- memset(bm, c, do_now * sizeof(long));
- bm_unmap(p_addr);
- bm_set_page_need_writeout(b->bm_pages[idx]);
- offset += do_now;
+ while (bit < bitmap->bm_bits) {
+ unsigned long last_bit = last_bit_on_page(bitmap, bitmap_index, bit);
+
+ bits_set += ___bm_op(device, bitmap, bitmap_index, bit, last_bit,
+ BM_OP_COUNT, NULL);
+ bit = last_bit + 1;
+ cond_resched();
+ }
+ bitmap->bm_set[bitmap_index] = bits_set;
+ if (bits_set == 0)
+ drbd_md_slot_emptied(device, bitmap_index);
}
}
/* For the layout, see comment above drbd_md_set_sector_offsets(). */
-static u64 drbd_md_on_disk_bits(struct drbd_backing_dev *ldev)
+static u64 drbd_md_on_disk_bits(struct drbd_device *device)
{
- u64 bitmap_sectors;
+ struct drbd_backing_dev *ldev = device->ldev;
+ u64 bitmap_sectors, word64_on_disk;
+
if (ldev->md.al_offset == 8)
bitmap_sectors = ldev->md.md_size_sect - ldev->md.bm_offset;
else
bitmap_sectors = ldev->md.al_offset - ldev->md.bm_offset;
- return bitmap_sectors << (9 + 3);
+
+ /* for interoperability between 32bit and 64bit architectures,
+ * we round on 64bit words. FIXME do we still need this? */
+ word64_on_disk = bitmap_sectors << (9 - 3); /* x * (512/8) */
+ do_div(word64_on_disk, ldev->md.max_peers);
+ return word64_on_disk << 6; /* x * 64 */;
}
/*
@@ -623,119 +951,220 @@ static u64 drbd_md_on_disk_bits(struct drbd_backing_dev *ldev)
* In case this is actually a resize, we copy the old bitmap into the new one.
* Otherwise, the bitmap is initialized to all bits set.
*/
-int drbd_bm_resize(struct drbd_device *device, sector_t capacity, int set_new_bits)
+int drbd_bm_resize(struct drbd_device *device, struct drbd_bitmap *b,
+ sector_t capacity, bool set_new_bits)
{
- struct drbd_bitmap *b = device->bitmap;
- unsigned long bits, words, owords, obits;
+ unsigned long bits, words, obits;
unsigned long want, have, onpages; /* number of pages */
- struct page **npages, **opages = NULL;
+ struct page **npages = NULL, **opages = NULL;
+ void *bm_on_pmem = NULL;
int err = 0;
bool growing;
- if (!expect(device, b))
- return -ENOMEM;
-
- drbd_bm_lock(device, "resize", BM_LOCKED_MASK);
-
- drbd_info(device, "drbd_bm_resize called with capacity == %llu\n",
- (unsigned long long)capacity);
+ lockdep_assert_held(&b->bm_change);
if (capacity == b->bm_dev_capacity)
goto out;
if (capacity == 0) {
+ unsigned int bitmap_index;
+
spin_lock_irq(&b->bm_lock);
opages = b->bm_pages;
onpages = b->bm_number_of_pages;
- owords = b->bm_words;
+ drbd_info(device, "Freeing bitmap of size %llu KiB\n", PAGES_TO_KIB(onpages));
b->bm_pages = NULL;
- b->bm_number_of_pages =
- b->bm_set =
- b->bm_bits =
- b->bm_words =
+ b->bm_number_of_pages = 0;
+ for (bitmap_index = 0; bitmap_index < b->bm_max_peers; bitmap_index++)
+ b->bm_set[bitmap_index] = 0;
+ b->bm_bits = 0;
+ b->bm_bits_4k = 0;
+ b->bm_words = 0;
b->bm_dev_capacity = 0;
spin_unlock_irq(&b->bm_lock);
- bm_free_pages(opages, onpages);
- bm_vk_free(opages);
+ if (!(b->bm_flags & BM_ON_DAX_PMEM)) {
+ bm_free_pages(opages, onpages);
+ kvfree(opages);
+ }
goto out;
}
- bits = BM_SECT_TO_BIT(ALIGN(capacity, BM_SECT_PER_BIT));
-
- /* if we would use
- words = ALIGN(bits,BITS_PER_LONG) >> LN2_BPL;
- a 32bit host could present the wrong number of words
- to a 64bit host.
- */
- words = ALIGN(bits, 64) >> LN2_BPL;
+ bits = bm_sect_to_bit(b, ALIGN(capacity, bm_sect_per_bit(b)));
+ words = (ALIGN(bits, 64) * b->bm_max_peers) / BITS_PER_LONG;
+ want = PFN_UP(words * sizeof(long));
+ have = b->bm_number_of_pages;
if (get_ldev(device)) {
- u64 bits_on_disk = drbd_md_on_disk_bits(device->ldev);
- put_ldev(device);
+ u64 bits_on_disk = drbd_md_on_disk_bits(device);
+
if (bits > bits_on_disk) {
- drbd_info(device, "bits = %lu\n", bits);
- drbd_info(device, "bits_on_disk = %llu\n", bits_on_disk);
+ put_ldev(device);
+ drbd_err(device, "Not enough space for bitmap: %lu > %lu\n",
+ (unsigned long)bits, (unsigned long)bits_on_disk);
err = -ENOSPC;
goto out;
}
+ if (drbd_md_dax_active(device->ldev)) {
+ drbd_info(device, "DAX/PMEM bitmap has size %llu KiB\n",
+ PAGES_TO_KIB(want));
+ bm_on_pmem = drbd_dax_bitmap(device, want);
+ }
+ put_ldev(device);
}
- want = PFN_UP(words*sizeof(long));
- have = b->bm_number_of_pages;
- if (want == have) {
- D_ASSERT(device, b->bm_pages != NULL);
- npages = b->bm_pages;
- } else {
- if (drbd_insert_fault(device, DRBD_FAULT_BM_ALLOC))
- npages = NULL;
- else
- npages = bm_realloc_pages(b, want);
- }
+ if (!bm_on_pmem) {
+ if (want == have) {
+ D_ASSERT(device, b->bm_pages != NULL);
+ drbd_info(device, "Bitmap size remains %llu KiB\n", PAGES_TO_KIB(have));
+ npages = b->bm_pages;
+ } else {
+ if (have == 0) {
+ drbd_info(device, "Allocating %llu KiB for new bitmap\n",
+ PAGES_TO_KIB(want));
+ } else if (want > have) {
+ drbd_info(device, "Allocating %llu KiB for bitmap, new size %llu KiB\n",
+ PAGES_TO_KIB(want - have), PAGES_TO_KIB(want));
+ }
- if (!npages) {
- err = -ENOMEM;
- goto out;
+ if (drbd_insert_fault(device, DRBD_FAULT_BM_ALLOC))
+ npages = NULL;
+ else
+ npages = bm_realloc_pages(device, b, want);
+ }
+
+ if (!npages) {
+ err = -ENOMEM;
+ goto out;
+ }
}
spin_lock_irq(&b->bm_lock);
- opages = b->bm_pages;
- owords = b->bm_words;
obits = b->bm_bits;
growing = bits > obits;
- if (opages && growing && set_new_bits)
- bm_set_surplus(b);
- b->bm_pages = npages;
+ if (bm_on_pmem) {
+ if (b->bm_on_pmem) {
+ void *src = b->bm_on_pmem;
+
+ memmove(bm_on_pmem, src, b->bm_words * sizeof(long));
+ arch_wb_cache_pmem(bm_on_pmem, b->bm_words * sizeof(long));
+ } else {
+ /* We are attaching a bitmap on PMEM. Since the memory
+ * is persistent, the bitmap is still valid. Do not
+ * overwrite it. */
+ growing = false;
+ }
+ b->bm_on_pmem = bm_on_pmem;
+ b->bm_flags |= BM_ON_DAX_PMEM;
+ } else {
+ opages = b->bm_pages;
+ b->bm_pages = npages;
+ }
b->bm_number_of_pages = want;
- b->bm_bits = bits;
+ b->bm_bits = bits;
+ b->bm_bits_4k = sect_to_bit(ALIGN(capacity, sect_per_bit(BM_BLOCK_SHIFT_4k)),
+ BM_BLOCK_SHIFT_4k);
b->bm_words = words;
b->bm_dev_capacity = capacity;
+ spin_unlock_irq(&b->bm_lock);
+ /* without holding the spinlock! Setting many bits is time-consuming */
if (growing) {
- if (set_new_bits) {
- bm_memset(b, owords, 0xff, words-owords);
- b->bm_set += bits - obits;
- } else
- bm_memset(b, owords, 0x00, words-owords);
+ unsigned int bitmap_index;
+ unsigned long first_bit_last_page = 0;
+
+ if (set_new_bits && want > have)
+ first_bit_last_page = ((want - 1) *
+ (PAGE_SIZE / sizeof(u32)) / b->bm_max_peers) << 5;
+
+ for (bitmap_index = 0; bitmap_index < b->bm_max_peers; bitmap_index++) {
+ unsigned long bm_set = b->bm_set[bitmap_index];
+
+ if (set_new_bits) {
+ /* Set bits in the tail of the last existing
+ * page [obits, page_end). New pages are
+ * __GFP_ZERO and get bulk-filled below.
+ * When obits == 0 all pages are new, so we
+ * skip this.
+ */
+ if (obits) {
+ unsigned long page_end =
+ last_bit_on_page(b, bitmap_index, obits);
+
+ if (page_end >= bits)
+ page_end = bits - 1;
+ bm_op(device, b, bitmap_index, obits, page_end,
+ BM_OP_SET, NULL);
+ }
+ /* Set bits on the last new page per-peer;
+ * bm_op() handles the partial last word
+ * correctly. Interior pages are bulk-filled
+ * below.
+ */
+ if (want > have)
+ bm_op(device, b, bitmap_index,
+ first_bit_last_page, bits - 1,
+ BM_OP_SET, NULL);
+ bm_set += bits - obits;
+ } else if (obits) {
+ /* Clear stale bits in the tail of the last existing
+ * page [obits, page_end). obits is rarely
+ * page-aligned, so this partial page may carry
+ * non-zero bits from a previous shrink or on-disk
+ * padding that must be cleared to avoid spurious
+ * resyncs.
+ *
+ * Newly allocated pages are __GFP_ZERO and need no
+ * clearing. When obits == 0 all pages are new, so
+ * we skip this branch entirely.
+ */
+ unsigned long page_end =
+ last_bit_on_page(b, bitmap_index, obits);
+
+ if (page_end >= bits)
+ page_end = bits - 1;
+ bm_op(device, b, bitmap_index, obits, page_end, BM_OP_CLEAR, NULL);
+ }
+ b->bm_set[bitmap_index] = bm_set;
+ }
+
+ if (set_new_bits && want > have + 1) {
+ /* New pages are __GFP_ZERO. Set all bits for all
+ * peers at once by filling interior pages with 0xFF
+ * rather than iterating per-peer word-by-word.
+ * The last new page is handled per-peer above.
+ */
+ unsigned long p;
+
+ for (p = have; p < want - 1; p++) {
+ void *addr;
+
+ spin_lock_irq(&b->bm_lock);
+ addr = bm_map(b, p);
+ memset(addr, 0xFF, PAGE_SIZE);
+ bm_unmap(b, addr);
+ bm_set_page_need_writeout(b, p);
+ spin_unlock_irq(&b->bm_lock);
+ cond_resched();
+ }
+ }
}
- if (want < have) {
+ if (want < have && !(b->bm_flags & BM_ON_DAX_PMEM)) {
/* implicit: (opages != NULL) && (opages != npages) */
+ drbd_info(device, "Freeing %llu KiB from bitmap, new size %llu KiB\n",
+ PAGES_TO_KIB(have - want), PAGES_TO_KIB(want));
bm_free_pages(opages + want, have - want);
}
-
- (void)bm_clear_surplus(b);
-
- spin_unlock_irq(&b->bm_lock);
if (opages != npages)
- bm_vk_free(opages);
+ kvfree(opages);
if (!growing)
- b->bm_set = bm_count_bits(b);
- drbd_info(device, "resync bitmap: bits=%lu words=%lu pages=%lu\n", bits, words, want);
+ bm_count_bits(device, b);
+ drbd_info(device, "resync bitmap: bits=%lu bits_4k=%lu words=%lu pages=%lu\n",
+ bits, b->bm_bits_4k, words, want);
out:
- drbd_bm_unlock(device);
return err;
}
@@ -744,10 +1173,8 @@ int drbd_bm_resize(struct drbd_device *device, sector_t capacity, int set_new_bi
* leaving this function...
* we still need to lock it, since it is important that this returns
* bm_set == 0 precisely.
- *
- * maybe bm_set should be atomic_t ?
*/
-unsigned long _drbd_bm_total_weight(struct drbd_device *device)
+unsigned long _drbd_bm_total_weight(struct drbd_device *device, int bitmap_index)
{
struct drbd_bitmap *b = device->bitmap;
unsigned long s;
@@ -759,172 +1186,102 @@ unsigned long _drbd_bm_total_weight(struct drbd_device *device)
return 0;
spin_lock_irqsave(&b->bm_lock, flags);
- s = b->bm_set;
+ s = b->bm_set[bitmap_index];
spin_unlock_irqrestore(&b->bm_lock, flags);
return s;
}
-unsigned long drbd_bm_total_weight(struct drbd_device *device)
+unsigned long drbd_bm_total_weight(struct drbd_peer_device *peer_device)
{
+ struct drbd_device *device = peer_device->device;
unsigned long s;
+
+ if (peer_device->bitmap_index == -1)
+ return 0;
+
/* if I don't have a disk, I don't know about out-of-sync status */
if (!get_ldev_if_state(device, D_NEGOTIATING))
return 0;
- s = _drbd_bm_total_weight(device);
+ s = _drbd_bm_total_weight(device, peer_device->bitmap_index);
put_ldev(device);
return s;
}
+/* Returns the number of unsigned long words per peer */
size_t drbd_bm_words(struct drbd_device *device)
{
struct drbd_bitmap *b = device->bitmap;
+
if (!expect(device, b))
return 0;
if (!expect(device, b->bm_pages))
return 0;
- return b->bm_words;
+ return b->bm_words / b->bm_max_peers;
}
unsigned long drbd_bm_bits(struct drbd_device *device)
{
struct drbd_bitmap *b = device->bitmap;
+
if (!expect(device, b))
return 0;
return b->bm_bits;
}
+unsigned long drbd_bm_bits_4k(struct drbd_device *device)
+{
+ struct drbd_bitmap *b = device->bitmap;
+
+ if (!expect(device, b))
+ return 0;
+
+ return b->bm_bits_4k;
+}
+
/* merge number words from buffer into the bitmap starting at offset.
* buffer[i] is expected to be little endian unsigned long.
* bitmap must be locked by drbd_bm_lock.
* currently only used from receive_bitmap.
*/
-void drbd_bm_merge_lel(struct drbd_device *device, size_t offset, size_t number,
+void drbd_bm_merge_lel(struct drbd_peer_device *peer_device, size_t offset, size_t number,
unsigned long *buffer)
{
- struct drbd_bitmap *b = device->bitmap;
- unsigned long *p_addr, *bm;
- unsigned long word, bits;
- unsigned int idx;
- size_t end, do_now;
-
- end = offset + number;
+ struct drbd_device *device = peer_device->device;
+ unsigned long start, end;
- if (!expect(device, b))
- return;
- if (!expect(device, b->bm_pages))
- return;
- if (number == 0)
- return;
- WARN_ON(offset >= b->bm_words);
- WARN_ON(end > b->bm_words);
-
- spin_lock_irq(&b->bm_lock);
- while (offset < end) {
- do_now = min_t(size_t, ALIGN(offset+1, LWPP), end) - offset;
- idx = bm_word_to_page_idx(b, offset);
- p_addr = bm_map_pidx(b, idx);
- bm = p_addr + MLPP(offset);
- offset += do_now;
- while (do_now--) {
- bits = hweight_long(*bm);
- word = *bm | *buffer++;
- *bm++ = word;
- b->bm_set += hweight_long(word) - bits;
- }
- bm_unmap(p_addr);
- bm_set_page_need_writeout(b->bm_pages[idx]);
- }
- /* with 32bit <-> 64bit cross-platform connect
- * this is only correct for current usage,
- * where we _know_ that we are 64 bit aligned,
- * and know that this function is used in this way, too...
- */
- if (end == b->bm_words)
- b->bm_set -= bm_clear_surplus(b);
- spin_unlock_irq(&b->bm_lock);
+ start = offset * BITS_PER_LONG;
+ end = start + number * BITS_PER_LONG - 1;
+ bm_op(device, device->bitmap, peer_device->bitmap_index, start, end,
+ BM_OP_MERGE, (__le32 *)buffer);
}
/* copy number words from the bitmap starting at offset into the buffer.
* buffer[i] will be little endian unsigned long.
*/
-void drbd_bm_get_lel(struct drbd_device *device, size_t offset, size_t number,
+void drbd_bm_get_lel(struct drbd_peer_device *peer_device, size_t offset, size_t number,
unsigned long *buffer)
{
- struct drbd_bitmap *b = device->bitmap;
- unsigned long *p_addr, *bm;
- size_t end, do_now;
-
- end = offset + number;
-
- if (!expect(device, b))
- return;
- if (!expect(device, b->bm_pages))
- return;
-
- spin_lock_irq(&b->bm_lock);
- if ((offset >= b->bm_words) ||
- (end > b->bm_words) ||
- (number <= 0))
- drbd_err(device, "offset=%lu number=%lu bm_words=%lu\n",
- (unsigned long) offset,
- (unsigned long) number,
- (unsigned long) b->bm_words);
- else {
- while (offset < end) {
- do_now = min_t(size_t, ALIGN(offset+1, LWPP), end) - offset;
- p_addr = bm_map_pidx(b, bm_word_to_page_idx(b, offset));
- bm = p_addr + MLPP(offset);
- offset += do_now;
- while (do_now--)
- *buffer++ = *bm++;
- bm_unmap(p_addr);
- }
- }
- spin_unlock_irq(&b->bm_lock);
-}
-
-/* set all bits in the bitmap */
-void drbd_bm_set_all(struct drbd_device *device)
-{
- struct drbd_bitmap *b = device->bitmap;
- if (!expect(device, b))
- return;
- if (!expect(device, b->bm_pages))
- return;
+ struct drbd_device *device = peer_device->device;
+ unsigned long start, end;
- spin_lock_irq(&b->bm_lock);
- bm_memset(b, 0, 0xff, b->bm_words);
- (void)bm_clear_surplus(b);
- b->bm_set = b->bm_bits;
- spin_unlock_irq(&b->bm_lock);
+ start = offset * BITS_PER_LONG;
+ end = start + number * BITS_PER_LONG - 1;
+ bm_op(device, device->bitmap, peer_device->bitmap_index, start, end,
+ BM_OP_EXTRACT, (__le32 *)buffer);
}
-/* clear all bits in the bitmap */
-void drbd_bm_clear_all(struct drbd_device *device)
-{
- struct drbd_bitmap *b = device->bitmap;
- if (!expect(device, b))
- return;
- if (!expect(device, b->bm_pages))
- return;
-
- spin_lock_irq(&b->bm_lock);
- bm_memset(b, 0, 0, b->bm_words);
- b->bm_set = 0;
- spin_unlock_irq(&b->bm_lock);
-}
static void drbd_bm_aio_ctx_destroy(struct kref *kref)
{
struct drbd_bm_aio_ctx *ctx = container_of(kref, struct drbd_bm_aio_ctx, kref);
unsigned long flags;
- spin_lock_irqsave(&ctx->device->resource->req_lock, flags);
+ spin_lock_irqsave(&ctx->device->pending_bmio_lock, flags);
list_del(&ctx->list);
- spin_unlock_irqrestore(&ctx->device->resource->req_lock, flags);
+ spin_unlock_irqrestore(&ctx->device->pending_bmio_lock, flags);
put_ldev(ctx->device);
kfree(ctx);
}
@@ -932,25 +1289,28 @@ static void drbd_bm_aio_ctx_destroy(struct kref *kref)
/* bv_page may be a copy, or may be the original */
static void drbd_bm_endio(struct bio *bio)
{
+ /* ldev_ref_transfer: ldev ref from bio submit in bitmap I/O path */
struct drbd_bm_aio_ctx *ctx = bio->bi_private;
struct drbd_device *device = ctx->device;
struct drbd_bitmap *b = device->bitmap;
- unsigned int idx = bm_page_to_idx(bio_first_page_all(bio));
+ unsigned int idx = bm_page_to_idx(bio->bi_io_vec[0].bv_page);
+ const bool copy_pages = ctx->flags & BM_AIO_COPY_PAGES;
+
+ blk_status_t status = bio->bi_status;
- if ((ctx->flags & BM_AIO_COPY_PAGES) == 0 &&
- !bm_test_page_unchanged(b->bm_pages[idx]))
+ if (!copy_pages && !bm_test_page_unchanged(b->bm_pages[idx]))
drbd_warn(device, "bitmap page idx %u changed during IO!\n", idx);
- if (bio->bi_status) {
+ if (status) {
/* ctx error will hold the completed-last non-zero error code,
* in case error codes differ. */
- ctx->error = blk_status_to_errno(bio->bi_status);
+ ctx->error = blk_status_to_errno(status);
bm_set_page_io_err(b->bm_pages[idx]);
/* Not identical to on disk version of it.
* Is BM_PAGE_IO_ERROR enough? */
if (drbd_device_ratelimit(device, BACKEND))
drbd_err(device, "IO ERROR %d on bitmap page idx %u\n",
- bio->bi_status, idx);
+ status, idx);
} else {
bm_clear_page_io_err(b->bm_pages[idx]);
dynamic_drbd_dbg(device, "bitmap page idx %u completed\n", idx);
@@ -958,7 +1318,15 @@ static void drbd_bm_endio(struct bio *bio)
bm_page_unlock_io(device, idx);
- if (ctx->flags & BM_AIO_COPY_PAGES)
+ /* this should not count as user activity and cause the resync to
+ * throttle -- see drbd_rs_c_min_rate_throttle(). Only where the
+ * meta-data shares the backing device's disk: the detector reads the
+ * part_stat counters of that whole disk.
+ */
+ if (device->ldev->md_bdev->bd_disk == device->ldev->backing_bdev->bd_disk)
+ atomic_add(bio->bi_io_vec[0].bv_len >> 9, &device->rs_sect_done);
+
+ if (copy_pages)
mempool_free(bio->bi_io_vec[0].bv_page, &drbd_md_io_page_pool);
bio_put(bio);
@@ -967,6 +1335,9 @@ static void drbd_bm_endio(struct bio *bio)
ctx->done = 1;
wake_up(&device->misc_wait);
kref_put(&ctx->kref, &drbd_bm_aio_ctx_destroy);
+ } else if (copy_pages) {
+ /* a submitter may be parked on the in-flight bound */
+ wake_up(&device->misc_wait);
}
}
@@ -976,24 +1347,24 @@ static inline sector_t drbd_md_last_bitmap_sector(struct drbd_backing_dev *bdev)
switch (bdev->md.meta_dev_idx) {
case DRBD_MD_INDEX_INTERNAL:
case DRBD_MD_INDEX_FLEX_INT:
- return bdev->md.md_offset + bdev->md.al_offset -1;
+ return bdev->md.md_offset + bdev->md.al_offset - 1;
case DRBD_MD_INDEX_FLEX_EXT:
default:
- return bdev->md.md_offset + bdev->md.md_size_sect -1;
+ return bdev->md.md_offset + bdev->md.md_size_sect - 1;
}
}
-static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_hold(local)
+static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr)
{
+ struct bio *bio;
struct drbd_device *device = ctx->device;
- enum req_op op = ctx->flags & BM_AIO_READ ? REQ_OP_READ : REQ_OP_WRITE;
struct drbd_bitmap *b = device->bitmap;
- struct bio *bio;
struct page *page;
sector_t last_bm_sect;
sector_t first_bm_sect;
sector_t on_disk_sector;
unsigned int len;
+ enum req_op op = ctx->flags & BM_AIO_READ ? REQ_OP_READ : REQ_OP_WRITE;
first_bm_sect = device->ldev->md.md_offset + device->ldev->md.bm_offset;
on_disk_sector = first_bm_sect + (((sector_t)page_nr) << (PAGE_SHIFT-SECTOR_SHIFT));
@@ -1004,6 +1375,7 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho
last_bm_sect = drbd_md_last_bitmap_sector(device->ldev);
if (first_bm_sect <= on_disk_sector && last_bm_sect >= on_disk_sector) {
sector_t len_sect = last_bm_sect - on_disk_sector + 1;
+
if (len_sect < PAGE_SIZE/SECTOR_SIZE)
len = (unsigned int)len_sect*SECTOR_SIZE;
else
@@ -1011,7 +1383,7 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho
} else {
if (drbd_device_ratelimit(device, METADATA)) {
drbd_err(device, "Invalid offset during on-disk bitmap access: "
- "page idx %u, sector %llu\n", page_nr, on_disk_sector);
+ "page idx %u, sector %llu\n", page_nr, (unsigned long long) on_disk_sector);
}
ctx->error = -EIO;
bm_set_page_io_err(b->bm_pages[page_nr]);
@@ -1036,35 +1408,93 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho
bm_store_page_idx(page, page_nr);
} else
page = b->bm_pages[page_nr];
+
bio = bio_alloc_bioset(device->ldev->md_bdev, 1, op, GFP_NOIO,
- &drbd_md_io_bio_set);
+ &drbd_md_io_bio_set);
bio->bi_iter.bi_sector = on_disk_sector;
__bio_add_page(bio, page, len, 0);
bio->bi_private = ctx;
bio->bi_end_io = drbd_bm_endio;
if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) {
- bio_io_error(bio);
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
} else {
submit_bio(bio);
- /* this should not count as user activity and cause the
- * resync to throttle -- see drbd_rs_should_slow_down(). */
- atomic_add(len >> 9, &device->rs_sect_ev);
+ if (op == REQ_OP_WRITE)
+ device->bm_writ_cnt++;
}
}
-/*
- * bm_rw: read/write the whole bitmap from/to its on disk location.
+/* A copying write-out allocates one page per page in flight, and the
+ * submitter fills the plug far faster than the backend drains it. Bound what
+ * one write-out may hold at once; the whole bitmap is gigabytes on a large
+ * volume with several bitmap slots.
+ */
+#define BM_AIO_COPY_PAGES_MAX_IN_FLIGHT 1024
+
+/* Wait until a copying write-out is back under its in-flight bound. Going to
+ * sleep flushes the plug, so what is queued reaches the backend.
+ * The disk-timeout applies as in wait_until_done_or_force_detached(): if no
+ * page completes within it, force-detach.
+ * Returns false if the local disk gave up and submitting more is pointless.
*/
-static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned lazy_writeout_upper_idx) __must_hold(local)
+static bool bm_wait_for_copy_page(struct drbd_device *device, struct drbd_bm_aio_ctx *ctx)
+{
+ long dt;
+
+ if (!(ctx->flags & BM_AIO_COPY_PAGES))
+ return true;
+
+ /* ldev_safe: get_ldev_if_state() in bm_rw_range(), put_ldev in drbd_bm_aio_ctx_destroy() */
+ dt = drbd_disk_timeout(device->ldev);
+
+ /* in_flight carries the submitter's own reference as well */
+ dt = wait_event_timeout(device->misc_wait,
+ atomic_read(&ctx->in_flight) <= BM_AIO_COPY_PAGES_MAX_IN_FLIGHT ||
+ test_bit(FORCE_DETACH, &device->flags) ||
+ test_bit(ABORT_MDIO, &device->flags),
+ dt);
+
+ if (dt == 0) {
+ drbd_err(device, "meta-data IO operation timed out\n");
+ drbd_handle_io_error(device, DRBD_FORCE_DETACH);
+ return false;
+ }
+
+ return atomic_read(&ctx->in_flight) <= BM_AIO_COPY_PAGES_MAX_IN_FLIGHT;
+}
+
+/**
+ * bm_rw_range() - read/write the specified range of bitmap pages
+ * @device: drbd device this bitmap is associated with
+ * @start_page: start of bitmap page indices to process
+ * @end_page: end of bitmap page indices to process
+ * @flags: BM_AIO_*, see struct bm_aio_ctx.
+ *
+ * Silently limits end_page to the current bitmap size.
+ *
+ * We don't want to special case on logical_block_size of the backend device,
+ * so we submit PAGE_SIZE aligned pieces.
+ * Note that on "most" systems, PAGE_SIZE is 4k.
+ *
+ * In case this becomes an issue on systems with larger PAGE_SIZE,
+ * we may want to change this again to do 4k aligned 4k pieces.
+ */
+static int bm_rw_range(struct drbd_device *device, unsigned int start_page, unsigned int end_page,
+ unsigned int flags)
{
struct drbd_bm_aio_ctx *ctx;
struct drbd_bitmap *b = device->bitmap;
- unsigned int num_pages, i, count = 0;
+ unsigned int i, count = 0;
unsigned long now;
- char ppb[10];
int err = 0;
+ if (b->bm_flags & BM_ON_DAX_PMEM) {
+ if (flags & (BM_AIO_WRITE_HINTED | BM_AIO_WRITE_ALL_PAGES | BM_AIO_WRITE_LAZY))
+ arch_wb_cache_pmem(b->bm_on_pmem, b->bm_words * sizeof(long));
+ return 0;
+ }
/*
* We are protected against bitmap disappearing/resizing by holding an
* ldev reference (caller must have called get_ldev()).
@@ -1074,6 +1504,10 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
* as we submit copies of pages anyways.
*/
+ /* if we reach this, we should have at least *some* bitmap pages. */
+ if (!expect(device, b->bm_number_of_pages))
+ return -ENODEV;
+
ctx = kmalloc_obj(struct drbd_bm_aio_ctx, GFP_NOIO);
if (!ctx)
return -ENOMEM;
@@ -1088,29 +1522,33 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
.kref = KREF_INIT(2),
};
- if (!get_ldev_if_state(device, D_ATTACHING)) { /* put is in drbd_bm_aio_ctx_destroy() */
- drbd_err(device, "ASSERT FAILED: get_ldev_if_state() == 1 in bm_rw()\n");
+ if (!expect(device, get_ldev_if_state(device, D_ATTACHING))) { /* put is in drbd_bm_aio_ctx_destroy() */
kfree(ctx);
return -ENODEV;
}
- /* Here D_ATTACHING is sufficient since drbd_bm_read() is called only from
- drbd_adm_attach(), after device->ldev was assigned. */
+ /* Here, D_ATTACHING is sufficient because drbd_bm_read() is only
+ * called from drbd_nl_attach_doit(), after device->ldev has been assigned.
+ *
+ * The corresponding put_ldev() happens in bm_aio_ctx_destroy().
+ */
if (0 == (ctx->flags & ~BM_AIO_READ))
- WARN_ON(!(BM_LOCKED_MASK & b->bm_flags));
+ WARN_ON(!(b->bm_flags & BM_LOCK_ALL));
- spin_lock_irq(&device->resource->req_lock);
- list_add_tail(&ctx->list, &device->pending_bitmap_io);
- spin_unlock_irq(&device->resource->req_lock);
+ if (end_page >= b->bm_number_of_pages)
+ end_page = b->bm_number_of_pages - 1;
- num_pages = b->bm_number_of_pages;
+ spin_lock_irq(&device->pending_bmio_lock);
+ list_add_tail(&ctx->list, &device->pending_bitmap_io);
+ spin_unlock_irq(&device->pending_bmio_lock);
now = jiffies;
- /* let the layers below us try to merge these bios... */
+ blk_start_plug(&ctx->bm_aio_plug);
+ /* implicit unplug if scheduled for whatever reason */
if (flags & BM_AIO_READ) {
- for (i = 0; i < num_pages; i++) {
+ for (i = start_page; i <= end_page; i++) {
atomic_inc(&ctx->in_flight);
bm_page_io_async(ctx, i);
++count;
@@ -1119,9 +1557,10 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
} else if (flags & BM_AIO_WRITE_HINTED) {
/* ASSERT: BM_AIO_WRITE_ALL_PAGES is not set. */
unsigned int hint;
+
for (hint = 0; hint < b->n_bitmap_hints; hint++) {
i = b->al_bitmap_hints[hint];
- if (i >= num_pages) /* == -1U: no hint here. */
+ if (i > end_page)
continue;
/* Several AL-extents may point to the same page. */
if (!test_and_clear_bit(BM_PAGE_HINT_WRITEOUT,
@@ -1130,15 +1569,16 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
/* Has it even changed? */
if (bm_test_page_unchanged(b->bm_pages[i]))
continue;
+ if (!bm_wait_for_copy_page(device, ctx))
+ break;
atomic_inc(&ctx->in_flight);
bm_page_io_async(ctx, i);
++count;
}
} else {
- for (i = 0; i < num_pages; i++) {
- /* ignore completely unchanged pages */
- if (lazy_writeout_upper_idx && i == lazy_writeout_upper_idx)
- break;
+ for (i = start_page; i <= end_page; i++) {
+ /* ignore completely unchanged pages,
+ * unless specifically requested to write ALL pages */
if (!(flags & BM_AIO_WRITE_ALL_PAGES) &&
bm_test_page_unchanged(b->bm_pages[i])) {
dynamic_drbd_dbg(device, "skipped bm write for idx %u\n", i);
@@ -1146,17 +1586,21 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
}
/* during lazy writeout,
* ignore those pages not marked for lazy writeout. */
- if (lazy_writeout_upper_idx &&
+ if ((flags & BM_AIO_WRITE_LAZY) &&
!bm_test_page_lazy_writeout(b->bm_pages[i])) {
dynamic_drbd_dbg(device, "skipped bm lazy write for idx %u\n", i);
continue;
}
+ if (!bm_wait_for_copy_page(device, ctx))
+ break;
atomic_inc(&ctx->in_flight);
bm_page_io_async(ctx, i);
++count;
cond_resched();
}
}
+ /* explicit unplug, we are done submitting */
+ blk_finish_plug(&ctx->bm_aio_plug);
/*
* We initialize ctx->in_flight to one to make sure drbd_bm_endio
@@ -1166,14 +1610,17 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
* no need to wait. Still, we need to put the kref associated with the
* "in_flight reached zero, all done" event.
*/
- if (!atomic_dec_and_test(&ctx->in_flight))
+ if (!atomic_dec_and_test(&ctx->in_flight)) {
+ /* ldev_safe: get_ldev_if_state() above, put_ldev in drbd_bm_aio_ctx_destroy() */
wait_until_done_or_force_detached(device, device->ldev, &ctx->done);
- else
+ } else {
kref_put(&ctx->kref, &drbd_bm_aio_ctx_destroy);
+ }
- /* summary for global bitmap IO */
- if (flags == 0) {
+ /* summary stats for global bitmap IO */
+ if ((flags & BM_AIO_NO_STATS) == 0 && count) {
unsigned int ms = jiffies_to_msecs(jiffies - now);
+
if (ms > 5) {
drbd_info(device, "bitmap %s of %u pages took %u ms\n",
(flags & BM_AIO_READ) ? "READ" : "WRITE",
@@ -1182,63 +1629,108 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
}
if (ctx->error) {
- drbd_alert(device, "we had at least one MD IO ERROR during bitmap IO\n");
- drbd_chk_io_error(device, 1, DRBD_META_IO_ERROR);
+ drbd_err(device, "we had at least one MD IO ERROR during bitmap IO\n");
+ drbd_handle_io_error(device, DRBD_META_IO_ERROR);
err = -EIO; /* ctx->error ? */
}
if (atomic_read(&ctx->in_flight))
err = -EIO; /* Disk timeout/force-detach during IO... */
- now = jiffies;
if (flags & BM_AIO_READ) {
- b->bm_set = bm_count_bits(b);
- drbd_info(device, "recounting of set bits took additional %lu jiffies\n",
- jiffies - now);
+ unsigned int ms;
+
+ now = jiffies;
+ bm_count_bits(device, b);
+ ms = jiffies_to_msecs(jiffies - now);
+ /* If we can count quickly, there is no need to report this either */
+ if (ms > 3)
+ drbd_info(device, "recounting of set bits took additional %ums\n", ms);
}
- now = b->bm_set;
-
- if ((flags & ~BM_AIO_READ) == 0)
- drbd_info(device, "%s (%lu bits) marked out-of-sync by on disk bit-map.\n",
- ppsize(ppb, now << (BM_BLOCK_SHIFT-10)), now);
kref_put(&ctx->kref, &drbd_bm_aio_ctx_destroy);
return err;
}
+static int bm_rw(struct drbd_device *device, unsigned int flags)
+{
+ return bm_rw_range(device, 0, -1U, flags);
+}
+
/*
* drbd_bm_read() - Read the whole bitmap from its on disk location.
* @device: DRBD device.
+ * @peer_device: parameter ignored
*/
int drbd_bm_read(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local)
+ struct drbd_peer_device *peer_device)
+{
+ return bm_rw(device, BM_AIO_READ);
+}
+static void push_al_bitmap_hint(struct drbd_device *device, unsigned int page_nr)
{
- return bm_rw(device, BM_AIO_READ, 0);
+ struct drbd_bitmap *b = device->bitmap;
+ struct page *page = b->bm_pages[page_nr];
+
+ BUG_ON(b->n_bitmap_hints >= ARRAY_SIZE(b->al_bitmap_hints));
+ if (!test_and_set_bit(BM_PAGE_HINT_WRITEOUT, &page_private(page)))
+ b->al_bitmap_hints[b->n_bitmap_hints++] = page_nr;
+}
+
+/**
+ * drbd_bm_mark_range_for_writeout() - mark with a "hint" to be considered for writeout
+ * @device: DRBD device.
+ * @start: Start index of the range to mark.
+ * @end: End index of the range to mark.
+ *
+ * From within an activity log transaction, we mark a few pages with these
+ * hints, then call drbd_bm_write_hinted(), which will only write out changed
+ * pages which are flagged with this mark.
+ */
+void drbd_bm_mark_range_for_writeout(struct drbd_device *device, unsigned long start, unsigned long end)
+{
+ struct drbd_bitmap *bitmap = device->bitmap;
+ unsigned int page_nr, last_page;
+
+ if (bitmap->bm_flags & BM_ON_DAX_PMEM)
+ return;
+
+ if (end >= bitmap->bm_bits)
+ end = bitmap->bm_bits - 1;
+
+ page_nr = bit_to_page_interleaved(bitmap, 0, start);
+ last_page = bit_to_page_interleaved(bitmap, bitmap->bm_max_peers - 1, end);
+ for (; page_nr <= last_page; page_nr++)
+ push_al_bitmap_hint(device, page_nr);
}
+
/*
* drbd_bm_write() - Write the whole bitmap to its on disk location.
* @device: DRBD device.
+ * @peer_device: parameter ignored
*
* Will only write pages that have changed since last IO.
*/
int drbd_bm_write(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local)
+ struct drbd_peer_device *peer_device)
{
- return bm_rw(device, 0, 0);
+ return bm_rw(device, 0);
}
/*
* drbd_bm_write_all() - Write the whole bitmap to its on disk location.
- * @device: DRBD device.
+ * @device: DRBD device.
+ * @peer_device: parameter ignored
*
- * Will write all pages.
+ * Will write all pages. Is used for online resize operations. The
+ * whole bitmap should be written into its new position.
*/
int drbd_bm_write_all(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local)
+ struct drbd_peer_device *peer_device)
{
- return bm_rw(device, BM_AIO_WRITE_ALL_PAGES, 0);
+ return bm_rw(device, BM_AIO_WRITE_ALL_PAGES);
}
/**
@@ -1246,14 +1738,16 @@ int drbd_bm_write_all(struct drbd_device *device,
* @device: DRBD device.
* @upper_idx: 0: write all changed pages; +ve: page index to stop scanning for changed pages
*/
-int drbd_bm_write_lazy(struct drbd_device *device, unsigned upper_idx) __must_hold(local)
+int drbd_bm_write_lazy(struct drbd_device *device, unsigned int upper_idx)
{
- return bm_rw(device, BM_AIO_COPY_PAGES, upper_idx);
+ return bm_rw_range(device, 0, upper_idx - 1,
+ BM_AIO_COPY_PAGES | BM_AIO_WRITE_LAZY | BM_AIO_NO_STATS);
}
/*
* drbd_bm_write_copy_pages() - Write the whole bitmap to its on disk location.
* @device: DRBD device.
+ * @peer_device: parameter ignored
*
* Will only write pages that have changed since last IO.
* In contrast to drbd_bm_write(), this will copy the bitmap pages
@@ -1263,431 +1757,224 @@ int drbd_bm_write_lazy(struct drbd_device *device, unsigned upper_idx) __must_ho
* pending resync acks are still being processed.
*/
int drbd_bm_write_copy_pages(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local)
+ struct drbd_peer_device *peer_device)
{
- return bm_rw(device, BM_AIO_COPY_PAGES, 0);
+ return bm_rw(device, BM_AIO_COPY_PAGES);
}
/*
* drbd_bm_write_hinted() - Write bitmap pages with "hint" marks, if they have changed.
* @device: DRBD device.
*/
-int drbd_bm_write_hinted(struct drbd_device *device) __must_hold(local)
+int drbd_bm_write_hinted(struct drbd_device *device)
{
- return bm_rw(device, BM_AIO_WRITE_HINTED | BM_AIO_COPY_PAGES, 0);
+ return bm_rw(device, BM_AIO_WRITE_HINTED | BM_AIO_COPY_PAGES | BM_AIO_NO_STATS);
}
-/* NOTE
- * find_first_bit returns int, we return unsigned long.
- * For this to work on 32bit arch with bitnumbers > (1<<32),
- * we'd need to return u64, and get a whole lot of other places
- * fixed where we still use unsigned long.
- *
- * this returns a bit number, NOT a sector!
- */
-static unsigned long __bm_find_next(struct drbd_device *device, unsigned long bm_fo,
- const int find_zero_bit)
+static __always_inline unsigned long
+__bm_many_bits_op(struct drbd_device *device, unsigned int bitmap_index,
+ unsigned long start, unsigned long end, enum bitmap_operations op)
{
- struct drbd_bitmap *b = device->bitmap;
- unsigned long *p_addr;
- unsigned long bit_offset;
- unsigned i;
+ struct drbd_bitmap *bitmap = device->bitmap;
+ unsigned long result = DRBD_END_OF_BITMAP;
+ unsigned long bit = start;
+ spin_lock_irq(&bitmap->bm_lock);
- if (bm_fo > b->bm_bits) {
- drbd_err(device, "bm_fo=%lu bm_bits=%lu\n", bm_fo, b->bm_bits);
- bm_fo = DRBD_END_OF_BITMAP;
- } else {
- while (bm_fo < b->bm_bits) {
- /* bit offset of the first bit in the page */
- bit_offset = bm_fo & ~BITS_PER_PAGE_MASK;
- p_addr = __bm_map_pidx(b, bm_bit_to_page_idx(b, bm_fo));
-
- if (find_zero_bit)
- i = find_next_zero_bit_le(p_addr,
- PAGE_SIZE*8, bm_fo & BITS_PER_PAGE_MASK);
- else
- i = find_next_bit_le(p_addr,
- PAGE_SIZE*8, bm_fo & BITS_PER_PAGE_MASK);
+ if (end >= bitmap->bm_bits)
+ end = bitmap->bm_bits - 1;
- __bm_unmap(p_addr);
- if (i < PAGE_SIZE*8) {
- bm_fo = bit_offset + i;
- if (bm_fo >= b->bm_bits)
- break;
- goto found;
- }
- bm_fo = bit_offset + PAGE_SIZE*8;
- }
- bm_fo = DRBD_END_OF_BITMAP;
- }
- found:
- return bm_fo;
-}
+ while (bit <= end && bit < bitmap->bm_bits) {
+ unsigned long last_bit = last_bit_on_page(bitmap, bitmap_index, bit);
-static unsigned long bm_find_next(struct drbd_device *device,
- unsigned long bm_fo, const int find_zero_bit)
-{
- struct drbd_bitmap *b = device->bitmap;
- unsigned long i = DRBD_END_OF_BITMAP;
-
- if (!expect(device, b))
- return i;
- if (!expect(device, b->bm_pages))
- return i;
-
- spin_lock_irq(&b->bm_lock);
- if (BM_DONT_TEST & b->bm_flags)
- bm_print_lock_info(device);
-
- i = __bm_find_next(device, bm_fo, find_zero_bit);
+ if (end < last_bit)
+ last_bit = end;
- spin_unlock_irq(&b->bm_lock);
- return i;
-}
+ result = __bm_op(device, bitmap, bitmap_index, bit, last_bit, op, NULL);
+ if ((op == BM_OP_FIND_BIT || op == BM_OP_FIND_ZERO_BIT) &&
+ result != DRBD_END_OF_BITMAP)
+ break;
+ bit = last_bit + 1;
+ spin_unlock_irq(&bitmap->bm_lock);
+ if (need_resched())
+ cond_resched();
+ spin_lock_irq(&bitmap->bm_lock);
+ }
+ spin_unlock_irq(&bitmap->bm_lock);
-unsigned long drbd_bm_find_next(struct drbd_device *device, unsigned long bm_fo)
-{
- return bm_find_next(device, bm_fo, 0);
+ return result;
}
-#if 0
-/* not yet needed for anything. */
-unsigned long drbd_bm_find_next_zero(struct drbd_device *device, unsigned long bm_fo)
+/*
+ * A single bm_op() find would scan up to the whole bitmap while holding
+ * bm_lock with interrupts disabled. With a large device the bitmap is
+ * large, too (a 64 TiB device has a 1 GiB bitmap), and scanning a mostly
+ * clear range keeps interrupts off for hundreds of milliseconds, stalling
+ * RCU and triggering soft lockups. Scan page by page instead.
+ */
+unsigned long drbd_bm_find_next(struct drbd_peer_device *peer_device, unsigned long start)
{
- return bm_find_next(device, bm_fo, 1);
+ return __bm_many_bits_op(peer_device->device, peer_device->bitmap_index, start, -1UL,
+ BM_OP_FIND_BIT);
}
-#endif
/* does not spin_lock_irqsave.
* you must take drbd_bm_lock() first */
-unsigned long _drbd_bm_find_next(struct drbd_device *device, unsigned long bm_fo)
+unsigned long _drbd_bm_find_next(struct drbd_peer_device *peer_device, unsigned long start)
{
- /* WARN_ON(!(BM_DONT_SET & device->b->bm_flags)); */
- return __bm_find_next(device, bm_fo, 0);
+ /* WARN_ON(!(device->b->bm_flags & BM_LOCK_SET)); */
+ return ____bm_op(peer_device->device, peer_device->device->bitmap,
+ peer_device->bitmap_index, start, -1UL, BM_OP_FIND_BIT, NULL);
}
-unsigned long _drbd_bm_find_next_zero(struct drbd_device *device, unsigned long bm_fo)
+unsigned long _drbd_bm_find_next_zero(struct drbd_peer_device *peer_device, unsigned long start)
{
- /* WARN_ON(!(BM_DONT_SET & device->b->bm_flags)); */
- return __bm_find_next(device, bm_fo, 1);
+ /* WARN_ON(!(device->b->bm_flags & BM_LOCK_SET)); */
+ return ____bm_op(peer_device->device, peer_device->device->bitmap,
+ peer_device->bitmap_index, start, -1UL, BM_OP_FIND_ZERO_BIT, NULL);
}
-/* returns number of bits actually changed.
- * for val != 0, we change 0 -> 1, return code positive
- * for val == 0, we change 1 -> 0, return code negative
- * wants bitnr, not sector.
- * expected to be called for only a few bits (e - s about BITS_PER_LONG).
- * Must hold bitmap lock already. */
-static int __bm_change_bits_to(struct drbd_device *device, const unsigned long s,
- unsigned long e, int val)
-{
- struct drbd_bitmap *b = device->bitmap;
- unsigned long *p_addr = NULL;
- unsigned long bitnr;
- unsigned int last_page_nr = -1U;
- int c = 0;
- int changed_total = 0;
-
- if (e >= b->bm_bits) {
- drbd_err(device, "ASSERT FAILED: bit_s=%lu bit_e=%lu bm_bits=%lu\n",
- s, e, b->bm_bits);
- e = b->bm_bits ? b->bm_bits -1 : 0;
- }
- for (bitnr = s; bitnr <= e; bitnr++) {
- unsigned int page_nr = bm_bit_to_page_idx(b, bitnr);
- if (page_nr != last_page_nr) {
- if (p_addr)
- __bm_unmap(p_addr);
- if (c < 0)
- bm_set_page_lazy_writeout(b->bm_pages[last_page_nr]);
- else if (c > 0)
- bm_set_page_need_writeout(b->bm_pages[last_page_nr]);
- changed_total += c;
- c = 0;
- p_addr = __bm_map_pidx(b, page_nr);
- last_page_nr = page_nr;
- }
- if (val)
- c += (0 == __test_and_set_bit_le(bitnr & BITS_PER_PAGE_MASK, p_addr));
- else
- c -= (0 != __test_and_clear_bit_le(bitnr & BITS_PER_PAGE_MASK, p_addr));
- }
- if (p_addr)
- __bm_unmap(p_addr);
- if (c < 0)
- bm_set_page_lazy_writeout(b->bm_pages[last_page_nr]);
- else if (c > 0)
- bm_set_page_need_writeout(b->bm_pages[last_page_nr]);
- changed_total += c;
- b->bm_set += changed_total;
- return changed_total;
-}
-
-/* returns number of bits actually changed.
- * for val != 0, we change 0 -> 1, return code positive
- * for val == 0, we change 1 -> 0, return code negative
- * wants bitnr, not sector */
-static int bm_change_bits_to(struct drbd_device *device, const unsigned long s,
- const unsigned long e, int val)
+unsigned int drbd_bm_set_bits(struct drbd_device *device, unsigned int bitmap_index,
+ unsigned long start, unsigned long end)
{
- unsigned long flags;
- struct drbd_bitmap *b = device->bitmap;
- int c = 0;
-
- if (!expect(device, b))
- return 1;
- if (!expect(device, b->bm_pages))
- return 0;
-
- spin_lock_irqsave(&b->bm_lock, flags);
- if ((val ? BM_DONT_SET : BM_DONT_CLEAR) & b->bm_flags)
- bm_print_lock_info(device);
-
- c = __bm_change_bits_to(device, s, e, val);
-
- spin_unlock_irqrestore(&b->bm_lock, flags);
- return c;
+ return bm_op(device, device->bitmap, bitmap_index, start, end, BM_OP_SET, NULL);
}
-/* returns number of bits changed 0 -> 1 */
-int drbd_bm_set_bits(struct drbd_device *device, const unsigned long s, const unsigned long e)
+void drbd_bm_set_many_bits(struct drbd_peer_device *peer_device, unsigned long start, unsigned long end)
{
- return bm_change_bits_to(device, s, e, 1);
+ if (peer_device->bitmap_index == -1)
+ return;
+ __bm_many_bits_op(peer_device->device, peer_device->bitmap_index, start, end, BM_OP_SET);
}
-/* returns number of bits changed 1 -> 0 */
-int drbd_bm_clear_bits(struct drbd_device *device, const unsigned long s, const unsigned long e)
+void drbd_bm_clear_many_bits(struct drbd_peer_device *peer_device, unsigned long start, unsigned long end)
{
- return -bm_change_bits_to(device, s, e, 0);
+ if (peer_device->bitmap_index == -1)
+ return;
+ __bm_many_bits_op(peer_device->device, peer_device->bitmap_index, start, end, BM_OP_CLEAR);
}
-/* sets all bits in full words,
- * from first_word up to, but not including, last_word */
-static inline void bm_set_full_words_within_one_page(struct drbd_bitmap *b,
- int page_nr, int first_word, int last_word)
+void
+_drbd_bm_clear_many_bits(struct drbd_device *device, int bitmap_index, unsigned long start, unsigned long end)
{
- int i;
- int bits;
- int changed = 0;
- unsigned long *paddr = kmap_atomic(b->bm_pages[page_nr]);
-
- /* I think it is more cache line friendly to hweight_long then set to ~0UL,
- * than to first bitmap_weight() all words, then bitmap_fill() all words */
- for (i = first_word; i < last_word; i++) {
- bits = hweight_long(paddr[i]);
- paddr[i] = ~0UL;
- changed += BITS_PER_LONG - bits;
- }
- kunmap_atomic(paddr);
- if (changed) {
- /* We only need lazy writeout, the information is still in the
- * remote bitmap as well, and is reconstructed during the next
- * bitmap exchange, if lost locally due to a crash. */
- bm_set_page_lazy_writeout(b->bm_pages[page_nr]);
- b->bm_set += changed;
- }
+ __bm_many_bits_op(device, bitmap_index, start, end, BM_OP_CLEAR);
}
-/* Same thing as drbd_bm_set_bits,
- * but more efficient for a large bit range.
- * You must first drbd_bm_lock().
- * Can be called to set the whole bitmap in one go.
- * Sets bits from s to e _inclusive_. */
-void _drbd_bm_set_bits(struct drbd_device *device, const unsigned long s, const unsigned long e)
+void
+_drbd_bm_set_many_bits(struct drbd_device *device, int bitmap_index, unsigned long start, unsigned long end)
{
- /* First set_bit from the first bit (s)
- * up to the next long boundary (sl),
- * then assign full words up to the last long boundary (el),
- * then set_bit up to and including the last bit (e).
- *
- * Do not use memset, because we must account for changes,
- * so we need to loop over the words with hweight() anyways.
- */
- struct drbd_bitmap *b = device->bitmap;
- unsigned long sl = ALIGN(s,BITS_PER_LONG);
- unsigned long el = (e+1) & ~((unsigned long)BITS_PER_LONG-1);
- int first_page;
- int last_page;
- int page_nr;
- int first_word;
- int last_word;
-
- if (e - s <= 3*BITS_PER_LONG) {
- /* don't bother; el and sl may even be wrong. */
- spin_lock_irq(&b->bm_lock);
- __bm_change_bits_to(device, s, e, 1);
- spin_unlock_irq(&b->bm_lock);
- return;
- }
-
- /* difference is large enough that we can trust sl and el */
-
- spin_lock_irq(&b->bm_lock);
-
- /* bits filling the current long */
- if (sl)
- __bm_change_bits_to(device, s, sl-1, 1);
-
- first_page = sl >> (3 + PAGE_SHIFT);
- last_page = el >> (3 + PAGE_SHIFT);
-
- /* MLPP: modulo longs per page */
- /* LWPP: long words per page */
- first_word = MLPP(sl >> LN2_BPL);
- last_word = LWPP;
+ __bm_many_bits_op(device, bitmap_index, start, end, BM_OP_SET);
+}
- /* first and full pages, unless first page == last page */
- for (page_nr = first_page; page_nr < last_page; page_nr++) {
- bm_set_full_words_within_one_page(device->bitmap, page_nr, first_word, last_word);
- spin_unlock_irq(&b->bm_lock);
- cond_resched();
- first_word = 0;
- spin_lock_irq(&b->bm_lock);
- }
- /* last page (respectively only page, for first page == last page) */
- last_word = MLPP(el >> LN2_BPL);
-
- /* consider bitmap->bm_bits = 32768, bitmap->bm_number_of_pages = 1. (or multiples).
- * ==> e = 32767, el = 32768, last_page = 2,
- * and now last_word = 0.
- * We do not want to touch last_page in this case,
- * as we did not allocate it, it is not present in bitmap->bm_pages.
- */
- if (last_word)
- bm_set_full_words_within_one_page(device->bitmap, last_page, first_word, last_word);
+/* set all bits in the bitmap */
+void drbd_bm_set_all(struct drbd_device *device)
+{
+ struct drbd_bitmap *bitmap = device->bitmap;
+ unsigned int bitmap_index;
- /* possibly trailing bits.
- * example: (e & 63) == 63, el will be e+1.
- * if that even was the very last bit,
- * it would trigger an assert in __bm_change_bits_to()
- */
- if (el <= e)
- __bm_change_bits_to(device, el, e, 1);
- spin_unlock_irq(&b->bm_lock);
+ for (bitmap_index = 0; bitmap_index < bitmap->bm_max_peers; bitmap_index++)
+ __bm_many_bits_op(device, bitmap_index, 0, -1, BM_OP_SET);
}
-/* returns bit state
- * wants bitnr, NOT sector.
- * inherently racy... area needs to be locked by means of {al,rs}_lru
- * 1 ... bit set
- * 0 ... bit not set
- * -1 ... first out of bounds access, stop testing for bits!
+/* clear all bits in the bitmap, except for the changed block tracking slot:
+ * the data did not change, so its record of changes stays valid
*/
-int drbd_bm_test_bit(struct drbd_device *device, const unsigned long bitnr)
+void drbd_bm_clear_all(struct drbd_device *device)
{
- unsigned long flags;
- struct drbd_bitmap *b = device->bitmap;
- unsigned long *p_addr;
- int i;
-
- if (!expect(device, b))
- return 0;
- if (!expect(device, b->bm_pages))
- return 0;
+ struct drbd_bitmap *bitmap = device->bitmap;
+ /* ldev_safe: callers clear the bitmap of an attached disk */
+ int cbt_index = drbd_cbt_bitmap_index(device->ldev);
+ unsigned int bitmap_index;
- spin_lock_irqsave(&b->bm_lock, flags);
- if (BM_DONT_TEST & b->bm_flags)
- bm_print_lock_info(device);
- if (bitnr < b->bm_bits) {
- p_addr = bm_map_pidx(b, bm_bit_to_page_idx(b, bitnr));
- i = test_bit_le(bitnr & BITS_PER_PAGE_MASK, p_addr) ? 1 : 0;
- bm_unmap(p_addr);
- } else if (bitnr == b->bm_bits) {
- i = -1;
- } else { /* (bitnr > b->bm_bits) */
- drbd_err(device, "bitnr=%lu > bm_bits=%lu\n", bitnr, b->bm_bits);
- i = 0;
- }
+ for (bitmap_index = 0; bitmap_index < bitmap->bm_max_peers; bitmap_index++)
+ if (bitmap_index != cbt_index)
+ __bm_many_bits_op(device, bitmap_index, 0, -1, BM_OP_CLEAR);
+}
- spin_unlock_irqrestore(&b->bm_lock, flags);
- return i;
+unsigned int drbd_bm_clear_bits(struct drbd_device *device, unsigned int bitmap_index,
+ unsigned long start, unsigned long end)
+{
+ return bm_op(device, device->bitmap, bitmap_index, start, end, BM_OP_CLEAR, NULL);
}
+
/* returns number of bits set in the range [s, e] */
-int drbd_bm_count_bits(struct drbd_device *device, const unsigned long s, const unsigned long e)
-{
- unsigned long flags;
- struct drbd_bitmap *b = device->bitmap;
- unsigned long *p_addr = NULL;
- unsigned long bitnr;
- unsigned int page_nr = -1U;
- int c = 0;
-
- /* If this is called without a bitmap, that is a bug. But just to be
- * robust in case we screwed up elsewhere, in that case pretend there
- * was one dirty bit in the requested area, so we won't try to do a
- * local read there (no bitmap probably implies no disk) */
- if (!expect(device, b))
- return 1;
- if (!expect(device, b->bm_pages))
- return 1;
+int drbd_bm_count_bits(struct drbd_device *device, unsigned int bitmap_index, unsigned long s, unsigned long e)
+{
+ return bm_op(device, device->bitmap, bitmap_index, s, e, BM_OP_COUNT, NULL);
+}
+
+enum bm_slot_op { BM_SLOT_COPY, BM_SLOT_OR };
+
+static void __bm_slot_op(struct drbd_device *device, unsigned int from_index,
+ unsigned int to_index, enum bm_slot_op op)
+{
+ struct drbd_bitmap *bitmap = device->bitmap;
+ unsigned long word_nr, from_word_nr, to_word_nr, words32_total;
+ unsigned int from_page_nr, to_page_nr, current_page_nr;
+ u32 data_word, *addr, *to_word;
+
+ words32_total = bitmap->bm_words * sizeof(unsigned long) / sizeof(u32);
+ spin_lock_irq(&bitmap->bm_all_slots_lock);
+ spin_lock(&bitmap->bm_lock);
+
+ if (op == BM_SLOT_COPY)
+ bitmap->bm_set[to_index] = 0;
+ current_page_nr = 0;
+ addr = bm_map(bitmap, current_page_nr);
+ for (word_nr = 0; word_nr < words32_total; word_nr += bitmap->bm_max_peers) {
+ from_word_nr = word_nr + from_index;
+ from_page_nr = word32_to_page(from_word_nr);
+ to_word_nr = word_nr + to_index;
+ to_page_nr = word32_to_page(to_word_nr);
+
+ if (current_page_nr != from_page_nr) {
+ bm_unmap(bitmap, addr);
+ spin_unlock(&bitmap->bm_lock);
+ spin_unlock_irq(&bitmap->bm_all_slots_lock);
+ if (need_resched())
+ cond_resched();
+ spin_lock_irq(&bitmap->bm_all_slots_lock);
+ spin_lock(&bitmap->bm_lock);
+ current_page_nr = from_page_nr;
+ addr = bm_map(bitmap, current_page_nr);
+ }
+ data_word = addr[word32_in_page(from_word_nr)];
- spin_lock_irqsave(&b->bm_lock, flags);
- if (BM_DONT_TEST & b->bm_flags)
- bm_print_lock_info(device);
- for (bitnr = s; bitnr <= e; bitnr++) {
- unsigned int idx = bm_bit_to_page_idx(b, bitnr);
- if (page_nr != idx) {
- page_nr = idx;
- if (p_addr)
- bm_unmap(p_addr);
- p_addr = bm_map_pidx(b, idx);
+ if (current_page_nr != to_page_nr) {
+ bm_unmap(bitmap, addr);
+ current_page_nr = to_page_nr;
+ addr = bm_map(bitmap, current_page_nr);
}
- if (expect(device, bitnr < b->bm_bits))
- c += (0 != test_bit_le(bitnr - (page_nr << (PAGE_SHIFT+3)), p_addr));
- else
- drbd_err(device, "bitnr=%lu bm_bits=%lu\n", bitnr, b->bm_bits);
+ to_word = &addr[word32_in_page(to_word_nr)];
+
+ if (op == BM_SLOT_OR) {
+ /* count only the bits that are new in the target */
+ data_word &= ~*to_word;
+ if (data_word)
+ bm_set_page_need_writeout(bitmap, current_page_nr);
+ *to_word |= data_word;
+ } else {
+ if (*to_word != data_word)
+ bm_set_page_need_writeout(bitmap, current_page_nr);
+ *to_word = data_word;
+ }
+ bitmap->bm_set[to_index] += hweight32(data_word);
}
- if (p_addr)
- bm_unmap(p_addr);
- spin_unlock_irqrestore(&b->bm_lock, flags);
- return c;
-}
+ bm_unmap(bitmap, addr);
+ if (op == BM_SLOT_COPY && bitmap->bm_set[to_index] == 0)
+ drbd_md_slot_emptied(device, to_index);
+ spin_unlock(&bitmap->bm_lock);
+ spin_unlock_irq(&bitmap->bm_all_slots_lock);
+}
-/* inherently racy...
- * return value may be already out-of-date when this function returns.
- * but the general usage is that this is only use during a cstate when bits are
- * only cleared, not set, and typically only care for the case when the return
- * value is zero, or we already "locked" this "bitmap extent" by other means.
- *
- * enr is bm-extent number, since we chose to name one sector (512 bytes)
- * worth of the bitmap a "bitmap extent".
- *
- * TODO
- * I think since we use it like a reference count, we should use the real
- * reference count of some bitmap extent element from some lru instead...
- *
- */
-int drbd_bm_e_weight(struct drbd_device *device, unsigned long enr)
+void drbd_bm_copy_slot(struct drbd_device *device, unsigned int from_index, unsigned int to_index)
{
- struct drbd_bitmap *b = device->bitmap;
- int count, s, e;
- unsigned long flags;
- unsigned long *p_addr, *bm;
-
- if (!expect(device, b))
- return 0;
- if (!expect(device, b->bm_pages))
- return 0;
+ __bm_slot_op(device, from_index, to_index, BM_SLOT_COPY);
+}
- spin_lock_irqsave(&b->bm_lock, flags);
- if (BM_DONT_TEST & b->bm_flags)
- bm_print_lock_info(device);
-
- s = S2W(enr);
- e = min((size_t)S2W(enr+1), b->bm_words);
- count = 0;
- if (s < b->bm_words) {
- int n = e-s;
- p_addr = bm_map_pidx(b, bm_word_to_page_idx(b, s));
- bm = p_addr + MLPP(s);
- count += bitmap_weight(bm, n * BITS_PER_LONG);
- bm_unmap(p_addr);
- } else {
- drbd_err(device, "start offset (%d) too large in drbd_bm_e_weight\n", s);
- }
- spin_unlock_irqrestore(&b->bm_lock, flags);
- return count;
+void drbd_bm_or_slot(struct drbd_device *device, unsigned int from_index, unsigned int to_index)
+{
+ __bm_slot_op(device, from_index, to_index, BM_SLOT_OR);
}
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 13/20] drbd: rework request processing for DRBD 9 multi-peer IO
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (11 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 12/20] drbd: rework activity log and bitmap for multi-peer replication Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 14/20] drbd: rework module core for DRBD 9 transport and multi-peer Christoph Böhmwalder
` (6 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Joel Colledge,
Andreas Gruenbacher
Restructure the request state model to support simultaneous replication
to multiple peers.
Split the single request state word into a local and per-peer-node
state, with a per-request spinlock protecting concurrent updates from
independent peer connections.
Move the transfer log from per-connection to per-resource scope, and
replace coarse request-path locking by fine-grained locks for the
interval tree, completion lists, and transfer log.
Track request lifetime with a three-level reference counting scheme
that separates upper-layer completion, bitmap/activity-log cleanup,
and peer-ack processing.
Destruction defers memory reclaim via call_rcu() to allow lock-free
transfer log traversal.
Replace blocking write conflict waits by an asynchronous conflict
resolution system that queues overlapping requests by type onto
per-device work lists for deferred submission.
A new peer acknowledgment subsystem batches cross-node write
confirmations using dagtag-based sequencing, ensuring all peers have
acknowledged a write window before its requests are reclaimed.
The write path independently decides replication or out-of-sync
notification for each peer, while the read path adds peer selection
with read balancing.
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_req.c | 3188 ++++++++++++++++++++++++---------
1 file changed, 2335 insertions(+), 853 deletions(-)
diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c
index 1412524e849c..8a403f019e96 100644
--- a/drivers/block/drbd/drbd_req.c
+++ b/drivers/block/drbd/drbd_req.c
@@ -13,7 +13,6 @@
#include "drbd_int.h"
#include "drbd_req.h"
-
static bool drbd_may_do_local_read(struct drbd_device *device, sector_t sector, int size);
static struct drbd_request *drbd_req_new(struct drbd_device *device, struct bio *bio_src)
@@ -23,159 +22,571 @@ static struct drbd_request *drbd_req_new(struct drbd_device *device, struct bio
req = mempool_alloc(&drbd_request_mempool, GFP_NOIO);
if (!req)
return NULL;
+
memset(req, 0, sizeof(*req));
- req->rq_state = (bio_data_dir(bio_src) == WRITE ? RQ_WRITE : 0)
- | (bio_op(bio_src) == REQ_OP_WRITE_ZEROES ? RQ_ZEROES : 0)
- | (bio_op(bio_src) == REQ_OP_DISCARD ? RQ_UNMAP : 0);
+ kref_get(&device->kref);
+
req->device = device;
req->master_bio = bio_src;
req->epoch = 0;
drbd_clear_interval(&req->i);
- req->i.sector = bio_src->bi_iter.bi_sector;
- req->i.size = bio_src->bi_iter.bi_size;
+ req->i.sector = bio_src->bi_iter.bi_sector;
+ req->i.size = bio_src->bi_iter.bi_size;
req->i.type = bio_data_dir(bio_src) == WRITE ? INTERVAL_LOCAL_WRITE : INTERVAL_LOCAL_READ;
INIT_LIST_HEAD(&req->tl_requests);
- INIT_LIST_HEAD(&req->w.list);
+ INIT_LIST_HEAD(&req->list);
INIT_LIST_HEAD(&req->req_pending_master_completion);
INIT_LIST_HEAD(&req->req_pending_local);
/* one reference to be put by __drbd_make_request */
atomic_set(&req->completion_ref, 1);
- /* one kref as long as completion_ref > 0 */
+ /* one reference as long as completion_ref > 0 */
+ refcount_set(&req->done_ref, 1);
+ /* one reference as long as done_ref > 0 */
+ refcount_set(&req->oos_send_ref, 1);
+ /* one kref as long as oos_send_ref > 0 */
kref_init(&req->kref);
+ spin_lock_init(&req->rq_lock);
+
+ req->local_rq_state = (bio_data_dir(bio_src) == WRITE ? RQ_WRITE : 0)
+ | (bio_op(bio_src) == REQ_OP_WRITE_ZEROES ? RQ_ZEROES : 0)
+ | (bio_op(bio_src) == REQ_OP_DISCARD ? RQ_UNMAP : 0);
+
return req;
}
+void drbd_reclaim_req(struct rcu_head *rp)
+{
+ struct drbd_request *req = container_of(rp, struct drbd_request, rcu);
+ kref_put(&req->device->kref, drbd_destroy_device);
+
+ mempool_free(req, &drbd_request_mempool);
+}
+
+static u64 peer_ack_mask(struct drbd_request *req)
+{
+ struct drbd_resource *resource = req->device->resource;
+ struct drbd_connection *connection;
+ u64 mask = 0;
+
+ spin_lock_irq(&req->rq_lock);
+ if (req->local_rq_state & RQ_LOCAL_OK)
+ mask |= NODE_MASK(resource->res_opts.node_id);
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ int node_id = connection->peer_node_id;
+
+ if (req->net_rq_state[node_id] & RQ_NET_OK)
+ mask |= NODE_MASK(node_id);
+ }
+ rcu_read_unlock();
+ spin_unlock_irq(&req->rq_lock);
+
+ return mask;
+}
+
+static void queue_peer_ack_send(struct drbd_resource *resource,
+ struct drbd_request *req, struct drbd_peer_ack *peer_ack)
+{
+ struct drbd_connection *connection;
+
+ lockdep_assert_held(&resource->state_rwlock);
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ unsigned int node_id = connection->peer_node_id;
+
+ if (connection->agreed_pro_version < 110 ||
+ connection->cstate[NOW] != C_CONNECTED) {
+ /* Do not jump back, as in process_peer_ack_list(). */
+ connection->last_peer_ack_dagtag_seen =
+ max(connection->last_peer_ack_dagtag_seen,
+ peer_ack->dagtag_sector);
+ continue;
+ }
+
+ if (req->net_rq_state[node_id] & RQ_NET_SENT)
+ peer_ack->pending_mask |= NODE_MASK(node_id);
+
+ peer_ack->queued_mask |= NODE_MASK(node_id);
+ queue_work(connection->ack_sender, &connection->peer_ack_work);
+ }
+ rcu_read_unlock();
+}
+
+void drbd_destroy_peer_ack_if_done(struct drbd_peer_ack *peer_ack)
+{
+ struct drbd_resource *resource = peer_ack->resource;
+
+ lockdep_assert_held(&resource->peer_ack_lock);
+
+ if (peer_ack->queued_mask)
+ return;
+
+ list_del(&peer_ack->list);
+ kfree(peer_ack);
+}
+
+int w_queue_peer_ack(struct drbd_work *w, int cancel)
+{
+ struct drbd_resource *resource =
+ container_of(w, struct drbd_resource, peer_ack_work);
+ LIST_HEAD(work_list);
+ struct drbd_request *req, *tmp;
+
+ spin_lock_irq(&resource->peer_ack_lock);
+ list_splice_init(&resource->peer_ack_req_list, &work_list);
+ spin_unlock_irq(&resource->peer_ack_lock);
+
+ list_for_each_entry_safe(req, tmp, &work_list, list) {
+ struct drbd_peer_ack *peer_ack =
+ kzalloc_obj(struct drbd_peer_ack);
+
+ peer_ack->resource = resource;
+ INIT_LIST_HEAD(&peer_ack->list);
+ peer_ack->mask = peer_ack_mask(req);
+ peer_ack->dagtag_sector = req->dagtag_sector;
+
+ /* lock against state changes for queue_peer_ack_send() */
+ read_lock_irq(&resource->state_rwlock);
+ spin_lock(&resource->peer_ack_lock);
+ list_add_tail(&peer_ack->list, &resource->peer_ack_list);
+ queue_peer_ack_send(resource, req, peer_ack);
+ drbd_destroy_peer_ack_if_done(peer_ack);
+ spin_unlock(&resource->peer_ack_lock);
+ read_unlock_irq(&resource->state_rwlock);
+
+ kref_put(&req->kref, drbd_req_destroy);
+ }
+ return 0;
+}
+
+void drbd_queue_peer_ack(struct drbd_resource *resource, struct drbd_request *req)
+{
+ lockdep_assert_held(&resource->peer_ack_lock);
+
+ list_add_tail(&req->list, &resource->peer_ack_req_list);
+ drbd_queue_work_if_unqueued(&resource->work, &resource->peer_ack_work);
+}
+
+static bool peer_ack_differs(struct drbd_request *req1, struct drbd_request *req2)
+{
+ unsigned int max_node_id = req1->device->resource->max_node_id;
+ unsigned int node_id;
+
+ for (node_id = 0; node_id <= max_node_id; node_id++)
+ if ((req1->net_rq_state[node_id] & RQ_NET_OK) !=
+ (req2->net_rq_state[node_id] & RQ_NET_OK))
+ return true;
+ return false;
+}
+
+static bool peer_ack_window_full(struct drbd_request *req)
+{
+ struct drbd_resource *resource = req->device->resource;
+ u32 peer_ack_window = resource->res_opts.peer_ack_window;
+ u64 last_dagtag = resource->last_peer_acked_dagtag + peer_ack_window;
+
+ return dagtag_newer_eq(req->dagtag_sector, last_dagtag);
+}
+
static void drbd_remove_request_interval(struct rb_root *root,
struct drbd_request *req)
{
struct drbd_device *device = req->device;
- struct drbd_interval *i = &req->i;
-
- drbd_remove_interval(root, i);
+ unsigned long flags;
- /* Wake up any processes waiting for this request to complete. */
- if (test_bit(INTERVAL_WAITING, &i->flags))
- wake_up(&device->misc_wait);
+ spin_lock_irqsave(&device->interval_lock, flags);
+ drbd_remove_interval(root, &req->i);
+ spin_unlock_irqrestore(&device->interval_lock, flags);
}
void drbd_req_destroy(struct kref *kref)
{
struct drbd_request *req = container_of(kref, struct drbd_request, kref);
+
+ call_rcu(&req->rcu, drbd_reclaim_req);
+}
+
+static void drbd_req_done(struct drbd_request *req)
+{
struct drbd_device *device = req->device;
- const unsigned s = req->rq_state;
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
+ unsigned int s = req->local_rq_state;
+ unsigned long modified_mask = 0;
+
+ lockdep_assert_held(&resource->state_rwlock);
+ lockdep_assert_irqs_disabled();
+
+#ifdef CONFIG_DRBD_TIMING_STATS
+ if (s & RQ_WRITE && req->i.size != 0) {
+ spin_lock(&device->timing_lock); /* local irq already disabled */
+ device->reqs++;
+ ktime_aggregate(device, req, in_actlog_kt);
+ ktime_aggregate(device, req, pre_submit_kt);
+ for_each_peer_device(peer_device, device) {
+ int node_id = peer_device->node_id;
+ unsigned int ns = req->net_rq_state[node_id];
+
+ if (!(ns & RQ_NET_MASK))
+ continue;
+ ktime_aggregate_pd(peer_device, node_id, req, pre_send_kt);
+ ktime_aggregate_pd(peer_device, node_id, req, acked_kt);
+ ktime_aggregate_pd(peer_device, node_id, req, net_done_kt);
+ }
+ spin_unlock(&device->timing_lock);
+ }
+#endif
- if ((req->master_bio && !(s & RQ_POSTPONED)) ||
- atomic_read(&req->completion_ref) ||
- (s & RQ_LOCAL_PENDING) ||
- ((s & RQ_NET_MASK) && !(s & RQ_NET_DONE))) {
- drbd_err(device, "drbd_req_destroy: Logic BUG rq_state = 0x%x, completion_ref = %d\n",
- s, atomic_read(&req->completion_ref));
+ /* paranoia */
+ for_each_peer_device(peer_device, device) {
+ unsigned int ns = req->net_rq_state[peer_device->node_id];
+
+ if (!(ns & RQ_NET_MASK))
+ continue;
+ if (ns & RQ_NET_DONE)
+ continue;
+
+ drbd_err(device,
+ "%s: Logic BUG rq_state: (0:%x, %d:%x), completion_ref = %d\n",
+ __func__, s, peer_device->node_id, ns, atomic_read(&req->completion_ref));
return;
}
- /* If called from mod_rq_state (expected normal case) or
- * drbd_send_and_submit (the less likely normal path), this holds the
- * req_lock, and req->tl_requests will typicaly be on ->transfer_log,
- * though it may be still empty (never added to the transfer log).
- *
- * If called from do_retry(), we do NOT hold the req_lock, but we are
- * still allowed to unconditionally list_del(&req->tl_requests),
- * because it will be on a local on-stack list only. */
- list_del_init(&req->tl_requests);
+ /* more paranoia */
+ if ((req->master_bio && !(s & RQ_POSTPONED)) ||
+ atomic_read(&req->completion_ref) || (s & RQ_LOCAL_PENDING)) {
+ drbd_err(device, "%s: Logic BUG master_bio:%d rq_state: %x, completion_ref = %d\n",
+ __func__, !!req->master_bio, s, atomic_read(&req->completion_ref));
+ return;
+ }
/* finally remove the request from the conflict detection
* respective block_id verification interval tree. */
- if (!drbd_interval_empty(&req->i)) {
- struct rb_root *root;
+ if (s & RQ_WRITE && !drbd_interval_empty(&req->i))
+ drbd_remove_request_interval(&device->requests, req);
+
+ /* There is a special case:
+ * we may notice late that IO was suspended, or that we may not
+ * write at all, and postpone, schedule for retry, or fail a
+ * write, before it even was submitted or sent. No peer lacks
+ * anything for such a write, so we do not want to touch the
+ * bitmap at all. Such a write left drbd_send_and_submit()
+ * before it got a dagtag, which is never 0 for a write with
+ * data.
+ */
+ if ((s & RQ_WRITE) && req->dagtag_sector &&
+ req->i.size && get_ldev_if_state(device, D_DETACHING)) {
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ unsigned long bits = -1, mask = -1;
+ int node_id, max_node_id = device->resource->max_node_id;
+
+ for (node_id = 0; node_id <= max_node_id; node_id++) {
+ unsigned int net_rq_state;
+
+ net_rq_state = req->net_rq_state[node_id];
+ if (net_rq_state & RQ_NET_OK) {
+ int bitmap_index = peer_md[node_id].bitmap_index;
+
+ if (bitmap_index == -1)
+ continue;
+
+ if (net_rq_state & RQ_NET_SIS)
+ clear_bit(bitmap_index, &bits);
+ else
+ clear_bit(bitmap_index, &mask);
+ }
+ }
+ modified_mask = drbd_set_sync(device, req->i.sector, req->i.size,
+ bits, mask);
+ put_ldev(device);
+ }
- if (s & RQ_WRITE)
- root = &device->write_requests;
- else
- root = &device->read_requests;
- drbd_remove_request_interval(root, req);
- } else if (s & (RQ_NET_MASK & ~RQ_NET_DONE) && req->i.size != 0)
- drbd_err(device, "drbd_req_destroy: Logic BUG: interval empty, but: rq_state=0x%x, sect=%llu, size=%u\n",
- s, (unsigned long long)req->i.sector, req->i.size);
-
- /* if it was a write, we may have to set the corresponding
- * bit(s) out-of-sync first. If it had a local part, we need to
- * release the reference to the activity log. */
if (s & RQ_WRITE) {
- /* Set out-of-sync unless both OK flags are set
- * (local only or remote failed).
- * Other places where we set out-of-sync:
- * READ with local io-error */
-
- /* There is a special case:
- * we may notice late that IO was suspended,
- * and postpone, or schedule for retry, a write,
- * before it even was submitted or sent.
- * In that case we do not want to touch the bitmap at all.
- */
- struct drbd_peer_device *peer_device = first_peer_device(device);
- if ((s & (RQ_POSTPONED|RQ_LOCAL_MASK|RQ_NET_MASK)) != RQ_POSTPONED) {
- if (!(s & RQ_NET_OK) || !(s & RQ_LOCAL_OK))
- drbd_set_out_of_sync(peer_device, req->i.sector, req->i.size);
+ for_each_peer_device(peer_device, device) {
+ if (!(req->net_rq_state[peer_device->node_id] & RQ_NET_PENDING_OOS))
+ continue;
- if ((s & RQ_NET_OK) && (s & RQ_LOCAL_OK) && (s & RQ_NET_SIS))
- drbd_set_in_sync(peer_device, req->i.sector, req->i.size);
+ if (s & RQ_POSTPONED) {
+ drbd_err(device, "%s: Logic BUG RQ_NET_PENDING_OOS|RQ_POSTPONED\n",
+ __func__);
+ continue;
+ }
+
+ /*
+ * As an optimization, we only send out-of-sync if we
+ * set some bit for this peer. If we are not
+ * replicating to this peer and the same block(s) are
+ * overwritten several times, the peer only needs to be
+ * informed of the first change.
+ */
+ if (peer_device->bitmap_index != -1 &&
+ test_bit(peer_device->bitmap_index, &modified_mask))
+ _req_mod(req, READY_FOR_NET, peer_device);
+ else
+ _req_mod(req, SKIP_OOS, peer_device);
+
+ wake_up(&peer_device->connection->sender_work.q_wait);
}
+ }
+
+ /* one might be tempted to move the drbd_al_complete_io
+ * to the local io completion callback drbd_request_endio.
+ * but, if this was a mirror write, we may only
+ * drbd_al_complete_io after this is RQ_NET_DONE,
+ * otherwise the extent could be dropped from the al
+ * before it has actually been written on the peer.
+ * if we crash before our peer knows about the request,
+ * but after the extent has been dropped from the al,
+ * we would forget to resync the corresponding extent.
+ */
+ if (s & RQ_IN_ACT_LOG) {
+ if (get_ldev_if_state(device, D_DETACHING)) {
+ if (drbd_al_complete_io(device, &req->i))
+ set_bit(INTERVAL_AL_EXTENT_LAST, &req->i.flags);
+ put_ldev(device);
+ } else if (drbd_device_ratelimit(device, BACKEND)) {
+ drbd_warn(device, "Should have called drbd_al_complete_io(, %llu, %u), but my Disk seems to have failed :(\n",
+ (unsigned long long) req->i.sector, req->i.size);
- /* one might be tempted to move the drbd_al_complete_io
- * to the local io completion callback drbd_request_endio.
- * but, if this was a mirror write, we may only
- * drbd_al_complete_io after this is RQ_NET_DONE,
- * otherwise the extent could be dropped from the al
- * before it has actually been written on the peer.
- * if we crash before our peer knows about the request,
- * but after the extent has been dropped from the al,
- * we would forget to resync the corresponding extent.
- */
- if (s & RQ_IN_ACT_LOG) {
- if (get_ldev_if_state(device, D_FAILED)) {
- drbd_al_complete_io(device, &req->i);
- put_ldev(device);
- } else if (drbd_device_ratelimit(device, BACKEND)) {
- drbd_warn(device, "Should have called drbd_al_complete_io(, %llu, %u), "
- "but my Disk seems to have failed :(\n",
- (unsigned long long) req->i.sector, req->i.size);
- }
}
}
+}
- mempool_free(req, &drbd_request_mempool);
+static void drbd_req_oos_sent(struct drbd_request *req)
+{
+ struct drbd_device *device = req->device;
+ struct drbd_resource *resource = device->resource;
+ unsigned int s = req->local_rq_state;
+
+ lockdep_assert_held(&resource->state_rwlock);
+ lockdep_assert_irqs_disabled();
+
+ /* A write without a dagtag never entered the change stream and has
+ * no position to acknowledge. Publishing its zero dagtag moves the
+ * peer-ack watermarks backwards, and a peer waiting for a higher one --
+ * a sync target waiting for P_FLUSH_REQUESTS_ACK -- then waits forever.
+ */
+ if (s & RQ_WRITE && req->i.size && req->dagtag_sector) {
+ struct drbd_request *peer_ack_req;
+
+ spin_lock(&resource->peer_ack_lock); /* local irq already disabled */
+ peer_ack_req = resource->peer_ack_req;
+ if (peer_ack_req) {
+ bool al_extent_last = test_bit(INTERVAL_AL_EXTENT_LAST, &req->i.flags);
+
+ if (peer_ack_differs(req, peer_ack_req) ||
+ (al_extent_last && atomic_read(&device->ap_actlog_cnt)) ||
+ peer_ack_window_full(req)) {
+ drbd_queue_peer_ack(resource, peer_ack_req);
+ peer_ack_req = NULL;
+ } else
+ kref_put(&peer_ack_req->kref, drbd_req_destroy);
+ }
+ resource->peer_ack_req = req;
+
+ if (!peer_ack_req)
+ resource->last_peer_acked_dagtag = req->dagtag_sector;
+ spin_unlock(&resource->peer_ack_lock);
+
+ mod_timer(&resource->peer_ack_timer,
+ jiffies + resource->res_opts.peer_ack_delay * HZ / 1000);
+ } else
+ kref_put(&req->kref, drbd_req_destroy);
}
-static void wake_all_senders(struct drbd_connection *connection)
+static void wake_all_senders(struct drbd_resource *resource)
{
- wake_up(&connection->sender_work.q_wait);
+ struct drbd_connection *connection;
+ /* We need make sure any update is visible before we wake up the
+ * threads that may check the values in their wait_event() condition.
+ * Do we need smp_mb here? Or rather switch to atomic_t? */
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource)
+ wake_up(&connection->sender_work.q_wait);
+ rcu_read_unlock();
}
-/* must hold resource->req_lock */
-void start_new_tl_epoch(struct drbd_connection *connection)
+bool start_new_tl_epoch(struct drbd_resource *resource)
{
+ unsigned long flags;
+ bool new_epoch_started;
+
+ spin_lock_irqsave(&resource->current_tle_lock, flags);
/* no point closing an epoch, if it is empty, anyways. */
- if (connection->current_tle_writes == 0)
- return;
+ if (resource->current_tle_writes == 0) {
+ new_epoch_started = false;
+ } else {
+ resource->current_tle_writes = 0;
+ atomic_inc(&resource->current_tle_nr);
+ wake_all_senders(resource);
+ new_epoch_started = true;
+ }
+ spin_unlock_irqrestore(&resource->current_tle_lock, flags);
- connection->current_tle_writes = 0;
- atomic_inc(&connection->current_tle_nr);
- wake_all_senders(connection);
+ return new_epoch_started;
}
void complete_master_bio(struct drbd_device *device,
struct bio_and_error *m)
{
+ int rw = bio_data_dir(m->bio);
+
if (unlikely(m->error))
m->bio->bi_status = errno_to_blk_status(m->error);
bio_endio(m->bio);
- dec_ap_bio(device);
+ dec_ap_bio(device, rw);
}
+static void queue_conflicting_resync_write(
+ struct conflict_worker *submit_conflict, struct drbd_interval *i)
+{
+ struct drbd_peer_request *peer_req = container_of(i, struct drbd_peer_request, i);
+
+ list_add_tail(&peer_req->w.list, &submit_conflict->resync_writes);
+}
+
+static void queue_conflicting_resync_read(
+ struct conflict_worker *submit_conflict, struct drbd_interval *i)
+{
+ struct drbd_peer_request *peer_req = container_of(i, struct drbd_peer_request, i);
+
+ list_add_tail(&peer_req->w.list, &submit_conflict->resync_reads);
+}
+
+static void queue_conflicting_write(
+ struct conflict_worker *submit_conflict, struct drbd_interval *i)
+{
+ struct drbd_request *req = container_of(i, struct drbd_request, i);
+
+ list_add_tail(&req->list, &submit_conflict->writes);
+}
+
+static void queue_conflicting_peer_write(
+ struct conflict_worker *submit_conflict, struct drbd_interval *i)
+{
+ struct drbd_peer_request *peer_req = container_of(i, struct drbd_peer_request, i);
+
+ list_add_tail(&peer_req->w.list, &submit_conflict->peer_writes);
+}
+
+/* Queue any conflicting requests in this interval to be submitted. */
+void drbd_release_conflicts(struct drbd_device *device, struct drbd_interval *release_interval)
+{
+ struct conflict_worker *submit_conflict = &device->submit_conflict;
+ struct drbd_interval *i;
+ bool any_queued = false;
+
+ lockdep_assert_held(&device->interval_lock);
+
+ drbd_for_each_overlap(i, &device->requests, release_interval->sector, release_interval->size) {
+ if (test_bit(INTERVAL_SUBMITTED, &i->flags))
+ continue;
+
+ /* If we are waiting for a reply from the peer, then there is
+ * no need to process the conflict. */
+ if (test_bit(INTERVAL_READY_TO_SEND, &i->flags) &&
+ !test_bit(INTERVAL_RECEIVED, &i->flags))
+ continue;
+
+ dynamic_drbd_dbg(device,
+ "%s %s request at %llus+%u after conflict with %llus+%u\n",
+ test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags) ? "Already queued" : "Queue",
+ drbd_interval_type_str(i),
+ (unsigned long long) i->sector, i->size,
+ (unsigned long long) release_interval->sector, release_interval->size);
+
+ if (test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags))
+ continue;
+
+ /* Verify requests never wait for conflicting intervals. If
+ * there are no conflicts, they are marked direcly as
+ * submitted. Hence we should not see any here. */
+ if (unlikely(drbd_interval_is_verify(i))) {
+ if (drbd_ratelimit())
+ drbd_err(device, "Found verify request that was not yet submitted\n");
+ continue;
+ }
+
+ set_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags);
+
+ spin_lock(&submit_conflict->lock);
+ /* Queue the request regardless of whether other conflicts
+ * remain. The conflict submitter will only actually submit the
+ * request if there are no conflicts. */
+ switch (i->type) {
+ case INTERVAL_LOCAL_WRITE:
+ queue_conflicting_write(submit_conflict, i);
+ break;
+ case INTERVAL_PEER_WRITE:
+ queue_conflicting_peer_write(submit_conflict, i);
+ break;
+ case INTERVAL_RESYNC_WRITE:
+ queue_conflicting_resync_write(submit_conflict, i);
+ break;
+ case INTERVAL_RESYNC_READ:
+ queue_conflicting_resync_read(submit_conflict, i);
+ break;
+ default:
+ BUG();
+ }
+ spin_unlock(&submit_conflict->lock);
+
+ any_queued = true;
+ }
+
+ if (any_queued)
+ queue_work(submit_conflict->wq, &submit_conflict->worker);
+}
+
+void drbd_put_ref_tl_walk(struct drbd_request *req, int done_put, int oos_send_put)
+{
+ struct drbd_resource *resource = req->device->resource;
+
+ lockdep_assert_held(&resource->state_rwlock);
+
+ while (req) {
+ struct drbd_request *next_write;
+ bool done = false;
+ bool oos_sent = false;
+
+ if (done_put && refcount_sub_and_test(done_put, &req->done_ref)) {
+ done = true;
+ drbd_req_done(req);
+ oos_send_put++;
+ }
+
+ if (oos_send_put && refcount_sub_and_test(oos_send_put, &req->oos_send_ref))
+ oos_sent = true;
+
+ if (!done && !oos_sent)
+ break;
+
+ spin_lock(&resource->tl_update_lock); /* local irq already disabled */
+ next_write = req->next_write;
+ if (oos_sent) {
+ list_del_rcu(&req->tl_requests);
+ if (resource->tl_previous_write == req)
+ resource->tl_previous_write = NULL;
+ } else if (done) {
+ set_bit(INTERVAL_DONE, &req->i.flags);
+ }
+ spin_unlock(&resource->tl_update_lock);
+
+ if (oos_sent)
+ /* potentially destroy */
+ drbd_req_oos_sent(req);
+
+ req = next_write;
+ done_put = done ? 1 : 0;
+ oos_send_put = oos_sent ? 1 : 0;
+ }
+}
/* Helper for __req_mod().
* Set m->bio to the master bio, if it is fit to be completed,
@@ -186,30 +597,11 @@ void complete_master_bio(struct drbd_device *device,
static
void drbd_req_complete(struct drbd_request *req, struct bio_and_error *m)
{
- const unsigned s = req->rq_state;
+ const unsigned int s = req->local_rq_state;
struct drbd_device *device = req->device;
- int error, ok;
-
- /* we must not complete the master bio, while it is
- * still being processed by _drbd_send_zc_bio (drbd_send_dblock)
- * not yet acknowledged by the peer
- * not yet completed by the local io subsystem
- * these flags may get cleared in any order by
- * the worker,
- * the receiver,
- * the bio_endio completion callbacks.
- */
- if ((s & RQ_LOCAL_PENDING && !(s & RQ_LOCAL_ABORTED)) ||
- (s & RQ_NET_QUEUED) || (s & RQ_NET_PENDING) ||
- (s & RQ_COMPLETION_SUSP)) {
- drbd_err(device, "drbd_req_complete: Logic BUG rq_state = 0x%x\n", s);
- return;
- }
-
- if (!req->master_bio) {
- drbd_err(device, "drbd_req_complete: Logic BUG, master_bio == NULL!\n");
- return;
- }
+ struct drbd_peer_device *peer_device;
+ unsigned long flags;
+ int error, ok = 0;
/*
* figure out whether to report success or failure.
@@ -224,69 +616,136 @@ void drbd_req_complete(struct drbd_request *req, struct bio_and_error *m)
* local completion error, if any, has been stored as ERR_PTR
* in private_bio within drbd_request_endio.
*/
- ok = (s & RQ_LOCAL_OK) || (s & RQ_NET_OK);
+ if (s & RQ_LOCAL_OK)
+ ++ok;
error = PTR_ERR(req->private_bio);
+ for_each_peer_device(peer_device, device) {
+ unsigned int ns = req->net_rq_state[peer_device->node_id];
+ /* any net ok ok local ok is good enough to complete this bio as OK */
+ if (ns & RQ_NET_OK)
+ ++ok;
+ /* paranoia */
+ /* we must not complete the master bio, while it is
+ * still being processed by _drbd_send_zc_bio (drbd_send_dblock),
+ * respectively still needed for the second drbd_csum_bio() there.
+ * not yet acknowledged by the peer
+ * not yet completed by the local io subsystem
+ * these flags may get cleared in any order by
+ * the worker,
+ * the sender,
+ * the receiver,
+ * the bio_endio completion callbacks.
+ */
+ if (!(ns & RQ_NET_MASK))
+ continue;
+ if (ns & RQ_NET_PENDING_OOS)
+ continue;
+ if (!(ns & (RQ_NET_PENDING|RQ_NET_QUEUED)))
+ continue;
+
+ drbd_err(device,
+ "drbd_req_complete: Logic BUG rq_state: (0:%x, %d:%x), completion_ref = %d\n",
+ s, peer_device->node_id, ns, atomic_read(&req->completion_ref));
+ return;
+ }
+
+ /* more paranoia */
+ if (atomic_read(&req->completion_ref) ||
+ ((s & RQ_LOCAL_PENDING) && !(s & RQ_LOCAL_ABORTED))) {
+ drbd_err(device, "drbd_req_complete: Logic BUG rq_state: %x, completion_ref = %d\n",
+ s, atomic_read(&req->completion_ref));
+ return;
+ }
+
+ if (!req->master_bio) {
+ drbd_err(device, "drbd_req_complete: Logic BUG, master_bio == NULL!\n");
+ return;
+ }
+
/* Before we can signal completion to the upper layers,
* we may need to close the current transfer log epoch.
- * We are within the request lock, so we can simply compare
- * the request epoch number with the current transfer log
- * epoch number. If they match, increase the current_tle_nr,
- * and reset the transfer log epoch write_cnt.
+ * We simply compare the request epoch number with the current
+ * transfer log epoch number.
+ * With very specific timing, this may cause unnecessary barriers
+ * to be sent, but that is harmless.
+ *
+ * There is no need to close the transfer log epoch for empty flushes.
+ * The completion of the previous requests had the required effect on
+ * the peers already.
*/
- if (op_is_write(bio_op(req->master_bio)) &&
- req->epoch == atomic_read(&first_peer_device(device)->connection->current_tle_nr))
- start_new_tl_epoch(first_peer_device(device)->connection);
+ if (bio_data_dir(req->master_bio) == WRITE &&
+ likely(req->i.size != 0) &&
+ req->epoch == atomic_read(&device->resource->current_tle_nr))
+ start_new_tl_epoch(device->resource);
/* Update disk stats */
bio_end_io_acct(req->master_bio, req->start_jif);
- /* If READ failed,
- * have it be pushed back to the retry work queue,
- * so it will re-enter __drbd_make_request(),
- * and be re-assigned to a suitable local or remote path,
- * or failed if we do not have access to good data anymore.
- *
- * Unless it was failed early by __drbd_make_request(),
- * because no path was available, in which case
- * it was not even added to the transfer_log.
- *
- * read-ahead may fail, and will not be retried.
- *
- * WRITE should have used all available paths already.
- */
- if (!ok &&
- bio_op(req->master_bio) == REQ_OP_READ &&
- !(req->master_bio->bi_opf & REQ_RAHEAD) &&
- !list_empty(&req->tl_requests))
- req->rq_state |= RQ_POSTPONED;
-
- if (!(req->rq_state & RQ_POSTPONED)) {
- m->error = ok ? 0 : (error ?: -EIO);
+ if (device->cached_err_io) {
+ ok = 0;
+ req->local_rq_state &= ~RQ_POSTPONED;
+ } else if (!ok &&
+ bio_op(req->master_bio) == REQ_OP_READ &&
+ !(req->master_bio->bi_opf & REQ_RAHEAD) &&
+ !list_empty(&req->tl_requests)) {
+ /* If READ failed,
+ * have it be pushed back to the retry work queue,
+ * so it will re-enter __drbd_make_request(),
+ * and be re-assigned to a suitable local or remote path,
+ * or failed if we do not have access to good data anymore.
+ *
+ * Unless it was failed early by __drbd_make_request(),
+ * because no path was available, in which case
+ * it was not even added to the transfer_log.
+ *
+ * read-ahead may fail, and will not be retried.
+ *
+ * WRITE should have used all available paths already.
+ */
+ req->local_rq_state |= RQ_POSTPONED;
+ }
+
+ if (!(req->local_rq_state & RQ_POSTPONED)) {
+ struct drbd_resource *resource = device->resource;
+ bool quorum =
+ resource->res_opts.on_no_quorum == ONQ_IO_ERROR ?
+ resource->cached_all_devices_have_quorum : true;
+
+ m->error = ok && quorum ? 0 : (error ?: -EIO);
+ if (!m->error && (req->local_rq_state & RQ_WRITE))
+ req->local_rq_state |= RQ_COMPLETED_OK;
m->bio = req->master_bio;
req->master_bio = NULL;
- /* We leave it in the tree, to be able to verify later
- * write-acks in protocol != C during resync.
- * But we mark it as "complete", so it won't be counted as
- * conflict in a multi-primary setup. */
- set_bit(INTERVAL_COMPLETED, &req->i.flags);
+
+ if (req->local_rq_state & RQ_WRITE) {
+ spin_lock_irqsave(&device->interval_lock, flags);
+ /* We leave it in the tree, to be able to verify later
+ * write-acks in protocol != C during resync.
+ * But we mark it as "complete", so it won't be counted as
+ * conflict in a multi-primary setup.
+ */
+ set_bit(INTERVAL_COMPLETED, &req->i.flags);
+ drbd_release_conflicts(device, &req->i);
+ spin_unlock_irqrestore(&device->interval_lock, flags);
+ }
}
- if (test_bit(INTERVAL_WAITING, &req->i.flags))
- wake_up(&device->misc_wait);
+ if (!(req->local_rq_state & RQ_WRITE))
+ drbd_remove_request_interval(&device->read_requests, req);
/* Either we are about to complete to upper layers,
* or we will restart this request.
* In either case, the request object will be destroyed soon,
* so better remove it from all lists. */
+ spin_lock_irqsave(&device->pending_completion_lock, flags);
list_del_init(&req->req_pending_master_completion);
+ spin_unlock_irqrestore(&device->pending_completion_lock, flags);
}
-/* still holds resource->req_lock */
static void drbd_req_put_completion_ref(struct drbd_request *req, struct bio_and_error *m, int put)
{
- struct drbd_device *device = req->device;
- D_ASSERT(device, m || (req->rq_state & RQ_POSTPONED));
+ D_ASSERT(req->device, m || (req->local_rq_state & RQ_POSTPONED));
if (!put)
return;
@@ -298,217 +757,356 @@ static void drbd_req_put_completion_ref(struct drbd_request *req, struct bio_and
/* local completion may still come in later,
* we need to keep the req object around. */
- if (req->rq_state & RQ_LOCAL_ABORTED)
+ if (req->local_rq_state & RQ_LOCAL_ABORTED)
return;
- if (req->rq_state & RQ_POSTPONED) {
+ if (req->local_rq_state & RQ_POSTPONED) {
/* don't destroy the req object just yet,
* but queue it for retry */
drbd_restart_request(req);
return;
}
- kref_put(&req->kref, drbd_req_destroy);
+ drbd_put_ref_tl_walk(req, 1, 0);
}
-static void set_if_null_req_next(struct drbd_peer_device *peer_device, struct drbd_request *req)
+void drbd_set_pending_out_of_sync(struct drbd_peer_device *peer_device)
{
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- if (!connection)
- return;
- if (connection->req_next == NULL)
- connection->req_next = req;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_resource *resource = device->resource;
+ const int node_id = peer_device->node_id;
+ struct drbd_request *req;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(req, &resource->transfer_log, tl_requests) {
+ unsigned int local_rq_state, net_rq_state;
+
+ /*
+ * This is similar to the bitmap modification performed in
+ * drbd_req_done(), but simplified for this special case.
+ */
+
+ spin_lock_irq(&req->rq_lock);
+ local_rq_state = req->local_rq_state;
+ net_rq_state = req->net_rq_state[node_id];
+ spin_unlock_irq(&req->rq_lock);
+
+ if (!(local_rq_state & RQ_WRITE))
+ continue;
+
+ if ((local_rq_state & (RQ_POSTPONED|RQ_LOCAL_MASK)) == RQ_POSTPONED)
+ continue;
+
+ if (!req->i.size)
+ continue;
+
+ if (net_rq_state & RQ_NET_OK)
+ continue;
+
+ drbd_set_out_of_sync(peer_device, req->i.sector, req->i.size);
+ }
+ rcu_read_unlock();
}
-static void advance_conn_req_next(struct drbd_peer_device *peer_device, struct drbd_request *req)
+static void advance_conn_req_next(struct drbd_connection *connection, struct drbd_request *req)
{
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- struct drbd_request *iter = req;
- if (!connection)
+ struct drbd_request *found_req = NULL;
+ /* Only the sender thread comes here. No other caller context of req_mod() ever arrives here */
+ if (connection->todo.req_next != req)
return;
- if (connection->req_next != req)
- return;
-
- req = NULL;
- list_for_each_entry_continue(iter, &connection->transfer_log, tl_requests) {
- const unsigned int s = iter->rq_state;
+ rcu_read_lock();
+ list_for_each_entry_continue_rcu(req, &connection->resource->transfer_log, tl_requests) {
+ const unsigned int s = req->net_rq_state[connection->peer_node_id];
- if (s & RQ_NET_QUEUED) {
- req = iter;
+ if (likely(s & RQ_NET_QUEUED)) {
+ found_req = req;
break;
}
}
- connection->req_next = req;
+ rcu_read_unlock();
+ connection->todo.req_next = found_req;
}
-static void set_if_null_req_ack_pending(struct drbd_peer_device *peer_device, struct drbd_request *req)
+/**
+ * set_cache_ptr_if_null() - Set caching pointer to given request if not currently set.
+ * @connection: DRBD connection to operate on.
+ * @cache_ptr: Pointer to set.
+ * @req: Request to potentially set the pointer to.
+ *
+ * The caching pointer system is designed to track the oldest request in the
+ * transfer log fulfilling some condition. In particular, a combination of
+ * flags towards a given peer. This condition must guarantee that the request
+ * will not be destroyed.
+ *
+ * This system is implemented by set_cache_ptr_if_null() and
+ * advance_cache_ptr(). A request must be in the transfer log and fulfil the
+ * condition before set_cache_ptr_if_null() is called. If
+ * set_cache_ptr_if_null() is called before this request is in the transfer log
+ * or before it fulfils the condition, the pointer may be advanced past this
+ * request, or unset, which also has the effect of skipping the request.
+ *
+ * Once the condition is no longer fulfilled for a request, advance_cache_ptr()
+ * must be called. If the caching pointer currently points to this request,
+ * this will advance it to the next request fulfilling the condition.
+ *
+ * set_cache_ptr_if_null() may be called concurrently with advance_cache_ptr().
+ */
+static void set_cache_ptr_if_null(struct drbd_connection *connection,
+ struct drbd_request **cache_ptr, struct drbd_request *req)
{
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- if (!connection)
- return;
- if (connection->req_ack_pending == NULL)
- connection->req_ack_pending = req;
+ spin_lock(&connection->advance_cache_ptr_lock); /* local IRQ already disabled */
+ if (*cache_ptr == NULL) {
+ smp_wmb(); /* make list_add_tail_rcu(req, transfer_log) visible before cache_ptr */
+ WRITE_ONCE(*cache_ptr, req);
+ }
+ spin_unlock(&connection->advance_cache_ptr_lock);
}
-static void advance_conn_req_ack_pending(struct drbd_peer_device *peer_device, struct drbd_request *req)
+/* See set_cache_ptr_if_null(). */
+static void advance_cache_ptr(struct drbd_connection *connection,
+ struct drbd_request __rcu **cache_ptr, struct drbd_request *req,
+ unsigned int is_set, unsigned int is_clear)
{
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- struct drbd_request *iter = req;
- if (!connection)
- return;
- if (connection->req_ack_pending != req)
- return;
+ struct drbd_request *old_req;
+ struct drbd_request *found_req = NULL;
+
+ /*
+ * Prevent concurrent updates of the same caching pointer. Otherwise if
+ * this function is called concurrently for a given caching pointer,
+ * the call for the older request may advance the pointer to the newer
+ * request, although the newer request has concurrently been modified
+ * such that it no longer fulfils the condition.
+ */
+ spin_lock(&connection->advance_cache_ptr_lock); /* local IRQ already disabled */
- req = NULL;
- list_for_each_entry_continue(iter, &connection->transfer_log, tl_requests) {
- const unsigned int s = iter->rq_state;
+ rcu_read_lock();
+ old_req = rcu_dereference(*cache_ptr);
+ if (old_req != req) {
+ rcu_read_unlock();
+ spin_unlock(&connection->advance_cache_ptr_lock);
+ return;
+ }
+ list_for_each_entry_continue_rcu(req, &connection->resource->transfer_log, tl_requests) {
+ const unsigned int s = READ_ONCE(req->net_rq_state[connection->peer_node_id]);
- if ((s & RQ_NET_SENT) && (s & RQ_NET_PENDING)) {
- req = iter;
+ if (!(s & RQ_NET_MASK))
+ continue;
+ if (((s & is_set) == is_set) && !(s & is_clear)) {
+ found_req = req;
break;
}
}
- connection->req_ack_pending = req;
+
+ WRITE_ONCE(*cache_ptr, found_req);
+ rcu_read_unlock();
+
+ spin_unlock(&connection->advance_cache_ptr_lock);
}
-static void set_if_null_req_not_net_done(struct drbd_peer_device *peer_device, struct drbd_request *req)
+/* for wsame, discard, and zero-out requests, the payload (amount of data we
+ * need to send) is much smaller than the number of storage sectors affected */
+static unsigned int req_payload_sectors(struct drbd_request *req)
{
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- if (!connection)
- return;
- if (connection->req_not_net_done == NULL)
- connection->req_not_net_done = req;
+ /* actually: physical_block_size,
+ * but lets just hardcode 4k in sectors: */
+ if (unlikely(req->local_rq_state & RQ_WSAME))
+ return 8;
+ /* really only a few bytes, but let's pretend one sector */
+ if (unlikely(req->local_rq_state & (RQ_UNMAP|RQ_ZEROES)))
+ return 1;
+ /* other have all the data as payload on the wire */
+ return req->i.size >> 9;
}
-static void advance_conn_req_not_net_done(struct drbd_peer_device *peer_device, struct drbd_request *req)
+static bool drbd_sender_needs_master_bio(unsigned int net_rq_state)
{
- struct drbd_connection *connection = peer_device ? peer_device->connection : NULL;
- struct drbd_request *iter = req;
- if (!connection)
- return;
- if (connection->req_not_net_done != req)
- return;
-
- req = NULL;
- list_for_each_entry_continue(iter, &connection->transfer_log, tl_requests) {
- const unsigned int s = iter->rq_state;
-
- if ((s & RQ_NET_SENT) && !(s & RQ_NET_DONE)) {
- req = iter;
- break;
- }
- }
- connection->req_not_net_done = req;
+ return (net_rq_state & RQ_NET_QUEUED) && !(net_rq_state & RQ_NET_DONE);
}
/* I'd like this to be the only place that manipulates
* req->completion_ref and req->kref. */
static void mod_rq_state(struct drbd_request *req, struct bio_and_error *m,
+ struct drbd_peer_device *peer_device,
int clear, int set)
{
- struct drbd_device *device = req->device;
- struct drbd_peer_device *peer_device = first_peer_device(device);
- unsigned s = req->rq_state;
+ unsigned int old_local, old_net = 0, new_net = 0;
+ unsigned int set_local = set & RQ_STATE_0_MASK;
+ unsigned int clear_local = clear & RQ_STATE_0_MASK;
int c_put = 0;
-
- if (drbd_suspended(device) && !((s | clear) & RQ_COMPLETION_SUSP))
- set |= RQ_COMPLETION_SUSP;
+ int d_put = 0;
+ int o_put = 0;
+ const int idx = peer_device ? peer_device->node_id : -1;
+ struct drbd_connection *connection = NULL;
+ bool unchanged;
+
+ set &= ~RQ_STATE_0_MASK;
+ clear &= ~RQ_STATE_0_MASK;
+
+ if (idx == -1) {
+ /* do not try to manipulate net state bits
+ * without an associated state slot! */
+ BUG_ON(set);
+ BUG_ON(clear);
+ }
/* apply */
+ spin_lock(&req->rq_lock); /* local IRQ already disabled */
+
+ old_local = req->local_rq_state;
+ req->local_rq_state &= ~clear_local;
+ req->local_rq_state |= set_local;
- req->rq_state &= ~clear;
- req->rq_state |= set;
+ if (idx != -1) {
+ old_net = req->net_rq_state[idx];
+ new_net = (req->net_rq_state[idx] & ~clear) | set;
+ WRITE_ONCE(req->net_rq_state[idx], new_net);
+ connection = peer_device->connection;
+ }
/* no change? */
- if (req->rq_state == s)
+ unchanged = req->local_rq_state == old_local &&
+ (idx == -1 || req->net_rq_state[idx] == old_net);
+
+ if (unchanged) {
+ spin_unlock(&req->rq_lock);
return;
+ }
/* intent: get references */
- kref_get(&req->kref);
-
- if (!(s & RQ_LOCAL_PENDING) && (set & RQ_LOCAL_PENDING))
+ if (!(old_local & RQ_LOCAL_PENDING) && (set_local & RQ_LOCAL_PENDING))
atomic_inc(&req->completion_ref);
- if (!(s & RQ_NET_PENDING) && (set & RQ_NET_PENDING)) {
- inc_ap_pending(device);
+ if (!(old_net & RQ_NET_PENDING) && (set & RQ_NET_PENDING)) {
+ inc_ap_pending(peer_device);
atomic_inc(&req->completion_ref);
}
- if (!(s & RQ_NET_QUEUED) && (set & RQ_NET_QUEUED)) {
+ if (!(old_net & RQ_NET_QUEUED) && (set & RQ_NET_QUEUED)) {
+ /* Keep request on transfer log while queued for sender */
+ refcount_inc(&req->oos_send_ref);
+ }
+
+ if (!drbd_sender_needs_master_bio(old_net) && drbd_sender_needs_master_bio(new_net)) {
+ /*
+ * This completion ref is necessary to avoid premature
+ * completion in case a WRITE_ACKED_BY_PEER comes in before the
+ * sender can do HANDED_OVER_TO_NETWORK.
+ */
atomic_inc(&req->completion_ref);
- set_if_null_req_next(peer_device, req);
}
- if (!(s & RQ_EXP_BARR_ACK) && (set & RQ_EXP_BARR_ACK))
- kref_get(&req->kref); /* wait for the DONE */
+ if (!(old_net & RQ_NET_READY) && (set & RQ_NET_READY) &&
+ !(req->net_rq_state[idx] & RQ_NET_DONE))
+ set_cache_ptr_if_null(connection, &connection->req_not_net_done, req);
+
+ if ((old_net & (RQ_NET_QUEUED | RQ_NET_READY)) != (RQ_NET_QUEUED | RQ_NET_READY) &&
+ (new_net & (RQ_NET_QUEUED | RQ_NET_READY)) == (RQ_NET_QUEUED | RQ_NET_READY))
+ set_cache_ptr_if_null(connection, &connection->req_next_ready, req);
- if (!(s & RQ_NET_SENT) && (set & RQ_NET_SENT)) {
+ if (!(old_net & RQ_EXP_BARR_ACK) && (set & RQ_EXP_BARR_ACK))
+ refcount_inc(&req->done_ref); /* wait for the DONE */
+
+ if (!(old_net & RQ_NET_SENT) && (set & RQ_NET_SENT)) {
/* potentially already completed in the ack_receiver thread */
- if (!(s & RQ_NET_DONE)) {
- atomic_add(req->i.size >> 9, &device->ap_in_flight);
- set_if_null_req_not_net_done(peer_device, req);
- }
- if (req->rq_state & RQ_NET_PENDING)
- set_if_null_req_ack_pending(peer_device, req);
+ if (!(old_net & RQ_NET_DONE))
+ atomic_add(req_payload_sectors(req), &peer_device->connection->ap_in_flight);
+ if (req->net_rq_state[idx] & RQ_NET_PENDING)
+ set_cache_ptr_if_null(connection, &connection->req_ack_pending, req);
}
- if (!(s & RQ_COMPLETION_SUSP) && (set & RQ_COMPLETION_SUSP))
+ if (!(old_local & RQ_COMPLETION_SUSP) && (set_local & RQ_COMPLETION_SUSP))
atomic_inc(&req->completion_ref);
+ spin_unlock(&req->rq_lock);
+
/* progress: put references */
- if ((s & RQ_COMPLETION_SUSP) && (clear & RQ_COMPLETION_SUSP))
+ if ((old_local & RQ_COMPLETION_SUSP) && (clear_local & RQ_COMPLETION_SUSP))
++c_put;
- if (!(s & RQ_LOCAL_ABORTED) && (set & RQ_LOCAL_ABORTED)) {
- D_ASSERT(device, req->rq_state & RQ_LOCAL_PENDING);
+ if (!(old_local & RQ_LOCAL_ABORTED) && (set_local & RQ_LOCAL_ABORTED)) {
+ D_ASSERT(req->device, req->local_rq_state & RQ_LOCAL_PENDING);
++c_put;
}
- if ((s & RQ_LOCAL_PENDING) && (clear & RQ_LOCAL_PENDING)) {
- if (req->rq_state & RQ_LOCAL_ABORTED)
- kref_put(&req->kref, drbd_req_destroy);
+ if ((old_local & RQ_LOCAL_PENDING) && (clear_local & RQ_LOCAL_PENDING)) {
+ struct drbd_device *device = req->device;
+
+ if (req->local_rq_state & RQ_LOCAL_ABORTED)
+ ++d_put;
else
++c_put;
+ spin_lock(&device->pending_completion_lock); /* local irq already disabled */
list_del_init(&req->req_pending_local);
+ spin_unlock(&device->pending_completion_lock);
}
- if ((s & RQ_NET_PENDING) && (clear & RQ_NET_PENDING)) {
- dec_ap_pending(device);
+ if ((old_net & RQ_NET_PENDING) && (clear & RQ_NET_PENDING)) {
+ dec_ap_pending(peer_device);
++c_put;
- req->acked_jif = jiffies;
- advance_conn_req_ack_pending(peer_device, req);
+ ktime_get_accounting(req->acked_kt[peer_device->node_id]);
+ advance_cache_ptr(connection, &connection->req_ack_pending,
+ req, RQ_NET_SENT | RQ_NET_PENDING, 0);
}
- if ((s & RQ_NET_QUEUED) && (clear & RQ_NET_QUEUED)) {
- ++c_put;
- advance_conn_req_next(peer_device, req);
+ if ((old_net & RQ_NET_QUEUED) && (clear & RQ_NET_QUEUED)) {
+ ++o_put;
+ advance_conn_req_next(connection, req);
+ advance_cache_ptr(connection, &connection->req_next_ready,
+ req, RQ_NET_QUEUED | RQ_NET_READY, 0);
}
- if (!(s & RQ_NET_DONE) && (set & RQ_NET_DONE)) {
- if (s & RQ_NET_SENT)
- atomic_sub(req->i.size >> 9, &device->ap_in_flight);
- if (s & RQ_EXP_BARR_ACK)
- kref_put(&req->kref, drbd_req_destroy);
- req->net_done_jif = jiffies;
+ if (drbd_sender_needs_master_bio(old_net) && !drbd_sender_needs_master_bio(new_net))
+ ++c_put;
- /* in ahead/behind mode, or just in case,
- * before we finally destroy this request,
- * the caching pointers must not reference it anymore */
- advance_conn_req_next(peer_device, req);
- advance_conn_req_ack_pending(peer_device, req);
- advance_conn_req_not_net_done(peer_device, req);
+ /* ap_in_flight holds the payload of the requests on the wire towards
+ * this peer, so a request that is taken off the wire for a resend has
+ * to give its sectors back here. The RQ_NET_SENT that follows the
+ * resend adds them again.
+ */
+ if ((old_net & RQ_NET_SENT) && (clear & RQ_NET_SENT) && !(old_net & RQ_NET_DONE))
+ atomic_sub(req_payload_sectors(req), &connection->ap_in_flight);
+
+ if (!(old_net & RQ_NET_DONE) && (set & RQ_NET_DONE)) {
+ if (old_net & RQ_NET_SENT)
+ atomic_sub(req_payload_sectors(req),
+ &peer_device->connection->ap_in_flight);
+ if (old_net & RQ_EXP_BARR_ACK)
+ ++d_put;
+ ktime_get_accounting(req->net_done_kt[peer_device->node_id]);
+
+ advance_cache_ptr(connection, &connection->req_not_net_done,
+ req, 0, RQ_NET_DONE);
}
- /* potentially complete and destroy */
-
- /* If we made progress, retry conflicting peer requests, if any. */
- if (test_bit(INTERVAL_WAITING, &req->i.flags))
- wake_up(&device->misc_wait);
+ if ((old_net & RQ_NET_PENDING_OOS) && (clear & RQ_NET_PENDING_OOS)) {
+ if (peer_device->repl_state[NOW] == L_AHEAD &&
+ atomic_read(&peer_device->connection->ap_in_flight) == 0) {
+ struct drbd_peer_device *pd;
+ int vnr;
+ /* The first peer device to notice that it is time to
+ * go Ahead -> SyncSource tries to trigger that
+ * transition for *all* peer devices currently in
+ * L_AHEAD for this connection. */
+ idr_for_each_entry(&peer_device->connection->peer_devices, pd, vnr) {
+ if (pd->repl_state[NOW] != L_AHEAD)
+ continue;
+ if (test_and_set_bit(AHEAD_TO_SYNC_SOURCE, pd->flags))
+ continue; /* already done */
+ pd->start_resync_side = L_SYNC_SOURCE;
+ mod_timer(&pd->start_resync_timer, jiffies + HZ);
+ }
+ }
+ }
+ /* potentially complete and destroy */
drbd_req_put_completion_ref(req, m, c_put);
- kref_put(&req->kref, drbd_req_destroy);
+
+ /* req cannot have been destroyed if there are still references */
+ if (d_put || o_put)
+ /* potentially destroy */
+ drbd_put_ref_tl_walk(req, d_put, o_put);
}
static void drbd_report_io_error(struct drbd_device *device, struct drbd_request *req)
@@ -517,10 +1115,25 @@ static void drbd_report_io_error(struct drbd_device *device, struct drbd_request
return;
drbd_warn(device, "local %s IO error sector %llu+%u on %pg\n",
- (req->rq_state & RQ_WRITE) ? "WRITE" : "READ",
- (unsigned long long)req->i.sector,
- req->i.size >> 9,
- device->ldev->backing_bdev);
+ (req->local_rq_state & RQ_WRITE) ? "WRITE" : "READ",
+ (unsigned long long)req->i.sector,
+ req->i.size >> 9,
+ device->ldev->backing_bdev);
+}
+
+static int drbd_protocol_state_bits(struct drbd_connection *connection)
+{
+ struct drbd_net_conf *nc;
+ int p;
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ p = nc->wire_protocol;
+ rcu_read_unlock();
+
+ return p == DRBD_PROT_C ? RQ_EXP_WRITE_ACK :
+ p == DRBD_PROT_B ? RQ_EXP_RECEIVE_ACK : 0;
+
}
/* Helper for HANDED_OVER_TO_NETWORK.
@@ -529,11 +1142,73 @@ static void drbd_report_io_error(struct drbd_device *device, struct drbd_request
* --> If so, clear PENDING and set NET_OK below.
* If it is a protocol A write, but not RQ_PENDING anymore, neg-ack was faster
* (and we must not set RQ_NET_OK) */
-static inline bool is_pending_write_protocol_A(struct drbd_request *req)
+static inline bool is_pending_write_protocol_A(struct drbd_request *req, int idx)
+{
+ return (req->local_rq_state & RQ_WRITE) == 0 ? 0 :
+ (req->net_rq_state[idx] &
+ (RQ_NET_PENDING|RQ_EXP_WRITE_ACK|RQ_EXP_RECEIVE_ACK))
+ == RQ_NET_PENDING;
+}
+
+/* Confirm-before-complete: while a diskless primary's optimistically rotated
+ * current generation is still unconfirmed (obligation state UNCONFIRMED), hold
+ * back completion of writes acknowledged in that generation, so we never
+ * acknowledge data to the upper layers in a generation no quorate partition has
+ * confirmed. Released by a NEW_UUID_CONFIRMED transfer-log walk once the
+ * generation is confirmed (drbd_maybe_release_rotated_gen).
+ *
+ * Only on on-no-quorum=suspend-io: with io-error the configured behaviour (error
+ * out) wins. Only the diskless optimistic bump reaches UNCONFIRMED; a diskful
+ * node self-confirms its bump synchronously and never arrives here in it.
+ */
+static bool hold_completion_for_unconfirmed_gen(struct drbd_request *req)
+{
+ struct drbd_device *device = req->device;
+
+ if (!(req->local_rq_state & RQ_WRITE))
+ return false;
+ if (device->resource->res_opts.on_no_quorum != ONQ_SUSPEND_IO)
+ return false;
+ if (drbd_gen_obligation_state(device) != GEN_OBL_UNCONFIRMED)
+ return false;
+ return (int)(req->epoch - device->exposed_gen_epoch) >= 0;
+}
+
+/* Clear CURRENT_UUID_UNCONFIRMED once this peer holds the rotated generation:
+ * every held write durable on it (RQ_NET_OK | RQ_NET_DONE), and a barrier ack
+ * at/past exposed_gen_epoch -- the in-order proof for the no-held case, where
+ * no held write vouches for receipt. Run from got_BarrierAck.
+ */
+void drbd_peer_maybe_confirm_rotated_gen(struct drbd_peer_device *peer_device,
+ unsigned int acked_epoch)
{
- return (req->rq_state &
- (RQ_WRITE|RQ_NET_PENDING|RQ_EXP_WRITE_ACK|RQ_EXP_RECEIVE_ACK))
- == (RQ_WRITE|RQ_NET_PENDING);
+ struct drbd_device *device = peer_device->device;
+ struct drbd_resource *resource = device->resource;
+ struct drbd_request *req;
+ int idx = peer_device->node_id;
+
+ if (!test_bit(CURRENT_UUID_UNCONFIRMED, peer_device->flags))
+ return;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(req, &resource->transfer_log, tl_requests) {
+ if (req->device != device)
+ continue;
+ if (!(req->local_rq_state & RQ_UNCONF_GEN))
+ continue;
+ /* durable == received and flushed: RQ_NET_OK | RQ_NET_DONE.
+ * RQ_NET_DONE without RQ_NET_OK is connection-loss cleanup, not data.
+ */
+ if ((req->net_rq_state[idx] & (RQ_NET_OK | RQ_NET_DONE)) !=
+ (RQ_NET_OK | RQ_NET_DONE)) {
+ rcu_read_unlock();
+ return;
+ }
+ }
+ rcu_read_unlock();
+
+ if ((int)(acked_epoch - device->exposed_gen_epoch) >= 0)
+ clear_bit(CURRENT_UUID_UNCONFIRMED, peer_device->flags);
}
/* obviously this could be coded as many single functions
@@ -544,95 +1219,76 @@ static inline bool is_pending_write_protocol_A(struct drbd_request *req)
* but having it this way
* enforces that it is all in this one place, where it is easier to audit,
* it makes it obvious that whatever "event" "happens" to a request should
- * happen "atomically" within the req_lock,
+ * happen with the state_rwlock read lock held,
* and it enforces that we have to think in a very structured manner
* about the "events" that may happen to a request during its life time ...
*
*
* peer_device == NULL means local disk
*/
-int __req_mod(struct drbd_request *req, enum drbd_req_event what,
+void __req_mod(struct drbd_request *req, enum drbd_req_event what,
struct drbd_peer_device *peer_device,
struct bio_and_error *m)
{
- struct drbd_device *const device = req->device;
- struct drbd_connection *const connection = peer_device ? peer_device->connection : NULL;
- struct net_conf *nc;
- int p, rv = 0;
+ struct drbd_device *device = req->device;
+ struct drbd_net_conf *nc;
+ unsigned long flags;
+ int p;
+ int idx;
+
+ lockdep_assert_held(&device->resource->state_rwlock);
if (m)
m->bio = NULL;
+ idx = peer_device ? peer_device->node_id : -1;
+
switch (what) {
default:
- drbd_err(device, "LOGIC BUG in %s:%u\n", __FILE__ , __LINE__);
- break;
-
- /* does not happen...
- * initialization done in drbd_req_new
- case CREATED:
- break;
- */
-
- case TO_BE_SENT: /* via network */
- /* reached via __drbd_make_request
- * and from w_read_retry_remote */
- D_ASSERT(device, !(req->rq_state & RQ_NET_MASK));
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- p = nc->wire_protocol;
- rcu_read_unlock();
- req->rq_state |=
- p == DRBD_PROT_C ? RQ_EXP_WRITE_ACK :
- p == DRBD_PROT_B ? RQ_EXP_RECEIVE_ACK : 0;
- mod_rq_state(req, m, 0, RQ_NET_PENDING);
+ drbd_err(device, "LOGIC BUG in %s:%u\n", __FILE__, __LINE__);
break;
case TO_BE_SUBMITTED: /* locally */
/* reached via __drbd_make_request */
- D_ASSERT(device, !(req->rq_state & RQ_LOCAL_MASK));
- mod_rq_state(req, m, 0, RQ_LOCAL_PENDING);
+ D_ASSERT(device, !(req->local_rq_state & RQ_LOCAL_MASK));
+ mod_rq_state(req, m, peer_device, 0, RQ_LOCAL_PENDING);
break;
case COMPLETED_OK:
- if (req->rq_state & RQ_WRITE)
+ if (req->local_rq_state & RQ_WRITE)
device->writ_cnt += req->i.size >> 9;
else
device->read_cnt += req->i.size >> 9;
- mod_rq_state(req, m, RQ_LOCAL_PENDING,
+ mod_rq_state(req, m, peer_device, RQ_LOCAL_PENDING,
RQ_LOCAL_COMPLETED|RQ_LOCAL_OK);
break;
case ABORT_DISK_IO:
- mod_rq_state(req, m, 0, RQ_LOCAL_ABORTED);
+ mod_rq_state(req, m, peer_device, 0, RQ_LOCAL_ABORTED);
break;
case WRITE_COMPLETED_WITH_ERROR:
drbd_report_io_error(device, req);
- __drbd_chk_io_error(device, DRBD_WRITE_ERROR);
- mod_rq_state(req, m, RQ_LOCAL_PENDING, RQ_LOCAL_COMPLETED);
+ mod_rq_state(req, m, peer_device, RQ_LOCAL_PENDING, RQ_LOCAL_COMPLETED);
break;
case READ_COMPLETED_WITH_ERROR:
- drbd_set_out_of_sync(first_peer_device(device),
- req->i.sector, req->i.size);
+ drbd_set_all_out_of_sync(device, -1, req->i.sector, req->i.size);
drbd_report_io_error(device, req);
- __drbd_chk_io_error(device, DRBD_READ_ERROR);
fallthrough;
case READ_AHEAD_COMPLETED_WITH_ERROR:
- /* it is legal to fail read-ahead, no __drbd_chk_io_error in that case. */
- mod_rq_state(req, m, RQ_LOCAL_PENDING, RQ_LOCAL_COMPLETED);
+ mod_rq_state(req, m, peer_device, RQ_LOCAL_PENDING, RQ_LOCAL_COMPLETED);
break;
case DISCARD_COMPLETED_NOTSUPP:
case DISCARD_COMPLETED_WITH_ERROR:
/* I'd rather not detach from local disk just because it
* failed a REQ_OP_DISCARD. */
- mod_rq_state(req, m, RQ_LOCAL_PENDING, RQ_LOCAL_COMPLETED);
+ mod_rq_state(req, m, peer_device, RQ_LOCAL_PENDING, RQ_LOCAL_COMPLETED);
break;
- case QUEUE_FOR_NET_READ:
+ case NEW_NET_READ:
/* READ, and
* no local disk,
* or target area marked as invalid,
@@ -644,27 +1300,19 @@ int __req_mod(struct drbd_request *req, enum drbd_req_event what,
* Corresponding drbd_remove_request_interval is in
* drbd_req_complete() */
D_ASSERT(device, drbd_interval_empty(&req->i));
+ spin_lock_irqsave(&device->interval_lock, flags);
drbd_insert_interval(&device->read_requests, &req->i);
+ spin_unlock_irqrestore(&device->interval_lock, flags);
- set_bit(UNPLUG_REMOTE, &device->flags);
-
- D_ASSERT(device, req->rq_state & RQ_NET_PENDING);
- D_ASSERT(device, (req->rq_state & RQ_LOCAL_MASK) == 0);
- mod_rq_state(req, m, 0, RQ_NET_QUEUED);
- req->w.cb = w_send_read_req;
- drbd_queue_work(&connection->sender_work,
- &req->w);
+ D_ASSERT(device, !(req->net_rq_state[idx] & RQ_NET_MASK));
+ D_ASSERT(device, !(req->local_rq_state & RQ_LOCAL_MASK));
+ mod_rq_state(req, m, peer_device, 0, RQ_NET_PENDING|RQ_NET_QUEUED);
break;
- case QUEUE_FOR_NET_WRITE:
+ case NEW_NET_WRITE:
/* assert something? */
/* from __drbd_make_request only */
- /* Corresponding drbd_remove_request_interval is in
- * drbd_req_complete() */
- D_ASSERT(device, drbd_interval_empty(&req->i));
- drbd_insert_interval(&device->write_requests, &req->i);
-
/* NOTE
* In case the req ended up on the transfer log before being
* queued on the worker, it could lead to this request being
@@ -679,85 +1327,109 @@ int __req_mod(struct drbd_request *req, enum drbd_req_event what,
*
* Add req to the (now) current epoch (barrier). */
- /* otherwise we may lose an unplug, which may cause some remote
- * io-scheduler timeout to expire, increasing maximum latency,
- * hurting performance. */
- set_bit(UNPLUG_REMOTE, &device->flags);
+ D_ASSERT(device, !(req->net_rq_state[idx] & RQ_NET_MASK));
/* queue work item to send data */
- D_ASSERT(device, req->rq_state & RQ_NET_PENDING);
- mod_rq_state(req, m, 0, RQ_NET_QUEUED|RQ_EXP_BARR_ACK);
- req->w.cb = w_send_dblock;
- drbd_queue_work(&connection->sender_work,
- &req->w);
+ mod_rq_state(req, m, peer_device, 0, RQ_NET_PENDING|RQ_NET_QUEUED|RQ_EXP_BARR_ACK|
+ drbd_protocol_state_bits(peer_device->connection));
+
+ /* Close the epoch, in case it outgrew the limit.
+ * Or if this is a "batch bio", and some of our peers is "old",
+ * because a batch bio "storm" (like, large scale discarding
+ * during mkfs time) would be likely to starve out the peers
+ * activity log, if it is smaller than ours (or we don't have
+ * any). And a fix for the resulting potential distributed
+ * deadlock was only implemented with P_CONFIRM_STABLE with
+ * protocol version 114.
+ */
+ if (device->resource->cached_min_aggreed_protocol_version < 114 &&
+ (req->local_rq_state & (RQ_UNMAP|RQ_WSAME|RQ_ZEROES)))
+ p = 1;
+ else {
+ rcu_read_lock();
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ p = nc->max_epoch_size;
+ rcu_read_unlock();
+ }
+ if (device->resource->current_tle_writes >= p)
+ start_new_tl_epoch(device->resource);
+ break;
- /* close the epoch, in case it outgrew the limit */
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- p = nc->max_epoch_size;
- rcu_read_unlock();
- if (connection->current_tle_writes >= p)
- start_new_tl_epoch(connection);
+ case NEW_NET_OOS:
+ /* We will just send P_OUT_OF_SYNC to this peer. The request is
+ * "done" from the start in the sense that everything necessary
+ * in the data stage has been done.
+ */
+ mod_rq_state(req, m, peer_device, 0, RQ_NET_PENDING_OOS|RQ_NET_QUEUED|RQ_NET_DONE);
+ break;
+ case READY_FOR_NET:
+ mod_rq_state(req, m, peer_device, 0, RQ_NET_READY);
break;
- case QUEUE_FOR_SEND_OOS:
- mod_rq_state(req, m, 0, RQ_NET_QUEUED);
- req->w.cb = w_send_out_of_sync;
- drbd_queue_work(&connection->sender_work,
- &req->w);
+ case SKIP_OOS:
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING_OOS, RQ_NET_READY);
break;
- case READ_RETRY_REMOTE_CANCELED:
+ case OOS_HANDED_TO_NETWORK:
case SEND_CANCELED:
case SEND_FAILED:
- /* real cleanup will be done from tl_clear. just update flags
- * so it is no longer marked as on the worker queue */
- mod_rq_state(req, m, RQ_NET_QUEUED, 0);
+ /* Sending P_OUT_OF_SYNC is irrelevant if the connection was
+ * lost. Hence, when the intention was to send P_OUT_OF_SYNC,
+ * the effect of successfully sending the packet and connection
+ * loss are the same.
+ *
+ * Otherwise just update flags so it is no longer marked as on
+ * the sender queue; real cleanup will be done from
+ * tl_walk(,CONNECTION_LOST*).
+ */
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING_OOS|RQ_NET_QUEUED, 0);
break;
case HANDED_OVER_TO_NETWORK:
/* assert something? */
- if (is_pending_write_protocol_A(req))
+ if (is_pending_write_protocol_A(req, idx))
/* this is what is dangerous about protocol A:
* pretend it was successfully written on the peer. */
- mod_rq_state(req, m, RQ_NET_QUEUED|RQ_NET_PENDING,
- RQ_NET_SENT|RQ_NET_OK);
+ mod_rq_state(req, m, peer_device, RQ_NET_QUEUED|RQ_NET_PENDING,
+ RQ_NET_SENT|RQ_NET_OK);
else
- mod_rq_state(req, m, RQ_NET_QUEUED, RQ_NET_SENT);
+ mod_rq_state(req, m, peer_device, RQ_NET_QUEUED, RQ_NET_SENT);
/* It is still not yet RQ_NET_DONE until the
* corresponding epoch barrier got acked as well,
* so we know what to dirty on connection loss. */
break;
- case OOS_HANDED_TO_NETWORK:
- /* Was not set PENDING, no longer QUEUED, so is now DONE
- * as far as this connection is concerned. */
- mod_rq_state(req, m, RQ_NET_QUEUED, RQ_NET_DONE);
- break;
-
- case CONNECTION_LOST_WHILE_PENDING:
- /* transfer log cleanup after connection loss */
- mod_rq_state(req, m,
- RQ_NET_OK|RQ_NET_PENDING|RQ_COMPLETION_SUSP,
- RQ_NET_DONE);
- break;
+ case CONNECTION_LOST:
+ case CONNECTION_LOST_WHILE_SUSPENDED:
+ /* Only apply to requests that were for this peer but not done. */
+ if (!(req->net_rq_state[idx] & RQ_NET_MASK) || req->net_rq_state[idx] & RQ_NET_DONE)
+ break;
- case CONFLICT_RESOLVED:
- /* for superseded conflicting writes of multiple primaries,
- * there is no need to keep anything in the tl, potential
- * node crashes are covered by the activity log.
+ /* For protocol A, or when not suspended, we consider the
+ * request to be lost towards this peer.
+ *
+ * Protocol B&C requests are kept while suspended because
+ * resending is allowed. If such a request is pending to this
+ * peer, we suspend its completion until IO is resumed. This is
+ * a conservative simplification. We could complete it while
+ * suspended once we know it has been received by "enough"
+ * peers. However, we do not track that.
*
- * If this request had been marked as RQ_POSTPONED before,
- * it will actually not be completed, but "restarted",
- * resubmitted from the retry worker context. */
- D_ASSERT(device, req->rq_state & RQ_NET_PENDING);
- D_ASSERT(device, req->rq_state & RQ_EXP_WRITE_ACK);
- mod_rq_state(req, m, RQ_NET_PENDING, RQ_NET_DONE|RQ_NET_OK);
+ * If the request is no longer pending to this peer, then we
+ * have already received the corresponding ack. The request may
+ * complete as far as this peer is concerned. */
+ if (what == CONNECTION_LOST ||
+ !(req->net_rq_state[idx] & (RQ_EXP_RECEIVE_ACK|RQ_EXP_WRITE_ACK)))
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING|RQ_NET_OK, RQ_NET_DONE);
+ else if (req->net_rq_state[idx] & RQ_NET_PENDING)
+ mod_rq_state(req, m, peer_device, 0, RQ_COMPLETION_SUSP);
break;
case WRITE_ACKED_BY_PEER_AND_SIS:
- req->rq_state |= RQ_NET_SIS;
+ spin_lock_irqsave(&req->rq_lock, flags);
+ req->net_rq_state[idx] |= RQ_NET_SIS;
+ spin_unlock_irqrestore(&req->rq_lock, flags);
fallthrough;
case WRITE_ACKED_BY_PEER:
/* Normal operation protocol C: successfully written on peer.
@@ -769,155 +1441,196 @@ int __req_mod(struct drbd_request *req, enum drbd_req_event what,
* for volatile write-back caches on lower level devices. */
goto ack_common;
case RECV_ACKED_BY_PEER:
- D_ASSERT(device, req->rq_state & RQ_EXP_RECEIVE_ACK);
+ D_ASSERT(device, req->net_rq_state[idx] & RQ_EXP_RECEIVE_ACK);
/* protocol B; pretends to be successfully written on peer.
* see also notes above in HANDED_OVER_TO_NETWORK about
* protocol != C */
- ack_common:
- mod_rq_state(req, m, RQ_NET_PENDING, RQ_NET_OK);
+ack_common:
+ if (hold_completion_for_unconfirmed_gen(req)) {
+ /* Mark RQ_NET_OK (the peer has the data), but hold master-bio
+ * completion via RQ_COMPLETION_SUSP until the generation is
+ * confirmed. Setting RQ_COMPLETION_SUSP takes a completion ref
+ * exactly as clearing RQ_NET_PENDING drops one, so the request
+ * does not complete now. RQ_UNCONF_GEN tags it for release by
+ * NEW_UUID_CONFIRMED.
+ */
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING,
+ RQ_NET_OK | RQ_COMPLETION_SUSP | RQ_UNCONF_GEN);
+ break;
+ }
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING, RQ_NET_OK);
+ break;
+
+ case NEG_ACKED:
+ mod_rq_state(req, m, peer_device, RQ_NET_OK|RQ_NET_PENDING,
+ (req->local_rq_state & RQ_WRITE) ? 0 : RQ_NET_DONE);
break;
- case POSTPONE_WRITE:
- D_ASSERT(device, req->rq_state & RQ_EXP_WRITE_ACK);
- /* If this node has already detected the write conflict, the
- * worker will be waiting on misc_wait. Wake it up once this
- * request has completed locally.
+ case POSTPONED_BY_PEER:
+ /* The peer did not process this write; retry it as a
+ * brand-new request once every reference drains. See
+ * drbd_restart_request().
*/
- D_ASSERT(device, req->rq_state & RQ_NET_PENDING);
- req->rq_state |= RQ_POSTPONED;
- if (test_bit(INTERVAL_WAITING, &req->i.flags))
- wake_up(&device->misc_wait);
- /* Do not clear RQ_NET_PENDING. This request will make further
- * progress via restart_conflicting_writes() or
- * fail_postponed_requests(). Hopefully. */
+ mod_rq_state(req, m, peer_device, RQ_NET_OK|RQ_NET_PENDING,
+ RQ_POSTPONED);
break;
- case NEG_ACKED:
- mod_rq_state(req, m, RQ_NET_OK|RQ_NET_PENDING, 0);
+ case COMPLETION_RESUMED:
+ mod_rq_state(req, m, peer_device, RQ_COMPLETION_SUSP, 0);
break;
- case FAIL_FROZEN_DISK_IO:
- if (!(req->rq_state & RQ_LOCAL_COMPLETED))
- break;
- mod_rq_state(req, m, RQ_COMPLETION_SUSP, 0);
+ case NEW_UUID_CONFIRMED:
+ /* The optimistically rotated data generation has been confirmed
+ * durable by an in-order barrier ack from a quorate peer. Release
+ * the writes whose completion we held back in that generation (see
+ * the RQ_UNCONF_GEN hold in ack_common). Once the generation is
+ * confirmed, every such write is safe to acknowledge: the peer holds
+ * the new current UUID, so the diskless-primary strand cannot occur.
+ */
+ if (req->local_rq_state & RQ_UNCONF_GEN)
+ mod_rq_state(req, m, peer_device,
+ RQ_COMPLETION_SUSP | RQ_UNCONF_GEN, 0);
break;
- case RESTART_FROZEN_DISK_IO:
- if (!(req->rq_state & RQ_LOCAL_COMPLETED))
+ case CANCEL_SUSPENDED_IO:
+ /* Only apply to requests that were for this peer but not done. */
+ if (!(req->net_rq_state[idx] & RQ_NET_MASK) || req->net_rq_state[idx] & RQ_NET_DONE)
break;
- mod_rq_state(req, m,
- RQ_COMPLETION_SUSP|RQ_LOCAL_COMPLETED,
- RQ_LOCAL_PENDING);
-
- rv = MR_READ;
- if (bio_data_dir(req->master_bio) == WRITE)
- rv = MR_WRITE;
-
- get_ldev(device); /* always succeeds in this call path */
- req->w.cb = w_restart_disk_io;
- drbd_queue_work(&connection->sender_work,
- &req->w);
+ /* CONNECTION_LOST_WHILE_SUSPENDED followed by
+ * CANCEL_SUSPENDED_IO should be essentially the same as
+ * CONNECTION_LOST. Make the corresponding changes. The
+ * RQ_COMPLETION_SUSP flag is handled by COMPLETION_RESUMED. */
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING|RQ_NET_OK, RQ_NET_DONE);
break;
case RESEND:
- /* Simply complete (local only) READs. */
- if (!(req->rq_state & RQ_WRITE) && !req->w.cb) {
- mod_rq_state(req, m, RQ_COMPLETION_SUSP, 0);
- break;
- }
-
/* If RQ_NET_OK is already set, we got a P_WRITE_ACK or P_RECV_ACK
before the connection loss (B&C only); only P_BARRIER_ACK
(or the local completion?) was missing when we suspended.
Throwing them out of the TL here by pretending we got a BARRIER_ACK.
- During connection handshake, we ensure that the peer was not rebooted. */
- if (!(req->rq_state & RQ_NET_OK)) {
- /* FIXME could this possibly be a req->dw.cb == w_send_out_of_sync?
- * in that case we must not set RQ_NET_PENDING. */
-
- mod_rq_state(req, m, RQ_COMPLETION_SUSP, RQ_NET_QUEUED|RQ_NET_PENDING);
- if (req->w.cb) {
- /* w.cb expected to be w_send_dblock, or w_send_read_req */
- drbd_queue_work(&connection->sender_work,
- &req->w);
- rv = req->rq_state & RQ_WRITE ? MR_WRITE : MR_READ;
- } /* else: FIXME can this happen? */
+ During connection handshake, we ensure that the peer was not rebooted.
+
+ Protocol A requests always have RQ_NET_OK removed when the
+ connection is lost, so this will never apply to them.
+
+ Resending is only allowed on synchronous connections,
+ where all requests not yet completed to upper layers would
+ be in the same "reorder-domain", there can not possibly be
+ any dependency between incomplete requests, and we are
+ allowed to complete this one "out-of-sequence".
+ */
+ if (req->net_rq_state[idx] & RQ_NET_OK)
+ goto barrier_acked;
+
+ /* Only apply to requests that are pending a response from
+ * this peer. */
+ if (!(req->net_rq_state[idx] & RQ_NET_PENDING))
break;
- }
- fallthrough; /* to BARRIER_ACKED */
+
+ D_ASSERT(device, !(req->net_rq_state[idx] & RQ_NET_QUEUED));
+ mod_rq_state(req, m, peer_device, RQ_NET_SENT, RQ_NET_QUEUED);
+ break;
case BARRIER_ACKED:
+barrier_acked:
/* barrier ack for READ requests does not make sense */
- if (!(req->rq_state & RQ_WRITE))
+ if (!(req->local_rq_state & RQ_WRITE))
break;
- if (req->rq_state & RQ_NET_PENDING) {
+ if (req->net_rq_state[idx] & RQ_NET_PENDING) {
/* barrier came in before all requests were acked.
* this is bad, because if the connection is lost now,
* we won't be able to clean them up... */
drbd_err(device, "FIXME (BARRIER_ACKED but pending)\n");
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING, RQ_NET_OK);
}
- /* Allowed to complete requests, even while suspended.
- * As this is called for all requests within a matching epoch,
+ /* As this is called for all requests within a matching epoch,
* we need to filter, and only set RQ_NET_DONE for those that
* have actually been on the wire. */
- mod_rq_state(req, m, RQ_COMPLETION_SUSP,
- (req->rq_state & RQ_NET_MASK) ? RQ_NET_DONE : 0);
+ if (req->net_rq_state[idx] & RQ_NET_MASK)
+ mod_rq_state(req, m, peer_device, 0, RQ_NET_DONE);
break;
case DATA_RECEIVED:
- D_ASSERT(device, req->rq_state & RQ_NET_PENDING);
- mod_rq_state(req, m, RQ_NET_PENDING, RQ_NET_OK|RQ_NET_DONE);
+ D_ASSERT(device, req->net_rq_state[idx] & RQ_NET_PENDING);
+ mod_rq_state(req, m, peer_device, RQ_NET_PENDING, RQ_NET_OK|RQ_NET_DONE);
break;
- case QUEUE_AS_DRBD_BARRIER:
- start_new_tl_epoch(connection);
- mod_rq_state(req, m, 0, RQ_NET_OK|RQ_NET_DONE);
+ case BARRIER_SENT:
+ mod_rq_state(req, m, peer_device, 0, RQ_NET_OK|RQ_NET_DONE);
break;
}
-
- return rv;
}
/* we may do a local read if:
* - we are consistent (of course),
* - or we are generally inconsistent,
- * BUT we are still/already IN SYNC for this area.
+ * BUT we are still/already IN SYNC with all peers for this area.
* since size may be bigger than BM_BLOCK_SIZE,
* we may need to check several bits.
*/
static bool drbd_may_do_local_read(struct drbd_device *device, sector_t sector, int size)
{
+ struct drbd_md *md = &device->ldev->md;
+ struct drbd_bitmap *bm;
+ unsigned int node_id;
+ unsigned int n_checked = 0;
+
unsigned long sbnr, ebnr;
sector_t esector, nr_sectors;
- if (device->state.disk == D_UP_TO_DATE)
+ if (device->disk_state[NOW] == D_UP_TO_DATE)
return true;
- if (device->state.disk != D_INCONSISTENT)
+ if (device->disk_state[NOW] != D_INCONSISTENT)
return false;
esector = sector + (size >> 9) - 1;
nr_sectors = get_capacity(device->vdisk);
D_ASSERT(device, sector < nr_sectors);
D_ASSERT(device, esector < nr_sectors);
- sbnr = BM_SECT_TO_BIT(sector);
- ebnr = BM_SECT_TO_BIT(esector);
+ bm = device->bitmap;
+ if (!bm)
+ return true;
+
+ sbnr = bm_sect_to_bit(bm, sector);
+ ebnr = bm_sect_to_bit(bm, esector);
- return drbd_bm_count_bits(device, sbnr, ebnr) == 0;
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &md->peers[node_id];
+
+ /* Skip bitmap indexes which are not assigned to a peer. */
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md->flags))
+ continue;
+
+ if (drbd_bm_count_bits(device, peer_md->bitmap_index, sbnr, ebnr))
+ return false;
+ ++n_checked;
+ }
+ if (n_checked == 0) {
+ drbd_err_ratelimit(device, "No valid bitmap slots found to check!\n");
+ return false;
+ }
+ return true;
}
-static bool remote_due_to_read_balancing(struct drbd_device *device, sector_t sector,
+/* TODO improve for more than one peer.
+ * also take into account the drbd protocol. */
+static bool remote_due_to_read_balancing(struct drbd_device *device,
+ struct drbd_peer_device *peer_device, sector_t sector,
enum drbd_read_balancing rbm)
{
int stripe_shift;
switch (rbm) {
case RB_CONGESTED_REMOTE:
+ /* originally, this used the bdi congestion framework,
+ * but that was removed in linux 5.18.
+ * so just never report the lower device as congested. */
return false;
case RB_LEAST_PENDING:
return atomic_read(&device->local_cnt) >
- atomic_read(&device->ap_pending_cnt) + atomic_read(&device->rs_pending_cnt);
+ atomic_read(&peer_device->ap_pending_cnt) + atomic_read(&peer_device->rs_pending_cnt);
case RB_32K_STRIPING: /* stripe_shift = 15 */
case RB_64K_STRIPING:
case RB_128K_STRIPING:
@@ -936,63 +1649,32 @@ static bool remote_due_to_read_balancing(struct drbd_device *device, sector_t se
}
}
-/*
- * complete_conflicting_writes - wait for any conflicting write requests
- *
- * The write_requests tree contains all active write requests which we
- * currently know about. Wait for any requests to complete which conflict with
- * the new one.
- *
- * Only way out: remove the conflicting intervals from the tree.
- */
-static void complete_conflicting_writes(struct drbd_request *req)
-{
- DEFINE_WAIT(wait);
- struct drbd_device *device = req->device;
- struct drbd_interval *i;
- sector_t sector = req->i.sector;
- int size = req->i.size;
-
- for (;;) {
- drbd_for_each_overlap(i, &device->write_requests, sector, size) {
- /* Ignore, if already completed to upper layers. */
- if (test_bit(INTERVAL_COMPLETED, &i->flags))
- continue;
- /* Handle the first found overlap. After the schedule
- * we have to restart the tree walk. */
- break;
- }
- if (!i) /* if any */
- break;
-
- /* Indicate to wake up device->misc_wait on progress. */
- prepare_to_wait(&device->misc_wait, &wait, TASK_UNINTERRUPTIBLE);
- set_bit(INTERVAL_WAITING, &i->flags);
- spin_unlock_irq(&device->resource->req_lock);
- schedule();
- spin_lock_irq(&device->resource->req_lock);
- }
- finish_wait(&device->misc_wait, &wait);
-}
-
-/* called within req_lock */
-static void maybe_pull_ahead(struct drbd_device *device)
+static void __maybe_pull_ahead(struct drbd_device *device, struct drbd_connection *connection)
{
- struct drbd_connection *connection = first_peer_device(device)->connection;
- struct net_conf *nc;
+ struct drbd_net_conf *nc;
bool congested = false;
enum drbd_on_congestion on_congestion;
+ u32 cong_fill = 0, cong_extents = 0;
+ struct drbd_peer_device *peer_device = conn_peer_device(connection, device->vnr);
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- on_congestion = nc ? nc->on_congestion : OC_BLOCK;
- rcu_read_unlock();
- if (on_congestion == OC_BLOCK ||
- connection->agreed_pro_version < 96)
+ lockdep_assert_held(&device->resource->state_rwlock);
+
+ if (connection->agreed_pro_version < 96)
return;
- if (on_congestion == OC_PULL_AHEAD && device->state.conn == C_AHEAD)
- return; /* nothing to do ... */
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (nc) {
+ on_congestion = nc->on_congestion;
+ cong_fill = nc->cong_fill;
+ cong_extents = nc->cong_extents;
+ } else {
+ on_congestion = OC_BLOCK;
+ }
+ if (on_congestion == OC_BLOCK)
+ return;
+
+ if (!drbd_should_do_remote(peer_device, NOW))
+ return; /* Ignore congestion if we are not replicating writes */
/* If I don't even have good local storage, we can not reasonably try
* to pull ahead of the peer. We also need the local reference to make
@@ -1001,44 +1683,125 @@ static void maybe_pull_ahead(struct drbd_device *device)
if (!get_ldev_if_state(device, D_UP_TO_DATE))
return;
- if (nc->cong_fill &&
- atomic_read(&device->ap_in_flight) >= nc->cong_fill) {
- drbd_info(device, "Congestion-fill threshold reached\n");
- congested = true;
+ if (test_and_set_bit(HANDLING_CONGESTION, peer_device->flags))
+ goto out;
+
+ /* if an other volume already found that we are congested, short circuit. */
+ congested = test_bit(CONN_CONGESTED, &connection->flags);
+
+ if (!congested && cong_fill) {
+ int n = atomic_read(&connection->ap_in_flight) +
+ atomic_read(&connection->rs_in_flight);
+ if (n >= cong_fill) {
+ drbd_info(device, "Congestion-fill threshold reached (%d >= %d)\n", n, cong_fill);
+ congested = true;
+ }
}
- if (device->act_log->used >= nc->cong_extents) {
- drbd_info(device, "Congestion-extents threshold reached\n");
+ if (!congested && device->act_log->used >= cong_extents) {
+ drbd_info(device, "Congestion-extents threshold reached (%d >= %d)\n",
+ device->act_log->used, cong_extents);
congested = true;
}
if (congested) {
- /* start a new epoch for non-mirrored writes */
- start_new_tl_epoch(first_peer_device(device)->connection);
-
- if (on_congestion == OC_PULL_AHEAD)
- _drbd_set_state(_NS(device, conn, C_AHEAD), 0, NULL);
- else /*nc->on_congestion == OC_DISCONNECT */
- _drbd_set_state(_NS(device, conn, C_DISCONNECTING), 0, NULL);
+ set_bit(CONN_CONGESTED, &connection->flags);
+ drbd_peer_device_post_work(peer_device, HANDLE_CONGESTION);
+ } else {
+ clear_bit(HANDLING_CONGESTION, peer_device->flags);
}
+out:
put_ldev(device);
}
-/* If this returns false, and req->private_bio is still set,
- * this should be submitted locally.
+static void maybe_pull_ahead(struct drbd_device *device)
+{
+ struct drbd_connection *connection;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, device->resource)
+ if (connection->cstate[NOW] == C_CONNECTED)
+ __maybe_pull_ahead(device, connection);
+ rcu_read_unlock();
+}
+
+bool drbd_should_do_remote(struct drbd_peer_device *peer_device, enum which_state which)
+{
+ enum drbd_disk_state peer_disk_state = peer_device->disk_state[which];
+ enum drbd_repl_state repl_state = peer_device->repl_state[which];
+ bool replication = peer_device->replication[which];
+
+ return peer_disk_state == D_UP_TO_DATE ||
+ (peer_disk_state == D_INCONSISTENT && replication &&
+ (repl_state == L_ESTABLISHED ||
+ (repl_state >= L_WF_BITMAP_T && repl_state < L_AHEAD)));
+ /* Before proto 96 that was >= CONNECTED instead of >= L_WF_BITMAP_T.
+ That is equivalent since before 96 IO was frozen in the L_WF_BITMAP*
+ states. */
+}
+
+static bool drbd_should_send_out_of_sync(struct drbd_peer_device *peer_device)
+{
+ enum drbd_disk_state peer_disk_state = peer_device->disk_state[NOW];
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+ bool replication = peer_device->replication[NOW];
+
+ return repl_state == L_AHEAD ||
+ repl_state == L_WF_BITMAP_S ||
+ (repl_state >= L_ESTABLISHED &&
+ (peer_disk_state == D_OUTDATED ||
+ (peer_disk_state == D_INCONSISTENT && !replication)));
+
+ /* proto 96 check omitted, there was no L_AHEAD back then,
+ * peer disk was never Outdated while connection was established,
+ * and IO was frozen during bitmap exchange */
+}
+
+/* Prefer to read from protcol C peers, then B, last A */
+static u64 calc_nodes_to_read_from(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ u64 candidates[DRBD_PROT_C] = {};
+ int wp;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ struct drbd_net_conf *nc;
+
+ if (peer_device->disk_state[NOW] != D_UP_TO_DATE)
+ continue;
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ if (!nc || !nc->allow_remote_read)
+ continue;
+ wp = nc->wire_protocol;
+ candidates[wp - 1] |= NODE_MASK(peer_device->node_id);
+ }
+ rcu_read_unlock();
+
+ for (wp = DRBD_PROT_C; wp >= DRBD_PROT_A; wp--) {
+ if (candidates[wp - 1])
+ return candidates[wp - 1];
+ }
+ return 0;
+}
+
+/* If this returns NULL, and req->private_bio is still set,
+ * the request should be submitted locally.
*
- * If it returns false, but req->private_bio is not set,
+ * If it returns NULL, but req->private_bio is not set,
* we do not have access to good data :(
*
* Otherwise, this destroys req->private_bio, if any,
- * and returns true.
+ * and returns the peer device which should be asked for data.
*/
-static bool do_remote_read(struct drbd_request *req)
+static struct drbd_peer_device *find_peer_device_for_read(struct drbd_request *req)
{
+ struct drbd_peer_device *peer_device;
struct drbd_device *device = req->device;
- enum drbd_read_balancing rbm;
+ enum drbd_read_balancing rbm = RB_PREFER_REMOTE;
if (req->private_bio) {
+ /* ldev_safe: have private_bio */
if (!drbd_may_do_local_read(device,
req->i.sector, req->i.size)) {
bio_put(req->private_bio);
@@ -1047,90 +1810,123 @@ static bool do_remote_read(struct drbd_request *req)
}
}
- if (device->state.pdsk != D_UP_TO_DATE)
- return false;
-
- if (req->private_bio == NULL)
- return true;
-
- /* TODO: improve read balancing decisions, take into account drbd
- * protocol, pending requests etc. */
-
- rcu_read_lock();
- rbm = rcu_dereference(device->ldev->disk_conf)->read_balancing;
- rcu_read_unlock();
+ if (device->disk_state[NOW] > D_DISKLESS) {
+ rcu_read_lock();
+ /* ldev_safe: checked disk_state while holding state_rwlock */
+ rbm = rcu_dereference(device->ldev->disk_conf)->read_balancing;
+ rcu_read_unlock();
+ if (rbm == RB_PREFER_LOCAL && req->private_bio)
+ return NULL; /* submit locally */
+ }
- if (rbm == RB_PREFER_LOCAL && req->private_bio)
- return false; /* submit locally */
+ /* TODO: improve read balancing decisions, allow user to configure node weights */
+ while (true) {
+ if (!device->read_nodes)
+ device->read_nodes = calc_nodes_to_read_from(device);
+ if (device->read_nodes) {
+ int peer_node_id = __ffs64(device->read_nodes);
- if (remote_due_to_read_balancing(device, req->i.sector, rbm)) {
- if (req->private_bio) {
- bio_put(req->private_bio);
- req->private_bio = NULL;
- put_ldev(device);
+ device->read_nodes &= ~NODE_MASK(peer_node_id);
+ peer_device = peer_device_by_node_id(device, peer_node_id);
+ if (!peer_device)
+ continue;
+ if (peer_device->disk_state[NOW] != D_UP_TO_DATE)
+ continue;
+ if (req->private_bio &&
+ !remote_due_to_read_balancing(device, peer_device, req->i.sector, rbm))
+ peer_device = NULL;
+ } else {
+ peer_device = NULL;
}
- return true;
+ break;
}
- return false;
+ if (peer_device && req->private_bio) {
+ bio_put(req->private_bio);
+ req->private_bio = NULL;
+ put_ldev(device);
+ }
+ return peer_device;
}
-bool drbd_should_do_remote(union drbd_dev_state s)
+static int drbd_process_empty_flush(struct drbd_request *req)
{
- return s.pdsk == D_UP_TO_DATE ||
- (s.pdsk >= D_INCONSISTENT &&
- s.conn >= C_WF_BITMAP_T &&
- s.conn < C_AHEAD);
- /* Before proto 96 that was >= CONNECTED instead of >= C_WF_BITMAP_T.
- That is equivalent since before 96 IO was frozen in the C_WF_BITMAP*
- states. */
-}
+ struct drbd_device *device = req->device;
+ struct drbd_peer_device *peer_device;
+ int count = 0;
+
+ for_each_peer_device(peer_device, device) {
+ /* When a flush is submitted, the expectation is that the data
+ * is written somewhere in a usable form. Hence only
+ * D_UP_TO_DATE peers are included and not all peers that
+ * receive the data. */
+ if (peer_device->disk_state[NOW] == D_UP_TO_DATE) {
+ ++count;
+
+ /* An empty flush indicates that all previously
+ * completed requests should be written out to stable
+ * storage. Request completion already triggers a
+ * barrier to be sent and the current epoch closed. The
+ * barrier causes the data to be written out unless
+ * that is configured not to be necessary.
+ *
+ * Hence there is nothing more to be done to cause the
+ * writing out to persistent storage which was
+ * requested. We just mark the request so that we know
+ * that a flush has effectively occurred on this peer
+ * so that we can complete it successfully.
+ *
+ * We _should_ wait for any outstanding barriers to
+ * protocol C peers to be acked before completing this
+ * request, so that we are sure that the previously
+ * completed requests have really been written out
+ * there too. However, DRBD has never yet implemented
+ * this. */
+ _req_mod(req, BARRIER_SENT, peer_device);
+ }
+ }
-static bool drbd_should_send_out_of_sync(union drbd_dev_state s)
-{
- return s.conn == C_AHEAD || s.conn == C_WF_BITMAP_S;
- /* pdsk = D_INCONSISTENT as a consequence. Protocol 96 check not necessary
- since we enter state C_AHEAD only if proto >= 96 */
+ return count;
}
-/* returns number of connections (== 1, for drbd 8.4)
- * expected to actually write this data,
+/* returns the number of connections expected to actually write this data,
* which does NOT include those that we are L_AHEAD for. */
static int drbd_process_write_request(struct drbd_request *req)
{
struct drbd_device *device = req->device;
- struct drbd_peer_device *peer_device = first_peer_device(device);
+ struct drbd_peer_device *peer_device;
int remote, send_oos;
+ int count = 0;
- remote = drbd_should_do_remote(device->state);
- send_oos = drbd_should_send_out_of_sync(device->state);
+ for_each_peer_device(peer_device, device) {
+ remote = drbd_should_do_remote(peer_device, NOW);
+ send_oos = drbd_should_send_out_of_sync(peer_device);
- /* Need to replicate writes. Unless it is an empty flush,
- * which is better mapped to a DRBD P_BARRIER packet,
- * also for drbd wire protocol compatibility reasons.
- * If this was a flush, just start a new epoch.
- * Unless the current epoch was empty anyways, or we are not currently
- * replicating, in which case there is no point. */
- if (unlikely(req->i.size == 0)) {
- /* The only size==0 bios we expect are empty flushes. */
- D_ASSERT(device, req->master_bio->bi_opf & REQ_PREFLUSH);
- if (remote)
- _req_mod(req, QUEUE_AS_DRBD_BARRIER, peer_device);
- return remote;
- }
+ if (!remote && !send_oos)
+ continue;
+
+ D_ASSERT(device, !(remote && send_oos));
- if (!remote && !send_oos)
- return 0;
+ if (remote) {
+ ++count;
+ _req_mod(req, NEW_NET_WRITE, peer_device);
+ } else
+ _req_mod(req, NEW_NET_OOS, peer_device);
+ }
- D_ASSERT(device, !(remote && send_oos));
+ return count;
+}
- if (remote) {
- _req_mod(req, TO_BE_SENT, peer_device);
- _req_mod(req, QUEUE_FOR_NET_WRITE, peer_device);
- } else if (drbd_set_out_of_sync(peer_device, req->i.sector, req->i.size))
- _req_mod(req, QUEUE_FOR_SEND_OOS, peer_device);
+static void drbd_request_ready_for_net(struct drbd_request *req)
+{
+ struct drbd_device *device = req->device;
+ struct drbd_peer_device *peer_device;
- return remote;
+ for_each_peer_device(peer_device, device) {
+ /* Do not mark RQ_NET_PENDING_OOS requests ready yet */
+ if (req->net_rq_state[peer_device->node_id] & RQ_NET_PENDING)
+ _req_mod(req, READY_FOR_NET, peer_device);
+ }
}
static void drbd_process_discard_or_zeroes_req(struct drbd_request *req, int flags)
@@ -1156,90 +1952,152 @@ drbd_submit_req_private_bio(struct drbd_request *req)
else
type = DRBD_FAULT_DT_RD;
+ /* ldev_safe: req->private_bio implies an ldev reference is held */
+ bio_set_dev(bio, device->ldev->backing_bdev);
+
/* State may have changed since we grabbed our reference on the
- * ->ldev member. Double check, and short-circuit to endio.
+ * device->ldev member. Double check, and short-circuit to endio.
* In case the last activity log transaction failed to get on
* stable storage, and this is a WRITE, we may not even submit
* this bio. */
if (get_ldev(device)) {
- if (drbd_insert_fault(device, type))
- bio_io_error(bio);
- else if (bio_op(bio) == REQ_OP_WRITE_ZEROES)
+ if (bio_op(bio) != REQ_OP_READ)
+ drbd_cbt_mark(device, req->i.sector, req->i.size >> SECTOR_SHIFT);
+
+ if (drbd_insert_fault(device, type)) {
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ } else if (bio_op(bio) == REQ_OP_WRITE_ZEROES) {
drbd_process_discard_or_zeroes_req(req, EE_ZEROOUT |
((bio->bi_opf & REQ_NOUNMAP) ? 0 : EE_TRIM));
- else if (bio_op(bio) == REQ_OP_DISCARD)
+ } else if (bio_op(bio) == REQ_OP_DISCARD) {
drbd_process_discard_or_zeroes_req(req, EE_TRIM);
- else
+ } else {
submit_bio_noacct(bio);
+ }
put_ldev(device);
- } else
- bio_io_error(bio);
+ } else {
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ }
+}
+
+static void drbd_queue_write(struct drbd_device *device, struct drbd_request *req)
+{
+ if (req->local_rq_state & RQ_WAIT_FOR_AL_ECNT) {
+ req->local_rq_state |= RQ_AP_ACTLOG_CNT;
+ atomic_inc(&device->ap_actlog_cnt);
+ }
+ spin_lock_irq(&device->pending_completion_lock);
+ list_add_tail(&req->req_pending_master_completion,
+ &device->pending_master_completion[1 /* WRITE */]);
+ spin_unlock_irq(&device->pending_completion_lock);
+ spin_lock(&device->submit.lock);
+ list_add_tail(&req->list, &device->submit.writes);
+ spin_unlock(&device->submit.lock);
+ queue_work(device->submit.wq, &device->submit.worker);
+ /* do_submit() may sleep internally on al_wait, too */
+ wake_up(&device->al_wait);
+}
+
+static void drbd_req_al_ecnt_done(struct drbd_request *req)
+{
+ if (!(req->local_rq_state & RQ_WAIT_FOR_AL_ECNT))
+ return;
+
+ req->local_rq_state &= ~RQ_WAIT_FOR_AL_ECNT;
+ atomic_sub(interval_to_al_extents(&req->i), &req->device->wait_for_actlog_ecnt);
}
-static void drbd_queue_write(struct drbd_device *device, struct drbd_request *req)
+static void drbd_req_in_actlog(struct drbd_request *req)
{
- spin_lock_irq(&device->resource->req_lock);
- list_add_tail(&req->tl_requests, &device->submit.writes);
- list_add_tail(&req->req_pending_master_completion,
- &device->pending_master_completion[1 /* WRITE */]);
- spin_unlock_irq(&device->resource->req_lock);
- queue_work(device->submit.wq, &device->submit.worker);
- /* do_submit() may sleep internally on al_wait, too */
- wake_up(&device->al_wait);
+ req->local_rq_state |= RQ_IN_ACT_LOG;
+ ktime_get_accounting(req->in_actlog_kt);
+ drbd_req_al_ecnt_done(req);
}
-/* returns the new drbd_request pointer, if the caller is expected to
- * drbd_send_and_submit() it (to save latency), or NULL if we queued the
- * request on the submitter thread.
+/* returns the new drbd_request pointer, if the caller is expected to submit it
+ * (to save latency), or NULL if we queued the request on the submitter thread.
* Returns ERR_PTR(-ENOMEM) if we cannot allocate a drbd_request.
*/
+#ifndef CONFIG_DRBD_TIMING_STATS
+#define drbd_request_prepare(d, b, k, j) drbd_request_prepare(d, b, j)
+#endif
static struct drbd_request *
-drbd_request_prepare(struct drbd_device *device, struct bio *bio)
+drbd_request_prepare(struct drbd_device *device, struct bio *bio,
+ ktime_t start_kt,
+ unsigned long start_jif)
{
const int rw = bio_data_dir(bio);
struct drbd_request *req;
+ bool al_suspended;
/* allocate outside of all locks; */
req = drbd_req_new(device, bio);
if (!req) {
- dec_ap_bio(device);
- /* only pass the error to the upper layers.
- * if user cannot handle io errors, that's not our business. */
drbd_err(device, "could not kmalloc() req\n");
- bio->bi_status = BLK_STS_RESOURCE;
- bio_endio(bio);
- return ERR_PTR(-ENOMEM);
+ goto no_mem;
}
/* Update disk stats */
req->start_jif = bio_start_io_acct(req->master_bio);
if (get_ldev(device)) {
- req->private_bio = bio_alloc_clone(device->ldev->backing_bdev,
- bio, GFP_NOIO,
- &drbd_io_bio_set);
+ req->private_bio = bio_alloc_clone(device->ldev->backing_bdev, bio, GFP_NOIO, &drbd_io_bio_set);
+ if (!req->private_bio) {
+ drbd_err(device, "could not bio_alloc_clone() req->private_bio\n");
+ kfree(req);
+ put_ldev(device);
+ goto no_mem;
+ }
req->private_bio->bi_private = req;
req->private_bio->bi_end_io = drbd_request_endio;
}
+ ktime_get_accounting_assign(req->start_kt, start_kt);
+
+ if (rw != WRITE || req->i.size == 0)
+ return req;
+
+ /* Let the activity log know we are about to use it...
+ * FIXME
+ * Needs to slow down to not congest on the activity log, in case we
+ * have multiple primaries and the peer sends huge scattered epochs.
+ * See also how peer_requests are handled
+ * in receive_Data() { ... drbd_wait_for_activity_log_extents(); ... }
+ */
+ al_suspended = test_bit(AL_SUSPENDED, &device->flags);
+ if (req->private_bio && !al_suspended) {
+ req->local_rq_state |= RQ_WAIT_FOR_AL_ECNT;
+ atomic_add(interval_to_al_extents(&req->i), &device->wait_for_actlog_ecnt);
+ }
+
/* process discards always from our submitter thread */
- if (bio_op(bio) == REQ_OP_WRITE_ZEROES ||
- bio_op(bio) == REQ_OP_DISCARD)
+ if ((bio_op(bio) == REQ_OP_WRITE_ZEROES) ||
+ (bio_op(bio) == REQ_OP_DISCARD))
goto queue_for_submitter_thread;
- if (rw == WRITE && req->private_bio && req->i.size
- && !test_bit(AL_SUSPENDED, &device->flags)) {
+ if (req->private_bio && !al_suspended) {
+ /* ldev_safe: have private_bio */
if (!drbd_al_begin_io_fastpath(device, &req->i))
goto queue_for_submitter_thread;
- req->rq_state |= RQ_IN_ACT_LOG;
- req->in_actlog_jif = jiffies;
+ drbd_req_in_actlog(req);
}
return req;
queue_for_submitter_thread:
- atomic_inc(&device->ap_actlog_cnt);
+ ktime_aggregate_delta(device, req->start_kt, before_queue_kt);
drbd_queue_write(device, req);
return NULL;
+
+ no_mem:
+ dec_ap_bio(device, rw);
+ /* only pass the error to the upper layers.
+ * if user cannot handle io errors, that's not our business.
+ */
+ bio->bi_status = BLK_STS_RESOURCE;
+ bio_endio(bio);
+ return ERR_PTR(-ENOMEM);
}
/* Require at least one path to current data.
@@ -1254,8 +2112,17 @@ drbd_request_prepare(struct drbd_device *device, struct bio *bio)
*/
static bool may_do_writes(struct drbd_device *device)
{
- const union drbd_dev_state s = device->state;
- return s.disk == D_UP_TO_DATE || s.pdsk == D_UP_TO_DATE;
+ struct drbd_peer_device *peer_device;
+
+ if (device->disk_state[NOW] == D_UP_TO_DATE)
+ return true;
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->disk_state[NOW] == D_UP_TO_DATE)
+ return true;
+ }
+
+ return false;
}
struct drbd_plug_cb {
@@ -1267,24 +2134,28 @@ struct drbd_plug_cb {
static void drbd_unplug(struct blk_plug_cb *cb, bool from_schedule)
{
struct drbd_plug_cb *plug = container_of(cb, struct drbd_plug_cb, cb);
- struct drbd_resource *resource = plug->cb.data;
struct drbd_request *req = plug->most_recent_req;
+ struct drbd_resource *resource;
kfree(cb);
if (!req)
return;
- spin_lock_irq(&resource->req_lock);
+ resource = req->device->resource;
+
+ read_lock_irq(&resource->state_rwlock);
/* In case the sender did not process it yet, raise the flag to
* have it followed with P_UNPLUG_REMOTE just after. */
- req->rq_state |= RQ_UNPLUG;
+ spin_lock(&req->rq_lock);
+ req->local_rq_state |= RQ_UNPLUG;
+ spin_unlock(&req->rq_lock);
/* but also queue a generic unplug */
drbd_queue_unplug(req->device);
kref_put(&req->kref, drbd_req_destroy);
- spin_unlock_irq(&resource->req_lock);
+ read_unlock_irq(&resource->state_rwlock);
}
-static struct drbd_plug_cb* drbd_check_plugged(struct drbd_resource *resource)
+static struct drbd_plug_cb *drbd_check_plugged(struct drbd_resource *resource)
{
/* A lot of text to say
* return (struct drbd_plug_cb*)blk_check_plugged(); */
@@ -1301,40 +2172,34 @@ static struct drbd_plug_cb* drbd_check_plugged(struct drbd_resource *resource)
static void drbd_update_plug(struct drbd_plug_cb *plug, struct drbd_request *req)
{
struct drbd_request *tmp = plug->most_recent_req;
- /* Will be sent to some peer.
- * Remember to tag it with UNPLUG_REMOTE on unplug */
+ /* Will be sent to some peer. */
kref_get(&req->kref);
plug->most_recent_req = req;
if (tmp)
kref_put(&tmp->kref, drbd_req_destroy);
}
-static void drbd_send_and_submit(struct drbd_device *device, struct drbd_request *req)
+static void drbd_send_and_submit(struct drbd_request *req)
{
+ struct drbd_device *device = req->device;
struct drbd_resource *resource = device->resource;
- struct drbd_peer_device *peer_device = first_peer_device(device);
+ struct drbd_peer_device *peer_device = NULL; /* for read */
const int rw = bio_data_dir(req->master_bio);
struct bio_and_error m = { NULL, };
bool no_remote = false;
bool submit_private_bio = false;
- spin_lock_irq(&resource->req_lock);
- if (rw == WRITE) {
- /* This may temporarily give up the req_lock,
- * but will re-aquire it before it returns here.
- * Needs to be before the check on drbd_suspended() */
- complete_conflicting_writes(req);
- /* no more giving up req_lock from now on! */
+ read_lock_irq(&resource->state_rwlock);
+ if (rw == WRITE) {
/* check for congestion, and potentially stop sending
* full data updates, but start sending "dirty bits" only. */
maybe_pull_ahead(device);
}
-
if (drbd_suspended(device)) {
/* push back and retry: */
- req->rq_state |= RQ_POSTPONED;
+ req->local_rq_state |= RQ_POSTPONED;
if (req->private_bio) {
bio_put(req->private_bio);
req->private_bio = NULL;
@@ -1343,76 +2208,144 @@ static void drbd_send_and_submit(struct drbd_device *device, struct drbd_request
goto out;
}
- /* We fail READ early, if we can not serve it.
- * We must do this before req is registered on any lists.
- * Otherwise, drbd_req_complete() will queue failed READ for retry. */
- if (rw != WRITE) {
- if (!do_remote_read(req) && !req->private_bio)
+ if (rw == WRITE) {
+ if (!may_do_writes(device)) {
+ if (req->private_bio) {
+ bio_put(req->private_bio);
+ req->private_bio = NULL;
+ put_ldev(device);
+ }
+ goto nodata;
+ }
+ } else {
+ /* We fail READ early, if we can not serve it.
+ * We must do this before req is registered on any lists.
+ * Otherwise, drbd_req_complete() will queue failed READ for retry. */
+ peer_device = find_peer_device_for_read(req);
+ if (!peer_device && !req->private_bio)
goto nodata;
}
- /* which transfer log epoch does this belong to? */
- req->epoch = atomic_read(&first_peer_device(device)->connection->current_tle_nr);
+ spin_lock(&resource->tl_update_lock); /* local irq already disabled */
+ if (rw == WRITE) {
+ /* Update dagtag_sector before determining current_tle_nr so
+ * that senders can detect if there are requests currently
+ * being submitted. Updates are protected by tl_update_lock,
+ * but reads are not, so WRITE_ONCE(). */
+ WRITE_ONCE(resource->dagtag_sector, resource->dagtag_sector + (req->i.size >> 9));
+ /* Ensure that the written value is visible to the senders. */
+ smp_wmb();
+ }
+ req->dagtag_sector = resource->dagtag_sector;
- /* no point in adding empty flushes to the transfer log,
- * they are mapped to drbd barriers already. */
- if (likely(req->i.size!=0)) {
- if (rw == WRITE)
- first_peer_device(device)->connection->current_tle_writes++;
+ spin_lock(&resource->current_tle_lock);
+ /* which transfer log epoch does this belong to? */
+ req->epoch = atomic_read(&resource->current_tle_nr);
+ if (rw == WRITE && likely(req->i.size != 0))
+ resource->current_tle_writes++;
+ spin_unlock(&resource->current_tle_lock);
- list_add_tail(&req->tl_requests, &first_peer_device(device)->connection->transfer_log);
- }
+ /* A size==0 bio can only be an empty flush, which is mapped to a DRBD
+ * P_BARRIER packet. */
+ if (unlikely(req->i.size == 0)) {
+ /* The only size==0 bios we expect are empty flushes. */
+ D_ASSERT(device, req->master_bio->bi_opf & REQ_PREFLUSH);
- if (rw == WRITE) {
- if (req->private_bio && !may_do_writes(device)) {
- bio_put(req->private_bio);
- req->private_bio = NULL;
- put_ldev(device);
- goto nodata;
- }
- if (!drbd_process_write_request(req))
+ if (!drbd_process_empty_flush(req))
no_remote = true;
} else {
- /* We either have a private_bio, or we can read from remote.
- * Otherwise we had done the goto nodata above. */
- if (req->private_bio == NULL) {
- _req_mod(req, TO_BE_SENT, peer_device);
- _req_mod(req, QUEUE_FOR_NET_READ, peer_device);
- } else
- no_remote = true;
+ if (rw == WRITE) {
+ struct drbd_request *prev_write = resource->tl_previous_write;
+
+ resource->tl_previous_write = req;
+
+ if (prev_write) {
+ if (!test_bit(INTERVAL_DONE, &prev_write->i.flags))
+ refcount_inc(&req->done_ref);
+ refcount_inc(&req->oos_send_ref);
+ prev_write->next_write = req;
+ }
+
+ if (!drbd_process_write_request(req))
+ no_remote = true;
+ } else {
+ if (peer_device)
+ _req_mod(req, NEW_NET_READ, peer_device);
+ else
+ no_remote = true;
+ }
+
+ /* req may now be accessed by other threads - do not modify
+ * "immutable" fields after this point */
+ list_add_tail_rcu(&req->tl_requests, &resource->transfer_log);
+
+ /* Do this after adding to the transfer log so that the
+ * caching pointer req_not_net_done is set if
+ * necessary. */
+ drbd_request_ready_for_net(req);
}
+ spin_unlock(&resource->tl_update_lock);
+
+ if (rw == WRITE)
+ wake_all_senders(resource);
+ else if (peer_device)
+ wake_up(&peer_device->connection->sender_work.q_wait);
+
+ /* Confirm-before-complete: a write admitted into an unconfirmed data
+ * generation has its completion held until a barrier ack confirms the
+ * generation durable (see hold_completion_for_unconfirmed_gen). That
+ * barrier is only emitted once the transfer-log epoch advances past this
+ * write's epoch. Request completion is what normally advances it -- but
+ * that is exactly what we hold, so for a trailing or lone write the epoch
+ * would never close and the confirming barrier would never be sent. Close
+ * the epoch here so the sender emits the barrier independent of completion.
+ * Self-limiting: the first barrier ack confirms the generation, after
+ * which no further write takes this path.
+ */
+ if (hold_completion_for_unconfirmed_gen(req))
+ start_new_tl_epoch(resource);
if (no_remote == false) {
struct drbd_plug_cb *plug = drbd_check_plugged(resource);
+
if (plug)
drbd_update_plug(plug, req);
}
/* If it took the fast path in drbd_request_prepare, add it here.
* The slow path has added it already. */
+ spin_lock(&device->pending_completion_lock); /* local irq already disabled */
if (list_empty(&req->req_pending_master_completion))
list_add_tail(&req->req_pending_master_completion,
&device->pending_master_completion[rw == WRITE]);
if (req->private_bio) {
- /* needs to be marked within the same spinlock */
+ /* pre_submit_jif is used in request_timer_fn() */
req->pre_submit_jif = jiffies;
+ ktime_get_accounting(req->pre_submit_kt);
list_add_tail(&req->req_pending_local,
&device->pending_completion[rw == WRITE]);
_req_mod(req, TO_BE_SUBMITTED, NULL);
- /* but we need to give up the spinlock to submit */
+ /* needs to be marked within the same spinlock
+ * but we need to give up the spinlock to submit */
submit_private_bio = true;
- } else if (no_remote) {
+ spin_unlock(&device->pending_completion_lock);
+ } else {
+ spin_unlock(&device->pending_completion_lock);
+ if (no_remote) {
nodata:
- drbd_err_ratelimit(device,
- "IO ERROR: neither local nor remote data, sector %llu+%u\n",
- (unsigned long long)req->i.sector, req->i.size >> 9);
- /* A write may have been queued for send_oos, however.
- * So we can not simply free it, we must go through drbd_req_put_completion_ref() */
+ drbd_err_ratelimit(req->device,
+ "IO ERROR: neither local nor remote data, sector %llu+%u\n",
+ (unsigned long long)req->i.sector, req->i.size >> 9);
+ /* A write may have been queued for send_oos, however.
+ * So we can not simply free it, we must go through
+ * drbd_req_put_completion_ref()
+ */
+ }
}
out:
drbd_req_put_completion_ref(req, &m, 1);
- spin_unlock_irq(&resource->req_lock);
+ read_unlock_irq(&resource->state_rwlock);
/* Even though above is a kref_put(), this is safe.
* As long as we still need to submit our private bio,
@@ -1422,114 +2355,420 @@ static void drbd_send_and_submit(struct drbd_device *device, struct drbd_request
* That's why we cannot check on req->private_bio. */
if (submit_private_bio)
drbd_submit_req_private_bio(req);
+
if (m.bio)
complete_master_bio(device, &m);
}
-void __drbd_make_request(struct drbd_device *device, struct bio *bio)
+/* Insert the request into the tree of writes. Pass it through to be submitted
+ * if possible. Otherwise it will be submitted asynchronously via
+ * drbd_release_conflicts once the conflict has been resolved. */
+static void drbd_conflict_submit_write(struct drbd_request *req)
+{
+ struct drbd_device *device = req->device;
+ struct drbd_interval *conflict;
+
+ spin_lock_irq(&device->interval_lock);
+ clear_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &req->i.flags);
+ conflict = drbd_find_conflict(device, &req->i, 0);
+ if (drbd_interval_empty(&req->i))
+ drbd_insert_interval(&device->requests, &req->i);
+ if (conflict)
+ drbd_rs_depth_exempt(device, &req->i);
+ if (!conflict) {
+ set_bit(INTERVAL_SUBMITTED, &req->i.flags);
+ } else if (drbd_interval_is_local(conflict)) {
+ struct drbd_request *conflicting_req =
+ container_of(conflict, struct drbd_request, i);
+
+ if (conflicting_req->local_rq_state & RQ_POSTPONED) {
+ req->local_rq_state |= RQ_POSTPONED;
+
+ /*
+ * Remove interval from tree to prevent req from being
+ * queued when conflicts are released.
+ */
+ drbd_remove_interval(&device->requests, &req->i);
+ drbd_clear_interval(&req->i);
+ }
+ }
+ spin_unlock_irq(&device->interval_lock);
+
+ /*
+ * If there is a conflict, the request will be submitted once the
+ * conflict has cleared.
+ */
+ if (!conflict) {
+ drbd_send_and_submit(req);
+ } else if (req->local_rq_state & RQ_POSTPONED) {
+ if (req->private_bio) {
+ bio_put(req->private_bio);
+ req->private_bio = NULL;
+ put_ldev(device);
+ }
+ drbd_req_put_completion_ref(req, NULL, 1);
+ }
+}
+
+static bool inc_ap_bio_cond(struct drbd_device *device, int rw)
+{
+ int ap_bio_cnt;
+ bool rv;
+
+ read_lock_irq(&device->resource->state_rwlock);
+ rv = may_inc_ap_bio(device);
+ read_unlock_irq(&device->resource->state_rwlock);
+ if (!rv)
+ return false;
+
+ /* check need for new current uuid _AFTER_ ensuring IO is not suspended via may_inc_ap_bio */
+ if (drbd_gen_obligation_outstanding(device)) {
+ if (drbd_gen_obligation_mint_start(device))
+ drbd_device_post_work(device, MAKE_NEW_CUR_UUID);
+
+ return false;
+ }
+
+ do {
+ unsigned int nr_requests = device->resource->res_opts.nr_requests;
+
+ ap_bio_cnt = atomic_read(&device->ap_bio_cnt[rw]);
+ if (ap_bio_cnt >= nr_requests)
+ return false;
+ } while (atomic_cmpxchg(&device->ap_bio_cnt[rw], ap_bio_cnt, ap_bio_cnt + 1) != ap_bio_cnt);
+
+ /* Re-check suspend_cnt after publishing our ap_bio_cnt increment.
+ * atomic_cmpxchg() is a full barrier, so this load is ordered after
+ * the increment; it pairs with the smp_mb__after_atomic() in
+ * drbd_suspend_io(). If a suspend raced in, at least one side sees
+ * the other: either drbd_suspend_io() observes our increment and
+ * waits, or we observe suspend_cnt here and roll back.
+ */
+ if (atomic_read(&device->suspend_cnt)) {
+ dec_ap_bio(device, rw);
+ return false;
+ }
+
+ return true;
+}
+
+static void inc_ap_bio(struct drbd_device *device, int rw)
+{
+ /* we wait here
+ * as long as the device is suspended
+ * until the bitmap is no longer on the fly during connection
+ * handshake as long as we would exceed the max_buffer limit.
+ *
+ * to avoid races with the reconnect code,
+ * we need to atomic_inc within the spinlock. */
+
+ wait_event(device->misc_wait, inc_ap_bio_cond(device, rw));
+}
+
+void __drbd_make_request(struct drbd_device *device, struct bio *bio,
+ ktime_t start_kt,
+ unsigned long start_jif)
{
- struct drbd_request *req = drbd_request_prepare(device, bio);
+ const int rw = bio_data_dir(bio);
+ struct drbd_request *req;
+
+ inc_ap_bio(device, bio_data_dir(bio));
+ req = drbd_request_prepare(device, bio, start_kt, start_jif);
if (IS_ERR_OR_NULL(req))
return;
- drbd_send_and_submit(device, req);
+
+ if (rw == WRITE)
+ drbd_conflict_submit_write(req);
+ else
+ drbd_send_and_submit(req);
+}
+
+/* Work function to submit requests once they are released after conflicts. The
+ * queued requests are processed and, if no other conflict is found, submitted. */
+void drbd_do_submit_conflict(struct work_struct *ws)
+{
+ struct drbd_device *device = container_of(ws, struct drbd_device, submit_conflict.worker);
+ struct drbd_peer_request *peer_req, *peer_req_tmp;
+ struct drbd_request *req, *tmp;
+ LIST_HEAD(resync_writes);
+ LIST_HEAD(resync_reads);
+ LIST_HEAD(writes);
+ LIST_HEAD(peer_writes);
+
+ spin_lock_irq(&device->submit_conflict.lock);
+ list_splice_init(&device->submit_conflict.resync_writes, &resync_writes);
+ list_splice_init(&device->submit_conflict.resync_reads, &resync_reads);
+ list_splice_init(&device->submit_conflict.writes, &writes);
+ list_splice_init(&device->submit_conflict.peer_writes, &peer_writes);
+ spin_unlock_irq(&device->submit_conflict.lock);
+
+ /* Delete the list entries when iterating them so that they can be re-used
+ * for adding them to the conflict lists again once the
+ * submit_conflict_queued flag has been cleared. */
+
+ list_for_each_entry_safe(peer_req, peer_req_tmp, &resync_writes, w.list) {
+ list_del_init(&peer_req->w.list);
+ if (!test_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags))
+ drbd_conflict_send_resync_request(peer_req);
+ else
+ drbd_conflict_submit_resync_request(peer_req);
+ }
+
+ list_for_each_entry_safe(peer_req, peer_req_tmp, &resync_reads, w.list) {
+ list_del_init(&peer_req->w.list);
+ drbd_conflict_submit_peer_read(peer_req);
+ }
+
+ list_for_each_entry_safe(req, tmp, &writes, list) {
+ list_del_init(&req->list);
+ drbd_conflict_submit_write(req);
+ }
+
+ list_for_each_entry_safe(peer_req, peer_req_tmp, &peer_writes, w.list) {
+ list_del_init(&peer_req->w.list);
+ /* ldev_safe: queued peer requests hold their own ldev references */
+ drbd_conflict_submit_peer_write(peer_req);
+ }
+}
+
+/* helpers for do_submit */
+
+struct incoming_pending {
+ /* from drbd_submit_bio() or receive_Data() */
+ struct list_head incoming;
+ /* for non-blocking fill-up # of updates in the transaction */
+ struct list_head more_incoming;
+ /* to be submitted after next AL-transaction commit */
+ struct list_head pending;
+ /* need cleanup */
+ struct list_head cleanup;
+};
+
+struct waiting_for_act_log {
+ struct incoming_pending requests;
+ struct incoming_pending peer_requests;
+};
+
+static void ipb_init(struct incoming_pending *ipb)
+{
+ INIT_LIST_HEAD(&ipb->incoming);
+ INIT_LIST_HEAD(&ipb->more_incoming);
+ INIT_LIST_HEAD(&ipb->pending);
+ INIT_LIST_HEAD(&ipb->cleanup);
+}
+
+static void wfa_init(struct waiting_for_act_log *wfa)
+{
+ ipb_init(&wfa->requests);
+ ipb_init(&wfa->peer_requests);
+}
+
+#define wfa_lists_empty(_wfa, name) \
+ (list_empty(&(_wfa)->requests.name) && list_empty(&(_wfa)->peer_requests.name))
+#define wfa_splice_tail_init(_wfa, from, to) do { \
+ list_splice_tail_init(&(_wfa)->requests.from, &(_wfa)->requests.to); \
+ list_splice_tail_init(&(_wfa)->peer_requests.from, &(_wfa)->peer_requests.to); \
+ } while (0)
+
+static void __drbd_submit_peer_request(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ int err;
+
+ peer_req->flags |= EE_IN_ACTLOG;
+ atomic_sub(interval_to_al_extents(&peer_req->i), &device->wait_for_actlog_ecnt);
+ atomic_dec(&device->wait_for_actlog);
+ list_del_init(&peer_req->w.list);
+
+ err = drbd_submit_peer_request(peer_req);
+
+ if (err)
+ drbd_cleanup_after_failed_submit_peer_write(peer_req);
}
-static void submit_fast_path(struct drbd_device *device, struct list_head *incoming)
+/* The only place where a request leaves the submitter thread's queue. */
+static void drbd_submit_queued_write(struct drbd_device *device, struct drbd_request *req)
+{
+ if (req->local_rq_state & RQ_AP_ACTLOG_CNT) {
+ req->local_rq_state &= ~RQ_AP_ACTLOG_CNT;
+ atomic_dec(&device->ap_actlog_cnt);
+ }
+ drbd_req_al_ecnt_done(req);
+
+ list_del_init(&req->list);
+ drbd_conflict_submit_write(req);
+}
+
+static void submit_fast_path(struct drbd_device *device, struct waiting_for_act_log *wfa)
{
struct blk_plug plug;
struct drbd_request *req, *tmp;
+ struct drbd_peer_request *pr, *pr_tmp;
blk_start_plug(&plug);
- list_for_each_entry_safe(req, tmp, incoming, tl_requests) {
+ list_for_each_entry_safe(pr, pr_tmp, &wfa->peer_requests.incoming, w.list) {
+ if (!drbd_al_begin_io_fastpath(pr->peer_device->device, &pr->i))
+ continue;
+
+ __drbd_submit_peer_request(pr);
+ }
+ list_for_each_entry_safe(req, tmp, &wfa->requests.incoming, list) {
const int rw = bio_data_dir(req->master_bio);
- if (rw == WRITE /* rw != WRITE should not even end up here! */
- && req->private_bio && req->i.size
- && !test_bit(AL_SUSPENDED, &device->flags)) {
+ if (rw == WRITE && req->private_bio && req->i.size
+ && !test_bit(AL_SUSPENDED, &device->flags)) {
if (!drbd_al_begin_io_fastpath(device, &req->i))
continue;
- req->rq_state |= RQ_IN_ACT_LOG;
- req->in_actlog_jif = jiffies;
- atomic_dec(&device->ap_actlog_cnt);
+ drbd_req_in_actlog(req);
}
- list_del_init(&req->tl_requests);
- drbd_send_and_submit(device, req);
+ drbd_submit_queued_write(device, req);
}
blk_finish_plug(&plug);
}
+static struct drbd_request *wfa_next_request(struct waiting_for_act_log *wfa)
+{
+ struct list_head *lh = !list_empty(&wfa->requests.more_incoming) ?
+ &wfa->requests.more_incoming : &wfa->requests.incoming;
+ return list_first_entry_or_null(lh, struct drbd_request, list);
+}
+
+static struct drbd_peer_request *wfa_next_peer_request(struct waiting_for_act_log *wfa)
+{
+ struct list_head *lh = !list_empty(&wfa->peer_requests.more_incoming) ?
+ &wfa->peer_requests.more_incoming : &wfa->peer_requests.incoming;
+ return list_first_entry_or_null(lh, struct drbd_peer_request, w.list);
+}
+
static bool prepare_al_transaction_nonblock(struct drbd_device *device,
- struct list_head *incoming,
- struct list_head *pending,
- struct list_head *later)
+ struct waiting_for_act_log *wfa)
{
+ struct drbd_peer_request *peer_req;
struct drbd_request *req;
- int wake = 0;
+ bool made_progress = false;
int err;
spin_lock_irq(&device->al_lock);
- while ((req = list_first_entry_or_null(incoming, struct drbd_request, tl_requests))) {
+
+ /* Don't even try, if someone has it locked right now. */
+ if (test_bit(__LC_LOCKED, &device->act_log->flags))
+ goto out;
+
+ while ((peer_req = wfa_next_peer_request(wfa))) {
+ if (peer_req->peer_device->connection->cstate[NOW] < C_CONNECTED) {
+ list_move_tail(&peer_req->w.list, &wfa->peer_requests.cleanup);
+ made_progress = true;
+ continue;
+ }
+ err = drbd_al_begin_io_nonblock(device, &peer_req->i);
+ if (err) {
+ if (err != -ENOBUFS && drbd_ratelimit())
+ drbd_err(device, "Unexpected error %d from drbd_al_begin_io_nonblock\n", err);
+ break;
+ }
+ list_move_tail(&peer_req->w.list, &wfa->peer_requests.pending);
+ made_progress = true;
+ }
+ while ((req = wfa_next_request(wfa))) {
+ ktime_aggregate_delta(device, req->start_kt, before_al_begin_io_kt);
err = drbd_al_begin_io_nonblock(device, &req->i);
- if (err == -ENOBUFS)
+ if (err) {
+ if (err != -ENOBUFS && drbd_ratelimit())
+ drbd_err(device, "Unexpected error %d from drbd_al_begin_io_nonblock\n", err);
break;
- if (err == -EBUSY)
- wake = 1;
- if (err)
- list_move_tail(&req->tl_requests, later);
- else
- list_move_tail(&req->tl_requests, pending);
+ }
+ list_move_tail(&req->list, &wfa->requests.pending);
+ made_progress = true;
}
+ out:
spin_unlock_irq(&device->al_lock);
- if (wake)
- wake_up(&device->al_wait);
- return !list_empty(pending);
+ return made_progress;
}
-static void send_and_submit_pending(struct drbd_device *device, struct list_head *pending)
+static void send_and_submit_pending(struct drbd_device *device, struct waiting_for_act_log *wfa)
{
struct blk_plug plug;
- struct drbd_request *req;
+ struct drbd_request *req, *tmp;
+ struct drbd_peer_request *pr, *pr_tmp;
blk_start_plug(&plug);
- while ((req = list_first_entry_or_null(pending, struct drbd_request, tl_requests))) {
- req->rq_state |= RQ_IN_ACT_LOG;
- req->in_actlog_jif = jiffies;
- atomic_dec(&device->ap_actlog_cnt);
- list_del_init(&req->tl_requests);
- drbd_send_and_submit(device, req);
+ list_for_each_entry_safe(pr, pr_tmp, &wfa->peer_requests.pending, w.list) {
+ __drbd_submit_peer_request(pr);
+ }
+ list_for_each_entry_safe(req, tmp, &wfa->requests.pending, list) {
+ drbd_req_in_actlog(req);
+ drbd_submit_queued_write(device, req);
}
blk_finish_plug(&plug);
}
+/* more: for non-blocking fill-up # of updates in the transaction */
+static bool grab_new_incoming_requests(struct drbd_device *device, struct waiting_for_act_log *wfa, bool more)
+{
+ /* grab new incoming requests */
+ struct list_head *reqs = more ? &wfa->requests.more_incoming : &wfa->requests.incoming;
+ struct list_head *peer_reqs = more ? &wfa->peer_requests.more_incoming : &wfa->peer_requests.incoming;
+ bool found_new = false;
+
+ spin_lock(&device->submit.lock);
+ found_new = !list_empty(&device->submit.writes);
+ list_splice_tail_init(&device->submit.writes, reqs);
+ found_new |= !list_empty(&device->submit.peer_writes);
+ list_splice_tail_init(&device->submit.peer_writes, peer_reqs);
+ spin_unlock(&device->submit.lock);
+
+ return found_new;
+}
+
void do_submit(struct work_struct *ws)
{
struct drbd_device *device = container_of(ws, struct drbd_device, submit.worker);
- LIST_HEAD(incoming); /* from drbd_make_request() */
- LIST_HEAD(pending); /* to be submitted after next AL-transaction commit */
- LIST_HEAD(busy); /* blocked by resync requests */
+ struct waiting_for_act_log wfa;
+ bool made_progress;
- /* grab new incoming requests */
- spin_lock_irq(&device->resource->req_lock);
- list_splice_tail_init(&device->submit.writes, &incoming);
- spin_unlock_irq(&device->resource->req_lock);
+ wfa_init(&wfa);
+
+ grab_new_incoming_requests(device, &wfa, false);
for (;;) {
DEFINE_WAIT(wait);
- /* move used-to-be-busy back to front of incoming */
- list_splice_init(&busy, &incoming);
- submit_fast_path(device, &incoming);
- if (list_empty(&incoming))
+ /* ldev_safe: queued requests acquired ldev in drbd_request_prepare() */
+ submit_fast_path(device, &wfa);
+ if (wfa_lists_empty(&wfa, incoming))
break;
for (;;) {
+ /*
+ * We put ourselves on device->al_wait, then check if
+ * we can need to actually sleep and wait for someone
+ * else to make progress.
+ *
+ * We need to sleep if we cannot activate enough
+ * activity log extents for even one single request.
+ * That would mean that all (peer-)requests in our
+ * incoming lists target "cold" activity log extents,
+ * all activity log extent slots are have on-going
+ * in-flight IO (are "hot"), and no idle or free slot
+ * is available.
+ *
+ * prepare_to_wait() can internally cause a wake_up()
+ * as well, though, so this may appear to busy-loop
+ * a couple times, but should settle down quickly.
+ *
+ * When application requests make sufficient progress,
+ * some refcount on some extent will eventually drop to
+ * zero, we will be woken up, and can try to move that
+ * now idle extent to "cold", and recycle its slot for
+ * one of the extents we'd like to become hot.
+ */
prepare_to_wait(&device->al_wait, &wait, TASK_UNINTERRUPTIBLE);
- list_splice_init(&busy, &incoming);
- prepare_al_transaction_nonblock(device, &incoming, &pending, &busy);
- if (!list_empty(&pending))
+ made_progress = prepare_al_transaction_nonblock(device, &wfa);
+ if (made_progress)
break;
schedule();
@@ -1545,15 +2784,12 @@ void do_submit(struct work_struct *ws)
* effectively blocking all new requests until we made
* at least _some_ progress with what we currently have.
*/
- if (!list_empty(&incoming))
+ if (!wfa_lists_empty(&wfa, incoming))
continue;
- /* Nothing moved to pending, but nothing left
- * on incoming: all moved to busy!
- * Grab new and iterate. */
- spin_lock_irq(&device->resource->req_lock);
- list_splice_tail_init(&device->submit.writes, &incoming);
- spin_unlock_irq(&device->resource->req_lock);
+ /* Nothing moved to pending, but nothing left on
+ * incoming. Grab new and iterate. */
+ grab_new_incoming_requests(device, &wfa, false);
}
finish_wait(&device->al_wait, &wait);
@@ -1561,81 +2797,216 @@ void do_submit(struct work_struct *ws)
* had been processed, skip ahead to commit, and iterate
* without splicing in more incoming requests from upper layers.
*
- * Else, if all incoming have been processed,
- * they have become either "pending" (to be submitted after
- * next transaction commit) or "busy" (blocked by resync).
+ * Else, if all incoming have been processed, they have become
+ * "pending" (to be submitted after next transaction commit).
*
* Maybe more was queued, while we prepared the transaction?
* Try to stuff those into this transaction as well.
* Be strictly non-blocking here,
* we already have something to commit.
*
- * Commit if we don't make any more progres.
+ * Commit as soon as we don't make any more progress.
*/
- while (list_empty(&incoming)) {
- LIST_HEAD(more_pending);
- LIST_HEAD(more_incoming);
- bool made_progress;
-
+ while (wfa_lists_empty(&wfa, incoming)) {
/* It is ok to look outside the lock,
* it's only an optimization anyways */
- if (list_empty(&device->submit.writes))
+ if (list_empty(&device->submit.writes) &&
+ list_empty(&device->submit.peer_writes))
break;
- spin_lock_irq(&device->resource->req_lock);
- list_splice_tail_init(&device->submit.writes, &more_incoming);
- spin_unlock_irq(&device->resource->req_lock);
-
- if (list_empty(&more_incoming))
+ if (!grab_new_incoming_requests(device, &wfa, true))
break;
- made_progress = prepare_al_transaction_nonblock(device, &more_incoming, &more_pending, &busy);
+ made_progress = prepare_al_transaction_nonblock(device, &wfa);
- list_splice_tail_init(&more_pending, &pending);
- list_splice_tail_init(&more_incoming, &incoming);
+ wfa_splice_tail_init(&wfa, more_incoming, incoming);
if (!made_progress)
break;
}
+ if (!list_empty(&wfa.peer_requests.cleanup))
+ drbd_cleanup_peer_requests_wfa(device, &wfa.peer_requests.cleanup);
+ /* ldev_safe: queued requests acquired ldev in drbd_request_prepare() */
drbd_al_begin_io_commit(device);
- send_and_submit_pending(device, &pending);
+
+ send_and_submit_pending(device, &wfa);
+ }
+}
+
+static bool drbd_reject_write_early(struct drbd_device *device, struct bio *bio)
+{
+ struct drbd_resource *resource = device->resource;
+
+ /* If you "mount -o ro", then later "mount -o remount,rw", you can end
+ * up with a DRBD "Secondary" receiving WRITE requests from the VFS.
+ * We cannot have that. */
+
+ if (bio_data_dir(bio) == READ)
+ return false;
+
+ if (resource->role[NOW] != R_PRIMARY) {
+ /* You can fsync() on an O_RDONLY fd. Only be noisy
+ * if there is data. Ratelimit on per device "unspec"
+ * ratelimit state before kmalloc / adding the specific
+ * openers hint.
+ */
+ if (bio_has_data(bio) && drbd_device_ratelimit(device, GENERIC)) {
+ char *buf = kmalloc(128, __GFP_NORETRY);
+
+ if (buf)
+ youngest_and_oldest_opener_to_str(device, buf, 128);
+ drbd_err(device,
+ "Rejected WRITE request, not in Primary role.%s\n", buf ?: "");
+ kfree(buf);
+ }
+ return true;
+ } else if (device->open_cnt == 0) {
+ drbd_err_ratelimit(device, "WRITE request, but open_cnt == 0!\n");
+ } else if (!device->writable && bio_has_data(bio)) {
+ /*
+ * If the resource was (temporarily, auto) promoted,
+ * a remount,rw may have succeeded without marking the device
+ * open_cnt as "writable". Once we let writes through, we need
+ * _all_ openers to release(), before we attempt to auto-demote
+ * again, so we mark it writable here. Grab the open_release
+ * mutex to protect against races with new openers.
+ */
+ mutex_lock(&resource->open_release);
+ drbd_info(device, "open_cnt:%d, implicitly promoted to writable\n",
+ device->open_cnt);
+ device->writable = true;
+ mutex_unlock(&resource->open_release);
+ }
+ return false;
+}
+
+/* Check if bio is "bad", likely to be rejected by lower layers or peers:
+ * Must not be too large, must not be unaligned.
+ */
+static bool bio_bad(struct drbd_device *device, struct bio *bio)
+{
+ unsigned int bss_mask = queue_logical_block_size(device->rq_queue) / SECTOR_SIZE - 1;
+ unsigned int bs_mask = queue_logical_block_size(device->rq_queue) - 1;
+ unsigned long long sector = bio->bi_iter.bi_sector;
+ unsigned int size = bio->bi_iter.bi_size;
+
+ if (size > DRBD_MAX_BATCH_BIO_SIZE || (size & bs_mask) || (sector & bss_mask)) {
+ char comm[TASK_COMM_LEN];
+
+ get_task_comm(comm, current);
+ drbd_warn(device, "bad bio: %llu +%u 0x%x submitted by %s[%u]\n",
+ sector, size, bio->bi_opf, comm, task_pid_nr(current));
+ return true;
}
+
+ return false;
}
+/* drbd_submit_bio() - entry point for data into DRBD
+ *
+ * Request handling flow:
+ *
+ * drbd_submit_bio
+ * |
+ * v wait for AL
+ * do_retry -----------------------> __drbd_make_request --------> drbd_queue_write
+ * ^ | |
+ * | | ...
+ * | | |
+ * | | v AL extent active
+ * | drbd_do_submit_conflict --------+ do_submit ----------------+
+ * | ^ | | |
+ * ... | | v v
+ * | ... | send_and_submit_pending submit_fast_path
+ * | | v | |
+ * | +----------- drbd_conflict_submit_write <-------+--------------------+
+ * | conflict |
+ * | v
+ * drbd_restart_request <----------- drbd_send_and_submit
+ * RQ_POSTPONED |
+ * v
+ * Request state machine
+ */
void drbd_submit_bio(struct bio *bio)
{
struct drbd_device *device = bio->bi_bdev->bd_disk->private_data;
+#ifdef CONFIG_DRBD_TIMING_STATS
+ ktime_t start_kt;
+#endif
+ unsigned long start_jif;
+
+ if (drbd_reject_write_early(device, bio)) {
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ return;
+ }
bio = bio_split_to_limits(bio);
if (!bio)
return;
- /*
- * what we "blindly" assume:
+ if (device->cached_err_io || bio_bad(device, bio)) {
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ return;
+ }
+
+ /* This is both an optimization: READ of size 0, nothing to do
+ * and a workaround: (older) ZFS explodes on size zero reads, see
+ * https://github.com/zfsonlinux/zfs/issues/8379
+ * Actually don't do anything for size zero bios.
+ * Add a "WARN_ONCE", so we can tell the caller to stop doing this.
*/
- D_ASSERT(device, IS_ALIGNED(bio->bi_iter.bi_size, 512));
+ if (bio_op(bio) == REQ_OP_READ && bio->bi_iter.bi_size == 0) {
+ WARN_ONCE(1, "size zero read from upper layers");
+ bio_endio(bio);
+ return;
+ }
+
+ ktime_get_accounting(start_kt);
+ start_jif = jiffies;
- inc_ap_bio(device);
- __drbd_make_request(device, bio);
+ __drbd_make_request(device, bio, start_kt, start_jif);
+}
+
+static unsigned long time_min_in_future(unsigned long now,
+ unsigned long t1, unsigned long t2)
+{
+ bool t1_in_future = time_after(t1, now);
+ bool t2_in_future = time_after(t2, now);
+
+ /* Ensure that we never return a time in the past. */
+ t1 = t1_in_future ? t1 : now;
+ t2 = t2_in_future ? t2 : now;
+
+ if (!t1_in_future)
+ return t2;
+
+ if (!t2_in_future)
+ return t1;
+
+ return time_after(t1, t2) ? t2 : t1;
}
static bool net_timeout_reached(struct drbd_request *net_req,
- struct drbd_connection *connection,
+ struct drbd_peer_device *peer_device,
unsigned long now, unsigned long ent,
unsigned int ko_count, unsigned int timeout)
{
- struct drbd_device *device = net_req->device;
+ struct drbd_connection *connection = peer_device->connection;
+ int peer_node_id = peer_device->node_id;
+ unsigned long pre_send_jif = net_req->pre_send_jif[peer_node_id];
- if (!time_after(now, net_req->pre_send_jif + ent))
+ if (!time_after(now, pre_send_jif + ent))
return false;
if (time_in_range(now, connection->last_reconnect_jif, connection->last_reconnect_jif + ent))
return false;
- if (net_req->rq_state & RQ_NET_PENDING) {
- drbd_warn(device, "Remote failed to finish a request within %ums > ko-count (%u) * timeout (%u * 0.1s)\n",
- jiffies_to_msecs(now - net_req->pre_send_jif), ko_count, timeout);
+ if (net_req->net_rq_state[peer_node_id] & RQ_NET_PENDING) {
+ drbd_warn(peer_device, "Remote failed to finish a request within %ums > ko-count (%u) * timeout (%u * 0.1s)\n",
+ jiffies_to_msecs(now - pre_send_jif), ko_count, timeout);
return true;
}
@@ -1644,9 +3015,12 @@ static bool net_timeout_reached(struct drbd_request *net_req,
* Check if we sent the barrier already. We should not blame the peer
* for being unresponsive, if we did not even ask it yet. */
if (net_req->epoch == connection->send.current_epoch_nr) {
- drbd_warn(device,
- "We did not send a P_BARRIER for %ums > ko-count (%u) * timeout (%u * 0.1s); drbd kernel thread blocked?\n",
- jiffies_to_msecs(now - net_req->pre_send_jif), ko_count, timeout);
+ /* It is OK for the barrier to be delayed for a long time for a
+ * suspended request. */
+ if (!(net_req->local_rq_state & RQ_COMPLETION_SUSP))
+ drbd_warn(peer_device,
+ "We did not send a P_BARRIER for %ums > ko-count (%u) * timeout (%u * 0.1s); drbd kernel thread blocked?\n",
+ jiffies_to_msecs(now - pre_send_jif), ko_count, timeout);
return false;
}
@@ -1667,7 +3041,7 @@ static bool net_timeout_reached(struct drbd_request *net_req,
* barrier packet is relevant enough.
*/
if (time_after(now, connection->send.last_sent_barrier_jif + ent)) {
- drbd_warn(device, "Remote failed to answer a P_BARRIER (sent at %lu jif; now=%lu jif) within %ums > ko-count (%u) * timeout (%u * 0.1s)\n",
+ drbd_warn(peer_device, "Remote failed to answer a P_BARRIER (sent at %lu jif; now=%lu jif) within %ums > ko-count (%u) * timeout (%u * 0.1s)\n",
connection->send.last_sent_barrier_jif, now,
jiffies_to_msecs(now - connection->send.last_sent_barrier_jif), ko_count, timeout);
return true;
@@ -1684,7 +3058,7 @@ static bool net_timeout_reached(struct drbd_request *net_req,
* - the connection was established (resp. disk was attached)
* for longer than the timeout already.
* Note that for 32bit jiffies and very stable connections/disks,
- * we may have a wrap around, which is catched by
+ * we may have a wrap around, which is caught by
* !time_in_range(now, last_..._jif, last_..._jif + timeout).
*
* Side effect: once per 32bit wrap-around interval, which means every
@@ -1694,92 +3068,200 @@ static bool net_timeout_reached(struct drbd_request *net_req,
void request_timer_fn(struct timer_list *t)
{
- struct drbd_device *device = timer_container_of(device, t,
- request_timer);
- struct drbd_connection *connection = first_peer_device(device)->connection;
- struct drbd_request *req_read, *req_write, *req_peer; /* oldest request */
- struct net_conf *nc;
- unsigned long oldest_submit_jif;
- unsigned long ent = 0, dt = 0, et, nt; /* effective timeout = ko_count * timeout */
- unsigned long now;
- unsigned int ko_count = 0, timeout = 0;
+ struct drbd_device *device = timer_container_of(device, t, request_timer);
+ struct drbd_resource *resource = device->resource;
+ struct drbd_connection *connection;
+ struct drbd_request *req_read, *req_write;
+ unsigned long oldest_submit_jif, irq_flags;
+ unsigned long disk_timeout = 0, effective_timeout = 0, now = jiffies, next_trigger_time = now;
+ bool restart_timer = false, io_error = false;
+ unsigned long timeout_peers = 0;
+ int node_id;
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- if (nc && device->state.conn >= C_WF_REPORT_PARAMS) {
- ko_count = nc->ko_count;
- timeout = nc->timeout;
- }
-
if (get_ldev(device)) { /* implicit state.disk >= D_INCONSISTENT */
- dt = rcu_dereference(device->ldev->disk_conf)->disk_timeout * HZ / 10;
+ disk_timeout = rcu_dereference(device->ldev->disk_conf)->disk_timeout * HZ / 10;
put_ldev(device);
}
rcu_read_unlock();
+ /* FIXME right now, this basically does a full transfer log walk *every time* */
+ read_lock_irq(&resource->state_rwlock);
+ if (disk_timeout) {
+ unsigned long write_pre_submit_jif = 0, read_pre_submit_jif = 0;
+
+ spin_lock(&device->pending_completion_lock); /* local irq already disabled */
+ req_read = list_first_entry_or_null(&device->pending_completion[0], struct drbd_request, req_pending_local);
+ req_write = list_first_entry_or_null(&device->pending_completion[1], struct drbd_request, req_pending_local);
+ spin_unlock(&device->pending_completion_lock);
+
+ if (req_write)
+ write_pre_submit_jif = req_write->pre_submit_jif;
+ if (req_read)
+ read_pre_submit_jif = req_read->pre_submit_jif;
+ oldest_submit_jif =
+ (req_write && req_read)
+ ? (time_before(write_pre_submit_jif, read_pre_submit_jif)
+ ? write_pre_submit_jif : read_pre_submit_jif)
+ : req_write ? write_pre_submit_jif
+ : req_read ? read_pre_submit_jif : now;
+
+ if (device->disk_state[NOW] > D_FAILED) {
+ effective_timeout = min_not_zero(effective_timeout, disk_timeout);
+ next_trigger_time = time_min_in_future(now,
+ next_trigger_time, oldest_submit_jif + disk_timeout);
+ restart_timer = true;
+ }
- ent = timeout * HZ/10 * ko_count;
- et = min_not_zero(dt, ent);
+ if (time_after(now, oldest_submit_jif + disk_timeout) &&
+ !time_in_range(now, device->last_reattach_jif, device->last_reattach_jif + disk_timeout))
+ io_error = true;
+ }
+ for_each_connection(connection, resource) {
+ struct drbd_peer_device *peer_device = conn_peer_device(connection, device->vnr);
+ struct drbd_net_conf *nc;
+ struct drbd_request *req;
+ unsigned long effective_net_timeout = 0;
+ unsigned long pre_send_jif = now;
+ unsigned int ko_count = 0, timeout = 0;
- if (!et)
- return; /* Recurring timer stopped */
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (nc) {
+ /* effective timeout = ko_count * timeout */
+ if (connection->cstate[NOW] == C_CONNECTED) {
+ ko_count = nc->ko_count;
+ timeout = nc->timeout;
+ effective_net_timeout = timeout * HZ/10 * ko_count;
+ }
+ }
+ rcu_read_unlock();
- now = jiffies;
- nt = now + et;
+ /* This connection is not established,
+ * or has the effective timeout disabled.
+ * no timer restart needed (for this connection). */
+ if (!effective_net_timeout)
+ continue;
- spin_lock_irq(&device->resource->req_lock);
- req_read = list_first_entry_or_null(&device->pending_completion[0], struct drbd_request, req_pending_local);
- req_write = list_first_entry_or_null(&device->pending_completion[1], struct drbd_request, req_pending_local);
+ /* maybe the oldest request waiting for the peer is in fact still
+ * blocking in tcp sendmsg. That's ok, though, that's handled via the
+ * socket send timeout, requesting a ping, and bumping ko-count in
+ * drbd_stream_send_timed_out().
+ */
- /* maybe the oldest request waiting for the peer is in fact still
- * blocking in tcp sendmsg. That's ok, though, that's handled via the
- * socket send timeout, requesting a ping, and bumping ko-count in
- * we_should_drop_the_connection().
- */
+ /* check the oldest request we did successfully sent,
+ * but which is still waiting for an ACK. */
+ req = connection->req_ack_pending;
+
+ /* If we don't have such request (e.g. protocol A)
+ * check the oldest request which is still waiting on its epoch
+ * closing barrier ack. */
+ if (!req) {
+ req = connection->req_not_net_done;
+
+ /* If we did not send the request yet then pre_send_jif
+ * is not set. Treat this the same as when there are no
+ * requests pending. */
+ if (req && !(req->net_rq_state[connection->peer_node_id] & RQ_NET_SENT))
+ req = NULL;
+ }
- /* check the oldest request we did successfully sent,
- * but which is still waiting for an ACK. */
- req_peer = connection->req_ack_pending;
+ if (req)
+ pre_send_jif = req->pre_send_jif[connection->peer_node_id];
+
+ effective_timeout = min_not_zero(effective_timeout, effective_net_timeout);
+ next_trigger_time = time_min_in_future(now,
+ next_trigger_time, pre_send_jif + effective_net_timeout);
+ /* Restart the timer, even if there are no pending requests at all.
+ * We currently do not re-arm from the submit path. */
+ restart_timer = true;
+
+ /* We have one timer per "device",
+ * but the "oldest" request is per "connection".
+ * Evaluate the oldest peer request only in one timer! */
+ if (req == NULL || req->device != device)
+ continue;
+
+ if (net_timeout_reached(req, peer_device, now, effective_net_timeout, ko_count, timeout)) {
+ dynamic_drbd_dbg(peer_device, "Request at %llus+%u timed out\n",
+ (unsigned long long) req->i.sector,
+ req->i.size);
+ timeout_peers |= NODE_MASK(connection->peer_node_id);
+ }
+ }
+ read_unlock_irq(&resource->state_rwlock);
- /* if we don't have such request (e.g. protocoll A)
- * check the oldest requests which is still waiting on its epoch
- * closing barrier ack. */
- if (!req_peer)
- req_peer = connection->req_not_net_done;
+ if (io_error) {
+ drbd_warn(device, "Local backing device failed to meet the disk-timeout\n");
+ drbd_handle_io_error(device, DRBD_FORCE_DETACH);
+ }
- /* evaluate the oldest peer request only in one timer! */
- if (req_peer && req_peer->device != device)
- req_peer = NULL;
+ BUILD_BUG_ON(sizeof(timeout_peers) * 8 < DRBD_NODE_ID_MAX);
+ for_each_set_bit(node_id, &timeout_peers, DRBD_NODE_ID_MAX) {
+ connection = drbd_get_connection_by_node_id(resource, node_id);
+ if (!connection)
+ continue;
+ begin_state_change(resource, &irq_flags, CS_VERBOSE | CS_HARD);
+ __change_cstate(connection, C_TIMEOUT);
+ end_state_change(resource, &irq_flags, "timeout");
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
- /* do we have something to evaluate? */
- if (req_peer == NULL && req_write == NULL && req_read == NULL)
- goto out;
+ if (restart_timer) {
+ next_trigger_time = time_min_in_future(now, next_trigger_time, now + effective_timeout);
+ mod_timer(&device->request_timer, next_trigger_time);
+ }
+}
- oldest_submit_jif =
- (req_write && req_read)
- ? ( time_before(req_write->pre_submit_jif, req_read->pre_submit_jif)
- ? req_write->pre_submit_jif : req_read->pre_submit_jif )
- : req_write ? req_write->pre_submit_jif
- : req_read ? req_read->pre_submit_jif : now;
+/**
+ * drbd_handle_io_error_: Handle the on_io_error setting, should be called from all io completion handlers
+ * @device: DRBD device.
+ * @df: Detach flags indicating the kind of IO that failed.
+ * @where: Calling function name.
+ */
+void drbd_handle_io_error_(struct drbd_device *device,
+ enum drbd_force_detach_flags df, const char *where)
+{
+ unsigned long flags;
+ enum drbd_io_error_p ep;
- if (ent && req_peer && net_timeout_reached(req_peer, connection, now, ent, ko_count, timeout))
- _conn_request_state(connection, NS(conn, C_TIMEOUT), CS_VERBOSE | CS_HARD);
+ write_lock_irqsave(&device->resource->state_rwlock, flags);
- if (dt && oldest_submit_jif != now &&
- time_after(now, oldest_submit_jif + dt) &&
- !time_in_range(now, device->last_reattach_jif, device->last_reattach_jif + dt)) {
- drbd_warn(device, "Local backing device failed to meet the disk-timeout\n");
- __drbd_chk_io_error(device, DRBD_FORCE_DETACH);
+ rcu_read_lock();
+ /* ldev_safe: called from endio handlers where ldev is still held */
+ ep = rcu_dereference(device->ldev->disk_conf)->on_io_error;
+ rcu_read_unlock();
+ switch (ep) {
+ case EP_PASS_ON: /* FIXME would this be better named "Ignore"? */
+ if (df == DRBD_READ_ERROR || df == DRBD_WRITE_ERROR) {
+ if (drbd_device_ratelimit(device, BACKEND))
+ drbd_err(device, "Local IO failed in %s.\n", where);
+ if (device->disk_state[NOW] > D_INCONSISTENT) {
+ begin_state_change_locked(device->resource, CS_HARD);
+ __change_disk_state(device, D_INCONSISTENT);
+ end_state_change_locked(device->resource, "local-io-error");
+ }
+ break;
+ }
+ fallthrough; /* for DRBD_META_IO_ERROR or DRBD_FORCE_DETACH */
+ case EP_DETACH:
+ case EP_CALL_HELPER:
+ /* Force-detach is not really an IO error, but rather a
+ * desperate measure to try to deal with a completely
+ * unresponsive lower level IO stack.
+ * Still it should be treated as a WRITE error.
+ */
+ if (df == DRBD_FORCE_DETACH)
+ set_bit(FORCE_DETACH, &device->flags);
+ if (device->disk_state[NOW] > D_FAILED) {
+ begin_state_change_locked(device->resource, CS_HARD);
+ __change_disk_state(device, D_FAILED);
+ end_state_change_locked(device->resource, "local-io-error");
+ drbd_err(device,
+ "Local IO failed in %s. Detaching...\n", where);
+ }
+ break;
}
- /* Reschedule timer for the nearest not already expired timeout.
- * Fallback to now + min(effective network timeout, disk timeout). */
- ent = (ent && req_peer && time_before(now, req_peer->pre_send_jif + ent))
- ? req_peer->pre_send_jif + ent : now + et;
- dt = (dt && oldest_submit_jif != now && time_before(now, oldest_submit_jif + dt))
- ? oldest_submit_jif + dt : now + et;
- nt = time_before(ent, dt) ? ent : dt;
-out:
- spin_unlock_irq(&device->resource->req_lock);
- mod_timer(&device->request_timer, nt);
+ write_unlock_irqrestore(&device->resource->state_rwlock, flags);
}
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 14/20] drbd: rework module core for DRBD 9 transport and multi-peer
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (12 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 13/20] drbd: rework request processing for DRBD 9 multi-peer IO Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 15/20] drbd: rework receiver for DRBD 9 transport and multi-peer protocol Christoph Böhmwalder
` (5 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Andreas Gruenbacher,
Joel Colledge, Moritz Tanner, Roland Kammerer
Rework drbd_main.c to align the module core with the DRBD 9 multi-peer
architecture introduced by the surrounding header and transport commits.
Refactor all packet sending around a page-based send buffer with
explicit cork/uncork semantics driven by the transport layer,
replacing the old per-socket static buffer and direct socket calls.
Move the transfer log from per-connection to per-resource scope, and
switch its traversal to RCU, allowing safe concurrent walks without
the coarse req_lock spinlock.
Rewrite UUID management for multi-peer: the fixed 4-slot layout is
replaced by a per-device current UUID, per-peer bitmap UUIDs, and a
history array.
This enables DRBD 9 to track resyncs across more than one peer
simultaneously.
Separate the resource and connection lifecycles so that resources and
connections are created, torn down, and reference-counted
independently, with threads scoped appropriately to each object.
Add quorum-aware auto-promote semantics to the block device
open/release path.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Moritz Tanner <moritz.tanner@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_main.c | 6812 +++++++++++++++++++++++---------
1 file changed, 5020 insertions(+), 1792 deletions(-)
diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c
index aed1d1ab6d0f..c890c806b35f 100644
--- a/drivers/block/drbd/drbd_main.c
+++ b/drivers/block/drbd/drbd_main.c
@@ -11,60 +11,67 @@
#include <linux/module.h>
#include <linux/jiffies.h>
#include <linux/drbd.h>
-#include <linux/uaccess.h>
#include <asm/types.h>
+#include <net/net_namespace.h>
#include <net/sock.h>
#include <linux/ctype.h>
-#include <linux/mutex.h>
#include <linux/fs.h>
#include <linux/file.h>
#include <linux/proc_fs.h>
#include <linux/init.h>
#include <linux/mm.h>
-#include <linux/memcontrol.h>
+#include <linux/memcontrol.h> /* needed on kernels <4.3 */
#include <linux/mm_inline.h>
#include <linux/slab.h>
-#include <linux/string.h>
-#include <linux/random.h>
-#include <linux/reboot.h>
#include <linux/notifier.h>
-#include <linux/kthread.h>
#include <linux/workqueue.h>
-#include <linux/unistd.h>
+#include <linux/kthread.h>
#include <linux/vmalloc.h>
-#include <linux/sched/signal.h>
+#include <linux/dynamic_debug.h>
+#include <linux/libnvdimm.h>
+#include <linux/string.h>
+#include <linux/swab.h>
+#include <linux/overflow.h>
+#include <linux/drbd_genl.h>
#include <linux/drbd_limits.h>
#include "drbd_int.h"
#include "drbd_protocol.h"
-#include "drbd_req.h" /* only for _req_mod in tl_release and tl_clear */
+#include "drbd_req.h"
#include "drbd_vli.h"
#include "drbd_debugfs.h"
+#include "drbd_meta_data.h"
+#include "drbd_legacy_84.h"
+#include "drbd_dax_pmem.h"
-static DEFINE_MUTEX(drbd_main_mutex);
-static int drbd_open(struct gendisk *disk, blk_mode_t mode);
+static int drbd_open(struct gendisk *gd, blk_mode_t mode);
static void drbd_release(struct gendisk *gd);
static void md_sync_timer_fn(struct timer_list *t);
static int w_bitmap_io(struct drbd_work *w, int unused);
+static int flush_send_buffer(struct drbd_connection *connection, enum drbd_stream drbd_stream);
+static u64 __set_bitmap_slots(struct drbd_device *device, u64 bitmap_uuid, u64 do_nodes);
+static u64 __test_bitmap_slots(struct drbd_device *device);
+static void drbd_send_ping_ack_wf(struct work_struct *ws);
+static void __net_exit __drbd_net_exit(struct net *net);
MODULE_AUTHOR("Philipp Reisner <phil@linbit.com>, "
"Lars Ellenberg <lars@linbit.com>");
MODULE_DESCRIPTION("drbd - Distributed Replicated Block Device v" REL_VERSION);
-MODULE_VERSION(REL_VERSION);
+MODULE_VERSION(DRBD_PROC_VERSION);
MODULE_LICENSE("GPL");
MODULE_PARM_DESC(minor_count, "Approximate number of drbd devices ("
__stringify(DRBD_MINOR_COUNT_MIN) "-" __stringify(DRBD_MINOR_COUNT_MAX) ")");
MODULE_ALIAS_BLOCKDEV_MAJOR(DRBD_MAJOR);
+MODULE_SOFTDEP("post: handshake");
#include <linux/moduleparam.h>
-/* thanks to these macros, if compiled into the kernel (not-module),
- * these become boot parameters (e.g., drbd.minor_count) */
#ifdef CONFIG_DRBD_FAULT_INJECTION
int drbd_enable_faults;
int drbd_fault_rate;
static int drbd_fault_count;
static int drbd_fault_devs;
+
/* bitmap of enabled faults */
module_param_named(enable_faults, drbd_enable_faults, int, 0664);
/* fault rate % value - applies to all enabled faults */
@@ -76,15 +83,12 @@ module_param_named(fault_devs, drbd_fault_devs, int, 0644);
#endif
/* module parameters we can keep static */
-static bool drbd_allow_oos; /* allow_open_on_secondary */
static bool drbd_disable_sendpage;
+static bool drbd_allow_oos; /* allow_open_on_secondary */
MODULE_PARM_DESC(allow_oos, "DONT USE!");
-module_param_named(allow_oos, drbd_allow_oos, bool, 0);
module_param_named(disable_sendpage, drbd_disable_sendpage, bool, 0644);
+module_param_named(allow_oos, drbd_allow_oos, bool, 0);
-/* module parameters we share */
-int drbd_proc_details; /* Detail level in proc drbd*/
-module_param_named(proc_details, drbd_proc_details, int, 0644);
/* module parameters shared with defaults */
unsigned int drbd_minor_count = DRBD_MINOR_COUNT_DEF;
/* Module parameter for setting the user mode helper program
@@ -93,16 +97,80 @@ char drbd_usermode_helper[80] = "/sbin/drbdadm";
module_param_named(minor_count, drbd_minor_count, uint, 0444);
module_param_string(usermode_helper, drbd_usermode_helper, sizeof(drbd_usermode_helper), 0644);
+static int param_set_drbd_protocol_version(const char *s, const struct kernel_param *kp)
+{
+ unsigned long long tmp;
+ unsigned int *res = kp->arg;
+ int rv;
+
+ rv = kstrtoull(s, 0, &tmp);
+ if (rv < 0)
+ return rv;
+ if (!drbd_protocol_version_acceptable(tmp))
+ return -ERANGE;
+ *res = tmp;
+ return 0;
+}
+
+#define param_check_drbd_protocol_version param_check_uint
+#define param_get_drbd_protocol_version param_get_uint
+
+static const struct kernel_param_ops param_ops_drbd_protocol_version = {
+ .set = param_set_drbd_protocol_version,
+ .get = param_get_drbd_protocol_version,
+};
+
+unsigned int drbd_protocol_version_min = PRO_VERSION_8_MIN;
+module_param_named(protocol_version_min, drbd_protocol_version_min, drbd_protocol_version, 0644);
+
+static int param_set_max_parallel_resyncs(const char *s, const struct kernel_param *kp)
+{
+ int rv = param_set_uint(s, kp);
+
+ if (rv == 0)
+ drbd_apply_resync_max_parallel();
+ return rv;
+}
+
+static const struct kernel_param_ops param_ops_max_parallel_resyncs = {
+ .set = param_set_max_parallel_resyncs,
+ .get = param_get_uint,
+};
+
+unsigned int drbd_max_parallel_resyncs; /* 0 = unlimited */
+module_param_cb(max_parallel_resyncs, ¶m_ops_max_parallel_resyncs,
+ &drbd_max_parallel_resyncs, 0644);
+MODULE_PARM_DESC(max_parallel_resyncs,
+ "Cap on volumes resyncing in parallel (0 = unlimited)");
+#define protocol_version_min_desc \
+ "\n\t\tReject DRBD dialects older than this.\n\t\t" \
+ "Supported: " \
+ "DRBD 8 [" __stringify(PRO_VERSION_8_MIN) "-" __stringify(PRO_VERSION_8_MAX) "]; " \
+ "DRBD 9 [" __stringify(PRO_VERSION_MIN) "-" __stringify(PRO_VERSION_MAX) "].\n\t\t" \
+ "Default: " __stringify(PRO_VERSION_8_MIN)
+MODULE_PARM_DESC(protocol_version_min, protocol_version_min_desc);
+
+#define param_check_drbd_strict_names param_check_bool
+#define param_get_drbd_strict_names param_get_bool
+const struct kernel_param_ops param_ops_drbd_strict_names = {
+ .set = param_set_drbd_strict_names,
+ .get = param_get_drbd_strict_names,
+};
+bool drbd_strict_names = true;
+MODULE_PARM_DESC(strict_names, "restrict resource and connection names to ascii alnum and a subset of punct");
+module_param_named(strict_names, drbd_strict_names, drbd_strict_names, 0644);
+
/* in 2.6.x, our device mapping and config info contains our virtual gendisks
* as member "struct gendisk *vdisk;"
*/
struct idr drbd_devices;
-struct list_head drbd_resources;
-struct mutex resources_mutex;
+LIST_HEAD(drbd_resources);
+static DEFINE_SPINLOCK(drbd_devices_lock);
+DEFINE_MUTEX(resources_mutex);
+struct workqueue_struct *ping_ack_sender;
struct kmem_cache *drbd_request_cache;
struct kmem_cache *drbd_ee_cache; /* peer requests */
-struct kmem_cache *drbd_bm_ext_cache; /* bitmap extents */
struct kmem_cache *drbd_al_ext_cache; /* activity log extents */
mempool_t drbd_request_mempool;
mempool_t drbd_ee_mempool;
@@ -110,8 +178,7 @@ mempool_t drbd_md_io_page_pool;
mempool_t drbd_buffer_page_pool;
struct bio_set drbd_md_io_bio_set;
struct bio_set drbd_io_bio_set;
-
-DEFINE_RATELIMIT_STATE(drbd_ratelimit_state, 5 * HZ, 5);
+struct bio_set drbd_peer_bio_set;
static const struct block_device_operations drbd_ops = {
.owner = THIS_MODULE,
@@ -120,71 +187,244 @@ static const struct block_device_operations drbd_ops = {
.release = drbd_release,
};
-#ifdef __CHECKER__
-/* When checking with sparse, and this is an inline function, sparse will
- give tons of false positives. When this is a real functions sparse works.
- */
-int _get_ldev_if_state(struct drbd_device *device, enum drbd_disk_state mins)
+static struct pernet_operations drbd_pernet_ops = {
+ .exit = __drbd_net_exit,
+};
+
+struct drbd_connection *__drbd_next_connection_ref(u64 *visited,
+ struct drbd_connection *connection,
+ struct drbd_resource *resource)
{
- int io_allowed;
+ int node_id;
- atomic_inc(&device->local_cnt);
- io_allowed = (device->state.disk >= mins);
- if (!io_allowed) {
- if (atomic_dec_and_test(&device->local_cnt))
- wake_up(&device->misc_wait);
+ rcu_read_lock();
+ if (!connection) {
+ connection = list_first_or_null_rcu(&resource->connections,
+ struct drbd_connection,
+ connections);
+ *visited = 0;
+ } else {
+ struct list_head *pos;
+ bool previous_visible; /* on the resources connections list */
+
+ pos = list_next_rcu(&connection->connections);
+ /* follow the pointer first, then check if the previous element was
+ still an element on the list of visible connections. */
+ smp_rmb();
+ previous_visible = !test_bit(C_UNREGISTERED, &connection->flags);
+ kref_put(&connection->kref, drbd_destroy_connection);
+
+ if (pos == &resource->connections) {
+ connection = NULL;
+ } else if (previous_visible) { /* visible -> we are now on a vital element */
+ connection = list_entry_rcu(pos, struct drbd_connection, connections);
+ } else { /* not visible -> pos might point to a dead element now */
+ for_each_connection_rcu(connection, resource) {
+ node_id = connection->peer_node_id;
+ if (!(*visited & NODE_MASK(node_id)))
+ goto found;
+ }
+ connection = NULL;
+ }
+ }
+
+ if (connection) {
+found:
+ node_id = connection->peer_node_id;
+ *visited |= NODE_MASK(node_id);
+
+ kref_get(&connection->kref);
}
- return io_allowed;
+
+ rcu_read_unlock();
+ return connection;
}
-#endif
+
+struct drbd_peer_device *__drbd_next_peer_device_ref(u64 *visited,
+ struct drbd_peer_device *peer_device,
+ struct drbd_device *device)
+{
+ rcu_read_lock();
+ if (!peer_device) {
+ peer_device = list_first_or_null_rcu(&device->peer_devices,
+ struct drbd_peer_device,
+ peer_devices);
+ *visited = 0;
+ } else {
+ struct list_head *pos;
+ bool previous_visible;
+
+ pos = list_next_rcu(&peer_device->peer_devices);
+ smp_rmb();
+ previous_visible = !test_bit(C_UNREGISTERED, &peer_device->connection->flags);
+ kref_put(&peer_device->connection->kref, drbd_destroy_connection);
+
+ if (pos == &device->peer_devices) {
+ peer_device = NULL;
+ } else if (previous_visible) {
+ peer_device = list_entry_rcu(pos, struct drbd_peer_device, peer_devices);
+ } else {
+ for_each_peer_device_rcu(peer_device, device) {
+ if (!(*visited & NODE_MASK(peer_device->node_id)))
+ goto found;
+ }
+ peer_device = NULL;
+ }
+ }
+
+ if (peer_device) {
+found:
+ *visited |= NODE_MASK(peer_device->node_id);
+
+ kref_get(&peer_device->connection->kref);
+ }
+
+ rcu_read_unlock();
+ return peer_device;
+}
+
+static void dump_epoch(struct drbd_resource *resource, int node_id, int epoch)
+{
+ struct drbd_request *req;
+ bool found_epoch = false;
+
+ list_for_each_entry_rcu(req, &resource->transfer_log, tl_requests) {
+ if (!found_epoch && req->epoch == epoch)
+ found_epoch = true;
+
+ if (found_epoch) {
+ if (req->epoch != epoch)
+ break;
+ drbd_info(req->device, "XXX %u %llu+%u 0x%x 0x%x\n",
+ req->epoch,
+ (unsigned long long)req->i.sector, req->i.size >> 9,
+ req->local_rq_state, req->net_rq_state[node_id]
+ );
+ }
+ }
+}
/**
* tl_release() - mark as BARRIER_ACKED all requests in the corresponding transfer log epoch
* @connection: DRBD connection.
+ * @o_block_id: "block id" aka expected pointer address of the oldest request
+ * @y_block_id: "block id" aka expected pointer address of the youngest request
+ * confirmed to be on stable storage.
* @barrier_nr: Expected identifier of the DRBD write barrier packet.
- * @set_size: Expected number of requests before that barrier.
+ * @set_size: Expected number of requests before that barrier, respectively
+ * number of requests in the interval [o_block_id;y_block_id]
+ *
+ * Called for both P_BARRIER_ACK and P_CONFIRM_STABLE,
+ * which is similar to an unsolicited partial barrier ack.
+ *
+ * Either barrier_nr (for barrier acks) or both o_block_id and y_blockid (for
+ * confirm stable) are given. For barrier acks, all requests in the epoch
+ * designated by "barrier_nr" are confirmed to be on stable storage.
+ *
+ * For confirm stable, both o_block_id and y_block_id are given, barrier_nr is
+ * ignored, and all requests from "o_block_id" up to and including y_block_id
+ * are confirmed to be on stable storage on the reporting peer.
*
* In case the passed barrier_nr or set_size does not match the oldest
* epoch of not yet barrier-acked requests, this function will cause a
* termination of the connection.
*/
-void tl_release(struct drbd_connection *connection, unsigned int barrier_nr,
+int tl_release(struct drbd_connection *connection,
+ uint64_t o_block_id,
+ uint64_t y_block_id,
+ unsigned int barrier_nr,
unsigned int set_size)
{
+ struct drbd_resource *resource = connection->resource;
+ const int idx = connection->peer_node_id;
struct drbd_request *r;
- struct drbd_request *req = NULL, *tmp = NULL;
+ struct drbd_request *req = NULL;
+ struct drbd_request *req_y = NULL;
int expect_epoch = 0;
int expect_size = 0;
- spin_lock_irq(&connection->resource->req_lock);
-
+ rcu_read_lock();
/* find oldest not yet barrier-acked write request,
* count writes in its epoch. */
- list_for_each_entry(r, &connection->transfer_log, tl_requests) {
- const unsigned s = r->rq_state;
+ r = READ_ONCE(connection->req_not_net_done);
+ if (r == NULL) {
+ drbd_err(connection, "BarrierAck #%u received, but req_not_net_done = NULL\n",
+ barrier_nr);
+ goto bail;
+ }
+ smp_rmb(); /* paired with smp_wmb() in set_cache_ptr_if_null() */
+ list_for_each_entry_from_rcu(r, &resource->transfer_log, tl_requests) {
+ unsigned int local_rq_state, net_rq_state;
+
+ spin_lock_irq(&r->rq_lock);
+ local_rq_state = r->local_rq_state;
+ net_rq_state = r->net_rq_state[idx];
+ spin_unlock_irq(&r->rq_lock);
+
if (!req) {
- if (!(s & RQ_WRITE))
+ if (!(local_rq_state & RQ_WRITE))
continue;
- if (!(s & RQ_NET_MASK))
+ if (!(net_rq_state & RQ_NET_MASK))
continue;
- if (s & RQ_NET_DONE)
+ if (net_rq_state & RQ_NET_DONE)
continue;
req = r;
expect_epoch = req->epoch;
- expect_size ++;
+ expect_size++;
} else {
+ const u16 s = r->net_rq_state[idx];
+
if (r->epoch != expect_epoch)
break;
- if (!(s & RQ_WRITE))
+ if (!(local_rq_state & RQ_WRITE))
continue;
- /* if (s & RQ_DONE): not expected */
- /* if (!(s & RQ_NET_MASK)): not expected */
- expect_size++;
+ /* probably a "send_out_of_sync", during Ahead/Behind mode,
+ * while at least one volume already started to resync again.
+ * Or a write that was not replicated during a resync, and
+ * replication has been enabled since it was submitted.
+ */
+ if ((s & RQ_NET_MASK) && !(s & RQ_EXP_BARR_ACK))
+ continue;
+ /* Count only what the peer counts: writes still outstanding
+ * towards it, not one already done or never sent to it.
+ */
+ if (s & RQ_NET_DONE || (s & RQ_NET_MASK) == 0)
+ dynamic_drbd_dbg(connection,
+ "not counting state flags: 0x%x during BarrierAck #%u\n",
+ s, barrier_nr);
+ else
+ expect_size++;
+ }
+ if (y_block_id && (struct drbd_request *)(unsigned long)y_block_id == r) {
+ req_y = r;
+ break;
}
}
/* first some paranoia code */
+ if (o_block_id) {
+ if ((struct drbd_request *)(unsigned long)o_block_id != req) {
+ drbd_err(connection, "BAD! ConfirmedStable: expected %p, found %p\n",
+ (struct drbd_request *)(unsigned long)o_block_id, req);
+ goto bail;
+ }
+ if (!req_y) {
+ drbd_err(connection, "BAD! ConfirmedStable: expected youngest request %p NOT found\n",
+ (struct drbd_req *)(unsigned long)y_block_id);
+ goto bail;
+ }
+ /* A P_CONFIRM_STABLE cannot tell me the to-be-expected barrier nr,
+ * it does not know it yet. But we just confirmed it knew the
+ * expected request, so just use that one. */
+ barrier_nr = expect_epoch;
+ /* Both requests referenced must be in the same epoch. */
+ if (req_y->epoch != expect_epoch) {
+ drbd_err(connection, "BAD! ConfirmedStable: reported requests not in the same epoch (%u != %u)\n",
+ req->epoch, req_y->epoch);
+ goto bail;
+ }
+ }
if (req == NULL) {
drbd_err(connection, "BAD! BarrierAck #%u received, but no epoch in tl!?\n",
barrier_nr);
@@ -197,111 +437,135 @@ void tl_release(struct drbd_connection *connection, unsigned int barrier_nr,
}
if (expect_size != set_size) {
- drbd_err(connection, "BAD! BarrierAck #%u received with n_writes=%u, expected n_writes=%u!\n",
- barrier_nr, set_size, expect_size);
+ if (!o_block_id) {
+ DEFINE_DYNAMIC_DEBUG_METADATA(ddm, "Bad barrier ack dump");
+
+ drbd_err(connection, "BAD! BarrierAck #%u received with n_writes=%u, expected n_writes=%u!\n",
+ barrier_nr, set_size, expect_size);
+
+ if (DYNAMIC_DEBUG_BRANCH(ddm))
+ dump_epoch(resource, connection->peer_node_id, expect_epoch);
+ } else
+ drbd_err(connection, "BAD! ConfirmedStable [%p,%p] received with n_writes=%u, expected n_writes=%u!\n",
+ req, req_y, set_size, expect_size);
goto bail;
}
/* Clean up list of requests processed during current epoch. */
- /* this extra list walk restart is paranoia,
- * to catch requests being barrier-acked "unexpectedly".
- * It usually should find the same req again, or some READ preceding it. */
- list_for_each_entry(req, &connection->transfer_log, tl_requests)
- if (req->epoch == expect_epoch) {
- tmp = req;
- break;
- }
- req = list_prepare_entry(tmp, &connection->transfer_log, tl_requests);
- list_for_each_entry_safe_from(req, r, &connection->transfer_log, tl_requests) {
+ list_for_each_entry_from_rcu(req, &resource->transfer_log, tl_requests) {
struct drbd_peer_device *peer_device;
+
if (req->epoch != expect_epoch)
break;
peer_device = conn_peer_device(connection, req->device->vnr);
- _req_mod(req, BARRIER_ACKED, peer_device);
+ req_mod(req, BARRIER_ACKED, peer_device);
+ if (req == req_y)
+ break;
+ }
+ rcu_read_unlock();
+
+ /* urgently flush out peer acks for P_CONFIRM_STABLE */
+ if (req_y) {
+ drbd_flush_peer_acks(resource);
+ } else {
+ WRITE_ONCE(connection->last_acked_epoch_nr, barrier_nr);
+ wake_up(&resource->barrier_wait);
}
- spin_unlock_irq(&connection->resource->req_lock);
- return;
+ return 0;
bail:
- spin_unlock_irq(&connection->resource->req_lock);
- conn_request_state(connection, NS(conn, C_PROTOCOL_ERROR), CS_HARD);
+ rcu_read_unlock();
+ return -EPROTO;
}
/**
- * _tl_restart() - Walks the transfer log, and applies an action to all requests
- * @connection: DRBD connection to operate on.
+ * __tl_walk() - Walks the transfer log, and applies an action to all requests
+ * @resource: DRBD resource to opterate on
+ * @connection: DRBD connection to operate on
+ * @from_req: If set, the walk starts from the request that this points to
* @what: The action/event to perform with all request objects
*
- * @what might be one of CONNECTION_LOST_WHILE_PENDING, RESEND, FAIL_FROZEN_DISK_IO,
- * RESTART_FROZEN_DISK_IO.
+ * @what might be one of CONNECTION_LOST, CONNECTION_LOST_WHILE_SUSPENDED,
+ * RESEND, CANCEL_SUSPENDED_IO, COMPLETION_RESUMED.
*/
-/* must hold resource->req_lock */
-void _tl_restart(struct drbd_connection *connection, enum drbd_req_event what)
+void __tl_walk(struct drbd_resource *const resource,
+ struct drbd_connection *const connection,
+ struct drbd_request **from_req,
+ const enum drbd_req_event what)
{
struct drbd_peer_device *peer_device;
- struct drbd_request *req, *r;
+ struct drbd_request *req = NULL;
- list_for_each_entry_safe(req, r, &connection->transfer_log, tl_requests) {
- peer_device = conn_peer_device(connection, req->device->vnr);
+ rcu_read_lock();
+ if (from_req)
+ req = READ_ONCE(*from_req);
+ if (!req)
+ req = list_entry_rcu(resource->transfer_log.next, struct drbd_request, tl_requests);
+ smp_rmb(); /* paired with smp_wmb() in set_cache_ptr_if_null() */
+ list_for_each_entry_from_rcu(req, &resource->transfer_log, tl_requests) {
+ /* Skip if the request has already been destroyed. */
+ if (!kref_get_unless_zero(&req->kref))
+ continue;
+
+ peer_device = connection == NULL ? NULL :
+ conn_peer_device(connection, req->device->vnr);
_req_mod(req, what, peer_device);
+ kref_put(&req->kref, drbd_req_destroy);
}
+ rcu_read_unlock();
}
-void tl_restart(struct drbd_connection *connection, enum drbd_req_event what)
+void tl_walk(struct drbd_connection *connection, struct drbd_request **from_req, enum drbd_req_event what)
{
- spin_lock_irq(&connection->resource->req_lock);
- _tl_restart(connection, what);
- spin_unlock_irq(&connection->resource->req_lock);
-}
+ struct drbd_resource *resource = connection->resource;
-/**
- * tl_clear() - Clears all requests and &struct drbd_tl_epoch objects out of the TL
- * @connection: DRBD connection.
- *
- * This is called after the connection to the peer was lost. The storage covered
- * by the requests on the transfer gets marked as our of sync. Called from the
- * receiver thread and the worker thread.
- */
-void tl_clear(struct drbd_connection *connection)
-{
- tl_restart(connection, CONNECTION_LOST_WHILE_PENDING);
+ read_lock_irq(&resource->state_rwlock);
+ __tl_walk(connection->resource, connection, from_req, what);
+ read_unlock_irq(&resource->state_rwlock);
}
/**
* tl_abort_disk_io() - Abort disk I/O for all requests for a certain device in the TL
- * @device: DRBD device.
+ * @device: DRBD device.
*/
void tl_abort_disk_io(struct drbd_device *device)
{
- struct drbd_connection *connection = first_peer_device(device)->connection;
- struct drbd_request *req, *r;
+ struct drbd_resource *resource = device->resource;
+ struct drbd_request *req;
- spin_lock_irq(&connection->resource->req_lock);
- list_for_each_entry_safe(req, r, &connection->transfer_log, tl_requests) {
- if (!(req->rq_state & RQ_LOCAL_PENDING))
+ rcu_read_lock();
+ list_for_each_entry_rcu(req, &resource->transfer_log, tl_requests) {
+ if (!(READ_ONCE(req->local_rq_state) & RQ_LOCAL_PENDING))
continue;
if (req->device != device)
continue;
- _req_mod(req, ABORT_DISK_IO, NULL);
+ /* Skip if the request has already been destroyed. */
+ if (!kref_get_unless_zero(&req->kref))
+ continue;
+
+ req_mod(req, ABORT_DISK_IO, NULL);
+ kref_put(&req->kref, drbd_req_destroy);
}
- spin_unlock_irq(&connection->resource->req_lock);
+ rcu_read_unlock();
}
static int drbd_thread_setup(void *arg)
{
struct drbd_thread *thi = (struct drbd_thread *) arg;
struct drbd_resource *resource = thi->resource;
+ struct drbd_connection *connection = thi->connection;
unsigned long flags;
int retval;
- snprintf(current->comm, sizeof(current->comm), "drbd_%c_%s",
- thi->name[0],
- resource->name);
-
allow_kernel_signal(DRBD_SIGKILL);
allow_kernel_signal(SIGXCPU);
+
+ if (connection)
+ kref_get(&connection->kref);
+ else
+ kref_get(&resource->kref);
restart:
retval = thi->function(thi);
@@ -318,26 +582,33 @@ static int drbd_thread_setup(void *arg)
*/
if (thi->t_state == RESTARTING) {
- drbd_info(resource, "Restarting %s thread\n", thi->name);
+ if (connection)
+ drbd_info(connection, "Restarting %s thread\n", thi->name);
+ else
+ drbd_info(resource, "Restarting %s thread\n", thi->name);
thi->t_state = RUNNING;
spin_unlock_irqrestore(&thi->t_lock, flags);
+ flush_signals(current); /* likely it got a signal to look at t_state... */
goto restart;
}
thi->task = NULL;
thi->t_state = NONE;
smp_mb();
- complete_all(&thi->stop);
- spin_unlock_irqrestore(&thi->t_lock, flags);
- drbd_info(resource, "Terminating %s\n", current->comm);
+ if (connection)
+ drbd_info(connection, "Terminating %s thread\n", thi->name);
+ else
+ drbd_info(resource, "Terminating %s thread\n", thi->name);
+
+ complete(&thi->stop);
+ spin_unlock_irqrestore(&thi->t_lock, flags);
- /* Release mod reference taken when thread was started */
+ if (connection)
+ kref_put(&connection->kref, drbd_destroy_connection);
+ else
+ kref_put(&resource->kref, drbd_destroy_resource);
- if (thi->connection)
- kref_put(&thi->connection->kref, drbd_destroy_connection);
- kref_put(&resource->kref, drbd_destroy_resource);
- module_put(THIS_MODULE);
return retval;
}
@@ -356,6 +627,7 @@ static void drbd_thread_init(struct drbd_resource *resource, struct drbd_thread
int drbd_thread_start(struct drbd_thread *thi)
{
struct drbd_resource *resource = thi->resource;
+ struct drbd_connection *connection = thi->connection;
struct task_struct *nt;
unsigned long flags;
@@ -365,36 +637,33 @@ int drbd_thread_start(struct drbd_thread *thi)
switch (thi->t_state) {
case NONE:
- drbd_info(resource, "Starting %s thread (from %s [%d])\n",
- thi->name, current->comm, current->pid);
-
- /* Get ref on module for thread - this is released when thread exits */
- if (!try_module_get(THIS_MODULE)) {
- drbd_err(resource, "Failed to get module reference in drbd_thread_start\n");
- spin_unlock_irqrestore(&thi->t_lock, flags);
- return false;
- }
-
- kref_get(&resource->kref);
- if (thi->connection)
- kref_get(&thi->connection->kref);
+ if (connection)
+ drbd_info(connection, "Starting %s thread (peer-node-id %d)\n",
+ thi->name, connection->peer_node_id);
+ else
+ drbd_info(resource, "Starting %s thread (node-id %d)\n",
+ thi->name, resource->res_opts.node_id);
init_completion(&thi->stop);
+ D_ASSERT(resource, thi->task == NULL);
thi->reset_cpu_mask = 1;
thi->t_state = RUNNING;
spin_unlock_irqrestore(&thi->t_lock, flags);
flush_signals(current); /* otherw. may get -ERESTARTNOINTR */
nt = kthread_create(drbd_thread_setup, (void *) thi,
- "drbd_%c_%s", thi->name[0], thi->resource->name);
+ "drbd_%c_%s", thi->name[0], resource->name);
if (IS_ERR(nt)) {
- drbd_err(resource, "Couldn't start thread\n");
+ if (connection)
+ drbd_err(connection, "Couldn't start thread: %ld\n", PTR_ERR(nt));
+ else
+ drbd_err(resource, "Couldn't start thread: %ld\n", PTR_ERR(nt));
+
+ spin_lock_irqsave(&thi->t_lock, flags);
+ thi->t_state = NONE;
+ spin_unlock_irqrestore(&thi->t_lock, flags);
- if (thi->connection)
- kref_put(&thi->connection->kref, drbd_destroy_connection);
- kref_put(&resource->kref, drbd_destroy_resource);
- module_put(THIS_MODULE);
return false;
}
spin_lock_irqsave(&thi->t_lock, flags);
@@ -405,8 +674,10 @@ int drbd_thread_start(struct drbd_thread *thi)
break;
case EXITING:
thi->t_state = RESTARTING;
- drbd_info(resource, "Restarting %s thread (from %s [%d])\n",
- thi->name, current->comm, current->pid);
+ if (connection)
+ drbd_info(connection, "Restarting %s thread\n", thi->name);
+ else
+ drbd_info(resource, "Restarting %s thread\n", thi->name);
fallthrough;
case RUNNING:
case RESTARTING:
@@ -419,6 +690,23 @@ int drbd_thread_start(struct drbd_thread *thi)
}
+/* Like the restart path of drbd_thread_setup(), for a thread restarting in place. */
+void drbd_thread_consume_restart(struct drbd_thread *thi)
+{
+ unsigned long flags;
+ bool restarted = false;
+
+ spin_lock_irqsave(&thi->t_lock, flags);
+ if (thi->t_state == RESTARTING) {
+ thi->t_state = RUNNING;
+ restarted = true;
+ }
+ spin_unlock_irqrestore(&thi->t_lock, flags);
+
+ if (restarted)
+ flush_signals(current);
+}
+
void _drbd_thread_stop(struct drbd_thread *thi, int restart, int wait)
{
unsigned long flags;
@@ -435,6 +723,12 @@ void _drbd_thread_stop(struct drbd_thread *thi, int restart, int wait)
return;
}
+ if (thi->t_state == EXITING && ns == RESTARTING) {
+ /* Do not abort a stop request, otherwise a waiter might never wake up */
+ spin_unlock_irqrestore(&thi->t_lock, flags);
+ return;
+ }
+
if (thi->t_state != ns) {
if (thi->task == NULL) {
spin_unlock_irqrestore(&thi->t_lock, flags);
@@ -447,7 +741,6 @@ void _drbd_thread_stop(struct drbd_thread *thi, int restart, int wait)
if (thi->task != current)
send_sig(DRBD_SIGKILL, thi->task, 1);
}
-
spin_unlock_irqrestore(&thi->t_lock, flags);
if (wait)
@@ -465,7 +758,7 @@ static void drbd_calc_cpu_mask(cpumask_var_t *cpu_mask)
{
unsigned int *resources_per_cpu, min_index = ~0;
- resources_per_cpu = kzalloc_objs(*resources_per_cpu, nr_cpu_ids);
+ resources_per_cpu = kcalloc(nr_cpu_ids, sizeof(*resources_per_cpu), GFP_KERNEL);
if (resources_per_cpu) {
struct drbd_resource *resource;
unsigned int cpu, min = ~0;
@@ -512,6 +805,46 @@ void drbd_thread_current_set_cpu(struct drbd_thread *thi)
#define drbd_calc_cpu_mask(A) ({})
#endif
+static bool drbd_all_neighbor_secondary(struct drbd_device *device, u64 *authoritative_ptr)
+{
+ struct drbd_peer_device *peer_device;
+ bool all_secondary = true;
+ u64 authoritative = 0;
+ int id;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED &&
+ peer_device->connection->peer_role[NOW] == R_PRIMARY) {
+ all_secondary = false;
+ id = peer_device->node_id;
+ authoritative |= NODE_MASK(id);
+ }
+ }
+ rcu_read_unlock();
+ if (authoritative_ptr)
+ *authoritative_ptr = authoritative;
+ return all_secondary;
+}
+
+/* This function is supposed to have the same semantics as calc_device_stable() in drbd_state.c
+ A primary is stable since it is authoritative.
+ Unstable are neighbors of a primary and resync target nodes.
+ Nodes further away from a primary are stable! */
+bool drbd_device_stable(struct drbd_device *device, u64 *authoritative_ptr)
+{
+ struct drbd_resource *resource = device->resource;
+ bool device_stable = true;
+
+ if (resource->role[NOW] == R_PRIMARY)
+ return true;
+
+ if (!drbd_all_neighbor_secondary(device, authoritative_ptr))
+ return false;
+
+ return device_stable;
+}
+
/*
* drbd_header_size - size of a packet header
*
@@ -523,197 +856,457 @@ unsigned int drbd_header_size(struct drbd_connection *connection)
{
if (connection->agreed_pro_version >= 100) {
BUILD_BUG_ON(!IS_ALIGNED(sizeof(struct p_header100), 8));
- return sizeof(struct p_header100);
+ return sizeof(struct p_header100); /* 16 */
} else {
BUILD_BUG_ON(sizeof(struct p_header80) !=
sizeof(struct p_header95));
BUILD_BUG_ON(!IS_ALIGNED(sizeof(struct p_header80), 8));
- return sizeof(struct p_header80);
+ return sizeof(struct p_header80); /* 8 */
}
}
-static unsigned int prepare_header80(struct p_header80 *h, enum drbd_packet cmd, int size)
+static void prepare_header80(struct p_header80 *h, enum drbd_packet cmd, int size)
{
h->magic = cpu_to_be32(DRBD_MAGIC);
h->command = cpu_to_be16(cmd);
- h->length = cpu_to_be16(size);
- return sizeof(struct p_header80);
+ h->length = cpu_to_be16(size - sizeof(struct p_header80));
}
-static unsigned int prepare_header95(struct p_header95 *h, enum drbd_packet cmd, int size)
+static void prepare_header95(struct p_header95 *h, enum drbd_packet cmd, int size)
{
h->magic = cpu_to_be16(DRBD_MAGIC_BIG);
h->command = cpu_to_be16(cmd);
- h->length = cpu_to_be32(size);
- return sizeof(struct p_header95);
+ h->length = cpu_to_be32(size - sizeof(struct p_header95));
}
-static unsigned int prepare_header100(struct p_header100 *h, enum drbd_packet cmd,
+static void prepare_header100(struct p_header100 *h, enum drbd_packet cmd,
int size, int vnr)
{
h->magic = cpu_to_be32(DRBD_MAGIC_100);
h->volume = cpu_to_be16(vnr);
h->command = cpu_to_be16(cmd);
- h->length = cpu_to_be32(size);
+ h->length = cpu_to_be32(size - sizeof(struct p_header100));
h->pad = 0;
- return sizeof(struct p_header100);
}
-static unsigned int prepare_header(struct drbd_connection *connection, int vnr,
- void *buffer, enum drbd_packet cmd, int size)
+static void prepare_header(struct drbd_connection *connection, int vnr,
+ void *buffer, enum drbd_packet cmd, int size)
{
if (connection->agreed_pro_version >= 100)
- return prepare_header100(buffer, cmd, size, vnr);
+ prepare_header100(buffer, cmd, size, vnr);
else if (connection->agreed_pro_version >= 95 &&
size > DRBD_MAX_SIZE_H80_PACKET)
- return prepare_header95(buffer, cmd, size);
+ prepare_header95(buffer, cmd, size);
else
- return prepare_header80(buffer, cmd, size);
-}
-
-static void *__conn_prepare_command(struct drbd_connection *connection,
- struct drbd_socket *sock)
-{
- if (!sock->socket)
- return NULL;
- return sock->sbuf + drbd_header_size(connection);
+ prepare_header80(buffer, cmd, size);
}
-void *conn_prepare_command(struct drbd_connection *connection, struct drbd_socket *sock)
+static void new_or_recycle_send_buffer_page(struct drbd_send_buffer *sbuf)
{
- void *p;
+ while (1) {
+ struct page *page;
+ int count = page_count(sbuf->page);
- mutex_lock(&sock->mutex);
- p = __conn_prepare_command(connection, sock);
- if (!p)
- mutex_unlock(&sock->mutex);
+ BUG_ON(count == 0);
+ if (count == 1)
+ goto have_page;
- return p;
-}
+ page = alloc_page(GFP_NOIO | __GFP_NORETRY | __GFP_NOWARN);
+ if (page) {
+ put_page(sbuf->page);
+ sbuf->page = page;
+ goto have_page;
+ }
-void *drbd_prepare_command(struct drbd_peer_device *peer_device, struct drbd_socket *sock)
-{
- return conn_prepare_command(peer_device->connection, sock);
+ schedule_timeout_uninterruptible(HZ / 10);
+ }
+have_page:
+ sbuf->unsent =
+ sbuf->pos = page_address(sbuf->page);
}
-static int __send_command(struct drbd_connection *connection, int vnr,
- struct drbd_socket *sock, enum drbd_packet cmd,
- unsigned int header_size, void *data,
- unsigned int size)
+static char * __must_check alloc_send_buffer(struct drbd_connection *connection, int size,
+ enum drbd_stream drbd_stream)
{
- int msg_flags;
+ struct drbd_send_buffer *sbuf = &connection->send_buffer[drbd_stream];
+ char *page_start = page_address(sbuf->page);
int err;
- /*
- * Called with @data == NULL and the size of the data blocks in @size
- * for commands that send data blocks. For those commands, omit the
- * MSG_MORE flag: this will increase the likelihood that data blocks
- * which are page aligned on the sender will end up page aligned on the
- * receiver.
- */
- msg_flags = data ? MSG_MORE : 0;
-
- header_size += prepare_header(connection, vnr, sock->sbuf, cmd,
- header_size + size);
- err = drbd_send_all(connection, sock->socket, sock->sbuf, header_size,
- msg_flags);
- if (data && !err)
- err = drbd_send_all(connection, sock->socket, data, size, 0);
- /* DRBD protocol "pings" are latency critical.
- * This is supposed to trigger tcp_push_pending_frames() */
- if (!err && (cmd == P_PING || cmd == P_PING_ACK))
- tcp_sock_set_nodelay(sock->socket->sk);
+ if (sbuf->pos - page_start + size > PAGE_SIZE) {
+ err = flush_send_buffer(connection, drbd_stream);
+ if (err)
+ return ERR_PTR(err);
+ new_or_recycle_send_buffer_page(sbuf);
+ }
- return err;
+ sbuf->allocated_size = size;
+ sbuf->additional_size = 0;
+
+ return sbuf->pos;
}
-static int __conn_send_command(struct drbd_connection *connection, struct drbd_socket *sock,
- enum drbd_packet cmd, unsigned int header_size,
- void *data, unsigned int size)
+/* If we called alloc_send_buffer(), possibly indirectly via __conn_prepare_command(),
+ * but then decide that we actually don't want to use it.
+ */
+static void cancel_send_buffer(struct drbd_connection *connection,
+ enum drbd_stream drbd_stream)
{
- return __send_command(connection, 0, sock, cmd, header_size, data, size);
+ connection->send_buffer[drbd_stream].allocated_size = 0;
+ connection->send_buffer[drbd_stream].additional_size = 0;
}
-int conn_send_command(struct drbd_connection *connection, struct drbd_socket *sock,
- enum drbd_packet cmd, unsigned int header_size,
- void *data, unsigned int size)
+/* Only used the shrink the previously allocated size. */
+static void resize_prepared_command(struct drbd_connection *connection,
+ enum drbd_stream drbd_stream,
+ int size)
{
- int err;
+ connection->send_buffer[drbd_stream].allocated_size =
+ size + drbd_header_size(connection);
+}
- err = __conn_send_command(connection, sock, cmd, header_size, data, size);
- mutex_unlock(&sock->mutex);
- return err;
+static void additional_size_command(struct drbd_connection *connection,
+ enum drbd_stream drbd_stream,
+ int additional_size)
+{
+ connection->send_buffer[drbd_stream].additional_size = additional_size;
}
-int drbd_send_command(struct drbd_peer_device *peer_device, struct drbd_socket *sock,
- enum drbd_packet cmd, unsigned int header_size,
- void *data, unsigned int size)
+void *__conn_prepare_command(struct drbd_connection *connection, int size,
+ enum drbd_stream drbd_stream)
{
- int err;
+ struct drbd_transport *transport = &connection->transport;
+ int header_size;
+ void *p;
- err = __send_command(peer_device->connection, peer_device->device->vnr,
- sock, cmd, header_size, data, size);
- mutex_unlock(&sock->mutex);
- return err;
+ if (connection->cstate[NOW] < C_CONNECTING)
+ return NULL;
+
+ if (connection->send_buffer[drbd_stream].dead)
+ return NULL;
+
+ if (!transport->class->ops.stream_ok(transport, drbd_stream))
+ return NULL;
+
+ header_size = drbd_header_size(connection);
+ p = alloc_send_buffer(connection, header_size + size, drbd_stream);
+ if (IS_ERR(p))
+ return NULL;
+ return p + header_size;
}
-int drbd_send_ping(struct drbd_connection *connection)
+/**
+ * conn_prepare_command() - Allocate a send buffer for a packet/command
+ * @connection: the connections the packet will be sent through
+ * @size: number of bytes to allocate
+ * @drbd_stream: DATA_STREAM or CONTROL_STREAM
+ *
+ * This allocates a buffer with capacity to hold the header, and
+ * the requested size. Upon success is return a pointer that points
+ * to the first byte behind the header. The caller is expected to
+ * call xxx_send_command() soon.
+ */
+void *conn_prepare_command(struct drbd_connection *connection, int size,
+ enum drbd_stream drbd_stream)
{
- struct drbd_socket *sock;
+ void *p;
- sock = &connection->meta;
- if (!conn_prepare_command(connection, sock))
- return -EIO;
- return conn_send_command(connection, sock, P_PING, 0, NULL, 0);
+ mutex_lock(&connection->mutex[drbd_stream]);
+ p = __conn_prepare_command(connection, size, drbd_stream);
+ if (!p)
+ mutex_unlock(&connection->mutex[drbd_stream]);
+
+ return p;
}
-int drbd_send_ping_ack(struct drbd_connection *connection)
+/**
+ * drbd_prepare_command() - Allocate a send buffer for a packet/command
+ * @peer_device: the DRBD peer device the packet will be sent to
+ * @size: number of bytes to allocate
+ * @drbd_stream: DATA_STREAM or CONTROL_STREAM
+ *
+ * This allocates a buffer with capacity to hold the header, and
+ * the requested size. Upon success is return a pointer that points
+ * to the first byte behind the header. The caller is expected to
+ * call xxx_send_command() soon.
+ */
+void *drbd_prepare_command(struct drbd_peer_device *peer_device, int size, enum drbd_stream drbd_stream)
{
- struct drbd_socket *sock;
-
- sock = &connection->meta;
- if (!conn_prepare_command(connection, sock))
- return -EIO;
- return conn_send_command(connection, sock, P_PING_ACK, 0, NULL, 0);
+ return conn_prepare_command(peer_device->connection, size, drbd_stream);
}
-int drbd_send_sync_param(struct drbd_peer_device *peer_device)
+static int flush_send_buffer(struct drbd_connection *connection, enum drbd_stream drbd_stream)
{
- struct drbd_socket *sock;
- struct p_rs_param_95 *p;
- int size;
- const int apv = peer_device->connection->agreed_pro_version;
- enum drbd_packet cmd;
- struct net_conf *nc;
- struct disk_conf *dc;
+ struct drbd_send_buffer *sbuf = &connection->send_buffer[drbd_stream];
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
+ unsigned int flags, offset, size;
+ int err;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
- if (!p)
+ if (sbuf->dead)
return -EIO;
- rcu_read_lock();
- nc = rcu_dereference(peer_device->connection->net_conf);
+ size = sbuf->pos - sbuf->unsent + sbuf->allocated_size;
+ if (size == 0)
+ return 0;
- size = apv <= 87 ? sizeof(struct p_rs_param)
- : apv == 88 ? sizeof(struct p_rs_param)
- + strlen(nc->verify_alg) + 1
- : apv <= 94 ? sizeof(struct p_rs_param_89)
- : /* apv >= 95 */ sizeof(struct p_rs_param_95);
+ if (drbd_stream == CONTROL_STREAM) {
+ connection->ctl_packets++;
+ if (check_add_overflow(connection->ctl_bytes, size, &connection->ctl_bytes)) {
+ connection->ctl_bytes = size;
+ connection->ctl_packets = 1;
+ }
+ }
- cmd = apv >= 89 ? P_SYNC_PARAM89 : P_SYNC_PARAM;
+ if (drbd_stream == DATA_STREAM) {
+ rcu_read_lock();
+ connection->transport.ko_count = rcu_dereference(connection->transport.net_conf)->ko_count;
+ rcu_read_unlock();
+ }
- /* initialize verify_alg and csums_alg */
- BUILD_BUG_ON(sizeof(p->algs) != 2 * SHARED_SECRET_MAX);
- memset(&p->algs, 0, sizeof(p->algs));
+ flags = (connection->cstate[NOW] < C_CONNECTING ? MSG_DONTWAIT : 0) |
+ (sbuf->additional_size ? MSG_MORE : 0);
+ offset = sbuf->unsent - (char *)page_address(sbuf->page);
+ err = tr_ops->send_page(transport, drbd_stream, sbuf->page, offset, size, flags);
+ if (err)
+ sbuf->dead = true;
+
+ /* Advance past this range even when the send failed. send_page() may
+ * have put part of it on the wire, so it must not be sent a second
+ * time, and this stream is broken anyway. Retaining it would make
+ * every later flush retry it and fail again, until the next connect
+ * re-initializes the buffer.
+ */
+ sbuf->unsent =
+ sbuf->pos += sbuf->allocated_size; /* send buffer submitted! */
+ sbuf->allocated_size = 0;
+
+ return err;
+}
+
+/*
+ * SFLAG_FLUSH makes sure the packet (and everything queued in front
+ * of it) gets sent immediately independently if it is currently
+ * corked.
+ *
+ * This is used for P_PING, P_PING_ACK, P_TWOPC_PREPARE, P_TWOPC_ABORT,
+ * P_TWOPC_YES, P_TWOPC_NO, P_TWOPC_RETRY and P_TWOPC_COMMIT.
+ *
+ * This quirk is necessary because it is corked while the worker
+ * thread processes work items. When it stops processing items, it
+ * uncorks. That works perfectly to coalesce ack packets etc..
+ * A work item doing two-phase commits needs to override that behavior.
+ */
+#define SFLAG_FLUSH 0x10
+#define DRBD_STREAM_FLAGS (SFLAG_FLUSH)
+
+static inline enum drbd_stream extract_stream(int stream_and_flags)
+{
+ return stream_and_flags & ~DRBD_STREAM_FLAGS;
+}
+
+int __send_command(struct drbd_connection *connection, int vnr,
+ enum drbd_packet cmd, int stream_and_flags)
+{
+ enum drbd_stream drbd_stream = extract_stream(stream_and_flags);
+ struct drbd_send_buffer *sbuf = &connection->send_buffer[drbd_stream];
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
+ /* CORKED + drbd_stream is either DATA_CORKED or CONTROL_CORKED */
+ bool corked = test_bit(CORKED + drbd_stream, &connection->flags);
+ bool flush = stream_and_flags & SFLAG_FLUSH;
+ int err;
+
+ if (connection->cstate[NOW] < C_CONNECTING) {
+ cancel_send_buffer(connection, drbd_stream);
+ return -EIO;
+ }
+ prepare_header(connection, vnr, sbuf->pos, cmd,
+ sbuf->allocated_size + sbuf->additional_size);
+
+ if (corked && !flush) {
+ sbuf->pos += sbuf->allocated_size;
+ sbuf->allocated_size = 0;
+ err = 0;
+ } else {
+ err = flush_send_buffer(connection, drbd_stream);
+
+ /* DRBD protocol "pings" are latency critical.
+ * This is supposed to trigger tcp_push_pending_frames() */
+ if (!err && flush)
+ tr_ops->hint(transport, drbd_stream, NODELAY);
+
+ }
+
+ return err;
+}
+
+void drbd_cork(struct drbd_connection *connection, enum drbd_stream stream)
+{
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
+
+ mutex_lock(&connection->mutex[stream]);
+ set_bit(CORKED + stream, &connection->flags);
+ /* only call into transport, if we expect it to work */
+ if (connection->cstate[NOW] >= C_CONNECTING)
+ tr_ops->hint(transport, stream, CORK);
+ mutex_unlock(&connection->mutex[stream]);
+}
+
+int drbd_uncork(struct drbd_connection *connection, enum drbd_stream stream)
+{
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
+ int err;
+
+ mutex_lock(&connection->mutex[stream]);
+ /* Without a transport there is nothing to flush and nobody to tell.
+ * Dropping the cork state is all that is left to do, and reporting an
+ * error would only tell the caller about the connection we already
+ * know is gone.
+ */
+ if (connection->cstate[NOW] < C_CONNECTING) {
+ clear_bit(CORKED + stream, &connection->flags);
+ mutex_unlock(&connection->mutex[stream]);
+ return 0;
+ }
+
+ err = flush_send_buffer(connection, stream);
+ if (!err) {
+ clear_bit(CORKED + stream, &connection->flags);
+ tr_ops->hint(transport, stream, UNCORK);
+ }
+ mutex_unlock(&connection->mutex[stream]);
+
+ if (err)
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+
+ return err;
+}
+
+int send_command(struct drbd_connection *connection, int vnr,
+ enum drbd_packet cmd, int stream_and_flags)
+{
+ enum drbd_stream drbd_stream = extract_stream(stream_and_flags);
+ int err;
+
+ err = __send_command(connection, vnr, cmd, stream_and_flags);
+ mutex_unlock(&connection->mutex[drbd_stream]);
+ return err;
+}
+
+int drbd_send_command(struct drbd_peer_device *peer_device,
+ enum drbd_packet cmd, enum drbd_stream drbd_stream)
+{
+ return send_command(peer_device->connection, peer_device->device->vnr,
+ cmd, drbd_stream);
+}
+
+int drbd_send_ping(struct drbd_connection *connection)
+{
+ if (!conn_prepare_command(connection, 0, CONTROL_STREAM))
+ return -EIO;
+ return send_command(connection, -1, P_PING, CONTROL_STREAM | SFLAG_FLUSH);
+}
+
+/*
+ * send_ping_work and send_ping_ack_work run on global workqueues and do not
+ * otherwise keep the connection alive. Take a connection reference for each
+ * queued instance so the connection cannot be freed while a ping is still
+ * pending or running
+ */
+void drbd_queue_ping(struct drbd_connection *connection)
+{
+ bool queued;
+
+ kref_get(&connection->kref);
+ queued = schedule_work(&connection->send_ping_work);
+ if (!queued) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+}
+
+void drbd_queue_ping_ack(struct drbd_connection *connection)
+{
+ bool queued;
+
+ kref_get(&connection->kref);
+ queued = queue_work(ping_ack_sender, &connection->send_ping_ack_work);
+ if (!queued) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+}
+
+void drbd_send_ping_ack_wf(struct work_struct *ws)
+{
+ struct drbd_connection *connection =
+ container_of(ws, struct drbd_connection, send_ping_ack_work);
+ int err;
+
+ err = conn_prepare_command(connection, 0, CONTROL_STREAM) ? 0 : -EIO;
+ if (!err)
+ err = send_command(connection, -1, P_PING_ACK, CONTROL_STREAM | SFLAG_FLUSH);
+ if (err)
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ kref_put(&connection->kref, drbd_destroy_connection);
+}
+
+int drbd_send_peer_ack(struct drbd_connection *connection, u64 mask, u64 dagtag_sector)
+{
+ struct p_peer_ack *p;
+
+ p = conn_prepare_command(connection, sizeof(*p), CONTROL_STREAM);
+ if (!p)
+ return -EIO;
+ p->mask = cpu_to_be64(mask);
+ p->dagtag = cpu_to_be64(dagtag_sector);
+
+ return send_command(connection, -1, P_PEER_ACK, CONTROL_STREAM);
+}
+
+int drbd_send_sync_param(struct drbd_peer_device *peer_device)
+{
+ struct p_rs_param_95 *p;
+ int size, err;
+ const int apv = peer_device->connection->agreed_pro_version;
+ enum drbd_packet cmd;
+ struct drbd_net_conf *nc;
+ struct drbd_peer_device_conf *pdc;
+
+ rcu_read_lock();
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+
+ size = apv <= 87 ? sizeof(struct p_rs_param)
+ : apv == 88 ? sizeof(struct p_rs_param)
+ + strlen(nc->verify_alg) + 1
+ : apv <= 94 ? sizeof(struct p_rs_param_89)
+ : /* apv >= 95 */ sizeof(struct p_rs_param_95);
+
+ cmd = apv >= 89 ? P_SYNC_PARAM89 : P_SYNC_PARAM;
+ rcu_read_unlock();
+
+ p = drbd_prepare_command(peer_device, size, DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ /* initialize verify_alg and csums_alg */
+ memset(p->verify_alg, 0, sizeof(p->verify_alg));
+ memset(p->csums_alg, 0, sizeof(p->csums_alg));
+
+ rcu_read_lock();
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
if (get_ldev(peer_device->device)) {
- dc = rcu_dereference(peer_device->device->ldev->disk_conf);
- p->resync_rate = cpu_to_be32(dc->resync_rate);
- p->c_plan_ahead = cpu_to_be32(dc->c_plan_ahead);
- p->c_delay_target = cpu_to_be32(dc->c_delay_target);
- p->c_fill_target = cpu_to_be32(dc->c_fill_target);
- p->c_max_rate = cpu_to_be32(dc->c_max_rate);
+ pdc = rcu_dereference(peer_device->conf);
+ /* These values will be ignored by peers running DRBD 9.2+, but
+ * we have to send something, so send the real values. We
+ * cannot omit the entire packet because we must verify that
+ * the algorithms match. */
+ p->resync_rate = cpu_to_be32(pdc->resync_rate);
+ p->c_plan_ahead = cpu_to_be32(pdc->c_plan_ahead);
+ p->c_delay_target = cpu_to_be32(pdc->c_delay_target);
+ p->c_fill_target = cpu_to_be32(pdc->c_fill_target);
+ p->c_max_rate = cpu_to_be32(pdc->c_max_rate);
put_ldev(peer_device->device);
} else {
p->resync_rate = cpu_to_be32(DRBD_RESYNC_RATE_DEF);
@@ -729,36 +1322,53 @@ int drbd_send_sync_param(struct drbd_peer_device *peer_device)
strscpy(p->csums_alg, nc->csums_alg);
rcu_read_unlock();
- return drbd_send_command(peer_device, sock, cmd, size, NULL, 0);
+ err = drbd_send_command(peer_device, cmd, DATA_STREAM);
+
+ /* A peer with established replication adopts the verify-alg we
+ * advertise, see receive_SyncParam(). Keep our record of the peer's
+ * algorithm current. Must not be called with conf_update held.
+ */
+ if (!err && apv >= 88 && peer_device->repl_state[NOW] != L_OFF) {
+ struct drbd_connection *connection = peer_device->connection;
+
+ mutex_lock(&connection->resource->conf_update);
+ nc = connection->transport.net_conf;
+ strscpy(connection->peer_verify_alg, nc->verify_alg,
+ sizeof(connection->peer_verify_alg));
+ mutex_unlock(&connection->resource->conf_update);
+ }
+
+ return err;
}
int __drbd_send_protocol(struct drbd_connection *connection, enum drbd_packet cmd)
{
- struct drbd_socket *sock;
struct p_protocol *p;
- struct net_conf *nc;
+ struct drbd_net_conf *nc;
size_t integrity_alg_len;
int size, cf;
- sock = &connection->data;
- p = __conn_prepare_command(connection, sock);
- if (!p)
- return -EIO;
-
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
-
- if (nc->tentative && connection->agreed_pro_version < 92) {
- rcu_read_unlock();
+ if (test_bit(CONN_DRY_RUN, &connection->flags) && connection->agreed_pro_version < 92) {
+ clear_bit(CONN_DRY_RUN, &connection->flags);
drbd_err(connection, "--dry-run is not supported by peer");
return -EOPNOTSUPP;
}
size = sizeof(*p);
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
if (connection->agreed_pro_version >= 87) {
integrity_alg_len = strlen(nc->integrity_alg) + 1;
size += integrity_alg_len;
}
+ rcu_read_unlock();
+
+ p = __conn_prepare_command(connection, size, DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
p->protocol = cpu_to_be32(nc->wire_protocol);
p->after_sb_0p = cpu_to_be32(nc->after_sb_0p);
@@ -766,9 +1376,9 @@ int __drbd_send_protocol(struct drbd_connection *connection, enum drbd_packet cm
p->after_sb_2p = cpu_to_be32(nc->after_sb_2p);
p->two_primaries = cpu_to_be32(nc->two_primaries);
cf = 0;
- if (nc->discard_my_data)
+ if (test_bit(CONN_DISCARD_MY_DATA, &connection->flags))
cf |= CF_DISCARD_MY_DATA;
- if (nc->tentative)
+ if (test_bit(CONN_DRY_RUN, &connection->flags))
cf |= CF_DRY_RUN;
p->conn_flags = cpu_to_be32(cf);
@@ -776,133 +1386,312 @@ int __drbd_send_protocol(struct drbd_connection *connection, enum drbd_packet cm
strscpy(p->integrity_alg, nc->integrity_alg, integrity_alg_len);
rcu_read_unlock();
- return __conn_send_command(connection, sock, cmd, size, NULL, 0);
-}
-
-int drbd_send_protocol(struct drbd_connection *connection)
-{
- int err;
-
- mutex_lock(&connection->data.mutex);
- err = __drbd_send_protocol(connection, P_PROTOCOL);
- mutex_unlock(&connection->data.mutex);
-
- return err;
+ return __send_command(connection, -1, cmd, DATA_STREAM);
}
static int _drbd_send_uuids(struct drbd_peer_device *peer_device, u64 uuid_flags)
{
struct drbd_device *device = peer_device->device;
- struct drbd_socket *sock;
struct p_uuids *p;
int i;
if (!get_ldev_if_state(device, D_NEGOTIATING))
return 0;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
if (!p) {
put_ldev(device);
return -EIO;
}
+
spin_lock_irq(&device->ldev->md.uuid_lock);
- for (i = UI_CURRENT; i < UI_SIZE; i++)
- p->uuid[i] = cpu_to_be64(device->ldev->md.uuid[i]);
+ p->current_uuid = cpu_to_be64(drbd_current_uuid(device));
+ p->bitmap_uuid = cpu_to_be64(drbd_bitmap_uuid(peer_device));
+ for (i = 0; i < ARRAY_SIZE(p->history_uuids); i++)
+ p->history_uuids[i] = cpu_to_be64(drbd_history_uuid(device, i));
spin_unlock_irq(&device->ldev->md.uuid_lock);
- device->comm_bm_set = drbd_bm_total_weight(device);
- p->uuid[UI_SIZE] = cpu_to_be64(device->comm_bm_set);
+ peer_device->comm_bm_set = drbd_bm_total_weight(peer_device);
+ p->dirty_bits = cpu_to_be64(peer_device->comm_bm_set);
+
+ if (test_bit(DISCARD_MY_DATA, peer_device->flags))
+ uuid_flags |= UUID_FLAG_DISCARD_MY_DATA;
+ if (test_bit(CRASHED_PRIMARY, &device->flags))
+ uuid_flags |= UUID_FLAG_CRASHED_PRIMARY;
+ if (!drbd_md_test_flag(device->ldev, MDF_CONSISTENT))
+ uuid_flags |= UUID_FLAG_INCONSISTENT;
+
+ /* Silently mask out any "too recent" flags,
+ * we cannot communicate those in old DRBD
+ * protocol versions. */
+ uuid_flags &= UUID_FLAG_MASK_COMPAT_84;
+
+ peer_device->comm_uuid_flags = uuid_flags;
+ p->uuid_flags = cpu_to_be64(uuid_flags);
+
+ put_ldev(device);
+
+ return drbd_send_command(peer_device, P_UUIDS, DATA_STREAM);
+}
+
+static u64 __bitmap_uuid(struct drbd_device *device, int node_id)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ u64 bitmap_uuid = peer_md[node_id].bitmap_uuid;
+
+ /* Sending a bitmap_uuid of 0 means that we are in sync with that peer.
+ The recipient of this message might use this assumption to throw away it's
+ bitmap to that peer.
+
+ Send -1 instead if we are (resync target from that peer) not at the same
+ current uuid.
+ This corner case is relevant if we finish resync from an UpToDate peer first,
+ and the second resync (which was paused first) is from an Outdated node.
+ And that second resync gets canceled by the resync target due to the first
+ resync finished successfully.
+
+ Exceptions to the above are when the peer's UUID is not known yet
+ */
+
rcu_read_lock();
- uuid_flags |= rcu_dereference(peer_device->connection->net_conf)->discard_my_data ? 1 : 0;
+ peer_device = peer_device_by_node_id(device, node_id);
+ if (peer_device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+
+ if (bitmap_uuid == 0 &&
+ (repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T) &&
+ peer_device->current_uuid != 0 &&
+ (peer_device->current_uuid & ~UUID_PRIMARY) !=
+ (drbd_current_uuid(device) & ~UUID_PRIMARY))
+ bitmap_uuid = -1;
+ }
rcu_read_unlock();
- uuid_flags |= test_bit(CRASHED_PRIMARY, &device->flags) ? 2 : 0;
- uuid_flags |= device->new_state_tmp.disk == D_INCONSISTENT ? 4 : 0;
- p->uuid[UI_FLAGS] = cpu_to_be64(uuid_flags);
- put_ldev(device);
- return drbd_send_command(peer_device, sock, P_UUIDS, sizeof(*p), NULL, 0);
+ return bitmap_uuid;
+}
+
+u64 drbd_collect_local_uuid_flags(struct drbd_peer_device *peer_device, u64 *authoritative_mask)
+{
+ struct drbd_device *device = peer_device->device;
+ u64 uuid_flags = 0;
+
+ if (test_bit(DISCARD_MY_DATA, peer_device->flags))
+ uuid_flags |= UUID_FLAG_DISCARD_MY_DATA;
+ if (test_bit(CRASHED_PRIMARY, &device->flags))
+ uuid_flags |= UUID_FLAG_CRASHED_PRIMARY;
+ if (!drbd_md_test_flag(device->ldev, MDF_CONSISTENT))
+ uuid_flags |= UUID_FLAG_INCONSISTENT;
+ if (test_bit(RECONNECT, &peer_device->connection->flags))
+ uuid_flags |= UUID_FLAG_RECONNECT;
+ if (test_bit(PRIMARY_LOST_QUORUM, &device->flags))
+ uuid_flags |= UUID_FLAG_PRIMARY_LOST_QUORUM;
+ if (drbd_device_stable(device, authoritative_mask))
+ uuid_flags |= UUID_FLAG_STABLE;
+ if (peer_device->connection->agreed_pro_version >= 125 &&
+ drbd_bitmap_slot_decides(peer_device))
+ uuid_flags |= UUID_FLAG_BITMAP_AUTHORITATIVE;
+
+ return uuid_flags;
+}
+
+/* sets UUID_FLAG_SYNC_TARGET on uuid_flags as appropriate (may be NULL) */
+u64 drbd_resolved_uuid(struct drbd_peer_device *peer_device_base, u64 *uuid_flags)
+{
+ struct drbd_device *device = peer_device_base->device;
+ struct drbd_peer_device *peer_device;
+ u64 uuid = drbd_current_uuid(device);
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->node_id == peer_device_base->node_id)
+ continue;
+ if (peer_device->repl_state[NOW] == L_SYNC_TARGET) {
+ uuid = peer_device->current_uuid;
+ if (uuid_flags)
+ *uuid_flags |= UUID_FLAG_SYNC_TARGET;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return uuid;
}
-int drbd_send_uuids(struct drbd_peer_device *peer_device)
+static int _drbd_send_uuids110(struct drbd_peer_device *peer_device, u64 uuid_flags, u64 node_mask)
{
- return _drbd_send_uuids(peer_device, 0);
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md;
+ struct p_uuids110 *p;
+ int i, pos = 0, day0 = -1, sent_day0 = -1;
+ u64 local_uuid_flags = 0, authoritative_mask, bitmap_uuids_mask = 0;
+ int p_size = sizeof(*p);
+
+ if (!get_ldev_if_state(device, D_NEGOTIATING)) {
+ WRITE_ONCE(peer_device->comm_day0_node_id, -1);
+ return drbd_send_current_uuid(peer_device,
+ diskless_primary_present_current_uuid(peer_device),
+ drbd_weak_nodes_device(device));
+ }
+
+ peer_md = device->ldev->md.peers;
+
+ p_size += (DRBD_PEERS_MAX + HISTORY_UUIDS) * sizeof(p->other_uuids[0]);
+ p = drbd_prepare_command(peer_device, p_size, DATA_STREAM);
+ if (!p) {
+ put_ldev(device);
+ return -EIO;
+ }
+
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+ peer_device->comm_current_uuid = drbd_resolved_uuid(peer_device, &local_uuid_flags);
+ p->current_uuid = cpu_to_be64(peer_device->comm_current_uuid);
+
+ /* Advertise an unallocated day0 bitmap slot only if we actually have
+ * one. drbd_find_bitmap_by_uuid() considers only the one we sent.
+ */
+ if (peer_device->connection->agreed_pro_version >= 116 &&
+ drbd_unallocated_index(device->ldev) != -1)
+ day0 = drbd_day0_node_id(&device->ldev->md, peer_device->node_id);
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ u64 val = __bitmap_uuid(device, i);
+ bool send_this = test_bit(__MDF_HAVE_BITMAP, &peer_md[i].flags) ||
+ test_bit(__MDF_NODE_EXISTS, &peer_md[i].flags);
+ if (i == day0 && val) {
+ send_this = true;
+ sent_day0 = i;
+ uuid_flags |= (u64)i << UUID_FLAG_UNALLOC_SHIFT;
+ uuid_flags |= UUID_FLAG_HAS_UNALLOC;
+ }
+ if (send_this) {
+ bitmap_uuids_mask |= NODE_MASK(i);
+ p->other_uuids[pos++] = cpu_to_be64(val);
+ }
+ }
+ WRITE_ONCE(peer_device->comm_day0_node_id, sent_day0);
+ peer_device->comm_bitmap_uuid = drbd_bitmap_uuid(peer_device);
+
+ for (i = 0; i < HISTORY_UUIDS; i++)
+ p->other_uuids[pos++] = cpu_to_be64(drbd_history_uuid(device, i));
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+
+ p->bitmap_uuids_mask = cpu_to_be64(bitmap_uuids_mask);
+
+ peer_device->comm_bm_set = drbd_bm_total_weight(peer_device);
+ p->dirty_bits = cpu_to_be64(peer_device->comm_bm_set);
+ local_uuid_flags |= drbd_collect_local_uuid_flags(peer_device, &authoritative_mask);
+ peer_device->comm_uuid_flags = local_uuid_flags;
+ uuid_flags |= local_uuid_flags;
+ if (uuid_flags & UUID_FLAG_STABLE) {
+ p->node_mask = cpu_to_be64(node_mask);
+ } else {
+ D_ASSERT(peer_device, node_mask == 0);
+ p->node_mask = cpu_to_be64(authoritative_mask);
+ }
+
+ p->uuid_flags = cpu_to_be64(uuid_flags);
+
+ put_ldev(device);
+
+ p_size = sizeof(*p) +
+ (hweight64(bitmap_uuids_mask) + HISTORY_UUIDS) * sizeof(p->other_uuids[0]);
+ resize_prepared_command(peer_device->connection, DATA_STREAM, p_size);
+ return drbd_send_command(peer_device, P_UUIDS110, DATA_STREAM);
}
-int drbd_send_uuids_skip_initial_sync(struct drbd_peer_device *peer_device)
+int drbd_send_uuids(struct drbd_peer_device *peer_device, u64 uuid_flags, u64 node_mask)
{
- return _drbd_send_uuids(peer_device, 8);
+ if (peer_device->connection->agreed_pro_version >= 110)
+ return _drbd_send_uuids110(peer_device, uuid_flags, node_mask);
+ else
+ return _drbd_send_uuids(peer_device, uuid_flags);
}
-void drbd_print_uuids(struct drbd_device *device, const char *text)
+void drbd_print_uuids(struct drbd_peer_device *peer_device, const char *text)
{
+ struct drbd_device *device = peer_device->device;
+
if (get_ldev_if_state(device, D_NEGOTIATING)) {
- u64 *uuid = device->ldev->md.uuid;
- drbd_info(device, "%s %016llX:%016llX:%016llX:%016llX\n",
- text,
- (unsigned long long)uuid[UI_CURRENT],
- (unsigned long long)uuid[UI_BITMAP],
- (unsigned long long)uuid[UI_HISTORY_START],
- (unsigned long long)uuid[UI_HISTORY_END]);
+ drbd_info(peer_device, "%s %016llX:%016llX:%016llX:%016llX\n",
+ text,
+ (unsigned long long)drbd_current_uuid(device),
+ (unsigned long long)drbd_bitmap_uuid(peer_device),
+ (unsigned long long)drbd_history_uuid(device, 0),
+ (unsigned long long)drbd_history_uuid(device, 1));
put_ldev(device);
} else {
- drbd_info(device, "%s effective data uuid: %016llX\n",
- text,
- (unsigned long long)device->ed_uuid);
+ drbd_info(peer_device, "%s exposed data uuid: %016llX\n",
+ text,
+ (unsigned long long)device->exposed_data_uuid);
}
}
+int drbd_send_current_uuid(struct drbd_peer_device *peer_device, u64 current_uuid, u64 weak_nodes)
+{
+ struct p_current_uuid *p;
+
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ peer_device->comm_current_uuid = current_uuid;
+ p->uuid = cpu_to_be64(current_uuid);
+ p->weak_nodes = cpu_to_be64(weak_nodes);
+ return drbd_send_command(peer_device, P_CURRENT_UUID, DATA_STREAM);
+}
+
void drbd_gen_and_send_sync_uuid(struct drbd_peer_device *peer_device)
{
struct drbd_device *device = peer_device->device;
- struct drbd_socket *sock;
- struct p_rs_uuid *p;
+ struct p_uuid *p;
u64 uuid;
- D_ASSERT(device, device->state.disk == D_UP_TO_DATE);
+ D_ASSERT(device, device->disk_state[NOW] == D_UP_TO_DATE);
- uuid = device->ldev->md.uuid[UI_BITMAP];
+ down_write(&device->uuid_sem);
+ uuid = drbd_bitmap_uuid(peer_device);
if (uuid && uuid != UUID_JUST_CREATED)
uuid = uuid + UUID_NEW_BM_OFFSET;
else
uuid = get_random_u64();
- drbd_uuid_set(device, UI_BITMAP, uuid);
- drbd_print_uuids(device, "updated sync UUID");
+ drbd_uuid_set_bitmap(peer_device, uuid);
+ drbd_print_uuids(peer_device, "updated sync UUID");
drbd_md_sync(device);
+ downgrade_write(&device->uuid_sem);
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
if (p) {
p->uuid = cpu_to_be64(uuid);
- drbd_send_command(peer_device, sock, P_SYNC_UUID, sizeof(*p), NULL, 0);
+ drbd_send_command(peer_device, P_SYNC_UUID, DATA_STREAM);
}
+ up_read(&device->uuid_sem);
}
-int drbd_send_sizes(struct drbd_peer_device *peer_device, int trigger_reply, enum dds_flags flags)
+int drbd_send_sizes(struct drbd_peer_device *peer_device,
+ uint64_t u_size_diskless, enum dds_flags flags)
{
+ struct drbd_connection *connection = peer_device->connection;
struct drbd_device *device = peer_device->device;
- struct drbd_socket *sock;
struct p_sizes *p;
sector_t d_size, u_size;
int q_order_type;
unsigned int max_bio_size;
unsigned int packet_size;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
- if (!p)
- return -EIO;
-
packet_size = sizeof(*p);
- if (peer_device->connection->agreed_features & DRBD_FF_WSAME)
+ if (connection->agreed_features & DRBD_FF_WSAME)
packet_size += sizeof(p->qlim[0]);
+ p = drbd_prepare_command(peer_device, packet_size, DATA_STREAM);
+ if (!p)
+ return -EIO;
+
memset(p, 0, packet_size);
if (get_ldev_if_state(device, D_NEGOTIATING)) {
struct block_device *bdev = device->ldev->backing_bdev;
struct request_queue *q = bdev_get_queue(bdev);
- d_size = drbd_get_max_capacity(device->ldev);
+ d_size = drbd_get_max_capacity(device, device->ldev, false);
rcu_read_lock();
u_size = rcu_dereference(device->ldev->disk_conf)->disk_size;
rcu_read_unlock();
@@ -918,6 +1707,10 @@ int drbd_send_sizes(struct drbd_peer_device *peer_device, int trigger_reply, enu
p->qlim->io_min = cpu_to_be32(bdev_io_min(bdev));
p->qlim->io_opt = cpu_to_be32(bdev_io_opt(bdev));
p->qlim->discard_enabled = !!bdev_max_discard_sectors(bdev);
+ p->qlim->write_same_capable = 0;
+ if (connection->agreed_features & DRBD_FF_BM_BLOCK_SHIFT)
+ p->qlim->bm_block_shift_minus_12 =
+ device->bitmap->bm_block_shift - BM_BLOCK_SHIFT_4k;
put_ldev(device);
} else {
struct request_queue *q = device->rq_queue;
@@ -930,128 +1723,332 @@ int drbd_send_sizes(struct drbd_peer_device *peer_device, int trigger_reply, enu
p->qlim->io_min = cpu_to_be32(queue_io_min(q));
p->qlim->io_opt = cpu_to_be32(queue_io_opt(q));
p->qlim->discard_enabled = 0;
+ p->qlim->write_same_capable = 0;
d_size = 0;
- u_size = 0;
+ u_size = u_size_diskless;
q_order_type = QUEUE_ORDERED_NONE;
max_bio_size = DRBD_MAX_BIO_SIZE; /* ... multiple BIOs per peer_request */
}
- if (peer_device->connection->agreed_pro_version <= 94)
+ if (connection->agreed_pro_version <= 94)
max_bio_size = min(max_bio_size, DRBD_MAX_SIZE_H80_PACKET);
- else if (peer_device->connection->agreed_pro_version < 100)
+ else if (connection->agreed_pro_version < 100)
max_bio_size = min(max_bio_size, DRBD_MAX_BIO_SIZE_P95);
+ /* 9.0.4 bumped pro_version to 112 and introduced 2PC resizes */
+ if (connection->agreed_pro_version >= 112)
+ d_size = drbd_partition_data_capacity(device);
+
p->d_size = cpu_to_be64(d_size);
p->u_size = cpu_to_be64(u_size);
- if (trigger_reply)
- p->c_size = 0;
- else
- p->c_size = cpu_to_be64(get_capacity(device->vdisk));
+ /*
+ TODO verify: this may be needed for v8 compatibility still.
+ p->c_size = cpu_to_be64(trigger_reply ? 0 : get_capacity(device->vdisk));
+ */
+ p->c_size = cpu_to_be64(get_capacity(device->vdisk));
p->max_bio_size = cpu_to_be32(max_bio_size);
p->queue_order_type = cpu_to_be16(q_order_type);
p->dds_flags = cpu_to_be16(flags);
- return drbd_send_command(peer_device, sock, P_SIZES, packet_size, NULL, 0);
+ return drbd_send_command(peer_device, P_SIZES, DATA_STREAM);
}
-/**
- * drbd_send_current_state() - Sends the drbd state to the peer
- * @peer_device: DRBD peer device.
- */
int drbd_send_current_state(struct drbd_peer_device *peer_device)
{
- struct drbd_socket *sock;
+ return drbd_send_state(peer_device, drbd_get_peer_device_state(peer_device, NOW));
+}
+
+static int send_state(struct drbd_connection *connection, int vnr, union drbd_state state)
+{
struct p_state *p;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
if (!p)
return -EIO;
- p->state = cpu_to_be32(peer_device->device->state.i); /* Within the send mutex */
- return drbd_send_command(peer_device, sock, P_STATE, sizeof(*p), NULL, 0);
+
+ if (connection->agreed_pro_version < 110)
+ drbd_disk_states_to_84(&state);
+
+ p->state = cpu_to_be32(state.i); /* Within the send mutex */
+ return send_command(connection, vnr, P_STATE, DATA_STREAM);
+}
+
+int conn_send_state(struct drbd_connection *connection, union drbd_state state)
+{
+ BUG_ON(connection->agreed_pro_version < 100);
+ return send_state(connection, -1, state);
}
/**
- * drbd_send_state() - After a state change, sends the new state to the peer
- * @peer_device: DRBD peer device.
- * @state: the state to send, not necessarily the current state.
- *
- * Each state change queues an "after_state_ch" work, which will eventually
- * send the resulting new state to the peer. If more state changes happen
- * between queuing and processing of the after_state_ch work, we still
- * want to send each intermediary state in the order it occurred.
+ * drbd_send_state() - Sends the drbd state to the peer
+ * @peer_device: Peer DRBD device to send the state to.
+ * @state: state to send
*/
int drbd_send_state(struct drbd_peer_device *peer_device, union drbd_state state)
{
- struct drbd_socket *sock;
- struct p_state *p;
+ peer_device->comm_state = state;
+ return send_state(peer_device->connection, peer_device->device->vnr, state);
+}
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
- if (!p)
- return -EIO;
- p->state = cpu_to_be32(state.i); /* Within the send mutex */
- return drbd_send_command(peer_device, sock, P_STATE, sizeof(*p), NULL, 0);
+/**
+ * drbd_send_uuids_peer_attach() - Send P_UUIDS and P_STATE for a peer attach
+ * @peer_device: Peer DRBD device that attached.
+ *
+ * The worker and the receiver both call this; whichever clears
+ * SEND_UUIDS_PEER_ATTACH first does the sending. The mutex covers the
+ * clear and the sends, so when this returns, P_UUIDS has been sent and
+ * comm_bm_set and comm_uuid_flags hold what the peer decides on, no
+ * matter which thread sent them.
+ *
+ * The state is read after the flag is cleared, so it includes the state
+ * change that set the flag. A worker item queued for an earlier attach
+ * that clears the flag of a later one therefore sends a state that
+ * matches the later attach, not its own stale one.
+ */
+void drbd_send_uuids_peer_attach(struct drbd_peer_device *peer_device)
+{
+ mutex_lock(&peer_device->peer_attach_send_mutex);
+ if (test_and_clear_bit(SEND_UUIDS_PEER_ATTACH, peer_device->flags)) {
+ drbd_send_uuids(peer_device, 0, 0);
+ drbd_send_current_state(peer_device);
+ }
+ mutex_unlock(&peer_device->peer_attach_send_mutex);
}
-int drbd_send_state_req(struct drbd_peer_device *peer_device, union drbd_state mask, union drbd_state val)
+int conn_send_state_req(struct drbd_connection *connection, int vnr, enum drbd_packet cmd,
+ union drbd_state mask, union drbd_state val)
{
- struct drbd_socket *sock;
struct p_req_state *p;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
+ /* Protocols before version 100 only support one volume and connection.
+ * All state change requests are via P_STATE_CHG_REQ. */
+ if (connection->agreed_pro_version < 100)
+ cmd = P_STATE_CHG_REQ;
+
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
if (!p)
return -EIO;
+
+ /* Only val holds disk states; mask.disk and mask.pdsk are bit masks. */
+ if (connection->agreed_pro_version < 110)
+ drbd_disk_states_to_84(&val);
+
p->mask = cpu_to_be32(mask.i);
p->val = cpu_to_be32(val.i);
- return drbd_send_command(peer_device, sock, P_STATE_CHG_REQ, sizeof(*p), NULL, 0);
+
+ return send_command(connection, vnr, cmd, DATA_STREAM);
}
-int conn_send_state_req(struct drbd_connection *connection, union drbd_state mask, union drbd_state val)
+int conn_send_twopc_request(struct drbd_connection *connection, struct twopc_request *request)
{
- enum drbd_packet cmd;
- struct drbd_socket *sock;
- struct p_req_state *p;
+ struct drbd_resource *resource = connection->resource;
+ struct p_twopc_request *p;
+
+ dynamic_drbd_dbg(connection, "Sending %s request for state change %u\n",
+ drbd_packet_name(request->cmd),
+ request->tid);
- cmd = connection->agreed_pro_version < 100 ? P_STATE_CHG_REQ : P_CONN_ST_CHG_REQ;
- sock = &connection->data;
- p = conn_prepare_command(connection, sock);
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
if (!p)
return -EIO;
- p->mask = cpu_to_be32(mask.i);
- p->val = cpu_to_be32(val.i);
- return conn_send_command(connection, sock, cmd, sizeof(*p), NULL, 0);
+ p->tid = cpu_to_be32(request->tid);
+ if (connection->agreed_features & DRBD_FF_2PC_V2) {
+ p->flags = cpu_to_be32(TWOPC_HAS_FLAGS | request->flags);
+ p->_pad = 0;
+ p->s8_initiator_node_id = request->initiator_node_id;
+ p->s8_target_node_id = request->target_node_id;
+ } else {
+ p->u32_initiator_node_id = cpu_to_be32(request->initiator_node_id);
+ p->u32_target_node_id = cpu_to_be32(request->target_node_id);
+ }
+ p->nodes_to_reach = cpu_to_be64(request->nodes_to_reach);
+ switch (resource->twopc.type) {
+ case TWOPC_STATE_CHANGE:
+ if (request->cmd == P_TWOPC_PREPARE) {
+ p->_compat_pad = 0;
+ p->mask = cpu_to_be32(resource->twopc.state_change.mask.i);
+ p->val = cpu_to_be32(resource->twopc.state_change.val.i);
+ } else { /* P_TWOPC_COMMIT */
+ p->primary_nodes = cpu_to_be64(resource->twopc.state_change.primary_nodes);
+ if (request->flags & TWOPC_HAS_REACHABLE &&
+ connection->agreed_features & DRBD_FF_2PC_V2) {
+ p->reachable_nodes = cpu_to_be64(
+ resource->twopc.state_change.reachable_nodes);
+ } else {
+ p->mask = cpu_to_be32(resource->twopc.state_change.mask.i);
+ p->val = cpu_to_be32(resource->twopc.state_change.val.i);
+ }
+ }
+ break;
+ case TWOPC_RESIZE:
+ if (request->cmd == P_TWOPC_PREP_RSZ) {
+ p->user_size = cpu_to_be64(resource->twopc.resize.user_size);
+ p->dds_flags = cpu_to_be16(resource->twopc.resize.dds_flags);
+ } else { /* P_TWOPC_COMMIT */
+ p->diskful_primary_nodes =
+ cpu_to_be64(resource->twopc.resize.diskful_primary_nodes);
+ p->exposed_size = cpu_to_be64(resource->twopc.resize.new_size);
+ }
+ }
+ return send_command(connection, request->vnr, request->cmd, DATA_STREAM | SFLAG_FLUSH);
}
-void drbd_send_sr_reply(struct drbd_peer_device *peer_device, enum drbd_state_rv retcode)
+void drbd_send_sr_reply(struct drbd_connection *connection, int vnr, enum drbd_state_rv retcode)
{
- struct drbd_socket *sock;
struct p_req_state_reply *p;
- sock = &peer_device->connection->meta;
- p = drbd_prepare_command(peer_device, sock);
+ p = conn_prepare_command(connection, sizeof(*p), CONTROL_STREAM);
if (p) {
+ enum drbd_packet cmd = P_STATE_CHG_REPLY;
+
+ if (connection->agreed_pro_version >= 100 && vnr < 0)
+ cmd = P_CONN_ST_CHG_REPLY;
+
p->retcode = cpu_to_be32(retcode);
- drbd_send_command(peer_device, sock, P_STATE_CHG_REPLY, sizeof(*p), NULL, 0);
+ send_command(connection, vnr, cmd, CONTROL_STREAM);
}
}
-void conn_send_sr_reply(struct drbd_connection *connection, enum drbd_state_rv retcode)
+void drbd_send_twopc_reply(struct drbd_connection *connection,
+ enum drbd_packet cmd, struct twopc_reply *reply)
{
- struct drbd_socket *sock;
- struct p_req_state_reply *p;
- enum drbd_packet cmd = connection->agreed_pro_version < 100 ? P_STATE_CHG_REPLY : P_CONN_ST_CHG_REPLY;
+ struct p_twopc_reply *p;
- sock = &connection->meta;
- p = conn_prepare_command(connection, sock);
+ p = conn_prepare_command(connection, sizeof(*p), CONTROL_STREAM);
if (p) {
- p->retcode = cpu_to_be32(retcode);
- conn_send_command(connection, sock, cmd, sizeof(*p), NULL, 0);
+ p->tid = cpu_to_be32(reply->tid);
+ p->initiator_node_id = cpu_to_be32(reply->initiator_node_id);
+ p->reachable_nodes = cpu_to_be64(reply->reachable_nodes);
+ switch (connection->resource->twopc.type) {
+ case TWOPC_STATE_CHANGE:
+ p->primary_nodes = cpu_to_be64(reply->primary_nodes);
+ p->weak_nodes = cpu_to_be64(reply->weak_nodes);
+ break;
+ case TWOPC_RESIZE:
+ p->diskful_primary_nodes = cpu_to_be64(reply->diskful_primary_nodes);
+ p->max_possible_size = cpu_to_be64(reply->max_possible_size);
+ break;
+ }
+ send_command(connection, reply->vnr, cmd, CONTROL_STREAM | SFLAG_FLUSH);
+ }
+}
+
+void drbd_send_peers_in_sync(struct drbd_peer_device *peer_device, u64 mask, sector_t sector, int size)
+{
+ struct p_peer_block_desc *p;
+
+ p = drbd_prepare_command(peer_device, sizeof(*p), CONTROL_STREAM);
+ if (p) {
+ p->sector = cpu_to_be64(sector);
+ p->mask = cpu_to_be64(mask);
+ p->size = cpu_to_be32(size);
+ p->pad = 0;
+ drbd_send_command(peer_device, P_PEERS_IN_SYNC, CONTROL_STREAM);
}
}
+int drbd_send_peer_dagtag(struct drbd_connection *connection, struct drbd_connection *lost_peer)
+{
+ struct p_peer_dagtag *p;
+
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ p->dagtag = cpu_to_be64(atomic64_read(&lost_peer->last_dagtag_sector));
+ p->node_id = cpu_to_be32(lost_peer->peer_node_id);
+
+ return send_command(connection, -1, P_PEER_DAGTAG, DATA_STREAM);
+}
+
+int drbd_send_flush_requests(struct drbd_connection *connection, u64 flush_sequence)
+{
+ struct p_flush_requests *p;
+
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ p->flush_sequence = cpu_to_be64(flush_sequence);
+
+ return send_command(connection, -1, P_FLUSH_REQUESTS, DATA_STREAM);
+}
+
+int drbd_send_flush_forward(struct drbd_connection *connection, u64 flush_sequence,
+ int initiator_node_id)
+{
+ struct p_flush_forward *p;
+
+ p = conn_prepare_command(connection, sizeof(*p), CONTROL_STREAM);
+ if (!p)
+ return -EIO;
+
+ p->flush_sequence = cpu_to_be64(flush_sequence);
+ p->initiator_node_id = cpu_to_be32(initiator_node_id);
+
+ return send_command(connection, -1, P_FLUSH_FORWARD, CONTROL_STREAM);
+}
+
+int drbd_send_flush_requests_ack(struct drbd_connection *connection, u64 flush_sequence,
+ int primary_node_id)
+{
+ struct p_flush_ack *p;
+
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ p->flush_sequence = cpu_to_be64(flush_sequence);
+ p->primary_node_id = cpu_to_be32(primary_node_id);
+
+ return send_command(connection, -1, P_FLUSH_REQUESTS_ACK, DATA_STREAM);
+}
+
+int drbd_send_enable_replication_next(struct drbd_peer_device *peer_device)
+{
+ struct p_enable_replication *p;
+ struct drbd_peer_device_conf *pdc;
+ bool resync_without_replication;
+
+ set_bit(PEER_REPLICATION_NEXT, peer_device->flags);
+ if (!(peer_device->connection->agreed_features & DRBD_FF_RESYNC_WITHOUT_REPLICATION))
+ return 0;
+
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ rcu_read_lock();
+ pdc = rcu_dereference(peer_device->conf);
+ resync_without_replication = pdc->resync_without_replication;
+ rcu_read_unlock();
+
+ if (resync_without_replication)
+ clear_bit(PEER_REPLICATION_NEXT, peer_device->flags);
+
+ p->enable = !resync_without_replication;
+ p->_pad1 = 0;
+ p->_pad2 = 0;
+
+ return drbd_send_command(peer_device, P_ENABLE_REPLICATION_NEXT, DATA_STREAM);
+}
+
+int drbd_send_enable_replication(struct drbd_peer_device *peer_device, bool enable)
+{
+ struct p_enable_replication *p;
+
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+
+ p->enable = enable;
+ p->_pad1 = 0;
+ p->_pad2 = 0;
+
+ return drbd_send_command(peer_device, P_ENABLE_REPLICATION, DATA_STREAM);
+}
+
static void dcbp_set_code(struct p_compressed_bm *p, enum drbd_bitmap_code code)
{
BUG_ON(code & ~0xf);
@@ -1069,24 +2066,28 @@ static void dcbp_set_pad_bits(struct p_compressed_bm *p, int n)
p->encoding = (p->encoding & (~0x7 << 4)) | (n << 4);
}
-static int fill_bitmap_rle_bits(struct drbd_device *device,
- struct p_compressed_bm *p,
- unsigned int size,
- struct bm_xfer_ctx *c)
+/* For compat reasons, encode bitmap as if it was 4k per bit!
+ * Easy: just scale the run length.
+ */
+static int fill_bitmap_rle_bits(struct drbd_peer_device *peer_device,
+ struct p_compressed_bm *p,
+ unsigned int size,
+ struct bm_xfer_ctx *c)
{
struct bitstream bs;
unsigned long plain_bits;
unsigned long tmp;
unsigned long rl;
- unsigned len;
- unsigned toggle;
+ unsigned long rl_4k;
+ unsigned int len;
+ unsigned int toggle;
int bits, use_rle;
/* may we use this feature? */
rcu_read_lock();
- use_rle = rcu_dereference(first_peer_device(device)->connection->net_conf)->use_rle;
+ use_rle = rcu_dereference(peer_device->connection->transport.net_conf)->use_rle;
rcu_read_unlock();
- if (!use_rle || first_peer_device(device)->connection->agreed_pro_version < 90)
+ if (!use_rle || peer_device->connection->agreed_pro_version < 90)
return 0;
if (c->bit_offset >= c->bm_bits)
@@ -1106,11 +2107,16 @@ static int fill_bitmap_rle_bits(struct drbd_device *device,
/* see how much plain bits we can stuff into one packet
* using RLE and VLI. */
do {
- tmp = (toggle == 0) ? _drbd_bm_find_next_zero(device, c->bit_offset)
- : _drbd_bm_find_next(device, c->bit_offset);
- if (tmp == -1UL)
+ tmp = (toggle == 0) ? _drbd_bm_find_next_zero(peer_device, c->bit_offset)
+ : _drbd_bm_find_next(peer_device, c->bit_offset);
+ if (tmp == -1UL) {
tmp = c->bm_bits;
- rl = tmp - c->bit_offset;
+ rl = tmp - c->bit_offset;
+ rl_4k = c->bm_bits_4k - (c->bit_offset << c->scale);
+ } else {
+ rl = tmp - c->bit_offset;
+ rl_4k = rl << c->scale;
+ }
if (toggle == 2) { /* first iteration */
if (rl == 0) {
@@ -1127,16 +2133,16 @@ static int fill_bitmap_rle_bits(struct drbd_device *device,
/* paranoia: catch zero runlength.
* can only happen if bitmap is modified while we scan it. */
if (rl == 0) {
- drbd_err(device, "unexpected zero runlength while encoding bitmap "
+ drbd_err(peer_device, "unexpected zero runlength while encoding bitmap "
"t:%u bo:%lu\n", toggle, c->bit_offset);
return -1;
}
- bits = vli_encode_bits(&bs, rl);
+ bits = vli_encode_bits(&bs, rl_4k);
if (bits == -ENOBUFS) /* buffer full */
break;
if (bits <= 0) {
- drbd_err(device, "error while encoding bitmap: %d\n", bits);
+ drbd_err(peer_device, "error while encoding bitmap: %d\n", bits);
return 0;
}
@@ -1147,7 +2153,7 @@ static int fill_bitmap_rle_bits(struct drbd_device *device,
len = bs.cur.b - p->code + !!bs.cur.bit;
- if (plain_bits < (len << 3)) {
+ if (plain_bits << c->scale < (len << 3)) {
/* incompressible with this method.
* we need to rewind both word and bit position. */
c->bit_offset -= plain_bits;
@@ -1166,33 +2172,69 @@ static int fill_bitmap_rle_bits(struct drbd_device *device,
return len;
}
+/* Repeat extracted bits by "peeling off" words from the end.
+ * scale != 0 implies that repeat >= 2.
+ * Feel free to optimize ...
+ */
+static void repeat_bits(unsigned long *base, unsigned long num, unsigned int scale)
+{
+ unsigned long *src, *dst;
+ unsigned int repeat = 1 << scale;
+ unsigned int n;
+ int sbit, dbit, i;
+
+ for (n = num - 1; n > 0; n--) {
+ src = &base[n];
+ for (i = 0; i < repeat; i++) {
+ dst = &base[n*repeat + i];
+ *dst = 0;
+ for (dbit = 0; dbit < BITS_PER_LONG; dbit++) {
+ sbit = (i * BITS_PER_LONG + dbit) >> scale;
+ if (test_bit(sbit, src))
+ *dst |= 1UL << dbit;
+ }
+ }
+ }
+}
+
/*
* send_bitmap_rle_or_plain
*
* Return 0 when done, 1 when another iteration is needed, and a negative error
* code upon failure.
+ *
+ * For compat reasons, send bitmap as if it was 4k per bit!
+ * Good thing that a "scaled" bitmap will always "compress".
*/
static int
send_bitmap_rle_or_plain(struct drbd_peer_device *peer_device, struct bm_xfer_ctx *c)
{
struct drbd_device *device = peer_device->device;
- struct drbd_socket *sock = &peer_device->connection->data;
unsigned int header_size = drbd_header_size(peer_device->connection);
- struct p_compressed_bm *p = sock->sbuf + header_size;
+ struct p_compressed_bm *pc;
+ char *p;
int len, err;
- len = fill_bitmap_rle_bits(device, p,
- DRBD_SOCKET_BUFFER_SIZE - header_size - sizeof(*p), c);
- if (len < 0)
+ p = alloc_send_buffer(peer_device->connection, DRBD_SOCKET_BUFFER_SIZE, DATA_STREAM);
+ if (IS_ERR(p))
+ return -EIO;
+
+ pc = (struct p_compressed_bm *)(p + header_size);
+
+ len = fill_bitmap_rle_bits(peer_device, pc,
+ DRBD_SOCKET_BUFFER_SIZE - header_size - sizeof(*pc), c);
+ if (len < 0) {
+ cancel_send_buffer(peer_device->connection, DATA_STREAM);
return -EIO;
+ }
if (len) {
- dcbp_set_code(p, RLE_VLI_Bits);
- err = __send_command(peer_device->connection, device->vnr, sock,
- P_COMPRESSED_BITMAP, sizeof(*p) + len,
- NULL, 0);
+ dcbp_set_code(pc, RLE_VLI_Bits);
+ resize_prepared_command(peer_device->connection, DATA_STREAM, sizeof(*pc) + len);
+ err = __send_command(peer_device->connection, device->vnr,
+ P_COMPRESSED_BITMAP, DATA_STREAM);
c->packets[0]++;
- c->bytes[0] += header_size + sizeof(*p) + len;
+ c->bytes[0] += header_size + sizeof(*pc) + len;
if (c->bit_offset >= c->bm_bits)
len = 0; /* DONE */
@@ -1201,16 +2243,40 @@ send_bitmap_rle_or_plain(struct drbd_peer_device *peer_device, struct bm_xfer_ct
* send a buffer full of plain text bits instead. */
unsigned int data_size;
unsigned long num_words;
- unsigned long *p = sock->sbuf + header_size;
-
+ unsigned long words_left = c->bm_words - c->word_offset;
+ unsigned long *pu = (unsigned long *)pc;
+
+ /* Only send full native bitmap words (actual granularity),
+ * scaled to what they would look like at 4k granularity.
+ * At maximum scale, which is (20 - 12), factor 256,
+ * to transfer at least one word of unscaled bitmap,
+ * we need data_size >= 256 (unsigned long) words,
+ * that is >= 2048 byte. Which we always have.
+ */
data_size = DRBD_SOCKET_BUFFER_SIZE - header_size;
- num_words = min_t(size_t, data_size / sizeof(*p),
- c->bm_words - c->word_offset);
- len = num_words * sizeof(*p);
- if (len)
- drbd_bm_get_lel(device, c->word_offset, num_words, p);
- err = __send_command(peer_device->connection, device->vnr, sock, P_BITMAP,
- len, NULL, 0);
+ data_size = ALIGN_DOWN(data_size, sizeof(*pu) * (1UL << c->scale));
+ num_words = (data_size / sizeof(*pu)) >> c->scale;
+ num_words = min_t(size_t, num_words, words_left);
+
+ len = num_words * sizeof(*pu);
+ if (len) {
+ drbd_bm_get_lel(peer_device, c->word_offset, num_words, pu);
+
+ if (c->scale) {
+ repeat_bits(pu, num_words, c->scale);
+ len <<= c->scale;
+ }
+ } else if (words_left != 0) {
+ drbd_err(peer_device,
+ "failed to scale %lu words by %u while sending bitmap\n",
+ words_left, c->scale);
+ cancel_send_buffer(peer_device->connection, DATA_STREAM);
+ return -ERANGE;
+ }
+
+ resize_prepared_command(peer_device->connection, DATA_STREAM, len);
+ err = __send_command(peer_device->connection, device->vnr, P_BITMAP, DATA_STREAM);
+
c->word_offset += num_words;
c->bit_offset = c->word_offset * BITS_PER_LONG;
@@ -1231,396 +2297,237 @@ send_bitmap_rle_or_plain(struct drbd_peer_device *peer_device, struct bm_xfer_ct
}
/* See the comment at receive_bitmap() */
-static int _drbd_send_bitmap(struct drbd_device *device,
- struct drbd_peer_device *peer_device)
+static bool _drbd_send_bitmap(struct drbd_device *device,
+ struct drbd_peer_device *peer_device)
{
struct bm_xfer_ctx c;
- int err;
-
- if (!expect(device, device->bitmap))
- return false;
+ int res;
if (get_ldev(device)) {
- if (drbd_md_test_flag(device->ldev, MDF_FULL_SYNC)) {
+ if (drbd_md_test_peer_flag(peer_device, __MDF_PEER_FULL_SYNC)) {
drbd_info(device, "Writing the whole bitmap, MDF_FullSync was set.\n");
- drbd_bm_set_all(device);
- if (drbd_bm_write(device, peer_device)) {
+ drbd_bm_set_many_bits(peer_device, 0, -1UL);
+ if (drbd_bm_write(device, NULL)) {
/* write_bm did fail! Leave full sync flag set in Meta P_DATA
* but otherwise process as per normal - need to tell other
* side that a full resync is required! */
drbd_err(device, "Failed to write bitmap to disk!\n");
} else {
- drbd_md_clear_flag(device, MDF_FULL_SYNC);
+ drbd_md_clear_peer_flag(peer_device, __MDF_PEER_FULL_SYNC);
drbd_md_sync(device);
}
}
+ c = (struct bm_xfer_ctx) {
+ .bm_bits_4k = drbd_bm_bits_4k(device),
+ .bm_bits = drbd_bm_bits(device),
+ .bm_words = drbd_bm_words(device),
+ .scale = device->bitmap->bm_block_shift - BM_BLOCK_SHIFT_4k,
+ };
+
put_ldev(device);
+ } else {
+ return false;
}
- c = (struct bm_xfer_ctx) {
- .bm_bits = drbd_bm_bits(device),
- .bm_words = drbd_bm_words(device),
- };
-
do {
- err = send_bitmap_rle_or_plain(peer_device, &c);
- } while (err > 0);
+ if (get_ldev(device)) {
+ res = send_bitmap_rle_or_plain(peer_device, &c);
+ put_ldev(device);
+ } else {
+ return false;
+ }
+ } while (res > 0);
- return err == 0;
+ return res == 0;
}
int drbd_send_bitmap(struct drbd_device *device, struct drbd_peer_device *peer_device)
{
- struct drbd_socket *sock = &peer_device->connection->data;
+ struct drbd_transport *peer_transport = &peer_device->connection->transport;
int err = -1;
- mutex_lock(&sock->mutex);
- if (sock->socket)
- err = !_drbd_send_bitmap(device, peer_device);
- mutex_unlock(&sock->mutex);
- return err;
-}
-
-void drbd_send_b_ack(struct drbd_connection *connection, u32 barrier_nr, u32 set_size)
-{
- struct drbd_socket *sock;
- struct p_barrier_ack *p;
-
- if (connection->cstate < C_WF_REPORT_PARAMS)
- return;
-
- sock = &connection->meta;
- p = conn_prepare_command(connection, sock);
- if (!p)
- return;
- p->barrier = barrier_nr;
- p->set_size = cpu_to_be32(set_size);
- conn_send_command(connection, sock, P_BARRIER_ACK, sizeof(*p), NULL, 0);
-}
-
-/**
- * _drbd_send_ack() - Sends an ack packet
- * @peer_device: DRBD peer device.
- * @cmd: Packet command code.
- * @sector: sector, needs to be in big endian byte order
- * @blksize: size in byte, needs to be in big endian byte order
- * @block_id: Id, big endian byte order
- */
-static int _drbd_send_ack(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
- u64 sector, u32 blksize, u64 block_id)
-{
- struct drbd_socket *sock;
- struct p_block_ack *p;
-
- if (peer_device->device->state.conn < C_CONNECTED)
- return -EIO;
-
- sock = &peer_device->connection->meta;
- p = drbd_prepare_command(peer_device, sock);
- if (!p)
+ if (peer_device->bitmap_index == -1) {
+ drbd_err(peer_device, "No bitmap allocated in drbd_send_bitmap()!\n");
return -EIO;
- p->sector = sector;
- p->block_id = block_id;
- p->blksize = blksize;
- p->seq_num = cpu_to_be32(atomic_inc_return(&peer_device->device->packet_seq));
- return drbd_send_command(peer_device, sock, cmd, sizeof(*p), NULL, 0);
-}
-
-/* dp->sector and dp->block_id already/still in network byte order,
- * data_size is payload size according to dp->head,
- * and may need to be corrected for digest size. */
-void drbd_send_ack_dp(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
- struct p_data *dp, int data_size)
-{
- if (peer_device->connection->peer_integrity_tfm)
- data_size -= crypto_shash_digestsize(peer_device->connection->peer_integrity_tfm);
- _drbd_send_ack(peer_device, cmd, dp->sector, cpu_to_be32(data_size),
- dp->block_id);
-}
-
-void drbd_send_ack_rp(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
- struct p_block_req *rp)
-{
- _drbd_send_ack(peer_device, cmd, rp->sector, rp->blksize, rp->block_id);
-}
+ }
-/**
- * drbd_send_ack() - Sends an ack packet
- * @peer_device: DRBD peer device
- * @cmd: packet command code
- * @peer_req: peer request
- */
-int drbd_send_ack(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
- struct drbd_peer_request *peer_req)
-{
- return _drbd_send_ack(peer_device, cmd,
- cpu_to_be64(peer_req->i.sector),
- cpu_to_be32(peer_req->i.size),
- peer_req->block_id);
-}
+ mutex_lock(&peer_device->connection->mutex[DATA_STREAM]);
+ if (peer_transport->class->ops.stream_ok(peer_transport, DATA_STREAM))
+ err = !_drbd_send_bitmap(device, peer_device);
+ mutex_unlock(&peer_device->connection->mutex[DATA_STREAM]);
-/* This function misuses the block_id field to signal if the blocks
- * are is sync or not. */
-int drbd_send_ack_ex(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
- sector_t sector, int blksize, u64 block_id)
-{
- return _drbd_send_ack(peer_device, cmd,
- cpu_to_be64(sector),
- cpu_to_be32(blksize),
- cpu_to_be64(block_id));
+ return err;
}
int drbd_send_rs_deallocated(struct drbd_peer_device *peer_device,
struct drbd_peer_request *peer_req)
{
- struct drbd_socket *sock;
- struct p_block_desc *p;
+ struct p_block_ack *p_id;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
- if (!p)
+ if (peer_device->connection->agreed_pro_version < 122) {
+ struct p_block_desc *p;
+
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+ p->sector = cpu_to_be64(peer_req->i.sector);
+ p->blksize = cpu_to_be32(peer_req->i.size);
+ p->pad = 0;
+ return drbd_send_command(peer_device, P_RS_DEALLOCATED, DATA_STREAM);
+ }
+
+ p_id = drbd_prepare_command(peer_device, sizeof(*p_id), DATA_STREAM);
+ if (!p_id)
return -EIO;
- p->sector = cpu_to_be64(peer_req->i.sector);
- p->blksize = cpu_to_be32(peer_req->i.size);
- p->pad = 0;
- return drbd_send_command(peer_device, sock, P_RS_DEALLOCATED, sizeof(*p), NULL, 0);
+ p_id->sector = cpu_to_be64(peer_req->i.sector);
+ p_id->blksize = cpu_to_be32(peer_req->i.size);
+ p_id->block_id = peer_req->block_id;
+ p_id->seq_num = 0;
+ return drbd_send_command(peer_device, P_RS_DEALLOCATED_ID, DATA_STREAM);
}
-int drbd_send_drequest(struct drbd_peer_device *peer_device, int cmd,
+int drbd_send_drequest(struct drbd_peer_device *peer_device,
sector_t sector, int size, u64 block_id)
{
- struct drbd_socket *sock;
struct p_block_req *p;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
if (!p)
return -EIO;
p->sector = cpu_to_be64(sector);
p->block_id = block_id;
p->blksize = cpu_to_be32(size);
- return drbd_send_command(peer_device, sock, cmd, sizeof(*p), NULL, 0);
-}
-
-int drbd_send_drequest_csum(struct drbd_peer_device *peer_device, sector_t sector, int size,
- void *digest, int digest_size, enum drbd_packet cmd)
-{
- struct drbd_socket *sock;
- struct p_block_req *p;
+ p->pad = 0;
+ return drbd_send_command(peer_device, P_DATA_REQUEST, DATA_STREAM);
+}
+
+static void *drbd_prepare_rs_req(struct drbd_peer_device *peer_device, enum drbd_packet cmd, int payload_size,
+ sector_t sector, int blksize, u64 block_id, unsigned int dagtag_node_id, u64 dagtag)
+{
+ void *payload;
+ struct p_block_req_common *req_common;
+
+ if (cmd == P_RS_DAGTAG_REQ || cmd == P_RS_CSUM_DAGTAG_REQ || cmd == P_RS_THIN_DAGTAG_REQ ||
+ cmd == P_OV_DAGTAG_REQ || cmd == P_OV_DAGTAG_REPLY ||
+ cmd == P_RS_DAGTAG_WAIT_REQ) {
+ struct p_rs_req *p;
+ /* Due to the slightly complicated nested struct definition,
+ * verify that the packet size is as expected. */
+ BUILD_BUG_ON(sizeof(struct p_rs_req) != 32);
+ p = drbd_prepare_command(peer_device, sizeof(*p) + payload_size, DATA_STREAM);
+ if (!p)
+ return NULL;
+ payload = p + 1;
+ req_common = &p->req_common;
+ p->dagtag_node_id = cpu_to_be32(dagtag_node_id);
+ p->dagtag = cpu_to_be64(dagtag);
+ } else {
+ struct p_block_req *p;
+ /* Due to the slightly complicated nested struct definition,
+ * verify that the packet size is as expected. */
+ BUILD_BUG_ON(sizeof(struct p_block_req) != 24);
+ p = drbd_prepare_command(peer_device, sizeof(*p) + payload_size, DATA_STREAM);
+ if (!p)
+ return NULL;
+ payload = p + 1;
+ req_common = &p->req_common;
+ p->pad = 0;
+ }
- /* FIXME: Put the digest into the preallocated socket buffer. */
+ req_common->sector = cpu_to_be64(sector);
+ req_common->block_id = block_id;
+ req_common->blksize = cpu_to_be32(blksize);
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
- if (!p)
- return -EIO;
- p->sector = cpu_to_be64(sector);
- p->block_id = ID_SYNCER /* unused */;
- p->blksize = cpu_to_be32(size);
- return drbd_send_command(peer_device, sock, cmd, sizeof(*p), digest, digest_size);
+ return payload;
}
-int drbd_send_ov_request(struct drbd_peer_device *peer_device, sector_t sector, int size)
+int drbd_send_rs_request(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
+ sector_t sector, int size, u64 block_id,
+ unsigned int dagtag_node_id, u64 dagtag)
{
- struct drbd_socket *sock;
- struct p_block_req *p;
-
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
- if (!p)
+ if (!drbd_prepare_rs_req(peer_device, cmd, 0,
+ sector, size, block_id, dagtag_node_id, dagtag))
return -EIO;
- p->sector = cpu_to_be64(sector);
- p->block_id = ID_SYNCER /* unused */;
- p->blksize = cpu_to_be32(size);
- return drbd_send_command(peer_device, sock, P_OV_REQUEST, sizeof(*p), NULL, 0);
+ return drbd_send_command(peer_device, cmd, DATA_STREAM);
}
-/* called on sndtimeo
- * returns false if we should retry,
- * true if we think connection is dead
- */
-static int we_should_drop_the_connection(struct drbd_connection *connection, struct socket *sock)
+void *drbd_prepare_drequest_csum(struct drbd_peer_request *peer_req, enum drbd_packet cmd,
+ int digest_size, unsigned int dagtag_node_id, u64 dagtag)
{
- int drop_it;
- /* long elapsed = (long)(jiffies - device->last_received); */
-
- drop_it = connection->meta.socket == sock
- || !connection->ack_receiver.task
- || get_t_state(&connection->ack_receiver) != RUNNING
- || connection->cstate < C_WF_REPORT_PARAMS;
-
- if (drop_it)
- return true;
-
- drop_it = !--connection->ko_count;
- if (!drop_it) {
- drbd_err(connection, "[%s/%d] sock_sendmsg time expired, ko = %u\n",
- current->comm, current->pid, connection->ko_count);
- request_ping(connection);
- }
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
- return drop_it; /* && (device->state == R_PRIMARY) */;
+ return drbd_prepare_rs_req(peer_device, cmd, digest_size,
+ peer_req->i.sector, peer_req->i.size, peer_req->block_id,
+ dagtag_node_id, dagtag);
}
-static void drbd_update_congested(struct drbd_connection *connection)
-{
- struct sock *sk = connection->data.socket->sk;
- if (sk->sk_wmem_queued > sk->sk_sndbuf * 4 / 5)
- set_bit(NET_CONGESTED, &connection->flags);
-}
-/* The idea of sendpage seems to be to put some kind of reference
- * to the page into the skb, and to hand it over to the NIC. In
- * this process get_page() gets called.
- *
- * As soon as the page was really sent over the network put_page()
- * gets called by some part of the network layer. [ NIC driver? ]
- *
- * [ get_page() / put_page() increment/decrement the count. If count
- * reaches 0 the page will be freed. ]
- *
- * This works nicely with pages from FSs.
- * But this means that in protocol A we might signal IO completion too early!
- *
- * In order not to corrupt data during a resync we must make sure
- * that we do not reuse our own buffer pages (EEs) to early, therefore
- * we have the net_ee list.
- *
- * XFS seems to have problems, still, it submits pages with page_count == 0!
- * As a workaround, we disable sendpage on pages
- * with page_count == 0 or PageSlab.
- */
-static int _drbd_no_send_page(struct drbd_peer_device *peer_device, struct page *page,
- int offset, size_t size, unsigned msg_flags)
+static int __send_bio(struct drbd_peer_device *peer_device, struct bio *bio, unsigned int msg_flags)
{
- struct socket *socket;
- void *addr;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
int err;
- socket = peer_device->connection->data.socket;
- addr = kmap(page) + offset;
- err = drbd_send_all(peer_device->connection, socket, addr, size, msg_flags);
- kunmap(page);
- if (!err)
- peer_device->device->send_cnt += size >> 9;
+ err = flush_send_buffer(connection, DATA_STREAM);
+ if (!err) {
+ err = tr_ops->send_bio(transport, bio, msg_flags);
+ if (err)
+ connection->send_buffer[DATA_STREAM].dead = true;
+ else
+ peer_device->send_cnt += bio->bi_iter.bi_size >> 9;
+ }
+
return err;
}
-static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *page,
- int offset, size_t size, unsigned msg_flags)
+/* sendmsg(MSG_SPLICE_PAGES) (former (sendpage()) increases the page ref_count
+ * and hands it to the network stack. After the NIC DMA sends the data, it
+ * decreases that page's ref_count.
+ * We may not do this for protocol A, where we could complete a write operation
+ * before the network stack sends the data.
+ */
+static int
+drbd_send_bio(struct drbd_peer_device *peer_device, struct bio *bio, unsigned int msg_flags)
{
- struct socket *socket = peer_device->connection->data.socket;
- struct msghdr msg = { .msg_flags = msg_flags, };
- struct bio_vec bvec;
- int len = size;
- int err = -EIO;
+ if (drbd_disable_sendpage)
+ msg_flags &= ~MSG_SPLICE_PAGES;
- /* e.g. XFS meta- & log-data is in slab pages, which have a
- * page_count of 0 and/or have PageSlab() set.
- * we cannot use send_page for those, as that does get_page();
- * put_page(); and would cause either a VM_BUG directly, or
- * __page_cache_release a page that would actually still be referenced
- * by someone, leading to some obscure delayed Oops somewhere else. */
- if (!drbd_disable_sendpage && sendpages_ok(page, len, offset))
- msg.msg_flags |= MSG_NOSIGNAL | MSG_SPLICE_PAGES;
+ /* e.g. XFS meta- & log-data is in slab pages have !sendpage_ok(page) */
+ if (msg_flags & MSG_SPLICE_PAGES) {
+ struct bvec_iter iter;
+ struct bio_vec bvec;
- drbd_update_congested(peer_device->connection);
- do {
- int sent;
-
- bvec_set_page(&bvec, page, len, offset);
- iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len);
+ bio_for_each_segment(bvec, bio, iter) {
+ struct page *page = bvec.bv_page;
- sent = sock_sendmsg(socket, &msg);
- if (sent <= 0) {
- if (sent == -EAGAIN) {
- if (we_should_drop_the_connection(peer_device->connection, socket))
- break;
- continue;
+ if (!sendpage_ok(page)) {
+ msg_flags &= ~MSG_SPLICE_PAGES;
+ break;
}
- drbd_warn(peer_device->device, "%s: size=%d len=%d sent=%d\n",
- __func__, (int)size, len, sent);
- if (sent < 0)
- err = sent;
- break;
}
- len -= sent;
- offset += sent;
- } while (len > 0 /* THINK && device->cstate >= C_CONNECTED*/);
- clear_bit(NET_CONGESTED, &peer_device->connection->flags);
-
- if (len == 0) {
- err = 0;
- peer_device->device->send_cnt += size >> 9;
}
- return err;
-}
-
-static int _drbd_send_bio(struct drbd_peer_device *peer_device, struct bio *bio)
-{
- struct bio_vec bvec;
- struct bvec_iter iter;
- /* hint all but last page with MSG_MORE */
- bio_for_each_segment(bvec, bio, iter) {
- int err;
-
- err = _drbd_no_send_page(peer_device, bvec.bv_page,
- bvec.bv_offset, bvec.bv_len,
- bio_iter_last(bvec, iter)
- ? 0 : MSG_MORE);
- if (err)
- return err;
- }
- return 0;
+ return __send_bio(peer_device, bio, msg_flags);
}
-static int _drbd_send_zc_bio(struct drbd_peer_device *peer_device, struct bio *bio)
+static int drbd_send_ee(struct drbd_peer_device *peer_device, struct drbd_peer_request *peer_req)
{
- struct bio_vec bvec;
- struct bvec_iter iter;
+ struct bio *bio;
+ int err = 0;
- /* hint all but last page with MSG_MORE */
- bio_for_each_segment(bvec, bio, iter) {
- int err;
-
- err = _drbd_send_page(peer_device, bvec.bv_page,
- bvec.bv_offset, bvec.bv_len,
- bio_iter_last(bvec, iter) ? 0 : MSG_MORE);
+ bio_list_for_each(bio, &peer_req->bios) {
+ err = __send_bio(peer_device, bio,
+ peer_req->flags & EE_RELEASE_TO_MEMPOOL ? 0 : MSG_SPLICE_PAGES);
if (err)
- return err;
+ break;
}
- return 0;
-}
-static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device,
- struct drbd_peer_request *peer_req)
-{
- bool use_sendpage = !(peer_req->flags & EE_RELEASE_TO_MEMPOOL);
- struct page *page = peer_req->pages;
- unsigned len = peer_req->i.size;
- int err;
-
- /* hint all but last page with MSG_MORE */
- page_chain_for_each(page) {
- unsigned l = min_t(unsigned, len, PAGE_SIZE);
-
- if (likely(use_sendpage))
- err = _drbd_send_page(peer_device, page, 0, l,
- page_chain_next(page) ? MSG_MORE : 0);
- else
- err = _drbd_no_send_page(peer_device, page, 0, l,
- page_chain_next(page) ? MSG_MORE : 0);
-
- if (err)
- return err;
- len -= l;
- }
- return 0;
+ return err;
}
-static u32 bio_flags_to_wire(struct drbd_connection *connection,
- struct bio *bio)
+/* see also wire_flags_to_bio() */
+static u32 bio_flags_to_wire(struct drbd_connection *connection, struct bio *bio)
{
if (connection->agreed_pro_version >= 95)
return (bio->bi_opf & REQ_SYNC ? DP_RW_SYNC : 0) |
@@ -1628,12 +2535,13 @@ static u32 bio_flags_to_wire(struct drbd_connection *connection,
(bio->bi_opf & REQ_PREFLUSH ? DP_FLUSH : 0) |
(bio_op(bio) == REQ_OP_DISCARD ? DP_DISCARD : 0) |
(bio_op(bio) == REQ_OP_WRITE_ZEROES ?
- ((connection->agreed_features & DRBD_FF_WZEROES) ?
- (DP_ZEROES |(!(bio->bi_opf & REQ_NOUNMAP) ? DP_DISCARD : 0))
- : DP_DISCARD)
- : 0);
- else
- return bio->bi_opf & REQ_SYNC ? DP_RW_SYNC : 0;
+ ((connection->agreed_features & DRBD_FF_WZEROES) ?
+ (DP_ZEROES | (!(bio->bi_opf & REQ_NOUNMAP) ? DP_DISCARD : 0))
+ : DP_DISCARD)
+ : 0);
+
+ /* else: we used to communicate one bit only in older DRBD */
+ return bio->bi_opf & REQ_SYNC ? DP_RW_SYNC : 0;
}
/* Used to send write or TRIM aka REQ_OP_DISCARD requests
@@ -1642,53 +2550,62 @@ static u32 bio_flags_to_wire(struct drbd_connection *connection,
int drbd_send_dblock(struct drbd_peer_device *peer_device, struct drbd_request *req)
{
struct drbd_device *device = peer_device->device;
- struct drbd_socket *sock;
+ struct drbd_connection *connection = peer_device->connection;
+ char *const before = connection->scratch_buffer.d.before;
+ char *const after = connection->scratch_buffer.d.after;
+ struct p_trim *trim = NULL;
struct p_data *p;
- void *digest_out;
+ void *digest_out = NULL;
unsigned int dp_flags = 0;
- int digest_size;
+ int digest_size = 0;
int err;
+ const unsigned int s = req->net_rq_state[peer_device->node_id];
+ const enum req_op op = bio_op(req->master_bio);
+
+ if (op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES) {
+ trim = drbd_prepare_command(peer_device, sizeof(*trim), DATA_STREAM);
+ if (!trim)
+ return -EIO;
+ p = &trim->p_data;
+ trim->size = cpu_to_be32(req->i.size);
+ } else {
+ if (connection->integrity_tfm)
+ digest_size = crypto_shash_digestsize(connection->integrity_tfm);
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
- digest_size = peer_device->connection->integrity_tfm ?
- crypto_shash_digestsize(peer_device->connection->integrity_tfm) : 0;
+ p = drbd_prepare_command(peer_device, sizeof(*p) + digest_size, DATA_STREAM);
+ if (!p)
+ return -EIO;
+ digest_out = p + 1;
+ }
- if (!p)
- return -EIO;
p->sector = cpu_to_be64(req->i.sector);
p->block_id = (unsigned long)req;
- p->seq_num = cpu_to_be32(atomic_inc_return(&device->packet_seq));
- dp_flags = bio_flags_to_wire(peer_device->connection, req->master_bio);
- if (device->state.conn >= C_SYNC_SOURCE &&
- device->state.conn <= C_PAUSED_SYNC_T)
+ p->seq_num = cpu_to_be32(atomic_inc_return(&peer_device->packet_seq));
+ dp_flags = bio_flags_to_wire(connection, req->master_bio);
+ if (peer_device->repl_state[NOW] >= L_SYNC_SOURCE && peer_device->repl_state[NOW] <= L_PAUSED_SYNC_T)
dp_flags |= DP_MAY_SET_IN_SYNC;
- if (peer_device->connection->agreed_pro_version >= 100) {
- if (req->rq_state & RQ_EXP_RECEIVE_ACK)
+ if (connection->agreed_pro_version >= 100) {
+ if (s & RQ_EXP_RECEIVE_ACK)
dp_flags |= DP_SEND_RECEIVE_ACK;
- /* During resync, request an explicit write ack,
- * even in protocol != C */
- if (req->rq_state & RQ_EXP_WRITE_ACK
- || (dp_flags & DP_MAY_SET_IN_SYNC))
+ if (s & RQ_EXP_WRITE_ACK || dp_flags & DP_MAY_SET_IN_SYNC)
dp_flags |= DP_SEND_WRITE_ACK;
}
p->dp_flags = cpu_to_be32(dp_flags);
- if (dp_flags & (DP_DISCARD|DP_ZEROES)) {
- enum drbd_packet cmd = (dp_flags & DP_ZEROES) ? P_ZEROES : P_TRIM;
- struct p_trim *t = (struct p_trim*)p;
- t->size = cpu_to_be32(req->i.size);
- err = __send_command(peer_device->connection, device->vnr, sock, cmd, sizeof(*t), NULL, 0);
+ if (trim) {
+ err = __send_command(connection, device->vnr,
+ (dp_flags & DP_ZEROES) ? P_ZEROES : P_TRIM, DATA_STREAM);
goto out;
}
- digest_out = p + 1;
- /* our digest is still only over the payload.
- * TRIM does not carry any payload. */
- if (digest_size)
- drbd_csum_bio(peer_device->connection->integrity_tfm, req->master_bio, digest_out);
- err = __send_command(peer_device->connection, device->vnr, sock, P_DATA,
- sizeof(*p) + digest_size, NULL, req->i.size);
+ if (digest_size && digest_out) {
+ WARN_ON(digest_size > sizeof(connection->scratch_buffer.d.before));
+ drbd_csum_bio(connection->integrity_tfm, req->master_bio, before);
+ memcpy(digest_out, before, digest_size);
+ }
+
+ additional_size_command(connection, DATA_STREAM, req->i.size);
+ err = __send_command(connection, device->vnr, P_DATA, DATA_STREAM);
if (!err) {
/* For protocol A, we have to memcpy the payload into
* socket buffers, as we may complete right away
@@ -1701,50 +2618,43 @@ int drbd_send_dblock(struct drbd_peer_device *peer_device, struct drbd_request *
* out ok after sending on this side, but does not fit on the
* receiving side, we sure have detected corruption elsewhere.
*/
- if (!(req->rq_state & (RQ_EXP_RECEIVE_ACK | RQ_EXP_WRITE_ACK)) || digest_size)
- err = _drbd_send_bio(peer_device, req->master_bio);
- else
- err = _drbd_send_zc_bio(peer_device, req->master_bio);
+ bool proto_b_or_c = (s & (RQ_EXP_RECEIVE_ACK | RQ_EXP_WRITE_ACK));
+ int msg_flags = proto_b_or_c && !digest_size ? MSG_SPLICE_PAGES : 0;
+
+ err = drbd_send_bio(peer_device, req->master_bio, msg_flags);
/* double check digest, sometimes buffers have been modified in flight. */
- if (digest_size > 0 && digest_size <= 64) {
- /* 64 byte, 512 bit, is the largest digest size
- * currently supported in kernel crypto. */
- unsigned char digest[64];
- drbd_csum_bio(peer_device->connection->integrity_tfm, req->master_bio, digest);
- if (memcmp(p + 1, digest, digest_size)) {
+ if (digest_size > 0) {
+ drbd_csum_bio(connection->integrity_tfm, req->master_bio, after);
+ if (memcmp(before, after, digest_size)) {
drbd_warn(device,
"Digest mismatch, buffer modified by upper layers during write: %llus +%u\n",
(unsigned long long)req->i.sector, req->i.size);
}
- } /* else if (digest_size > 64) {
- ... Be noisy about digest too large ...
- } */
+ }
}
out:
- mutex_unlock(&sock->mutex); /* locked by drbd_prepare_command() */
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
return err;
}
/* answer packet, used to send data back for read requests:
* Peer -> (diskless) R_PRIMARY (P_DATA_REPLY)
- * C_SYNC_SOURCE -> C_SYNC_TARGET (P_RS_DATA_REPLY)
+ * L_SYNC_SOURCE -> L_SYNC_TARGET (P_RS_DATA_REPLY)
*/
int drbd_send_block(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
struct drbd_peer_request *peer_req)
{
- struct drbd_device *device = peer_device->device;
- struct drbd_socket *sock;
+ struct drbd_connection *connection = peer_device->connection;
struct p_data *p;
int err;
int digest_size;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
+ digest_size = connection->integrity_tfm ?
+ crypto_shash_digestsize(connection->integrity_tfm) : 0;
- digest_size = peer_device->connection->integrity_tfm ?
- crypto_shash_digestsize(peer_device->connection->integrity_tfm) : 0;
+ p = drbd_prepare_command(peer_device, sizeof(*p) + digest_size, DATA_STREAM);
if (!p)
return -EIO;
@@ -1752,300 +2662,763 @@ int drbd_send_block(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
p->block_id = peer_req->block_id;
p->seq_num = 0; /* unused */
p->dp_flags = 0;
+
+ /* Older peers expect block_id for P_RS_DATA_REPLY to be ID_SYNCER. */
+ if (connection->agreed_pro_version < 122 && cmd == P_RS_DATA_REPLY)
+ p->block_id = ID_SYNCER;
+
if (digest_size)
- drbd_csum_ee(peer_device->connection->integrity_tfm, peer_req, p + 1);
- err = __send_command(peer_device->connection, device->vnr, sock, cmd, sizeof(*p) + digest_size, NULL, peer_req->i.size);
+ drbd_csum_bios(connection->integrity_tfm, &peer_req->bios, p + 1);
+ additional_size_command(connection, DATA_STREAM, peer_req->i.size);
+ err = __send_command(connection,
+ peer_device->device->vnr, cmd, DATA_STREAM);
if (!err)
- err = _drbd_send_zc_ee(peer_device, peer_req);
- mutex_unlock(&sock->mutex); /* locked by drbd_prepare_command() */
+ err = drbd_send_ee(peer_device, peer_req);
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
return err;
}
-int drbd_send_out_of_sync(struct drbd_peer_device *peer_device, struct drbd_request *req)
+int drbd_send_out_of_sync(struct drbd_peer_device *peer_device, sector_t sector, unsigned int size)
{
- struct drbd_socket *sock;
struct p_block_desc *p;
- sock = &peer_device->connection->data;
- p = drbd_prepare_command(peer_device, sock);
+ p = drbd_prepare_command(peer_device, sizeof(*p), DATA_STREAM);
if (!p)
return -EIO;
- p->sector = cpu_to_be64(req->i.sector);
- p->blksize = cpu_to_be32(req->i.size);
- return drbd_send_command(peer_device, sock, P_OUT_OF_SYNC, sizeof(*p), NULL, 0);
+ p->sector = cpu_to_be64(sector);
+ p->blksize = cpu_to_be32(size);
+ return drbd_send_command(peer_device, P_OUT_OF_SYNC, DATA_STREAM);
}
-/*
- drbd_send distinguishes two cases:
+int drbd_send_dagtag(struct drbd_connection *connection, u64 dagtag)
+{
+ struct p_dagtag *p;
- Packets sent via the data socket "sock"
- and packets sent via the meta data socket "msock"
+ if (connection->agreed_pro_version < 110)
+ return 0;
- sock msock
- -----------------+-------------------------+------------------------------
- timeout conf.timeout / 2 conf.timeout / 2
- timeout action send a ping via msock Abort communication
- and close all sockets
-*/
+ p = conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+ p->dagtag = cpu_to_be64(dagtag);
+ return send_command(connection, -1, P_DAGTAG, DATA_STREAM);
+}
-/*
- * you must have down()ed the appropriate [m]sock_mutex elsewhere!
- */
-int drbd_send(struct drbd_connection *connection, struct socket *sock,
- void *buf, size_t size, unsigned msg_flags)
+/* primary_peer_present_and_not_two_primaries_allowed() */
+static bool primary_peer_present(struct drbd_resource *resource)
{
- struct kvec iov = {.iov_base = buf, .iov_len = size};
- struct msghdr msg = {.msg_flags = msg_flags | MSG_NOSIGNAL};
- int rv, sent = 0;
-
- if (!sock)
- return -EBADR;
-
- /* THINK if (signal_pending) return ... ? */
+ struct drbd_connection *connection;
+ struct drbd_net_conf *nc;
+ bool two_primaries, rv = false;
- iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &iov, 1, size);
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ nc = rcu_dereference(connection->transport.net_conf);
+ two_primaries = nc ? nc->two_primaries : false;
- if (sock == connection->data.socket) {
- rcu_read_lock();
- connection->ko_count = rcu_dereference(connection->net_conf)->ko_count;
- rcu_read_unlock();
- drbd_update_congested(connection);
- }
- do {
- rv = sock_sendmsg(sock, &msg);
- if (rv == -EAGAIN) {
- if (we_should_drop_the_connection(connection, sock))
- break;
- else
- continue;
- }
- if (rv == -EINTR) {
- flush_signals(current);
- rv = 0;
- }
- if (rv < 0)
+ if (connection->peer_role[NOW] == R_PRIMARY && !two_primaries) {
+ rv = true;
break;
- sent += rv;
- } while (sent < size);
-
- if (sock == connection->data.socket)
- clear_bit(NET_CONGESTED, &connection->flags);
-
- if (rv <= 0) {
- if (rv != -EAGAIN) {
- drbd_err(connection, "%s_sendmsg returned %d\n",
- sock == connection->meta.socket ? "msock" : "sock",
- rv);
- conn_request_state(connection, NS(conn, C_BROKEN_PIPE), CS_HARD);
- } else
- conn_request_state(connection, NS(conn, C_TIMEOUT), CS_HARD);
+ }
}
+ rcu_read_unlock();
- return sent;
+ return rv;
}
-/*
- * drbd_send_all - Send an entire buffer
- *
- * Returns 0 upon success and a negative error value otherwise.
+static bool any_disk_is_uptodate(struct drbd_device *device)
+{
+ bool ret = false;
+
+ rcu_read_lock();
+ if (device->disk_state[NOW] == D_UP_TO_DATE)
+ ret = true;
+ else {
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->disk_state[NOW] == D_UP_TO_DATE) {
+ ret = true;
+ break;
+ }
+ }
+ }
+ rcu_read_unlock();
+
+ return ret;
+}
+
+/* If we are trying to (re-)establish some connection,
+ * it may be useful to re-try the conditions in drbd_open().
+ * But if we have no connection at all (yet/anymore),
+ * or are disconnected and not trying to (re-)establish,
+ * or are established already, retrying won't help at all.
+ * Asking the same peer(s) the same question
+ * is unlikely to change their answer.
+ * Almost always triggered by udev (and the configured probes) while bringing
+ * the resource "up", just after "new-minor", even before "attach" or any
+ * "peers"/"paths" are configured.
*/
-int drbd_send_all(struct drbd_connection *connection, struct socket *sock, void *buffer,
- size_t size, unsigned msg_flags)
+static bool connection_state_may_improve_soon(struct drbd_resource *resource)
{
- int err;
+ struct drbd_connection *connection;
+ bool ret = false;
- err = drbd_send(connection, sock, buffer, size, msg_flags);
- if (err < 0)
- return err;
- if (err != size)
- return -EIO;
- return 0;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ enum drbd_conn_state cstate = connection->cstate[NOW];
+
+ if (C_DISCONNECTING < cstate && cstate < C_CONNECTED) {
+ ret = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+ return ret;
}
-static int drbd_open(struct gendisk *disk, blk_mode_t mode)
+/* TASK_COMM_LEN reserves one '\0', sizeof("") both include '\0',
+ * that's room enough for ':' and ' ' separators and the EOS.
+ */
+union comm_pid_tag_buf {
+ char comm[TASK_COMM_LEN];
+ char buf[TASK_COMM_LEN + sizeof("2147483647") + sizeof("auto-promote")];
+};
+
+static void snprintf_current_comm_pid_tag(union comm_pid_tag_buf *s, const char *tag)
{
- struct drbd_device *device = disk->private_data;
- unsigned long flags;
- int rv = 0;
+ int len;
+
+ get_task_comm(s->comm, current);
+ len = strlen(s->buf);
+ snprintf(s->buf + len, sizeof(s->buf)-len, ":%d %s", task_pid_nr(current), tag);
+}
+
+static int try_to_promote(struct drbd_device *device, long timeout, bool ndelay)
+{
+ struct drbd_resource *resource = device->resource;
+ int rv;
+
+ do {
+ union comm_pid_tag_buf tag;
+ unsigned long start = jiffies;
+ long t;
+
+ snprintf_current_comm_pid_tag(&tag, "auto-promote");
+ rv = drbd_set_role(resource, R_PRIMARY, 0, tag.buf, NULL);
+ timeout -= jiffies - start;
+
+ if (ndelay || rv >= SS_SUCCESS || timeout <= 0) {
+ break;
+ } else if (rv == SS_CW_FAILED_BY_PEER) {
+ /* Probably udev has it open read-only on one of the peers;
+ since commit cbcbb50a65 from 2017 it waits on the peer;
+ retry only if the timeout permits */
+ if (jiffies - start < HZ / 10) {
+ t = schedule_timeout_interruptible(HZ / 10);
+ if (t)
+ break;
+ timeout -= HZ / 10;
+ }
+ } else if (rv == SS_TWO_PRIMARIES) {
+ /* Wait till the peer demoted itself */
+ t = wait_event_interruptible_timeout(resource->state_wait,
+ resource->role[NOW] == R_PRIMARY ||
+ (!primary_peer_present(resource) && any_disk_is_uptodate(device)),
+ timeout);
+ if (t <= 0)
+ break;
+ timeout = t;
+ } else if (rv == SS_NO_UP_TO_DATE_DISK && connection_state_may_improve_soon(resource)) {
+ /* Wait until we get a connection established */
+ t = wait_event_interruptible_timeout(resource->state_wait,
+ any_disk_is_uptodate(device), timeout);
+ if (t <= 0)
+ break;
+ timeout = t;
+ } else {
+ break;
+ }
+ } while (timeout > 0);
+ return rv;
+}
+
+static int ro_open_cond(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
+
+ if (!device->have_quorum[NOW])
+ return -ENODATA;
+ else if (resource->role[NOW] != R_PRIMARY &&
+ primary_peer_present(resource) && !drbd_allow_oos)
+ return -EMEDIUMTYPE;
+ else if (any_disk_is_uptodate(device))
+ return 0;
+ else if (connection_state_may_improve_soon(resource))
+ return -EAGAIN;
+ else
+ return -ENODATA;
+}
- mutex_lock(&drbd_main_mutex);
- spin_lock_irqsave(&device->resource->req_lock, flags);
- /* to have a stable device->state.role
- * and no race with updating open_cnt */
+enum ioc_rv {
+ IOC_SLEEP = 0,
+ IOC_OK = 1,
+ IOC_ABORT = 2,
+};
- if (device->state.role != R_PRIMARY) {
+/* If we are in the middle of a cluster wide state change, we don't want
+ * to change (open_cnt == 0), as that then could cause a failure to commit
+ * some already promised peer auto-promote locally.
+ * So we wait until the pending remote_state_change is finalized,
+ * or give up when the timeout is reached.
+ *
+ * But we don't want to fail an open on a Primary just because it happens
+ * during some unrelated remote state change.
+ * If we are already Primary, or already have an open count != 0,
+ * we don't need to wait, it won't change anything.
+ */
+static enum ioc_rv inc_open_count(struct drbd_device *device, blk_mode_t mode)
+{
+ struct drbd_resource *resource = device->resource;
+ enum ioc_rv r;
+
+ if (test_bit(DOWN_IN_PROGRESS, &resource->flags))
+ return IOC_ABORT;
+
+ read_lock_irq(&resource->state_rwlock);
+ if (test_bit(UNREGISTERED, &device->flags))
+ r = IOC_ABORT;
+ else if (resource->remote_state_change &&
+ resource->role[NOW] != R_PRIMARY &&
+ (device->open_cnt == 0 || mode & BLK_OPEN_WRITE)) {
+ if (mode & BLK_OPEN_NDELAY)
+ r = IOC_ABORT;
+ else
+ r = IOC_SLEEP;
+ } else {
+ r = IOC_OK;
+ device->open_cnt++;
if (mode & BLK_OPEN_WRITE)
- rv = -EROFS;
- else if (!drbd_allow_oos)
- rv = -EMEDIUMTYPE;
+ device->writable = true;
}
+ read_unlock_irq(&resource->state_rwlock);
- if (!rv)
- device->open_cnt++;
- spin_unlock_irqrestore(&device->resource->req_lock, flags);
- mutex_unlock(&drbd_main_mutex);
+ return r;
+}
- return rv;
+static void __prune_or_free_openers(struct drbd_device *device, pid_t pid)
+{
+ struct opener *pos, *tmp;
+
+ list_for_each_entry_safe(pos, tmp, &device->openers, list) {
+ // if pid == 0, i.e., counts were 0, delete all entries, else the matching one
+ if (pid == 0 || pid == pos->pid) {
+ dynamic_drbd_dbg(device, "%sopeners del: %s(%d)\n", pid == 0 ? "" : "all ",
+ pos->comm, pos->pid);
+ list_del(&pos->list);
+ kfree(pos);
+
+ /* in case we remove a real process, stop here, there might be multiple openers with the same pid */
+ /* this assumes that the oldest opener with the same pid releases first. "as good as it gets" */
+ if (pid != 0)
+ break;
+ }
+ }
}
-static void drbd_release(struct gendisk *gd)
+static void free_openers(struct drbd_device *device)
+{
+ __prune_or_free_openers(device, 0);
+}
+
+static void prune_or_free_openers(struct drbd_device *device, pid_t pid)
+{
+ spin_lock(&device->openers_lock);
+ __prune_or_free_openers(device, pid);
+ spin_unlock(&device->openers_lock);
+}
+
+static void add_opener(struct drbd_device *device, bool did_auto_promote)
+{
+ struct opener *opener, *tmp;
+ ktime_t now = ktime_get_real();
+ int len = 0;
+
+ if (did_auto_promote) {
+ struct drbd_resource *resource = device->resource;
+
+ resource->auto_promoted_by.minor = device->minor;
+ resource->auto_promoted_by.pid = task_pid_nr(current);
+ resource->auto_promoted_by.opened = now;
+ get_task_comm(resource->auto_promoted_by.comm, current);
+ }
+ opener = kmalloc_obj(*opener, GFP_NOIO);
+ if (!opener)
+ return;
+ get_task_comm(opener->comm, current);
+ opener->pid = task_pid_nr(current);
+ opener->opened = now;
+
+ spin_lock(&device->openers_lock);
+ list_for_each_entry(tmp, &device->openers, list)
+ if (++len > 100) { /* 100 ought to be enough for everybody */
+ dynamic_drbd_dbg(device, "openers: list full, do not add new opener\n");
+ kfree(opener);
+ goto out;
+ }
+
+ list_add(&opener->list, &device->openers);
+ dynamic_drbd_dbg(device, "openers add: %s(%d)\n", opener->comm, opener->pid);
+out:
+ spin_unlock(&device->openers_lock);
+}
+
+static int drbd_open(struct gendisk *gd, blk_mode_t mode)
{
struct drbd_device *device = gd->private_data;
+ struct drbd_resource *resource = device->resource;
+ long timeout = resource->res_opts.auto_promote_timeout * HZ / 10;
+ enum drbd_state_rv rv = SS_UNKNOWN_ERROR;
+ bool was_writable;
+ enum ioc_rv r;
+ int err = 0;
+
+ /* Fail read-only open from systemd-udev (version <= 238) */
+ if (!(mode & BLK_OPEN_WRITE) && !drbd_allow_oos) {
+ char comm[TASK_COMM_LEN];
+
+ get_task_comm(comm, current);
+ if (!strcmp("systemd-udevd", comm))
+ return -EACCES;
+ }
- mutex_lock(&drbd_main_mutex);
- device->open_cnt--;
- mutex_unlock(&drbd_main_mutex);
+ /* Fail read-write open early,
+ * in case someone explicitly set us read-only (blockdev --setro) */
+ if (bdev_read_only(gd->part0) && (mode & BLK_OPEN_WRITE))
+ return -EACCES;
+
+ if (resource->fail_io[NOW])
+ return -ENOTRECOVERABLE;
+
+ kref_get(&device->kref);
+
+ mutex_lock(&resource->open_release);
+ was_writable = device->writable;
+
+ timeout = wait_event_interruptible_timeout(resource->twopc_wait,
+ (r = inc_open_count(device, mode)),
+ timeout);
+
+ if (r == IOC_ABORT || (r == IOC_SLEEP && timeout <= 0)) {
+ mutex_unlock(&resource->open_release);
+ kref_put(&device->kref, drbd_destroy_device);
+ return -EAGAIN;
+ }
+
+ if (resource->res_opts.auto_promote) {
+ /* Allow opening in read-only mode on an unconnected secondary.
+ This avoids split brain when the drbd volume gets opened
+ temporarily by udev while it scans for PV signatures. */
+
+ if (mode & BLK_OPEN_WRITE) {
+ if (resource->role[NOW] == R_SECONDARY) {
+ rv = try_to_promote(device, timeout, (mode & BLK_OPEN_NDELAY));
+ if (rv < SS_SUCCESS)
+ drbd_info(resource, "Auto-promote failed: %s (%d)\n",
+ drbd_set_st_err_str(rv), rv);
+ }
+ } else if ((mode & BLK_OPEN_NDELAY) == 0) {
+ /* Double check peers
+ *
+ * Some services may try to first open ro, and only if that
+ * works open rw. An attempt to failover immediately after
+ * primary crash, before DRBD has noticed that the primary peer
+ * is gone, would result in open failure, thus failure to take
+ * over services. */
+ err = ro_open_cond(device);
+ if (err == -EMEDIUMTYPE) {
+ drbd_check_peers(resource);
+ err = -EAGAIN;
+ }
+ if (err == -EAGAIN) {
+ wait_event_interruptible_timeout(resource->state_wait,
+ ro_open_cond(device) != -EAGAIN,
+ resource->res_opts.auto_promote_timeout * HZ / 10);
+ }
+ }
+ } else if (resource->role[NOW] != R_PRIMARY &&
+ !(mode & BLK_OPEN_WRITE) && !drbd_allow_oos) {
+ err = -EMEDIUMTYPE;
+ goto out;
+ }
+
+ if (test_bit(UNREGISTERED, &device->flags)) {
+ err = -ENODEV;
+ } else if (mode & BLK_OPEN_WRITE) {
+ if (resource->role[NOW] != R_PRIMARY)
+ err = rv == SS_INTERRUPTED ? -ERESTARTSYS : -EROFS;
+ } else /* READ access only */ {
+ err = ro_open_cond(device);
+ }
+out:
+ /* still keep mutex, but release ASAP */
+ if (!err) {
+ add_opener(device, rv >= SS_SUCCESS);
+ /* Only interested in first open and last close. */
+ if (device->open_cnt == 1) {
+ struct drbd_device_info info;
+
+ device_to_info(&info, device);
+ mutex_lock(¬ification_mutex);
+ notify_device_state(NULL, 0, device, &info, NOTIFY_CHANGE);
+ mutex_unlock(¬ification_mutex);
+ }
+ } else
+ device->writable = was_writable;
+
+ mutex_unlock(&resource->open_release);
+ if (err) {
+ drbd_release(gd);
+ if (err == -EAGAIN && !(mode & BLK_OPEN_NDELAY))
+ err = -EMEDIUMTYPE;
+ }
+
+ return err;
}
-/* need to hold resource->req_lock */
-void drbd_queue_unplug(struct drbd_device *device)
+void drbd_open_counts(struct drbd_resource *resource, int *rw_count_ptr, int *ro_count_ptr)
+{
+ struct drbd_device *device;
+ int vnr, rw_count = 0, ro_count = 0;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (device->writable)
+ rw_count += device->open_cnt;
+ else
+ ro_count += device->open_cnt;
+ }
+ rcu_read_unlock();
+ *rw_count_ptr = rw_count;
+ *ro_count_ptr = ro_count;
+}
+
+static void wait_for_peer_disk_updates(struct drbd_resource *resource)
{
- if (device->state.pdsk >= D_INCONSISTENT && device->state.conn >= C_CONNECTED) {
- D_ASSERT(device, device->state.role == R_PRIMARY);
- if (test_and_clear_bit(UNPLUG_REMOTE, &device->flags)) {
- drbd_queue_work_if_unqueued(
- &first_peer_device(device)->connection->sender_work,
- &device->unplug_work);
+ struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+ int vnr;
+
+restart:
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ for_each_peer_device_rcu(peer_device, device) {
+ if (test_bit(GOT_NEG_ACK, peer_device->flags)) {
+ clear_bit(GOT_NEG_ACK, peer_device->flags);
+ rcu_read_unlock();
+ wait_event(resource->state_wait, peer_device->disk_state[NOW] < D_UP_TO_DATE);
+ goto restart;
+ }
}
}
+ rcu_read_unlock();
}
-static void drbd_set_defaults(struct drbd_device *device)
+/* Barrier numbers are epoch numbers, with gaps. Once the sender is past @epoch,
+ * the barriers of earlier epochs are acked if all are, or if one >= @epoch - 1 is.
+ */
+static bool barriers_acked(struct drbd_resource *resource, unsigned int epoch, u64 *done)
{
- /* Beware! The actual layout differs
- * between big endian and little endian */
- device->state = (union drbd_dev_state) {
- { .role = R_SECONDARY,
- .peer = R_UNKNOWN,
- .conn = C_STANDALONE,
- .disk = D_DISKLESS,
- .pdsk = D_UNKNOWN,
- } };
+ struct drbd_connection *connection;
+ bool acked = true;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ u64 mask = NODE_MASK(connection->peer_node_id);
+ bool seen;
+ int epoch_nr, sent, ack;
+
+ if (*done & mask)
+ continue;
+ /* without the connection, the peer gets the writes by resync */
+ if (connection->cstate[NOW] != C_CONNECTED) {
+ *done |= mask;
+ continue;
+ }
+ /* pairs with smp_store_release() in re_init_if_first_write() */
+ seen = smp_load_acquire(&connection->send.seen_any_write_yet);
+ /* pairs with smp_store_release() in maybe_send_barrier() */
+ epoch_nr = smp_load_acquire(&connection->send.current_epoch_nr);
+ if (seen && (int)(epoch_nr - epoch) < 0) {
+ acked = false;
+ break;
+ }
+ sent = READ_ONCE(connection->send.last_sent_epoch_nr);
+ ack = READ_ONCE(connection->last_acked_epoch_nr);
+ if (ack != sent && (int)(ack - (epoch - 1)) < 0) {
+ acked = false;
+ break;
+ }
+ *done |= mask;
+ }
+ rcu_read_unlock();
+
+ return acked;
}
-void drbd_init_set_defaults(struct drbd_device *device)
+/* Close the epoch, wait for each connected peer's barrier acks of the writes
+ * in earlier epochs. A later write delays it by at most one barrier ack.
+ */
+static void wait_for_barrier_acks(struct drbd_resource *resource)
{
- /* the memset(,0,) did most of this.
- * note: only assignments, no allocation in here */
+ unsigned int epoch;
+ u64 done = 0;
- drbd_set_defaults(device);
+ start_new_tl_epoch(resource);
+ epoch = atomic_read(&resource->current_tle_nr);
+ wait_event(resource->barrier_wait, barriers_acked(resource, epoch, &done));
+}
- atomic_set(&device->ap_bio_cnt, 0);
- atomic_set(&device->ap_actlog_cnt, 0);
- atomic_set(&device->ap_pending_cnt, 0);
- atomic_set(&device->rs_pending_cnt, 0);
- atomic_set(&device->unacked_cnt, 0);
- atomic_set(&device->local_cnt, 0);
- atomic_set(&device->pp_in_use_by_net, 0);
- atomic_set(&device->rs_sect_in, 0);
- atomic_set(&device->rs_sect_ev, 0);
- atomic_set(&device->ap_in_flight, 0);
- atomic_set(&device->md_io.in_use, 0);
+/* Like wait_for_barrier_acks(); returns -ERESTARTSYS when interrupted */
+int wait_for_barrier_acks_interruptible(struct drbd_resource *resource)
+{
+ unsigned int epoch;
+ u64 done = 0;
- mutex_init(&device->own_state_mutex);
- device->state_mutex = &device->own_state_mutex;
+ start_new_tl_epoch(resource);
+ epoch = atomic_read(&resource->current_tle_nr);
+ return wait_event_interruptible(resource->barrier_wait,
+ barriers_acked(resource, epoch, &done));
+}
- spin_lock_init(&device->al_lock);
- spin_lock_init(&device->peer_seq_lock);
-
- INIT_LIST_HEAD(&device->active_ee);
- INIT_LIST_HEAD(&device->sync_ee);
- INIT_LIST_HEAD(&device->done_ee);
- INIT_LIST_HEAD(&device->read_ee);
- INIT_LIST_HEAD(&device->resync_reads);
- INIT_LIST_HEAD(&device->resync_work.list);
- INIT_LIST_HEAD(&device->unplug_work.list);
- INIT_LIST_HEAD(&device->bm_io_work.w.list);
- INIT_LIST_HEAD(&device->pending_master_completion[0]);
- INIT_LIST_HEAD(&device->pending_master_completion[1]);
- INIT_LIST_HEAD(&device->pending_completion[0]);
- INIT_LIST_HEAD(&device->pending_completion[1]);
+static void drbd_fsync_device(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
- device->resync_work.cb = w_resync_timer;
- device->unplug_work.cb = w_send_write_hint;
- device->bm_io_work.w.cb = w_bitmap_io;
+ sync_blockdev(device->vdisk->part0);
+ /* Prevent writes occurring after demotion, at least
+ * the writes already submitted in this context. This
+ * covers the case where DRBD auto-demotes on release,
+ * which is important because it often occurs
+ * immediately after a write. */
+ wait_event(device->misc_wait, !atomic_read(&device->ap_bio_cnt[WRITE]));
+
+ wait_for_barrier_acks(resource);
+ /* After waiting for pending barriers, we got any possible NEG_ACKs,
+ and see them in wait_for_peer_disk_updates() */
+ wait_for_peer_disk_updates(resource);
+
+ /* In case switching from R_PRIMARY to R_SECONDARY works
+ out, there is no rw opener at this point. Thus, no new
+ writes can come in. -> Flushing queued peer acks is
+ necessary and sufficient.
+ The cluster wide role change required packets to be
+ received by the sender. -> We can be sure that the
+ peer_acks queued on a sender's TODO list go out before
+ we send the two phase commit packet.
+ */
+ drbd_flush_peer_acks(resource);
+}
- timer_setup(&device->resync_timer, resync_timer_fn, 0);
- timer_setup(&device->md_sync_timer, md_sync_timer_fn, 0);
- timer_setup(&device->start_resync_timer, start_resync_timer_fn, 0);
- timer_setup(&device->request_timer, request_timer_fn, 0);
+static void drbd_release(struct gendisk *gd)
+{
+ struct drbd_device *device = gd->private_data;
+ struct drbd_resource *resource = device->resource;
+ int open_rw_cnt, open_ro_cnt;
+
+ mutex_lock(&resource->open_release);
+ /* The last one to close already called sync_blockdevice(), generic
+ * bdev_release() respectively blkdev_put_whole() takes care of that.
+ * We still want our side effects of drbd_fsync_device():
+ * wait until all peers confirmed they have all the data, regardless of
+ * replication protocol, even if that is asynchronous.
+ * Still, do it before decreasing the open_cnt, just in case, so we
+ * won't confuse drbd_reject_write_early() or other code paths that may
+ * check for open_cnt != 0 when they see write requests.
+ */
+ if (device->writable && device->open_cnt == 1) {
+ drbd_fsync_device(device);
+ device->writable = false;
+ }
+ device->open_cnt--;
+ drbd_open_counts(resource, &open_rw_cnt, &open_ro_cnt);
- init_waitqueue_head(&device->misc_wait);
- init_waitqueue_head(&device->state_wait);
- init_waitqueue_head(&device->ee_wait);
- init_waitqueue_head(&device->al_wait);
- init_waitqueue_head(&device->seq_wait);
+ if (open_ro_cnt == 0)
+ wake_up_all(&resource->state_wait);
+
+ if (test_bit(UNREGISTERED, &device->flags) && device->open_cnt == 0 &&
+ !test_and_set_bit(DESTROYING_DEV, &device->flags))
+ call_rcu(&device->rcu, drbd_reclaim_device);
+
+ if (resource->res_opts.auto_promote &&
+ open_rw_cnt == 0 &&
+ resource->role[NOW] == R_PRIMARY &&
+ !test_bit(EXPLICIT_PRIMARY, &resource->flags)) {
+ union comm_pid_tag_buf tag;
+ int rv;
+
+ snprintf_current_comm_pid_tag(&tag, "auto-demote");
+
+ /*
+ * Auto-demote is triggered by the last opener releasing the
+ * DRBD device. However, it is an implicit action, so it should
+ * not be affected by the state of the process. In particular,
+ * it should ignore any pending signals. It may be the case
+ * that the process is releasing DRBD because it is being
+ * terminated using a signal. Masking signals is not enough:
+ * a group exit's SIGKILL and TIF_NOTIFY_SIGNAL are pending
+ * through any mask, so the waits themselves ignore signals.
+ */
+ rv = drbd_set_role(resource, R_SECONDARY, CS_NO_SIGNALS, tag.buf, NULL);
+ if (rv < SS_SUCCESS)
+ drbd_warn(resource, "Auto-demote failed: %s (%d)\n",
+ drbd_set_st_err_str(rv), rv);
+ }
+
+ if (open_ro_cnt == 0 && open_rw_cnt == 0 && resource->fail_io[NOW]) {
+ unsigned long irq_flags;
+
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ resource->fail_io[NEW] = false;
+ end_state_change(resource, &irq_flags, "release");
+ }
- device->resync_wenr = LC_FREE;
- device->peer_max_bio_size = DRBD_MAX_BIO_SIZE_SAFE;
- device->local_max_bio_size = DRBD_MAX_BIO_SIZE_SAFE;
+ /* if the open count is 0, we free the whole list, otherwise we remove the specific pid */
+ prune_or_free_openers(device, (device->open_cnt == 0) ? 0 : task_pid_nr(current));
+ if (open_rw_cnt == 0 && open_ro_cnt == 0 && resource->auto_promoted_by.pid != 0)
+ memset(&resource->auto_promoted_by, 0, sizeof(resource->auto_promoted_by));
+ if (device->open_cnt == 0) {
+ struct drbd_device_info info;
+
+ device_to_info(&info, device);
+ mutex_lock(¬ification_mutex);
+ notify_device_state(NULL, 0, device, &info, NOTIFY_CHANGE);
+ mutex_unlock(¬ification_mutex);
+ }
+ mutex_unlock(&resource->open_release);
+ kref_put(&device->kref, drbd_destroy_device); /* might destroy the resource as well */
}
-void drbd_set_my_capacity(struct drbd_device *device, sector_t size)
+static void drbd_remove_all_paths(struct drbd_connection *connection)
{
- char ppb[10];
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_path *path, *tmp;
+
+ lockdep_assert_held(&resource->conf_update);
+
+ list_for_each_entry(path, &transport->paths, list)
+ set_bit(TR_UNREGISTERED, &path->flags);
- set_capacity_and_notify(device->vdisk, size);
+ /* Ensure flag visible before list manipulation. */
+ smp_wmb();
- drbd_info(device, "size = %s (%llu KB)\n",
- ppsize(ppb, size>>1), (unsigned long long)size>>1);
+ list_for_each_entry_safe(path, tmp, &transport->paths, list) {
+ /* Exclusive with reading state, in particular remember_state_change() */
+ write_lock_irq(&resource->state_rwlock);
+ list_del_rcu(&path->list);
+ write_unlock_irq(&resource->state_rwlock);
+
+ transport->class->ops.remove_path(path);
+ notify_path(connection, path, NOTIFY_DESTROY);
+ call_rcu(&path->rcu, drbd_reclaim_path);
+ }
}
-void drbd_device_cleanup(struct drbd_device *device)
+/** __drbd_net_exit is called when a network namespace is removed.
+ *
+ * For DRBD this means it needs to remove any sockets assigned to that namespace,
+ * i.e. it needs to disconnect some connections. It also needs to remove those
+ * paths associated with the to be removed namespace, so the connection can be
+ * reconfigured from a new namespace.
+ */
+static void __net_exit __drbd_net_exit(struct net *net)
{
- int i;
- if (first_peer_device(device)->connection->receiver.t_state != NONE)
- drbd_err(device, "ASSERT FAILED: receiver t_state == %d expected 0.\n",
- first_peer_device(device)->connection->receiver.t_state);
-
- device->al_writ_cnt =
- device->bm_writ_cnt =
- device->read_cnt =
- device->recv_cnt =
- device->send_cnt =
- device->writ_cnt =
- device->p_size =
- device->rs_start =
- device->rs_total =
- device->rs_failed = 0;
- device->rs_last_events = 0;
- device->rs_last_sect_ev = 0;
- for (i = 0; i < DRBD_SYNC_MARKS; i++) {
- device->rs_mark_left[i] = 0;
- device->rs_mark_time[i] = 0;
- }
- D_ASSERT(device, first_peer_device(device)->connection->net_conf == NULL);
-
- set_capacity_and_notify(device->vdisk, 0);
- if (device->bitmap) {
- /* maybe never allocated. */
- drbd_bm_resize(device, 0, 1);
- drbd_bm_cleanup(device);
+ struct drbd_resource *resource;
+ struct drbd_connection *connection, *n;
+ enum drbd_state_rv rv;
+ LIST_HEAD(connections_wait_list);
+
+ /* Disconnect and removal of paths works in 3 steps:
+ * 1. Find all connections associated with the namespace, add it to a separate list.
+ * 2. Iterate over all connections in the new list and start the disconnect.
+ * 3. Iterate again over all connections, waiting for them to disconnect and remove the path configuration.*/
+
+ /* Step 1 */
+ rcu_read_lock();
+ for_each_resource_rcu(resource, &drbd_resources) {
+ for_each_connection_rcu(connection, resource) {
+ /* We don't have to worry about any races here:
+ * For a connection to be "missed", it would need to be configured
+ * from the namespace to be removed. Since netlink does keep the
+ * namespace alive for the duration of it's connection, we can
+ * assume the namespace assignment can no longer be changed. */
+ if (net_eq(net, drbd_net_assigned_to_connection(connection))) {
+ drbd_info(connection, "Disconnect because network namespace is exiting\n");
+ kref_get(&connection->kref);
+
+ list_add(&connection->remove_net_list, &connections_wait_list);
+ }
+ }
}
+ rcu_read_unlock();
- drbd_backing_dev_free(device, device->ldev);
- device->ldev = NULL;
+ /* Step 2 */
+ list_for_each_entry(connection, &connections_wait_list, remove_net_list) {
+ /* We just start the disconnect here. We have to use force=true here,
+ * otherwise the disconnect might fail waiting for some openers to disappear.
+ *
+ * Actually waiting for the disconnect is relegated to step 3, so we disconnect
+ * in parallel. */
+ rv = change_cstate(connection, C_DISCONNECTING, CS_HARD);
+ if (rv < SS_SUCCESS && rv != SS_ALREADY_STANDALONE)
+ drbd_err(connection, "Failed to disconnect: %s\n", drbd_set_st_err_str(rv));
+ }
- clear_bit(AL_SUSPENDED, &device->flags);
+ /* Step 3 */
+ list_for_each_entry_safe(connection, n, &connections_wait_list, remove_net_list) {
+ list_del_init(&connection->remove_net_list);
- D_ASSERT(device, list_empty(&device->active_ee));
- D_ASSERT(device, list_empty(&device->sync_ee));
- D_ASSERT(device, list_empty(&device->done_ee));
- D_ASSERT(device, list_empty(&device->read_ee));
- D_ASSERT(device, list_empty(&device->resync_reads));
- D_ASSERT(device, list_empty(&first_peer_device(device)->connection->sender_work.q));
- D_ASSERT(device, list_empty(&device->resync_work.list));
- D_ASSERT(device, list_empty(&device->unplug_work.list));
+ /* Wait here for StandAlone: a path can only be removed if it's not established */
+ wait_event(connection->resource->state_wait, connection->cstate[NOW] == C_STANDALONE);
- drbd_set_defaults(device);
+ mutex_lock(&connection->resource->adm_mutex);
+ mutex_lock(&connection->resource->conf_update);
+ drbd_remove_all_paths(connection);
+ mutex_unlock(&connection->resource->conf_update);
+ mutex_unlock(&connection->resource->adm_mutex);
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+}
+
+void drbd_queue_unplug(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
+ struct drbd_connection *connection;
+ u64 dagtag_sector;
+
+ dagtag_sector = resource->dagtag_sector;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ /* use the "next" slot */
+ unsigned int i = !connection->todo.unplug_slot;
+
+ connection->todo.unplug_dagtag_sector[i] = dagtag_sector;
+ wake_up(&connection->sender_work.q_wait);
+ }
+ rcu_read_unlock();
}
+static void drbd_set_defaults(struct drbd_device *device)
+{
+ device->disk_state[NOW] = D_DISKLESS;
+}
static void drbd_destroy_mempools(void)
{
- /* D_ASSERT(device, atomic_read(&drbd_pp_vacant)==0); */
-
+ bioset_exit(&drbd_peer_bio_set);
bioset_exit(&drbd_io_bio_set);
bioset_exit(&drbd_md_io_bio_set);
mempool_exit(&drbd_buffer_page_pool);
@@ -2054,15 +3427,11 @@ static void drbd_destroy_mempools(void)
mempool_exit(&drbd_request_mempool);
kmem_cache_destroy(drbd_ee_cache);
kmem_cache_destroy(drbd_request_cache);
- kmem_cache_destroy(drbd_bm_ext_cache);
kmem_cache_destroy(drbd_al_ext_cache);
drbd_ee_cache = NULL;
drbd_request_cache = NULL;
- drbd_bm_ext_cache = NULL;
drbd_al_ext_cache = NULL;
-
- return;
}
static int drbd_create_mempools(void)
@@ -2081,11 +3450,6 @@ static int drbd_create_mempools(void)
if (drbd_ee_cache == NULL)
goto Enomem;
- drbd_bm_ext_cache = kmem_cache_create(
- "drbd_bm", sizeof(struct bm_extent), 0, 0, NULL);
- if (drbd_bm_ext_cache == NULL)
- goto Enomem;
-
drbd_al_ext_cache = kmem_cache_create(
"drbd_al", sizeof(struct lc_element), 0, 0, NULL);
if (drbd_al_ext_cache == NULL)
@@ -2101,10 +3465,14 @@ static int drbd_create_mempools(void)
if (ret)
goto Enomem;
- ret = mempool_init_page_pool(&drbd_md_io_page_pool, DRBD_MIN_POOL_PAGES, 0);
+ ret = bioset_init(&drbd_peer_bio_set, BIO_POOL_SIZE,
+ offsetof(struct drbd_peer_bio, bio), BIOSET_NEED_BVECS);
if (ret)
goto Enomem;
+ ret = mempool_init_page_pool(&drbd_md_io_page_pool, DRBD_MIN_POOL_PAGES, 0);
+ if (ret)
+ goto Enomem;
ret = mempool_init_page_pool(&drbd_buffer_page_pool, number, 0);
if (ret)
goto Enomem;
@@ -2125,71 +3493,76 @@ static int drbd_create_mempools(void)
return -ENOMEM;
}
-static void drbd_release_all_peer_reqs(struct drbd_device *device)
+static void free_peer_device(struct drbd_peer_device *peer_device)
{
- int rr;
+ if (test_and_clear_bit(HOLDING_UUID_READ_LOCK, peer_device->flags))
+ up_read_non_owner(&peer_device->device->uuid_sem);
- rr = drbd_free_peer_reqs(device, &device->active_ee);
- if (rr)
- drbd_err(device, "%d EEs in active list found!\n", rr);
+ kfree(peer_device->rs_plan_s);
+ kfree(peer_device->conf);
+ kfree(peer_device);
+}
- rr = drbd_free_peer_reqs(device, &device->sync_ee);
- if (rr)
- drbd_err(device, "%d EEs in sync list found!\n", rr);
+static void drbd_device_finalize_work_fn(struct work_struct *work)
+{
+ struct drbd_device *device = container_of(work, struct drbd_device, finalize_work);
+ struct drbd_resource *resource = device->resource;
- rr = drbd_free_peer_reqs(device, &device->read_ee);
- if (rr)
- drbd_err(device, "%d EEs in read list found!\n", rr);
+ /* ldev_safe: no other contexts can access */
+ drbd_bm_free(device);
- rr = drbd_free_peer_reqs(device, &device->done_ee);
- if (rr)
- drbd_err(device, "%d EEs in done list found!\n", rr);
+ put_disk(device->vdisk);
+
+ kfree(device);
+ kref_put(&resource->kref, drbd_destroy_resource);
}
-/* caution. no locking. */
+/* may not sleep, called from call_rcu. */
void drbd_destroy_device(struct kref *kref)
{
struct drbd_device *device = container_of(kref, struct drbd_device, kref);
- struct drbd_resource *resource = device->resource;
- struct drbd_peer_device *peer_device, *tmp_peer_device;
-
- timer_shutdown_sync(&device->request_timer);
- timer_shutdown_sync(&device->start_resync_timer);
-
- /* paranoia asserts */
- D_ASSERT(device, device->open_cnt == 0);
- /* end paranoia asserts */
+ struct drbd_peer_device *peer_device, *tmp;
/* cleanup stuff that may have been allocated during
* device (re-)configuration or state changes */
- drbd_backing_dev_free(device, device->ldev);
- device->ldev = NULL;
+#ifdef CONFIG_DRBD_COMPAT_84
+ if (device->resource->res_opts.drbd8_compat_mode)
+ atomic_dec(&nr_drbd8_devices);
+#endif
- drbd_release_all_peer_reqs(device);
+ free_openers(device);
lc_destroy(device->act_log);
- lc_destroy(device->resync);
-
- kfree(device->p_uuid);
- /* device->p_uuid = NULL; */
+ for_each_peer_device_safe(peer_device, tmp, device) {
+ kref_put(&peer_device->connection->kref, drbd_destroy_connection);
+ free_peer_device(peer_device);
+ }
- if (device->bitmap) /* should no longer be there. */
- drbd_bm_cleanup(device);
__free_page(device->md_io.page);
- put_disk(device->vdisk);
- kfree(device->rs_plan_s);
- /* not for_each_connection(connection, resource):
- * those may have been cleaned up and disassociated already.
- */
- for_each_peer_device_safe(peer_device, tmp_peer_device, device) {
- kref_put(&peer_device->connection->kref, drbd_destroy_connection);
- kfree(peer_device);
- }
- if (device->submit.wq)
- destroy_workqueue(device->submit.wq);
- kfree(device);
+ INIT_WORK(&device->finalize_work, drbd_device_finalize_work_fn);
+ schedule_work(&device->finalize_work);
+}
+
+void drbd_destroy_resource(struct kref *kref)
+{
+ struct drbd_resource *resource = container_of(kref, struct drbd_resource, kref);
+
+ idr_destroy(&resource->devices);
+ free_cpumask_var(resource->cpu_mask);
+ kfree(resource->name);
+ kfree(resource);
+ module_put(THIS_MODULE);
+}
+
+void drbd_reclaim_resource(struct rcu_head *rp)
+{
+ struct drbd_resource *resource = container_of(rp, struct drbd_resource, rcu);
+
+ drbd_thread_stop_nowait(&resource->worker);
+
+ mempool_free(resource->peer_ack_req, &drbd_request_mempool);
kref_put(&resource->kref, drbd_destroy_resource);
}
@@ -2214,97 +3587,88 @@ static void do_retry(struct work_struct *ws)
list_splice_init(&retry->writes, &writes);
spin_unlock_irq(&retry->lock);
- list_for_each_entry_safe(req, tmp, &writes, tl_requests) {
+ list_for_each_entry_safe(req, tmp, &writes, list) {
struct drbd_device *device = req->device;
+ struct drbd_resource *resource = device->resource;
struct bio *bio = req->master_bio;
+ unsigned long start_jif = req->start_jif;
bool expected;
+ ktime_var_assign_for_accounting(start_kt, req->start_kt);
+
+ /* No locking when accessing local_rq_state & net_rq_state, since
+ * this request is not active at the moment. */
expected =
expect(device, atomic_read(&req->completion_ref) == 0) &&
- expect(device, req->rq_state & RQ_POSTPONED) &&
- expect(device, (req->rq_state & RQ_LOCAL_PENDING) == 0 ||
- (req->rq_state & RQ_LOCAL_ABORTED) != 0);
+ expect(device, req->local_rq_state & RQ_POSTPONED) &&
+ expect(device, (req->local_rq_state & RQ_LOCAL_PENDING) == 0 ||
+ (req->local_rq_state & RQ_LOCAL_ABORTED) != 0);
if (!expected)
drbd_err(device, "req=%p completion_ref=%d rq_state=%x\n",
req, atomic_read(&req->completion_ref),
- req->rq_state);
+ req->local_rq_state);
- /* We still need to put one kref associated with the
+ /* We still need to put one done reference associated with the
* "completion_ref" going zero in the code path that queued it
* here. The request object may still be referenced by a
* frozen local req->private_bio, in case we force-detached.
*/
- kref_put(&req->kref, drbd_req_destroy);
+ read_lock_irq(&resource->state_rwlock);
+ drbd_put_ref_tl_walk(req, 1, 0);
+ read_unlock_irq(&resource->state_rwlock);
/* A single suspended or otherwise blocking device may stall
- * all others as well. Fortunately, this code path is to
- * recover from a situation that "should not happen":
- * concurrent writes in multi-primary setup.
- * In a "normal" lifecycle, this workqueue is supposed to be
- * destroyed without ever doing anything.
- * If it turns out to be an issue anyways, we can do per
+ * all others as well. This code path is to recover from a
+ * situation that "should not happen": concurrent writes in
+ * multi-primary setup. It is also used for retrying failed
+ * reads. If it turns out to be an issue, we can do per
* resource (replication group) or per device (minor) retry
* workqueues instead.
*/
/* We are not just doing submit_bio_noacct(),
* as we want to keep the start_time information. */
- inc_ap_bio(device);
- __drbd_make_request(device, bio);
+ __drbd_make_request(device, bio, start_kt, start_jif);
}
}
-/* called via drbd_req_put_completion_ref(),
- * holds resource->req_lock */
+/* called via drbd_req_put_completion_ref() */
void drbd_restart_request(struct drbd_request *req)
{
+ struct drbd_device *device = req->device;
+ struct drbd_resource *resource = device->resource;
+ bool susp = drbd_suspended(device);
unsigned long flags;
+
spin_lock_irqsave(&retry.lock, flags);
- list_move_tail(&req->tl_requests, &retry.writes);
+ list_move_tail(&req->list, susp ? &resource->suspended_reqs : &retry.writes);
spin_unlock_irqrestore(&retry.lock, flags);
/* Drop the extra reference that would otherwise
* have been dropped by complete_master_bio.
* do_retry() needs to grab a new one. */
- dec_ap_bio(req->device);
+ dec_ap_bio(req->device, bio_data_dir(req->master_bio));
- queue_work(retry.wq, &retry.worker);
+ if (!susp)
+ queue_work(retry.wq, &retry.worker);
}
-void drbd_destroy_resource(struct kref *kref)
+void drbd_restart_suspended_reqs(struct drbd_resource *resource)
{
- struct drbd_resource *resource =
- container_of(kref, struct drbd_resource, kref);
-
- idr_destroy(&resource->devices);
- free_cpumask_var(resource->cpu_mask);
- kfree(resource->name);
- kfree(resource);
-}
+ unsigned long flags;
-void drbd_free_resource(struct drbd_resource *resource)
-{
- struct drbd_connection *connection, *tmp;
+ spin_lock_irqsave(&retry.lock, flags);
+ list_splice_init(&resource->suspended_reqs, &retry.writes);
+ spin_unlock_irqrestore(&retry.lock, flags);
- for_each_connection_safe(connection, tmp, resource) {
- set_bit(C_UNREGISTERED, &connection->flags);
- list_del(&connection->connections);
- drbd_debugfs_connection_cleanup(connection);
- kref_put(&connection->kref, drbd_destroy_connection);
- }
- drbd_debugfs_resource_cleanup(resource);
- kref_put(&resource->kref, drbd_destroy_resource);
+ queue_work(retry.wq, &retry.worker);
}
static void drbd_cleanup(void)
{
- unsigned int i;
- struct drbd_device *device;
- struct drbd_resource *resource, *tmp;
-
/* first remove proc,
- * drbdsetup uses it's presence to detect
+ * drbdsetup uses its presence to detect
* whether DRBD is loaded.
* If we would get stuck in proc removal,
* but have netlink already deregistered,
@@ -2317,20 +3681,15 @@ static void drbd_cleanup(void)
if (retry.wq)
destroy_workqueue(retry.wq);
- genl_unregister_family(&drbd_nl_family);
-
- idr_for_each_entry(&drbd_devices, device, i)
- drbd_delete_device(device);
-
- /* not _rcu since, no other updater anymore. Genl already unregistered */
- for_each_resource_safe(resource, tmp, &drbd_resources) {
- list_del(&resource->resources);
- drbd_free_resource(resource);
- }
-
+ drbd_nl_drbd2_exit();
+ drbd_nl_legacy_exit();
drbd_debugfs_cleanup();
+ unregister_pernet_device(&drbd_pernet_ops);
+
drbd_destroy_mempools();
+ if (ping_ack_sender)
+ destroy_workqueue(ping_ack_sender);
unregister_blkdev(DRBD_MAJOR, "drbd");
idr_destroy(&drbd_devices);
@@ -2338,7 +3697,7 @@ static void drbd_cleanup(void)
pr_info("module cleanup done.\n");
}
-static void drbd_init_workqueue(struct drbd_work_queue* wq)
+static void drbd_init_workqueue(struct drbd_work_queue *wq)
{
spin_lock_init(&wq->q_lock);
INIT_LIST_HEAD(&wq->q);
@@ -2359,6 +3718,16 @@ static int w_complete(struct drbd_work *w, int cancel)
return 0;
}
+void drbd_queue_work(struct drbd_work_queue *q, struct drbd_work *w)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&q->q_lock, flags);
+ list_add_tail(&w->list, &q->q);
+ spin_unlock_irqrestore(&q->q_lock, flags);
+ wake_up(&q->q_wait);
+}
+
void drbd_flush_workqueue(struct drbd_work_queue *work_queue)
{
struct completion_work completion_work;
@@ -2369,6 +3738,23 @@ void drbd_flush_workqueue(struct drbd_work_queue *work_queue)
wait_for_completion(&completion_work.done);
}
+void drbd_flush_workqueue_interruptible(struct drbd_device *device)
+{
+ struct completion_work completion_work;
+ int err;
+
+ completion_work.w.cb = w_complete;
+ init_completion(&completion_work.done);
+ drbd_queue_work(&device->resource->work, &completion_work.w);
+ err = wait_for_completion_interruptible(&completion_work.done);
+ if (err == -ERESTARTSYS) {
+ set_bit(ABORT_MDIO, &device->flags);
+ wake_up_all(&device->misc_wait);
+ wait_for_completion(&completion_work.done);
+ clear_bit(ABORT_MDIO, &device->flags);
+ }
+}
+
struct drbd_resource *drbd_find_resource(const char *name)
{
struct drbd_resource *resource;
@@ -2389,51 +3775,58 @@ struct drbd_resource *drbd_find_resource(const char *name)
return resource;
}
-struct drbd_connection *conn_get_by_addrs(void *my_addr, int my_addr_len,
- void *peer_addr, int peer_addr_len)
+static void drbd_put_send_buffers(struct drbd_connection *connection)
{
- struct drbd_resource *resource;
- struct drbd_connection *connection;
+ unsigned int i;
- rcu_read_lock();
- for_each_resource_rcu(resource, &drbd_resources) {
- for_each_connection_rcu(connection, resource) {
- if (connection->my_addr_len == my_addr_len &&
- connection->peer_addr_len == peer_addr_len &&
- !memcmp(&connection->my_addr, my_addr, my_addr_len) &&
- !memcmp(&connection->peer_addr, peer_addr, peer_addr_len)) {
- kref_get(&connection->kref);
- goto found;
- }
+ for (i = DATA_STREAM; i <= CONTROL_STREAM ; i++) {
+ if (connection->send_buffer[i].page) {
+ put_page(connection->send_buffer[i].page);
+ connection->send_buffer[i].page = NULL;
}
}
- connection = NULL;
-found:
- rcu_read_unlock();
- return connection;
}
-static int drbd_alloc_socket(struct drbd_socket *socket)
+static int drbd_alloc_send_buffers(struct drbd_connection *connection)
{
- socket->rbuf = (void *) __get_free_page(GFP_KERNEL);
- if (!socket->rbuf)
- return -ENOMEM;
- socket->sbuf = (void *) __get_free_page(GFP_KERNEL);
- if (!socket->sbuf)
- return -ENOMEM;
+ unsigned int i;
+
+ for (i = DATA_STREAM; i <= CONTROL_STREAM ; i++) {
+ struct page *page;
+
+ page = alloc_page(GFP_KERNEL);
+ if (!page) {
+ drbd_put_send_buffers(connection);
+ return -ENOMEM;
+ }
+ connection->send_buffer[i].page = page;
+ connection->send_buffer[i].unsent =
+ connection->send_buffer[i].pos = page_address(page);
+ }
+
return 0;
}
-static void drbd_free_socket(struct drbd_socket *socket)
+void drbd_flush_peer_acks(struct drbd_resource *resource)
{
- free_page((unsigned long) socket->sbuf);
- free_page((unsigned long) socket->rbuf);
+ spin_lock_irq(&resource->peer_ack_lock);
+ if (resource->peer_ack_req) {
+ resource->last_peer_acked_dagtag = resource->peer_ack_req->dagtag_sector;
+ drbd_queue_peer_ack(resource, resource->peer_ack_req);
+ resource->peer_ack_req = NULL;
+ }
+ spin_unlock_irq(&resource->peer_ack_lock);
}
-void conn_free_crypto(struct drbd_connection *connection)
+static void peer_ack_timer_fn(struct timer_list *t)
{
- drbd_free_sock(connection);
+ struct drbd_resource *resource = timer_container_of(resource, t, peer_ack_timer);
+
+ drbd_flush_peer_acks(resource);
+}
+void conn_free_crypto(struct drbd_connection *connection)
+{
crypto_free_shash(connection->csums_tfm);
crypto_free_shash(connection->verify_tfm);
crypto_free_shash(connection->cram_hmac_tfm);
@@ -2451,11 +3844,27 @@ void conn_free_crypto(struct drbd_connection *connection)
connection->int_dig_vv = NULL;
}
-int set_resource_options(struct drbd_resource *resource, struct res_opts *res_opts)
+static void wake_all_device_misc(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr)
+ wake_up(&device->misc_wait);
+ rcu_read_unlock();
+}
+
+int set_resource_options(struct drbd_resource *resource, struct drbd_res_opts *res_opts,
+ const char *tag)
{
struct drbd_connection *connection;
cpumask_var_t new_cpu_mask;
int err;
+ bool wake_device_misc = false;
+ bool force_state_recalc = false;
+ unsigned long irq_flags;
+ struct drbd_res_opts *old_opts = &resource->res_opts;
if (!zalloc_cpumask_var(&new_cpu_mask, GFP_KERNEL))
return -ENOMEM;
@@ -2467,6 +3876,7 @@ int set_resource_options(struct drbd_resource *resource, struct res_opts *res_op
if (err == -EOVERFLOW) {
/* So what. mask it out. */
cpumask_var_t tmp_cpu_mask;
+
if (zalloc_cpumask_var(&tmp_cpu_mask, GFP_KERNEL)) {
cpumask_setall(tmp_cpu_mask);
cpumask_and(new_cpu_mask, new_cpu_mask, tmp_cpu_mask);
@@ -2484,26 +3894,47 @@ int set_resource_options(struct drbd_resource *resource, struct res_opts *res_op
goto fail;
}
}
+ if (res_opts->nr_requests < DRBD_NR_REQUESTS_MIN)
+ res_opts->nr_requests = DRBD_NR_REQUESTS_MIN;
+
+ if (old_opts->quorum != res_opts->quorum ||
+ old_opts->on_no_quorum != res_opts->on_no_quorum)
+ force_state_recalc = true;
+
+ if (resource->res_opts.nr_requests < res_opts->nr_requests)
+ wake_device_misc = true;
+
resource->res_opts = *res_opts;
if (cpumask_empty(new_cpu_mask))
drbd_calc_cpu_mask(&new_cpu_mask);
if (!cpumask_equal(resource->cpu_mask, new_cpu_mask)) {
cpumask_copy(resource->cpu_mask, new_cpu_mask);
+ resource->worker.reset_cpu_mask = 1;
+ rcu_read_lock();
for_each_connection_rcu(connection, resource) {
connection->receiver.reset_cpu_mask = 1;
- connection->ack_receiver.reset_cpu_mask = 1;
- connection->worker.reset_cpu_mask = 1;
+ connection->sender.reset_cpu_mask = 1;
}
+ rcu_read_unlock();
}
err = 0;
+ if (force_state_recalc) {
+ begin_state_change(resource, &irq_flags, CS_VERBOSE | CS_FORCE_RECALC);
+ end_state_change(resource, &irq_flags, tag);
+ }
+
+ if (wake_device_misc)
+ wake_all_device_misc(resource);
+
fail:
free_cpumask_var(new_cpu_mask);
return err;
}
-struct drbd_resource *drbd_create_resource(const char *name)
+struct drbd_resource *drbd_create_resource(const char *name,
+ struct drbd_res_opts *res_opts)
{
struct drbd_resource *resource;
@@ -2516,15 +3947,63 @@ struct drbd_resource *drbd_create_resource(const char *name)
if (!zalloc_cpumask_var(&resource->cpu_mask, GFP_KERNEL))
goto fail_free_name;
kref_init(&resource->kref);
- ratelimit_state_init(&resource->ratelimit[D_RL_R_GENERIC], 5 * HZ, 10);
idr_init(&resource->devices);
INIT_LIST_HEAD(&resource->connections);
- resource->write_ordering = WO_BDEV_FLUSH;
- list_add_tail_rcu(&resource->resources, &drbd_resources);
+ spin_lock_init(&resource->tl_update_lock);
+ INIT_LIST_HEAD(&resource->transfer_log);
+ spin_lock_init(&resource->peer_ack_lock);
+ INIT_LIST_HEAD(&resource->peer_ack_req_list);
+ INIT_LIST_HEAD(&resource->peer_ack_list);
+ INIT_LIST_HEAD(&resource->peer_ack_work.list);
+ spin_lock_init(&resource->send_oos_lock);
+ INIT_LIST_HEAD(&resource->send_oos);
+ resource->peer_ack_work.cb = w_queue_peer_ack;
+ timer_setup(&resource->peer_ack_timer, peer_ack_timer_fn, 0);
+ spin_lock_init(&resource->initiator_flush_lock);
+ sema_init(&resource->state_sem, 1);
+ resource->role[NOW] = R_SECONDARY;
+ resource->max_node_id = res_opts->drbd8_compat_mode ? 1 : res_opts->node_id;
+ resource->twopc_reply.initiator_node_id = -1;
mutex_init(&resource->conf_update);
mutex_init(&resource->adm_mutex);
- spin_lock_init(&resource->req_lock);
+ mutex_init(&resource->open_release);
+ rwlock_init(&resource->state_rwlock);
+ INIT_LIST_HEAD(&resource->listeners);
+ spin_lock_init(&resource->listeners_lock);
+ init_waitqueue_head(&resource->state_wait);
+ init_waitqueue_head(&resource->twopc_wait);
+ init_waitqueue_head(&resource->barrier_wait);
+ timer_setup(&resource->twopc_timer, twopc_timer_fn, 0);
+ INIT_WORK(&resource->twopc_work, nested_twopc_work);
+ drbd_init_workqueue(&resource->work);
+ drbd_thread_init(resource, &resource->worker, drbd_worker, "worker");
+ spin_lock_init(&resource->current_tle_lock);
drbd_debugfs_resource_add(resource);
+ resource->cached_min_aggreed_protocol_version = drbd_protocol_version_min;
+ /* members is a bit mask of the "seen" nodes in this resource.
+ * In drbd8 compatibility mode, we only have one peer, so we can
+ * set this to 1. */
+ resource->members = res_opts->drbd8_compat_mode ? 1 : NODE_MASK(res_opts->node_id);
+ INIT_WORK(&resource->empty_twopc, drbd_empty_twopc_work_fn);
+ INIT_WORK(&resource->resume_twopc, drbd_resume_twopc_work_fn);
+ INIT_LIST_HEAD(&resource->suspended_reqs);
+
+ ratelimit_state_init(&resource->ratelimit[D_RL_R_GENERIC], 5*HZ, 10);
+
+
+ /* lock_order_ok: drbd_adm_new_resource() holds resources_mutex here,
+ * and the state change in set_resource_options() takes state_sem,
+ * while drbd_start_resync() takes resources_mutex under state_sem.
+ * This resource is not on drbd_resources yet, so nobody else can
+ * hold its state_sem, and the two orders can not deadlock.
+ */
+ if (set_resource_options(resource, res_opts, "create-resource"))
+ goto fail_free_name;
+
+ drbd_thread_start(&resource->worker);
+
+ list_add_tail_rcu(&resource->resources, &drbd_resources);
+
return resource;
fail_free_name:
@@ -2536,129 +4015,306 @@ struct drbd_resource *drbd_create_resource(const char *name)
}
/* caller must be under adm_mutex */
-struct drbd_connection *conn_create(const char *name, struct res_opts *res_opts)
+struct drbd_connection *drbd_create_connection(struct drbd_resource *resource,
+ struct drbd_transport_class *tc)
{
- struct drbd_resource *resource;
struct drbd_connection *connection;
+ int size;
- connection = kzalloc_obj(struct drbd_connection);
+ size = sizeof(*connection) - sizeof(connection->transport) + tc->instance_size;
+ connection = kzalloc(size, GFP_KERNEL);
if (!connection)
return NULL;
- if (drbd_alloc_socket(&connection->data))
- goto fail;
- if (drbd_alloc_socket(&connection->meta))
+ ratelimit_state_init(&connection->ratelimit[D_RL_C_GENERIC], 5*HZ, /* no burst */ 1);
+
+ if (drbd_alloc_send_buffers(connection))
goto fail;
connection->current_epoch = kzalloc_obj(struct drbd_epoch);
if (!connection->current_epoch)
goto fail;
- INIT_LIST_HEAD(&connection->transfer_log);
-
INIT_LIST_HEAD(&connection->current_epoch->list);
connection->epochs = 1;
spin_lock_init(&connection->epoch_lock);
+ INIT_LIST_HEAD(&connection->todo.work_list);
+ connection->todo.req = NULL;
+
+ atomic_set(&connection->ap_in_flight, 0);
+ atomic_set(&connection->rs_in_flight, 0);
connection->send.seen_any_write_yet = false;
connection->send.current_epoch_nr = 0;
connection->send.current_epoch_writes = 0;
+ connection->send.current_dagtag_sector =
+ resource->dagtag_sector - ((BIO_MAX_VECS << PAGE_SHIFT) >> SECTOR_SHIFT) - 1;
- resource = drbd_create_resource(name);
- if (!resource)
- goto fail;
-
- connection->cstate = C_STANDALONE;
- mutex_init(&connection->cstate_mutex);
- init_waitqueue_head(&connection->ping_wait);
+ connection->cstate[NOW] = C_STANDALONE;
+ connection->peer_role[NOW] = R_UNKNOWN;
idr_init(&connection->peer_devices);
drbd_init_workqueue(&connection->sender_work);
- mutex_init(&connection->data.mutex);
- mutex_init(&connection->meta.mutex);
+ mutex_init(&connection->mutex[DATA_STREAM]);
+ mutex_init(&connection->mutex[CONTROL_STREAM]);
+
+ INIT_LIST_HEAD(&connection->connect_timer_work.list);
+ timer_setup(&connection->connect_timer, connect_timer_fn, 0);
drbd_thread_init(resource, &connection->receiver, drbd_receiver, "receiver");
connection->receiver.connection = connection;
- drbd_thread_init(resource, &connection->worker, drbd_worker, "worker");
- connection->worker.connection = connection;
- drbd_thread_init(resource, &connection->ack_receiver, drbd_ack_receiver, "ack_recv");
- connection->ack_receiver.connection = connection;
+ drbd_thread_init(resource, &connection->sender, drbd_sender, "sender");
+ connection->sender.connection = connection;
+ spin_lock_init(&connection->primary_flush_lock);
+ spin_lock_init(&connection->flush_ack_lock);
+ spin_lock_init(&connection->peer_reqs_lock);
+ INIT_LIST_HEAD(&connection->peer_requests);
+ INIT_LIST_HEAD(&connection->peer_reads);
+ INIT_LIST_HEAD(&connection->connections);
+ INIT_LIST_HEAD(&connection->done_ee);
+ INIT_LIST_HEAD(&connection->dagtag_wait_ee);
+ INIT_LIST_HEAD(&connection->source_wait_ee);
+ INIT_LIST_HEAD(&connection->dagtag_wait_reqs);
+ INIT_LIST_HEAD(&connection->remove_net_list);
+ init_waitqueue_head(&connection->ee_wait);
kref_init(&connection->kref);
- ratelimit_state_init(&connection->ratelimit[D_RL_C_GENERIC], 5 * HZ, /* no burst */ 1);
- connection->resource = resource;
+ INIT_WORK(&connection->peer_ack_work, drbd_send_peer_ack_wf);
+ INIT_LIST_HEAD(&connection->send_oos_work.list);
+ connection->send_oos_work.cb = drbd_send_out_of_sync_wf;
+ INIT_LIST_HEAD(&connection->flush_ack_work.list);
+ connection->flush_ack_work.cb = drbd_flush_ack_wf;
+ INIT_WORK(&connection->send_acks_work, drbd_send_acks_wf);
+ INIT_WORK(&connection->send_ping_ack_work, drbd_send_ping_ack_wf);
+ INIT_WORK(&connection->send_ping_work, drbd_send_ping_wf);
+
+ INIT_LIST_HEAD(&connection->send_dagtag_work.list);
+ connection->send_dagtag_work.cb = w_send_dagtag;
- if (set_resource_options(resource, res_opts))
- goto fail_resource;
+ spin_lock_init(&connection->advance_cache_ptr_lock);
kref_get(&resource->kref);
- list_add_tail_rcu(&connection->connections, &resource->connections);
- drbd_debugfs_connection_add(connection);
+ connection->resource = resource;
+ connection->after_reconciliation.lost_node_id = -1;
+ connection->reconcile_handshake.lost_node_id = -1;
+ connection->reconcile_handshake.sent_lost_node = false;
+
+ connection->reassemble_buffer.buffer = connection->reassemble_buffer_bytes.bytes;
+
+ INIT_LIST_HEAD(&connection->transport.paths);
+ connection->transport.log_prefix = resource->name;
+ if (tc->ops.init(&connection->transport))
+ goto fail;
+ drbd_connection_update_rx_alignment(resource, connection);
+
return connection;
-fail_resource:
- list_del(&resource->resources);
- drbd_free_resource(resource);
fail:
+ drbd_put_send_buffers(connection);
kfree(connection->current_epoch);
- drbd_free_socket(&connection->meta);
- drbd_free_socket(&connection->data);
kfree(connection);
+
return NULL;
}
+/**
+ * drbd_transport_shutdown() - Free the transport specific members (e.g., sockets) of a connection
+ * @connection: The connection to shut down
+ * @op: The operation. Only close the connection or destroy the whole transport
+ *
+ * Must be called with conf_update held.
+ */
+void drbd_transport_shutdown(struct drbd_connection *connection, enum drbd_tr_free_op op)
+{
+ struct drbd_transport *transport = &connection->transport;
+
+ lockdep_assert_held(&connection->resource->conf_update);
+
+ mutex_lock(&connection->mutex[DATA_STREAM]);
+ mutex_lock(&connection->mutex[CONTROL_STREAM]);
+
+ /* With both stream mutexes held, a prepared packet was abandoned:
+ * its header was never written, so it must not reach the wire.
+ */
+ cancel_send_buffer(connection, DATA_STREAM);
+ cancel_send_buffer(connection, CONTROL_STREAM);
+
+ /* Ignore send errors, if any: we are shutting down. */
+ flush_send_buffer(connection, DATA_STREAM);
+ flush_send_buffer(connection, CONTROL_STREAM);
+
+ /* Holding conf_update ensures that paths list is not modified concurrently. */
+ transport->class->ops.free(transport, op);
+ if (op == DESTROY_TRANSPORT) {
+ drbd_remove_all_paths(connection);
+
+ /* Wait for the delayed drbd_reclaim_path() calls. */
+ rcu_barrier();
+ drbd_put_transport_class(transport->class);
+ }
+
+ mutex_unlock(&connection->mutex[CONTROL_STREAM]);
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+}
+
+void drbd_destroy_path(struct kref *kref)
+{
+ struct drbd_path *path = container_of(kref, struct drbd_path, kref);
+ struct drbd_connection *connection =
+ container_of(path->transport, struct drbd_connection, transport);
+ kref_put(&connection->kref, drbd_destroy_connection);
+ kfree(path);
+}
+
+/* The last ref might be dropped from atomic context, no sleeping functions here */
void drbd_destroy_connection(struct kref *kref)
{
struct drbd_connection *connection = container_of(kref, struct drbd_connection, kref);
struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ int vnr;
if (atomic_read(&connection->current_epoch->epoch_size) != 0)
drbd_err(connection, "epoch_size:%d\n", atomic_read(&connection->current_epoch->epoch_size));
kfree(connection->current_epoch);
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ free_peer_device(peer_device);
+ kref_put(&device->kref, drbd_destroy_device);
+ }
idr_destroy(&connection->peer_devices);
- drbd_free_socket(&connection->meta);
- drbd_free_socket(&connection->data);
- kfree(connection->int_dig_in);
- kfree(connection->int_dig_vv);
+ kfree(connection->transport.net_conf);
kfree(connection);
kref_put(&resource->kref, drbd_destroy_resource);
}
+struct drbd_peer_device *create_peer_device(struct drbd_device *device, struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ int err;
+
+ peer_device = kzalloc_obj(struct drbd_peer_device);
+ if (!peer_device)
+ return NULL;
+
+ peer_device->connection = connection;
+ peer_device->device = device;
+ peer_device->disk_state[NOW] = D_UNKNOWN;
+ peer_device->repl_state[NOW] = L_OFF;
+ peer_device->replication[NOW] = true;
+ peer_device->peer_replication[NOW] = true;
+ spin_lock_init(&peer_device->peer_seq_lock);
+ mutex_init(&peer_device->peer_attach_send_mutex);
+
+ ratelimit_state_init(&peer_device->ratelimit[D_RL_PD_GENERIC], 5*HZ, /* no burst */ 1);
+
+ err = drbd_create_peer_device_default_config(peer_device);
+ if (err) {
+ kfree(peer_device);
+ return NULL;
+ }
+
+ timer_setup(&peer_device->start_resync_timer, start_resync_timer_fn, 0);
+
+ INIT_LIST_HEAD(&peer_device->resync_work.list);
+ peer_device->resync_work.cb = w_resync_timer;
+ timer_setup(&peer_device->resync_timer, resync_timer_fn, 0);
+
+ INIT_LIST_HEAD(&peer_device->propagate_uuids_work.list);
+ peer_device->propagate_uuids_work.cb = w_send_uuids;
+
+ atomic_set(&peer_device->ap_pending_cnt, 0);
+ atomic_set(&peer_device->unacked_cnt, 0);
+ atomic_set(&peer_device->rs_pending_cnt, 0);
+
+ INIT_LIST_HEAD(&peer_device->resync_requests);
+
+ atomic_set(&peer_device->rs_sect_in, 0);
+ atomic_set(&peer_device->rs_sect_parked, 0);
+
+ peer_device->bitmap_index = -1;
+ peer_device->comm_day0_node_id = -1;
+ peer_device->resync_finished_pdsk = D_UNKNOWN;
+
+ peer_device->q_limits.physical_block_size = SECTOR_SIZE;
+ peer_device->q_limits.logical_block_size = SECTOR_SIZE;
+ peer_device->q_limits.alignment_offset = 0;
+ peer_device->q_limits.io_min = SECTOR_SIZE;
+ peer_device->q_limits.io_opt = PAGE_SIZE;
+ peer_device->q_limits.max_bio_size = DRBD_MAX_BIO_SIZE;
+
+ return peer_device;
+}
+
+static void drbd_ldev_destroy(struct work_struct *ws)
+{
+ struct drbd_device *device = container_of(ws, struct drbd_device, ldev_destroy_work);
+
+ /* ldev_safe: destroying the bitmap */
+ drbd_bm_free(device);
+ lc_destroy(device->act_log);
+ device->act_log = NULL;
+ /* ldev_safe: destroying ldev */
+ drbd_backing_dev_free(device, device->ldev);
+ /* ldev_safe: final teardown, no other user possible */
+ device->ldev = NULL;
+ drbd_resource_update_rx_alignment(device->resource);
+
+ clear_bit(GOING_DISKLESS, &device->flags);
+ wake_up(&device->misc_wait);
+ kref_put(&device->kref, drbd_destroy_device);
+}
+
+static int init_conflict_submitter(struct drbd_device *device)
+{
+ /* Short name so that it is recognizable from the first 15 characters. */
+ device->submit_conflict.wq =
+ alloc_ordered_workqueue("drbd%u_sc", WQ_MEM_RECLAIM, device->minor);
+ if (!device->submit_conflict.wq)
+ return -ENOMEM;
+ INIT_WORK(&device->submit_conflict.worker, drbd_do_submit_conflict);
+ INIT_LIST_HEAD(&device->submit_conflict.resync_writes);
+ INIT_LIST_HEAD(&device->submit_conflict.resync_reads);
+ INIT_LIST_HEAD(&device->submit_conflict.writes);
+ INIT_LIST_HEAD(&device->submit_conflict.peer_writes);
+ spin_lock_init(&device->submit_conflict.lock);
+ return 0;
+}
+
static int init_submitter(struct drbd_device *device)
{
- /* opencoded create_singlethread_workqueue(),
- * to be able to say "drbd%d", ..., minor */
device->submit.wq =
alloc_ordered_workqueue("drbd%u_submit", WQ_MEM_RECLAIM, device->minor);
if (!device->submit.wq)
return -ENOMEM;
-
INIT_WORK(&device->submit.worker, do_submit);
INIT_LIST_HEAD(&device->submit.writes);
+ INIT_LIST_HEAD(&device->submit.peer_writes);
+ spin_lock_init(&device->submit.lock);
return 0;
}
-enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsigned int minor)
+enum drbd_ret_code drbd_create_device(struct drbd_adm_ctx *adm_ctx, unsigned int minor,
+ struct drbd_device_conf *device_conf,
+ struct drbd_device **p_device)
{
struct drbd_resource *resource = adm_ctx->resource;
- struct drbd_connection *connection, *n;
+ struct drbd_connection *connection;
struct drbd_device *device;
struct drbd_peer_device *peer_device, *tmp_peer_device;
struct gendisk *disk;
+ LIST_HEAD(peer_devices);
+ LIST_HEAD(tmp);
int id;
int vnr = adm_ctx->volume;
enum drbd_ret_code err = ERR_NOMEM;
- struct queue_limits lim = {
- /*
- * Setting the max_hw_sectors to an odd value of 8kibyte here.
- * This triggers a max_bio_size message upon first attach or
- * connect.
- */
- .max_hw_sectors = DRBD_MAX_BIO_SIZE_SAFE >> 8,
- };
+ bool locked = false;
+
+ lockdep_assert_held(&resource->conf_update);
device = minor_to_device(minor);
if (device)
@@ -2670,28 +4326,70 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig
return ERR_NOMEM;
kref_init(&device->kref);
- ratelimit_state_init(&device->ratelimit[D_RL_D_GENERIC], 5 * HZ, /* no burst */ 1);
- ratelimit_state_init(&device->ratelimit[D_RL_D_METADATA], 5 * HZ, 10);
- ratelimit_state_init(&device->ratelimit[D_RL_D_BACKEND], 5 * HZ, 10);
+ ratelimit_state_init(&device->ratelimit[D_RL_D_GENERIC], 5*HZ, /* no burst */ 1);
+ ratelimit_state_init(&device->ratelimit[D_RL_D_METADATA], 5*HZ, 10);
+ ratelimit_state_init(&device->ratelimit[D_RL_D_BACKEND], 5*HZ, 10);
kref_get(&resource->kref);
device->resource = resource;
device->minor = minor;
device->vnr = vnr;
+ device->device_conf = *device_conf;
+
+ drbd_set_defaults(device);
+
+ atomic_set(&device->ap_bio_cnt[READ], 0);
+ atomic_set(&device->ap_bio_cnt[WRITE], 0);
+ atomic_set(&device->ap_actlog_cnt, 0);
+ atomic_set(&device->wait_for_actlog, 0);
+ atomic_set(&device->wait_for_actlog_ecnt, 0);
+ atomic_set(&device->local_cnt, 0);
+ atomic_set(&device->rs_sect_done, 0);
+ atomic_set(&device->rs_sect_queued, 0);
+ atomic_set(&device->rs_drain_sect, 0);
+ device->rs_drain_since = ktime_get();
+ atomic_set(&device->md_io.in_use, 0);
+
+#ifdef CONFIG_DRBD_TIMING_STATS
+ spin_lock_init(&device->timing_lock);
+#endif
+ spin_lock_init(&device->al_lock);
+ spin_lock_init(&device->gen_obligation_lock);
+
+ spin_lock_init(&device->pending_completion_lock);
+ INIT_LIST_HEAD(&device->pending_master_completion[0]);
+ INIT_LIST_HEAD(&device->pending_master_completion[1]);
+ INIT_LIST_HEAD(&device->pending_completion[0]);
+ INIT_LIST_HEAD(&device->pending_completion[1]);
+ INIT_LIST_HEAD(&device->openers);
+ spin_lock_init(&device->openers_lock);
+ spin_lock_init(&device->peer_req_bio_completion_lock);
+
+ atomic_set(&device->pending_bitmap_work.n, 0);
+ spin_lock_init(&device->pending_bitmap_work.q_lock);
+ INIT_LIST_HEAD(&device->pending_bitmap_work.q);
+
+ timer_setup(&device->md_sync_timer, md_sync_timer_fn, 0);
+ timer_setup(&device->request_timer, request_timer_fn, 0);
+
+ init_waitqueue_head(&device->misc_wait);
+ init_waitqueue_head(&device->al_wait);
+ init_waitqueue_head(&device->seq_wait);
- drbd_init_set_defaults(device);
+ init_rwsem(&device->uuid_sem);
- disk = blk_alloc_disk(&lim, NUMA_NO_NODE);
+ disk = blk_alloc_disk(NULL, NUMA_NO_NODE);
if (IS_ERR(disk)) {
err = PTR_ERR(disk);
goto out_no_disk;
}
+ INIT_WORK(&device->ldev_destroy_work, drbd_ldev_destroy);
+ INIT_WORK(&device->try_get_resynced_work, drbd_try_get_resynced_work_fn);
+
device->vdisk = disk;
device->rq_queue = disk->queue;
- set_disk_ro(disk, true);
-
disk->major = DRBD_MAJOR;
disk->first_minor = minor;
disk->minors = 1;
@@ -2704,12 +4402,39 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig
if (!device->md_io.page)
goto out_no_io_page;
- if (drbd_bm_init(device))
- goto out_no_bitmap;
+ /* Just put in some sane default; should never be used. */
+ device->last_bm_block_shift = BM_BLOCK_SHIFT_MIN;
+
+ spin_lock_init(&device->interval_lock);
device->read_requests = RB_ROOT;
- device->write_requests = RB_ROOT;
+ device->requests = RB_ROOT;
+
+ BUG_ON(!mutex_is_locked(&resource->conf_update));
+ for_each_connection(connection, resource) {
+ peer_device = create_peer_device(device, connection);
+ if (!peer_device)
+ goto out_no_peer_device;
+ list_add(&peer_device->peer_devices, &peer_devices);
+ }
+
+ /* Insert the new device into all idrs under state_rwlock write lock
+ to guarantee a consistent object model. idr_preload() doesn't help
+ because it can only guarantee that a single idr_alloc() will
+ succeed. This fails (and will be retried) if no memory is
+ immediately available.
+ Keep in mid that RCU readers might find the device in the moment
+ we add it to the resources->devices IDR!
+ */
+
+ INIT_LIST_HEAD(&device->peer_devices);
+ spin_lock_init(&device->pending_bmio_lock);
+ INIT_LIST_HEAD(&device->pending_bitmap_io);
- id = idr_alloc(&drbd_devices, device, minor, minor + 1, GFP_KERNEL);
+ locked = true;
+ write_lock_irq(&resource->state_rwlock);
+ spin_lock(&drbd_devices_lock);
+ id = idr_alloc(&drbd_devices, device, minor, minor + 1, GFP_NOWAIT);
+ spin_unlock(&drbd_devices_lock);
if (id < 0) {
if (id == -ENOSPC)
err = ERR_MINOR_OR_VOLUME_EXISTS;
@@ -2717,7 +4442,7 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig
}
kref_get(&device->kref);
- id = idr_alloc(&resource->devices, device, vnr, vnr + 1, GFP_KERNEL);
+ id = idr_alloc(&resource->devices, device, vnr, vnr + 1, GFP_NOWAIT);
if (id < 0) {
if (id == -ENOSPC)
err = ERR_MINOR_OR_VOLUME_EXISTS;
@@ -2725,70 +4450,82 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig
}
kref_get(&device->kref);
- INIT_LIST_HEAD(&device->peer_devices);
- INIT_LIST_HEAD(&device->pending_bitmap_io);
- for_each_connection(connection, resource) {
- peer_device = kzalloc_obj(struct drbd_peer_device);
- if (!peer_device)
- goto out_idr_remove_from_resource;
- peer_device->connection = connection;
- peer_device->device = device;
- ratelimit_state_init(&peer_device->ratelimit[D_RL_PD_GENERIC],
- 5 * HZ, /* no burst */ 1);
-
- list_add(&peer_device->peer_devices, &device->peer_devices);
+ list_for_each_entry_safe(peer_device, tmp_peer_device, &peer_devices, peer_devices) {
+ connection = peer_device->connection;
+ id = idr_alloc(&connection->peer_devices, peer_device,
+ device->vnr, device->vnr + 1, GFP_NOWAIT);
+ if (id < 0)
+ goto out_remove_peer_device;
+ list_del(&peer_device->peer_devices);
+ list_add_rcu(&peer_device->peer_devices, &device->peer_devices);
+ kref_get(&connection->kref);
kref_get(&device->kref);
+ }
+ write_unlock_irq(&resource->state_rwlock);
+ locked = false;
- id = idr_alloc(&connection->peer_devices, peer_device, vnr, vnr + 1, GFP_KERNEL);
- if (id < 0) {
- if (id == -ENOSPC)
- err = ERR_INVALID_REQUEST;
- goto out_idr_remove_from_resource;
- }
- kref_get(&connection->kref);
- INIT_WORK(&peer_device->send_acks_work, drbd_send_acks_wf);
+ if (init_conflict_submitter(device)) {
+ err = ERR_NOMEM;
+ goto out_remove_peer_device;
}
if (init_submitter(device)) {
err = ERR_NOMEM;
- goto out_idr_remove_from_resource;
+ goto out_remove_peer_device;
}
err = add_disk(disk);
if (err)
- goto out_destroy_workqueue;
-
- /* inherit the connection state */
- device->state.conn = first_connection(resource)->cstate;
- if (device->state.conn == C_WF_REPORT_PARAMS) {
- for_each_peer_device(peer_device, device)
+ goto out_destroy_submitter;
+ device->have_quorum[OLD] =
+ device->have_quorum[NEW] =
+ (resource->res_opts.quorum == QOU_OFF);
+ /* Real quorum, recomputed on the first state change. */
+ device->quorum[OLD] =
+ device->quorum[NEW] = false;
+
+ for_each_peer_device(peer_device, device) {
+ connection = peer_device->connection;
+ peer_device->node_id = connection->peer_node_id;
+
+ if (connection->cstate[NOW] >= C_CONNECTED)
drbd_connected(peer_device);
}
- /* move to create_peer_device() */
- for_each_peer_device(peer_device, device)
- drbd_debugfs_peer_device_add(peer_device);
+
drbd_debugfs_device_add(device);
+ *p_device = device;
return NO_ERROR;
-out_destroy_workqueue:
+out_destroy_submitter:
destroy_workqueue(device->submit.wq);
-out_idr_remove_from_resource:
- for_each_connection_safe(connection, n, resource) {
- peer_device = idr_remove(&connection->peer_devices, vnr);
- if (peer_device)
- kref_put(&connection->kref, drbd_destroy_connection);
- }
- for_each_peer_device_safe(peer_device, tmp_peer_device, device) {
+ device->submit.wq = NULL;
+out_remove_peer_device:
+ list_splice_init_rcu(&device->peer_devices, &tmp, synchronize_rcu);
+ list_for_each_entry_safe(peer_device, tmp_peer_device, &tmp, peer_devices) {
+ struct drbd_connection *connection = peer_device->connection;
+
+ idr_remove(&connection->peer_devices, device->vnr);
list_del(&peer_device->peer_devices);
kfree(peer_device);
+ kref_put(&connection->kref, drbd_destroy_connection);
}
idr_remove(&resource->devices, vnr);
+
out_idr_remove_minor:
+ spin_lock(&drbd_devices_lock);
idr_remove(&drbd_devices, minor);
- synchronize_rcu();
+ spin_unlock(&drbd_devices_lock);
out_no_minor_idr:
- drbd_bm_cleanup(device);
-out_no_bitmap:
+ if (locked)
+ write_unlock_irq(&resource->state_rwlock);
+ synchronize_rcu();
+
+out_no_peer_device:
+ list_for_each_entry_safe(peer_device, tmp_peer_device, &peer_devices, peer_devices) {
+ list_del(&peer_device->peer_devices);
+ kfree(peer_device);
+ }
+
__free_page(device->md_io.page);
out_no_io_page:
put_disk(disk);
@@ -2798,34 +4535,135 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig
return err;
}
-void drbd_delete_device(struct drbd_device *device)
+/**
+ * drbd_unregister_device() - make a device "invisible"
+ * @device: DRBD device to unregister
+ *
+ * Remove the device from the drbd object model and unregister it in the
+ * kernel. Keep reference counts on device->kref; they are dropped in
+ * drbd_reclaim_device().
+ */
+void drbd_unregister_device(struct drbd_device *device)
{
struct drbd_resource *resource = device->resource;
struct drbd_connection *connection;
struct drbd_peer_device *peer_device;
- /* move to free_peer_device() */
- for_each_peer_device(peer_device, device)
- drbd_debugfs_peer_device_cleanup(peer_device);
- drbd_debugfs_device_cleanup(device);
+ write_lock_irq(&resource->state_rwlock);
for_each_connection(connection, resource) {
idr_remove(&connection->peer_devices, device->vnr);
- kref_put(&device->kref, drbd_destroy_device);
}
idr_remove(&resource->devices, device->vnr);
- kref_put(&device->kref, drbd_destroy_device);
- idr_remove(&drbd_devices, device_to_minor(device));
- kref_put(&device->kref, drbd_destroy_device);
+ spin_lock(&drbd_devices_lock);
+ idr_remove(&drbd_devices, device->minor);
+ spin_unlock(&drbd_devices_lock);
+ write_unlock_irq(&resource->state_rwlock);
+
+ for_each_peer_device(peer_device, device)
+ drbd_debugfs_peer_device_cleanup(peer_device);
+ drbd_debugfs_device_cleanup(device);
del_gendisk(device->vdisk);
- synchronize_rcu();
- kref_put(&device->kref, drbd_destroy_device);
+
+ destroy_workqueue(device->submit_conflict.wq);
+ device->submit_conflict.wq = NULL;
+ destroy_workqueue(device->submit.wq);
+ device->submit.wq = NULL;
+ timer_shutdown_sync(&device->request_timer);
+}
+
+void drbd_reclaim_device(struct rcu_head *rp)
+{
+ struct drbd_device *device = container_of(rp, struct drbd_device, rcu);
+ struct drbd_peer_device *peer_device;
+ int i;
+
+ for_each_peer_device(peer_device, device) {
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+
+ for (i = 0; i < 3; i++) {
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+}
+
+static void shutdown_connect_timer(struct drbd_connection *connection)
+{
+ if (timer_shutdown_sync(&connection->connect_timer)) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+}
+
+void del_connect_timer(struct drbd_connection *connection)
+{
+ if (timer_delete_sync(&connection->connect_timer)) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+}
+
+/**
+ * drbd_unregister_connection() - make a connection "invisible"
+ * @connection: DRBD connection to unregister
+ *
+ * Remove the connection from the drbd object model. Keep reference counts on
+ * connection->kref; they are dropped in drbd_reclaim_connection().
+ */
+void drbd_unregister_connection(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ int vnr, rr;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ drbd_debugfs_peer_device_cleanup(peer_device);
+
+ write_lock_irq(&resource->state_rwlock);
+ set_bit(C_UNREGISTERED, &connection->flags);
+ smp_wmb();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ list_del_rcu(&peer_device->peer_devices);
+ list_del_rcu(&connection->connections);
+ write_unlock_irq(&resource->state_rwlock);
+
+ drbd_debugfs_connection_cleanup(connection);
+
+ shutdown_connect_timer(connection);
+
+ rr = drbd_free_peer_reqs(connection, &connection->done_ee);
+ if (rr)
+ drbd_err(connection, "%d EEs in done list found!\n", rr);
+
+ drbd_transport_shutdown(connection, DESTROY_TRANSPORT);
+ drbd_put_send_buffers(connection);
+ conn_free_crypto(connection);
+}
+
+void drbd_reclaim_connection(struct rcu_head *rp)
+{
+ struct drbd_connection *connection =
+ container_of(rp, struct drbd_connection, rcu);
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+ kref_put(&connection->kref, drbd_destroy_connection);
+}
+
+void drbd_reclaim_path(struct rcu_head *rp)
+{
+ struct drbd_path *path = container_of(rp, struct drbd_path, rcu);
+
+ INIT_LIST_HEAD(&path->list);
+ kref_put(&path->kref, drbd_destroy_path);
}
static int __init drbd_init(void)
{
int err;
- if (drbd_minor_count < DRBD_MINOR_COUNT_MIN || drbd_minor_count > DRBD_MINOR_COUNT_MAX) {
+ if (drbd_minor_count < DRBD_MINOR_COUNT_MIN
+ || drbd_minor_count > DRBD_MINOR_COUNT_MAX) {
pr_err("invalid minor_count (%d)\n", drbd_minor_count);
#ifdef MODULE
return -EINVAL;
@@ -2841,24 +4679,44 @@ static int __init drbd_init(void)
return err;
}
+ /*
+ * allocate all necessary structs
+ */
drbd_proc = NULL; /* play safe for drbd_cleanup */
idr_init(&drbd_devices);
- mutex_init(&resources_mutex);
- INIT_LIST_HEAD(&drbd_resources);
+ err = register_pernet_device(&drbd_pernet_ops);
+ if (err) {
+ pr_err("unable to register net namespace handlers\n");
+ goto fail;
+ }
- err = genl_register_family(&drbd_nl_family);
+ err = drbd_nl_legacy_init();
if (err) {
pr_err("unable to register generic netlink family\n");
goto fail;
}
+ err = drbd_nl_drbd2_init();
+ if (err) {
+ pr_err("unable to register the drbd2 generic netlink family\n");
+ goto fail;
+ }
+
+ err = -ENOMEM;
+ ping_ack_sender = alloc_workqueue("drbd_pas",
+ WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
+ if (!ping_ack_sender)
+ goto fail;
+
err = drbd_create_mempools();
if (err)
goto fail;
err = -ENOMEM;
- drbd_proc = proc_create_single("drbd", S_IFREG | 0444 , NULL, drbd_seq_show);
+ drbd_proc = proc_create_single("drbd", S_IFREG | 0444, NULL,
+ drbd_seq_show);
+
if (!drbd_proc) {
pr_err("unable to register proc file\n");
goto fail;
@@ -2879,7 +4737,15 @@ static int __init drbd_init(void)
"Version: " REL_VERSION " (api:%d/proto:%d-%d)\n",
DRBD_FAMILY_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX);
pr_info("%s\n", drbd_buildtag());
+#if defined(CONFIG_DRBD_COMPAT_84) && !defined(DRBD_NL_FAMILY_V2)
+ pr_info("serving the DRBD 8.4 netlink API, advertised as version " DRBD_PROC_VERSION "\n");
+#endif
pr_info("registered as block device major %d\n", DRBD_MAJOR);
+
+#ifdef CONFIG_DRBD_COMPAT_84
+ atomic_set(&nr_drbd8_devices, 0);
+#endif
+
return 0; /* Success! */
fail:
@@ -2891,584 +4757,1758 @@ static int __init drbd_init(void)
return err;
}
-static void drbd_free_one_sock(struct drbd_socket *ds)
-{
- struct socket *s;
- mutex_lock(&ds->mutex);
- s = ds->socket;
- ds->socket = NULL;
- mutex_unlock(&ds->mutex);
- if (s) {
- /* so debugfs does not need to mutex_lock() */
- synchronize_rcu();
- kernel_sock_shutdown(s, SHUT_RDWR);
- sock_release(s);
- }
-}
-
-void drbd_free_sock(struct drbd_connection *connection)
-{
- if (connection->data.socket)
- drbd_free_one_sock(&connection->data);
- if (connection->meta.socket)
- drbd_free_one_sock(&connection->meta);
-}
-
/* meta data management */
-void conn_md_sync(struct drbd_connection *connection)
+static
+void drbd_md_encode_9(struct drbd_device *device, struct meta_data_on_disk_9 *buffer)
{
- struct drbd_peer_device *peer_device;
- int vnr;
+ int i;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
+ buffer->effective_size = cpu_to_be64(device->ldev->md.effective_size);
+ buffer->current_uuid = cpu_to_be64(device->ldev->md.current_uuid);
+ buffer->members = cpu_to_be64(device->ldev->md.members);
+ buffer->features = cpu_to_be64(DRBD_MD_FEATURES);
+ buffer->flags = cpu_to_be32(device->ldev->md.flags);
+ buffer->magic = cpu_to_be32(DRBD_MD_MAGIC_09);
- kref_get(&device->kref);
- rcu_read_unlock();
- drbd_md_sync(device);
- kref_put(&device->kref, drbd_destroy_device);
- rcu_read_lock();
- }
- rcu_read_unlock();
-}
+ buffer->md_size_sect = cpu_to_be32(device->ldev->md.md_size_sect);
+ buffer->al_offset = cpu_to_be32(device->ldev->md.al_offset);
+ buffer->al_nr_extents = cpu_to_be32(device->act_log->nr_elements);
+ buffer->bm_bytes_per_bit = cpu_to_be32(device->ldev->md.bm_block_size);
+ buffer->device_uuid = cpu_to_be64(device->ldev->md.device_uuid);
-/* aligned 4kByte */
-struct meta_data_on_disk {
- u64 la_size_sect; /* last agreed size. */
- u64 uuid[UI_SIZE]; /* UUIDs. */
- u64 device_uuid;
- u64 reserved_u64_1;
- u32 flags; /* MDF */
- u32 magic;
- u32 md_size_sect;
- u32 al_offset; /* offset to this block */
- u32 al_nr_extents; /* important for restoring the AL (userspace) */
- /* `-- act_log->nr_elements <-- ldev->dc.al_extents */
- u32 bm_offset; /* offset to the bitmap, from here */
- u32 bm_bytes_per_bit; /* BM_BLOCK_SIZE */
- u32 la_peer_max_bio_size; /* last peer max_bio_size */
+ buffer->bm_offset = cpu_to_be32(device->ldev->md.bm_offset);
+ buffer->la_peer_max_bio_size = cpu_to_be32(device->device_conf.max_bio_size);
+ buffer->bm_max_peers = cpu_to_be32(device->ldev->md.max_peers);
+ buffer->node_id = cpu_to_be32(device->ldev->md.node_id);
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[i];
+
+ buffer->peers[i].bitmap_uuid = cpu_to_be64(peer_md->bitmap_uuid);
+ buffer->peers[i].bitmap_dagtag = cpu_to_be64(peer_md->bitmap_dagtag);
+ buffer->peers[i].flags = cpu_to_be32(peer_md->flags & ~MDF_HAVE_BITMAP);
+ buffer->peers[i].bitmap_index = cpu_to_be32(peer_md->bitmap_index);
+ }
+ BUILD_BUG_ON(ARRAY_SIZE(device->ldev->md.history_uuids) != ARRAY_SIZE(buffer->history_uuids));
+ for (i = 0; i < ARRAY_SIZE(buffer->history_uuids); i++)
+ buffer->history_uuids[i] = cpu_to_be64(device->ldev->md.history_uuids[i]);
- /* see al_tr_number_to_on_disk_sector() */
- u32 al_stripes;
- u32 al_stripe_size_4k;
+ buffer->al_stripes = cpu_to_be32(device->ldev->md.al_stripes);
+ buffer->al_stripe_size_4k = cpu_to_be32(device->ldev->md.al_stripe_size_4k);
- u8 reserved_u8[4096 - (7*8 + 10*4)];
-} __packed;
+ buffer->cbt_bookmark.id = cpu_to_be64(device->ldev->md.cbt.id);
+ buffer->cbt_bookmark.bitmap_uuid = cpu_to_be64(device->ldev->md.cbt.bitmap_uuid);
+ buffer->cbt_bookmark.bitmap_dagtag = cpu_to_be64(device->ldev->md.cbt.bitmap_dagtag);
+ buffer->cbt_bookmark.since = cpu_to_be64(device->ldev->md.cbt.since);
+ buffer->cbt_bookmark.flags = cpu_to_be32(device->ldev->md.cbt.flags);
+ buffer->cbt_bookmark.bitmap_index = cpu_to_be32(device->ldev->md.cbt.bitmap_index);
+ if (device->bitmap == NULL)
+ for (i = 0; i < DRBD_PEERS_MAX; i++)
+ buffer->peers[i].flags |= cpu_to_be32(MDF_PEER_FULL_SYNC);
+}
+static void drbd_md_encode(struct drbd_device *device, void *buffer)
+{
+ if (test_bit(LEGACY_84_MD, &device->flags))
+ drbd_md_encode_84(device, buffer);
+ else
+ drbd_md_encode_9(device, buffer);
+}
-void drbd_md_write(struct drbd_device *device, void *b)
+int drbd_md_write(struct drbd_device *device, struct meta_data_on_disk_9 *buffer)
{
- struct meta_data_on_disk *buffer = b;
sector_t sector;
- int i;
+ int err;
memset(buffer, 0, sizeof(*buffer));
- buffer->la_size_sect = cpu_to_be64(get_capacity(device->vdisk));
- for (i = UI_CURRENT; i < UI_SIZE; i++)
- buffer->uuid[i] = cpu_to_be64(device->ldev->md.uuid[i]);
- buffer->flags = cpu_to_be32(device->ldev->md.flags);
- buffer->magic = cpu_to_be32(DRBD_MD_MAGIC_84_UNCLEAN);
-
- buffer->md_size_sect = cpu_to_be32(device->ldev->md.md_size_sect);
- buffer->al_offset = cpu_to_be32(device->ldev->md.al_offset);
- buffer->al_nr_extents = cpu_to_be32(device->act_log->nr_elements);
- buffer->bm_bytes_per_bit = cpu_to_be32(BM_BLOCK_SIZE);
- buffer->device_uuid = cpu_to_be64(device->ldev->md.device_uuid);
+ drbd_md_encode(device, buffer);
- buffer->bm_offset = cpu_to_be32(device->ldev->md.bm_offset);
- buffer->la_peer_max_bio_size = cpu_to_be32(device->peer_max_bio_size);
+ if (drbd_md_dax_active(device->ldev)) {
+ struct meta_data_on_disk_9 *md_on_pmem = drbd_dax_md_addr(device->ldev);
- buffer->al_stripes = cpu_to_be32(device->ldev->md.al_stripes);
- buffer->al_stripe_size_4k = cpu_to_be32(device->ldev->md.al_stripe_size_4k);
+ /* Copy the encoded super block, rather than encoding into
+ * persistent memory directly. That way the reserved fields are
+ * zeroed here as well, so that a field which a later version
+ * places there is cleared by this version. Copying also leaves
+ * every byte at either its old or its new value, whereas
+ * zeroing in place would make the meta data unreadable should a
+ * crash hit that window.
+ */
+ memcpy(md_on_pmem, buffer, sizeof(*buffer));
+ arch_wb_cache_pmem(md_on_pmem, sizeof(*buffer));
+ return 0;
+ }
D_ASSERT(device, drbd_md_ss(device->ldev) == device->ldev->md.md_offset);
sector = device->ldev->md.md_offset;
- if (drbd_md_sync_page_io(device, device->ldev, sector, REQ_OP_WRITE)) {
- /* this was a try anyways ... */
+ err = drbd_md_sync_page_io(device, device->ldev, sector, REQ_OP_WRITE);
+ if (err) {
drbd_err(device, "meta data update failed!\n");
- drbd_chk_io_error(device, 1, DRBD_META_IO_ERROR);
+ drbd_handle_io_error(device, DRBD_META_IO_ERROR);
}
+
+ return err;
}
/**
- * drbd_md_sync() - Writes the meta data super block if the MD_DIRTY flag bit is set
+ * __drbd_md_sync() - Writes the meta data super block (conditionally) if the MD_DIRTY flag bit is set
* @device: DRBD device.
+ * @maybe: meta data may in fact be "clean", the actual write may be skipped.
*/
-void drbd_md_sync(struct drbd_device *device)
+static int __drbd_md_sync(struct drbd_device *device, bool maybe)
{
- struct meta_data_on_disk *buffer;
+ struct meta_data_on_disk_9 *buffer;
+ int err = -EIO;
/* Don't accidentally change the DRBD meta data layout. */
- BUILD_BUG_ON(UI_SIZE != 4);
- BUILD_BUG_ON(sizeof(struct meta_data_on_disk) != 4096);
+ BUILD_BUG_ON(DRBD_PEERS_MAX != 32);
+ BUILD_BUG_ON(HISTORY_UUIDS != 32);
+ BUILD_BUG_ON(sizeof(struct meta_data_on_disk_9) != 4096);
+
+ if (!get_ldev_if_state(device, D_DETACHING))
+ return -EIO;
+
+ /*
+ * A disk that failed may take this write or not, by chance. Keep the
+ * last superblock it did take rather than let that chance decide.
+ */
+ if (device->disk_state[NOW] == D_FAILED ||
+ (device->disk_state[NOW] == D_DETACHING &&
+ test_bit(FORCE_DETACH, &device->flags))) {
+ timer_delete(&device->md_sync_timer);
+ goto out;
+ }
+
+ buffer = drbd_md_get_buffer(device, __func__);
+ if (!buffer)
+ goto out;
timer_delete(&device->md_sync_timer);
/* timer may be rearmed by drbd_md_mark_dirty() now. */
- if (!test_and_clear_bit(MD_DIRTY, &device->flags))
+
+ if (test_and_clear_bit(MD_DIRTY, &device->flags) || !maybe) {
+ err = drbd_md_write(device, buffer);
+ if (err)
+ set_bit(MD_DIRTY, &device->flags);
+ }
+
+ drbd_md_put_buffer(device);
+out:
+ put_ldev(device);
+
+ return err;
+}
+
+int drbd_md_sync(struct drbd_device *device)
+{
+ return __drbd_md_sync(device, false);
+}
+
+int drbd_md_sync_if_dirty(struct drbd_device *device)
+{
+ return __drbd_md_sync(device, true);
+}
+
+/**
+ * drbd_md_mark_dirty() - Mark meta data super block as dirty
+ * @device: DRBD device.
+ *
+ * Call this function if you change anything that should be written to
+ * the meta-data super block. This function sets MD_DIRTY, and starts a
+ * timer that ensures that within five seconds you have to call drbd_md_sync().
+ */
+void drbd_md_mark_dirty(struct drbd_device *device)
+{
+ if (!test_and_set_bit(MD_DIRTY, &device->flags))
+ mod_timer(&device->md_sync_timer, jiffies + 5*HZ);
+}
+
+void _drbd_uuid_push_history(struct drbd_device *device, u64 val)
+{
+ struct drbd_md *md = &device->ldev->md;
+ int node_id, i;
+
+ if (val == UUID_JUST_CREATED || val == 0)
return;
- /* We use here D_FAILED and not D_ATTACHING because we try to write
- * metadata even if we detach due to a disk failure! */
- if (!get_ldev_if_state(device, D_FAILED))
+ val &= ~UUID_PRIMARY;
+
+ if (val == (md->current_uuid & ~UUID_PRIMARY))
return;
- buffer = drbd_md_get_buffer(device, __func__);
- if (!buffer)
- goto out;
+ /* day0 UUID: keep in history even if also held in a bitmap slot */
+ if (md->history_uuids[0] == 0) {
+ md->history_uuids[0] = val;
+ return;
+ }
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (node_id == md->node_id)
+ continue;
+ if (val == (md->peers[node_id].bitmap_uuid & ~UUID_PRIMARY))
+ return;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(md->history_uuids); i++) {
+ if (md->history_uuids[i] == val)
+ return;
+ }
+
+ for (i = ARRAY_SIZE(md->history_uuids) - 1; i > 0; i--)
+ md->history_uuids[i] = md->history_uuids[i - 1];
+ md->history_uuids[i] = val;
+}
+
+u64 _drbd_uuid_pull_history(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_md *md = &device->ldev->md;
+ u64 first_history_uuid;
+ int i;
+
+ first_history_uuid = md->history_uuids[0];
+ for (i = 0; i < ARRAY_SIZE(md->history_uuids) - 1; i++)
+ md->history_uuids[i] = md->history_uuids[i + 1];
+ md->history_uuids[i] = 0;
+
+ return first_history_uuid;
+}
+
+static void __drbd_uuid_set_current(struct drbd_device *device, u64 val)
+{
+ drbd_md_mark_dirty(device);
+ if (device->resource->role[NOW] == R_PRIMARY)
+ val |= UUID_PRIMARY;
+ else
+ val &= ~UUID_PRIMARY;
+
+ device->ldev->md.current_uuid = val;
+ drbd_uuid_set_exposed(device, val, false);
+}
+
+/* Assign a peer's bitmap UUID, with its dagtag and MDF_PEER_DIVERGENCE_BITMAP.
+ * A bitmap_uuid of 0 means the bitmap is unused.
+ */
+void drbd_set_peer_bitmap_uuid(struct drbd_peer_md *peer_md, u64 bitmap_uuid, u64 dagtag)
+{
+ u64 previous = peer_md->bitmap_uuid;
+
+ peer_md->bitmap_uuid = bitmap_uuid;
+ peer_md->bitmap_dagtag = bitmap_uuid ? dagtag : 0;
+ /* Mark it a divergence bitmap only when we begin tracking a fresh one
+ * (0 -> non-0). A non-0 -> non-0 re-assignment rebases a bitmap that a
+ * resync already owns onto a new data generation; that bitmap is a
+ * convergence bitmap and its flag, cleared at resync start, must stay
+ * cleared. This relies on a resync's bitmap UUID being non-zero
+ * throughout -- see drbd_run_resync().
+ */
+ if (bitmap_uuid && !previous)
+ set_bit(__MDF_PEER_DIVERGENCE_BITMAP, &peer_md->flags);
+ else if (!bitmap_uuid)
+ clear_bit(__MDF_PEER_DIVERGENCE_BITMAP, &peer_md->flags);
+}
+
+/**
+ * drbd_day0_node_id() - Pick the day0 slot that stands for all of them
+ * @md: the meta data
+ * @exclude: a node id not to pick, or -1
+ *
+ * All day0 slots should hold the same UUID, but meta data can hold day0 slots
+ * that disagree. The sender of our UUIDs advertises only one of them, so the
+ * search for a bitmap to copy and forget-peer must use that same one, or the
+ * two ends of a handshake decide differently.
+ *
+ * When they disagree, take the UUID most of them hold: disagreeing slots belong
+ * to the few node ids that were used and forgotten, while the node ids never
+ * used keep the day0 UUID. Between UUIDs held by equally many slots, take the
+ * one set at the lower dagtag, the older one.
+ *
+ * Returns the node id of a day0 slot holding that UUID, else of one with
+ * bitmap UUID 0, else -1.
+ */
+int drbd_day0_node_id(struct drbd_md *md, int exclude)
+{
+ int node_id, other, best = -1, best_count = 0, unset = -1;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ u64 uuid = md->peers[node_id].bitmap_uuid & ~UUID_PRIMARY;
+ int count = 0;
+
+ if (node_id == exclude || !drbd_is_day0_slot(md, node_id))
+ continue;
+ if (!uuid) {
+ if (unset == -1)
+ unset = node_id;
+ continue;
+ }
+ for (other = 0; other < DRBD_NODE_ID_MAX; other++) {
+ if (drbd_is_day0_slot(md, other) &&
+ (md->peers[other].bitmap_uuid & ~UUID_PRIMARY) == uuid)
+ count++;
+ }
+ if (best == -1 || count > best_count ||
+ (count == best_count &&
+ md->peers[node_id].bitmap_dagtag < md->peers[best].bitmap_dagtag)) {
+ best = node_id;
+ best_count = count;
+ }
+ }
+ return best != -1 ? best : unset;
+}
+
+void __drbd_uuid_set_bitmap(struct drbd_peer_device *peer_device, u64 val)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[peer_device->node_id];
+
+ drbd_md_mark_dirty(device);
+ drbd_set_peer_bitmap_uuid(peer_md, val, device->resource->dagtag_sector);
+}
+
+void _drbd_uuid_set_current(struct drbd_device *device, u64 val)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
+ __drbd_uuid_set_current(device, val);
+ spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
+}
+
+void _drbd_uuid_set_bitmap(struct drbd_peer_device *peer_device, u64 val)
+{
+ struct drbd_device *device = peer_device->device;
+ unsigned long flags;
+
+ down_write(&device->uuid_sem);
+ spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
+ __drbd_uuid_set_bitmap(peer_device, val);
+ spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
+ up_write(&device->uuid_sem);
+}
+
+/* call holding down_write(uuid_sem) */
+void drbd_uuid_set_bitmap(struct drbd_peer_device *peer_device, u64 uuid)
+{
+ struct drbd_device *device = peer_device->device;
+ unsigned long flags;
+ u64 previous_uuid;
+
+ spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
+ previous_uuid = drbd_bitmap_uuid(peer_device);
+ __drbd_uuid_set_bitmap(peer_device, uuid);
+ if (previous_uuid)
+ _drbd_uuid_push_history(device, previous_uuid);
+ spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
+}
+
+/**
+ * drbd_uuid_is_day0() - Check if device is in "day0" UUID state
+ * @device: DRBD device (caller must hold ldev reference)
+ *
+ * Returns true if the current UUID appears to be a "day0" UUID:
+ * a real UUID value was set (e.g. by linstor during create-md),
+ * but no UUID rotation has ever happened (all history and bitmap
+ * UUIDs are still zero).
+ *
+ * The same notion of day0 as a day0 peer slot (see struct drbd_peer_md), seen
+ * differently: here, the local current UUID is still day0; there, a bitmap
+ * tracks divergence since day0.
+ */
+bool drbd_uuid_is_day0(struct drbd_device *device)
+{
+ struct drbd_md *md = &device->ldev->md;
+ int i;
+
+ if ((md->current_uuid & ~UUID_PRIMARY) == UUID_JUST_CREATED ||
+ md->current_uuid == 0)
+ return false;
+
+ for (i = 0; i < ARRAY_SIZE(md->history_uuids); i++)
+ if (md->history_uuids[i])
+ return false;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ if (i == md->node_id)
+ continue;
+ if (md->peers[i].bitmap_uuid)
+ return false;
+ }
+
+ return true;
+}
+
+static u64 rotate_current_into_bitmap(struct drbd_device *device, u64 weak_nodes, u64 dagtag)
+{
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ struct drbd_peer_device *peer_device;
+ int node_id;
+ u64 bm_uuid, prev_c_uuid;
+ u64 node_mask = 0; /* bit mask of node-ids processed */
+ u64 slot_mask = 0; /* bit mask of on-disk bitmap slots processed */
+ /* return value, bit mask of node-ids for which we
+ * actually set a new bitmap uuid */
+ u64 got_new_bitmap_uuid = 0;
+
+ if (device->ldev->md.current_uuid != UUID_JUST_CREATED) {
+ prev_c_uuid = device->ldev->md.current_uuid;
+ } else {
+ prev_c_uuid = get_random_u64();
+ drbd_info(device, "Bitmap UUID for initial resync: %016llX\n",
+ (unsigned long long)prev_c_uuid);
+ }
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_disk_state pdsk;
+
+ node_id = peer_device->node_id;
+ node_mask |= NODE_MASK(node_id);
+ if (peer_device->bitmap_index != -1)
+ __set_bit(peer_device->bitmap_index, (unsigned long *)&slot_mask);
+ bm_uuid = peer_md[node_id].bitmap_uuid;
+ if (bm_uuid && bm_uuid != prev_c_uuid)
+ continue;
+
+ /* Defensive: a sync source's bitmap is a convergence bitmap, so
+ * never establish a divergence bitmap on it.
+ */
+ if (bm_uuid == 0 && is_sync_source_state(peer_device, NOW))
+ continue;
+
+ pdsk = peer_device->disk_state[NOW];
+
+ /* Create a new current UUID for a peer that is diskless but usually has a backing disk.
+ * Do not create a new current UUID for a CONNECTED intentional diskless peer.
+ * Create one for an intentional diskless peer that is currently away. */
+ if (pdsk == D_DISKLESS && !test_bit(__MDF_HAVE_BITMAP, &peer_md[node_id].flags))
+ continue;
+
+ if ((pdsk <= D_UNKNOWN && pdsk != D_NEGOTIATING) ||
+ (NODE_MASK(node_id) & weak_nodes)) {
+ drbd_set_peer_bitmap_uuid(&peer_md[node_id], prev_c_uuid, dagtag);
+ drbd_md_mark_dirty(device);
+ got_new_bitmap_uuid |= NODE_MASK(node_id);
+ }
+ }
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ int slot_nr;
+
+ if (node_id == device->ldev->md.node_id)
+ continue;
+ if (node_mask & NODE_MASK(node_id))
+ continue;
+ slot_nr = peer_md[node_id].bitmap_index;
+ if (slot_nr != -1) {
+ if (test_bit(slot_nr, (unsigned long *)&slot_mask))
+ continue;
+ __set_bit(slot_nr, (unsigned long *)&slot_mask);
+ }
+ bm_uuid = peer_md[node_id].bitmap_uuid;
+ if (bm_uuid && bm_uuid != prev_c_uuid)
+ continue;
+ if (slot_nr == -1) {
+ slot_nr = find_first_zero_bit((unsigned long *)&slot_mask, sizeof(slot_mask) * BITS_PER_BYTE);
+ __set_bit(slot_nr, (unsigned long *)&slot_mask);
+ /* A slot without a bitmap needs no divergence flag, and a
+ * day0 slot must keep flags == 0.
+ */
+ peer_md[node_id].bitmap_uuid = prev_c_uuid;
+ peer_md[node_id].bitmap_dagtag = dagtag;
+ } else {
+ drbd_set_peer_bitmap_uuid(&peer_md[node_id], prev_c_uuid, dagtag);
+ }
+ drbd_md_mark_dirty(device);
+ /* count, but only if that bitmap index exists. */
+ if (slot_nr < device->ldev->md.max_peers)
+ got_new_bitmap_uuid |= NODE_MASK(node_id);
+ }
+ rcu_read_unlock();
+
+ return got_new_bitmap_uuid;
+}
+
+static u64 initial_resync_nodes(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ u64 nodes = 0;
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->disk_state[NOW] == D_INCONSISTENT &&
+ peer_device->repl_state[NOW] == L_ESTABLISHED)
+ nodes |= NODE_MASK(peer_device->node_id);
+ }
+
+ return nodes;
+}
+
+u64 drbd_weak_nodes_device(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ u64 not_weak = 0;
+
+ if (device->disk_state[NOW] == D_UP_TO_DATE)
+ not_weak = NODE_MASK(device->resource->res_opts.node_id);
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_disk_state pdsk = peer_device->disk_state[NOW];
+
+ if (!(pdsk <= D_FAILED || pdsk == D_UNKNOWN || pdsk == D_OUTDATED))
+ not_weak |= NODE_MASK(peer_device->node_id);
+
+ }
+ rcu_read_unlock();
+
+ return ~not_weak;
+}
+
+
+static enum drbd_mint_outcome __new_current_uuid_prepare(struct drbd_device *device, bool forced)
+{
+ u64 got_new_bitmap_uuid, val, old_current_uuid;
+ bool day0;
+ int err;
+
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+ day0 = drbd_uuid_is_day0(device);
+ got_new_bitmap_uuid = rotate_current_into_bitmap(device,
+ forced ? initial_resync_nodes(device) : 0,
+ device->resource->dagtag_sector);
+
+ /* No slot to rotate the current UUID into, and not the first bump:
+ * every absent peer already sees a current UUID it does not have, so
+ * the divergence is recorded and a further one records nothing.
+ */
+ if (!got_new_bitmap_uuid && !day0) {
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+ return MINT_UNNECESSARY;
+ }
+
+ old_current_uuid = device->ldev->md.current_uuid;
+ val = get_random_u64();
+ __drbd_uuid_set_current(device, val);
+ /* first bump: retire day0 into history (after __drbd_uuid_set_current) */
+ if (day0)
+ _drbd_uuid_push_history(device, old_current_uuid);
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+
+ /* get it to stable storage _now_ */
+ err = drbd_md_sync(device);
+ if (err) {
+ _drbd_uuid_set_current(device, old_current_uuid);
+ return MINT_FAILED;
+ }
+
+ return MINT_MINTED;
+}
+
+static void __new_current_uuid_info(struct drbd_device *device, u64 weak_nodes)
+{
+ drbd_info(device, "new current UUID: %016llX weak: %016llX\n",
+ device->ldev->md.current_uuid, weak_nodes);
+}
+
+static void __new_current_uuid_send(struct drbd_device *device, u64 weak_nodes, bool forced)
+{
+ struct drbd_peer_device *peer_device;
+ u64 im;
+
+ for_each_peer_device_ref(peer_device, im, device) {
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED)
+ drbd_send_uuids(peer_device, forced ? 0 : UUID_FLAG_NEW_DATAGEN, weak_nodes);
+ }
+}
+
+static enum drbd_mint_outcome __drbd_uuid_new_current_send(struct drbd_device *device, bool forced)
+{
+ enum drbd_mint_outcome outcome;
+ u64 weak_nodes;
+
+ down_write(&device->uuid_sem);
+ outcome = __new_current_uuid_prepare(device, forced);
+ if (outcome != MINT_MINTED) {
+ up_write(&device->uuid_sem);
+ return outcome;
+ }
+ downgrade_write(&device->uuid_sem);
+ /* New data generation: separate pre- and post-bump writes into distinct
+ * write epochs (see the diskless path in drbd_uuid_new_current).
+ */
+ start_new_tl_epoch(device->resource);
+ weak_nodes = drbd_weak_nodes_device(device);
+ __new_current_uuid_info(device, weak_nodes);
+ __new_current_uuid_send(device, weak_nodes, forced);
+ up_read(&device->uuid_sem);
+
+ return MINT_MINTED;
+}
+
+static void __drbd_uuid_new_current_holding_uuid_sem(struct drbd_device *device)
+{
+ u64 weak_nodes;
+
+ if (__new_current_uuid_prepare(device, false) != MINT_MINTED)
+ return;
+ weak_nodes = drbd_weak_nodes_device(device);
+ __new_current_uuid_info(device, weak_nodes);
+}
+
+static bool peer_can_fill_a_bitmap_slot(struct drbd_peer_device *peer_device,
+ bool intentional_diskless)
+{
+ struct drbd_device *device = peer_device->device;
+ const int my_node_id = device->resource->res_opts.node_id;
+ int node_id;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_device *p2;
+
+ if (node_id == peer_device->node_id)
+ continue;
+ if (peer_device->bitmap_uuids[node_id] != 0)
+ continue;
+ /* bitmap_uuids[my_node_id] is the peer's record of *us*, from the
+ * handshake. When intentionally diskless (bitmap=no) the peer
+ * never rotates into it, so it is perpetually zero and never
+ * fillable -- skip it. A detached (bitmap=yes) primary keeps
+ * MDF_HAVE_BITMAP on the peer, so the slot self-corrects after a
+ * bump and a zero there is a real signal of divergence to resync
+ * on reattach -- do not skip it.
+ */
+ if (node_id == my_node_id && intentional_diskless)
+ continue;
+ p2 = peer_device_by_node_id(device, node_id);
+ /* Diskless peers (bitmap=no) have no bitmap slot tracking
+ * changes; rotate_current_into_bitmap() skips them.
+ */
+ if (p2 && !want_bitmap(p2))
+ continue;
+ return true;
+ }
+
+ return false;
+}
+
+/* For an intentionally-diskless primary (bitmap=no): returns true if the peer
+ * reports no prior bump -- i.e. the very first write session, or a prior
+ * bump's entries have since been cleared by resync -- so a new generation must
+ * be signalled. We scan peer_device->bitmap_uuids[n], the connected peer's
+ * peer_md[n].bitmap_uuid from the handshake; holding no disk, that is all we
+ * have.
+ *
+ * The peer's record of *us* (bitmap_uuids[my_node_id]) is useless here: the
+ * peer has no MDF_HAVE_BITMAP for us, so it is perpetually zero and the slot
+ * scan in peer_can_fill_a_bitmap_slot() skips it, missing the first-write /
+ * 2-node case. (A merely-detached primary keeps MDF_HAVE_BITMAP on the peer,
+ * so that slot self-corrects and the slot scan handles it without this helper.)
+ * We instead detect prior bumps through THIRD PARTY configured peers
+ * (n != my_node_id, n != peer_device->node_id): rotate_current_into_bitmap()
+ * on the receiving peer records the old current UUID in each absent or weak
+ * peer's slot.
+ *
+ * Once such entries are present this returns false, and the bump path calls
+ * a_lost_peer_is_on_same_cur_uuid() to handle an absent peer's current UUID
+ * being lost after a restart.
+ */
+static bool diskless_primary_needs_uuid_bump(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ const int my_node_id = device->resource->res_opts.node_id;
+ int n;
+
+ for (n = 0; n < DRBD_NODE_ID_MAX; n++) {
+ struct drbd_peer_device *pn;
+
+ if (n == peer_device->node_id || n == my_node_id)
+ continue;
+ /* Skip peer_md entries for node IDs not in the current cluster
+ * configuration. When a peer is removed from a cluster its
+ * peer_md[n].bitmap_uuid on remaining nodes is preserved for
+ * potential re-join, so a non-zero entry there records a bump
+ * from when that peer was still a member -- not evidence that a
+ * bump is needed now.
+ */
+ pn = peer_device_by_node_id(device, n);
+ if (!pn)
+ continue;
+ /* A bitmap_uuid of (u64)-1 is a sentinel emitted by
+ * drbd_bitmap_uuid_for_node() when peer_md[n].bitmap_uuid is
+ * physically zero but we are currently a sync target from that
+ * peer at a different generation. It is not evidence of a prior
+ * UUID bump.
+ */
+ if (peer_device->bitmap_uuids[n] != (u64)-1 &&
+ peer_device->bitmap_uuids[n] != 0)
+ return false;
+ }
+ return true;
+}
+
+static bool diskfull_peers_need_new_cur_uuid(struct drbd_device *device)
+{
+ const bool intentional_diskless = device->device_conf.intentional_diskless;
+ struct drbd_peer_device *peer_device;
+ bool rv = false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ /* In protocol < 110 (DRBD 8, strictly two peers), the primary
+ * does not generate or send a new current UUID. Instead the
+ * secondary detects the new primary session and creates the UUID
+ * on its own side. Nothing for us to do here.
+ */
+ if (peer_device->connection->agreed_pro_version < 110)
+ continue;
+
+ /* Only an up-to-date peer persists a new current uuid! */
+ if (peer_device->disk_state[NOW] < D_UP_TO_DATE)
+ continue;
+
+ /* peer_can_fill_a_bitmap_slot() scans the connected peer's
+ * bitmap_uuid records (from the handshake, not our own metadata):
+ * a zero slot for a configured bitmap peer means that peer is on
+ * our generation and a bump marks a new divergence. Authoritative
+ * even for a diskless primary -- it catches a peer an absent third
+ * party resynced to our generation behind our back, which our own
+ * UUID memory misses. A detached primary stays bitmap=yes, so the
+ * peer's record of us self-corrects after a bump and the scan
+ * decides on its own.
+ *
+ * An intentionally-diskless primary is bitmap=no: the peer's
+ * record of us stays zero and is skipped, so the scan misses the
+ * first-write / 2-node case; diskless_primary_needs_uuid_bump()
+ * covers that from third-party slots instead.
+ */
+ if (peer_can_fill_a_bitmap_slot(peer_device, intentional_diskless) ||
+ (intentional_diskless && diskless_primary_needs_uuid_bump(peer_device))) {
+ rv = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+static bool a_lost_peer_is_on_same_cur_uuid(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ bool rv = false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_disk_state pdsk = peer_device->disk_state[NOW];
+ enum drbd_repl_state repl = peer_device->repl_state[NOW];
+
+ /* We don't need to create a new data generation for a directly connected peer */
+ if (repl >= L_ESTABLISHED ||
+ pdsk < D_INCONSISTENT || pdsk > D_UNKNOWN)
+ continue;
+
+ /* Or if UUID_FLAG_SYNC_TARGET is set */
+ if (peer_device->uuid_flags & UUID_FLAG_SYNC_TARGET)
+ continue;
+
+ /* If the peer is unreachable and we know it was at the same data
+ * generation, we must bump so it can detect the divergence on
+ * reconnect.
+ *
+ * If current_uuid is zero, we have no memory of the peer's current
+ * UUID (e.g. after a restart). The peer may have caught up to our
+ * current generation in the interim -- for example, it reconnected
+ * after a prior bump, synced, then disconnected again before we
+ * restarted. Without a bump, that peer would see no UUID divergence
+ * on reconnect: a suspended primary resumes into a split-brain, a
+ * secondary skips the resync its diverged data requires.
+ * Bump conservatively.
+ */
+ if (((device->exposed_data_uuid & ~UUID_PRIMARY) ==
+ (peer_device->current_uuid & ~UUID_PRIMARY)) ||
+ peer_device->current_uuid == 0) {
+ rv = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+/**
+ * drbd_uuid_new_current() - Creates a new current UUID
+ * @device: DRBD device.
+ * @forced: Force UUID creation
+ *
+ * For a diskful device, rotates the old current UUID into peer_md[n].bitmap_uuid
+ * for every absent or weak peer, generates a fresh current UUID, persists it,
+ * and notifies connected peers via P_UUIDS110. Causes an incremental resync
+ * on reconnect for peers that missed writes.
+ *
+ * For a diskless device, generates a fresh exposed_data_uuid and sends
+ * P_CURRENT_UUID to connected peers if a bump is warranted. The decision
+ * is made by two complementary helpers:
+ *
+ * diskfull_peers_need_new_cur_uuid(): bump if a connected diskful peer's
+ * metadata shows that some peer still sits on our generation.
+ *
+ * a_lost_peer_is_on_same_cur_uuid(): bump if an absent peer is, or might
+ * be, on our generation.
+ *
+ * Key scenarios (A = diskless primary, B = diskful secondary connected,
+ * C = third peer, UUID0/1/2 = successive current UUIDs):
+ *
+ * S1. 2-node (A, B), A promotes:
+ * Only the connected peer B exists -> bump.
+ *
+ * S2. 3-node (A, B, C diskful absent), first write:
+ * C has not recorded a bump from us, so it still sits on our generation
+ * -> bump (UUID0->UUID1). C resyncs when it reconnects.
+ *
+ * S3. 3-node, second write (C still absent at UUID0):
+ * We already know C diverged on the first bump (UUID0 != UUID1) -> no
+ * bump. C still sees UUID1 != UUID0 and resyncs on reconnect.
+ *
+ * S4. 3-node, A restarts, promotes (C still absent; C may be at UUID0 or UUID1):
+ * A has lost its knowledge of C's UUID across the restart -> bump.
+ * Necessary because C may have caught up to UUID1 and disconnected again
+ * before A restarted; without the bump, C at UUID1 would see no UUID
+ * divergence -- a suspended primary resumes into a split-brain, a
+ * secondary skips the resync its diverged data requires.
+ *
+ * S5. 3-node, C synced to UUID1 then disconnected; A still running:
+ * C is back on our current generation -> bump (UUID1->UUID2). Without it,
+ * C at UUID1 would reconnect and see no divergence.
+ *
+ * S6. 4-node (A diskless primary; B, C diskful connected; D diskful absent),
+ * a third party resyncs D behind A's back:
+ * A's first write bumps to UUID1, taken by B and C; D is absent. A resync
+ * C->D then completes out of A's sight, so D is on UUID1 but A never
+ * learned of it. On A's next write D still sits on our generation and must
+ * be marked diverged -> bump (UUID1->UUID2). D detects the divergence on
+ * reconnect. A's own per-peer UUID memory cannot see a resync a third node
+ * performed on an absent peer; only the connected peers' metadata can.
+ *
+ * The combined behaviour for an absent peer D:
+ * - First write session: bump triggered because exposed_data_uuid matches
+ * D's last-known UUID.
+ * - Subsequent write sessions in the same primary up-cycle: no bump --
+ * the primary already knows D's stale UUID differs from exposed, so D
+ * will detect the change on reconnect regardless.
+ * - After the primary restarts: bump again via the current_uuid == 0
+ * path, since the primary no longer has any knowledge of D's state.
+ *
+ * Return: what the attempt did, see enum drbd_mint_outcome. MINT_DEFERRED and
+ * MINT_FAILED are retryable and leave the obligation with the caller;
+ * MINT_MINTED and MINT_UNNECESSARY meet it. MINT_EXPOSED moved the obligation
+ * to UNCONFIRMED itself, where the confirm meets it.
+ */
+enum drbd_mint_outcome drbd_uuid_new_current(struct drbd_device *device, bool forced)
+{
+ if (get_ldev_if_state(device, D_UP_TO_DATE)) {
+ enum drbd_mint_outcome outcome;
+
+ /* gen-rotate reason: per call site (promotion=OTHER, others DEGRADE).
+ * Diskful: persists md.current_uuid synchronously -> self-confirms.
+ */
+ outcome = __drbd_uuid_new_current_send(device, forced);
+ put_ldev(device);
+ return outcome;
+ }
+
+ if (diskfull_peers_need_new_cur_uuid(device) ||
+ a_lost_peer_is_on_same_cur_uuid(device)) {
+ /* gen-rotate reason: DEGRADE (diskless primary lost a diskful peer);
+ * no local disk -> not self-confirming, relies on peer acks.
+ */
+ struct drbd_peer_device *peer_device;
+ /* The peers will store the new current UUID... */
+ u64 current_uuid, weak_nodes;
+
+ /* At most one unconfirmed generation (and one predecessor) exists
+ * at a time. The obligation state enforces that on its own: an
+ * event arriving while one is unconfirmed re-arms instead, and the
+ * mint executor is not dispatched from UNCONFIRMED. A caller that
+ * mints without going through the executor still needs the decline.
+ */
+ if (drbd_gen_obligation_state(device) == GEN_OBL_UNCONFIRMED)
+ return MINT_DEFERRED;
+
+ current_uuid = get_random_u64();
+ if (device->resource->role[NOW] == R_PRIMARY)
+ current_uuid |= UUID_PRIMARY;
+ else
+ current_uuid &= ~UUID_PRIMARY;
+
+ /* A new data generation begins here: close the current write
+ * epoch so that writes issued under the old current UUID and
+ * writes issued under the new one never share an epoch. The
+ * post-bump epoch's barrier-ack then tells us a peer actually
+ * persisted the new generation (a confirmation the diskless
+ * primary otherwise lacks).
+ */
+ start_new_tl_epoch(device->resource);
+
+ down_write(&device->uuid_sem);
+ /* Keep the generation we are leaving as the predecessor, tagged with
+ * the epoch the new one begins in, so a peer that reconnects on it can
+ * be recognised and replayed forward instead of rejected.
+ */
+ device->exposed_data_uuid_predecessor = device->exposed_data_uuid;
+ device->exposed_gen_epoch = atomic_read(&device->resource->current_tle_nr);
+
+ /* Publish ordering (paired with the smp_rmb() in
+ * drbd_maybe_release_rotated_gen()):
+ *
+ * set per-peer CURRENT_UUID_UNCONFIRMED (all peers)
+ * smp_wmb()
+ * obligation -> UNCONFIRMED <- the gate, last
+ * send P_CURRENT_UUID <- only after the gate
+ *
+ * Marks before gate: a releaser reads gate then marks, so it
+ * cannot see the gate set with a survivor's mark already clear.
+ */
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED)
+ set_bit(CURRENT_UUID_UNCONFIRMED, peer_device->flags);
+ }
+ smp_wmb(); /* marks before gate; paired with smp_rmb() in releaser */
+ /* Entered from MINTING by the mint executor, from NONE or ARMED by a
+ * caller that mints the obligation itself, and from DISCHARGED when a
+ * suspended resource armed none. Not a dispatch guard: that is the
+ * ARMED -> MINTING transition.
+ */
+ drbd_gen_obligation_transition(device,
+ GEN_OBL_IN(GEN_OBL_MINTING) |
+ GEN_OBL_IN(GEN_OBL_NONE) |
+ GEN_OBL_IN(GEN_OBL_ARMED) |
+ GEN_OBL_IN(GEN_OBL_DISCHARGED),
+ GEN_OBL_UNCONFIRMED, 0, 0);
+ drbd_uuid_set_exposed(device, current_uuid, false);
+ downgrade_write(&device->uuid_sem);
+ drbd_info(device, "sending new current UUID: %016llX\n", current_uuid);
+
+ weak_nodes = drbd_weak_nodes_device(device);
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED) {
+ /* Optimistically assume the peer received and persisted the
+ * new current UUID. It stays marked unconfirmed (above): if
+ * the peer never actually got it (lost P_CURRENT_UUID, stalled
+ * backend) and later reconnects on the old generation, the
+ * handshake must trust the peer's reported UUID over this
+ * optimistic value -- otherwise the diskless primary would
+ * treat the peer as UpToDate at a generation no one persisted.
+ */
+ peer_device->current_uuid = current_uuid;
+ drbd_send_current_uuid(peer_device, current_uuid, weak_nodes);
+ }
+ }
+ up_read(&device->uuid_sem);
+
+ /* We just opened an unconfirmed data generation. The
+ * confirm-before-complete IO hold needs no action here: it engages
+ * lazily, per write, when a write in this generation is acked while still
+ * unconfirmed (see hold_completion_for_unconfirmed_gen in drbd_req.c), and
+ * is released by the in-order barrier ack (NEW_UUID_CONFIRMED).
+ */
+ return MINT_EXPOSED;
+ }
+
+ /* Without local UpToDate data the two helpers above decide, and neither
+ * found a peer that could still sit on our generation.
+ */
+ return MINT_UNNECESSARY;
+}
+
+void drbd_uuid_new_current_by_user(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ u64 peer_mask = 0;
+ int node_id;
+
+ down_write(&device->uuid_sem);
+ for_each_peer_device(peer_device, device) {
+ drbd_uuid_set_bitmap(peer_device, 0); /* Rotate UI_BITMAP to History 1, etc... */
+ peer_mask |= NODE_MASK(peer_device->node_id);
+ }
+
+ /* rotate_current_into_bitmap()'s second loop skips peer_md entries
+ * whose bitmap_uuid is non-zero but does not match prev_c_uuid (the
+ * "stale" case: a previous bump wrote a UUID into the slot and a
+ * subsequent bump left it unchanged). Such a stale value can later
+ * cause a false RULE_BITMAP_SELF match in drbd_sync_handshake() --
+ * SYNC_SOURCE_USE_BITMAP against an empty bitmap -- silently skipping
+ * a resync the peer actually needs.
+ *
+ * For every node_id that is not a currently configured peer (and
+ * therefore not already handled by drbd_uuid_set_bitmap() above),
+ * apply the same treatment: push the old UUID to history so that
+ * reconnect handshakes can still find it, and zero the slot. With
+ * bitmap_uuid == 0 the second loop in rotate_current_into_bitmap()
+ * treats the entry as fresh and advances it to prev_c_uuid together
+ * with all other absent peers.
+ */
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *pm = &device->ldev->md.peers[node_id];
+ u64 bm_uuid;
+
+ if (node_id == device->ldev->md.node_id)
+ continue;
+ if (peer_mask & NODE_MASK(node_id))
+ continue; /* covered by for_each_peer_device above */
+ bm_uuid = pm->bitmap_uuid;
+ if (!bm_uuid)
+ continue;
+ drbd_set_peer_bitmap_uuid(pm, 0, 0);
+ drbd_md_mark_dirty(device);
+ _drbd_uuid_push_history(device, bm_uuid);
+ }
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+
+ if (get_ldev(device)) {
+ __drbd_uuid_new_current_holding_uuid_sem(device);
+ put_ldev(device);
+ }
+ up_write(&device->uuid_sem);
+}
+
+static void drbd_propagate_uuids(struct drbd_device *device, u64 nodes)
+{
+ struct drbd_peer_device *peer_device;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (!(nodes & NODE_MASK(peer_device->node_id)))
+ continue;
+
+ if (peer_device->repl_state[NOW] < L_ESTABLISHED)
+ continue;
+
+ if (list_empty(&peer_device->propagate_uuids_work.list))
+ drbd_queue_work(&peer_device->connection->sender_work,
+ &peer_device->propagate_uuids_work);
+ }
+ rcu_read_unlock();
+}
+
+void drbd_uuid_received_new_current(struct drbd_peer_device *from_pd, u64 val, u64 weak_nodes)
+{
+ struct drbd_device *device = from_pd->device;
+ u64 dagtag = atomic64_read(&from_pd->connection->last_dagtag_sector);
+ struct drbd_peer_device *peer_device;
+ u64 recipients = 0;
+ bool set_current = true;
+
+ down_write(&device->uuid_sem);
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->repl_state[NOW] == L_SYNC_TARGET ||
+ peer_device->repl_state[NOW] == L_BEHIND ||
+ peer_device->repl_state[NOW] == L_PAUSED_SYNC_T) {
+ peer_device->current_uuid = val;
+ set_current = false;
+ }
+ if (peer_device->repl_state[NOW] == L_WF_BITMAP_S ||
+ peer_device->repl_state[NOW] == L_SYNC_SOURCE ||
+ peer_device->repl_state[NOW] == L_PAUSED_SYNC_S ||
+ peer_device->repl_state[NOW] == L_ESTABLISHED)
+ recipients |= NODE_MASK(peer_device->node_id);
+
+ if (peer_device->disk_state[NOW] == D_DISKLESS)
+ recipients |= NODE_MASK(peer_device->node_id);
+ }
+ rcu_read_unlock();
+
+ /* Cleared above when we are a resync target: the UUID is now recorded
+ * in the sync source's peer-device and adopted at resync end.
+ */
+ if (!set_current)
+ drbd_warn(from_pd, "received new current UUID: %016llX weak_nodes=%016llX (deferred to resync end)\n",
+ val, weak_nodes);
+
+ /* Neither up to date nor a resync target: this generation can not be
+ * adopted, and no resync end will adopt it later either.
+ */
+ if (set_current && device->disk_state[NOW] != D_UP_TO_DATE) {
+ drbd_warn(from_pd, "not adopting new current UUID %016llX on %s disk\n",
+ val, drbd_disk_str(device->disk_state[NOW]));
+ set_current = false;
+ }
+
+ if (set_current) {
+ u64 old_current = device->ldev->md.current_uuid;
+ u64 upd;
+
+ drbd_warn(from_pd, "received new current UUID: %016llX weak_nodes=%016llX\n",
+ val, weak_nodes);
+
+ if (device->disk_state[NOW] == D_UP_TO_DATE)
+ recipients |= rotate_current_into_bitmap(device, weak_nodes, dagtag);
+
+ upd = ~weak_nodes; /* These nodes are connected to the primary */
+ upd &= __test_bitmap_slots(device); /* of those, I have a bitmap for */
+ __set_bitmap_slots(device, val, upd);
+ /* Setting bitmap to the (new) current-UUID, means, at this moment
+ we know that we are at the same data as this not connected peer. */
+
+ __drbd_uuid_set_current(device, val);
+
+ /* Even when the old current UUID was not used as any bitmap
+ * UUID, we still add it to the history. This is relevant, in
+ * particular, when we afterwards perform a sync handshake with
+ * a peer which is not one of the "weak_nodes", but hasn't
+ * received the new current UUID. If we do not add the current
+ * UUID to the history, we will end up with a spurious
+ * unrelated data or split-brain decision. */
+ _drbd_uuid_push_history(device, old_current);
+ }
+
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+ downgrade_write(&device->uuid_sem);
+ if (set_current)
+ drbd_propagate_uuids(device, recipients);
+ up_read(&device->uuid_sem);
+}
+
+static u64 __set_bitmap_slots(struct drbd_device *device, u64 bitmap_uuid, u64 do_nodes)
+{
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ u64 modified = 0;
+ int node_id;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (node_id == device->ldev->md.node_id)
+ continue;
+ if (!(do_nodes & NODE_MASK(node_id)))
+ continue;
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md[node_id].flags))
+ continue;
+ if (peer_md[node_id].bitmap_uuid != bitmap_uuid) {
+ u64 previous_bitmap_uuid = peer_md[node_id].bitmap_uuid;
+ /* drbd_info(device, "XXX bitmap[node_id=%d] = %llX\n", node_id, bitmap_uuid); */
+ drbd_set_peer_bitmap_uuid(&peer_md[node_id], bitmap_uuid,
+ device->resource->dagtag_sector);
+ _drbd_uuid_push_history(device, previous_bitmap_uuid);
+ drbd_md_mark_dirty(device);
+ modified |= NODE_MASK(node_id);
+ }
+ }
+
+ return modified;
+}
+
+static u64 __test_bitmap_slots(struct drbd_device *device)
+{
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int node_id;
+ u64 rv = 0;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (peer_md[node_id].bitmap_uuid)
+ rv |= NODE_MASK(node_id);
+ }
+
+ return rv;
+}
+
+/* __test_bitmap_slots_of_peer() operates on view of the world I know the
+ SyncSource had. It might be that in the mean time some peers sent more
+ recent UUIDs to me. Remove all peers that are on the same UUID as I am
+ now from the set of nodes */
+static u64 __test_bitmap_slots_of_peer(struct drbd_peer_device *peer_device)
+{
+ u64 set_bitmap_slots = 0;
+ int node_id;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ u64 bitmap_uuid = peer_device->bitmap_uuids[node_id];
+
+ if (bitmap_uuid != 0 && bitmap_uuid != -1)
+ set_bitmap_slots |= NODE_MASK(node_id);
+ }
+
+ return set_bitmap_slots;
+}
+
+static u64
+peers_with_current_uuid(struct drbd_device *device, u64 current_uuid)
+{
+ struct drbd_peer_device *peer_device;
+ u64 nodes = 0;
+
+ current_uuid &= ~UUID_PRIMARY;
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_disk_state peer_disk_state = peer_device->disk_state[NOW];
+
+ if (peer_disk_state < D_INCONSISTENT || peer_disk_state == D_UNKNOWN)
+ continue;
+ if (current_uuid == (peer_device->current_uuid & ~UUID_PRIMARY))
+ nodes |= NODE_MASK(peer_device->node_id);
+ }
+ rcu_read_unlock();
+
+ return nodes;
+}
+
+/* Prepare a sync target's peer slot at resync start: rotate our current UUID
+ * into the bitmap, then leave divergence. As a resync proceeds, bits in the
+ * bitmap towards the peer are cleared -- on both the sync source and the sync
+ * target. From then on the bitmap only records the blocks not yet in sync, so
+ * it is a convergence bitmap and must no longer be copied to another peer slot.
+ * Clear the flag after rotate_current_into_bitmap(), which may (re-)set it on
+ * this slot, and persist it before any bit is cleared. Both under one
+ * uuid_lock hold, as the other rotate_current_into_bitmap() callers do, so that
+ * a concurrent UUID update can not re-establish divergence on this slot.
+ *
+ * The caller persists the cleared flag with drbd_md_sync_if_dirty().
+ * WARNING: Bits may already be cleared while that write is in flight, so a
+ * crash in that window can leave a partially cleared bitmap on disk with the
+ * flag still set. The bitmap would then be taken for a divergence bitmap and
+ * could be copied to another peer slot.
+ */
+void drbd_uuid_resync_starting(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ unsigned long flags;
+
+ if (peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY)
+ set_bit(SYNC_SRC_CRASHED_PRI, peer_device->flags);
+
+ spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
+ peer_device->rs_start_uuid = drbd_current_uuid(device);
+ rotate_current_into_bitmap(device, 0, device->resource->dagtag_sector);
+ clear_bit(__MDF_PEER_DIVERGENCE_BITMAP,
+ &device->ldev->md.peers[peer_device->node_id].flags);
+ drbd_md_mark_dirty(device);
+ spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
+}
+
+/* Prepare a sync source's peer slot at resync start. The bitmap UUID must be
+ * non-zero throughout the resync. A divergent-generation resync already has a
+ * non-zero bitmap UUID. A same-current forced full resync -- the peer ran
+ * "invalidate", so source and target still share a current UUID yet the whole
+ * device is out of sync -- has no handshake to set it, so set it to our current
+ * UUID here.
+ *
+ * Then leave divergence, as drbd_uuid_resync_starting() does for the target.
+ * Both under one uuid_lock hold.
+ */
+void drbd_uuid_resync_starting_source(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[peer_device->node_id];
+ unsigned long flags;
+
+ spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
+ if (peer_md->bitmap_uuid == 0 &&
+ (peer_device->current_uuid & ~UUID_PRIMARY) ==
+ (drbd_current_uuid(device) & ~UUID_PRIMARY))
+ drbd_set_peer_bitmap_uuid(peer_md, drbd_current_uuid(device),
+ device->resource->dagtag_sector);
+ clear_bit(__MDF_PEER_DIVERGENCE_BITMAP, &peer_md->flags);
+ drbd_md_mark_dirty(device);
+ spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
+}
+
+u64 drbd_uuid_resync_finished(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ unsigned long flags;
+ int i;
+ u64 ss_nz_bm; /* sync_source has non zero bitmap for. expressed as nodemask */
+ u64 pwcu; /* peers with current uuid */
+ u64 newer;
+
+ spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
+ // Inherit history from the sync source
+ for (i = 0; i < ARRAY_SIZE(peer_device->history_uuids); i++)
+ _drbd_uuid_push_history(device, peer_device->history_uuids[i] & ~UUID_PRIMARY);
+
+ // Inherit history in bitmap UUIDs from the sync source
+ for (i = 0; i < DRBD_PEERS_MAX; i++)
+ if (peer_device->bitmap_uuids[i] != -1)
+ _drbd_uuid_push_history(device,
+ peer_device->bitmap_uuids[i] & ~UUID_PRIMARY);
+
+ ss_nz_bm = __test_bitmap_slots_of_peer(peer_device);
+ pwcu = peers_with_current_uuid(device, peer_device->current_uuid);
+
+ newer = __set_bitmap_slots(device, peer_device->rs_start_uuid, ss_nz_bm & ~pwcu);
+ __set_bitmap_slots(device, 0, ~ss_nz_bm | pwcu);
+ __drbd_uuid_set_current(device, peer_device->current_uuid);
+ spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
+
+ return newer;
+}
+
+bool drbd_uuid_set_exposed(struct drbd_device *device, u64 val, bool log)
+{
+ if ((device->exposed_data_uuid & ~UUID_PRIMARY) == (val & ~UUID_PRIMARY) ||
+ val == UUID_JUST_CREATED)
+ return false;
+
+ if (device->resource->role[NOW] == R_PRIMARY)
+ val |= UUID_PRIMARY;
+ else
+ val &= ~UUID_PRIMARY;
+
+ device->exposed_data_uuid = val;
+
+ if (log)
+ drbd_info(device, "Setting exposed data uuid: %016llX\n", (unsigned long long)val);
+
+ return true;
+}
+
+static const char *name_of_node_id(struct drbd_resource *resource, int node_id)
+{
+ /* Caller need to hold rcu_read_lock */
+ struct drbd_connection *connection = drbd_connection_by_node_id(resource, node_id);
+
+ return connection ? rcu_dereference(connection->transport.net_conf)->name : "";
+}
+
+static void forget_bitmap(struct drbd_device *device, int node_id)
+{
+ int bitmap_index = device->ldev->md.peers[node_id].bitmap_index;
+ const char *name;
+
+ if (_drbd_bm_total_weight(device, bitmap_index) == 0)
+ return;
+
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+ rcu_read_lock();
+ name = name_of_node_id(device->resource, node_id);
+ drbd_info(device, "clearing bitmap UUID and content (%lu bits) for node %d (%s)(slot %d)\n",
+ _drbd_bm_total_weight(device, bitmap_index), node_id, name, bitmap_index);
+ rcu_read_unlock();
+ drbd_suspend_io(device, WRITE_ONLY);
+ drbd_bm_lock(device, "forget_bitmap()", BM_LOCK_TEST | BM_LOCK_SET);
+ _drbd_bm_clear_many_bits(device, bitmap_index, 0, -1UL);
+ drbd_bm_unlock(device);
+ drbd_resume_io(device);
+ drbd_md_mark_dirty(device);
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+}
+
+static void copy_bitmap(struct drbd_device *device, int from_id, int to_id)
+{
+ struct drbd_peer_device *peer_device = peer_device_by_node_id(device, to_id);
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ u64 previous_bitmap_uuid = peer_md[to_id].bitmap_uuid;
+ int from_index = peer_md[from_id].bitmap_index;
+ int to_index = peer_md[to_id].bitmap_index;
+ const char *from_name, *to_name;
+
+ drbd_set_peer_bitmap_uuid(&peer_md[to_id], peer_md[from_id].bitmap_uuid,
+ peer_md[from_id].bitmap_dagtag);
+ _drbd_uuid_push_history(device, previous_bitmap_uuid);
+
+ /* Pretending that the updated UUID was sent is a hack.
+ Unfortunately Necessary to not interrupt the handshake */
+ if (peer_device && peer_device->comm_bitmap_uuid == previous_bitmap_uuid)
+ peer_device->comm_bitmap_uuid = peer_md[from_id].bitmap_uuid;
+
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+ rcu_read_lock();
+ from_name = name_of_node_id(device->resource, from_id);
+ to_name = name_of_node_id(device->resource, to_id);
+ drbd_info(device, "Node %d (%s) synced up to node %d (%s). copying bitmap slot %d to %d.\n",
+ to_id, to_name, from_id, from_name, from_index, to_index);
+ rcu_read_unlock();
+ drbd_suspend_io(device, WRITE_ONLY);
+ drbd_bm_lock(device, "copy_bitmap()", BM_LOCK_ALL);
+ drbd_bm_copy_slot(device, from_index, to_index);
+ /* the copy carries the reason the bits were set for */
+ assign_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md[to_id].flags,
+ test_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md[from_id].flags) &&
+ _drbd_bm_total_weight(device, to_index) != 0);
+ peer_md[to_id].placeholder_src = peer_md[from_id].placeholder_src;
+ peer_md[to_id].placeholder_src_complete = peer_md[from_id].placeholder_src_complete;
+ drbd_bm_unlock(device);
+ drbd_resume_io(device);
+ drbd_md_mark_dirty(device);
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+}
+
+/* Find a peer slot holding the given bitmap UUID, to be used as a copy source.
+ * Prefer, in order:
+ * 1. a divergence bitmap with an allocated bitmap (MDF_HAVE_BITMAP);
+ * 2. any other divergence bitmap (e.g. a day0 slot);
+ * 3. any matching slot, so callers can still distinguish "UUID unknown" (-1)
+ * from "UUID known but only as a convergence bitmap being cleared by a
+ * resync", which cannot be copied from.
+ */
+static int find_node_id_by_bitmap_uuid(struct drbd_device *device, u64 bm_uuid)
+{
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int node_id, any = -1, divergence = -1;
- drbd_md_write(device, buffer);
+ bm_uuid &= ~UUID_PRIMARY;
- /* Update device->ldev->md.la_size_sect,
- * since we updated it on metadata. */
- device->ldev->md.la_size_sect = get_capacity(device->vdisk);
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if ((peer_md[node_id].bitmap_uuid & ~UUID_PRIMARY) != bm_uuid)
+ continue;
+ if (any == -1)
+ any = node_id;
+ if (!is_divergence_bitmap(&peer_md[node_id]))
+ continue;
+ if (test_bit(__MDF_HAVE_BITMAP, &peer_md[node_id].flags))
+ return node_id;
+ if (divergence == -1)
+ divergence = node_id;
+ }
- drbd_md_put_buffer(device);
-out:
- put_ldev(device);
+ return divergence != -1 ? divergence : any;
}
-static int check_activity_log_stripe_size(struct drbd_device *device,
- struct meta_data_on_disk *on_disk,
- struct drbd_md *in_core)
+static bool node_connected(struct drbd_resource *resource, int node_id)
{
- u32 al_stripes = be32_to_cpu(on_disk->al_stripes);
- u32 al_stripe_size_4k = be32_to_cpu(on_disk->al_stripe_size_4k);
- u64 al_size_4k;
+ struct drbd_connection *connection;
+ bool r = false;
- /* both not set: default to old fixed size activity log */
- if (al_stripes == 0 && al_stripe_size_4k == 0) {
- al_stripes = 1;
- al_stripe_size_4k = MD_32kB_SECT/8;
- }
+ rcu_read_lock();
+ connection = drbd_connection_by_node_id(resource, node_id);
+ if (connection)
+ r = connection->cstate[NOW] == C_CONNECTED;
+ rcu_read_unlock();
- /* some paranoia plausibility checks */
+ return r;
+}
- /* we need both values to be set */
- if (al_stripes == 0 || al_stripe_size_4k == 0)
- goto err;
+static bool detect_copy_ops_on_peer(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ struct drbd_resource *resource = device->resource;
+ int node_id1, node_id2, from_id;
+ u64 peer_bm_uuid;
+ bool modified = false;
- al_size_4k = (u64)al_stripes * al_stripe_size_4k;
+ for (node_id1 = 0; node_id1 < DRBD_NODE_ID_MAX; node_id1++) {
+ if (device->ldev->md.peers[node_id1].bitmap_index == -1)
+ continue;
- /* Upper limit of activity log area, to avoid potential overflow
- * problems in al_tr_number_to_on_disk_sector(). As right now, more
- * than 72 * 4k blocks total only increases the amount of history,
- * limiting this arbitrarily to 16 GB is not a real limitation ;-) */
- if (al_size_4k > (16 * 1024 * 1024/4))
- goto err;
+ if (node_connected(resource, node_id1))
+ continue;
- /* Lower limit: we need at least 8 transaction slots (32kB)
- * to not break existing setups */
- if (al_size_4k < MD_32kB_SECT/8)
- goto err;
+ peer_bm_uuid = peer_device->bitmap_uuids[node_id1];
+ if (peer_bm_uuid == 0 || peer_bm_uuid == -1ULL)
+ continue;
- in_core->al_stripe_size_4k = al_stripe_size_4k;
- in_core->al_stripes = al_stripes;
- in_core->al_size_4k = al_size_4k;
+ peer_bm_uuid &= ~UUID_PRIMARY;
+ for (node_id2 = node_id1 + 1; node_id2 < DRBD_NODE_ID_MAX; node_id2++) {
+ if (device->ldev->md.peers[node_id2].bitmap_index == -1)
+ continue;
- return 0;
-err:
- drbd_err(device, "invalid activity log striping: al_stripes=%u, al_stripe_size_4k=%u\n",
- al_stripes, al_stripe_size_4k);
- return -EINVAL;
-}
-
-static int check_offsets_and_sizes(struct drbd_device *device, struct drbd_backing_dev *bdev)
-{
- sector_t capacity = drbd_get_capacity(bdev->md_bdev);
- struct drbd_md *in_core = &bdev->md;
- s32 on_disk_al_sect;
- s32 on_disk_bm_sect;
-
- /* The on-disk size of the activity log, calculated from offsets, and
- * the size of the activity log calculated from the stripe settings,
- * should match.
- * Though we could relax this a bit: it is ok, if the striped activity log
- * fits in the available on-disk activity log size.
- * Right now, that would break how resize is implemented.
- * TODO: make drbd_determine_dev_size() (and the drbdmeta tool) aware
- * of possible unused padding space in the on disk layout. */
- if (in_core->al_offset < 0) {
- if (in_core->bm_offset > in_core->al_offset)
- goto err;
- on_disk_al_sect = -in_core->al_offset;
- on_disk_bm_sect = in_core->al_offset - in_core->bm_offset;
- } else {
- if (in_core->al_offset != MD_4kB_SECT)
- goto err;
- if (in_core->bm_offset < in_core->al_offset + in_core->al_size_4k * MD_4kB_SECT)
- goto err;
+ if (node_connected(resource, node_id2))
+ continue;
- on_disk_al_sect = in_core->bm_offset - MD_4kB_SECT;
- on_disk_bm_sect = in_core->md_size_sect - in_core->bm_offset;
+ if (peer_bm_uuid == (peer_device->bitmap_uuids[node_id2] & ~UUID_PRIMARY))
+ goto found;
+ }
}
+ return false;
- /* old fixed size meta data is exactly that: fixed. */
- if (in_core->meta_dev_idx >= 0) {
- if (in_core->md_size_sect != MD_128MB_SECT
- || in_core->al_offset != MD_4kB_SECT
- || in_core->bm_offset != MD_4kB_SECT + MD_32kB_SECT
- || in_core->al_stripes != 1
- || in_core->al_stripe_size_4k != MD_32kB_SECT/8)
- goto err;
+found:
+ from_id = find_node_id_by_bitmap_uuid(device, peer_bm_uuid);
+ if (from_id == -1) {
+ if (peer_md[node_id1].bitmap_uuid == 0 && peer_md[node_id2].bitmap_uuid == 0)
+ return false;
+ drbd_err(peer_device, "unexpected\n");
+ drbd_err(peer_device, "In UUIDs from node %d found equal UUID (%llX) for nodes %d %d\n",
+ peer_device->node_id, peer_bm_uuid, node_id1, node_id2);
+ drbd_err(peer_device, "I have %llX for node_id=%d\n",
+ peer_md[node_id1].bitmap_uuid, node_id1);
+ drbd_err(peer_device, "I have %llX for node_id=%d\n",
+ peer_md[node_id2].bitmap_uuid, node_id2);
+ return false;
}
- if (capacity < in_core->md_size_sect)
- goto err;
- if (capacity - in_core->md_size_sect < drbd_md_first_sector(bdev))
- goto err;
-
- /* should be aligned, and at least 32k */
- if ((on_disk_al_sect & 7) || (on_disk_al_sect < MD_32kB_SECT))
- goto err;
-
- /* should fit (for now: exactly) into the available on-disk space;
- * overflow prevention is in check_activity_log_stripe_size() above. */
- if (on_disk_al_sect != in_core->al_size_4k * MD_4kB_SECT)
- goto err;
-
- /* again, should be aligned */
- if (in_core->bm_offset & 7)
- goto err;
-
- /* FIXME check for device grow with flex external meta data? */
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md[from_id].flags))
+ return false;
- /* can the available bitmap space cover the last agreed device size? */
- if (on_disk_bm_sect < (in_core->la_size_sect+7)/MD_4kB_SECT/8/512)
- goto err;
+ /* Only copy from a divergence bitmap, never from a convergence bitmap
+ * that is being cleared by an ongoing resync.
+ */
+ if (!is_divergence_bitmap(&peer_md[from_id]))
+ return false;
- return 0;
+ if (from_id != node_id1 &&
+ peer_md[node_id1].bitmap_uuid != peer_bm_uuid) {
+ copy_bitmap(device, from_id, node_id1);
+ modified = true;
-err:
- drbd_err(device, "meta data offsets don't make sense: idx=%d "
- "al_s=%u, al_sz4k=%u, al_offset=%d, bm_offset=%d, "
- "md_size_sect=%u, la_size=%llu, md_capacity=%llu\n",
- in_core->meta_dev_idx,
- in_core->al_stripes, in_core->al_stripe_size_4k,
- in_core->al_offset, in_core->bm_offset, in_core->md_size_sect,
- (unsigned long long)in_core->la_size_sect,
- (unsigned long long)capacity);
+ }
+ if (from_id != node_id2 &&
+ peer_md[node_id2].bitmap_uuid != peer_bm_uuid) {
+ copy_bitmap(device, from_id, node_id2);
+ modified = true;
+ }
- return -EINVAL;
+ return modified;
}
-
-/**
- * drbd_md_read() - Reads in the meta data super block
- * @device: DRBD device.
- * @bdev: Device from which the meta data should be read in.
- *
- * Return NO_ERROR on success, and an enum drbd_ret_code in case
- * something goes wrong.
- *
- * Called exactly once during drbd_adm_attach(), while still being D_DISKLESS,
- * even before @bdev is assigned to @device->ldev.
- */
-int drbd_md_read(struct drbd_device *device, struct drbd_backing_dev *bdev)
+void drbd_uuid_detect_finished_resyncs(struct drbd_peer_device *peer_device)
{
- struct meta_data_on_disk *buffer;
- u32 magic, flags;
- int i, rv = NO_ERROR;
+ u64 peer_current_uuid = peer_device->current_uuid & ~UUID_PRIMARY;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ const int my_node_id = device->resource->res_opts.node_id;
+ bool write_bm = false;
+ bool filled = false;
+ bool current_equal;
+ int node_id;
- if (device->state.disk != D_DISKLESS)
- return ERR_DISK_CONFIGURED;
+ current_equal = peer_current_uuid == (drbd_resolved_uuid(peer_device, NULL) & ~UUID_PRIMARY) &&
+ !(peer_device->uuid_flags & UUID_FLAG_SYNC_TARGET) &&
+ !(peer_device->comm_uuid_flags & UUID_FLAG_SYNC_TARGET);
- buffer = drbd_md_get_buffer(device, __func__);
- if (!buffer)
- return ERR_NOMEM;
+ spin_lock_irq(&device->ldev->md.uuid_lock);
- /* First, figure out where our meta data superblock is located,
- * and read it. */
- bdev->md.meta_dev_idx = bdev->disk_conf->meta_dev_idx;
- bdev->md.md_offset = drbd_md_ss(bdev);
- /* Even for (flexible or indexed) external meta data,
- * initially restrict us to the 4k superblock for now.
- * Affects the paranoia out-of-range access check in drbd_md_sync_page_io(). */
- bdev->md.md_size_sect = 8;
-
- if (drbd_md_sync_page_io(device, bdev, bdev->md.md_offset,
- REQ_OP_READ)) {
- /* NOTE: can't do normal error processing here as this is
- called BEFORE disk is attached */
- drbd_err(device, "Error while reading metadata.\n");
- rv = ERR_IO_MD_DISK;
- goto err;
- }
-
- magic = be32_to_cpu(buffer->magic);
- flags = be32_to_cpu(buffer->flags);
- if (magic == DRBD_MD_MAGIC_84_UNCLEAN ||
- (magic == DRBD_MD_MAGIC_08 && !(flags & MDF_AL_CLEAN))) {
- /* btw: that's Activity Log clean, not "all" clean. */
- drbd_err(device, "Found unclean meta data. Did you \"drbdadm apply-al\"?\n");
- rv = ERR_MD_UNCLEAN;
- goto err;
- }
-
- rv = ERR_MD_INVALID;
- if (magic != DRBD_MD_MAGIC_08) {
- if (magic == DRBD_MD_MAGIC_07)
- drbd_err(device, "Found old (0.7) meta data magic. Did you \"drbdadm create-md\"?\n");
- else
- drbd_err(device, "Meta data magic not found. Did you \"drbdadm create-md\"?\n");
- goto err;
- }
+ if (peer_device->repl_state[NOW] == L_OFF && current_equal) {
+ u64 bm_to_peer = peer_device->comm_bitmap_uuid & ~UUID_PRIMARY;
+ u64 bm_towards_me = peer_device->bitmap_uuids[my_node_id] & ~UUID_PRIMARY;
- if (be32_to_cpu(buffer->bm_bytes_per_bit) != BM_BLOCK_SIZE) {
- drbd_err(device, "unexpected bm_bytes_per_bit: %u (expected %u)\n",
- be32_to_cpu(buffer->bm_bytes_per_bit), BM_BLOCK_SIZE);
- goto err;
+ if (bm_towards_me != 0 && bm_to_peer == 0 &&
+ bm_towards_me != peer_current_uuid) {
+ if (peer_device->comm_bm_set == 0 && peer_device->dirty_bits == 0) {
+ drbd_info(peer_device, "Peer missed end of resync, 0 to sync\n");
+ if (peer_device->connection->agreed_pro_version < 124)
+ set_bit(RS_PEER_MISSED_END, peer_device->flags);
+ } else {
+ drbd_info(peer_device, "Peer missed end of resync\n");
+ set_bit(RS_PEER_MISSED_END, peer_device->flags);
+ }
+ }
+ if (bm_towards_me == 0 && bm_to_peer != 0 &&
+ bm_to_peer != peer_current_uuid) {
+ if (peer_device->comm_bm_set == 0 && peer_device->dirty_bits == 0) {
+ int peer_node_id = peer_device->node_id;
+ u64 previous = peer_md[peer_node_id].bitmap_uuid;
+
+ drbd_info(peer_device,
+ "Missed end of resync as sync-source, no bits to sync\n");
+ drbd_set_peer_bitmap_uuid(&peer_md[peer_node_id], 0, 0);
+ _drbd_uuid_push_history(device, previous);
+ peer_device->comm_bitmap_uuid = 0;
+ drbd_md_mark_dirty(device);
+ if (peer_device->connection->agreed_pro_version < 124)
+ set_bit(RS_SOURCE_MISSED_END, peer_device->flags);
+ } else {
+ drbd_info(peer_device, "Missed end of resync as sync-source\n");
+ set_bit(RS_SOURCE_MISSED_END, peer_device->flags);
+ }
+ }
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+ return;
}
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_device *pd2;
- /* convert to in_core endian */
- bdev->md.la_size_sect = be64_to_cpu(buffer->la_size_sect);
- for (i = UI_CURRENT; i < UI_SIZE; i++)
- bdev->md.uuid[i] = be64_to_cpu(buffer->uuid[i]);
- bdev->md.flags = be32_to_cpu(buffer->flags);
- bdev->md.device_uuid = be64_to_cpu(buffer->device_uuid);
-
- bdev->md.md_size_sect = be32_to_cpu(buffer->md_size_sect);
- bdev->md.al_offset = be32_to_cpu(buffer->al_offset);
- bdev->md.bm_offset = be32_to_cpu(buffer->bm_offset);
+ if (node_id == device->ldev->md.node_id)
+ continue;
- if (check_activity_log_stripe_size(device, buffer, &bdev->md))
- goto err;
- if (check_offsets_and_sizes(device, bdev))
- goto err;
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md[node_id].flags) &&
+ !test_bit(__MDF_NODE_EXISTS, &peer_md[node_id].flags))
+ continue;
- if (be32_to_cpu(buffer->bm_offset) != bdev->md.bm_offset) {
- drbd_err(device, "unexpected bm_offset: %d (expected %d)\n",
- be32_to_cpu(buffer->bm_offset), bdev->md.bm_offset);
- goto err;
- }
- if (be32_to_cpu(buffer->md_size_sect) != bdev->md.md_size_sect) {
- drbd_err(device, "unexpected md_size: %u (expected %u)\n",
- be32_to_cpu(buffer->md_size_sect), bdev->md.md_size_sect);
- goto err;
- }
+ pd2 = peer_device_by_node_id(device, node_id);
+ if (pd2 && pd2 != peer_device && pd2->repl_state[NOW] > L_ESTABLISHED)
+ continue;
- rv = NO_ERROR;
+ if (peer_device->bitmap_uuids[node_id] == 0 && peer_md[node_id].bitmap_uuid != 0) {
+ int from_node_id;
+
+ if (current_equal) {
+ u64 previous_bitmap_uuid = peer_md[node_id].bitmap_uuid;
+
+ drbd_set_peer_bitmap_uuid(&peer_md[node_id], 0, 0);
+ _drbd_uuid_push_history(device, previous_bitmap_uuid);
+ if (node_id == peer_device->node_id)
+ drbd_print_uuids(peer_device, "updated UUIDs");
+ else if (test_bit(__MDF_HAVE_BITMAP, &peer_md[node_id].flags))
+ forget_bitmap(device, node_id);
+ else
+ drbd_info(device, "Clearing bitmap UUID for node %d\n",
+ node_id);
+ drbd_md_mark_dirty(device);
+ write_bm = true;
+ }
- spin_lock_irq(&device->resource->req_lock);
- if (device->state.conn < C_CONNECTED) {
- unsigned int peer;
- peer = be32_to_cpu(buffer->la_peer_max_bio_size);
- peer = max(peer, DRBD_MAX_BIO_SIZE_SAFE);
- device->peer_max_bio_size = peer;
+ from_node_id = find_node_id_by_bitmap_uuid(device, peer_current_uuid);
+ if (from_node_id != -1 && node_id != from_node_id &&
+ is_divergence_bitmap(&peer_md[from_node_id]) &&
+ dagtag_newer(peer_md[from_node_id].bitmap_dagtag,
+ peer_md[node_id].bitmap_dagtag)) {
+ if (test_bit(__MDF_HAVE_BITMAP, &peer_md[node_id].flags) &&
+ test_bit(__MDF_HAVE_BITMAP, &peer_md[from_node_id].flags))
+ copy_bitmap(device, from_node_id, node_id);
+ else
+ drbd_info(device, "Node %d synced up to node %d.\n",
+ node_id, from_node_id);
+ drbd_md_mark_dirty(device);
+ filled = true;
+ }
+ }
}
- spin_unlock_irq(&device->resource->req_lock);
-
- err:
- drbd_md_put_buffer(device);
-
- return rv;
-}
-
-/**
- * drbd_md_mark_dirty() - Mark meta data super block as dirty
- * @device: DRBD device.
- *
- * Call this function if you change anything that should be written to
- * the meta-data super block. This function sets MD_DIRTY, and starts a
- * timer that ensures that within five seconds you have to call drbd_md_sync().
- */
-void drbd_md_mark_dirty(struct drbd_device *device)
-{
- if (!test_and_set_bit(MD_DIRTY, &device->flags))
- mod_timer(&device->md_sync_timer, jiffies + 5*HZ);
-}
-
-void drbd_uuid_move_history(struct drbd_device *device) __must_hold(local)
-{
- int i;
- for (i = UI_HISTORY_START; i < UI_HISTORY_END; i++)
- device->ldev->md.uuid[i+1] = device->ldev->md.uuid[i];
-}
+ write_bm |= detect_copy_ops_on_peer(peer_device);
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
-void __drbd_uuid_set(struct drbd_device *device, int idx, u64 val) __must_hold(local)
-{
- if (idx == UI_CURRENT) {
- if (device->state.role == R_PRIMARY)
- val |= 1;
- else
- val &= ~((u64)1);
+ if (write_bm || filled) {
+ u64 to_nodes = filled ? -1 : ~NODE_MASK(peer_device->node_id);
- drbd_set_ed_uuid(device, val);
+ drbd_propagate_uuids(device, to_nodes);
+ drbd_bitmap_io(device, &drbd_bm_write_copy_pages,
+ "detect_finished_resyncs()", BM_LOCK_BULK, NULL);
}
-
- device->ldev->md.uuid[idx] = val;
- drbd_md_mark_dirty(device);
-}
-
-void _drbd_uuid_set(struct drbd_device *device, int idx, u64 val) __must_hold(local)
-{
- unsigned long flags;
- spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
- __drbd_uuid_set(device, idx, val);
- spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
}
-void drbd_uuid_set(struct drbd_device *device, int idx, u64 val) __must_hold(local)
+int drbd_bmio_set_all_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device)
{
- unsigned long flags;
- spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
- if (device->ldev->md.uuid[idx]) {
- drbd_uuid_move_history(device);
- device->ldev->md.uuid[UI_HISTORY_START] = device->ldev->md.uuid[idx];
- }
- __drbd_uuid_set(device, idx, val);
- spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
+ drbd_bm_set_all(device);
+ return drbd_bm_write(device, NULL);
}
/**
- * drbd_uuid_new_current() - Creates a new current UUID
+ * drbd_bmio_set_n_write() - io_fn for drbd_queue_bitmap_io() or drbd_bitmap_io()
* @device: DRBD device.
+ * @peer_device: Peer DRBD device.
*
- * Creates a new current UUID, and rotates the old current UUID into
- * the bitmap slot. Causes an incremental resync upon next connect.
+ * Sets all bits in the bitmap towards one peer and writes the whole bitmap to stable storage.
*/
-void drbd_uuid_new_current(struct drbd_device *device) __must_hold(local)
+int drbd_bmio_set_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device)
{
- u64 val;
- unsigned long long bm_uuid;
-
- val = get_random_u64();
-
- spin_lock_irq(&device->ldev->md.uuid_lock);
- bm_uuid = device->ldev->md.uuid[UI_BITMAP];
-
- if (bm_uuid)
- drbd_warn(device, "bm UUID was already set: %llX\n", bm_uuid);
-
- device->ldev->md.uuid[UI_BITMAP] = device->ldev->md.uuid[UI_CURRENT];
- __drbd_uuid_set(device, UI_CURRENT, val);
- spin_unlock_irq(&device->ldev->md.uuid_lock);
+ int rv = -EIO;
- drbd_print_uuids(device, "new current UUID");
- /* get it to stable storage _now_ */
+ drbd_md_set_peer_flag(peer_device, __MDF_PEER_FULL_SYNC);
drbd_md_sync(device);
-}
+ drbd_bm_set_many_bits(peer_device, 0, -1UL);
-void drbd_uuid_set_bm(struct drbd_device *device, u64 val) __must_hold(local)
-{
- unsigned long flags;
- spin_lock_irqsave(&device->ldev->md.uuid_lock, flags);
- if (device->ldev->md.uuid[UI_BITMAP] == 0 && val == 0) {
- spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
- return;
- }
-
- if (val == 0) {
- drbd_uuid_move_history(device);
- device->ldev->md.uuid[UI_HISTORY_START] = device->ldev->md.uuid[UI_BITMAP];
- device->ldev->md.uuid[UI_BITMAP] = 0;
- } else {
- unsigned long long bm_uuid = device->ldev->md.uuid[UI_BITMAP];
- if (bm_uuid)
- drbd_warn(device, "bm UUID was already set: %llX\n", bm_uuid);
+ rv = drbd_bm_write(device, NULL);
- device->ldev->md.uuid[UI_BITMAP] = val & ~((u64)1);
+ if (!rv) {
+ drbd_md_clear_peer_flag(peer_device, __MDF_PEER_FULL_SYNC);
+ drbd_md_sync(device);
}
- spin_unlock_irqrestore(&device->ldev->md.uuid_lock, flags);
- drbd_md_mark_dirty(device);
+ return rv;
}
/**
- * drbd_bmio_set_n_write() - io_fn for drbd_queue_bitmap_io() or drbd_bitmap_io()
+ * drbd_bmio_set_allocated_n_write() - io_fn for drbd_queue_bitmap_io() or drbd_bitmap_io()
* @device: DRBD device.
- * @peer_device: Peer DRBD device.
+ * @peer_device: parameter ignored
*
- * Sets all bits in the bitmap and writes the whole bitmap to stable storage.
+ * Sets all bits in all allocated bitmap slots and writes it to stable storage.
*/
-int drbd_bmio_set_n_write(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local)
-
+int drbd_bmio_set_allocated_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device)
{
+ const int my_node_id = device->resource->res_opts.node_id;
+ struct drbd_md *md = &device->ldev->md;
int rv = -EIO;
+ int node_id, bitmap_index;
- drbd_md_set_flag(device, MDF_FULL_SYNC);
- drbd_md_sync(device);
- drbd_bm_set_all(device);
-
- rv = drbd_bm_write(device, peer_device);
-
- if (!rv) {
- drbd_md_clear_flag(device, MDF_FULL_SYNC);
- drbd_md_sync(device);
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (node_id == my_node_id)
+ continue;
+ bitmap_index = md->peers[node_id].bitmap_index;
+ if (bitmap_index == -1)
+ continue;
+ _drbd_bm_set_many_bits(device, bitmap_index, 0, -1UL);
}
+ rv = drbd_bm_write(device, NULL);
return rv;
}
/**
- * drbd_bmio_clear_n_write() - io_fn for drbd_queue_bitmap_io() or drbd_bitmap_io()
+ * drbd_bmio_clear_all_n_write() - io_fn for drbd_queue_bitmap_io() or drbd_bitmap_io()
* @device: DRBD device.
* @peer_device: Peer DRBD device.
*
* Clears all bits in the bitmap and writes the whole bitmap to stable storage.
*/
-int drbd_bmio_clear_n_write(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local)
-
+int drbd_bmio_clear_all_n_write(struct drbd_device *device,
+ struct drbd_peer_device *peer_device)
{
drbd_resume_al(device);
drbd_bm_clear_all(device);
- return drbd_bm_write(device, peer_device);
+ return drbd_bm_write(device, NULL);
+}
+
+int drbd_bmio_clear_one_peer(struct drbd_device *device,
+ struct drbd_peer_device *peer_device)
+{
+ drbd_bm_clear_many_bits(peer_device, 0, -1UL);
+ return drbd_bm_write(device, NULL);
}
static int w_bitmap_io(struct drbd_work *w, int unused)
{
- struct drbd_device *device =
- container_of(w, struct drbd_device, bm_io_work.w);
- struct bm_io_work *work = &device->bm_io_work;
+ struct bm_io_work *work =
+ container_of(w, struct bm_io_work, w);
+ struct drbd_device *device = work->device;
int rv = -EIO;
- if (work->flags != BM_LOCKED_CHANGE_ALLOWED) {
- int cnt = atomic_read(&device->ap_bio_cnt);
- if (cnt)
- drbd_err(device, "FIXME: ap_bio_cnt %d, expected 0; queued for '%s'\n",
- cnt, work->why);
- }
-
if (get_ldev(device)) {
- drbd_bm_lock(device, work->why, work->flags);
+ if (work->flags & BM_LOCK_SINGLE_SLOT)
+ drbd_bm_slot_lock(work->peer_device, work->why, work->flags);
+ else
+ drbd_bm_lock(device, work->why, work->flags);
rv = work->io_fn(device, work->peer_device);
- drbd_bm_unlock(device);
+ if (work->flags & BM_LOCK_SINGLE_SLOT)
+ drbd_bm_slot_unlock(work->peer_device);
+ else
+ drbd_bm_unlock(device);
put_ldev(device);
}
- clear_bit_unlock(BITMAP_IO, &device->flags);
- wake_up(&device->misc_wait);
-
if (work->done)
- work->done(device, rv);
+ work->done(device, work->peer_device, rv);
- clear_bit(BITMAP_IO_QUEUED, &device->flags);
- work->why = NULL;
- work->flags = 0;
+ if (atomic_dec_and_test(&device->pending_bitmap_work.n))
+ wake_up(&device->misc_wait);
+ kfree(work);
return 0;
}
+void drbd_queue_pending_bitmap_work(struct drbd_device *device)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&device->pending_bitmap_work.q_lock, flags);
+ spin_lock(&device->resource->work.q_lock);
+ list_splice_tail_init(&device->pending_bitmap_work.q, &device->resource->work.q);
+ spin_unlock(&device->resource->work.q_lock);
+ spin_unlock_irqrestore(&device->pending_bitmap_work.q_lock, flags);
+ wake_up(&device->resource->work.q_wait);
+}
+
/**
* drbd_queue_bitmap_io() - Queues an IO operation on the whole bitmap
* @device: DRBD device.
* @io_fn: IO callback to be called when bitmap IO is possible
* @done: callback to be called after the bitmap IO was performed
* @why: Descriptive text of the reason for doing the IO
- * @flags: Bitmap flags
+ * @flags: Bitmap operation flags
* @peer_device: Peer DRBD device.
*
* While IO on the bitmap happens we freeze application IO thus we ensure
* that drbd_set_out_of_sync() can not be called. This function MAY ONLY be
- * called from worker context. It MUST NOT be used while a previous such
+ * called from sender context. It MUST NOT be used while a previous such
* work is still pending!
*
* Its worker function encloses the call of io_fn() by get_ldev() and
@@ -3476,35 +6516,63 @@ static int w_bitmap_io(struct drbd_work *w, int unused)
*/
void drbd_queue_bitmap_io(struct drbd_device *device,
int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
- void (*done)(struct drbd_device *, int),
+ void (*done)(struct drbd_device *, struct drbd_peer_device *, int),
char *why, enum bm_flag flags,
struct drbd_peer_device *peer_device)
{
- D_ASSERT(device, current == peer_device->connection->worker.task);
-
- D_ASSERT(device, !test_bit(BITMAP_IO_QUEUED, &device->flags));
- D_ASSERT(device, !test_bit(BITMAP_IO, &device->flags));
- D_ASSERT(device, list_empty(&device->bm_io_work.w.list));
- if (device->bm_io_work.why)
- drbd_err(device, "FIXME going to queue '%s' but '%s' still pending?\n",
- why, device->bm_io_work.why);
-
- device->bm_io_work.peer_device = peer_device;
- device->bm_io_work.io_fn = io_fn;
- device->bm_io_work.done = done;
- device->bm_io_work.why = why;
- device->bm_io_work.flags = flags;
-
- spin_lock_irq(&device->resource->req_lock);
- set_bit(BITMAP_IO, &device->flags);
- /* don't wait for pending application IO if the caller indicates that
- * application IO does not conflict anyways. */
- if (flags == BM_LOCKED_CHANGE_ALLOWED || atomic_read(&device->ap_bio_cnt) == 0) {
- if (!test_and_set_bit(BITMAP_IO_QUEUED, &device->flags))
- drbd_queue_work(&peer_device->connection->sender_work,
- &device->bm_io_work.w);
+ struct bm_io_work *bm_io_work;
+
+ D_ASSERT(device, current == device->resource->worker.task);
+
+ bm_io_work = kmalloc_obj(*bm_io_work, GFP_NOIO);
+ if (!bm_io_work) {
+ if (done)
+ done(device, peer_device, -ENOMEM);
+ return;
}
- spin_unlock_irq(&device->resource->req_lock);
+ bm_io_work->w.cb = w_bitmap_io;
+ bm_io_work->device = device;
+ bm_io_work->peer_device = peer_device;
+ bm_io_work->io_fn = io_fn;
+ bm_io_work->done = done;
+ bm_io_work->why = why;
+ bm_io_work->flags = flags;
+
+ /*
+ * Whole-bitmap operations can only take place when there is no
+ * concurrent application I/O. We ensure exclusion between the two
+ * types of I/O with the following mechanism:
+ *
+ * - device->ap_bio_cnt keeps track of the number of application I/O
+ * requests in progress.
+ *
+ * - A non-empty device->pending_bitmap_work list indicates that
+ * whole-bitmap I/O operations are pending, and no new application
+ * I/O should be started. We make sure that the list doesn't appear
+ * empty system wide before trying to queue the whole-bitmap I/O.
+ *
+ * - In dec_ap_bio(), we decrement device->ap_bio_cnt. If it reaches
+ * zero and the device->pending_bitmap_work list is non-empty, we
+ * queue the whole-bitmap operations.
+ *
+ * - In inc_ap_bio(), we increment device->ap_bio_cnt before checking
+ * if the device->pending_bitmap_work list is non-empty. If
+ * device->pending_bitmap_work is non-empty, we immediately call
+ * dec_ap_bio().
+ *
+ * This ensures that whenever there is pending whole-bitmap I/O, we
+ * realize in dec_ap_bio().
+ *
+ */
+
+ /* no one should accidentally schedule the next bitmap IO
+ * when it is only half-queued yet */
+ atomic_inc(&device->ap_bio_cnt[WRITE]);
+ atomic_inc(&device->pending_bitmap_work.n);
+ spin_lock_irq(&device->pending_bitmap_work.q_lock);
+ list_add_tail(&bm_io_work->w.list, &device->pending_bitmap_work.q);
+ spin_unlock_irq(&device->pending_bitmap_work.q_lock);
+ dec_ap_bio(device, WRITE); /* may move to actual work queue */
}
/**
@@ -3512,11 +6580,11 @@ void drbd_queue_bitmap_io(struct drbd_device *device,
* @device: DRBD device.
* @io_fn: IO callback to be called when bitmap IO is possible
* @why: Descriptive text of the reason for doing the IO
- * @flags: Bitmap flags
+ * @flags: Bitmap operation flags
* @peer_device: Peer DRBD device.
*
* freezes application IO while that the actual IO operations runs. This
- * functions MAY NOT be called from worker context.
+ * functions MAY NOT be called from sender context.
*/
int drbd_bitmap_io(struct drbd_device *device,
int (*io_fn)(struct drbd_device *, struct drbd_peer_device *),
@@ -3524,17 +6592,28 @@ int drbd_bitmap_io(struct drbd_device *device,
struct drbd_peer_device *peer_device)
{
/* Only suspend io, if some operation is supposed to be locked out */
- const bool do_suspend_io = flags & (BM_DONT_CLEAR|BM_DONT_SET|BM_DONT_TEST);
+ const bool do_suspend_io = flags & (BM_LOCK_CLEAR|BM_LOCK_SET|BM_LOCK_TEST);
int rv;
- D_ASSERT(device, current != first_peer_device(device)->connection->worker.task);
+ D_ASSERT(device, current != device->resource->worker.task);
+
+ if (!device->bitmap)
+ return 0;
if (do_suspend_io)
- drbd_suspend_io(device);
+ drbd_suspend_io(device, WRITE_ONLY);
+
+ if (flags & BM_LOCK_SINGLE_SLOT)
+ drbd_bm_slot_lock(peer_device, why, flags);
+ else
+ drbd_bm_lock(device, why, flags);
- drbd_bm_lock(device, why, flags);
rv = io_fn(device, peer_device);
- drbd_bm_unlock(device);
+
+ if (flags & BM_LOCK_SINGLE_SLOT)
+ drbd_bm_slot_unlock(peer_device);
+ else
+ drbd_bm_unlock(device);
if (do_suspend_io)
drbd_resume_io(device);
@@ -3542,68 +6621,85 @@ int drbd_bitmap_io(struct drbd_device *device,
return rv;
}
-void drbd_md_set_flag(struct drbd_device *device, int flag) __must_hold(local)
+void drbd_md_set_peer_flag(struct drbd_peer_device *peer_device,
+ enum mdf_peer_flag_bit flag_bit)
{
- if ((device->ldev->md.flags & flag) != flag) {
+ struct drbd_device *device = peer_device->device;
+ struct drbd_md *md = &device->ldev->md;
+
+ if (!test_and_set_bit(flag_bit, &md->peers[peer_device->node_id].flags))
drbd_md_mark_dirty(device);
- device->ldev->md.flags |= flag;
- }
}
-void drbd_md_clear_flag(struct drbd_device *device, int flag) __must_hold(local)
+void drbd_md_clear_peer_flag(struct drbd_peer_device *peer_device,
+ enum mdf_peer_flag_bit flag_bit)
{
- if ((device->ldev->md.flags & flag) != 0) {
+ struct drbd_device *device = peer_device->device;
+ struct drbd_md *md = &device->ldev->md;
+
+ if (test_and_clear_bit(flag_bit, &md->peers[peer_device->node_id].flags))
drbd_md_mark_dirty(device);
- device->ldev->md.flags &= ~flag;
- }
}
-int drbd_md_test_flag(struct drbd_backing_dev *bdev, int flag)
+
+int drbd_md_test_flag(struct drbd_backing_dev *bdev, enum mdf_flag flag)
{
return (bdev->md.flags & flag) != 0;
}
-static void md_sync_timer_fn(struct timer_list *t)
+/* The last out-of-sync bit toward this slot is gone, and with it the reason
+ * the bits were set for. Called under the bitmap lock, with the disk attached.
+ */
+void drbd_md_slot_emptied(struct drbd_device *device, int bitmap_index)
{
- struct drbd_device *device = timer_container_of(device, t,
- md_sync_timer);
- drbd_device_post_work(device, MD_SYNC);
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int node_id;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (peer_md[node_id].bitmap_index != bitmap_index)
+ continue;
+ peer_md[node_id].placeholder_src = 0;
+ peer_md[node_id].placeholder_src_complete = true;
+ if (test_and_clear_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md[node_id].flags))
+ drbd_md_mark_dirty(device);
+ }
}
-/**
- * drbd_wait_misc - wait for a request to make progress
- * @device: device associated with the request
- * @i: the struct drbd_interval embedded in struct drbd_request or
- * struct drbd_peer_request
+/* Every bit standing toward a peer may be one that peer lacks: a crashed
+ * Primary's activity log replay marks the crash window toward all of them.
*/
-int drbd_wait_misc(struct drbd_device *device, struct drbd_interval *i)
+void drbd_md_set_bitmaps_authoritative(struct drbd_device *device)
{
- struct net_conf *nc;
- DEFINE_WAIT(wait);
- long timeout;
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int node_id;
- rcu_read_lock();
- nc = rcu_dereference(first_peer_device(device)->connection->net_conf);
- if (!nc) {
- rcu_read_unlock();
- return -ETIMEDOUT;
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md[node_id].flags))
+ continue;
+ if (_drbd_bm_total_weight(device, peer_md[node_id].bitmap_index) == 0)
+ continue;
+ if (!test_and_set_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md[node_id].flags))
+ drbd_md_mark_dirty(device);
}
- timeout = nc->ko_count ? nc->timeout * HZ / 10 * nc->ko_count : MAX_SCHEDULE_TIMEOUT;
- rcu_read_unlock();
+}
- /* Indicate to wake up device->misc_wait on progress. */
- set_bit(INTERVAL_WAITING, &i->flags);
- prepare_to_wait(&device->misc_wait, &wait, TASK_INTERRUPTIBLE);
- spin_unlock_irq(&device->resource->req_lock);
- timeout = schedule_timeout(timeout);
- finish_wait(&device->misc_wait, &wait);
- spin_lock_irq(&device->resource->req_lock);
- if (!timeout || device->state.conn < C_CONNECTED)
- return -ETIMEDOUT;
- if (signal_pending(current))
- return -ERESTARTSYS;
- return 0;
+bool drbd_md_test_peer_flag(struct drbd_peer_device *peer_device, enum mdf_peer_flag_bit flag_bit)
+{
+ struct drbd_md *md = &peer_device->device->ldev->md;
+
+ if (peer_device->bitmap_index == -1)
+ return false;
+
+ return test_bit(flag_bit, &md->peers[peer_device->node_id].flags);
+}
+
+static void md_sync_timer_fn(struct timer_list *t)
+{
+ struct drbd_device *device = timer_container_of(device, t, md_sync_timer);
+
+ drbd_device_post_work(device, MD_SYNC);
}
+
void lock_all_resources(void)
{
struct drbd_resource *resource;
@@ -3612,7 +6708,7 @@ void lock_all_resources(void)
mutex_lock(&resources_mutex);
local_irq_disable();
for_each_resource(resource, &drbd_resources)
- spin_lock_nested(&resource->req_lock, i++);
+ read_lock(&resource->state_rwlock);
}
void unlock_all_resources(void)
@@ -3620,11 +6716,142 @@ void unlock_all_resources(void)
struct drbd_resource *resource;
for_each_resource(resource, &drbd_resources)
- spin_unlock(&resource->req_lock);
+ read_unlock(&resource->state_rwlock);
local_irq_enable();
mutex_unlock(&resources_mutex);
}
+long twopc_timeout(struct drbd_resource *resource)
+{
+ return resource->res_opts.twopc_timeout * HZ/10;
+}
+
+u64 directly_connected_nodes(struct drbd_resource *resource, enum which_state which)
+{
+ u64 directly_connected = 0;
+ struct drbd_connection *connection;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->cstate[which] < C_CONNECTED)
+ continue;
+ directly_connected |= NODE_MASK(connection->peer_node_id);
+ }
+ rcu_read_unlock();
+
+ return directly_connected;
+}
+
+static sector_t bm_sect_to_max_capacity(const struct drbd_md *md, sector_t bm_sect)
+{
+ /* we do our meta data IO in 4k units */
+ u64 bm_bytes = ALIGN_DOWN(bm_sect << SECTOR_SHIFT, 4096);
+ u64 bm_bytes_per_peer = div_u64(bm_bytes, md->max_peers);
+ u64 bm_bits_per_peer = bm_bytes_per_peer * BITS_PER_BYTE;
+
+ return bm_bits_per_peer << (md->bm_block_shift - SECTOR_SHIFT);
+}
+
+
+/**
+ * drbd_get_max_capacity() - Returns the capacity for user-data on the local backing device
+ * @device: The DRBD device.
+ * @bdev: Meta data block device.
+ * @warn: Whether to warn when size is clipped.
+ *
+ * This function returns the capacity for user-data on the local backing
+ * device. In the case of internal meta-data, this is the backing disk size
+ * reduced by the meta-data size. In the case of external meta-data, this is
+ * the size of the backing disk.
+ */
+sector_t drbd_get_max_capacity(
+ struct drbd_device *device, struct drbd_backing_dev *bdev, bool warn)
+{
+ unsigned int bm_max_peers = bdev->md.max_peers;
+ unsigned int bm_block_size = bdev->md.bm_block_size;
+ sector_t backing_bdev_capacity = drbd_get_capacity(bdev->backing_bdev);
+ sector_t bm_sect;
+ sector_t backing_capacity_remaining;
+ sector_t metadata_limit;
+ sector_t max_capacity;
+
+ switch (bdev->md.meta_dev_idx) {
+ case DRBD_MD_INDEX_INTERNAL:
+ case DRBD_MD_INDEX_FLEX_INT:
+ bm_sect = bdev->md.al_offset - bdev->md.bm_offset;
+ backing_capacity_remaining = drbd_md_first_sector(bdev);
+ break;
+ case DRBD_MD_INDEX_FLEX_EXT:
+ bm_sect = bdev->md.md_size_sect - bdev->md.bm_offset;
+ backing_capacity_remaining = backing_bdev_capacity;
+ break;
+ default:
+ bm_sect = DRBD_BM_SECTORS_INDEXED;
+ backing_capacity_remaining = backing_bdev_capacity;
+ }
+
+ metadata_limit = bm_sect_to_max_capacity(&bdev->md, bm_sect);
+
+ dynamic_drbd_dbg(device,
+ "Backing device capacity: %llus, remaining: %llus, bitmap sectors: %llus\n",
+ (unsigned long long) backing_bdev_capacity,
+ (unsigned long long) backing_capacity_remaining,
+ (unsigned long long) bm_sect);
+ dynamic_drbd_dbg(device,
+ "Max peers: %u, bytes_per_bit: %u, metadata limit: %llus, hard limit: %llus\n",
+ bm_max_peers, bm_block_size,
+ (unsigned long long) metadata_limit,
+ (unsigned long long) DRBD_MAX_SECTORS);
+
+ max_capacity = backing_capacity_remaining;
+ if (max_capacity > DRBD_MAX_SECTORS) {
+ if (warn)
+ drbd_warn(device, "Device size clipped from %llus to %llus due to DRBD limitations\n",
+ (unsigned long long) max_capacity,
+ (unsigned long long) DRBD_MAX_SECTORS);
+ max_capacity = DRBD_MAX_SECTORS;
+ }
+ if (max_capacity > metadata_limit) {
+ if (warn)
+ drbd_warn(device, "Device size clipped from %llus to %llus due to metadata size\n",
+ (unsigned long long) max_capacity,
+ (unsigned long long) metadata_limit);
+ max_capacity = metadata_limit;
+ }
+ return max_capacity;
+}
+
+/* this is about cluster partitions, not block device partitions */
+sector_t drbd_partition_data_capacity(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ sector_t capacity = (sector_t)(-1);
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (test_bit(HAVE_SIZES, peer_device->flags)) {
+ dynamic_drbd_dbg(peer_device, "d_size: %llus\n",
+ (unsigned long long)peer_device->d_size);
+ capacity = min_not_zero(capacity, peer_device->d_size);
+ }
+ }
+ rcu_read_unlock();
+
+ if (get_ldev_if_state(device, D_ATTACHING)) {
+ /* In case we somehow end up here while attaching, but before
+ * we even assigned the ldev, pretend to still be diskless.
+ */
+ if (device->ldev != NULL) {
+ sector_t local_capacity = drbd_local_max_size(device);
+
+ capacity = min_not_zero(capacity, local_capacity);
+ }
+ put_ldev(device);
+ }
+
+ return capacity != (sector_t)(-1) ? capacity : 0;
+}
+
#ifdef CONFIG_DRBD_FAULT_INJECTION
/* Fault insertion support including random number generator shamelessly
* stolen from kernel/rcutorture.c */
@@ -3656,7 +6883,8 @@ _drbd_fault_random(struct fault_random_state *rsp)
}
static char *
-_drbd_fault_str(unsigned int type) {
+_drbd_fault_str(unsigned int type)
+{
static char *_faults[] = {
[DRBD_FAULT_MD_WR] = "Meta-data write",
[DRBD_FAULT_MD_RD] = "Meta-data read",
@@ -3668,6 +6896,7 @@ _drbd_fault_str(unsigned int type) {
[DRBD_FAULT_BM_ALLOC] = "BM allocation",
[DRBD_FAULT_AL_EE] = "EE allocation",
[DRBD_FAULT_RECEIVE] = "receive data corruption",
+ [DRBD_FAULT_BIO_TOO_SMALL] = "BIO too small",
};
return (type < DRBD_FAULT_MAX) ? _faults[type] : "**Unknown**";
@@ -3680,14 +6909,14 @@ _drbd_insert_fault(struct drbd_device *device, unsigned int type)
unsigned int ret = (
(drbd_fault_devs == 0 ||
- ((1 << device_to_minor(device)) & drbd_fault_devs) != 0) &&
+ ((1 << device->minor) & drbd_fault_devs) != 0) &&
(((_drbd_fault_random(&rrs) % 100) + 1) <= drbd_fault_rate));
if (ret) {
drbd_fault_count++;
drbd_warn_ratelimit(device, "***Simulating %s failure\n",
- _drbd_fault_str(type));
+ _drbd_fault_str(type));
}
return ret;
@@ -3697,7 +6926,6 @@ _drbd_insert_fault(struct drbd_device *device, unsigned int type)
module_init(drbd_init)
module_exit(drbd_cleanup)
-EXPORT_SYMBOL(drbd_conn_str);
-EXPORT_SYMBOL(drbd_role_str);
-EXPORT_SYMBOL(drbd_disk_str);
-EXPORT_SYMBOL(drbd_set_st_err_str);
+/* For transport layer */
+EXPORT_SYMBOL(drbd_destroy_connection);
+EXPORT_SYMBOL(drbd_destroy_path);
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 15/20] drbd: rework receiver for DRBD 9 transport and multi-peer protocol
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (13 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 14/20] drbd: rework module core for DRBD 9 transport and multi-peer Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 16/20] drbd: rework the netlink core for DRBD 9 Christoph Böhmwalder
` (4 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Joel Colledge,
Andreas Gruenbacher, Moritz Tanner, Roland Kammerer
Adapt the receiver to the DRBD 9 multi-peer architecture.
Replace all direct socket I/O by calls through the transport abstraction
layer, with the transport managing buffer allocation.
Move peer request tracking from per-device lists to connection-level
structures, enabling a single receiver thread to serve all volumes on
a connection.
UUID-based resync decisions replace the old integer heuristic with enums
for strategies and rules, so each sync handshake outcome and its reason
are self-describing and logged by name.
Update UUID comparison for multi-peer: each peer now carries per-node
bitmap UUIDs and a history array, replacing the fixed four-slot layout.
Introduce DAG-tag ordering as a causal consistency mechanism, letting
peer requests declare dependencies on writes seen at another node and
waiting until those dependencies are resolved.
Add two-phase commit handling so that coordinated state changes (role
transitions, resync initiation, resize) can be propagated to all nodes
atomically.
Write conflict detection moves from a flag-based approach to an
interval-tree with typed intervals, using asynchronous deferred
submission for conflicting requests.
The disconnect path is restructured at the connection level: it
cancels dagtag-dependent requests, drains resync activity, flushes
workqueues, and performs per-peer-device teardown before returning
the connection to the unconnected state.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Moritz Tanner <moritz.tanner@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_receiver.c | 17304 ++++++++++++++++++++-------
1 file changed, 13128 insertions(+), 4176 deletions(-)
diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index 101192d8398f..95cf8db197d8 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -6,41 +6,53 @@
* Copyright (C) 2008, LINBIT HA-Solutions GmbH.
*/
-#include <linux/module.h>
-
-#include <linux/uaccess.h>
#include <net/sock.h>
+#include <linux/bio.h>
#include <linux/drbd.h>
#include <linux/fs.h>
#include <linux/file.h>
#include <linux/in.h>
#include <linux/mm.h>
-#include <linux/memcontrol.h>
+#include <linux/memcontrol.h> /* needed on kernels <4.3 */
#include <linux/mm_inline.h>
#include <linux/slab.h>
-#include <uapi/linux/sched/types.h>
-#include <linux/sched/signal.h>
#include <linux/pkt_sched.h>
-#include <linux/unistd.h>
#include <linux/vmalloc.h>
#include <linux/random.h>
-#include <linux/string.h>
-#include <linux/scatterlist.h>
+#include <net/ipv6.h>
#include <linux/part_stat.h>
-#include <linux/mempool.h>
+
#include "drbd_int.h"
+#include "drbd_meta_data.h"
#include "drbd_protocol.h"
#include "drbd_req.h"
#include "drbd_vli.h"
-#define PRO_FEATURES (DRBD_FF_TRIM|DRBD_FF_THIN_RESYNC|DRBD_FF_WSAME|DRBD_FF_WZEROES)
-struct packet_info {
- enum drbd_packet cmd;
- unsigned int size;
- unsigned int vnr;
- void *data;
+enum ao_op {
+ OUTDATE_DISKS,
+ OUTDATE_DISKS_AND_DISCONNECT,
+};
+
+struct flush_work {
+ struct drbd_work w;
+ struct drbd_epoch *epoch;
+};
+
+struct update_peers_work {
+ struct drbd_work w;
+ struct drbd_peer_device *peer_device;
+ sector_t sector_start;
+ sector_t sector_end;
+};
+
+enum epoch_event {
+ EV_PUT,
+ EV_GOT_BARRIER_NR,
+ EV_BARRIER_DONE,
+ EV_BECAME_LAST,
+ EV_CLEANUP = 32, /* used as flag */
};
enum finish_epoch {
@@ -49,201 +61,600 @@ enum finish_epoch {
FE_RECYCLED,
};
-static int drbd_do_features(struct drbd_connection *connection);
-static int drbd_do_auth(struct drbd_connection *connection);
-static int drbd_disconnected(struct drbd_peer_device *);
-static void conn_wait_active_ee_empty(struct drbd_connection *connection);
+enum resync_reason {
+ AFTER_UNSTABLE,
+ DISKLESS_PRIMARY,
+};
+
+enum sync_rule {
+ RULE_SYNC_SOURCE_MISSED_FINISH,
+ RULE_SYNC_SOURCE_PEER_MISSED_FINISH,
+ RULE_SYNC_TARGET_MISSED_FINISH,
+ RULE_SYNC_TARGET_PEER_MISSED_FINISH,
+ RULE_SYNC_TARGET_MISSED_START,
+ RULE_SYNC_SOURCE_MISSED_START,
+ RULE_INITIAL_HANDSHAKE_CHANGED,
+ RULE_JUST_CREATED_PEER,
+ RULE_JUST_CREATED_SELF,
+ RULE_JUST_CREATED_BOTH,
+ RULE_CRASHED_PRIMARY,
+ RULE_LOST_QUORUM,
+ RULE_RECONNECTED,
+ RULE_BOTH_OFF,
+ RULE_RECONCILE_BITMAP,
+ RULE_RECONCILE_DAGTAG,
+ RULE_BITMAP_PEER,
+ RULE_BITMAP_PEER_OTHER,
+ RULE_BITMAP_SELF,
+ RULE_BITMAP_SELF_OTHER,
+ RULE_BITMAP_BOTH,
+ RULE_HISTORY_PEER,
+ RULE_HISTORY_SELF,
+ RULE_HISTORY_BOTH,
+};
+
+static const char * const sync_rule_names[] = {
+ [RULE_SYNC_SOURCE_MISSED_FINISH] = "sync-source-missed-finish",
+ [RULE_SYNC_SOURCE_PEER_MISSED_FINISH] = "sync-source-peer-missed-finish",
+ [RULE_SYNC_TARGET_MISSED_FINISH] = "sync-target-missed-finish",
+ [RULE_SYNC_TARGET_PEER_MISSED_FINISH] = "sync-target-peer-missed-finish",
+ [RULE_SYNC_TARGET_MISSED_START] = "sync-target-missed-start",
+ [RULE_SYNC_SOURCE_MISSED_START] = "sync-source-missed-start",
+ [RULE_INITIAL_HANDSHAKE_CHANGED] = "initial-handshake-changed",
+ [RULE_JUST_CREATED_PEER] = "just-created-peer",
+ [RULE_JUST_CREATED_SELF] = "just-created-self",
+ [RULE_JUST_CREATED_BOTH] = "just-created-both",
+ [RULE_CRASHED_PRIMARY] = "crashed-primary",
+ [RULE_LOST_QUORUM] = "lost-quorum",
+ [RULE_RECONNECTED] = "reconnected",
+ [RULE_BOTH_OFF] = "both-off",
+ [RULE_RECONCILE_BITMAP] = "reconcile-bitmap",
+ [RULE_RECONCILE_DAGTAG] = "reconcile-dagtag",
+ [RULE_BITMAP_PEER] = "bitmap-peer",
+ [RULE_BITMAP_PEER_OTHER] = "bitmap-peer-other",
+ [RULE_BITMAP_SELF] = "bitmap-self",
+ [RULE_BITMAP_SELF_OTHER] = "bitmap-self-other",
+ [RULE_BITMAP_BOTH] = "bitmap-both",
+ [RULE_HISTORY_PEER] = "history-peer",
+ [RULE_HISTORY_SELF] = "history-self",
+ [RULE_HISTORY_BOTH] = "history-both",
+};
+
+enum sync_strategy {
+ UNDETERMINED = 0,
+ NO_SYNC,
+ SYNC_SOURCE_IF_BOTH_FAILED,
+ SYNC_SOURCE_USE_BITMAP,
+ SYNC_SOURCE_SET_BITMAP,
+ SYNC_SOURCE_COPY_BITMAP,
+ SYNC_TARGET_IF_BOTH_FAILED,
+ SYNC_TARGET_USE_BITMAP,
+ SYNC_TARGET_SET_BITMAP,
+ SYNC_TARGET_CLEAR_BITMAP,
+ SPLIT_BRAIN_AUTO_RECOVER,
+ SPLIT_BRAIN_DISCONNECT,
+ UNRELATED_DATA,
+ RETRY_CONNECT,
+ REQUIRES_PROTO_91,
+ REQUIRES_PROTO_96,
+ REQUIRES_PROTO_124,
+ SYNC_TARGET_PRIMARY_RECONNECT,
+ SYNC_TARGET_PRIMARY_DISCONNECT,
+ DRY_RUN,
+};
+
+struct sync_descriptor {
+ char * const name;
+ int required_protocol;
+ bool is_split_brain;
+ bool is_sync_source;
+ bool is_sync_target;
+ bool reconnect;
+ bool disconnect;
+ int resync_peer_preference;
+ enum sync_strategy full_sync_equivalent;
+ enum sync_strategy reverse;
+};
+
+static const struct sync_descriptor sync_descriptors[] = {
+ [UNDETERMINED] = {
+ .name = "?",
+ },
+ [NO_SYNC] = {
+ .name = "no-sync",
+ .resync_peer_preference = 5,
+ },
+ [SYNC_SOURCE_IF_BOTH_FAILED] = {
+ .name = "source-if-both-failed",
+ .is_sync_source = true,
+ .reverse = SYNC_TARGET_IF_BOTH_FAILED,
+ },
+ [SYNC_SOURCE_USE_BITMAP] = {
+ .name = "source-use-bitmap",
+ .is_sync_source = true,
+ .full_sync_equivalent = SYNC_SOURCE_SET_BITMAP,
+ .reverse = SYNC_TARGET_USE_BITMAP,
+ },
+ [SYNC_SOURCE_SET_BITMAP] = {
+ .name = "source-set-bitmap",
+ .is_sync_source = true,
+ .reverse = SYNC_TARGET_SET_BITMAP,
+ },
+ [SYNC_SOURCE_COPY_BITMAP] = {
+ .name = "source-copy-other-bitmap",
+ .is_sync_source = true,
+ },
+ [SYNC_TARGET_IF_BOTH_FAILED] = {
+ .name = "target-if-both-failed",
+ .is_sync_target = true,
+ .resync_peer_preference = 4,
+ .reverse = SYNC_SOURCE_IF_BOTH_FAILED,
+ },
+ [SYNC_TARGET_USE_BITMAP] = {
+ .name = "target-use-bitmap",
+ .is_sync_target = true,
+ .full_sync_equivalent = SYNC_TARGET_SET_BITMAP,
+ .resync_peer_preference = 3,
+ .reverse = SYNC_SOURCE_USE_BITMAP,
+ },
+ [SYNC_TARGET_SET_BITMAP] = {
+ .name = "target-set-bitmap",
+ .is_sync_target = true,
+ .resync_peer_preference = 2,
+ .reverse = SYNC_SOURCE_SET_BITMAP,
+ },
+ [SYNC_TARGET_CLEAR_BITMAP] = {
+ .name = "target-clear-bitmap",
+ .is_sync_target = true,
+ .resync_peer_preference = 1,
+ },
+ [SPLIT_BRAIN_AUTO_RECOVER] = {
+ .name = "split-brain-auto-recover",
+ .is_split_brain = true,
+ .disconnect = true,
+ },
+ [SPLIT_BRAIN_DISCONNECT] = {
+ .name = "split-brain-disconnect",
+ .is_split_brain = true,
+ .disconnect = true,
+ },
+ [UNRELATED_DATA] = {
+ .name = "unrelated-data",
+ .disconnect = true,
+ },
+ [RETRY_CONNECT] = {
+ .name = "retry-connect",
+ .reconnect = true,
+ },
+ [REQUIRES_PROTO_91] = {
+ .name = "requires-proto-91",
+ .required_protocol = 91,
+ .disconnect = true,
+ },
+ [REQUIRES_PROTO_96] = {
+ .name = "requires-proto-96",
+ .required_protocol = 96,
+ .disconnect = true,
+ },
+ [REQUIRES_PROTO_124] = {
+ .name = "requires-proto-124",
+ .required_protocol = 124,
+ .disconnect = true,
+ },
+ [SYNC_TARGET_PRIMARY_RECONNECT] = {
+ .name = "sync-target-primary-reconnect",
+ .is_sync_target = true,
+ .reconnect = true,
+ },
+ [SYNC_TARGET_PRIMARY_DISCONNECT] = {
+ .name = "sync-target-primary-disconnect",
+ .is_sync_target = true,
+ .disconnect = true,
+ },
+ [DRY_RUN] = {
+ .name = "dry-run",
+ .disconnect = true,
+ },
+};
+
+enum rcv_timeou_kind {
+ PING_TIMEOUT,
+ REGULAR_TIMEOUT,
+};
+
+int drbd_do_features(struct drbd_connection *connection);
+int drbd_do_auth(struct drbd_connection *connection);
+static void conn_disconnect(struct drbd_connection *connection);
+
static enum finish_epoch drbd_may_finish_epoch(struct drbd_connection *, struct drbd_epoch *, enum epoch_event);
static int e_end_block(struct drbd_work *, int);
+static void cleanup_unacked_peer_requests(struct drbd_connection *connection);
+static void cleanup_peer_ack_list(struct drbd_connection *connection);
+static u64 node_ids_to_bitmap(struct drbd_device *device, u64 node_ids);
+static void process_twopc(struct drbd_connection *, struct twopc_reply *, struct packet_info *, unsigned long);
+static void drbd_resync(struct drbd_peer_device *, enum resync_reason);
+static void drbd_unplug_all_devices(struct drbd_connection *connection);
+static int decode_header(struct drbd_connection *, const void *, struct packet_info *);
+static void check_resync_source(struct drbd_peer_device *primary_pd, u64 weak_nodes);
+static void set_rcvtimeo(struct drbd_connection *connection, enum rcv_timeou_kind kind);
+static bool disconnect_expected(struct drbd_connection *connection);
+static bool uuid_in_peer_history(struct drbd_peer_device *peer_device, u64 uuid);
+static bool uuid_in_my_history(struct drbd_device *device, u64 uuid);
+static int find_common_lost_primary_node_id(struct drbd_connection *self);
+static void drbd_cancel_conflicting_resync_requests(struct drbd_peer_device *peer_device);
+
+static const char *drbd_sync_rule_str(enum sync_rule rule)
+{
+ if (rule < 0 || rule >= ARRAY_SIZE(sync_rule_names)) {
+ WARN_ON(true);
+ return "?";
+ }
+ return sync_rule_names[rule];
+}
+
+static struct sync_descriptor strategy_descriptor(enum sync_strategy strategy)
+{
+ if (strategy < 0 || strategy >= ARRAY_SIZE(sync_descriptors)) {
+ WARN_ON(true);
+ return sync_descriptors[UNDETERMINED];
+ }
+ return sync_descriptors[strategy];
+}
+
+static bool is_strategy_determined(enum sync_strategy strategy)
+{
+ return strategy == NO_SYNC ||
+ strategy_descriptor(strategy).is_sync_source ||
+ strategy_descriptor(strategy).is_sync_target;
+}
+
+static struct drbd_epoch *previous_epoch(struct drbd_connection *connection, struct drbd_epoch *epoch)
+{
+ struct drbd_epoch *prev;
+
+ spin_lock(&connection->epoch_lock);
+ prev = list_entry(epoch->list.prev, struct drbd_epoch, list);
+ if (prev == epoch || prev == connection->current_epoch)
+ prev = NULL;
+ spin_unlock(&connection->epoch_lock);
+ return prev;
+}
+static void rs_sectors_came_in(struct drbd_peer_device *peer_device, int size)
+{
+ int rs_sect_in = atomic_add_return(size >> 9, &peer_device->rs_sect_in);
-#define GFP_TRY (__GFP_HIGHMEM | __GFP_NOWARN)
+ /* When resync runs faster than anticipated, consider running the
+ * resync_work early. */
+ if (rs_sect_in >= drbd_rs_outstanding(peer_device))
+ drbd_rs_all_in_flight_came_back(peer_device, rs_sect_in);
+}
-static struct page *__drbd_alloc_pages(unsigned int number)
+static void drbd_strip_bio_list(struct drbd_transport *transport, struct bio_list *bios)
{
- struct page *page = NULL;
- struct page *tmp = NULL;
- unsigned int i = 0;
-
- /* GFP_TRY, because we must not cause arbitrary write-out: in a DRBD
- * "criss-cross" setup, that might cause write-out on some other DRBD,
- * which in turn might block on the other node at this very place. */
- for (i = 0; i < number; i++) {
- tmp = mempool_alloc(&drbd_buffer_page_pool, GFP_TRY);
- if (!tmp)
- goto fail;
- set_page_private(tmp, (unsigned long)page);
- page = tmp;
+ struct bvec_iter iter;
+ struct bio_vec bvec;
+ struct bio *bio;
+
+ while ((bio = bio_list_pop(bios))) {
+ bio_for_each_bvec(bvec, bio, iter) {
+ struct page *page = bvec.bv_page;
+ unsigned int end = bvec.bv_offset + bvec.bv_len;
+ unsigned int pos = 0;
+
+ /* bio_add_page() may have merged contiguous pages from
+ * separate allocations into a single bvec, and a bvec may
+ * start mid-page. Drop one reference per allocation unit the bvec
+ * touches, stepping by compound_order.
+ */
+ while (pos < end) {
+ unsigned int order = compound_order(page);
+
+ drbd_free_page(transport, page);
+ page += 1 << order;
+ pos += PAGE_SIZE << order;
+ }
+ }
+ bio_put(bio);
}
+}
+
+void drbd_peer_req_strip_bio(struct drbd_peer_request *peer_req)
+{
+ drbd_strip_bio_list(&peer_req->peer_device->connection->transport, &peer_req->bios);
+}
+
+static struct page *
+__drbd_alloc_pages(struct drbd_connection *connection, gfp_t gfp_mask, int order)
+{
+ struct page *page;
+ unsigned int mxb;
+
+ rcu_read_lock();
+ mxb = rcu_dereference(connection->transport.net_conf)->max_buffers;
+ rcu_read_unlock();
+
+ if (atomic_read(&connection->pp_in_use) >= mxb)
+ schedule_timeout_interruptible(HZ / 10);
+
+ if (order == 0)
+ page = mempool_alloc(&drbd_buffer_page_pool, gfp_mask);
+ else
+ page = alloc_pages(gfp_mask | __GFP_COMP | __GFP_NORETRY, order);
+
+ if (page)
+ atomic_add(1 << order, &connection->pp_in_use);
+
return page;
-fail:
- page_chain_for_each_safe(page, tmp) {
- set_page_private(page, 0);
- mempool_free(page, &drbd_buffer_page_pool);
- }
- return NULL;
}
/**
- * drbd_alloc_pages() - Returns @number pages, retries forever (or until signalled)
- * @peer_device: DRBD device.
- * @number: number of pages requested
- * @retry: whether to retry, if not enough pages are available right now
- *
- * Tries to allocate number pages, first from our own page pool, then from
- * the kernel.
- * Possibly retry until DRBD frees sufficient pages somewhere else.
+ * drbd_alloc_pages() - Returns a page, which might be a single or compound page
+ * @transport: DRBD transport
+ * @gfp_mask: how to allocate and whether to loop until we succeed
+ * @size: Desired size, gets rounded down to the closest power of two
*
- * If this allocation would exceed the max_buffers setting, we throttle
- * allocation (schedule_timeout) to give the system some room to breathe.
+ * Allocates a page from the kernel or from the private mempool. When this
+ * allocation exceeds the max_buffers setting, throttle the allocation via
+ * schedule_timeout.
*
- * We do not use max-buffers as hard limit, because it could lead to
- * congestion and further to a distributed deadlock during online-verify or
- * (checksum based) resync, if the max-buffers, socket buffer sizes and
+ * We do not use max-buffers as a hard limit, because it could lead to
+ * congestion and, further, to a distributed deadlock during online-verify or
+ * (checksum-based) resync, if the max-buffers, socket buffer sizes, and
* resync-rate settings are mis-configured.
- *
- * Returns a page chain linked via page->private.
*/
-struct page *drbd_alloc_pages(struct drbd_peer_device *peer_device, unsigned int number,
- bool retry)
+struct page *drbd_alloc_pages(struct drbd_transport *transport, gfp_t gfp_mask, unsigned int size)
{
- struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+ int order = max(ilog2(size) - PAGE_SHIFT, 0);
struct page *page;
- struct net_conf *nc;
+
+ if (order && drbd_insert_fault_conn(connection, DRBD_FAULT_BIO_TOO_SMALL))
+ order = 0;
+
+ page = __drbd_alloc_pages(connection, gfp_mask | __GFP_NOWARN, order);
+ if (!page && order)
+ page = __drbd_alloc_pages(connection, gfp_mask, 0);
+
+ return page;
+}
+EXPORT_SYMBOL(drbd_alloc_pages); /* for transports */
+
+/**
+ * drbd_alloc_pages_split() - Allocate a physically contiguous, non-compound region
+ * @transport: DRBD transport
+ * @gfp_mask: how to allocate
+ * @order: in/out: on entry the desired (maximum) order, on return the order
+ * actually allocated
+ *
+ * Allocates 2^*order physically contiguous pages and runs split_page() so that
+ * each constituent page becomes an independently refcounted order-0 page.
+ *
+ * Tries the requested order first and falls back to successively smaller orders
+ * when the high-order, physically-contiguous allocation fails.
+ * The caller uses *order to learn how large a region it actually got, so it can
+ * fill its receive window with as few, as-large-as-possible regions.
+ *
+ * The same max_buffers throttle as __drbd_alloc_pages() applies.
+ *
+ * Returns the head page (representing 1 << *order linear pages), or NULL.
+ */
+struct page *drbd_alloc_pages_split(struct drbd_transport *transport, gfp_t gfp_mask, int *order)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+ struct page *page = NULL;
unsigned int mxb;
+ int o = *order;
rcu_read_lock();
- nc = rcu_dereference(peer_device->connection->net_conf);
- mxb = nc ? nc->max_buffers : 1000000;
+ mxb = rcu_dereference(connection->transport.net_conf)->max_buffers;
rcu_read_unlock();
- if (atomic_read(&device->pp_in_use) >= mxb)
+ if (atomic_read(&connection->pp_in_use) >= mxb)
schedule_timeout_interruptible(HZ / 10);
- page = __drbd_alloc_pages(number);
- if (page)
- atomic_add(number, &device->pp_in_use);
+ for (; o > 0; o--) {
+ page = alloc_pages(gfp_mask | __GFP_NORETRY | __GFP_NOWARN, o);
+ if (page)
+ break;
+ }
+ if (!page) {
+ o = 0;
+ page = alloc_pages(gfp_mask | __GFP_NOWARN, 0);
+ if (!page)
+ return NULL;
+ }
+
+ if (o)
+ split_page(page, o);
+
+ atomic_add(1 << o, &connection->pp_in_use);
+ *order = o;
+
return page;
}
+EXPORT_SYMBOL(drbd_alloc_pages_split); /* for transports */
-/* Must not be used from irq, as that may deadlock: see drbd_alloc_pages.
- * Is also used from inside an other spin_lock_irq(&resource->req_lock);
- * Either links the page chain back to the global pool,
+/* Must not be used from irq, as that may deadlock: see drbd_alloc_pages().
+ * Either links the page chain back to the pool of free pages,
* or returns all pages to the system. */
-static void drbd_free_pages(struct drbd_device *device, struct page *page)
+void drbd_free_page(struct drbd_transport *transport, struct page *page)
{
- struct page *tmp;
- int i = 0;
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+ int order = compound_order(page), i = 0;
if (page == NULL)
return;
- page_chain_for_each_safe(page, tmp) {
- set_page_private(page, 0);
- if (page_count(page) == 1)
- mempool_free(page, &drbd_buffer_page_pool);
- else
- put_page(page);
- i++;
- }
- i = atomic_sub_return(i, &device->pp_in_use);
+ if (page_count(page) == 1 && order == 0)
+ mempool_free(page, &drbd_buffer_page_pool);
+ else
+ put_page(page);
+
+ i = atomic_sub_return(1 << order, &connection->pp_in_use);
if (i < 0)
- drbd_warn(device, "ASSERTION FAILED: pp_in_use: %d < 0\n", i);
+ drbd_warn(connection, "ASSERTION FAILED: pp_in_use: %d < 0\n", i);
}
+EXPORT_SYMBOL(drbd_free_page);
-/*
-You need to hold the req_lock:
- _drbd_wait_ee_list_empty()
-
-You must not have the req_lock:
- drbd_free_peer_req()
- drbd_alloc_peer_req()
- drbd_free_peer_reqs()
- drbd_ee_fix_bhs()
- drbd_finish_peer_reqs()
- drbd_clear_done_ee()
- drbd_wait_ee_list_empty()
-*/
+/* Take an additional reference on a page from drbd_alloc_pages() /
+ * drbd_alloc_pages_split(), to be released with drbd_free_page(). pp_in_use
+ * counts references, so a page shared by two holders counts twice while
+ * shared. Safe from irq context.
+ */
+void drbd_get_page(struct drbd_transport *transport, struct page *page)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+
+ get_page(page);
+ atomic_add(1 << compound_order(page), &connection->pp_in_use);
+}
+EXPORT_SYMBOL(drbd_get_page);
+
+static int
+peer_req_alloc_bio(struct drbd_peer_request *peer_req, size_t size, gfp_t gfp_mask, blk_opf_t opf)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_transport *transport = &peer_device->connection->transport;
+ struct drbd_device *device = peer_device->device;
+ enum req_op op = opf & REQ_OP_MASK;
+ unsigned short nr_vecs;
+ struct page *page;
+ struct bio *bio;
+
+ nr_vecs = DIV_ROUND_UP(size, PAGE_SIZE);
+ if (nr_vecs > BIO_MAX_VECS)
+ nr_vecs = BIO_MAX_VECS;
+
+ if (drbd_insert_fault(device, DRBD_FAULT_BIO_TOO_SMALL))
+ nr_vecs = DIV_ROUND_UP(nr_vecs, 4);
+
+ bio = bio_alloc_bioset(device->ldev->backing_bdev, nr_vecs, opf, gfp_mask,
+ &drbd_peer_bio_set);
+ if (!bio)
+ return -ENOMEM;
+
+ bio_list_add(&peer_req->bios, bio);
-/* normal: payload_size == request size (bi_size)
- * w_same: payload_size == logical_block_size
- * trim: payload_size == 0 */
+ if (op == REQ_OP_READ) {
+ while (size) {
+ int len;
+
+ page = drbd_alloc_pages(transport, gfp_mask, size);
+ if (!page)
+ goto out_free_pages;
+ len = min(PAGE_SIZE << compound_order(page), size);
+
+ len = drbd_bio_add_page(transport, &peer_req->bios, page, len, 0);
+ if (len < 0)
+ goto out_free_pages;
+ size -= len;
+ }
+ if (!mempool_is_saturated(&drbd_buffer_page_pool))
+ peer_req->flags |= EE_RELEASE_TO_MEMPOOL;
+ }
+ return 0;
+
+out_free_pages:
+ drbd_peer_req_strip_bio(peer_req);
+ return -ENOMEM;
+}
+
+/**
+ * drbd_alloc_peer_req() - Allocate a drbd_peer_request
+ * @peer_device: peer device object
+ * @gfp_mask: how to allocate and whether to loop until we succeed
+ * @size: size (normal I/O), logical_block_size (w_same), 0 (trim)
+ * @opf: REQ_OP_READ or REQ_OP_WRITE
+ *
+ * For REQ_OP_READ, it allocates the peer_req with a BIO and populates it
+ * entirely with buffer pages. Otherwise it allocates the peer_req with
+ * an empty BIO.
+ *
+ * Return: the new peer request, or NULL if the allocation failed.
+ */
struct drbd_peer_request *
-drbd_alloc_peer_req(struct drbd_peer_device *peer_device, u64 id, sector_t sector,
- unsigned int request_size, unsigned int payload_size, gfp_t gfp_mask) __must_hold(local)
+drbd_alloc_peer_req(struct drbd_peer_device *peer_device, gfp_t gfp_mask,
+ size_t size, blk_opf_t opf)
{
struct drbd_device *device = peer_device->device;
struct drbd_peer_request *peer_req;
- struct page *page = NULL;
- unsigned int nr_pages = PFN_UP(payload_size);
+ int err;
+ gfp_mask &= ~__GFP_HIGHMEM;
if (drbd_insert_fault(device, DRBD_FAULT_AL_EE))
return NULL;
- peer_req = mempool_alloc(&drbd_ee_mempool, gfp_mask & ~__GFP_HIGHMEM);
+ peer_req = mempool_alloc(&drbd_ee_mempool, gfp_mask);
if (!peer_req) {
if (!(gfp_mask & __GFP_NOWARN))
drbd_err(device, "%s: allocation failed\n", __func__);
return NULL;
}
-
- if (nr_pages) {
- page = drbd_alloc_pages(peer_device, nr_pages,
- gfpflags_allow_blocking(gfp_mask));
- if (!page)
- goto fail;
- if (!mempool_is_saturated(&drbd_buffer_page_pool))
- peer_req->flags |= EE_RELEASE_TO_MEMPOOL;
- }
-
memset(peer_req, 0, sizeof(*peer_req));
+
INIT_LIST_HEAD(&peer_req->w.list);
drbd_clear_interval(&peer_req->i);
- peer_req->i.size = request_size;
- peer_req->i.sector = sector;
+ INIT_LIST_HEAD(&peer_req->recv_order);
peer_req->submit_jif = jiffies;
+ kref_get(&device->kref); /* this kref holds the peer_req->peer_device object alive */
peer_req->peer_device = peer_device;
- peer_req->pages = page;
- /*
- * The block_id is opaque to the receiver. It is not endianness
- * converted, and sent back to the sender unchanged.
- */
- peer_req->block_id = id;
+ peer_req->block_id = (unsigned long) peer_req;
+
+ if (opf == REQ_NO_BIO)
+ return peer_req;
+
+ err = peer_req_alloc_bio(peer_req, size, gfp_mask, opf);
+ if (err)
+ goto out_free_peer_req;
return peer_req;
- fail:
+out_free_peer_req:
+ kref_put(&device->kref, drbd_destroy_device);
mempool_free(peer_req, &drbd_ee_mempool);
return NULL;
}
-void drbd_free_peer_req(struct drbd_device *device, struct drbd_peer_request *peer_req)
+void drbd_free_peer_req(struct drbd_peer_request *peer_req)
{
- might_sleep();
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
+
+ if (peer_req->flags & EE_ON_RECV_ORDER) {
+ spin_lock_irq(&connection->peer_reqs_lock);
+ if (peer_req->i.type == INTERVAL_RESYNC_WRITE)
+ drbd_list_del_resync_request(peer_req);
+ else
+ list_del(&peer_req->recv_order);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ }
+ D_ASSERT(peer_device, !(peer_req->flags & EE_ON_SEND_OOS));
if (peer_req->flags & EE_HAS_DIGEST)
kfree(peer_req->digest);
- drbd_free_pages(device, peer_req->pages);
- D_ASSERT(device, atomic_read(&peer_req->pending_bios) == 0);
- D_ASSERT(device, drbd_interval_empty(&peer_req->i));
- if (!expect(device, !(peer_req->flags & EE_CALL_AL_COMPLETE_IO))) {
- peer_req->flags &= ~EE_CALL_AL_COMPLETE_IO;
- drbd_al_complete_io(device, &peer_req->i);
- }
+ D_ASSERT(peer_device, atomic_read(&peer_req->pending_bios) == 0);
+ D_ASSERT(peer_device, drbd_interval_empty(&peer_req->i));
+ drbd_peer_req_strip_bio(peer_req);
+ kref_put(&peer_device->device->kref, drbd_destroy_device);
mempool_free(peer_req, &drbd_ee_mempool);
}
-int drbd_free_peer_reqs(struct drbd_device *device, struct list_head *list)
+int drbd_free_peer_reqs(struct drbd_connection *connection, struct list_head *list)
{
LIST_HEAD(work_list);
struct drbd_peer_request *peer_req, *t;
int count = 0;
- spin_lock_irq(&device->resource->req_lock);
+ spin_lock_irq(&connection->peer_reqs_lock);
list_splice_init(list, &work_list);
- spin_unlock_irq(&device->resource->req_lock);
+ spin_unlock_irq(&connection->peer_reqs_lock);
list_for_each_entry_safe(peer_req, t, &work_list, w.list) {
- drbd_free_peer_req(device, peer_req);
+ drbd_free_peer_req(peer_req);
count++;
}
return count;
@@ -252,109 +663,139 @@ int drbd_free_peer_reqs(struct drbd_device *device, struct list_head *list)
/*
* See also comments in _req_mod(,BARRIER_ACKED) and receive_Barrier.
*/
-static int drbd_finish_peer_reqs(struct drbd_device *device)
+static int drbd_finish_peer_reqs(struct drbd_connection *connection)
{
LIST_HEAD(work_list);
struct drbd_peer_request *peer_req, *t;
int err = 0;
+ int n = 0;
- spin_lock_irq(&device->resource->req_lock);
- list_splice_init(&device->done_ee, &work_list);
- spin_unlock_irq(&device->resource->req_lock);
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_splice_init(&connection->done_ee, &work_list);
+ spin_unlock_irq(&connection->peer_reqs_lock);
/* possible callbacks here:
- * e_end_block, and e_end_resync_block, e_send_superseded.
+ * e_end_block, and e_end_resync_block.
* all ignore the last argument.
*/
list_for_each_entry_safe(peer_req, t, &work_list, w.list) {
int err2;
+ ++n;
/* list_del not necessary, next/prev members not touched */
+ /* The callback may free peer_req. */
err2 = peer_req->w.cb(&peer_req->w, !!err);
if (!err)
err = err2;
- drbd_free_peer_req(device, peer_req);
}
- wake_up(&device->ee_wait);
+ if (atomic_sub_and_test(n, &connection->done_ee_cnt))
+ wake_up(&connection->ee_wait);
return err;
}
-static void _drbd_wait_ee_list_empty(struct drbd_device *device,
- struct list_head *head)
+/* conn_connect() arms a data stream receive timeout to bound the connect;
+ * receive_state() disarms it per peer device, so a connection that has none
+ * keeps it over a data stream that is now legitimately idle. Disarm here
+ * instead, from the timeout it caused: a receive already waiting took its
+ * timeout when it started, so disarming elsewhere comes too late for it.
+ */
+static bool disarm_stale_connect_timeout(struct drbd_connection *connection)
{
- DEFINE_WAIT(wait);
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
- /* avoids spin_lock/unlock
- * and calling prepare_to_wait in the fast path */
- while (!list_empty(head)) {
- prepare_to_wait(&device->ee_wait, &wait, TASK_UNINTERRUPTIBLE);
- spin_unlock_irq(&device->resource->req_lock);
- io_schedule();
- finish_wait(&device->ee_wait, &wait);
- spin_lock_irq(&device->resource->req_lock);
- }
-}
+ if (connection->cstate[NOW] != C_CONNECTED)
+ return false;
-static void drbd_wait_ee_list_empty(struct drbd_device *device,
- struct list_head *head)
-{
- spin_lock_irq(&device->resource->req_lock);
- _drbd_wait_ee_list_empty(device, head);
- spin_unlock_irq(&device->resource->req_lock);
+ if (tr_ops->get_rcvtimeo(transport, DATA_STREAM) == MAX_SCHEDULE_TIMEOUT)
+ return false;
+
+ drbd_info(connection, "Idle data stream on an established connection; "
+ "disarming the connect receive timeout\n");
+ tr_ops->set_rcvtimeo(transport, DATA_STREAM, MAX_SCHEDULE_TIMEOUT);
+
+ return true;
}
-static int drbd_recv_short(struct socket *sock, void *buf, size_t size, int flags)
+/* Before C_CONNECTED, the peer cannot announce a disconnect with a
+ * two-phase commit; closing the socket is its regular way to give up.
+ */
+static void conn_closed_by_peer(struct drbd_connection *connection)
{
- struct kvec iov = {
- .iov_base = buf,
- .iov_len = size,
- };
- struct msghdr msg = {
- .msg_flags = (flags ? flags : MSG_WAITALL | MSG_NOSIGNAL)
- };
- iov_iter_kvec(&msg.msg_iter, ITER_DEST, &iov, 1, size);
- return sock_recvmsg(sock, &msg, msg.msg_flags);
+ struct drbd_resource *resource = connection->resource;
+ unsigned long irq_flags;
+
+ /* lock_order_ok: no CS_SERIALIZE, state_sem is not taken */
+ begin_state_change(resource, &irq_flags, CS_HARD);
+ __change_cstate(connection, connection->cstate[NOW] == C_CONNECTING ?
+ C_TEAR_DOWN : C_BROKEN_PIPE);
+ end_state_change(resource, &irq_flags, NULL);
}
-static int drbd_recv(struct drbd_connection *connection, void *buf, size_t size)
+static int drbd_recv(struct drbd_connection *connection, void **buf, size_t size, int flags)
{
+ struct drbd_transport_ops *tr_ops = &connection->transport.class->ops;
int rv;
- rv = drbd_recv_short(connection->data.socket, buf, size, 0);
+retry:
+ rv = tr_ops->recv(&connection->transport, DATA_STREAM, buf, size, flags);
+
+ /* At most one retry: the disarm makes the second attempt untimed. */
+ if (rv == -EAGAIN && disarm_stale_connect_timeout(connection))
+ goto retry;
if (rv < 0) {
if (rv == -ECONNRESET)
drbd_info(connection, "sock was reset by peer\n");
else if (rv != -ERESTARTSYS)
- drbd_err(connection, "sock_recvmsg returned %d\n", rv);
+ drbd_info(connection, "sock_recvmsg returned %d\n", rv);
} else if (rv == 0) {
- if (test_bit(DISCONNECT_SENT, &connection->flags)) {
+ if (test_bit(DISCONNECT_EXPECTED, &connection->flags)) {
long t;
+
rcu_read_lock();
- t = rcu_dereference(connection->net_conf)->ping_timeo * HZ/10;
+ t = rcu_dereference(connection->transport.net_conf)->ping_timeo * HZ/10;
rcu_read_unlock();
- t = wait_event_timeout(connection->ping_wait, connection->cstate < C_WF_REPORT_PARAMS, t);
+ t = wait_event_timeout(connection->resource->state_wait,
+ connection->cstate[NOW] < C_CONNECTED, t);
if (t)
goto out;
}
drbd_info(connection, "sock was shut down by peer\n");
+ conn_closed_by_peer(connection);
+ goto out;
}
if (rv != size)
- conn_request_state(connection, NS(conn, C_BROKEN_PIPE), CS_HARD);
+ change_cstate(connection, C_BROKEN_PIPE, CS_HARD);
out:
return rv;
}
-static int drbd_recv_all(struct drbd_connection *connection, void *buf, size_t size)
+static int drbd_recv_into(struct drbd_connection *connection, void *buf, size_t size)
+{
+ int err;
+
+ err = drbd_recv(connection, &buf, size, CALLER_BUFFER);
+
+ if (err != size) {
+ if (err >= 0)
+ err = -EIO;
+ } else
+ err = 0;
+ return err;
+}
+
+static int drbd_recv_all(struct drbd_connection *connection, void **buf, size_t size)
{
int err;
- err = drbd_recv(connection, buf, size);
+ err = drbd_recv(connection, buf, size, 0);
+
if (err != size) {
if (err >= 0)
err = -EIO;
@@ -363,7 +804,7 @@ static int drbd_recv_all(struct drbd_connection *connection, void *buf, size_t s
return err;
}
-static int drbd_recv_all_warn(struct drbd_connection *connection, void *buf, size_t size)
+static int drbd_recv_all_warn(struct drbd_connection *connection, void **buf, size_t size)
{
int err;
@@ -373,628 +814,746 @@ static int drbd_recv_all_warn(struct drbd_connection *connection, void *buf, siz
return err;
}
-/* quoting tcp(7):
- * On individual connections, the socket buffer size must be set prior to the
- * listen(2) or connect(2) calls in order to have it take effect.
- * This is our wrapper to do so.
- */
-static void drbd_setbufsize(struct socket *sock, unsigned int snd,
- unsigned int rcv)
+static int drbd_send_disconnect(struct drbd_connection *connection)
{
- /* open coded SO_SNDBUF, SO_RCVBUF */
- if (snd) {
- sock->sk->sk_sndbuf = snd;
- sock->sk->sk_userlocks |= SOCK_SNDBUF_LOCK;
- }
- if (rcv) {
- sock->sk->sk_rcvbuf = rcv;
- sock->sk->sk_userlocks |= SOCK_RCVBUF_LOCK;
- }
+ if (connection->agreed_pro_version < 118)
+ return 0;
+
+ if (!conn_prepare_command(connection, 0, DATA_STREAM))
+ return -EIO;
+ return send_command(connection, -1, P_DISCONNECT, DATA_STREAM);
}
-static struct socket *drbd_try_connect(struct drbd_connection *connection)
+static void initialize_send_buffer(struct drbd_connection *connection, enum drbd_stream drbd_stream)
{
- const char *what;
- struct socket *sock;
- struct sockaddr_in6 src_in6;
- struct sockaddr_in6 peer_in6;
- struct net_conf *nc;
- int err, peer_addr_len, my_addr_len;
- int sndbuf_size, rcvbuf_size, connect_int;
- int disconnect_on_error = 1;
+ struct drbd_send_buffer *sbuf = &connection->send_buffer[drbd_stream];
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- if (!nc) {
- rcu_read_unlock();
- return NULL;
- }
- sndbuf_size = nc->sndbuf_size;
- rcvbuf_size = nc->rcvbuf_size;
- connect_int = nc->connect_int;
- rcu_read_unlock();
+ sbuf->unsent =
+ sbuf->pos = page_address(sbuf->page);
+ sbuf->allocated_size = 0;
+ sbuf->additional_size = 0;
+ sbuf->dead = false;
+}
+
+/* Gets called if a connection is established, or if a new minor gets created
+ in a connection */
+int drbd_connected(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ u64 weak_nodes = 0;
+ int err;
- my_addr_len = min_t(int, connection->my_addr_len, sizeof(src_in6));
- memcpy(&src_in6, &connection->my_addr, my_addr_len);
+ atomic_set(&peer_device->packet_seq, 0);
+ peer_device->peer_seq = 0;
- if (((struct sockaddr *)&connection->my_addr)->sa_family == AF_INET6)
- src_in6.sin6_port = 0;
- else
- ((struct sockaddr_in *)&src_in6)->sin_port = 0; /* AF_INET & AF_SCI */
+ if (device->resource->role[NOW] == R_PRIMARY)
+ weak_nodes = drbd_weak_nodes_device(device);
- peer_addr_len = min_t(int, connection->peer_addr_len, sizeof(src_in6));
- memcpy(&peer_in6, &connection->peer_addr, peer_addr_len);
+ err = drbd_send_sync_param(peer_device);
- what = "sock_create_kern";
- err = sock_create_kern(&init_net, ((struct sockaddr *)&src_in6)->sa_family,
- SOCK_STREAM, IPPROTO_TCP, &sock);
- if (err < 0) {
- sock = NULL;
- goto out;
+ if (!err)
+ err = drbd_send_enable_replication_next(peer_device);
+ if (!err)
+ err = drbd_send_sizes(peer_device, 0, 0);
+ if (!err)
+ err = drbd_send_uuids(peer_device, 0, weak_nodes);
+ if (!err) {
+ set_bit(INITIAL_STATE_SENT, peer_device->flags);
+ err = drbd_send_current_state(peer_device);
}
- sock->sk->sk_rcvtimeo =
- sock->sk->sk_sndtimeo = connect_int * HZ;
- drbd_setbufsize(sock, sndbuf_size, rcvbuf_size);
-
- /* explicitly bind to the configured IP as source IP
- * for the outgoing connections.
- * This is needed for multihomed hosts and to be
- * able to use lo: interfaces for drbd.
- * Make sure to use 0 as port number, so linux selects
- * a free one dynamically.
- */
- what = "bind before connect";
- err = sock->ops->bind(sock, (struct sockaddr_unsized *) &src_in6, my_addr_len);
- if (err < 0)
- goto out;
+ clear_bit(USE_DEGR_WFC_T, peer_device->flags);
+ clear_bit(RESIZE_PENDING, peer_device->flags);
+ mod_timer(&device->request_timer, jiffies + HZ); /* just start it here. */
+ return err;
+}
- /* connect may fail, peer not yet available.
- * stay C_WF_CONNECTION, don't go Disconnecting! */
- disconnect_on_error = 0;
- what = "connect";
- err = sock->ops->connect(sock, (struct sockaddr_unsized *) &peer_in6, peer_addr_len, 0);
+void conn_connect2(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
-out:
- if (err < 0) {
- if (sock) {
- sock_release(sock);
- sock = NULL;
- }
- switch (-err) {
- /* timeout, busy, signal pending */
- case ETIMEDOUT: case EAGAIN: case EINPROGRESS:
- case EINTR: case ERESTARTSYS:
- /* peer not (yet) available, network problem */
- case ECONNREFUSED: case ENETUNREACH:
- case EHOSTDOWN: case EHOSTUNREACH:
- disconnect_on_error = 0;
- break;
- default:
- drbd_err(connection, "%s failed, err = %d\n", what, err);
+ /*
+ * If a common primary is currently absent and we have a dagtag toward
+ * it, hand the peer our position in its change stream before the UUID
+ * and state packets below. drbd_uuid_compare() on the peer uses it to
+ * roll an equal-UUID both-dirty reconcile forward (the node further
+ * along the stream becomes the source). Feature-gated; a peer without
+ * the feature never receives it and falls back to a node-id choice.
+ */
+ if (connection->agreed_features & DRBD_FF_RECONCILE_RECONNECT) {
+ int lost_node_id = find_common_lost_primary_node_id(connection);
+ struct drbd_connection *lost_peer = lost_node_id == -1 ? NULL :
+ drbd_get_connection_by_node_id(connection->resource, lost_node_id);
+
+ if (lost_peer) {
+ if (!drbd_send_peer_dagtag(connection, lost_peer))
+ connection->reconcile_handshake.sent_lost_node = true;
+ kref_put(&lost_peer->kref, drbd_destroy_connection);
}
- if (disconnect_on_error)
- conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_HARD);
}
- return sock;
-}
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
-struct accept_wait_data {
- struct drbd_connection *connection;
- struct socket *s_listen;
- struct completion door_bell;
- void (*original_sk_state_change)(struct sock *sk);
+ kref_get(&device->kref);
-};
+ /* connection cannot go away: caller holds a reference. */
+ rcu_read_unlock();
-static void drbd_incoming_connection(struct sock *sk)
-{
- struct accept_wait_data *ad = sk->sk_user_data;
- void (*state_change)(struct sock *sk);
+ /* In the compatibility case with protocol version < 110, that
+ * is DRBD 8.4, we do not hold uuid_sem while exchanging the
+ * initial UUID and state packets. There is no need because
+ * there are no other peers which could interfere. */
+ if (connection->agreed_pro_version >= 110) {
+ down_read_non_owner(&device->uuid_sem);
+ set_bit(HOLDING_UUID_READ_LOCK, peer_device->flags);
+ /* since drbd_connected() is also called from drbd_create_device()
+ aquire lock here before calling drbd_connected(). */
+ }
+ drbd_connected(peer_device);
- state_change = ad->original_sk_state_change;
- if (sk->sk_state == TCP_ESTABLISHED)
- complete(&ad->door_bell);
- state_change(sk);
+ rcu_read_lock();
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+ rcu_read_unlock();
+ drbd_uncork(connection, DATA_STREAM);
}
-static int prepare_listen_socket(struct drbd_connection *connection, struct accept_wait_data *ad)
+static bool initial_states_received(struct drbd_connection *connection)
{
- int err, sndbuf_size, rcvbuf_size, my_addr_len;
- struct sockaddr_in6 my_addr;
- struct socket *s_listen;
- struct net_conf *nc;
- const char *what;
-
+ struct drbd_peer_device *peer_device;
+ int vnr;
+ bool rv = true;
+
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- if (!nc) {
- rcu_read_unlock();
- return -EIO;
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (!test_bit(INITIAL_STATE_RECEIVED, peer_device->flags)) {
+ rv = false;
+ break;
+ }
}
- sndbuf_size = nc->sndbuf_size;
- rcvbuf_size = nc->rcvbuf_size;
rcu_read_unlock();
- my_addr_len = min_t(int, connection->my_addr_len, sizeof(struct sockaddr_in6));
- memcpy(&my_addr, &connection->my_addr, my_addr_len);
-
- what = "sock_create_kern";
- err = sock_create_kern(&init_net, ((struct sockaddr *)&my_addr)->sa_family,
- SOCK_STREAM, IPPROTO_TCP, &s_listen);
- if (err) {
- s_listen = NULL;
- goto out;
- }
-
- s_listen->sk->sk_reuse = SK_CAN_REUSE; /* SO_REUSEADDR */
- drbd_setbufsize(s_listen, sndbuf_size, rcvbuf_size);
-
- what = "bind before listen";
- err = s_listen->ops->bind(s_listen, (struct sockaddr_unsized *)&my_addr, my_addr_len);
- if (err < 0)
- goto out;
+ return rv;
+}
- ad->s_listen = s_listen;
- write_lock_bh(&s_listen->sk->sk_callback_lock);
- ad->original_sk_state_change = s_listen->sk->sk_state_change;
- s_listen->sk->sk_state_change = drbd_incoming_connection;
- s_listen->sk->sk_user_data = ad;
- write_unlock_bh(&s_listen->sk->sk_callback_lock);
+void wait_initial_states_received(struct drbd_connection *connection)
+{
+ struct drbd_net_conf *nc;
+ long timeout;
- what = "listen";
- err = s_listen->ops->listen(s_listen, 5);
- if (err < 0)
- goto out;
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ timeout = nc->ping_timeo * HZ/10;
+ rcu_read_unlock();
+ wait_event_interruptible_timeout(connection->ee_wait,
+ initial_states_received(connection),
+ timeout);
+}
- return 0;
-out:
- if (s_listen)
- sock_release(s_listen);
- if (err < 0) {
- if (err != -EAGAIN && err != -EINTR && err != -ERESTARTSYS) {
- drbd_err(connection, "%s failed, err = %d\n", what, err);
- conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_HARD);
- }
- }
+void connect_timer_fn(struct timer_list *t)
+{
+ struct drbd_connection *connection = timer_container_of(connection, t, connect_timer);
- return -EIO;
+ drbd_queue_work(&connection->sender_work, &connection->connect_timer_work);
}
-static void unregister_state_change(struct sock *sk, struct accept_wait_data *ad)
+static void arm_connect_timer(struct drbd_connection *connection, unsigned long expires)
{
- write_lock_bh(&sk->sk_callback_lock);
- sk->sk_state_change = ad->original_sk_state_change;
- sk->sk_user_data = NULL;
- write_unlock_bh(&sk->sk_callback_lock);
+ bool was_pending = mod_timer(&connection->connect_timer, expires);
+
+ if (was_pending) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
}
-static struct socket *drbd_wait_for_connect(struct drbd_connection *connection, struct accept_wait_data *ad)
+static bool retry_by_rr_conflict(struct drbd_connection *connection)
{
- int timeo, connect_int, err = 0;
- struct socket *s_estab = NULL;
- struct net_conf *nc;
+ enum drbd_after_sb_p rr_conflict;
+ struct drbd_net_conf *nc;
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- if (!nc) {
- rcu_read_unlock();
- return NULL;
- }
- connect_int = nc->connect_int;
+ nc = rcu_dereference(connection->transport.net_conf);
+ rr_conflict = nc->rr_conflict;
rcu_read_unlock();
- timeo = connect_int * HZ;
- /* 28.5% random jitter */
- timeo += get_random_u32_below(2) ? timeo / 7 : -timeo / 7;
-
- err = wait_for_completion_interruptible_timeout(&ad->door_bell, timeo);
- if (err <= 0)
- return NULL;
+ return rr_conflict == ASB_RETRY_CONNECT;
+}
- err = kernel_accept(ad->s_listen, &s_estab, 0);
- if (err < 0) {
- if (err != -EAGAIN && err != -EINTR && err != -ERESTARTSYS) {
- drbd_err(connection, "accept failed, err = %d\n", err);
- conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_HARD);
- }
- }
+static void apply_local_state_change(struct drbd_connection *connection, enum ao_op ao_op, bool force_demote)
+{
+ /* Although the connect failed, outdate local disks if we learn from the
+ * handshake that the peer has more recent data */
+ struct drbd_resource *resource = connection->resource;
+ unsigned long irq_flags;
+ int vnr;
- if (s_estab)
- unregister_state_change(s_estab->sk, ad);
+ mutex_lock(&resource->open_release);
+ begin_state_change(resource, &irq_flags, CS_HARD | (force_demote ? CS_FS_IGN_OPENERS : 0));
+ if (ao_op == OUTDATE_DISKS_AND_DISCONNECT)
+ __change_cstate(connection, C_DISCONNECTING);
+ if (resource->role[NOW] == R_SECONDARY ||
+ (resource->cached_susp && (
+ resource->res_opts.on_no_data == OND_IO_ERROR ||
+ resource->res_opts.on_susp_primary_outdated == SPO_FORCE_SECONDARY))) {
+ /* One day we might relax the above condition to
+ * resource->role[NOW] == R_SECONDARY || resource->cached_susp
+ * Right now it is that way, because we do not offer a way to gracefully
+ * get out of a Primary/Outdated state */
+ struct drbd_peer_device *peer_device;
+ bool set_fail_io = false;
- return s_estab;
-}
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ enum drbd_repl_state r = peer_device->connect_state.conn;
+ struct drbd_device *device = peer_device->device;
-static int decode_header(struct drbd_connection *, void *, struct packet_info *);
+ if (r == L_WF_BITMAP_T || r == L_SYNC_TARGET || r == L_PAUSED_SYNC_T)
+ __change_disk_state(device, D_OUTDATED);
-static int send_first_packet(struct drbd_connection *connection, struct drbd_socket *sock,
- enum drbd_packet cmd)
-{
- if (!conn_prepare_command(connection, sock))
- return -EIO;
- return conn_send_command(connection, sock, cmd, 0, NULL, 0);
+ if (device->open_cnt)
+ set_fail_io = true;
+ }
+ if (resource->role[NOW] == R_PRIMARY && force_demote) {
+ drbd_warn(connection, "Remote node has more recent data;"
+ " force secondary!\n");
+ resource->role[NEW] = R_SECONDARY;
+ if (set_fail_io)
+ resource->fail_io[NEW] = true;
+ }
+ }
+ end_state_change(resource, &irq_flags, "connect-failed");
+ mutex_unlock(&resource->open_release);
}
-static int receive_first_packet(struct drbd_connection *connection, struct socket *sock)
+/* Without the initial state of every volume finish_nested_twopc() never sets
+ * CONN_HANDSHAKE_READY, and the connect runs to twopc-timeout for good.
+ */
+static void report_missing_initial_states(struct drbd_connection *connection)
{
- unsigned int header_size = drbd_header_size(connection);
- struct packet_info pi;
- struct net_conf *nc;
- int err;
+ struct drbd_peer_device *peer_device;
+ int vnr;
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- if (!nc) {
- rcu_read_unlock();
- return -EIO;
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (!test_bit(INITIAL_STATE_RECEIVED, peer_device->flags))
+ drbd_warn(connection,
+ "Connect cannot complete: the peer sent no state for volume %d\n",
+ vnr);
}
- sock->sk->sk_rcvtimeo = nc->ping_timeo * 4 * HZ / 10;
rcu_read_unlock();
-
- err = drbd_recv_short(sock, connection->data.rbuf, header_size, 0);
- if (err != header_size) {
- if (err >= 0)
- err = -EIO;
- return err;
- }
- err = decode_header(connection, connection->data.rbuf, &pi);
- if (err)
- return err;
- return pi.cmd;
}
-/**
- * drbd_socket_okay() - Free the socket if its connection is not okay
- * @sock: pointer to the pointer to the socket.
+/* Ends a burst of connect attempts: the transport session is given up, so
+ * connect-int becomes the cool-down instead of another 50 ms re-arm.
*/
-static bool drbd_socket_okay(struct socket **sock)
-{
- int rr;
- char tb[4];
+#define CONNECT_TRIES_PER_SESSION 10
- if (!*sock)
- return false;
+static unsigned long connect_retry_delay(struct drbd_connection *connection,
+ unsigned long elapsed)
+{
+ /* Idle at least as long as the attempt was busy, so a connect that keeps
+ * failing cannot hold the transaction more than half of the time.
+ */
+ return clamp_t(unsigned long, elapsed, HZ / 20,
+ twopc_timeout(connection->resource));
+}
- rr = drbd_recv_short(*sock, tb, 4, MSG_DONTWAIT | MSG_PEEK);
+static bool connect_retry_allowed(struct drbd_connection *connection,
+ enum drbd_state_rv rv, unsigned long elapsed)
+{
+ /* Yielding to a concurrent state change gives up before the prepare, so
+ * such an attempt costs nothing and does not spend one of the tries.
+ */
+ if (rv == SS_CONCURRENT_ST_CHG && elapsed < HZ / 20)
+ return true;
- if (rr > 0 || rr == -EAGAIN) {
+ if (++connection->connect_tries <= CONNECT_TRIES_PER_SESSION)
return true;
- } else {
- sock_release(*sock);
- *sock = NULL;
- return false;
- }
+
+ drbd_info(connection, "Connect keeps failing (%s); retrying after connect-int\n",
+ drbd_set_st_err_str(rv));
+ return false;
}
-static bool connection_established(struct drbd_connection *connection,
- struct socket **sock1,
- struct socket **sock2)
+static int connect_work(struct drbd_work *work, int cancel)
{
- struct net_conf *nc;
- int timeout;
- bool ok;
+ struct drbd_connection *connection =
+ container_of(work, struct drbd_connection, connect_timer_work);
+ struct drbd_resource *resource = connection->resource;
+ enum drbd_state_rv rv;
+ long t = resource->res_opts.auto_promote_timeout * HZ / 10;
+ bool retry = retry_by_rr_conflict(connection);
+ bool incompat_states, force_demote;
+ unsigned long start_time;
- if (!*sock1 || !*sock2)
- return false;
+ if (connection->cstate[NOW] != C_CONNECTING)
+ goto out_put;
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- timeout = (nc->sock_check_timeo ?: nc->ping_timeo) * HZ / 10;
- rcu_read_unlock();
- schedule_timeout_interruptible(timeout);
+ if (connection->agreed_pro_version == 117)
+ wait_initial_states_received(connection);
+
+ start_time = jiffies;
+ do {
+ /* Carefully check if it is okay to do a two_phase_commit from sender context */
+ if (down_trylock(&resource->state_sem)) {
+ rv = SS_CONCURRENT_ST_CHG;
+ break;
+ }
+ rv = change_cstate_tag(connection, C_CONNECTED, CS_SERIALIZE |
+ CS_ALREADY_SERIALIZED | CS_VERBOSE | CS_DONT_RETRY,
+ "connected", NULL);
+ up(&resource->state_sem);
+ if (rv != SS_PRIMARY_READER)
+ break;
- ok = drbd_socket_okay(sock1);
- ok = drbd_socket_okay(sock2) && ok;
+ /* We have a connection established, peer is primary. On my side is a
+ read-only opener, probably udev or some other scanning after device creating.
+ This short lived read-only open prevents now that we can continue.
+ Better retry after the read-only opener goes away. */
+
+ t = wait_event_interruptible_timeout(resource->state_wait,
+ !drbd_open_ro_count(resource),
+ t);
+ } while (t > 0);
+
+ incompat_states = (rv == SS_CW_FAILED_BY_PEER || rv == SS_TWO_PRIMARIES);
+ force_demote = resource->role[NOW] == R_PRIMARY &&
+ resource->res_opts.on_susp_primary_outdated == SPO_FORCE_SECONDARY;
+ retry = retry || force_demote;
+
+ if (rv >= SS_SUCCESS) {
+ if (connection->agreed_pro_version < 117)
+ conn_connect2(connection);
+ } else if (rv == SS_TIMEOUT || rv == SS_CONCURRENT_ST_CHG) {
+ unsigned long elapsed = jiffies - start_time;
+
+ if (connection->cstate[NOW] != C_CONNECTING)
+ goto out_put;
+ /* Only after a timeout: SS_CONCURRENT_ST_CHG gives up before the
+ * prepare, so no state can have arrived for this attempt.
+ */
+ if (rv == SS_TIMEOUT)
+ report_missing_initial_states(connection);
+ if (connect_retry_allowed(connection, rv, elapsed)) {
+ arm_connect_timer(connection, jiffies +
+ connect_retry_delay(connection, elapsed));
+ return 0; /* Return early. Keep the reference on the connection! */
+ }
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ } else if (rv == SS_HANDSHAKE_RETRY || (incompat_states && retry)) {
+ bool again = connect_retry_allowed(connection, rv, jiffies - start_time);
+
+ if (again)
+ arm_connect_timer(connection, jiffies + HZ);
+ apply_local_state_change(connection, OUTDATE_DISKS, force_demote);
+ if (again)
+ return 0; /* Keep reference */
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ } else if (rv == SS_HANDSHAKE_DISCONNECT || (incompat_states && !retry)) {
+ drbd_send_disconnect(connection);
+ apply_local_state_change(connection, OUTDATE_DISKS_AND_DISCONNECT, force_demote);
+ } else {
+ drbd_info(connection, "Failure to connect: %s (%d); retrying\n",
+ drbd_set_st_err_str(rv), rv);
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ }
- return ok;
+ out_put:
+ kref_put(&connection->kref, drbd_destroy_connection);
+ return 0;
}
-/* Gets called if a connection is established, or if a new minor gets created
- in a connection */
-int drbd_connected(struct drbd_peer_device *peer_device)
+static int drbd_transport_connect(struct drbd_connection *connection)
{
- struct drbd_device *device = peer_device->device;
- int err;
-
- atomic_set(&device->packet_seq, 0);
- device->peer_seq = 0;
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_resource *resource = connection->resource;
+ int err = 0;
- device->state_mutex = peer_device->connection->agreed_pro_version < 100 ?
- &peer_device->connection->cstate_mutex :
- &device->own_state_mutex;
+ mutex_lock(&resource->conf_update);
+ err = transport->class->ops.prepare_connect(transport);
+ mutex_unlock(&resource->conf_update);
- err = drbd_send_sync_param(peer_device);
if (!err)
- err = drbd_send_sizes(peer_device, 0, 0);
- if (!err)
- err = drbd_send_uuids(peer_device);
- if (!err)
- err = drbd_send_current_state(peer_device);
- clear_bit(USE_DEGR_WFC_T, &device->flags);
- clear_bit(RESIZE_PENDING, &device->flags);
- atomic_set(&device->ap_in_flight, 0);
- mod_timer(&device->request_timer, jiffies + HZ); /* just start it here. */
+ err = transport->class->ops.connect(transport);
+
+ mutex_lock(&resource->conf_update);
+ transport->class->ops.finish_connect(transport);
+ mutex_unlock(&resource->conf_update);
+
return err;
}
-/*
- * return values:
- * 1 yes, we have a valid connection
- * 0 oops, did not work out, please try again
- * -1 peer talks different language,
- * no point in trying again, please go standalone.
- * -2 We do not have a network config...
+/* A peer returning while this volume owes a new data generation for writes that
+ * already completed is accepted at the still equal current UUID and then adopts
+ * the generation that is minted a moment later: once it is connected
+ * drbd_weak_nodes_device() no longer counts it weak, so no bitmap slot is
+ * stamped toward it, and the writes it missed survive only as other replicas'
+ * out-of-sync bits, which order no resync at equal current UUIDs. Order the
+ * local handshake behind the mint, so the UUIDs sent below carry it and the
+ * returner resyncs. A latent obligation owes nothing to a returner -- the data
+ * is identical -- and is left to the next write.
+ *
+ * Sleeps. Called from the connection's receiver thread while the peer is still
+ * C_CONNECTING and D_UNKNOWN, holding no lock.
*/
-static int conn_connect(struct drbd_connection *connection)
+static void device_mint_before_handshake(struct drbd_device *device, long timeout)
{
- struct drbd_socket sock, msock;
- struct drbd_peer_device *peer_device;
- struct net_conf *nc;
- int vnr, timeout, h;
- bool discard_my_data, ok;
- enum drbd_state_rv rv;
- struct accept_wait_data ad = {
- .connection = connection,
- .door_bell = COMPLETION_INITIALIZER_ONSTACK(ad.door_bell),
- };
+ switch (drbd_gen_obligation_state(device)) {
+ case GEN_OBL_ARMED:
+ /* Latent: no completed write is missing on the returner, so it is
+ * data identical at the equal current UUID -- no hold, no mint.
+ */
+ if (!drbd_gen_obligation_materialized(device))
+ break;
+ /* Materialized, and still armed: its mandatory mint failed at the
+ * completing edge. Retry it here, in the receiver thread, before
+ * the UUID exchange; the executor's exit decides what remains.
+ */
+ if (drbd_gen_obligation_mint_start(device))
+ drbd_gen_obligation_mint_run(device);
+ break;
+ case GEN_OBL_MINTING:
+ /* Another consumer is evaluating this obligation; wait for its
+ * outcome. On timeout proceed -- the connect retry loop comes
+ * around again.
+ */
+ wait_event_timeout(device->misc_wait,
+ drbd_gen_obligation_state(device) != GEN_OBL_MINTING,
+ timeout);
+ break;
+ case GEN_OBL_UNCONFIRMED:
+ /* Wait for the confirmation, but proceed when it does not come:
+ * it may be one only this peer can give. A diskless primary
+ * whose sole peer is the returner needs that peer's barrier ack,
+ * which needs this connect to complete -- an unbounded wait
+ * would deadlock against it. A returner one generation behind
+ * is then handled by the diskless reconnect handshake, which
+ * replays it forward or parks it until a resync.
+ */
+ wait_event_timeout(device->misc_wait,
+ drbd_gen_obligation_state(device) != GEN_OBL_UNCONFIRMED,
+ timeout);
+ break;
+ case GEN_OBL_PARKED:
+ /* No hold: while writers fail fast no write completes, so a
+ * returner at an equal current UUID is data identical, and the
+ * mint of the unpark reaches it in order once connected.
+ */
+ break;
+ case GEN_OBL_NONE:
+ case GEN_OBL_DISCHARGED:
+ break;
+ }
+}
- clear_bit(DISCONNECT_SENT, &connection->flags);
- if (conn_request_state(connection, NS(conn, C_WF_CONNECTION), CS_VERBOSE) < SS_SUCCESS)
- return -2;
+static void conn_mint_before_handshake(struct drbd_connection *connection, long timeout)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
- mutex_init(&sock.mutex);
- sock.sbuf = connection->data.sbuf;
- sock.rbuf = connection->data.rbuf;
- sock.socket = NULL;
- mutex_init(&msock.mutex);
- msock.sbuf = connection->meta.sbuf;
- msock.rbuf = connection->meta.rbuf;
- msock.socket = NULL;
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
- /* Assume that the peer only understands protocol 80 until we know better. */
- connection->agreed_pro_version = 80;
+ kref_get(&device->kref);
- if (prepare_listen_socket(connection, &ad))
- return 0;
+ /* connection cannot go away: caller holds a reference. */
+ rcu_read_unlock();
- do {
- struct socket *s;
-
- s = drbd_try_connect(connection);
- if (s) {
- if (!sock.socket) {
- sock.socket = s;
- send_first_packet(connection, &sock, P_INITIAL_DATA);
- } else if (!msock.socket) {
- clear_bit(RESOLVE_CONFLICTS, &connection->flags);
- msock.socket = s;
- send_first_packet(connection, &msock, P_INITIAL_META);
- } else {
- drbd_err(connection, "Logic error in conn_connect()\n");
- goto out_release_sockets;
- }
- }
+ device_mint_before_handshake(device, timeout);
- if (connection_established(connection, &sock.socket, &msock.socket))
- break;
+ rcu_read_lock();
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+ rcu_read_unlock();
+}
-retry:
- s = drbd_wait_for_connect(connection, &ad);
- if (s) {
- int fp = receive_first_packet(connection, s);
- drbd_socket_okay(&sock.socket);
- drbd_socket_okay(&msock.socket);
- switch (fp) {
- case P_INITIAL_DATA:
- if (sock.socket) {
- drbd_warn(connection, "initial packet S crossed\n");
- sock_release(sock.socket);
- sock.socket = s;
- goto randomize;
- }
- sock.socket = s;
- break;
- case P_INITIAL_META:
- set_bit(RESOLVE_CONFLICTS, &connection->flags);
- if (msock.socket) {
- drbd_warn(connection, "initial packet M crossed\n");
- sock_release(msock.socket);
- msock.socket = s;
- goto randomize;
- }
- msock.socket = s;
- break;
- default:
- drbd_warn(connection, "Error receiving initial packet\n");
- sock_release(s);
-randomize:
- if (get_random_u32_below(2))
- goto retry;
- }
- }
+/*
+ * Returns true if we have a valid connection.
+ */
+static bool conn_connect(struct drbd_connection *connection)
+{
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_resource *resource = connection->resource;
+ int ping_timeo, ping_int, h, err, vnr, in_flight;
+ struct drbd_peer_device *peer_device;
+ enum drbd_stream stream;
+ struct drbd_net_conf *nc;
+ bool discard_my_data;
+ bool have_mutex;
+ bool no_addr = false;
+
+start:
+ have_mutex = false;
+ clear_bit(PING_PENDING, &connection->flags);
+ clear_bit(DISCONNECT_EXPECTED, &connection->flags);
+ /* A send that failed on the previous socket must not fail this attempt:
+ * C_CONNECTING lets other threads flush before the transport is up.
+ */
+ for (stream = DATA_STREAM; stream <= CONTROL_STREAM; stream++) {
+ mutex_lock(&connection->mutex[stream]);
+ initialize_send_buffer(connection, stream);
+ mutex_unlock(&connection->mutex[stream]);
+ }
+ if (change_cstate_tag(connection, C_CONNECTING, CS_VERBOSE, "connecting", NULL)
+ < SS_SUCCESS) {
+ /* We do not have a network config. */
+ return false;
+ }
- if (connection->cstate <= C_DISCONNECTING)
- goto out_release_sockets;
- if (signal_pending(current)) {
- flush_signals(current);
- smp_rmb();
- if (get_t_state(&connection->receiver) == EXITING)
- goto out_release_sockets;
- }
+ /* Assume that the peer only understands our minimum supported
+ * protocol version; until we know better. */
+ connection->agreed_pro_version = drbd_protocol_version_min;
- ok = connection_established(connection, &sock.socket, &msock.socket);
- } while (!ok);
+ err = drbd_transport_connect(connection);
+ if (err == -EAGAIN) {
+ enum drbd_conn_state cstate;
- if (ad.s_listen)
- sock_release(ad.s_listen);
+ read_lock_irq(&resource->state_rwlock); /* See commit message */
+ cstate = connection->cstate[NOW];
+ read_unlock_irq(&resource->state_rwlock);
+ if (cstate == C_DISCONNECTING)
+ return false;
+ goto retry;
+ } else if (err == -EADDRNOTAVAIL) {
+ struct drbd_net_conf *nc;
+ int connect_int;
+ long t;
- sock.socket->sk->sk_reuse = SK_CAN_REUSE; /* SO_REUSEADDR */
- msock.socket->sk->sk_reuse = SK_CAN_REUSE; /* SO_REUSEADDR */
+ rcu_read_lock();
+ nc = rcu_dereference(transport->net_conf);
+ connect_int = nc ? nc->connect_int : 10;
+ rcu_read_unlock();
- sock.socket->sk->sk_allocation = GFP_NOIO;
- msock.socket->sk->sk_allocation = GFP_NOIO;
+ if (!no_addr) {
+ drbd_warn(connection,
+ "Configured local address not found, retrying every %d sec, "
+ "err=%d\n", connect_int, err);
+ no_addr = true;
+ }
- sock.socket->sk->sk_use_task_frag = false;
- msock.socket->sk->sk_use_task_frag = false;
+ t = schedule_timeout_interruptible(connect_int * HZ);
+ if (t || connection->cstate[NOW] == C_DISCONNECTING)
+ return false;
+ goto start;
+ } else if (err == -EDESTADDRREQ) {
+ /*
+ * No destination address, we cannot possibly make a connection.
+ * Maybe a resource was partially left over due to some other bug?
+ * Either way, abort here and go StandAlone to prevent reconnection.
+ */
+ drbd_err(connection, "No destination address, err=%d\n", err);
+ change_cstate_tag(connection, C_STANDALONE, CS_HARD, "no-dest-addr", NULL);
+ return false;
+ } else if (err < 0) {
+ drbd_warn(connection, "Failed to initiate connection, err=%d\n", err);
+ goto abort;
+ }
- sock.socket->sk->sk_priority = TC_PRIO_INTERACTIVE_BULK;
- msock.socket->sk->sk_priority = TC_PRIO_INTERACTIVE;
+ connection->reassemble_buffer.avail = 0;
- /* NOT YET ...
- * sock.socket->sk->sk_sndtimeo = connection->net_conf->timeout*HZ/10;
- * sock.socket->sk->sk_rcvtimeo = MAX_SCHEDULE_TIMEOUT;
- * first set it to the P_CONNECTION_FEATURES timeout,
- * which we set to 4x the configured ping_timeout. */
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
-
- sock.socket->sk->sk_sndtimeo =
- sock.socket->sk->sk_rcvtimeo = nc->ping_timeo*4*HZ/10;
-
- msock.socket->sk->sk_rcvtimeo = nc->ping_int*HZ;
- timeout = nc->timeout * HZ / 10;
- discard_my_data = nc->discard_my_data;
+ nc = rcu_dereference(connection->transport.net_conf);
+ ping_timeo = nc->ping_timeo;
+ ping_int = nc->ping_int;
rcu_read_unlock();
- msock.socket->sk->sk_sndtimeo = timeout;
-
- /* we don't want delays.
- * we use TCP_CORK where appropriate, though */
- tcp_sock_set_nodelay(sock.socket->sk);
- tcp_sock_set_nodelay(msock.socket->sk);
+ /* Make sure we are "uncorked", otherwise we risk timeouts,
+ * in case this is a reconnect and we had been corked before. */
+ for (stream = DATA_STREAM; stream <= CONTROL_STREAM; stream++) {
+ initialize_send_buffer(connection, stream);
+ drbd_uncork(connection, stream);
+ }
- connection->data.socket = sock.socket;
- connection->meta.socket = msock.socket;
- connection->last_received = jiffies;
+ /* Make sure the handshake happens without interference from other threads,
+ * or the challenge response authentication could be garbled. */
+ mutex_lock(&connection->mutex[DATA_STREAM]);
+ have_mutex = true;
+ clear_bit(PING_TIMEOUT_ACTIVE, &connection->flags);
+ transport->class->ops.set_rcvtimeo(transport, DATA_STREAM, ping_timeo * 4 * HZ/10);
+ transport->class->ops.set_rcvtimeo(transport, CONTROL_STREAM, ping_int * HZ);
h = drbd_do_features(connection);
- if (h <= 0)
- return h;
+ if (h < 0)
+ goto abort;
+ if (h == 0)
+ goto retry;
if (connection->cram_hmac_tfm) {
- /* drbd_request_state(device, NS(conn, WFAuth)); */
switch (drbd_do_auth(connection)) {
case -1:
drbd_err(connection, "Authentication of peer failed\n");
- return -1;
+ goto abort;
case 0:
drbd_err(connection, "Authentication of peer failed, trying again.\n");
- return 0;
+ goto retry;
}
}
- connection->data.socket->sk->sk_sndtimeo = timeout;
- connection->data.socket->sk->sk_rcvtimeo = MAX_SCHEDULE_TIMEOUT;
-
- if (drbd_send_protocol(connection) == -EOPNOTSUPP)
- return -1;
-
- /* Prevent a race between resync-handshake and
- * being promoted to Primary.
- *
- * Grab and release the state mutex, so we know that any current
- * drbd_set_role() is finished, and any incoming drbd_set_role
- * will see the STATE_SENT flag, and wait for it to be cleared.
- */
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
- mutex_lock(peer_device->device->state_mutex);
-
- /* avoid a race with conn_request_state( C_DISCONNECTING ) */
- spin_lock_irq(&connection->resource->req_lock);
- set_bit(STATE_SENT, &connection->flags);
- spin_unlock_irq(&connection->resource->req_lock);
+ discard_my_data = test_bit(CONN_DISCARD_MY_DATA, &connection->flags);
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
- mutex_unlock(peer_device->device->state_mutex);
+ if (__drbd_send_protocol(connection, P_PROTOCOL) == -EOPNOTSUPP)
+ goto abort;
rcu_read_lock();
idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- kref_get(&device->kref);
- rcu_read_unlock();
-
- if (discard_my_data)
- set_bit(DISCARD_MY_DATA, &device->flags);
- else
- clear_bit(DISCARD_MY_DATA, &device->flags);
-
- drbd_connected(peer_device);
- kref_put(&device->kref, drbd_destroy_device);
- rcu_read_lock();
+ set_bit(REPLICATION_NEXT, peer_device->flags);
+ assign_bit(DISCARD_MY_DATA, peer_device->flags,
+ discard_my_data);
}
rcu_read_unlock();
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+ have_mutex = false;
- rv = conn_request_state(connection, NS(conn, C_WF_REPORT_PARAMS), CS_VERBOSE);
- if (rv < SS_SUCCESS || connection->cstate != C_WF_REPORT_PARAMS) {
- clear_bit(STATE_SENT, &connection->flags);
- return 0;
- }
-
- drbd_thread_start(&connection->ack_receiver);
- /* opencoded create_singlethread_workqueue(),
- * to be able to use format string arguments */
connection->ack_sender =
- alloc_ordered_workqueue("drbd_as_%s", WQ_MEM_RECLAIM, connection->resource->name);
+ alloc_ordered_workqueue("drbd_as_%s", WQ_MEM_RECLAIM, resource->name);
if (!connection->ack_sender) {
drbd_err(connection, "Failed to create workqueue ack_sender\n");
- return 0;
+ schedule_timeout_uninterruptible(HZ);
+ goto retry;
}
- mutex_lock(&connection->resource->conf_update);
- /* The discard_my_data flag is a single-shot modifier to the next
- * connection attempt, the handshake of which is now well underway.
- * No need for rcu style copying of the whole struct
- * just to clear a single value. */
- connection->net_conf->discard_my_data = 0;
- mutex_unlock(&connection->resource->conf_update);
+ atomic_set(&connection->rs_in_flight, 0);
+ /* A teardown while IO was suspended keeps requests for a resend, and
+ * their sectors stay counted until the resend sends them again. With
+ * no such request left, or with a count below zero, what is left is an
+ * accounting error from an earlier connection: do not carry it over.
+ */
+ in_flight = atomic_read(&connection->ap_in_flight);
+ if (in_flight < 0 || (in_flight > 0 && !READ_ONCE(connection->req_not_net_done))) {
+ drbd_warn(connection, "ap_in_flight = %d on connect, expected 0\n", in_flight);
+ atomic_set(&connection->ap_in_flight, 0);
+ }
+ clear_bit(DAGTAG_STREAM_GONE, &connection->flags);
- return h;
+ /* The last point before both the UUID exchange and arm_connect_timer():
+ * conn_connect2() sends the UUIDs, and on every path but the pre-110 one
+ * it runs inside the connect two-phase commit's prepare phase.
+ */
+ conn_mint_before_handshake(connection, ping_timeo * HZ / 10);
+
+ if (connection->agreed_pro_version >= 110) {
+ /* Allow 10 times the ping_timeo for two-phase commits. That is
+ * 5 seconds by default. The unit of ping_timeo is tenths of a
+ * second. */
+ transport->class->ops.set_rcvtimeo(transport, DATA_STREAM, ping_timeo * HZ);
+
+ if (connection->agreed_pro_version == 117)
+ conn_connect2(connection);
+
+ if (resource->res_opts.node_id < connection->peer_node_id) {
+ kref_get(&connection->kref);
+ connection->connect_timer_work.cb = connect_work;
+ connection->connect_tries = 0;
+ arm_connect_timer(connection, jiffies);
+ }
+ } else {
+ enum drbd_state_rv rv;
-out_release_sockets:
- if (ad.s_listen)
- sock_release(ad.s_listen);
- if (sock.socket)
- sock_release(sock.socket);
- if (msock.socket)
- sock_release(msock.socket);
- return -1;
+ rv = change_cstate(connection, C_CONNECTED,
+ CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE | CS_LOCAL_ONLY);
+ if (rv < SS_SUCCESS || connection->cstate[NOW] != C_CONNECTED)
+ goto retry;
+ conn_connect2(connection);
+ }
+
+ clear_bit(PING_TIMEOUT_ACTIVE, &connection->flags);
+ return true;
+
+retry:
+ if (have_mutex)
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+ conn_disconnect(connection);
+ schedule_timeout_interruptible(HZ);
+ drbd_thread_consume_restart(&connection->receiver);
+ goto start;
+
+abort:
+ if (have_mutex)
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+ change_cstate(connection, C_DISCONNECTING, CS_HARD);
+ return false;
}
-static int decode_header(struct drbd_connection *connection, void *header, struct packet_info *pi)
+static unsigned int decode_header_size(const void *header)
{
- unsigned int header_size = drbd_header_size(connection);
+ const u32 first_dword = *(u32 *)header;
+ const u16 first_word = *(u16 *)header;
- if (header_size == sizeof(struct p_header100) &&
- *(__be32 *)header == cpu_to_be32(DRBD_MAGIC_100)) {
- struct p_header100 *h = header;
- if (h->pad != 0) {
- drbd_err(connection, "Header padding is not zero\n");
- return -EINVAL;
- }
- pi->vnr = be16_to_cpu(h->volume);
+ return first_dword == cpu_to_be32(DRBD_MAGIC_100) ? sizeof(struct p_header100) :
+ first_word == cpu_to_be16(DRBD_MAGIC_BIG) ? sizeof(struct p_header95) :
+ sizeof(struct p_header80);
+}
+
+static int __decode_header(const void *header, struct packet_info *pi)
+{
+ const u32 first_dword = *(u32 *)header;
+ const u16 first_word = *(u16 *)header;
+ unsigned int header_size;
+ int header_version;
+
+ if (first_dword == cpu_to_be32(DRBD_MAGIC_100)) {
+ const struct p_header100 *h = header;
+ u16 vnr = be16_to_cpu(h->volume);
+
+ if (h->pad != 0)
+ return -ENOENT;
+
+ pi->vnr = vnr == ((u16) 0xFFFF) ? -1 : vnr;
pi->cmd = be16_to_cpu(h->command);
pi->size = be32_to_cpu(h->length);
- } else if (header_size == sizeof(struct p_header95) &&
- *(__be16 *)header == cpu_to_be16(DRBD_MAGIC_BIG)) {
- struct p_header95 *h = header;
+ header_size = sizeof(*h);
+ header_version = 100;
+ } else if (first_word == cpu_to_be16(DRBD_MAGIC_BIG)) {
+ const struct p_header95 *h = header;
+
pi->cmd = be16_to_cpu(h->command);
pi->size = be32_to_cpu(h->length);
pi->vnr = 0;
- } else if (header_size == sizeof(struct p_header80) &&
- *(__be32 *)header == cpu_to_be32(DRBD_MAGIC)) {
- struct p_header80 *h = header;
+ header_size = sizeof(*h);
+ header_version = 95;
+ } else if (first_dword == cpu_to_be32(DRBD_MAGIC)) {
+ const struct p_header80 *h = header;
+
pi->cmd = be16_to_cpu(h->command);
pi->size = be16_to_cpu(h->length);
pi->vnr = 0;
+ header_size = sizeof(*h);
+ header_version = 80;
} else {
- drbd_err(connection, "Wrong magic value 0x%08x in protocol version %d\n",
- be32_to_cpu(*(__be32 *)header),
- connection->agreed_pro_version);
return -EINVAL;
}
- pi->data = header + header_size;
+
+ pi->data = (void *)(header + header_size); /* casting away 'const'! */
+ return header_version;
+}
+
+static bool header_version_good(int header_version, int protocol_version)
+{
+ switch (header_version) {
+ case 100: return protocol_version >= 100;
+ case 95: return protocol_version < 100;
+ case 80: return protocol_version < 95;
+ default: return false;
+ }
+}
+
+static int decode_header(struct drbd_connection *connection, const void *header,
+ struct packet_info *pi)
+{
+ const int agreed_pro_version = connection->agreed_pro_version;
+ int header_version = __decode_header(header, pi);
+
+ if (header_version == -ENOENT) {
+ drbd_err(connection, "Header padding is not zero\n");
+ return -EINVAL;
+ } else if (header_version < 0 || !header_version_good(header_version, agreed_pro_version)) {
+ drbd_err(connection, "Wrong magic value 0x%08x in protocol version %d, %d [data]\n",
+ be32_to_cpu(*(__be32 *)header), agreed_pro_version, header_version);
+ return -EINVAL;
+ }
return 0;
}
@@ -1008,49 +1567,58 @@ static void drbd_unplug_all_devices(struct drbd_connection *connection)
static int drbd_recv_header(struct drbd_connection *connection, struct packet_info *pi)
{
- void *buffer = connection->data.rbuf;
+ void *buffer;
int err;
- err = drbd_recv_all_warn(connection, buffer, drbd_header_size(connection));
+ err = drbd_recv_all_warn(connection, &buffer, drbd_header_size(connection));
if (err)
return err;
err = decode_header(connection, buffer, pi);
- connection->last_received = jiffies;
return err;
}
static int drbd_recv_header_maybe_unplug(struct drbd_connection *connection, struct packet_info *pi)
{
- void *buffer = connection->data.rbuf;
+ struct drbd_transport_ops *tr_ops = &connection->transport.class->ops;
unsigned int size = drbd_header_size(connection);
+ void *buffer;
int err;
- err = drbd_recv_short(connection->data.socket, buffer, size, MSG_NOSIGNAL|MSG_DONTWAIT);
+ err = tr_ops->recv(&connection->transport, DATA_STREAM, &buffer,
+ size, MSG_NOSIGNAL | MSG_DONTWAIT);
if (err != size) {
+ int rflags = 0;
+
/* If we have nothing in the receive buffer now, to reduce
* application latency, try to drain the backend queues as
* quickly as possible, and let remote TCP know what we have
* received so far. */
if (err == -EAGAIN) {
- tcp_sock_set_quickack(connection->data.socket->sk, 2);
+ tr_ops->hint(&connection->transport, DATA_STREAM, QUICKACK);
drbd_unplug_all_devices(connection);
- }
- if (err > 0) {
- buffer += err;
+ } else if (err > 0) {
size -= err;
+ rflags |= GROW_BUFFER;
}
- err = drbd_recv_all_warn(connection, buffer, size);
+
+ err = drbd_recv(connection, &buffer, size, rflags);
+ if (err != size) {
+ if (err >= 0)
+ err = -EIO;
+ } else
+ err = 0;
+
if (err)
return err;
}
- err = decode_header(connection, connection->data.rbuf, pi);
- connection->last_received = jiffies;
+ err = decode_header(connection, buffer, pi);
return err;
}
+
/* This is blkdev_issue_flush, but asynchronous.
* We want to submit to all component volumes in parallel,
* then wait for all completions.
@@ -1071,9 +1639,11 @@ static void one_flush_endio(struct bio *bio)
struct drbd_device *device = octx->device;
struct issue_flush_context *ctx = octx->ctx;
- if (bio->bi_status) {
- ctx->error = blk_status_to_errno(bio->bi_status);
- drbd_info(device, "local disk FLUSH FAILED with status %d\n", bio->bi_status);
+ blk_status_t status = bio->bi_status;
+
+ if (status) {
+ ctx->error = blk_status_to_errno(status);
+ drbd_info(device, "local disk FLUSH FAILED with status %d\n", status);
}
kfree(octx);
bio_put(bio);
@@ -1089,7 +1659,7 @@ static void one_flush_endio(struct bio *bio)
static void submit_one_flush(struct drbd_device *device, struct issue_flush_context *ctx)
{
struct bio *bio = bio_alloc(device->ldev->backing_bdev, 0,
- REQ_OP_WRITE | REQ_PREFLUSH, GFP_NOIO);
+ REQ_OP_WRITE | REQ_PREFLUSH, GFP_NOIO);
struct one_flush_context *octx = kmalloc_obj(*octx, GFP_NOIO);
if (!octx) {
@@ -1116,10 +1686,12 @@ static void submit_one_flush(struct drbd_device *device, struct issue_flush_cont
submit_bio(bio);
}
-static void drbd_flush(struct drbd_connection *connection)
+static enum finish_epoch drbd_flush_after_epoch(struct drbd_connection *connection, struct drbd_epoch *epoch)
{
- if (connection->resource->write_ordering >= WO_BDEV_FLUSH) {
- struct drbd_peer_device *peer_device;
+ struct drbd_resource *resource = connection->resource;
+
+ if (resource->write_ordering >= WO_BDEV_FLUSH) {
+ struct drbd_device *device;
struct issue_flush_context ctx;
int vnr;
@@ -1128,9 +1700,7 @@ static void drbd_flush(struct drbd_connection *connection)
init_completion(&ctx.done);
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
-
+ idr_for_each_entry(&resource->devices, device, vnr) {
if (!get_ldev(device))
continue;
kref_get(&device->kref);
@@ -1155,6 +1725,88 @@ static void drbd_flush(struct drbd_connection *connection)
drbd_bump_write_ordering(connection->resource, NULL, WO_DRAIN_IO);
}
}
+
+ /* If called before sending P_CONFIRM_STABLE, we don't have the epoch
+ * (and must not finish it yet, anyways) */
+ if (epoch == NULL)
+ return FE_STILL_LIVE;
+ return drbd_may_finish_epoch(connection, epoch, EV_BARRIER_DONE);
+}
+
+static int w_flush(struct drbd_work *w, int cancel)
+{
+ struct flush_work *fw = container_of(w, struct flush_work, w);
+ struct drbd_epoch *epoch = fw->epoch;
+ struct drbd_connection *connection = epoch->connection;
+
+ kfree(fw);
+
+ if (!test_and_set_bit(DE_BARRIER_IN_NEXT_EPOCH_ISSUED, &epoch->flags))
+ drbd_flush_after_epoch(connection, epoch);
+
+ drbd_may_finish_epoch(connection, epoch, EV_PUT |
+ (connection->cstate[NOW] < C_CONNECTED ? EV_CLEANUP : 0));
+
+ return 0;
+}
+
+static void drbd_send_b_ack(struct drbd_connection *connection, u32 barrier_nr, u32 set_size)
+{
+ struct p_barrier_ack *p;
+
+ if (connection->cstate[NOW] < C_CONNECTED)
+ return;
+
+ p = conn_prepare_command(connection, sizeof(*p), CONTROL_STREAM);
+ if (!p)
+ return;
+ p->barrier = barrier_nr;
+ p->set_size = cpu_to_be32(set_size);
+ send_command(connection, -1, P_BARRIER_ACK, CONTROL_STREAM);
+}
+
+static void drbd_send_confirm_stable(struct drbd_peer_request *peer_req)
+{
+ struct drbd_connection *connection = peer_req->peer_device->connection;
+ struct drbd_epoch *epoch = peer_req->epoch;
+ struct drbd_peer_request *oldest, *youngest;
+ struct p_confirm_stable *p;
+ int count;
+
+ if (connection->cstate[NOW] < C_CONNECTED)
+ return;
+
+ /* peer_req is not on stable storage yet, but the only one in this epoch.
+ * Nothing to confirm, just wait for the normal barrier_ack and peer_ack
+ * to do their work. */
+ oldest = epoch->oldest_unconfirmed_peer_req;
+ if (oldest == peer_req)
+ return;
+
+ p = conn_prepare_command(connection, sizeof(*p), CONTROL_STREAM);
+ if (!p)
+ return;
+
+ /* peer_req has not been added to connection->peer_requests yet, so
+ * connection->peer_requests.prev is the youngest request that should
+ * now be on stable storage. */
+ spin_lock_irq(&connection->peer_reqs_lock);
+ youngest = list_entry(connection->peer_requests.prev, struct drbd_peer_request, recv_order);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ count = atomic_read(&epoch->epoch_size) - atomic_read(&epoch->confirmed) - 1;
+ atomic_add(count, &epoch->confirmed);
+ epoch->oldest_unconfirmed_peer_req = peer_req;
+
+ D_ASSERT(connection, oldest->epoch == youngest->epoch);
+ D_ASSERT(connection, count > 0);
+
+ p->oldest_block_id = oldest->block_id;
+ p->youngest_block_id = youngest->block_id;
+ p->set_size = cpu_to_be32(count);
+ p->pad = 0;
+
+ send_command(connection, -1, P_CONFIRM_STABLE, CONTROL_STREAM);
}
/**
@@ -1167,13 +1819,16 @@ static enum finish_epoch drbd_may_finish_epoch(struct drbd_connection *connectio
struct drbd_epoch *epoch,
enum epoch_event ev)
{
- int epoch_size;
+ int finish, epoch_size;
struct drbd_epoch *next_epoch;
+ int schedule_flush = 0;
enum finish_epoch rv = FE_STILL_LIVE;
+ struct drbd_resource *resource = connection->resource;
spin_lock(&connection->epoch_lock);
do {
next_epoch = NULL;
+ finish = 0;
epoch_size = atomic_read(&epoch->epoch_size);
@@ -1183,6 +1838,16 @@ static enum finish_epoch drbd_may_finish_epoch(struct drbd_connection *connectio
break;
case EV_GOT_BARRIER_NR:
set_bit(DE_HAVE_BARRIER_NUMBER, &epoch->flags);
+
+ /* Special case: If we just switched from WO_BIO_BARRIER to
+ WO_BDEV_FLUSH we should not finish the current epoch */
+ if (test_bit(DE_CONTAINS_A_BARRIER, &epoch->flags) && epoch_size == 1 &&
+ resource->write_ordering != WO_BIO_BARRIER &&
+ epoch == connection->current_epoch)
+ clear_bit(DE_CONTAINS_A_BARRIER, &epoch->flags);
+ break;
+ case EV_BARRIER_DONE:
+ set_bit(DE_BARRIER_IN_NEXT_EPOCH_DONE, &epoch->flags);
break;
case EV_BECAME_LAST:
/* nothing to do*/
@@ -1191,18 +1856,30 @@ static enum finish_epoch drbd_may_finish_epoch(struct drbd_connection *connectio
if (epoch_size != 0 &&
atomic_read(&epoch->active) == 0 &&
- (test_bit(DE_HAVE_BARRIER_NUMBER, &epoch->flags) || ev & EV_CLEANUP)) {
+ (test_bit(DE_HAVE_BARRIER_NUMBER, &epoch->flags) || ev & EV_CLEANUP) &&
+ epoch->list.prev == &connection->current_epoch->list &&
+ !test_bit(DE_IS_FINISHING, &epoch->flags)) {
+ /* Nearly all conditions are met to finish that epoch... */
+ if (test_bit(DE_BARRIER_IN_NEXT_EPOCH_DONE, &epoch->flags) ||
+ resource->write_ordering == WO_NONE ||
+ (epoch_size == 1 && test_bit(DE_CONTAINS_A_BARRIER, &epoch->flags)) ||
+ ev & EV_CLEANUP) {
+ finish = 1;
+ set_bit(DE_IS_FINISHING, &epoch->flags);
+ } else if (!test_bit(DE_BARRIER_IN_NEXT_EPOCH_ISSUED, &epoch->flags) &&
+ resource->write_ordering == WO_BIO_BARRIER) {
+ atomic_inc(&epoch->active);
+ schedule_flush = 1;
+ }
+ }
+ if (finish) {
if (!(ev & EV_CLEANUP)) {
+ /* adjust for nr requests already confirmed via P_CONFIRM_STABLE, if any. */
+ epoch_size -= atomic_read(&epoch->confirmed);
spin_unlock(&connection->epoch_lock);
drbd_send_b_ack(epoch->connection, epoch->barrier_nr, epoch_size);
spin_lock(&connection->epoch_lock);
}
-#if 0
- /* FIXME: dec unacked on connection, once we have
- * something to count pending connection packets in. */
- if (test_bit(DE_HAVE_BARRIER_NUMBER, &epoch->flags))
- dec_unacked(epoch->connection);
-#endif
if (connection->current_epoch != epoch) {
next_epoch = list_entry(epoch->list.next, struct drbd_epoch, list);
@@ -1214,9 +1891,11 @@ static enum finish_epoch drbd_may_finish_epoch(struct drbd_connection *connectio
if (rv == FE_STILL_LIVE)
rv = FE_DESTROYED;
} else {
+ epoch->oldest_unconfirmed_peer_req = NULL;
epoch->flags = 0;
atomic_set(&epoch->epoch_size, 0);
- /* atomic_set(&epoch->active, 0); is already zero */
+ atomic_set(&epoch->confirmed, 0);
+ /* atomic_set(&epoch->active, 0); is alrady zero */
if (rv == FE_STILL_LIVE)
rv = FE_RECYCLED;
}
@@ -1230,16 +1909,35 @@ static enum finish_epoch drbd_may_finish_epoch(struct drbd_connection *connectio
spin_unlock(&connection->epoch_lock);
+ if (schedule_flush) {
+ struct flush_work *fw;
+
+ fw = kmalloc_obj(*fw, GFP_ATOMIC);
+ if (fw) {
+ fw->w.cb = w_flush;
+ fw->epoch = epoch;
+ drbd_queue_work(&resource->work, &fw->w);
+ } else {
+ drbd_warn(resource, "Could not kmalloc a flush_work obj\n");
+ set_bit(DE_BARRIER_IN_NEXT_EPOCH_ISSUED, &epoch->flags);
+ /* That is not a recursion, only one level */
+ drbd_may_finish_epoch(connection, epoch, EV_BARRIER_DONE);
+ drbd_may_finish_epoch(connection, epoch, EV_PUT);
+ }
+ }
+
return rv;
}
static enum write_ordering_e
max_allowed_wo(struct drbd_backing_dev *bdev, enum write_ordering_e wo)
{
- struct disk_conf *dc;
+ struct drbd_disk_conf *dc;
dc = rcu_dereference(bdev->disk_conf);
+ if (wo == WO_BIO_BARRIER && !dc->disk_barrier)
+ wo = WO_BDEV_FLUSH;
if (wo == WO_BDEV_FLUSH && !dc->disk_flushes)
wo = WO_DRAIN_IO;
if (wo == WO_DRAIN_IO && !dc->disk_drain)
@@ -1257,18 +1955,22 @@ void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backin
{
struct drbd_device *device;
enum write_ordering_e pwo;
- int vnr;
+ int vnr, i = 0;
static char *write_ordering_str[] = {
[WO_NONE] = "none",
[WO_DRAIN_IO] = "drain",
[WO_BDEV_FLUSH] = "flush",
+ [WO_BIO_BARRIER] = "barrier",
};
pwo = resource->write_ordering;
- if (wo != WO_BDEV_FLUSH)
+ if (wo != WO_BIO_BARRIER)
wo = min(pwo, wo);
rcu_read_lock();
idr_for_each_entry(&resource->devices, device, vnr) {
+ if (i++ == 1 && wo == WO_BIO_BARRIER)
+ wo = WO_BDEV_FLUSH; /* WO = barrier does not handle multiple volumes */
+
if (get_ldev(device)) {
wo = max_allowed_wo(device->ldev, wo);
if (device->ldev == bdev)
@@ -1283,21 +1985,11 @@ void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backin
rcu_read_unlock();
resource->write_ordering = wo;
- if (pwo != resource->write_ordering || wo == WO_BDEV_FLUSH)
+ if (pwo != resource->write_ordering || wo == WO_BIO_BARRIER)
drbd_info(resource, "Method to ensure write ordering: %s\n", write_ordering_str[resource->write_ordering]);
}
/*
- * Mapping "discard" to ZEROOUT with UNMAP does not work for us:
- * Drivers have to "announce" q->limits.max_write_zeroes_sectors, or it
- * will directly go to fallback mode, submitting normal writes, and
- * never even try to UNMAP.
- *
- * And dm-thin does not do this (yet), mostly because in general it has
- * to assume that "skip_block_zeroing" is set. See also:
- * https://www.mail-archive.com/dm-devel%40redhat.com/msg07965.html
- * https://www.redhat.com/archives/dm-devel/2018-January/msg00271.html
- *
* We *may* ignore the discard-zeroes-data setting, if so configured.
*
* Assumption is that this "discard_zeroes_data=0" is only because the backend
@@ -1320,6 +2012,7 @@ void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backin
int drbd_issue_discard_or_zero_out(struct drbd_device *device, sector_t start, unsigned int nr_sectors, int flags)
{
struct block_device *bdev = device->ldev->backing_bdev;
+ struct request_queue *q = bdev_get_queue(bdev);
sector_t tmp, nr;
unsigned int max_discard_sectors, granularity;
int alignment;
@@ -1329,7 +2022,7 @@ int drbd_issue_discard_or_zero_out(struct drbd_device *device, sector_t start, u
goto zero_out;
/* Zero-sector (unknown) and one-sector granularities are the same. */
- granularity = max(bdev_discard_granularity(bdev) >> 9, 1U);
+ granularity = max(q->limits.discard_granularity >> 9, 1U);
alignment = (bdev_discard_alignment(bdev) >> 9) % granularity;
max_discard_sectors = min(bdev_max_discard_sectors(bdev), (1U << 22));
@@ -1356,8 +2049,7 @@ int drbd_issue_discard_or_zero_out(struct drbd_device *device, sector_t start, u
start = tmp;
}
while (nr_sectors >= max_discard_sectors) {
- err |= blkdev_issue_discard(bdev, start, max_discard_sectors,
- GFP_NOIO);
+ err |= blkdev_issue_discard(bdev, start, max_discard_sectors, GFP_NOIO);
nr_sectors -= max_discard_sectors;
start += max_discard_sectors;
}
@@ -1384,7 +2076,7 @@ int drbd_issue_discard_or_zero_out(struct drbd_device *device, sector_t start, u
static bool can_do_reliable_discards(struct drbd_device *device)
{
- struct disk_conf *dc;
+ struct drbd_disk_conf *dc;
bool can_do;
if (!bdev_max_discard_sectors(device->ldev->backing_bdev))
@@ -1414,11 +2106,11 @@ static void drbd_issue_peer_discard_or_zero_out(struct drbd_device *device, stru
static int peer_request_fault_type(struct drbd_peer_request *peer_req)
{
- if (peer_req_op(peer_req) == REQ_OP_READ) {
- return peer_req->flags & EE_APPLICATION ?
+ if (bio_op(peer_req->bios.head) == REQ_OP_READ) {
+ return drbd_interval_is_application(&peer_req->i) ?
DRBD_FAULT_DT_RD : DRBD_FAULT_RS_RD;
} else {
- return peer_req->flags & EE_APPLICATION ?
+ return drbd_interval_is_application(&peer_req->i) ?
DRBD_FAULT_DT_WR : DRBD_FAULT_RS_WR;
}
}
@@ -1436,18 +2128,28 @@ static int peer_request_fault_type(struct drbd_peer_request *peer_req)
* single page to an empty bio (which should never happen and likely indicates
* that the lower level IO stack is in some way broken). This has been observed
* on certain Xen deployments.
+ *
+ * When this function returns 0, it "consumes" an ldev reference; the
+ * reference is released when the request completes.
*/
/* TODO allocate from our own bio_set. */
int drbd_submit_peer_request(struct drbd_peer_request *peer_req)
{
struct drbd_device *device = peer_req->peer_device->device;
- struct bio *bios = NULL;
- struct bio *bio;
- struct page *page = peer_req->pages;
+ struct bio *bio, *next_bio;
sector_t sector = peer_req->i.sector;
- unsigned int data_size = peer_req->i.size;
- unsigned int n_bios = 0;
- unsigned int nr_pages = PFN_UP(data_size);
+ struct bio_list bios;
+ int fault_type, err, nr_bios = 0;
+
+ if (peer_req->flags & EE_SET_OUT_OF_SYNC)
+ drbd_set_out_of_sync(peer_req->peer_device,
+ sector, peer_req->i.size);
+
+ if (peer_req->flags & (EE_TRIM|EE_ZEROOUT) ||
+ bio_op(bio_list_peek(&peer_req->bios)) == REQ_OP_WRITE) {
+ /* ldev_safe: the caller passes its ldev reference to this function */
+ drbd_cbt_mark(device, sector, peer_req->i.size >> SECTOR_SHIFT);
+ }
/* TRIM/DISCARD: for now, always use the helper function
* blkdev_issue_zeroout(..., discard=true).
@@ -1455,27 +2157,18 @@ int drbd_submit_peer_request(struct drbd_peer_request *peer_req)
* Correctness first, performance later. Next step is to code an
* asynchronous variant of the same.
*/
- if (peer_req->flags & (EE_TRIM | EE_ZEROOUT)) {
- /* wait for all pending IO completions, before we start
- * zeroing things out. */
- conn_wait_active_ee_empty(peer_req->peer_device->connection);
- /* add it to the active list now,
- * so we can find it to present it in debugfs */
+ if (peer_req->flags & (EE_TRIM|EE_ZEROOUT)) {
peer_req->submit_jif = jiffies;
- peer_req->flags |= EE_SUBMITTED;
-
- /* If this was a resync request from receive_rs_deallocated(),
- * it is already on the sync_ee list */
- if (list_empty(&peer_req->w.list)) {
- spin_lock_irq(&device->resource->req_lock);
- list_add_tail(&peer_req->w.list, &device->active_ee);
- spin_unlock_irq(&device->resource->req_lock);
- }
+ /* ldev_safe: a peer_req has a ldev reference */
drbd_issue_peer_discard_or_zero_out(device, peer_req);
return 0;
}
+ fault_type = peer_request_fault_type(peer_req);
+ bios = peer_req->bios;
+ bio_list_init(&peer_req->bios);
+
/* In most cases, we will only need one bio. But in case the lower
* level restrictions happen to be different at this offset on this
* side than those of the sending peer, we may need to submit the
@@ -1484,90 +2177,166 @@ int drbd_submit_peer_request(struct drbd_peer_request *peer_req)
* Plain bio_alloc is good enough here, this is no DRBD internally
* generated bio, but a bio allocated on behalf of the peer.
*/
-next_bio:
/* _DISCARD, _WRITE_ZEROES handled above.
* REQ_OP_FLUSH (empty flush) not expected,
* should have been mapped to a "drbd protocol barrier".
* REQ_OP_SECURE_ERASE: I don't see how we could ever support that.
*/
- if (!(peer_req_op(peer_req) == REQ_OP_WRITE ||
- peer_req_op(peer_req) == REQ_OP_READ)) {
- drbd_err(device, "Invalid bio op received: 0x%x\n", peer_req->opf);
- return -EINVAL;
+ bio = bio_list_peek(&bios);
+ if (!(bio_op(bio) == REQ_OP_WRITE || bio_op(bio) == REQ_OP_READ)) {
+ drbd_err(device, "Invalid bio op received: 0x%x\n", bio->bi_opf);
+ err = -EINVAL;
+ goto fail;
}
- bio = bio_alloc(device->ldev->backing_bdev, nr_pages, peer_req->opf, GFP_NOIO);
- /* > peer_req->i.sector, unless this is the first bio */
- bio->bi_iter.bi_sector = sector;
- bio->bi_private = peer_req;
- bio->bi_end_io = drbd_peer_request_endio;
+ /* we special case some flags in the multi-bio case, see below
+ * (REQ_PREFLUSH, or BIO_RW_BARRIER in older kernels) */
- bio->bi_next = bios;
- bios = bio;
- ++n_bios;
+ /* Get reference for the first bio */
+ atomic_inc(&peer_req->pending_bios);
- page_chain_for_each(page) {
- unsigned len = min_t(unsigned, data_size, PAGE_SIZE);
- if (!bio_add_page(bio, page, len, 0))
- goto next_bio;
- data_size -= len;
- sector += len >> 9;
- --nr_pages;
- }
- D_ASSERT(device, data_size == 0);
- D_ASSERT(device, page == NULL);
-
- atomic_set(&peer_req->pending_bios, n_bios);
/* for debugfs: update timestamp, mark as submitted */
peer_req->submit_jif = jiffies;
- peer_req->flags |= EE_SUBMITTED;
- do {
- bio = bios;
- bios = bios->bi_next;
- bio->bi_next = NULL;
+ while ((bio = bio_list_pop(&bios))) {
+ /* bio_list_pop() clears bio->bi_next; it is a kernel-private
+ * field used during I/O; used temprorarily by DRBD pre submit
+ * and post completion
+ */
+ bio->bi_iter.bi_sector = sector;
+ bio->bi_private = peer_req;
+ bio->bi_end_io = drbd_peer_request_endio;
+
+ /* Save sector offset and size for restoration after I/O. */
+ to_drbd_peer_bio(bio)->sector_offset = sector - peer_req->i.sector;
+ to_drbd_peer_bio(bio)->size = bio->bi_iter.bi_size;
+
+ sector += bio_sectors(bio);
+
+ nr_bios++;
+
+ /* Get reference for the next bio (if any) now to prevent premature completion */
+ next_bio = bio_list_peek(&bios);
+ if (next_bio)
+ atomic_inc(&peer_req->pending_bios);
+ drbd_submit_bio_noacct(device, fault_type, bio);
+
+ /* strip off REQ_PREFLUSH,
+ * unless it is the first or last bio */
+ if (next_bio && next_bio->bi_next)
+ next_bio->bi_opf &= ~REQ_PREFLUSH;
+ }
+ if (nr_bios > 1)
+ device->multi_bio_cnt++;
- drbd_submit_bio_noacct(device, peer_request_fault_type(peer_req), bio);
- } while (bios);
return 0;
+
+fail:
+ while ((bio = bio_list_pop(&bios)))
+ bio_put(bio);
+ return err;
}
-static void drbd_remove_epoch_entry_interval(struct drbd_device *device,
- struct drbd_peer_request *peer_req)
+void drbd_remove_peer_req_interval(struct drbd_peer_request *peer_req)
{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
struct drbd_interval *i = &peer_req->i;
+ unsigned long flags;
- drbd_remove_interval(&device->write_requests, i);
+ spin_lock_irqsave(&device->interval_lock, flags);
+ D_ASSERT(device, !drbd_interval_empty(i));
+ drbd_remove_interval(&device->requests, i);
drbd_clear_interval(i);
-
- /* Wake up any processes waiting for this peer request to complete. */
- if (test_bit(INTERVAL_WAITING, &i->flags))
- wake_up(&device->misc_wait);
+ if (!drbd_interval_is_verify(&peer_req->i))
+ drbd_release_conflicts(device, i);
+ spin_unlock_irqrestore(&device->interval_lock, flags);
}
-static void conn_wait_active_ee_empty(struct drbd_connection *connection)
+/**
+ * w_e_reissue() - Worker callback; Resubmit a bio
+ * @w: work object.
+ * @cancel: The connection will be closed anyways (unused in this callback)
+ */
+int w_e_reissue(struct drbd_work *w, int cancel)
{
- struct drbd_peer_device *peer_device;
- int vnr;
+ struct drbd_peer_request *peer_req =
+ container_of(w, struct drbd_peer_request, w);
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ int err;
+ /* We leave DE_CONTAINS_A_BARRIER and EE_IS_BARRIER in place,
+ (and DE_BARRIER_IN_NEXT_EPOCH_ISSUED in the previous Epoch)
+ so that we can finish that epoch in drbd_may_finish_epoch().
+ That is necessary if we already have a long chain of Epochs, before
+ we realize that BARRIER is actually not supported */
+
+ /* As long as the -ENOTSUPP on the barrier is reported immediately
+ that will never trigger. If it is reported late, we will just
+ print that warning and continue correctly for all future requests
+ with WO_BDEV_FLUSH */
+ if (previous_epoch(connection, peer_req->epoch))
+ drbd_warn(device, "Write ordering was not enforced (one time event)\n");
+
+ /* we still have a local reference,
+ * get_ldev was done in receive_Data. */
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
+ peer_req->w.cb = e_end_block;
+ err = drbd_submit_peer_request(peer_req);
+ switch (err) {
+ case -ENOMEM:
+ peer_req->w.cb = w_e_reissue;
+ drbd_queue_work(&connection->sender_work,
+ &peer_req->w);
+ /* retry later */
+ fallthrough;
+ case 0:
+ /* keep worker happy and connection up */
+ return 0;
- kref_get(&device->kref);
- rcu_read_unlock();
- drbd_wait_ee_list_empty(device, &device->active_ee);
- kref_put(&device->kref, drbd_destroy_device);
- rcu_read_lock();
+ case -ENOSPC:
+ /* no other error expected, but anyways: */
+ default:
+ /* forget the object,
+ * and cause a "Network failure" */
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_al_complete_io(device, &peer_req->i);
+ drbd_may_finish_epoch(connection, peer_req->epoch, EV_PUT | EV_CLEANUP);
+ drbd_free_peer_req(peer_req);
+ drbd_err(device, "submit failed, triggering re-connect\n");
+ return err;
}
- rcu_read_unlock();
+}
+
+static void conn_wait_done_ee_empty_or_disconnect(struct drbd_connection *connection)
+{
+ wait_event(connection->ee_wait,
+ atomic_read(&connection->done_ee_cnt) == 0
+ || connection->cstate[NOW] < C_CONNECTED);
+}
+
+static void conn_wait_active_ee_empty_or_disconnect(struct drbd_connection *connection)
+{
+ if (atomic_read(&connection->active_ee_cnt) == 0)
+ return;
+
+ drbd_unplug_all_devices(connection);
+
+ wait_event(connection->ee_wait,
+ atomic_read(&connection->active_ee_cnt) == 0
+ || connection->cstate[NOW] < C_CONNECTED);
}
static int receive_Barrier(struct drbd_connection *connection, struct packet_info *pi)
{
- int rv;
+ struct drbd_transport_ops *tr_ops = &connection->transport.class->ops;
+ int rv, issue_flush;
struct p_barrier *p = pi->data;
struct drbd_epoch *epoch;
+ tr_ops->hint(&connection->transport, DATA_STREAM, QUICKACK);
+ drbd_unplug_all_devices(connection);
+
/* FIXME these are unacked on connection,
* not a specific (peer)device.
*/
@@ -1581,41 +2350,48 @@ static int receive_Barrier(struct drbd_connection *connection, struct packet_inf
* Therefore we must send the barrier_ack after the barrier request was
* completed. */
switch (connection->resource->write_ordering) {
+ case WO_BIO_BARRIER:
case WO_NONE:
if (rv == FE_RECYCLED)
return 0;
-
- /* receiver context, in the writeout path of the other node.
- * avoid potential distributed deadlock */
- epoch = kmalloc_obj(struct drbd_epoch, GFP_NOIO);
- if (epoch)
- break;
- else
- drbd_warn(connection, "Allocation of an epoch failed, slowing down\n");
- fallthrough;
+ break;
case WO_BDEV_FLUSH:
case WO_DRAIN_IO:
- conn_wait_active_ee_empty(connection);
- drbd_flush(connection);
+ if (rv == FE_STILL_LIVE) {
+ set_bit(DE_BARRIER_IN_NEXT_EPOCH_ISSUED, &connection->current_epoch->flags);
+ conn_wait_active_ee_empty_or_disconnect(connection);
+ rv = drbd_flush_after_epoch(connection, connection->current_epoch);
+ }
+ if (rv == FE_RECYCLED)
+ return 0;
- if (atomic_read(&connection->current_epoch->epoch_size)) {
- epoch = kmalloc_obj(struct drbd_epoch, GFP_NOIO);
- if (epoch)
- break;
+ /*
+ * The ack_sender will send all the ACKs and barrier ACKs out,
+ * since all EEs added to done_ee. We need to provide a new
+ * epoch object for the EEs that come in soon.
+ */
+ break;
+ }
+
+ /* receiver context, in the writeout path of the other node.
+ * avoid potential distributed deadlock */
+ epoch = kzalloc_obj(struct drbd_epoch, GFP_NOIO);
+ if (!epoch) {
+ drbd_warn(connection, "Allocation of an epoch failed, slowing down\n");
+ issue_flush = !test_and_set_bit(DE_BARRIER_IN_NEXT_EPOCH_ISSUED, &connection->current_epoch->flags);
+ conn_wait_active_ee_empty_or_disconnect(connection);
+ if (issue_flush) {
+ rv = drbd_flush_after_epoch(connection, connection->current_epoch);
+ if (rv == FE_RECYCLED)
+ return 0;
}
+ conn_wait_done_ee_empty_or_disconnect(connection);
+
return 0;
- default:
- drbd_err(connection, "Strangeness in connection->write_ordering %d\n",
- connection->resource->write_ordering);
- return -EIO;
}
- epoch->flags = 0;
- atomic_set(&epoch->epoch_size, 0);
- atomic_set(&epoch->active, 0);
-
spin_lock(&connection->epoch_lock);
if (atomic_read(&connection->current_epoch->epoch_size)) {
list_add(&epoch->list, &connection->current_epoch->list);
@@ -1630,15 +2406,158 @@ static int receive_Barrier(struct drbd_connection *connection, struct packet_inf
return 0;
}
-/* quick wrapper in case payload size != request_size (write same) */
-static void drbd_csum_ee_size(struct crypto_shash *h,
- struct drbd_peer_request *r, void *d,
- unsigned int payload_size)
+/* pi->data points into some recv buffer, which may be
+ * re-used/recycled/overwritten by the next receive operation.
+ * (read_in_block via recv_resync_read) */
+static void p_req_detail_from_pi(struct drbd_connection *connection,
+ struct drbd_peer_request_details *d, struct packet_info *pi)
+{
+ struct p_trim *p = pi->data;
+ bool is_trim_or_zeroes = pi->cmd == P_TRIM || pi->cmd == P_ZEROES;
+ unsigned int digest_size =
+ pi->cmd != P_TRIM && connection->peer_integrity_tfm ?
+ crypto_shash_digestsize(connection->peer_integrity_tfm) : 0;
+
+ d->sector = be64_to_cpu(p->p_data.sector);
+ d->block_id = p->p_data.block_id;
+ d->peer_seq = be32_to_cpu(p->p_data.seq_num);
+ d->dp_flags = be32_to_cpu(p->p_data.dp_flags);
+ d->length = pi->size;
+ d->bi_size = is_trim_or_zeroes ? be32_to_cpu(p->size) : pi->size - digest_size;
+ d->digest_size = digest_size;
+}
+
+/* A cluster-wide size change is agreed by a two-phase commit and then applied
+ * node by node, and applying it takes a different amount of time everywhere: a
+ * diskless node only sets the new capacity, a diskful one first grows its resync
+ * bitmap. So a peer that is further along legitimately sends us requests for the
+ * newly added tail of the device while we still consider that tail out of range.
+ * Wait for our own half of the size change and look again, instead of failing the
+ * request: an error return from a packet handler destroys the connection.
+ *
+ * remote_state_change plus TWOPC_RESIZE covers the whole window on every node:
+ * it is set when this node joins the resize two-phase commit and cleared after
+ * drbd_commit_size_change() has applied the new size locally.
+ *
+ * The wait is bounded, and it is skipped while an application read is
+ * outstanding here: a diskful node's size change waits for its application IO to
+ * drain (drbd_suspend_io), and a remote read completes through this very receiver
+ * thread, so waiting for both at once would deadlock. Falling through in either
+ * case leaves the old behaviour, which is what a peer sending a genuinely
+ * out-of-range request deserves.
+ */
+#define DRBD_SIZE_CHANGE_TIMEOUT (HZ)
+
+static bool resize_in_progress(struct drbd_resource *resource)
+{
+ bool rv;
+
+ read_lock_irq(&resource->state_rwlock);
+ rv = resource->remote_state_change && resource->twopc.type == TWOPC_RESIZE;
+ read_unlock_irq(&resource->state_rwlock);
+
+ return rv;
+}
+
+static bool in_range_after_size_change(struct drbd_device *device, sector_t sector,
+ unsigned int size)
+{
+ struct drbd_resource *resource = device->resource;
+
+ if (atomic_read(&device->ap_bio_cnt[READ]))
+ return false;
+
+ wait_event_timeout(resource->twopc_wait, !resize_in_progress(resource),
+ DRBD_SIZE_CHANGE_TIMEOUT);
+
+ return sector + (size >> 9) <= get_capacity(device->vdisk);
+}
+
+static void copy_bio_list(struct bio_list *dst, struct bio_list *src)
+{
+ struct bio *sb = src->head, *db = dst->head;
+ struct bvec_iter si = sb->bi_iter, di = db->bi_iter;
+
+ while (sb && db) {
+ struct bio_vec sv = bio_iter_iovec(sb, si);
+ struct bio_vec dv = bio_iter_iovec(db, di);
+ unsigned int len = min(sv.bv_len, dv.bv_len);
+ void *s, *d;
+
+ s = kmap_local_page(sv.bv_page);
+ d = kmap_local_page(dv.bv_page);
+ memcpy(d + dv.bv_offset, s + sv.bv_offset, len);
+ kunmap_local(d);
+ kunmap_local(s);
+
+ bio_advance_iter(sb, &si, len);
+ if (!si.bi_size) {
+ sb = sb->bi_next;
+ if (sb)
+ si = sb->bi_iter;
+ }
+ bio_advance_iter(db, &di, len);
+ if (!di.bi_size) {
+ db = db->bi_next;
+ if (db)
+ di = db->bi_iter;
+ }
+ }
+}
+
+/* The backing device's queue dma_alignment constrains the offset and length of
+ * every bvec submitted to it; a violating bio fails in the block layer. A
+ * transport that lands received payload at sub-page offsets (rdma2) aligns it
+ * to the value the core announces via set_rx_alignment(), but a region
+ * announced before that value changed (a later attach of a device with a
+ * coarser requirement) can still deliver misaligned payload for a while. Copy
+ * such payload into freshly allocated, page-aligned pages.
+ */
+static int peer_req_align_bios(struct drbd_peer_request *peer_req, unsigned int size)
{
- unsigned int tmp = r->i.size;
- r->i.size = payload_size;
- drbd_csum_ee(h, r, d);
- r->i.size = tmp;
+ struct drbd_connection *connection = peer_req->peer_device->connection;
+ struct drbd_transport *transport = &connection->transport;
+ struct bio_list received;
+ unsigned int remaining;
+ int err;
+
+ received = peer_req->bios;
+ bio_list_init(&peer_req->bios);
+
+ err = peer_req_alloc_bio(peer_req, size, GFP_NOIO, received.head->bi_opf);
+ if (err)
+ goto out_restore;
+
+ remaining = size;
+ while (remaining) {
+ struct page *page;
+ int len;
+
+ page = drbd_alloc_pages(transport, GFP_NOIO, remaining);
+ if (!page) {
+ err = -ENOMEM;
+ goto out_strip_restore;
+ }
+ len = min_t(unsigned int, PAGE_SIZE << compound_order(page), remaining);
+ len = drbd_bio_add_page(transport, &peer_req->bios, page, len, 0);
+ if (len < 0) {
+ drbd_free_page(transport, page);
+ err = len;
+ goto out_strip_restore;
+ }
+ remaining -= len;
+ }
+
+ copy_bio_list(&peer_req->bios, &received);
+ drbd_strip_bio_list(transport, &received);
+ connection->rx_misaligned_copies++;
+ return 0;
+
+out_strip_restore:
+ drbd_strip_bio_list(transport, &peer_req->bios);
+out_restore:
+ peer_req->bios = received;
+ return err;
}
/* used from receive_RSDataReply (recv_resync_read)
@@ -1650,140 +2569,130 @@ static void drbd_csum_ee_size(struct crypto_shash *h,
* both trim and write same have the bi_size ("data len to be affected")
* as extra argument in the packet header.
*/
-static struct drbd_peer_request *
-read_in_block(struct drbd_peer_device *peer_device, u64 id, sector_t sector,
- struct packet_info *pi) __must_hold(local)
+static int
+read_in_block(struct drbd_peer_request *peer_req, struct drbd_peer_request_details *d)
{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
- const sector_t capacity = get_capacity(device->vdisk);
- struct drbd_peer_request *peer_req;
- struct page *page;
- int digest_size, err;
- unsigned int data_size = pi->size, ds;
- void *dig_in = peer_device->connection->int_dig_in;
- void *dig_vv = peer_device->connection->int_dig_vv;
- unsigned long *data;
- struct p_trim *trim = (pi->cmd == P_TRIM) ? pi->data : NULL;
- struct p_trim *zeroes = (pi->cmd == P_ZEROES) ? pi->data : NULL;
-
- digest_size = 0;
- if (!trim && peer_device->connection->peer_integrity_tfm) {
- digest_size = crypto_shash_digestsize(peer_device->connection->peer_integrity_tfm);
- /*
- * FIXME: Receive the incoming digest into the receive buffer
- * here, together with its struct p_data?
- */
- err = drbd_recv_all_warn(peer_device->connection, dig_in, digest_size);
+ struct drbd_connection *connection = peer_device->connection;
+ const uint64_t capacity = get_capacity(device->vdisk);
+ void *dig_in = connection->int_dig_in;
+ void *dig_vv = connection->int_dig_vv;
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
+ unsigned int misalign_bits;
+ int size, err;
+
+ if (d->digest_size) {
+ err = drbd_recv_into(connection, dig_in, d->digest_size);
if (err)
- return NULL;
- data_size -= digest_size;
- }
-
- /* assume request_size == data_size, but special case trim. */
- ds = data_size;
- if (trim) {
- if (!expect(peer_device, data_size == 0))
- return NULL;
- ds = be32_to_cpu(trim->size);
- } else if (zeroes) {
- if (!expect(peer_device, data_size == 0))
- return NULL;
- ds = be32_to_cpu(zeroes->size);
+ return err;
}
- if (!expect(peer_device, IS_ALIGNED(ds, 512)))
- return NULL;
- if (trim || zeroes) {
- if (!expect(peer_device, ds <= (DRBD_MAX_BBIO_SECTORS << 9)))
- return NULL;
- } else if (!expect(peer_device, ds <= DRBD_MAX_BIO_SIZE))
- return NULL;
+ if (!expect(peer_device, IS_ALIGNED(d->bi_size, 512)))
+ return -EINVAL;
+ /* The WSAME mechanism was removed in Linux 5.18,
+ * and subsequently from drbd.
+ * In theory, a "modern" drbd will never advertise support for
+ * WRITE_SAME, so a compliant peer should never send a DP_WSAME
+ * packet. If we receive one anyway, that's a protocol error.
+ */
+ if (!expect(peer_device, (d->dp_flags & DP_WSAME) == 0))
+ return -EINVAL;
+ if (d->dp_flags & (DP_DISCARD|DP_ZEROES)) {
+ if (!expect(peer_device, d->bi_size <= (DRBD_MAX_BBIO_SECTORS << 9)))
+ return -EINVAL;
+ } else if (!expect(peer_device, d->bi_size <= DRBD_MAX_BIO_SIZE))
+ return -EINVAL;
- /* even though we trust out peer,
+ /* even though we trust our peer,
* we sometimes have to double check. */
- if (sector + (ds>>9) > capacity) {
+ if (d->sector + (d->bi_size>>9) > capacity &&
+ !in_range_after_size_change(device, d->sector, d->bi_size)) {
drbd_err(device, "request from peer beyond end of local disk: "
"capacity: %llus < sector: %llus + size: %u\n",
- (unsigned long long)capacity,
- (unsigned long long)sector, ds);
- return NULL;
+ capacity, d->sector, d->bi_size);
+ return -EINVAL;
}
- /* GFP_NOIO, because we must not cause arbitrary write-out: in a DRBD
- * "criss-cross" setup, that might cause write-out on some other DRBD,
- * which in turn might block on the other node at this very place. */
- peer_req = drbd_alloc_peer_req(peer_device, id, sector, ds, data_size, GFP_NOIO);
- if (!peer_req)
- return NULL;
+ peer_req->block_id = d->block_id;
- peer_req->flags |= EE_WRITE;
- if (trim) {
- peer_req->flags |= EE_TRIM;
- return peer_req;
+ if (d->length == 0)
+ return 0;
+
+ size = d->length - d->digest_size;
+ if (bio_list_empty(&peer_req->bios)) {
+ /* For a checksum resync, the bio was consumed for reading. */
+ err = peer_req_alloc_bio(peer_req, size, GFP_NOIO, REQ_OP_WRITE);
+ if (err)
+ return err;
}
- if (zeroes) {
- peer_req->flags |= EE_ZEROOUT;
- return peer_req;
+
+ err = tr_ops->recv_bio(transport, &peer_req->bios, size, &misalign_bits);
+ if (err < 0) {
+ if (err == -ECONNRESET)
+ drbd_info(connection, "sock was reset by peer while receiving data\n");
+ else if (err != -ERESTARTSYS)
+ drbd_info(connection, "recvmsg returned %d while receiving data\n", err);
+ } else if (err == 0) {
+ drbd_info(connection, "sock was shut down by peer while receiving data\n");
+ err = -EIO;
}
- /* receive payload size bytes into page chain */
- ds = data_size;
- page = peer_req->pages;
- page_chain_for_each(page) {
- unsigned len = min_t(int, ds, PAGE_SIZE);
- data = kmap_local_page(page);
- err = drbd_recv_all_warn(peer_device->connection, data, len);
- if (drbd_insert_fault(device, DRBD_FAULT_RECEIVE)) {
- drbd_err(device, "Fault injection: Corrupting data on receive\n");
- data[0] = data[0] ^ (unsigned long)-1;
- }
- kunmap_local(data);
- if (err) {
- drbd_free_peer_req(device, peer_req);
- return NULL;
- }
- ds -= len;
+ if (err != size) {
+ change_cstate(connection, C_BROKEN_PIPE, CS_HARD);
+ return err;
}
- if (digest_size) {
- drbd_csum_ee_size(peer_device->connection->peer_integrity_tfm, peer_req, dig_vv, data_size);
- if (memcmp(dig_in, dig_vv, digest_size)) {
+ if (misalign_bits & queue_dma_alignment(bdev_get_queue(device->ldev->backing_bdev))) {
+ err = peer_req_align_bios(peer_req, size);
+ if (err)
+ return err;
+ }
+
+ if (drbd_insert_fault(device, DRBD_FAULT_RECEIVE)) {
+ struct bio *bio = bio_list_peek(&peer_req->bios);
+ unsigned long *data;
+
+ drbd_err(device, "Fault injection: Corrupting data on receive, sector %llu\n",
+ d->sector);
+
+ data = bvec_virt(&bio->bi_io_vec[0]);
+ data[0] = ~data[0];
+ }
+
+ if (d->digest_size) {
+ drbd_csum_bios(connection->peer_integrity_tfm, &peer_req->bios, dig_vv);
+ if (memcmp(dig_in, dig_vv, d->digest_size)) {
drbd_err(device, "Digest integrity check FAILED: %llus +%u\n",
- (unsigned long long)sector, data_size);
- drbd_free_peer_req(device, peer_req);
- return NULL;
+ d->sector, d->bi_size);
+ return -EINVAL;
}
}
- device->recv_cnt += data_size >> 9;
- return peer_req;
+ peer_device->recv_cnt += d->bi_size >> 9;
+ return 0;
}
-/* drbd_drain_block() just takes a data block
- * out of the socket input buffer, and discards it.
- */
-static int drbd_drain_block(struct drbd_peer_device *peer_device, int data_size)
+static int ignore_remaining_packet(struct drbd_connection *connection, int size)
{
- struct page *page;
- int err = 0;
- void *data;
+ void *data_to_ignore;
- if (!data_size)
- return 0;
+ if (size < 0) {
+ drbd_err(connection, "Invalid packet size\n");
+ return -EIO;
+ }
- page = drbd_alloc_pages(peer_device, 1, 1);
+ while (size) {
+ int s = min_t(int, size, DRBD_SOCKET_BUFFER_SIZE);
+ int rv = drbd_recv(connection, &data_to_ignore, s, 0);
- data = kmap_local_page(page);
- while (data_size) {
- unsigned int len = min_t(int, data_size, PAGE_SIZE);
+ if (rv < 0)
+ return rv;
- err = drbd_recv_all_warn(peer_device->connection, data, len);
- if (err)
- break;
- data_size -= len;
+ size -= rv;
}
- kunmap_local(data);
- drbd_free_pages(peer_device->device, page);
- return err;
+
+ return 0;
}
static int recv_dless_read(struct drbd_peer_device *peer_device, struct drbd_request *req,
@@ -1799,7 +2708,7 @@ static int recv_dless_read(struct drbd_peer_device *peer_device, struct drbd_req
digest_size = 0;
if (peer_device->connection->peer_integrity_tfm) {
digest_size = crypto_shash_digestsize(peer_device->connection->peer_integrity_tfm);
- err = drbd_recv_all_warn(peer_device->connection, dig_in, digest_size);
+ err = drbd_recv_into(peer_device->connection, dig_in, digest_size);
if (err)
return err;
data_size -= digest_size;
@@ -1812,15 +2721,16 @@ static int recv_dless_read(struct drbd_peer_device *peer_device, struct drbd_req
/* optimistically update recv_cnt. if receiving fails below,
* we disconnect anyways, and counters will be reset. */
- peer_device->device->recv_cnt += data_size>>9;
+ peer_device->recv_cnt += data_size >> 9;
bio = req->master_bio;
D_ASSERT(peer_device->device, sector == bio->bi_iter.bi_sector);
bio_for_each_segment(bvec, bio, iter) {
void *mapped = bvec_kmap_local(&bvec);
+
expect = min_t(int, data_size, bvec.bv_len);
- err = drbd_recv_all_warn(peer_device->connection, mapped, expect);
+ err = drbd_recv_into(peer_device->connection, mapped, expect);
kunmap_local(mapped);
if (err)
return err;
@@ -1839,3901 +2749,11878 @@ static int recv_dless_read(struct drbd_peer_device *peer_device, struct drbd_req
return 0;
}
-/*
- * e_end_resync_block() is called in ack_sender context via
- * drbd_finish_peer_reqs().
- */
-static int e_end_resync_block(struct drbd_work *w, int unused)
+static bool bits_in_sync(struct drbd_peer_device *peer_device, sector_t sector_start, sector_t sector_end)
{
- struct drbd_peer_request *peer_req =
- container_of(w, struct drbd_peer_request, w);
- struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
- sector_t sector = peer_req->i.sector;
- int err;
+ struct drbd_bitmap *bm = device->bitmap;
+
+ if (peer_device->repl_state[NOW] == L_ESTABLISHED ||
+ peer_device->repl_state[NOW] == L_SYNC_SOURCE ||
+ peer_device->repl_state[NOW] == L_SYNC_TARGET ||
+ peer_device->repl_state[NOW] == L_PAUSED_SYNC_S ||
+ peer_device->repl_state[NOW] == L_PAUSED_SYNC_T) {
+ /* An Inconsistent peer is receiving a resync. Unless we are the
+ * source, that data is not ours and our bitmap does not know it.
+ */
+ if (peer_device->disk_state[NOW] == D_INCONSISTENT &&
+ peer_device->repl_state[NOW] != L_SYNC_SOURCE)
+ return false;
+ if (drbd_bm_total_weight(peer_device) == 0)
+ return true;
+ if (drbd_bm_count_bits(device, peer_device->bitmap_index,
+ bm_sect_to_bit(bm, sector_start),
+ bm_sect_to_bit(bm, sector_end - 1)) == 0)
+ return true;
+ }
+ return false;
+}
- D_ASSERT(device, drbd_interval_empty(&peer_req->i));
+static void update_peers_for_interval(struct drbd_peer_device *peer_device,
+ struct drbd_interval *interval)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_bitmap *bm = device->bitmap;
+ u64 mask = NODE_MASK(peer_device->node_id), im;
+ struct drbd_peer_device *p;
+ sector_t sector_end = interval->sector + (interval->size >> SECTOR_SHIFT);
+
+ /* Only send P_PEERS_IN_SYNC if we are actually in sync with this peer. */
+ if (drbd_bm_count_bits(device, peer_device->bitmap_index,
+ bm_sect_to_bit(bm, interval->sector),
+ bm_sect_to_bit(bm, sector_end - 1)))
+ return;
- if (likely((peer_req->flags & EE_WAS_ERROR) == 0)) {
- drbd_set_in_sync(peer_device, sector, peer_req->i.size);
- err = drbd_send_ack(peer_device, P_RS_WRITE_ACK, peer_req);
- } else {
- /* Record failure to sync */
- drbd_rs_failed_io(peer_device, sector, peer_req->i.size);
+ for_each_peer_device_ref(p, im, device) {
+ if (p == peer_device)
+ continue;
- err = drbd_send_ack(peer_device, P_NEG_ACK, peer_req);
+ if (bits_in_sync(p, interval->sector, sector_end))
+ mask |= NODE_MASK(p->node_id);
}
- dec_unacked(device);
- return err;
+ for_each_peer_device_ref(p, im, device) {
+ /* Only send to the peer whose bitmap bits have been cleared if
+ * we are connected to that peer. The bits may have been
+ * cleared by a P_PEERS_IN_SYNC from another peer while we are
+ * connecting to this one. We mustn't send P_PEERS_IN_SYNC
+ * during the initial connection handshake. */
+ if (p == peer_device && p->connection->cstate[NOW] != C_CONNECTED)
+ continue;
+
+ if (mask & NODE_MASK(p->node_id))
+ drbd_send_peers_in_sync(p, mask, interval->sector, interval->size);
+ }
}
-static int recv_resync_read(struct drbd_peer_device *peer_device, sector_t sector,
- struct packet_info *pi) __releases(local)
+/* Potentially send P_PEERS_IN_SYNC for a range with size that fits in an int. */
+static void update_peers_for_small_range(struct drbd_peer_device *peer_device,
+ sector_t sector, int size)
{
struct drbd_device *device = peer_device->device;
- struct drbd_peer_request *peer_req;
+ struct drbd_interval interval;
- peer_req = read_in_block(peer_device, ID_SYNCER, sector, pi);
- if (!peer_req)
- goto fail;
+ memset(&interval, 0, sizeof(interval));
+ drbd_clear_interval(&interval);
+ interval.sector = sector;
+ interval.size = size;
+ interval.type = INTERVAL_PEERS_IN_SYNC_LOCK;
- peer_req->i.type = INTERVAL_RESYNC_WRITE;
+ spin_lock_irq(&device->interval_lock);
+ if (drbd_find_conflict(device, &interval, 0)) {
+ spin_unlock_irq(&device->interval_lock);
+ return;
+ }
+ drbd_insert_interval(&device->requests, &interval);
+ /* Interval is not waiting for conflicts to resolve so mark as "submitted". */
+ set_bit(INTERVAL_SUBMITTED, &interval.flags);
+ spin_unlock_irq(&device->interval_lock);
- dec_rs_pending(peer_device);
+ /* Check for activity in the activity log extent _after_ locking the
+ * interval. Otherwise a write might occur between checking and
+ * locking. */
+ if (!drbd_al_active(device, sector, size))
+ update_peers_for_interval(peer_device, &interval);
- inc_unacked(device);
- /* corresponding dec_unacked() in e_end_resync_block()
- * respective _drbd_clear_done_ee */
+ spin_lock_irq(&device->interval_lock);
+ drbd_remove_interval(&device->requests, &interval);
+ drbd_release_conflicts(device, &interval);
+ spin_unlock_irq(&device->interval_lock);
+}
- peer_req->w.cb = e_end_resync_block;
- peer_req->opf = REQ_OP_WRITE;
- peer_req->submit_jif = jiffies;
+static void update_peers_for_range(struct drbd_peer_device *peer_device,
+ sector_t sector_start, sector_t sector_end)
+{
+ struct drbd_device *device = peer_device->device;
+ unsigned int enr_start = sector_start >> (AL_EXTENT_SHIFT - SECTOR_SHIFT);
+ unsigned int enr_end = ((sector_end - 1) >> (AL_EXTENT_SHIFT - SECTOR_SHIFT)) + 1;
+ unsigned int enr;
- spin_lock_irq(&device->resource->req_lock);
- list_add_tail(&peer_req->w.list, &device->sync_ee);
- spin_unlock_irq(&device->resource->req_lock);
+ if (!get_ldev(device))
+ return;
- atomic_add(pi->size >> 9, &device->rs_sect_ev);
- if (drbd_submit_peer_request(peer_req) == 0)
- return 0;
+ for (enr = enr_start; enr < enr_end; enr++) {
+ sector_t enr_start_sector = max(sector_start,
+ ((sector_t) enr) << (AL_EXTENT_SHIFT - SECTOR_SHIFT));
+ sector_t enr_end_sector = min(sector_end,
+ ((sector_t) (enr + 1)) << (AL_EXTENT_SHIFT - SECTOR_SHIFT));
- /* don't care for the reason here */
- drbd_err(device, "submit failed, triggering re-connect\n");
- spin_lock_irq(&device->resource->req_lock);
- list_del(&peer_req->w.list);
- spin_unlock_irq(&device->resource->req_lock);
+ update_peers_for_small_range(peer_device,
+ enr_start_sector, (enr_end_sector - enr_start_sector) << SECTOR_SHIFT);
+ }
- drbd_free_peer_req(device, peer_req);
-fail:
put_ldev(device);
- return -EIO;
}
-static struct drbd_request *
-find_request(struct drbd_device *device, struct rb_root *root, u64 id,
- sector_t sector, bool missing_ok, const char *func)
+static int w_update_peers(struct drbd_work *w, int unused)
{
- struct drbd_request *req;
+ struct update_peers_work *upw = container_of(w, struct update_peers_work, w);
+ struct drbd_peer_device *peer_device = upw->peer_device;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
- /* Request object according to our peer */
- req = (struct drbd_request *)(unsigned long)id;
- if (drbd_contains_interval(root, sector, &req->i) && drbd_interval_is_local(&req->i))
- return req;
- if (!missing_ok) {
- drbd_err(device, "%s: failed to find request 0x%lx, sector %llus\n", func,
- (unsigned long)id, (unsigned long long)sector);
- }
- return NULL;
+ if (connection->agreed_pro_version >= 110)
+ update_peers_for_range(peer_device, upw->sector_start, upw->sector_end);
+
+ kfree(upw);
+ kref_put(&device->kref, drbd_destroy_device);
+ kref_put(&connection->kref, drbd_destroy_connection);
+
+ return 0;
}
-static int receive_DataReply(struct drbd_connection *connection, struct packet_info *pi)
+void drbd_queue_update_peers(struct drbd_peer_device *peer_device,
+ sector_t sector_start, sector_t sector_end)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct drbd_request *req;
- sector_t sector;
- int err;
- struct p_data *p = pi->data;
-
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ struct drbd_device *device = peer_device->device;
+ struct update_peers_work *upw;
- sector = be64_to_cpu(p->sector);
+ upw = kmalloc_obj(*upw, GFP_ATOMIC | __GFP_NOWARN);
+ if (upw) {
+ upw->sector_start = sector_start;
+ upw->sector_end = sector_end;
+ upw->w.cb = w_update_peers;
- spin_lock_irq(&device->resource->req_lock);
- req = find_request(device, &device->read_requests, p->block_id, sector, false, __func__);
- spin_unlock_irq(&device->resource->req_lock);
- if (unlikely(!req))
- return -EIO;
+ kref_get(&peer_device->device->kref);
- err = recv_dless_read(peer_device, req, sector, pi->size);
- if (!err)
- req_mod(req, DATA_RECEIVED, peer_device);
- /* else: nothing. handled from drbd_disconnect...
- * I don't think we may complete this just yet
- * in case we are "on-disconnect: freeze" */
+ kref_get(&peer_device->connection->kref);
- return err;
+ upw->peer_device = peer_device;
+ drbd_queue_work(&device->resource->work, &upw->w);
+ } else {
+ if (drbd_ratelimit())
+ drbd_warn(peer_device, "kmalloc(upw) failed.\n");
+ }
}
-static int receive_RSDataReply(struct drbd_connection *connection, struct packet_info *pi)
+static void drbd_peers_in_sync_progress(struct drbd_peer_device *peer_device,
+ sector_t sector_start, sector_t sector_end)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- sector_t sector;
- int err;
- struct p_data *p = pi->data;
+ /* P_PEERS_IN_SYNC "steps" are represented by their start sector */
+ sector_t step = sector_start & ~PEERS_IN_SYNC_STEP_SECT_MASK;
+ sector_t end_step = sector_end & ~PEERS_IN_SYNC_STEP_SECT_MASK;
+ sector_t last_end = peer_device->last_in_sync_end;
+ sector_t last_step = last_end & ~PEERS_IN_SYNC_STEP_SECT_MASK;
+ sector_t last_step_end = min(get_capacity(peer_device->device->vdisk),
+ last_step + PEERS_IN_SYNC_STEP_SECT);
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ /* Send for last request if it was part way through a different step */
+ if (last_end > last_step && step != last_step)
+ drbd_queue_update_peers(peer_device, last_step, last_step_end);
- sector = be64_to_cpu(p->sector);
- D_ASSERT(device, p->block_id == ID_SYNCER);
+ /* Send if the request reaches or passes a step boundary */
+ if (end_step != step)
+ drbd_queue_update_peers(peer_device, step, end_step);
- if (get_ldev(device)) {
- /* data is submitted to disk within recv_resync_read.
- * corresponding put_ldev done below on error,
- * or in drbd_peer_request_endio. */
- err = recv_resync_read(peer_device, sector, pi);
- } else {
- drbd_err_ratelimit(device, "Can not write resync data to local disk.\n");
+ peer_device->last_in_sync_end = sector_end;
+
+ /*
+ * Consider scheduling a bitmap update to reduce the size of the next
+ * resync if this one is disrupted.
+ */
+ if (drbd_lazy_bitmap_update_due(peer_device))
+ drbd_peer_device_post_work(peer_device, RS_LAZY_BM_WRITE);
+}
- err = drbd_drain_block(peer_device, pi->size);
+static void drbd_check_peers_in_sync_progress(struct drbd_peer_device *peer_device)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ LIST_HEAD(completed);
+ struct drbd_peer_request *peer_req, *tmp;
- drbd_send_ack_dp(peer_device, P_NEG_ACK, p, pi->size);
- }
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry_safe(peer_req, tmp, &peer_device->resync_requests, recv_order) {
+ if (!test_bit(INTERVAL_COMPLETED, &peer_req->i.flags))
+ break;
- atomic_add(pi->size >> 9, &device->rs_sect_in);
+ drbd_peers_in_sync_progress(peer_device, peer_req->i.sector,
+ peer_req->i.sector + (peer_req->i.size >> SECTOR_SHIFT));
- return err;
+ drbd_list_del_resync_request(peer_req);
+ list_add_tail(&peer_req->recv_order, &completed);
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ list_for_each_entry_safe(peer_req, tmp, &completed, recv_order)
+ drbd_free_peer_req(peer_req);
}
-static void restart_conflicting_writes(struct drbd_device *device,
- sector_t sector, int size)
+static void drbd_resync_request_complete(struct drbd_peer_request *peer_req)
{
- struct drbd_interval *i;
- struct drbd_request *req;
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
- drbd_for_each_overlap(i, &device->write_requests, sector, size) {
- if (!drbd_interval_is_local(i))
- continue;
- req = container_of(i, struct drbd_request, i);
- if (req->rq_state & RQ_LOCAL_PENDING ||
- !(req->rq_state & RQ_POSTPONED))
- continue;
- /* as it is RQ_POSTPONED, this will cause it to
- * be queued on the retry workqueue. */
- __req_mod(req, CONFLICT_RESOLVED, NULL, NULL);
- }
+ /*
+ * Free the pages now but leave the peer request until the
+ * corresponding peers-in-sync has been scheduled.
+ */
+ drbd_peer_req_strip_bio(peer_req);
+
+ /*
+ * The interval is no longer in the tree, but use this flag anyway,
+ * since it has an appropriate meaning. After setting the flag,
+ * peer_req may be freed by another thread.
+ */
+ set_bit(INTERVAL_COMPLETED, &peer_req->i.flags);
+ peer_req = NULL;
+
+ drbd_check_peers_in_sync_progress(peer_device);
}
/*
- * e_end_block() is called in ack_sender context via drbd_finish_peer_reqs().
+ * e_end_resync_block() is called in ack_sender context via
+ * drbd_finish_peer_reqs().
*/
-static int e_end_block(struct drbd_work *w, int cancel)
+static int e_end_resync_block(struct drbd_work *w, int unused)
{
struct drbd_peer_request *peer_req =
container_of(w, struct drbd_peer_request, w);
struct drbd_peer_device *peer_device = peer_req->peer_device;
- struct drbd_device *device = peer_device->device;
sector_t sector = peer_req->i.sector;
- int err = 0, pcmd;
+ unsigned int size = peer_req->requested_size;
+ u64 block_id = peer_req->block_id;
+ int err;
- if (peer_req->flags & EE_SEND_WRITE_ACK) {
- if (likely((peer_req->flags & EE_WAS_ERROR) == 0)) {
- pcmd = (device->state.conn >= C_SYNC_SOURCE &&
- device->state.conn <= C_PAUSED_SYNC_T &&
- peer_req->flags & EE_MAY_SET_IN_SYNC) ?
- P_RS_WRITE_ACK : P_WRITE_ACK;
- err = drbd_send_ack(peer_device, pcmd, peer_req);
- if (pcmd == P_RS_WRITE_ACK)
- drbd_set_in_sync(peer_device, sector, peer_req->i.size);
- } else {
- err = drbd_send_ack(peer_device, P_NEG_ACK, peer_req);
- /* we expect it to be marked out of sync anyways...
- * maybe assert this? */
- }
- dec_unacked(device);
+ if (likely((peer_req->flags & EE_WAS_ERROR) == 0)) {
+ drbd_set_in_sync(peer_device, sector, size);
+ err = drbd_send_ack_be(peer_device, P_RS_WRITE_ACK, sector, size, block_id);
+ } else {
+ /* Record failure to sync */
+ drbd_rs_failed_io(peer_device, sector, size);
+
+ err = drbd_send_ack_be(peer_device, P_RS_NEG_ACK, sector, size, block_id);
}
- /* we delete from the conflict detection hash _after_ we sent out the
- * P_WRITE_ACK / P_NEG_ACK, to get the sequence number right. */
- if (peer_req->flags & EE_IN_INTERVAL_TREE) {
- spin_lock_irq(&device->resource->req_lock);
- D_ASSERT(device, !drbd_interval_empty(&peer_req->i));
- drbd_remove_epoch_entry_interval(device, peer_req);
- if (peer_req->flags & EE_RESTART_REQUESTS)
- restart_conflicting_writes(device, sector, peer_req->i.size);
- spin_unlock_irq(&device->resource->req_lock);
- } else
- D_ASSERT(device, drbd_interval_empty(&peer_req->i));
+ dec_unacked(peer_device);
- drbd_may_finish_epoch(peer_device->connection, peer_req->epoch, EV_PUT + (cancel ? EV_CLEANUP : 0));
+ /*
+ * If INTERVAL_SUBMITTED is not set, this request was merged into
+ * another discard. It has already been removed from the interval tree.
+ */
+ if (test_bit(INTERVAL_SUBMITTED, &peer_req->i.flags))
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_resync_request_complete(peer_req);
return err;
}
-static int e_send_ack(struct drbd_work *w, enum drbd_packet ack)
+static struct drbd_peer_request *__find_resync_request(struct drbd_peer_device *peer_device,
+ unsigned long type_mask, sector_t sector, unsigned int size, u64 block_id,
+ bool expected)
{
- struct drbd_peer_request *peer_req =
- container_of(w, struct drbd_peer_request, w);
- struct drbd_peer_device *peer_device = peer_req->peer_device;
- int err;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_interval *i;
+ struct drbd_peer_request *peer_req = NULL;
- err = drbd_send_ack(peer_device, ack, peer_req);
- dec_unacked(peer_device->device);
+ spin_lock_irq(&device->interval_lock);
+ drbd_for_each_overlap(i, &device->requests, sector, size) {
+ struct drbd_peer_request *pr;
- return err;
-}
+ if (!test_bit(INTERVAL_READY_TO_SEND, &i->flags))
+ continue;
-static int e_send_superseded(struct drbd_work *w, int unused)
-{
- return e_send_ack(w, P_SUPERSEDED);
-}
+ if (!(INTERVAL_TYPE_MASK(i->type) & type_mask))
+ continue;
-static int e_send_retry_write(struct drbd_work *w, int unused)
-{
- struct drbd_peer_request *peer_req =
- container_of(w, struct drbd_peer_request, w);
- struct drbd_connection *connection = peer_req->peer_device->connection;
+ if (i->sector != sector || i->size != size)
+ continue;
- return e_send_ack(w, connection->agreed_pro_version >= 100 ?
- P_RETRY_WRITE : P_SUPERSEDED);
-}
+ pr = container_of(i, struct drbd_peer_request, i);
+ /* With agreed_pro_version < 122, block_id is always ID_SYNCER. */
+ if (pr->peer_device == peer_device &&
+ (block_id == ID_SYNCER || pr->block_id == block_id)) {
+ peer_req = pr;
+ break;
+ }
+ }
+ spin_unlock_irq(&device->interval_lock);
-static bool seq_greater(u32 a, u32 b)
-{
- /*
- * We assume 32-bit wrap-around here.
- * For 24-bit wrap-around, we would have to shift:
- * a <<= 8; b <<= 8;
- */
- return (s32)a - (s32)b > 0;
+ if (peer_req)
+ D_ASSERT(peer_device, peer_req->i.size == size);
+ else if (expected && drbd_ratelimit())
+ drbd_err(peer_device, "Unexpected resync reply at %llus+%u\n",
+ (unsigned long long) sector, size);
+
+ return peer_req;
}
-static u32 seq_max(u32 a, u32 b)
+static struct drbd_peer_request *find_resync_request(struct drbd_peer_device *peer_device,
+ unsigned long type_mask, sector_t sector, unsigned int size, u64 block_id)
{
- return seq_greater(a, b) ? a : b;
+ return __find_resync_request(peer_device, type_mask, sector, size, block_id, true);
}
-static void update_peer_seq(struct drbd_peer_device *peer_device, unsigned int peer_seq)
+static void drbd_cleanup_received_resync_write(struct drbd_peer_request *peer_req)
{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
struct drbd_device *device = peer_device->device;
- unsigned int newest_peer_seq;
- if (test_bit(RESOLVE_CONFLICTS, &peer_device->connection->flags)) {
- spin_lock(&device->peer_seq_lock);
- newest_peer_seq = seq_max(device->peer_seq, peer_seq);
- device->peer_seq = newest_peer_seq;
- spin_unlock(&device->peer_seq_lock);
- /* wake up only if we actually changed device->peer_seq */
- if (peer_seq == newest_peer_seq)
- wake_up(&device->seq_wait);
- }
-}
-
-static inline int overlaps(sector_t s1, int l1, sector_t s2, int l2)
-{
- return !((s1 + (l1>>9) <= s2) || (s1 >= s2 + (l2>>9)));
-}
+ drbd_remove_peer_req_interval(peer_req);
-/* maybe change sync_ee into interval trees as well? */
-static bool overlapping_resync_write(struct drbd_device *device, struct drbd_peer_request *peer_req)
-{
- struct drbd_peer_request *rs_req;
- bool rv = false;
+ dec_unacked(peer_device);
- spin_lock_irq(&device->resource->req_lock);
- list_for_each_entry(rs_req, &device->sync_ee, w.list) {
- if (overlaps(peer_req->i.sector, peer_req->i.size,
- rs_req->i.sector, rs_req->i.size)) {
- rv = true;
- break;
- }
- }
- spin_unlock_irq(&device->resource->req_lock);
+ drbd_free_peer_req(peer_req);
+ put_ldev(device);
- return rv;
+ if (atomic_dec_and_test(&connection->backing_ee_cnt))
+ wake_up(&connection->ee_wait);
}
-/* Called from receive_Data.
- * Synchronize packets on sock with packets on msock.
+/* How much resync IO may be queued at the backing device at once, in sectors.
*
- * This is here so even when a P_DATA packet traveling via sock overtook an Ack
- * packet traveling on msock, they are still processed in the order they have
- * been sent.
- *
- * Note: we don't care for Ack packets overtaking P_DATA packets.
+ * We try to keep the queue of "in-flight" resync IO small-ish even
+ * while we do not see application activity so application IO does not
+ * suffer extreme latency tails even after "quiet" periods.
*
- * In case packet_seq is larger than device->peer_seq number, there are
- * outstanding packets on the msock. We wait for them to arrive.
- * In case we are the logically next packet, we update device->peer_seq
- * ourselves. Correctly handles 32bit wrap around.
+ * drbd_rs_depth_completed() measures how fast resync IO is worked off.
+ * RS_DEPTH_TARGET_MS of that rate is the delay an ongoing resync may add
+ * to one application request.
*
- * Assume we have a 10 GBit connection, that is about 1<<30 byte per second,
- * about 1<<21 sectors per second. So "worst" case, we have 1<<3 == 8 seconds
- * for the 24bit wrap (historical atomic_t guarantee on some archs), and we have
- * 1<<9 == 512 seconds aka ages for the 32bit wrap around...
+ * The rate is measured rather than taken from c-max-rate because a device that
+ * does not deliver what c-max-rate permits would get a target overstating its
+ * queue by the same factor -- widest on the slow device where an application
+ * request suffers most. A device that needs longer than RS_DEPTH_TARGET_MS for
+ * RS_DEPTH_MIN_SECT ends up at that floor, which is as far as bounding a queue
+ * can go: below it a resync queues no more than the request the device is
+ * working on, and an application request waits for that one in any case.
*
- * returns 0 if we may process the packet,
- * -ERESTARTSYS if we were interrupted (by disconnect signal). */
-static int wait_for_and_update_peer_seq(struct drbd_peer_device *peer_device, const u32 peer_seq)
+ * On a high latency device (clat > RS_DEPTH_TARGET_MS), that alone
+ * would not keep it busy. Add the device's service time.
+ */
+static unsigned int rs_depth_target_sect(struct drbd_device *device)
{
- struct drbd_device *device = peer_device->device;
- DEFINE_WAIT(wait);
- long timeout;
- int ret = 0, tp;
-
- if (!test_bit(RESOLVE_CONFLICTS, &peer_device->connection->flags))
- return 0;
-
- spin_lock(&device->peer_seq_lock);
- for (;;) {
- if (!seq_greater(peer_seq - 1, device->peer_seq)) {
- device->peer_seq = seq_max(device->peer_seq, peer_seq);
- break;
- }
-
- if (signal_pending(current)) {
- ret = -ERESTARTSYS;
- break;
- }
-
- rcu_read_lock();
- tp = rcu_dereference(peer_device->connection->net_conf)->two_primaries;
- rcu_read_unlock();
+ u64 ns = (u64)RS_DEPTH_TARGET_MS * NSEC_PER_MSEC + READ_ONCE(device->rs_lat_ns);
+ u64 sect = (u64)READ_ONCE(device->rs_drain_rate) * ns;
- if (!tp)
- break;
+ sect = div64_u64(sect, NSEC_PER_SEC);
- /* Only need to wait if two_primaries is enabled */
- prepare_to_wait(&device->seq_wait, &wait, TASK_INTERRUPTIBLE);
- spin_unlock(&device->peer_seq_lock);
- rcu_read_lock();
- timeout = rcu_dereference(peer_device->connection->net_conf)->ping_timeo*HZ/10;
- rcu_read_unlock();
- timeout = schedule_timeout(timeout);
- spin_lock(&device->peer_seq_lock);
- if (!timeout) {
- ret = -ETIMEDOUT;
- drbd_err(device, "Timed out waiting for missing ack packets; disconnecting\n");
- break;
- }
- }
- spin_unlock(&device->peer_seq_lock);
- finish_wait(&device->seq_wait, &wait);
- return ret;
+ return clamp_t(u64, sect, RS_DEPTH_MIN_SECT, INT_MAX);
}
-static enum req_op wire_flags_to_bio_op(u32 dpf)
+/* Busy time the drain rate is measured over: at least one service time,
+ * so that a sample does not end on a burst of completions the device
+ * worked on concurrently.
+ */
+static u64 rs_drain_sample_ns(struct drbd_device *device)
{
- if (dpf & DP_ZEROES)
- return REQ_OP_WRITE_ZEROES;
- if (dpf & DP_DISCARD)
- return REQ_OP_DISCARD;
- else
- return REQ_OP_WRITE;
+ return max_t(u64, READ_ONCE(device->rs_lat_ns), RS_DRAIN_SAMPLE_NS);
}
-/* see also bio_flags_to_wire() */
-static blk_opf_t wire_flags_to_bio(struct drbd_connection *connection, u32 dpf)
+/* Sectors the backing device delivers in one sample interval at the rate
+ * measured so far, and RS_DEPTH_MIN_SECT until there is a rate.
+ * Computed where the rate or the service time changes, not per IO.
+ */
+static unsigned int rs_drain_sample_sect(struct drbd_device *device)
{
- return wire_flags_to_bio_op(dpf) |
- (dpf & DP_RW_SYNC ? REQ_SYNC : 0) |
- (dpf & DP_FUA ? REQ_FUA : 0) |
- (dpf & DP_FLUSH ? REQ_PREFLUSH : 0);
+ u64 sect = (u64)READ_ONCE(device->rs_drain_rate) * rs_drain_sample_ns(device);
+
+ sect = div64_u64(sect, NSEC_PER_SEC);
+
+ return clamp_t(u64, sect, RS_DEPTH_MIN_SECT, UINT_MAX);
}
-static void fail_postponed_requests(struct drbd_device *device, sector_t sector,
- unsigned int size)
+/* May more resync IO go to the backing device?
+ * While a probe is due, only into an empty queue.
+ */
+static bool rs_depth_room(struct drbd_device *device)
{
- struct drbd_peer_device *peer_device = first_peer_device(device);
- struct drbd_interval *i;
-
- repeat:
- drbd_for_each_overlap(i, &device->write_requests, sector, size) {
- struct drbd_request *req;
- struct bio_and_error m;
+ unsigned int queued = atomic_read(&device->rs_sect_queued);
- if (!drbd_interval_is_local(i))
- continue;
- req = container_of(i, struct drbd_request, i);
- if (!(req->rq_state & RQ_POSTPONED))
- continue;
- req->rq_state &= ~RQ_POSTPONED;
- __req_mod(req, NEG_ACKED, peer_device, &m);
- spin_unlock_irq(&device->resource->req_lock);
- if (m.bio)
- complete_master_bio(device, &m);
- spin_lock_irq(&device->resource->req_lock);
- goto repeat;
- }
+ if (READ_ONCE(device->rs_lat_probe) == RS_LAT_DRAIN)
+ return queued == 0;
+ return queued < rs_depth_target_sect(device);
}
-static int handle_write_conflicts(struct drbd_device *device,
- struct drbd_peer_request *peer_req)
+bool drbd_rs_depth_exceeded(struct drbd_peer_device *peer_device)
{
- struct drbd_connection *connection = peer_req->peer_device->connection;
- bool resolve_conflicts = test_bit(RESOLVE_CONFLICTS, &connection->flags);
- sector_t sector = peer_req->i.sector;
- const unsigned int size = peer_req->i.size;
+ struct drbd_device *device = peer_device->device;
+ unsigned int c_min_rate;
+
+ rcu_read_lock();
+ c_min_rate = rcu_dereference(peer_device->conf)->c_min_rate;
+ rcu_read_unlock();
+
+ /* the extra throttle is disabled, and with it this bound */
+ if (c_min_rate == 0)
+ return false;
+
+ return !rs_depth_room(device);
+}
+
+static void rs_depth_resume(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+
+ if (!test_and_clear_bit(RS_DEPTH_WAITERS, &device->flags))
+ return;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (test_and_clear_bit(RS_DEPTH_DEFERRED, peer_device->flags))
+ drbd_queue_work_if_unqueued(&peer_device->connection->sender_work,
+ &peer_device->resync_work);
+ }
+ rcu_read_unlock();
+}
+
+/* A checksum resync round that stopped at the depth bound starts again as soon
+ * as the queue has room, instead of at the next resync timer tick.
+ */
+void drbd_rs_depth_defer(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+
+ set_bit(RS_DEPTH_DEFERRED, peer_device->flags);
+ set_bit(RS_DEPTH_WAITERS, &device->flags);
+
+ /* The completion that makes room may have run since the check. */
+ smp_mb__after_atomic(); /* pairs with drbd_rs_depth_release() */
+ if (!drbd_rs_depth_exceeded(peer_device))
+ rs_depth_resume(device);
+}
+
+/* An application write waits for the resync requests it conflicts with.
+ * Do not let it wait for the depth bound as well: exempt those
+ * requests from the bound, and release the ones already parked there.
+ */
+void drbd_rs_depth_exempt(struct drbd_device *device, struct drbd_interval *interval)
+{
+ struct conflict_worker *submit_conflict = &device->submit_conflict;
struct drbd_interval *i;
- bool equal;
- int err;
+ bool parked = false;
- /*
- * Inserting the peer request into the write_requests tree will prevent
- * new conflicting local requests from being added.
- */
- drbd_insert_interval(&device->write_requests, &peer_req->i);
+ lockdep_assert_held(&device->interval_lock);
- repeat:
- drbd_for_each_overlap(i, &device->write_requests, sector, size) {
- if (i == &peer_req->i)
+ drbd_for_each_overlap(i, &device->requests, interval->sector, interval->size) {
+ if (i == interval || test_bit(INTERVAL_SUBMITTED, &i->flags) ||
+ !drbd_should_defer_to_resync(interval, i))
continue;
- if (test_bit(INTERVAL_COMPLETED, &i->flags))
+ if (test_and_set_bit(INTERVAL_APP_WAITS, &i->flags))
continue;
+ if (test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags))
+ parked = true;
+ }
- if (!drbd_interval_is_local(i)) {
- /*
- * Our peer has sent a conflicting remote request; this
- * should not happen in a two-node setup. Wait for the
- * earlier peer request to complete.
- */
- err = drbd_wait_misc(device, i);
- if (err)
- goto out;
- goto repeat;
- }
+ if (parked)
+ queue_work(submit_conflict->wq, &submit_conflict->worker);
+}
- equal = i->sector == sector && i->size == size;
- if (resolve_conflicts) {
- /*
- * If the peer request is fully contained within the
- * overlapping request, it can be considered overwritten
- * and thus superseded; otherwise, it will be retried
- * once all overlapping requests have completed.
- */
- bool superseded = i->sector <= sector && i->sector +
- (i->size >> 9) >= sector + (size >> 9);
-
- if (!equal)
- drbd_alert(device, "Concurrent writes detected: "
- "local=%llus +%u, remote=%llus +%u, "
- "assuming %s came first\n",
- (unsigned long long)i->sector, i->size,
- (unsigned long long)sector, size,
- superseded ? "local" : "remote");
-
- peer_req->w.cb = superseded ? e_send_superseded :
- e_send_retry_write;
- list_add_tail(&peer_req->w.list, &device->done_ee);
- /* put is in drbd_send_acks_wf() */
- kref_get(&device->kref);
- if (!queue_work(connection->ack_sender,
- &peer_req->peer_device->send_acks_work))
- kref_put(&device->kref, drbd_destroy_device);
+/* Resync IO completed at the backing device, so the queue is shorter than it
+ * was. Let the conflict submitter look again at what it is holding.
+ */
+void drbd_rs_depth_release(struct drbd_device *device)
+{
+ struct conflict_worker *submit_conflict = &device->submit_conflict;
- err = -ENOENT;
- goto out;
- } else {
- struct drbd_request *req =
- container_of(i, struct drbd_request, i);
+ if (!list_empty_careful(&submit_conflict->resync_writes) ||
+ !list_empty_careful(&submit_conflict->resync_reads))
+ queue_work(submit_conflict->wq, &submit_conflict->worker);
- if (!equal)
- drbd_alert(device, "Concurrent writes detected: "
- "local=%llus +%u, remote=%llus +%u\n",
- (unsigned long long)i->sector, i->size,
- (unsigned long long)sector, size);
+ if (test_bit(RS_DEPTH_WAITERS, &device->flags) && rs_depth_room(device))
+ rs_depth_resume(device);
+}
- if (req->rq_state & RQ_LOCAL_PENDING ||
- !(req->rq_state & RQ_POSTPONED)) {
- /*
- * Wait for the node with the discard flag to
- * decide if this request has been superseded
- * or needs to be retried.
- * Requests that have been superseded will
- * disappear from the write_requests tree.
- *
- * In addition, wait for the conflicting
- * request to finish locally before submitting
- * the conflicting peer request.
- */
- err = drbd_wait_misc(device, &req->i);
- if (err) {
- _conn_request_state(connection, NS(conn, C_TIMEOUT), CS_HARD);
- fail_postponed_requests(device, sector, size);
- goto out;
- }
- goto repeat;
- }
- /*
- * Remember to restart the conflicting requests after
- * the new peer request has completed.
- */
- peer_req->flags |= EE_RESTART_REQUESTS;
+/* Resync IO submitted to the backing device. */
+void drbd_rs_depth_queued(struct drbd_device *device, unsigned int sect)
+{
+ if (!sect)
+ return;
+
+ /* Time the device from where it becomes busy: how long it was idle
+ * says nothing about how fast it works resync IO off.
+ */
+ if (atomic_add_return(sect, &device->rs_sect_queued) == sect) {
+ ktime_t now = ktime_get();
+
+ device->rs_drain_since = now;
+ if (READ_ONCE(device->rs_lat_probe) == RS_LAT_DRAIN) {
+ device->rs_lat_probe_kt = now;
+ WRITE_ONCE(device->rs_lat_probe, RS_LAT_TIMING);
}
}
- err = 0;
-
- out:
- if (err)
- drbd_remove_epoch_entry_interval(device, peer_req);
- return err;
}
-/* mirrored write */
-static int receive_Data(struct drbd_connection *connection, struct packet_info *pi)
+/* Does resync completion latency, read in jiffies, show a service time other
+ * than the one probed? Only one more than a jiffy away in either direction
+ * counts: lat_jif is a difference of two jiffies values.
+ */
+static bool rs_lat_changed(struct drbd_device *device, unsigned long lat_jif)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct net_conf *nc;
- sector_t sector;
- struct drbd_peer_request *peer_req;
- struct p_data *p = pi->data;
- u32 peer_seq = be32_to_cpu(p->seq_num);
- u32 dp_flags;
- int err, tp;
+ u64 lat_ns = READ_ONCE(device->rs_lat_ns);
+ u64 lo = lat_jif ? (u64)jiffies_to_msecs(lat_jif - 1) * NSEC_PER_MSEC : 0;
+ u64 hi = (u64)jiffies_to_msecs(lat_jif + 1) * NSEC_PER_MSEC;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ return lo > lat_ns + (u64)RS_DEPTH_TARGET_MS * NSEC_PER_MSEC || hi < lat_ns;
+}
- if (!get_ldev(device)) {
- int err2;
+/* Watch resync completion latency in jiffies, which costs no clock read.
+ * Its minimum over a window is the service time plus the queue the
+ * bound allows; probe again when that disagrees with the probed
+ * service time, and every RS_LAT_PROBE_INTV in any case.
+ */
+static void rs_lat_watch(struct drbd_device *device, unsigned long submit_jif)
+{
+ unsigned long now = jiffies, lat = now - submit_jif + 1;
+ unsigned long wmin = device->rs_lat_wmin;
- err = wait_for_and_update_peer_seq(peer_device, peer_seq);
- drbd_send_ack_dp(peer_device, P_NEG_ACK, p, pi->size);
- atomic_inc(&connection->current_epoch->epoch_size);
- err2 = drbd_drain_block(peer_device, pi->size);
- if (!err)
- err = err2;
- return err;
+ if (!wmin || lat < wmin) {
+ wmin = lat;
+ device->rs_lat_wmin = wmin;
}
+ if (time_before(now, device->rs_lat_wend))
+ return;
- /*
- * Corresponding put_ldev done either below (on various errors), or in
- * drbd_peer_request_endio, if we successfully submit the data at the
- * end of this function.
+ device->rs_lat_wmin = 0;
+ device->rs_lat_wend = now + RS_LAT_WINDOW;
+ if (rs_lat_changed(device, wmin - 1) ||
+ time_after_eq(now, device->rs_lat_due))
+ WRITE_ONCE(device->rs_lat_probe, RS_LAT_DRAIN);
+}
+
+/* Resync IO completed at the backing device: the queue is shorter than it was,
+ * and the device just showed how fast it works that queue off. Measure that
+ * over busy time, so the rate is the device's own and not a function of how
+ * much resync work happened to be there -- at the RS_DEPTH_MIN_SECT floor a
+ * completion empties the queue, and that has to measure as well as any depth.
+ *
+ * The service time is timed on a probe: hold resync IO until the queue
+ * is empty, then time the first completion after the next submission.
+ * Every request that can complete first was submitted no earlier than
+ * the probe and completes no later, so that time is the service time
+ * of an idle queue.
+ */
+void drbd_rs_depth_completed(struct drbd_device *device, unsigned int sect,
+ unsigned long submit_jif)
+{
+ unsigned int drained, sect_since;
+ s64 busy_ns, elapsed_ns;
+ ktime_t now;
+ bool idle;
+ int probe;
+
+ drained = atomic_add_return(sect, &device->rs_drain_sect);
+ idle = atomic_sub_return(sect, &device->rs_sect_queued) == 0;
+ sect_since = drained - device->rs_drain_mark;
+
+ probe = READ_ONCE(device->rs_lat_probe);
+ if (probe == RS_LAT_IDLE)
+ rs_lat_watch(device, submit_jif);
+
+ /* Reading the clock can be costly.
+ * Read it only where the answer is used: when the queue just ran empty
+ * and the busy time has to stop there, when this completion carries
+ * enough sectors to end the sample, or when it ends a probe.
*/
+ if (!idle && sect_since < READ_ONCE(device->rs_drain_sample_sect) &&
+ probe != RS_LAT_TIMING)
+ goto release;
- sector = be64_to_cpu(p->sector);
- peer_req = read_in_block(peer_device, p->block_id, sector, pi);
- if (!peer_req) {
- put_ldev(device);
- return -EIO;
+ now = ktime_get();
+ if (probe == RS_LAT_TIMING) {
+ WRITE_ONCE(device->rs_lat_ns,
+ clamp_t(s64, ktime_to_ns(ktime_sub(now, device->rs_lat_probe_kt)),
+ 1, RS_LAT_MAX_NS));
+ device->rs_lat_due = jiffies + RS_LAT_PROBE_INTV;
+ device->rs_lat_wend = jiffies + RS_LAT_WINDOW;
+ device->rs_lat_wmin = 0;
+ WRITE_ONCE(device->rs_lat_probe, RS_LAT_IDLE);
+ WRITE_ONCE(device->rs_drain_sample_sect, rs_drain_sample_sect(device));
}
+ elapsed_ns = ktime_to_ns(ktime_sub(now, device->rs_drain_since));
+ busy_ns = device->rs_drain_busy_ns + (elapsed_ns > 0 ? elapsed_ns : 0);
- peer_req->i.type = INTERVAL_PEER_WRITE;
- peer_req->w.cb = e_end_block;
- peer_req->submit_jif = jiffies;
- peer_req->flags |= EE_APPLICATION;
+ if (idle)
+ device->rs_drain_busy_ns = busy_ns; /* idle from here */
- dp_flags = be32_to_cpu(p->dp_flags);
- peer_req->opf = wire_flags_to_bio(connection, dp_flags);
- if (pi->cmd == P_TRIM) {
- D_ASSERT(peer_device, peer_req->i.size > 0);
- D_ASSERT(peer_device, peer_req_op(peer_req) == REQ_OP_DISCARD);
- D_ASSERT(peer_device, peer_req->pages == NULL);
- /* need to play safe: an older DRBD sender
- * may mean zero-out while sending P_TRIM. */
- if (0 == (connection->agreed_features & DRBD_FF_WZEROES))
- peer_req->flags |= EE_ZEROOUT;
- } else if (pi->cmd == P_ZEROES) {
- D_ASSERT(peer_device, peer_req->i.size > 0);
- D_ASSERT(peer_device, peer_req_op(peer_req) == REQ_OP_WRITE_ZEROES);
- D_ASSERT(peer_device, peer_req->pages == NULL);
- /* Do (not) pass down BLKDEV_ZERO_NOUNMAP? */
- if (dp_flags & DP_DISCARD)
- peer_req->flags |= EE_TRIM;
- } else if (peer_req->pages == NULL) {
- D_ASSERT(device, peer_req->i.size == 0);
- D_ASSERT(device, dp_flags & DP_FLUSH);
- }
+ if (busy_ns >= rs_drain_sample_ns(device)) {
+ u64 rate = (u64)sect_since * NSEC_PER_SEC;
- if (dp_flags & DP_MAY_SET_IN_SYNC)
- peer_req->flags |= EE_MAY_SET_IN_SYNC;
+ WRITE_ONCE(device->rs_drain_rate,
+ min_t(u64, div64_u64(rate, busy_ns), RS_DRAIN_RATE_MAX));
+ device->rs_drain_mark = drained;
+ device->rs_drain_busy_ns = 0;
+ device->rs_drain_since = now;
+ WRITE_ONCE(device->rs_drain_sample_sect, rs_drain_sample_sect(device));
+ }
- spin_lock(&connection->epoch_lock);
- peer_req->epoch = connection->current_epoch;
- atomic_inc(&peer_req->epoch->epoch_size);
- atomic_inc(&peer_req->epoch->active);
- spin_unlock(&connection->epoch_lock);
+release:
+ drbd_rs_depth_release(device);
+}
- rcu_read_lock();
- nc = rcu_dereference(peer_device->connection->net_conf);
- tp = nc->two_primaries;
- if (peer_device->connection->agreed_pro_version < 100) {
- switch (nc->wire_protocol) {
- case DRBD_PROT_C:
- dp_flags |= DP_SEND_WRITE_ACK;
- break;
- case DRBD_PROT_B:
- dp_flags |= DP_SEND_RECEIVE_ACK;
- break;
- }
- }
- rcu_read_unlock();
+void drbd_conflict_submit_resync_request(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ struct conflict_worker *submit_conflict = &device->submit_conflict;
+ bool conflict;
+ bool canceled;
+ bool too_deep;
+
+ spin_lock_irq(&device->interval_lock);
+ if (peer_req->flags & EE_RS_DEPTH_PARKED) {
+ peer_req->flags &= ~EE_RS_DEPTH_PARKED;
+ atomic_sub(peer_req->i.size >> 9, &peer_device->rs_sect_parked);
+ }
+ clear_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &peer_req->i.flags);
+ canceled = test_bit(INTERVAL_CANCELED, &peer_req->i.flags);
+ set_bit(INTERVAL_RECEIVED, &peer_req->i.flags);
+ conflict = drbd_find_conflict(device, &peer_req->i, 0);
+ /* A discard queues nothing at the device, so holding it back would
+ * shorten no queue. Its zero-out fallback does, and is not bounded.
+ */
+ too_deep = !conflict && !canceled && !(peer_req->flags & EE_TRIM) &&
+ !test_bit(INTERVAL_APP_WAITS, &peer_req->i.flags) &&
+ drbd_rs_depth_exceeded(peer_device);
+ if (too_deep)
+ set_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &peer_req->i.flags);
+ else if (!conflict && !canceled)
+ set_bit(INTERVAL_SUBMITTED, &peer_req->i.flags);
+ if (too_deep) {
+ peer_req->flags |= EE_RS_DEPTH_PARKED;
+ atomic_add(peer_req->i.size >> 9, &peer_device->rs_sect_parked);
+ }
+ spin_unlock_irq(&device->interval_lock);
+
+ if (too_deep) {
+ /* Wait for the backing device the same way a conflicting interval
+ * waits: on the conflict submitter's list, so the receiver goes
+ * back to the stream. It must -- on a shared data stream the
+ * application's writes are queued behind this one.
+ */
+ spin_lock_irq(&submit_conflict->lock);
+ list_add_tail(&peer_req->w.list, &submit_conflict->resync_writes);
+ spin_unlock_irq(&submit_conflict->lock);
- if (dp_flags & DP_SEND_WRITE_ACK) {
- peer_req->flags |= EE_SEND_WRITE_ACK;
- inc_unacked(device);
- /* corresponding dec_unacked() in e_end_block()
- * respective _drbd_clear_done_ee */
+ /* The completion that empties the queue may have run since the
+ * check above, and then it did not see this request.
+ */
+ smp_mb(); /* list_add vs. re-check; see drbd_rs_depth_completed() */
+ if (!drbd_rs_depth_exceeded(peer_device))
+ queue_work(submit_conflict->wq, &submit_conflict->worker);
+ return;
}
- if (dp_flags & DP_SEND_RECEIVE_ACK) {
- /* I really don't like it that the receiver thread
- * sends on the msock, but anyways */
- drbd_send_ack(peer_device, P_RECV_ACK, peer_req);
- }
+ if (!conflict && !canceled) {
+ unsigned int size = peer_req->flags & EE_TRIM ? 0 : peer_req->i.size;
+ int err;
- if (tp) {
- /* two primaries implies protocol C */
- D_ASSERT(device, dp_flags & DP_SEND_WRITE_ACK);
- peer_req->flags |= EE_IN_INTERVAL_TREE;
- err = wait_for_and_update_peer_seq(peer_device, peer_seq);
- if (err)
- goto out_interrupted;
- spin_lock_irq(&device->resource->req_lock);
- err = handle_write_conflicts(device, peer_req);
+ drbd_rs_depth_queued(device, size >> 9);
+ err = drbd_submit_peer_request(peer_req);
if (err) {
- spin_unlock_irq(&device->resource->req_lock);
- if (err == -ENOENT) {
- put_ldev(device);
- return 0;
- }
- goto out_interrupted;
- }
- } else {
- update_peer_seq(peer_device, peer_seq);
- spin_lock_irq(&device->resource->req_lock);
- }
- /* TRIM and is processed synchronously,
- * we wait for all pending requests, respectively wait for
- * active_ee to become empty in drbd_submit_peer_request();
- * better not add ourselves here. */
- if ((peer_req->flags & (EE_TRIM | EE_ZEROOUT)) == 0)
- list_add_tail(&peer_req->w.list, &device->active_ee);
- spin_unlock_irq(&device->resource->req_lock);
-
- if (device->state.conn == C_SYNC_TARGET)
- wait_event(device->ee_wait, !overlapping_resync_write(device, peer_req));
-
- if (device->state.pdsk < D_INCONSISTENT) {
- /* In case we have the only disk of the cluster, */
- drbd_set_out_of_sync(peer_device, peer_req->i.sector, peer_req->i.size);
- peer_req->flags &= ~EE_MAY_SET_IN_SYNC;
- drbd_al_begin_io(device, &peer_req->i);
- peer_req->flags |= EE_CALL_AL_COMPLETE_IO;
- }
-
- err = drbd_submit_peer_request(peer_req);
- if (!err)
- return 0;
+ atomic_sub(size >> 9, &device->rs_sect_queued);
+ if (drbd_ratelimit())
+ drbd_err(device, "submit failed, triggering re-connect\n");
- /* don't care for the reason here */
- drbd_err(device, "submit failed, triggering re-connect\n");
- spin_lock_irq(&device->resource->req_lock);
- list_del(&peer_req->w.list);
- drbd_remove_epoch_entry_interval(device, peer_req);
- spin_unlock_irq(&device->resource->req_lock);
- if (peer_req->flags & EE_CALL_AL_COMPLETE_IO) {
- peer_req->flags &= ~EE_CALL_AL_COMPLETE_IO;
- drbd_al_complete_io(device, &peer_req->i);
+ drbd_cleanup_received_resync_write(peer_req);
+ change_cstate(peer_device->connection, C_PROTOCOL_ERROR, CS_HARD);
+ }
+ } else if (canceled) {
+ drbd_cleanup_received_resync_write(peer_req);
}
-
-out_interrupted:
- drbd_may_finish_epoch(connection, peer_req->epoch, EV_PUT | EV_CLEANUP);
- put_ldev(device);
- drbd_free_peer_req(device, peer_req);
- return err;
}
-/* We may throttle resync, if the lower device seems to be busy,
- * and current sync rate is above c_min_rate.
- *
- * To decide whether or not the lower device is busy, we use a scheme similar
- * to MD RAID is_mddev_idle(): if the partition stats reveal "significant"
- * (more than 64 sectors) of activity we cannot account for with our own resync
- * activity, it obviously is "busy".
- *
- * The current sync rate used here uses only the most recent two step marks,
- * to have a short time average so we can react faster.
- */
-bool drbd_rs_should_slow_down(struct drbd_peer_device *peer_device, sector_t sector,
- bool throttle_if_app_is_waiting)
+static int recv_resync_read(struct drbd_peer_device *peer_device,
+ struct drbd_peer_request *peer_req,
+ struct drbd_peer_request_details *d)
{
+ struct drbd_connection *connection = peer_device->connection;
struct drbd_device *device = peer_device->device;
- struct lc_element *tmp;
- bool throttle = drbd_rs_c_min_rate_throttle(device);
+ unsigned int size;
+ sector_t sector;
+ int err;
+ u64 im;
- if (!throttle || throttle_if_app_is_waiting)
- return throttle;
+ err = read_in_block(peer_req, d);
+ if (err)
+ return err;
- spin_lock_irq(&device->al_lock);
- tmp = lc_find(device->resync, BM_SECT_TO_EXT(sector));
- if (tmp) {
- struct bm_extent *bm_ext = lc_entry(tmp, struct bm_extent, lce);
- if (test_bit(BME_PRIORITY, &bm_ext->flags))
- throttle = false;
- /* Do not slow down if app IO is already waiting for this extent,
- * and our progress is necessary for application IO to complete. */
- }
- spin_unlock_irq(&device->al_lock);
+ if (test_bit(UNSTABLE_RESYNC, peer_device->flags))
+ clear_bit(STABLE_RESYNC, &device->flags);
- return throttle;
-}
+ dec_rs_pending(peer_device);
-bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)
-{
- struct gendisk *disk = device->ldev->backing_bdev->bd_disk;
- unsigned long db, dt, dbdt;
- unsigned int c_min_rate;
- int curr_events;
+ inc_unacked(peer_device);
+ /* corresponding dec_unacked() in e_end_resync_block()
+ * respective _drbd_clear_done_ee */
- rcu_read_lock();
- c_min_rate = rcu_dereference(device->ldev->disk_conf)->c_min_rate;
- rcu_read_unlock();
+ peer_req->w.cb = e_end_resync_block;
+ peer_req->submit_jif = jiffies;
- /* feature disabled? */
- if (c_min_rate == 0)
- return false;
+ sector = peer_req->i.sector;
+ size = peer_req->i.size;
- curr_events = (int)part_stat_read_accum(disk->part0, sectors) -
- atomic_read(&device->rs_sect_ev);
+ /* Setting all peers out of sync here. The sync source peer will be
+ * set in sync when the write completes. The sync source will soon
+ * set other peers in sync with a P_PEERS_IN_SYNC packet.
+ */
+ drbd_set_all_out_of_sync(device, peer_device->node_id, sector, size);
- if (atomic_read(&device->ap_actlog_cnt)
- || curr_events - device->rs_last_events > 64) {
- unsigned long rs_left;
- int i;
+ atomic_inc(&connection->backing_ee_cnt);
+ drbd_conflict_submit_resync_request(peer_req);
+ peer_req = NULL; /* since submitted, might be destroyed already */
- device->rs_last_events = curr_events;
+ drbd_process_rs_discards(peer_device, false);
- /* sync speed average over the last 2*DRBD_SYNC_MARK_STEP,
- * approx. */
- i = (device->rs_last_mark + DRBD_SYNC_MARKS-1) % DRBD_SYNC_MARKS;
+ for_each_peer_device_ref(peer_device, im, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
- if (device->state.conn == C_VERIFY_S || device->state.conn == C_VERIFY_T)
- rs_left = device->ov_left;
- else
- rs_left = drbd_bm_total_weight(device) - device->rs_failed;
+ if (repl_is_sync_source(repl_state) || repl_state == L_WF_BITMAP_S)
+ drbd_send_out_of_sync(peer_device, sector, size);
+ }
+ return 0;
+}
- dt = ((long)jiffies - (long)device->rs_mark_time[i]) / HZ;
- if (!dt)
- dt++;
- db = device->rs_mark_left[i] - rs_left;
- dbdt = Bit2KB(db/dt);
+/* caller must hold interval_lock */
+static struct drbd_request *
+find_request(struct drbd_device *device, enum drbd_interval_type type, u64 id,
+ sector_t sector, bool missing_ok, const char *func)
+{
+ struct rb_root *root = type == INTERVAL_LOCAL_READ ? &device->read_requests : &device->requests;
+ struct drbd_request *req;
- if (dbdt > c_min_rate)
- return true;
+ /* Request object according to our peer */
+ req = (struct drbd_request *)(unsigned long)id;
+ if (drbd_contains_interval(root, sector, &req->i) && req->i.type == type)
+ return req;
+ if (!missing_ok) {
+ drbd_err(device, "%s: failed to find request 0x%lx, sector %llus\n", func,
+ (unsigned long)id, (unsigned long long)sector);
}
- return false;
+ return NULL;
}
-static int receive_DataRequest(struct drbd_connection *connection, struct packet_info *pi)
+static int receive_DataReply(struct drbd_connection *connection, struct packet_info *pi)
{
struct drbd_peer_device *peer_device;
struct drbd_device *device;
+ struct drbd_request *req;
sector_t sector;
- sector_t capacity;
- struct drbd_peer_request *peer_req;
- struct digest_info *di = NULL;
- int size, verb;
- struct p_block_req *p = pi->data;
+ int err;
+ struct p_data *p = pi->data;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
return -EIO;
device = peer_device->device;
- capacity = get_capacity(device->vdisk);
sector = be64_to_cpu(p->sector);
- size = be32_to_cpu(p->blksize);
- if (size <= 0 || !IS_ALIGNED(size, 512) || size > DRBD_MAX_BIO_SIZE) {
- drbd_err(device, "%s:%d: sector: %llus, size: %u\n", __FILE__, __LINE__,
- (unsigned long long)sector, size);
- return -EINVAL;
- }
- if (sector + (size>>9) > capacity) {
- drbd_err(device, "%s:%d: sector: %llus, size: %u\n", __FILE__, __LINE__,
- (unsigned long long)sector, size);
- return -EINVAL;
- }
+ spin_lock_irq(&device->interval_lock);
+ req = find_request(device, INTERVAL_LOCAL_READ, p->block_id, sector, false, __func__);
+ spin_unlock_irq(&device->interval_lock);
+ if (unlikely(!req))
+ return -EIO;
- if (!get_ldev_if_state(device, D_UP_TO_DATE)) {
- verb = 1;
- switch (pi->cmd) {
- case P_DATA_REQUEST:
- drbd_send_ack_rp(peer_device, P_NEG_DREPLY, p);
- break;
- case P_RS_THIN_REQ:
- case P_RS_DATA_REQUEST:
- case P_CSUM_RS_REQUEST:
- case P_OV_REQUEST:
- drbd_send_ack_rp(peer_device, P_NEG_RS_DREPLY , p);
- break;
- case P_OV_REPLY:
- verb = 0;
- dec_rs_pending(peer_device);
- drbd_send_ack_ex(peer_device, P_OV_RESULT, sector, size, ID_IN_SYNC);
- break;
- default:
- BUG();
- }
- if (verb)
- drbd_err_ratelimit(device,
- "Can not satisfy peer's read request, no local data.\n");
-
- /* drain possibly payload */
- return drbd_drain_block(peer_device, pi->size);
- }
-
- /* GFP_NOIO, because we must not cause arbitrary write-out: in a DRBD
- * "criss-cross" setup, that might cause write-out on some other DRBD,
- * which in turn might block on the other node at this very place. */
- peer_req = drbd_alloc_peer_req(peer_device, p->block_id, sector, size,
- size, GFP_NOIO);
- if (!peer_req) {
- put_ldev(device);
- return -ENOMEM;
- }
- peer_req->opf = REQ_OP_READ;
+ err = recv_dless_read(peer_device, req, sector, pi->size);
+ if (!err)
+ req_mod(req, DATA_RECEIVED, peer_device);
+ /* else: nothing. handled from drbd_disconnect...
+ * I don't think we may complete this just yet
+ * in case we are "on-disconnect: freeze" */
- switch (pi->cmd) {
- case P_DATA_REQUEST:
- peer_req->i.type = INTERVAL_PEER_READ;
- peer_req->w.cb = w_e_end_data_req;
- /* application IO, don't drbd_rs_begin_io */
- peer_req->flags |= EE_APPLICATION;
- goto submit;
+ return err;
+}
- case P_RS_THIN_REQ:
- /* If at some point in the future we have a smart way to
- find out if this data block is completely deallocated,
- then we would do something smarter here than reading
- the block... */
- peer_req->flags |= EE_RS_THIN_REQ;
- fallthrough;
- case P_RS_DATA_REQUEST:
- peer_req->i.type = INTERVAL_RESYNC_READ;
- peer_req->w.cb = w_e_end_rsdata_req;
- /* used in the sector offset progress display */
- device->bm_resync_fo = BM_SECT_TO_BIT(sector);
- break;
+/**
+ * _drbd_send_ack() - Sends an ack packet
+ * @peer_device: DRBD peer device.
+ * @cmd: Packet command code.
+ * @sector: sector, needs to be in big endian byte order
+ * @blksize: size in byte, needs to be in big endian byte order
+ * @block_id: Id, big endian byte order
+ */
+static int _drbd_send_ack(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
+ u64 sector, u32 blksize, u64 block_id)
+{
+ struct p_block_ack *p;
- case P_OV_REPLY:
- case P_CSUM_RS_REQUEST:
- di = kmalloc(sizeof(*di) + pi->size, GFP_NOIO);
- if (!di)
- goto out_free_e;
-
- di->digest_size = pi->size;
- di->digest = (((char *)di)+sizeof(struct digest_info));
-
- peer_req->digest = di;
- peer_req->flags |= EE_HAS_DIGEST;
-
- if (drbd_recv_all(peer_device->connection, di->digest, pi->size))
- goto out_free_e;
-
- if (pi->cmd == P_CSUM_RS_REQUEST) {
- D_ASSERT(device, peer_device->connection->agreed_pro_version >= 89);
- peer_req->i.type = INTERVAL_RESYNC_READ;
- peer_req->w.cb = w_e_end_csum_rs_req;
- /* used in the sector offset progress display */
- device->bm_resync_fo = BM_SECT_TO_BIT(sector);
- /* remember to report stats in drbd_resync_finished */
- device->use_csums = true;
- } else if (pi->cmd == P_OV_REPLY) {
- /* track progress, we may need to throttle */
- atomic_add(size >> 9, &device->rs_sect_in);
- peer_req->i.type = INTERVAL_OV_READ_SOURCE;
- peer_req->w.cb = w_e_end_ov_reply;
- dec_rs_pending(peer_device);
- /* drbd_rs_begin_io done when we sent this request,
- * but accounting still needs to be done. */
- goto submit_for_resync;
- }
- break;
+ if (peer_device->repl_state[NOW] < L_ESTABLISHED)
+ return -EIO;
- case P_OV_REQUEST:
- if (device->ov_start_sector == ~(sector_t)0 &&
- peer_device->connection->agreed_pro_version >= 90) {
- unsigned long now = jiffies;
- int i;
- device->ov_start_sector = sector;
- device->ov_position = sector;
- device->ov_left = drbd_bm_bits(device) - BM_SECT_TO_BIT(sector);
- device->rs_total = device->ov_left;
- for (i = 0; i < DRBD_SYNC_MARKS; i++) {
- device->rs_mark_left[i] = device->ov_left;
- device->rs_mark_time[i] = now;
- }
- drbd_info(device, "Online Verify start sector: %llu\n",
- (unsigned long long)sector);
+ p = drbd_prepare_command(peer_device, sizeof(*p), CONTROL_STREAM);
+ if (!p)
+ return -EIO;
+ p->sector = sector;
+ p->block_id = block_id;
+ p->blksize = blksize;
+ p->seq_num = cpu_to_be32(atomic_inc_return(&peer_device->packet_seq));
+
+ if (peer_device->connection->agreed_pro_version < 122) {
+ switch (cmd) {
+ case P_RS_NEG_ACK:
+ cmd = P_NEG_ACK;
+ p->block_id = ID_SYNCER;
+ break;
+ case P_WRITE_ACK_IN_SYNC:
+ cmd = P_RS_WRITE_ACK;
+ break;
+ case P_RS_WRITE_ACK:
+ p->block_id = ID_SYNCER;
+ break;
+ default:
+ break;
}
- peer_req->i.type = INTERVAL_OV_READ_TARGET;
- peer_req->w.cb = w_e_end_ov_req;
- break;
-
- default:
- BUG();
}
- /* Throttle, drbd_rs_begin_io and submit should become asynchronous
- * wrt the receiver, but it is not as straightforward as it may seem.
- * Various places in the resync start and stop logic assume resync
- * requests are processed in order, requeuing this on the worker thread
- * introduces a bunch of new code for synchronization between threads.
- *
- * Unlimited throttling before drbd_rs_begin_io may stall the resync
- * "forever", throttling after drbd_rs_begin_io will lock that extent
- * for application writes for the same time. For now, just throttle
- * here, where the rest of the code expects the receiver to sleep for
- * a while, anyways.
- */
+ return drbd_send_command(peer_device, cmd, CONTROL_STREAM);
+}
- /* Throttle before drbd_rs_begin_io, as that locks out application IO;
- * this defers syncer requests for some time, before letting at least
- * on request through. The resync controller on the receiving side
- * will adapt to the incoming rate accordingly.
- *
- * We cannot throttle here if remote is Primary/SyncTarget:
- * we would also throttle its application reads.
- * In that case, throttling is done on the SyncTarget only.
- */
+static int drbd_send_ack_dp(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
+ struct drbd_peer_request_details *d)
+{
+ return _drbd_send_ack(peer_device, cmd,
+ cpu_to_be64(d->sector),
+ cpu_to_be32(d->bi_size),
+ d->block_id);
+}
- /* Even though this may be a resync request, we do add to "read_ee";
- * "sync_ee" is only used for resync WRITEs.
- * Add to list early, so debugfs can find this request
- * even if we have to sleep below. */
- spin_lock_irq(&device->resource->req_lock);
- list_add_tail(&peer_req->w.list, &device->read_ee);
- spin_unlock_irq(&device->resource->req_lock);
-
- update_receiver_timing_details(connection, drbd_rs_should_slow_down);
- if (device->state.peer != R_PRIMARY
- && drbd_rs_should_slow_down(peer_device, sector, false))
- schedule_timeout_uninterruptible(HZ/10);
- update_receiver_timing_details(connection, drbd_rs_begin_io);
- if (drbd_rs_begin_io(device, sector))
- goto out_free_e;
+/* Send an ack packet wih a block ID that is already in big endian byte order. */
+int drbd_send_ack_be(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
+ sector_t sector, int size, u64 block_id)
+{
+ return _drbd_send_ack(peer_device, cmd, cpu_to_be64(sector), cpu_to_be32(size), block_id);
+}
-submit_for_resync:
- atomic_add(size >> 9, &device->rs_sect_ev);
+/**
+ * drbd_send_ack() - Sends an ack packet
+ * @peer_device: DRBD peer device
+ * @cmd: packet command code
+ * @peer_req: peer request
+ */
+int drbd_send_ack(struct drbd_peer_device *peer_device, enum drbd_packet cmd,
+ struct drbd_peer_request *peer_req)
+{
+ return _drbd_send_ack(peer_device, cmd,
+ cpu_to_be64(peer_req->i.sector),
+ cpu_to_be32(peer_req->i.size),
+ peer_req->block_id);
+}
-submit:
- update_receiver_timing_details(connection, drbd_submit_peer_request);
- inc_unacked(device);
- if (drbd_submit_peer_request(peer_req) == 0)
- return 0;
+int drbd_send_ov_result(struct drbd_peer_device *peer_device, sector_t sector, int blksize,
+ u64 block_id, enum ov_result result)
+{
+ struct p_ov_result *p;
- /* don't care for the reason here */
- drbd_err(device, "submit failed, triggering re-connect\n");
+ if (peer_device->connection->agreed_pro_version < 122)
+ /* Misuse the block_id field to signal if the blocks are is sync or not. */
+ return _drbd_send_ack(peer_device, P_OV_RESULT,
+ cpu_to_be64(sector),
+ cpu_to_be32(blksize),
+ cpu_to_be64(drbd_ov_result_to_block_id(result)));
-out_free_e:
- spin_lock_irq(&device->resource->req_lock);
- list_del(&peer_req->w.list);
- spin_unlock_irq(&device->resource->req_lock);
- /* no drbd_rs_complete_io(), we are dropping the connection anyways */
+ if (peer_device->repl_state[NOW] < L_ESTABLISHED)
+ return -EIO;
- put_ldev(device);
- drbd_free_peer_req(device, peer_req);
- return -EIO;
+ p = drbd_prepare_command(peer_device, sizeof(*p), CONTROL_STREAM);
+ if (!p)
+ return -EIO;
+ p->sector = cpu_to_be64(sector);
+ p->block_id = block_id;
+ p->blksize = cpu_to_be32(blksize);
+ p->seq_num = cpu_to_be32(atomic_inc_return(&peer_device->packet_seq));
+ p->result = cpu_to_be32(result);
+ p->pad = 0;
+
+ return drbd_send_command(peer_device, P_OV_RESULT_ID, CONTROL_STREAM);
}
-/*
- * drbd_asb_recover_0p - Recover after split-brain with no remaining primaries
- */
-static int drbd_asb_recover_0p(struct drbd_peer_device *peer_device) __must_hold(local)
+static int receive_RSDataReply(struct drbd_connection *connection, struct packet_info *pi)
{
- struct drbd_device *device = peer_device->device;
- int self, peer, rv = -100;
- unsigned long ch_self, ch_peer;
- enum drbd_after_sb_p after_sb_0p;
+ struct drbd_peer_request_details d;
+ struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+ struct drbd_peer_request *peer_req;
+ int err;
- self = device->ldev->md.uuid[UI_BITMAP] & 1;
- peer = device->p_uuid[UI_BITMAP] & 1;
+ p_req_detail_from_pi(connection, &d, pi);
+ pi->data = NULL;
- ch_peer = device->p_uuid[UI_SIZE];
- ch_self = device->comm_bm_set;
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
+ device = peer_device->device;
- rcu_read_lock();
- after_sb_0p = rcu_dereference(peer_device->connection->net_conf)->after_sb_0p;
- rcu_read_unlock();
- switch (after_sb_0p) {
- case ASB_CONSENSUS:
- case ASB_DISCARD_SECONDARY:
- case ASB_CALL_HELPER:
- case ASB_VIOLENTLY:
- drbd_err(device, "Configuration error.\n");
- break;
- case ASB_DISCONNECT:
- break;
- case ASB_DISCARD_YOUNGER_PRI:
- if (self == 0 && peer == 1) {
- rv = -1;
- break;
- }
- if (self == 1 && peer == 0) {
- rv = 1;
- break;
- }
- fallthrough; /* to one of the other strategies */
- case ASB_DISCARD_OLDER_PRI:
- if (self == 0 && peer == 1) {
- rv = 1;
- break;
- }
- if (self == 1 && peer == 0) {
- rv = -1;
- break;
- }
- /* Else fall through to one of the other strategies... */
- drbd_warn(device, "Discard younger/older primary did not find a decision\n"
- "Using discard-least-changes instead\n");
- fallthrough;
- case ASB_DISCARD_ZERO_CHG:
- if (ch_peer == 0 && ch_self == 0) {
- rv = test_bit(RESOLVE_CONFLICTS, &peer_device->connection->flags)
- ? -1 : 1;
- break;
- } else {
- if (ch_peer == 0) { rv = 1; break; }
- if (ch_self == 0) { rv = -1; break; }
- }
- if (after_sb_0p == ASB_DISCARD_ZERO_CHG)
- break;
- fallthrough;
- case ASB_DISCARD_LEAST_CHG:
- if (ch_self < ch_peer)
- rv = -1;
- else if (ch_self > ch_peer)
- rv = 1;
- else /* ( ch_self == ch_peer ) */
- /* Well, then use something else. */
- rv = test_bit(RESOLVE_CONFLICTS, &peer_device->connection->flags)
- ? -1 : 1;
- break;
- case ASB_DISCARD_LOCAL:
- rv = -1;
- break;
- case ASB_DISCARD_REMOTE:
- rv = 1;
- }
+ peer_req = find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_RESYNC_WRITE),
+ d.sector, d.bi_size, d.block_id);
+ if (!peer_req)
+ return -EIO;
- return rv;
-}
+ if (get_ldev(device)) {
+ err = recv_resync_read(peer_device, peer_req, &d);
+ if (err)
+ put_ldev(device);
+ } else {
+ drbd_err_ratelimit(device, "Cannot write resync data to local disk.\n");
-/*
- * drbd_asb_recover_1p - Recover after split-brain with one remaining primary
- */
-static int drbd_asb_recover_1p(struct drbd_peer_device *peer_device) __must_hold(local)
-{
- struct drbd_device *device = peer_device->device;
- int hg, rv = -100;
- enum drbd_after_sb_p after_sb_1p;
+ err = ignore_remaining_packet(connection, pi->size);
- rcu_read_lock();
- after_sb_1p = rcu_dereference(peer_device->connection->net_conf)->after_sb_1p;
- rcu_read_unlock();
- switch (after_sb_1p) {
- case ASB_DISCARD_YOUNGER_PRI:
- case ASB_DISCARD_OLDER_PRI:
- case ASB_DISCARD_LEAST_CHG:
- case ASB_DISCARD_LOCAL:
- case ASB_DISCARD_REMOTE:
- case ASB_DISCARD_ZERO_CHG:
- drbd_err(device, "Configuration error.\n");
- break;
- case ASB_DISCONNECT:
- break;
- case ASB_CONSENSUS:
- hg = drbd_asb_recover_0p(peer_device);
- if (hg == -1 && device->state.role == R_SECONDARY)
- rv = hg;
- if (hg == 1 && device->state.role == R_PRIMARY)
- rv = hg;
- break;
- case ASB_VIOLENTLY:
- rv = drbd_asb_recover_0p(peer_device);
- break;
- case ASB_DISCARD_SECONDARY:
- return device->state.role == R_PRIMARY ? 1 : -1;
- case ASB_CALL_HELPER:
- hg = drbd_asb_recover_0p(peer_device);
- if (hg == -1 && device->state.role == R_PRIMARY) {
- enum drbd_state_rv rv2;
-
- /* drbd_change_state() does not sleep while in SS_IN_TRANSIENT_STATE,
- * we might be here in C_WF_REPORT_PARAMS which is transient.
- * we do not need to wait for the after state change work either. */
- rv2 = drbd_change_state(device, CS_VERBOSE, NS(role, R_SECONDARY));
- if (rv2 != SS_SUCCESS) {
- drbd_khelper(device, "pri-lost-after-sb");
- } else {
- drbd_warn(device, "Successfully gave up primary role.\n");
- rv = hg;
- }
- } else
- rv = hg;
+ drbd_send_ack_dp(peer_device, P_RS_NEG_ACK, &d);
+
+ dec_rs_pending(peer_device);
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
}
- return rv;
+ rs_sectors_came_in(peer_device, d.bi_size);
+
+ return err;
}
-/*
- * drbd_asb_recover_2p - Recover after split-brain with two remaining primaries
+/* The tail of e_end_block(): send the answer this write earned, then release
+ * the peer request. Also reached from teardown, where no answer goes out.
*/
-static int drbd_asb_recover_2p(struct drbd_peer_device *peer_device) __must_hold(local)
+static int e_end_block_tail(struct drbd_peer_request *peer_req, int cancel)
{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
- int hg, rv = -100;
- enum drbd_after_sb_p after_sb_2p;
+ struct drbd_connection *connection = peer_device->connection;
+ sector_t sector = peer_req->i.sector;
+ int err = 0, pcmd;
- rcu_read_lock();
- after_sb_2p = rcu_dereference(peer_device->connection->net_conf)->after_sb_2p;
- rcu_read_unlock();
- switch (after_sb_2p) {
- case ASB_DISCARD_YOUNGER_PRI:
- case ASB_DISCARD_OLDER_PRI:
- case ASB_DISCARD_LEAST_CHG:
- case ASB_DISCARD_LOCAL:
- case ASB_DISCARD_REMOTE:
- case ASB_CONSENSUS:
- case ASB_DISCARD_SECONDARY:
- case ASB_DISCARD_ZERO_CHG:
- drbd_err(device, "Configuration error.\n");
- break;
- case ASB_VIOLENTLY:
- rv = drbd_asb_recover_0p(peer_device);
- break;
- case ASB_DISCONNECT:
- break;
- case ASB_CALL_HELPER:
- hg = drbd_asb_recover_0p(peer_device);
- if (hg == -1) {
- enum drbd_state_rv rv2;
-
- /* drbd_change_state() does not sleep while in SS_IN_TRANSIENT_STATE,
- * we might be here in C_WF_REPORT_PARAMS which is transient.
- * we do not need to wait for the after state change work either. */
- rv2 = drbd_change_state(device, CS_VERBOSE, NS(role, R_SECONDARY));
- if (rv2 != SS_SUCCESS) {
- drbd_khelper(device, "pri-lost-after-sb");
- } else {
- drbd_warn(device, "Successfully gave up primary role.\n");
- rv = hg;
- }
+ if (peer_req->flags & EE_SEND_WRITE_ACK) {
+ if (unlikely(peer_req->flags & EE_WAS_ERROR)) {
+ pcmd = P_NEG_ACK;
+ /* we expect it to be marked out of sync anyways...
+ * maybe assert this? */
+ } else if (peer_req->flags & EE_POSTPONE) {
+ /* The sync source can not get this write; the writer
+ * counts it as not processed and retries it.
+ */
+ pcmd = P_RETRY_WRITE;
+ } else if (peer_device->repl_state[NOW] >= L_SYNC_SOURCE &&
+ peer_device->repl_state[NOW] <= L_PAUSED_SYNC_T &&
+ peer_req->flags & EE_MAY_SET_IN_SYNC) {
+ pcmd = P_WRITE_ACK_IN_SYNC;
+ drbd_set_in_sync(peer_device, sector, peer_req->i.size);
} else
- rv = hg;
+ pcmd = P_WRITE_ACK;
+ err = drbd_send_ack(peer_device, pcmd, peer_req);
+ dec_unacked(peer_device);
}
- return rv;
-}
+ drbd_remove_peer_req_interval(peer_req);
-static void drbd_uuid_dump(struct drbd_device *device, char *text, u64 *uuid,
- u64 bits, u64 flags)
-{
- if (!uuid) {
- drbd_info(device, "%s uuid info vanished while I was looking!\n", text);
- return;
+ if (connection->agreed_pro_version < 110) {
+ drbd_al_complete_io(device, &peer_req->i);
+ drbd_may_finish_epoch(connection, peer_req->epoch, EV_PUT + (cancel ? EV_CLEANUP : 0));
+ drbd_free_peer_req(peer_req);
+ } else {
+ drbd_peer_req_strip_bio(peer_req);
+ /* The write stays on peer_requests until the peer ack. The
+ * interval is no longer in the tree; the flag only tells
+ * debugfs that this write is done here.
+ */
+ set_bit(INTERVAL_COMPLETED, &peer_req->i.flags);
+ drbd_may_finish_epoch(connection, peer_req->epoch, EV_PUT + (cancel ? EV_CLEANUP : 0));
+ /* Do not use peer_req after this point. We may have sent the
+ * corresponding barrier and received the corresponding peer ack. As a
+ * result, peer_req may have been freed. */
}
- drbd_info(device, "%s %016llX:%016llX:%016llX:%016llX bits:%llu flags:%llX\n",
- text,
- (unsigned long long)uuid[UI_CURRENT],
- (unsigned long long)uuid[UI_BITMAP],
- (unsigned long long)uuid[UI_HISTORY_START],
- (unsigned long long)uuid[UI_HISTORY_END],
- (unsigned long long)bits,
- (unsigned long long)flags);
+
+ return err;
}
-/*
- 100 after split brain try auto recover
- 2 C_SYNC_SOURCE set BitMap
- 1 C_SYNC_SOURCE use BitMap
- 0 no Sync
- -1 C_SYNC_TARGET use BitMap
- -2 C_SYNC_TARGET set BitMap
- -100 after split brain, disconnect
--1000 unrelated data
--1091 requires proto 91
--1096 requires proto 96
+/* Resumed on the sender once the exchange with the sync source is decided.
+ * The epoch arm of e_end_block() already ran before this write was parked.
*/
-
-static int drbd_uuid_compare(struct drbd_peer_device *const peer_device,
- enum drbd_role const peer_role, int *rule_nr) __must_hold(local)
+static int e_end_block_resume(struct drbd_work *w, int cancel)
{
- struct drbd_connection *const connection = peer_device->connection;
- struct drbd_device *device = peer_device->device;
- u64 self, peer;
- int i, j;
-
- self = device->ldev->md.uuid[UI_CURRENT] & ~((u64)1);
- peer = device->p_uuid[UI_CURRENT] & ~((u64)1);
+ struct drbd_peer_request *peer_req =
+ container_of(w, struct drbd_peer_request, w);
- *rule_nr = 10;
- if (self == UUID_JUST_CREATED && peer == UUID_JUST_CREATED)
- return 0;
+ return e_end_block_tail(peer_req, cancel);
+}
- *rule_nr = 20;
- if ((self == UUID_JUST_CREATED || self == (u64)0) &&
- peer != UUID_JUST_CREATED)
- return -2;
+/* The write's full bitmap-granularity range. The resync replies in this unit,
+ * so this is the unit that can roll the write back, and the unit that must
+ * follow the source's content once the write is refused.
+ */
+static void unsecured_write_range(struct drbd_peer_request *peer_req,
+ sector_t *sector, unsigned int *size)
+{
+ struct drbd_bitmap *bm = peer_req->peer_device->device->bitmap;
+ unsigned int sectors = peer_req->i.size >> SECTOR_SHIFT;
+ unsigned long first_bit = bm_sect_to_bit(bm, peer_req->i.sector);
+ unsigned long last_bit = bm_sect_to_bit(bm, peer_req->i.sector + sectors - 1);
- *rule_nr = 30;
- if (self != UUID_JUST_CREATED &&
- (peer == UUID_JUST_CREATED || peer == (u64)0))
- return 2;
+ *sector = bm_bit_to_sect(bm, first_bit);
+ *size = (last_bit - first_bit + 1) * bm_block_size(bm);
+}
- if (self == peer) {
- int rct, dc; /* roles at crash time */
+static void drbd_mark_unsecured_write_out_of_sync(struct drbd_peer_request *peer_req)
+{
+ struct drbd_device *device = peer_req->peer_device->device;
+ struct drbd_peer_device *source;
+ struct drbd_connection *pinned = NULL;
- if (device->p_uuid[UI_BITMAP] == (u64)0 && device->ldev->md.uuid[UI_BITMAP] != (u64)0) {
+ rcu_read_lock();
+ source = peer_device_by_node_id(device, peer_req->wait_source_node_id);
+ if (source) {
+ pinned = source->connection;
+ kref_get(&pinned->kref);
+ }
+ rcu_read_unlock();
- if (connection->agreed_pro_version < 91)
- return -1091;
+ if (!source)
+ return;
- if ((device->ldev->md.uuid[UI_BITMAP] & ~((u64)1)) == (device->p_uuid[UI_HISTORY_START] & ~((u64)1)) &&
- (device->ldev->md.uuid[UI_HISTORY_START] & ~((u64)1)) == (device->p_uuid[UI_HISTORY_START + 1] & ~((u64)1))) {
- drbd_info(device, "was SyncSource, missed the resync finished event, corrected myself:\n");
- drbd_uuid_move_history(device);
- device->ldev->md.uuid[UI_HISTORY_START] = device->ldev->md.uuid[UI_BITMAP];
- device->ldev->md.uuid[UI_BITMAP] = 0;
+ if (get_ldev(device)) {
+ sector_t sector;
+ unsigned int size;
- drbd_uuid_dump(device, "self", device->ldev->md.uuid,
- device->state.disk >= D_NEGOTIATING ? drbd_bm_total_weight(device) : 0, 0);
- *rule_nr = 34;
- } else {
- drbd_info(device, "was SyncSource (peer failed to write sync_uuid)\n");
- *rule_nr = 36;
- }
+ unsecured_write_range(peer_req, §or, &size);
+ drbd_set_out_of_sync(source, sector, size);
+ put_ldev(device);
+ }
+ kref_put(&pinned->kref, drbd_destroy_connection);
+}
- return 1;
- }
+/* Whether drbd_refuse_unsecured_write() can take this write back from its
+ * writer: a protocol C write, still unacknowledged. The writer retries it
+ * (DRBD_FF_WRITE_POSTPONE) or holds it pending across a severed connection;
+ * either way the range may be overwritten. A protocol A or B write is
+ * complete for the writer already.
+ */
+static bool unsecured_write_refusable(struct drbd_peer_request *peer_req)
+{
+ return peer_req->flags & EE_SEND_WRITE_ACK;
+}
- if (device->ldev->md.uuid[UI_BITMAP] == (u64)0 && device->p_uuid[UI_BITMAP] != (u64)0) {
+/* A protocol A or B write in a range the sync source can not get: the writer
+ * already completed it and can not take it back, so it can not be refused. The
+ * source's copy would roll it back, so end the resync towards that source
+ * instead, the way check_resync_source() ends one from a weak source; the
+ * write then survives on this node, unacknowledged (protocol A and B send no
+ * write acknowledgment).
+ */
+static void drbd_end_resync_for_unsecured_write(struct drbd_peer_request *peer_req)
+{
+ struct drbd_device *device = peer_req->peer_device->device;
+ struct drbd_peer_device *source;
+ struct drbd_connection *pinned = NULL;
- if (connection->agreed_pro_version < 91)
- return -1091;
+ rcu_read_lock();
+ source = peer_device_by_node_id(device, peer_req->wait_source_node_id);
+ if (source) {
+ pinned = source->connection;
+ kref_get(&pinned->kref);
+ }
+ rcu_read_unlock();
- if ((device->ldev->md.uuid[UI_HISTORY_START] & ~((u64)1)) == (device->p_uuid[UI_BITMAP] & ~((u64)1)) &&
- (device->ldev->md.uuid[UI_HISTORY_START + 1] & ~((u64)1)) == (device->p_uuid[UI_HISTORY_START] & ~((u64)1))) {
- drbd_info(device, "was SyncTarget, peer missed the resync finished event, corrected peer:\n");
+ if (!source)
+ return;
- device->p_uuid[UI_HISTORY_START + 1] = device->p_uuid[UI_HISTORY_START];
- device->p_uuid[UI_HISTORY_START] = device->p_uuid[UI_BITMAP];
- device->p_uuid[UI_BITMAP] = 0UL;
+ if (repl_is_sync_target(source->repl_state[NOW])) {
+ drbd_info(source, "Holding a write the sync source can not get, ending the resync\n");
+ change_repl_state(source, L_ESTABLISHED, CS_VERBOSE, "unsecured-write");
+ }
+ kref_put(&pinned->kref, drbd_destroy_connection);
+}
- drbd_uuid_dump(device, "peer", device->p_uuid, device->p_uuid[UI_SIZE], device->p_uuid[UI_FLAGS]);
- *rule_nr = 35;
- } else {
- drbd_info(device, "was SyncTarget (failed to write sync_uuid)\n");
- *rule_nr = 37;
- }
+/* Outcome "unreachable" for a write withheld from acknowledgment: the sync
+ * source can not get this write any more, so this node must not let it stand
+ * as it is. The write's full range is marked out of sync toward the source, so
+ * the running resync overwrites every part of it with the source's content.
+ *
+ * A protocol C write that agreed DRBD_FF_WRITE_POSTPONE is answered
+ * P_RETRY_WRITE: the writer retries it once the cluster can order it again.
+ *
+ * A protocol C write that did not agree the feature can not be told to retry,
+ * and it can not be answered either way: a positive ack would confirm a write
+ * this node can not keep, and a negative ack tells the writer this node's disk
+ * failed -- it did not -- and makes the writer distrust it and refuse to hand
+ * over the primary role. The writer is instead an older primary that has lost
+ * every up-to-date peer and suspended, so it is only waiting for this leg.
+ * Sever that leg with a stated reason: the writer holds the write pending and
+ * retransmits it once a peer holds the data again, the way it holds any write
+ * across a lost connection -- honest, and not mistaken for a network fault or
+ * a failed disk. Leaving the write unacknowledged would reach the same
+ * connection loss through the writer's request timeout, only later and blamed
+ * on the network.
+ *
+ * A protocol A or B write can not be answered (its writer already completed
+ * it); it ends the resync and survives on this node instead.
+ */
+static void drbd_refuse_unsecured_write(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *writer = peer_req->peer_device;
- return -1;
+ if (peer_req->flags & EE_SEND_WRITE_ACK) {
+ drbd_mark_unsecured_write_out_of_sync(peer_req);
+ if (writer->connection->agreed_features & DRBD_FF_WRITE_POSTPONE) {
+ peer_req->flags |= EE_POSTPONE;
+ drbd_info_ratelimit(writer,
+ "Sync source can not get this write, postponing it towards the writer\n");
+ } else {
+ drbd_warn(writer,
+ "Sync source can not get this write and the writer can not retry it; disconnecting so it holds the write pending\n");
+ peer_req->flags &= ~EE_SEND_WRITE_ACK;
+ dec_unacked(writer);
+ change_cstate_tag(writer->connection, C_NETWORK_FAILURE,
+ CS_HARD, "unsecured-write", NULL);
}
+ } else {
+ drbd_end_resync_for_unsecured_write(peer_req);
+ }
+ peer_req->flags &= ~(EE_WAIT_FOR_SOURCE | EE_SOURCE_UNREACHABLE);
+ peer_req->w.cb = e_end_block_resume;
+ drbd_queue_work(&writer->connection->sender_work, &peer_req->w);
+}
- /* Common power [off|failure] */
- rct = (test_bit(CRASHED_PRIMARY, &device->flags) ? 1 : 0) +
- (device->p_uuid[UI_FLAGS] & 2);
- /* lowest bit is set when we were primary,
- * next bit (weight 2) is set when peer was primary */
- *rule_nr = 40;
+/* Whether the resync this write was withheld for still runs. Serializes
+ * against drbd_refuse_unsecured_writes() through peer_reqs_lock: hold
+ * it across this test and the parking.
+ */
+static bool wait_source_still_syncing(struct drbd_peer_request *peer_req)
+{
+ struct drbd_device *device = peer_req->peer_device->device;
+ struct drbd_peer_device *source;
+ bool syncing = false;
- /* Neither has the "crashed primary" flag set,
- * only a replication link hickup. */
- if (rct == 0)
- return 0;
+ rcu_read_lock();
+ source = peer_device_by_node_id(device, peer_req->wait_source_node_id);
+ if (source)
+ syncing = repl_is_sync_target(source->repl_state[NOW]);
+ rcu_read_unlock();
- /* Current UUID equal and no bitmap uuid; does not necessarily
- * mean this was a "simultaneous hard crash", maybe IO was
- * frozen, so no UUID-bump happened.
- * This is a protocol change, overload DRBD_FF_WSAME as flag
- * for "new-enough" peer DRBD version. */
- if (device->state.role == R_PRIMARY || peer_role == R_PRIMARY) {
- *rule_nr = 41;
- if (!(connection->agreed_features & DRBD_FF_WSAME)) {
- drbd_warn(peer_device, "Equivalent unrotated UUIDs, but current primary present.\n");
- return -(0x10000 | PRO_VERSION_MAX | (DRBD_FF_WSAME << 8));
- }
- if (device->state.role == R_PRIMARY && peer_role == R_PRIMARY) {
- /* At least one has the "crashed primary" bit set,
- * both are primary now, but neither has rotated its UUIDs?
- * "Can not happen." */
- drbd_err(peer_device, "Equivalent unrotated UUIDs, but both are primary. Can not resolve this.\n");
- return -100;
+ return syncing;
+}
+
+/*
+ * e_end_block() is called in ack_sender context via drbd_finish_peer_reqs().
+ */
+static int e_end_block(struct drbd_work *w, int cancel)
+{
+ struct drbd_peer_request *peer_req =
+ container_of(w, struct drbd_peer_request, w);
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_epoch *epoch;
+
+ if (peer_req->flags & EE_IS_BARRIER) {
+ epoch = previous_epoch(connection, peer_req->epoch);
+ if (epoch)
+ drbd_may_finish_epoch(connection, epoch,
+ EV_BARRIER_DONE + (cancel ? EV_CLEANUP : 0));
+ }
+
+ if (!cancel) {
+ unsigned long answer = 0;
+ bool parked = false;
+
+ /* The sync source may have answered before this write
+ * completed locally (the answer is recorded on the request,
+ * which is findable on peer_requests): act on it now. Only
+ * park while the answer is still out and the resync this
+ * write was withheld for still runs; once that resync is
+ * gone, no answer settles the exchange any more.
+ *
+ * A write that is not parked has its answer decided here,
+ * under the lock drbd_secure_resync_range() takes to arm a
+ * write it still finds undecided.
+ */
+ spin_lock_irq(&connection->peer_reqs_lock);
+ if (peer_req->flags & EE_WAIT_FOR_SOURCE) {
+ answer = peer_req->flags &
+ (EE_SOURCE_REACHED | EE_SOURCE_UNREACHABLE);
+ if (!answer && wait_source_still_syncing(peer_req)) {
+ peer_req->w.cb = e_end_block_resume;
+ list_add_tail(&peer_req->w.list, &connection->source_wait_ee);
+ parked = true;
}
- if (device->state.role == R_PRIMARY)
- return 1;
- return -1;
}
+ if (!parked && !(answer & EE_SOURCE_UNREACHABLE))
+ peer_req->flags |= EE_ACK_DECIDED;
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ if (parked)
+ return 0;
- /* Both are secondary.
- * Really looks like recovery from simultaneous hard crash.
- * Check which had been primary before, and arbitrate. */
- switch (rct) {
- case 0: /* !self_pri && !peer_pri */ return 0; /* already handled */
- case 1: /* self_pri && !peer_pri */ return 1;
- case 2: /* !self_pri && peer_pri */ return -1;
- case 3: /* self_pri && peer_pri */
- dc = test_bit(RESOLVE_CONFLICTS, &connection->flags);
- return dc ? -1 : 1;
+ if (peer_req->flags & EE_WAIT_FOR_SOURCE) {
+ if (answer & EE_SOURCE_REACHED) {
+ peer_req->flags &= ~(EE_WAIT_FOR_SOURCE | EE_SOURCE_REACHED);
+ } else {
+ drbd_refuse_unsecured_write(peer_req);
+ return 0;
+ }
}
}
- *rule_nr = 50;
- peer = device->p_uuid[UI_BITMAP] & ~((u64)1);
- if (self == peer)
- return -1;
-
- *rule_nr = 51;
- peer = device->p_uuid[UI_HISTORY_START] & ~((u64)1);
- if (self == peer) {
- if (connection->agreed_pro_version < 96 ?
- (device->ldev->md.uuid[UI_HISTORY_START] & ~((u64)1)) ==
- (device->p_uuid[UI_HISTORY_START + 1] & ~((u64)1)) :
- peer + UUID_NEW_BM_OFFSET == (device->p_uuid[UI_BITMAP] & ~((u64)1))) {
- /* The last P_SYNC_UUID did not get though. Undo the last start of
- resync as sync source modifications of the peer's UUIDs. */
+ return e_end_block_tail(peer_req, cancel);
+}
- if (connection->agreed_pro_version < 91)
- return -1091;
+static bool seq_greater(u32 a, u32 b)
+{
+ /*
+ * We assume 32-bit wrap-around here.
+ * For 24-bit wrap-around, we would have to shift:
+ * a <<= 8; b <<= 8;
+ */
+ return (s32)a - (s32)b > 0;
+}
- device->p_uuid[UI_BITMAP] = device->p_uuid[UI_HISTORY_START];
- device->p_uuid[UI_HISTORY_START] = device->p_uuid[UI_HISTORY_START + 1];
+static u32 seq_max(u32 a, u32 b)
+{
+ return seq_greater(a, b) ? a : b;
+}
- drbd_info(device, "Lost last syncUUID packet, corrected:\n");
- drbd_uuid_dump(device, "peer", device->p_uuid, device->p_uuid[UI_SIZE], device->p_uuid[UI_FLAGS]);
+static void update_peer_seq(struct drbd_peer_device *peer_device, unsigned int peer_seq)
+{
+ unsigned int newest_peer_seq;
- return -1;
- }
+ if (test_bit(RESOLVE_CONFLICTS, &peer_device->connection->transport.flags)) {
+ spin_lock_bh(&peer_device->peer_seq_lock);
+ newest_peer_seq = seq_max(peer_device->peer_seq, peer_seq);
+ peer_device->peer_seq = newest_peer_seq;
+ spin_unlock_bh(&peer_device->peer_seq_lock);
+ /* wake up only if we actually changed peer_device->peer_seq */
+ if (peer_seq == newest_peer_seq)
+ wake_up(&peer_device->device->seq_wait);
}
+}
- *rule_nr = 60;
- self = device->ldev->md.uuid[UI_CURRENT] & ~((u64)1);
- for (i = UI_HISTORY_START; i <= UI_HISTORY_END; i++) {
- peer = device->p_uuid[i] & ~((u64)1);
- if (self == peer)
- return -2;
- }
+/* Called from receive_Data.
+ * Synchronize packets on sock with packets on msock.
+ *
+ * This is here so even when a P_DATA packet traveling via sock overtook an Ack
+ * packet traveling on msock, they are still processed in the order they have
+ * been sent.
+ *
+ * Note: we don't care for Ack packets overtaking P_DATA packets.
+ *
+ * In case packet_seq is larger than peer_device->peer_seq number, there are
+ * outstanding packets on the msock. We wait for them to arrive.
+ * In case we are the logically next packet, we update peer_device->peer_seq
+ * ourselves. Correctly handles 32bit wrap around.
+ *
+ * Assume we have a 10 GBit connection, that is about 1<<30 byte per second,
+ * about 1<<21 sectors per second. So "worst" case, we have 1<<3 == 8 seconds
+ * for the 24bit wrap (historical atomic_t guarantee on some archs), and we have
+ * 1<<9 == 512 seconds aka ages for the 32bit wrap around...
+ *
+ * returns 0 if we may process the packet,
+ * -ERESTARTSYS if we were interrupted (by disconnect signal). */
+static int wait_for_and_update_peer_seq(struct drbd_peer_device *peer_device, const u32 peer_seq)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ DEFINE_WAIT(wait);
+ long timeout;
+ int ret = 0, tp;
- *rule_nr = 70;
- self = device->ldev->md.uuid[UI_BITMAP] & ~((u64)1);
- peer = device->p_uuid[UI_CURRENT] & ~((u64)1);
- if (self == peer)
- return 1;
+ if (!test_bit(RESOLVE_CONFLICTS, &connection->transport.flags))
+ return 0;
- *rule_nr = 71;
- self = device->ldev->md.uuid[UI_HISTORY_START] & ~((u64)1);
- if (self == peer) {
- if (connection->agreed_pro_version < 96 ?
- (device->ldev->md.uuid[UI_HISTORY_START + 1] & ~((u64)1)) ==
- (device->p_uuid[UI_HISTORY_START] & ~((u64)1)) :
- self + UUID_NEW_BM_OFFSET == (device->ldev->md.uuid[UI_BITMAP] & ~((u64)1))) {
- /* The last P_SYNC_UUID did not get though. Undo the last start of
- resync as sync source modifications of our UUIDs. */
+ spin_lock_bh(&peer_device->peer_seq_lock);
+ for (;;) {
+ if (!seq_greater(peer_seq - 1, peer_device->peer_seq)) {
+ peer_device->peer_seq = seq_max(peer_device->peer_seq, peer_seq);
+ break;
+ }
- if (connection->agreed_pro_version < 91)
- return -1091;
+ if (signal_pending(current)) {
+ ret = -ERESTARTSYS;
+ break;
+ }
- __drbd_uuid_set(device, UI_BITMAP, device->ldev->md.uuid[UI_HISTORY_START]);
- __drbd_uuid_set(device, UI_HISTORY_START, device->ldev->md.uuid[UI_HISTORY_START + 1]);
+ rcu_read_lock();
+ tp = rcu_dereference(connection->transport.net_conf)->two_primaries;
+ rcu_read_unlock();
- drbd_info(device, "Last syncUUID did not get through, corrected:\n");
- drbd_uuid_dump(device, "self", device->ldev->md.uuid,
- device->state.disk >= D_NEGOTIATING ? drbd_bm_total_weight(device) : 0, 0);
+ if (!tp)
+ break;
- return 1;
+ /* Only need to wait if two_primaries is enabled */
+ prepare_to_wait(&peer_device->device->seq_wait, &wait, TASK_INTERRUPTIBLE);
+ spin_unlock_bh(&peer_device->peer_seq_lock);
+ rcu_read_lock();
+ timeout = rcu_dereference(connection->transport.net_conf)->ping_timeo*HZ/10;
+ rcu_read_unlock();
+ timeout = schedule_timeout(timeout);
+ spin_lock_bh(&peer_device->peer_seq_lock);
+ if (!timeout) {
+ ret = -ETIMEDOUT;
+ drbd_err(peer_device, "Timed out waiting for missing ack packets; disconnecting\n");
+ break;
}
}
+ spin_unlock_bh(&peer_device->peer_seq_lock);
+ finish_wait(&peer_device->device->seq_wait, &wait);
+ return ret;
+}
+static enum req_op wire_flags_to_bio_op(u32 dpf)
+{
+ if (dpf & DP_ZEROES)
+ return REQ_OP_WRITE_ZEROES;
+ if (dpf & DP_DISCARD)
+ return REQ_OP_DISCARD;
+ return REQ_OP_WRITE;
+}
- *rule_nr = 80;
- peer = device->p_uuid[UI_CURRENT] & ~((u64)1);
- for (i = UI_HISTORY_START; i <= UI_HISTORY_END; i++) {
- self = device->ldev->md.uuid[i] & ~((u64)1);
- if (self == peer)
- return 2;
- }
-
- *rule_nr = 90;
- self = device->ldev->md.uuid[UI_BITMAP] & ~((u64)1);
- peer = device->p_uuid[UI_BITMAP] & ~((u64)1);
- if (self == peer && self != ((u64)0))
- return 100;
+/* see also bio_flags_to_wire() */
+static blk_opf_t wire_flags_to_bio(struct drbd_connection *connection, u32 dpf)
+{
+ blk_opf_t opf = wire_flags_to_bio_op(dpf) |
+ (dpf & DP_RW_SYNC ? REQ_SYNC : 0);
- *rule_nr = 100;
- for (i = UI_HISTORY_START; i <= UI_HISTORY_END; i++) {
- self = device->ldev->md.uuid[i] & ~((u64)1);
- for (j = UI_HISTORY_START; j <= UI_HISTORY_END; j++) {
- peer = device->p_uuid[j] & ~((u64)1);
- if (self == peer)
- return -100;
- }
- }
+ /* we used to communicate one bit only in older DRBD */
+ if (connection->agreed_pro_version >= 95)
+ opf |= (dpf & DP_FUA ? REQ_FUA : 0) |
+ (dpf & DP_FLUSH ? REQ_PREFLUSH : 0);
- return -1000;
+ return opf;
}
-/* drbd_sync_handshake() returns the new conn state on success, or
- CONN_MASK (-1) on failure.
- */
-static enum drbd_conns drbd_sync_handshake(struct drbd_peer_device *peer_device,
- enum drbd_role peer_role,
- enum drbd_disk_state peer_disk) __must_hold(local)
+static void drbd_wait_for_activity_log_extents(struct drbd_peer_request *peer_req)
{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
struct drbd_device *device = peer_device->device;
- enum drbd_conns rv = C_MASK;
- enum drbd_disk_state mydisk;
- struct net_conf *nc;
- int hg, rule_nr, rr_conflict, tentative, always_asbp;
+ struct lru_cache *al;
+ int nr_al_extents;
+ int nr, used, ecnt;
- mydisk = device->state.disk;
- if (mydisk == D_NEGOTIATING)
- mydisk = device->new_state_tmp.disk;
+ /* Let the activity log know we are about to use it.
+ * See also drbd_request_prepare() for the "request" entry point. */
+ nr_al_extents = interval_to_al_extents(&peer_req->i);
+ ecnt = atomic_add_return(nr_al_extents, &device->wait_for_actlog_ecnt);
- drbd_info(device, "drbd_sync_handshake:\n");
+ spin_lock_irq(&device->al_lock);
+ al = device->act_log;
+ nr = al->nr_elements;
+ used = al->used;
+ spin_unlock_irq(&device->al_lock);
- spin_lock_irq(&device->ldev->md.uuid_lock);
- drbd_uuid_dump(device, "self", device->ldev->md.uuid, device->comm_bm_set, 0);
- drbd_uuid_dump(device, "peer", device->p_uuid,
- device->p_uuid[UI_SIZE], device->p_uuid[UI_FLAGS]);
+ /* note: due to the slight delay between being accounted in "used" after
+ * being committed to the activity log with drbd_al_begin_io_commit(),
+ * and being subtracted from "wait_for_actlog_ecnt" in __drbd_submit_peer_request(),
+ * this can err, but only on the conservative side (overestimating ecnt).
+ * ecnt also includes any requests which are held due to conflicts,
+ * conservatively overestimating the number of activity log extents
+ * required. */
+ if (ecnt > nr - used) {
+ conn_wait_active_ee_empty_or_disconnect(connection);
+ drbd_flush_after_epoch(connection, NULL);
+ conn_wait_done_ee_empty_or_disconnect(connection);
- hg = drbd_uuid_compare(peer_device, peer_role, &rule_nr);
- spin_unlock_irq(&device->ldev->md.uuid_lock);
+ /* would this peer even understand me? */
+ if (connection->agreed_pro_version >= 114)
+ drbd_send_confirm_stable(peer_req);
+ }
+}
- drbd_info(device, "uuid_compare()=%d by rule %d\n", hg, rule_nr);
+static int drbd_peer_write_conflicts(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ sector_t sector = peer_req->i.sector;
+ const unsigned int size = peer_req->i.size;
+ struct drbd_interval *i;
- if (hg == -1000) {
- drbd_alert(device, "Unrelated data, aborting!\n");
- return C_MASK;
- }
- if (hg < -0x10000) {
- int proto, fflags;
- hg = -hg;
- proto = hg & 0xff;
- fflags = (hg >> 8) & 0xff;
- drbd_alert(device, "To resolve this both sides have to support at least protocol %d and feature flags 0x%x\n",
- proto, fflags);
- return C_MASK;
- }
- if (hg < -1000) {
- drbd_alert(device, "To resolve this both sides have to support at least protocol %d\n", -hg - 1000);
- return C_MASK;
- }
+ i = drbd_find_conflict(device, &peer_req->i, CONFLICT_FLAG_APPLICATION_ONLY);
- if ((mydisk == D_INCONSISTENT && peer_disk > D_INCONSISTENT) ||
- (peer_disk == D_INCONSISTENT && mydisk > D_INCONSISTENT)) {
- int f = (hg == -100) || abs(hg) == 2;
- hg = mydisk > D_INCONSISTENT ? 1 : -1;
- if (f)
- hg = hg*2;
- drbd_info(device, "Becoming sync %s due to disk states.\n",
- hg > 0 ? "source" : "target");
+ if (i) {
+ drbd_alert(device, "Concurrent writes detected: "
+ "local=%llus +%u, remote=%llus +%u\n",
+ (unsigned long long) i->sector, i->size,
+ (unsigned long long) sector, size);
+ return -EBUSY;
}
- if (abs(hg) == 100)
- drbd_khelper(device, "initial-split-brain");
+ return 0;
+}
- rcu_read_lock();
- nc = rcu_dereference(peer_device->connection->net_conf);
- always_asbp = nc->always_asbp;
- rr_conflict = nc->rr_conflict;
- tentative = nc->tentative;
- rcu_read_unlock();
+static void drbd_queue_peer_request(struct drbd_device *device, struct drbd_peer_request *peer_req)
+{
+ atomic_inc(&device->wait_for_actlog);
+ spin_lock(&device->submit.lock);
+ list_add_tail(&peer_req->w.list, &device->submit.peer_writes);
+ spin_unlock(&device->submit.lock);
+ queue_work(device->submit.wq, &device->submit.worker);
+ /* do_submit() may sleep internally on al_wait, too */
+ wake_up(&device->al_wait);
+}
- if (hg == 100 || (hg == -100 && always_asbp)) {
- int pcount = (device->state.role == R_PRIMARY)
- + (peer_role == R_PRIMARY);
- int forced = (hg == -100);
+static struct drbd_peer_request *find_released_peer_request(struct drbd_resource *resource, unsigned int node_id, u64 dagtag)
+{
+ struct drbd_connection *connection;
+ struct drbd_peer_request *released_peer_req = NULL;
- switch (pcount) {
- case 0:
- hg = drbd_asb_recover_0p(peer_device);
- break;
- case 1:
- hg = drbd_asb_recover_1p(peer_device);
- break;
- case 2:
- hg = drbd_asb_recover_2p(peer_device);
+ read_lock_irq(&resource->state_rwlock);
+ for_each_connection(connection, resource) {
+ struct drbd_peer_request *peer_req;
+
+ /* Skip if we are not connected. If we are in the process of
+ * disconnecting, the requests on dagtag_wait_ee will be
+ * cleared up. Do not interfere with that process. */
+ if (connection->cstate[NOW] < C_CONNECTED)
+ continue;
+
+ spin_lock(&connection->peer_reqs_lock);
+ list_for_each_entry(peer_req, &connection->dagtag_wait_ee, w.list) {
+ if (!peer_req->depend_dagtag ||
+ peer_req->depend_dagtag_node_id != node_id ||
+ peer_req->depend_dagtag > dagtag)
+ continue;
+
+ dynamic_drbd_dbg(peer_req->peer_device, "%s at %llus+%u: Wait for dagtag %llus from peer %u complete\n",
+ drbd_interval_type_str(&peer_req->i),
+ (unsigned long long) peer_req->i.sector, peer_req->i.size,
+ (unsigned long long) peer_req->depend_dagtag,
+ peer_req->depend_dagtag_node_id);
+
+ list_del(&peer_req->w.list);
+ released_peer_req = peer_req;
break;
}
- if (abs(hg) < 100) {
- drbd_warn(device, "Split-Brain detected, %d primaries, "
- "automatically solved. Sync from %s node\n",
- pcount, (hg < 0) ? "peer" : "this");
- if (forced) {
- drbd_warn(device, "Doing a full sync, since"
- " UUIDs where ambiguous.\n");
- hg = hg*2;
- }
- }
+ spin_unlock(&connection->peer_reqs_lock);
+
+ if (released_peer_req)
+ break;
}
+ read_unlock_irq(&resource->state_rwlock);
- if (hg == -100) {
- if (test_bit(DISCARD_MY_DATA, &device->flags) && !(device->p_uuid[UI_FLAGS]&1))
- hg = -1;
- if (!test_bit(DISCARD_MY_DATA, &device->flags) && (device->p_uuid[UI_FLAGS]&1))
- hg = 1;
+ return released_peer_req;
+}
- if (abs(hg) < 100)
- drbd_warn(device, "Split-Brain detected, manually solved. "
- "Sync from %s node\n",
- (hg < 0) ? "peer" : "this");
- }
+static void release_dagtag_wait(struct drbd_resource *resource, unsigned int node_id, u64 dagtag)
+{
+ struct drbd_peer_request *peer_req;
- if (hg == -100) {
- /* FIXME this log message is not correct if we end up here
- * after an attempted attach on a diskless node.
- * We just refuse to attach -- well, we drop the "connection"
- * to that disk, in a way... */
- drbd_alert(device, "Split-Brain detected but unresolved, dropping connection!\n");
- drbd_khelper(device, "split-brain");
- return C_MASK;
+ while ((peer_req = find_released_peer_request(resource, node_id, dagtag))) {
+ atomic_inc(&peer_req->peer_device->connection->backing_ee_cnt);
+ drbd_conflict_submit_peer_read(peer_req);
}
+}
- if (hg > 0 && mydisk <= D_INCONSISTENT) {
- drbd_err(device, "I shall become SyncSource, but I am inconsistent!\n");
- return C_MASK;
- }
+/* A dagtag wait request of a sync target, parked until this connection's
+ * write stream reaches the position the request names. Waits live on the
+ * connection that feeds the stream: its dagtag advance and its teardown answer
+ * them; the requester's own teardown discards its waits unanswered.
+ */
+struct drbd_dagtag_wait_req {
+ struct list_head list;
+ struct drbd_peer_device *peer_device;
+ sector_t sector;
+ unsigned int size;
+ u64 block_id;
+ u64 dagtag;
+};
- if (hg < 0 && /* by intention we do not use mydisk here. */
- device->state.role == R_PRIMARY && device->state.disk >= D_CONSISTENT) {
- switch (rr_conflict) {
- case ASB_CALL_HELPER:
- drbd_khelper(device, "pri-lost");
- fallthrough;
- case ASB_DISCONNECT:
- drbd_err(device, "I shall become SyncTarget, but I am primary!\n");
- return C_MASK;
- case ASB_VIOLENTLY:
- drbd_warn(device, "Becoming SyncTarget, violating the stable-data"
- "assumption\n");
- }
- }
+static int queue_dagtag_wait_req(struct drbd_peer_device *peer_device, sector_t sector,
+ unsigned int size, u64 block_id, unsigned int node_id,
+ u64 dagtag)
+{
+ struct drbd_resource *resource = peer_device->device->resource;
+ struct drbd_connection *connection;
+ struct drbd_dagtag_wait_req *wait;
+ bool queued = false;
- if (tentative || test_bit(CONN_DRY_RUN, &peer_device->connection->flags)) {
- if (hg == 0)
- drbd_info(device, "dry-run connect: No resync, would become Connected immediately.\n");
- else
- drbd_info(device, "dry-run connect: Would become %s, doing a %s resync.",
- drbd_conn_str(hg > 0 ? C_SYNC_SOURCE : C_SYNC_TARGET),
- abs(hg) >= 2 ? "full" : "bit-map based");
- return C_MASK;
+ wait = kmalloc_obj(*wait, GFP_NOIO);
+ if (!wait)
+ return -ENOMEM;
+
+ wait->peer_device = peer_device;
+ wait->sector = sector;
+ wait->size = size;
+ wait->block_id = block_id;
+ wait->dagtag = dagtag;
+
+ rcu_read_lock();
+ connection = drbd_connection_by_node_id(resource, node_id);
+ if (connection) {
+ /* answer_dagtag_wait_reqs() empties this list under this lock
+ * once the stream is gone, and the teardown marks the stream
+ * before it does. Take the decision again here, so a wait
+ * either lands before that walk or is never queued at all.
+ */
+ spin_lock_irq(&connection->peer_reqs_lock);
+ queued = !test_bit(DAGTAG_STREAM_GONE, &connection->flags);
+ if (queued)
+ list_add_tail(&wait->list, &connection->dagtag_wait_reqs);
+ spin_unlock_irq(&connection->peer_reqs_lock);
}
+ rcu_read_unlock();
- if (abs(hg) >= 2) {
- drbd_info(device, "Writing the whole bitmap, full sync required after drbd_sync_handshake.\n");
- if (drbd_bitmap_io(device, &drbd_bmio_set_n_write, "set_n_write from sync_handshake",
- BM_LOCKED_SET_ALLOWED, NULL))
- return C_MASK;
+ if (!queued) {
+ /* The stream went away between the reach test and here. */
+ kfree(wait);
+ return drbd_send_ack_be(peer_device, P_RS_DAGTAG_UNREACHABLE,
+ sector, size, block_id);
}
- if (hg > 0) { /* become sync source. */
- rv = C_WF_BITMAP_S;
- } else if (hg < 0) { /* become sync target */
- rv = C_WF_BITMAP_T;
- } else {
- rv = C_CONNECTED;
- if (drbd_bm_total_weight(device)) {
- drbd_info(device, "No resync, but %lu bits in bitmap!\n",
- drbd_bm_total_weight(device));
+ return 0;
+}
+
+/* Answer every dagtag wait request this connection's stream has now reached,
+ * and, when the stream itself is gone, every request that is left.
+ */
+static void answer_dagtag_wait_reqs(struct drbd_connection *connection, u64 dagtag, bool gone)
+{
+ struct drbd_dagtag_wait_req *wait, *t;
+ LIST_HEAD(work_list);
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry_safe(wait, t, &connection->dagtag_wait_reqs, list) {
+ if (gone || wait->dagtag <= dagtag) {
+ /* Pin the requester across the send below; only list
+ * membership keeps it referenced otherwise.
+ */
+ kref_get(&wait->peer_device->connection->kref);
+ list_move_tail(&wait->list, &work_list);
}
}
+ spin_unlock_irq(&connection->peer_reqs_lock);
- return rv;
+ list_for_each_entry_safe(wait, t, &work_list, list) {
+ struct drbd_connection *requester = wait->peer_device->connection;
+
+ drbd_send_ack_be(wait->peer_device,
+ gone ? P_RS_DAGTAG_UNREACHABLE : P_RS_DAGTAG_REACHED,
+ wait->sector, wait->size, wait->block_id);
+ kfree(wait);
+ kref_put(&requester->kref, drbd_destroy_connection);
+ }
}
-static enum drbd_after_sb_p convert_after_sb(enum drbd_after_sb_p peer)
+/* The requester of these waits is going away; discard them from every
+ * stream's list before a del-peer could free the peer_device they name.
+ */
+static void drop_dagtag_wait_reqs_of(struct drbd_connection *requester)
{
- /* ASB_DISCARD_REMOTE - ASB_DISCARD_LOCAL is valid */
- if (peer == ASB_DISCARD_REMOTE)
- return ASB_DISCARD_LOCAL;
+ struct drbd_resource *resource = requester->resource;
+ struct drbd_connection *connection;
+ struct drbd_dagtag_wait_req *wait, *t;
+ LIST_HEAD(work_list);
- /* any other things with ASB_DISCARD_REMOTE or ASB_DISCARD_LOCAL are invalid */
- if (peer == ASB_DISCARD_LOCAL)
- return ASB_DISCARD_REMOTE;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry_safe(wait, t, &connection->dagtag_wait_reqs, list) {
+ if (wait->peer_device->connection == requester)
+ list_move_tail(&wait->list, &work_list);
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ }
+ rcu_read_unlock();
- /* everything else is valid if they are equal on both sides. */
- return peer;
+ list_for_each_entry_safe(wait, t, &work_list, list)
+ kfree(wait);
}
-static int receive_protocol(struct drbd_connection *connection, struct packet_info *pi)
+/* This node stopped being the sync source of this peer, so no stream
+ * position it still reaches means anything to the writes that peer withholds.
+ * Answer every wait request of that peer unreachable: a peer still waiting
+ * refuses the writes, a peer that left sync target ignores the answer.
+ */
+void drbd_dagtag_wait_reqs_source_gone(struct drbd_peer_device *requester)
{
- struct p_protocol *p = pi->data;
- enum drbd_after_sb_p p_after_sb_0p, p_after_sb_1p, p_after_sb_2p;
- int p_proto, p_discard_my_data, p_two_primaries, cf;
- struct net_conf *nc, *old_net_conf, *new_net_conf = NULL;
- char integrity_alg[SHARED_SECRET_MAX] = "";
- struct crypto_shash *peer_integrity_tfm = NULL;
- void *int_dig_in = NULL, *int_dig_vv = NULL;
-
- p_proto = be32_to_cpu(p->protocol);
- p_after_sb_0p = be32_to_cpu(p->after_sb_0p);
- p_after_sb_1p = be32_to_cpu(p->after_sb_1p);
- p_after_sb_2p = be32_to_cpu(p->after_sb_2p);
- p_two_primaries = be32_to_cpu(p->two_primaries);
- cf = be32_to_cpu(p->conn_flags);
- p_discard_my_data = cf & CF_DISCARD_MY_DATA;
-
- if (connection->agreed_pro_version >= 87) {
- int err;
-
- if (pi->size > sizeof(integrity_alg))
- return -EIO;
- err = drbd_recv_all(connection, integrity_alg, pi->size);
- if (err)
- return err;
- integrity_alg[SHARED_SECRET_MAX - 1] = 0;
- }
-
- if (pi->cmd != P_PROTOCOL_UPDATE) {
- clear_bit(CONN_DRY_RUN, &connection->flags);
-
- if (cf & CF_DRY_RUN)
- set_bit(CONN_DRY_RUN, &connection->flags);
-
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
-
- if (p_proto != nc->wire_protocol) {
- drbd_err(connection, "incompatible %s settings\n", "protocol");
- goto disconnect_rcu_unlock;
- }
-
- if (convert_after_sb(p_after_sb_0p) != nc->after_sb_0p) {
- drbd_err(connection, "incompatible %s settings\n", "after-sb-0pri");
- goto disconnect_rcu_unlock;
- }
-
- if (convert_after_sb(p_after_sb_1p) != nc->after_sb_1p) {
- drbd_err(connection, "incompatible %s settings\n", "after-sb-1pri");
- goto disconnect_rcu_unlock;
- }
+ struct drbd_resource *resource = requester->device->resource;
+ struct drbd_connection *connection;
+ struct drbd_dagtag_wait_req *wait, *t;
+ LIST_HEAD(work_list);
- if (convert_after_sb(p_after_sb_2p) != nc->after_sb_2p) {
- drbd_err(connection, "incompatible %s settings\n", "after-sb-2pri");
- goto disconnect_rcu_unlock;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry_safe(wait, t, &connection->dagtag_wait_reqs, list) {
+ if (wait->peer_device == requester)
+ list_move_tail(&wait->list, &work_list);
}
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ }
+ rcu_read_unlock();
- if (p_discard_my_data && nc->discard_my_data) {
- drbd_err(connection, "incompatible %s settings\n", "discard-my-data");
- goto disconnect_rcu_unlock;
- }
+ list_for_each_entry_safe(wait, t, &work_list, list) {
+ drbd_send_ack_be(requester, P_RS_DAGTAG_UNREACHABLE,
+ wait->sector, wait->size, wait->block_id);
+ kfree(wait);
+ }
+}
- if (p_two_primaries != nc->two_primaries) {
- drbd_err(connection, "incompatible %s settings\n", "allow-two-primaries");
- goto disconnect_rcu_unlock;
- }
+/* The exchange toward this sync source can no longer complete: the resync
+ * ended or was aborted, the source detached, or the connection to it is
+ * gone. Resolve every write of this device still withheld from
+ * acknowledgment as "unreachable": mark and refuse. A refusal is always
+ * safe; the cost is a retry. Runs from the after-state-change work when the
+ * replication state leaves sync target.
+ */
+void drbd_refuse_unsecured_writes(struct drbd_peer_device *source)
+{
+ struct drbd_device *device = source->device;
+ struct drbd_resource *resource = device->resource;
+ struct drbd_connection *connection;
+ struct drbd_peer_request *peer_req, *t;
+ LIST_HEAD(work_list);
- if (strcmp(integrity_alg, nc->integrity_alg)) {
- drbd_err(connection, "incompatible %s settings\n", "data-integrity-alg");
- goto disconnect_rcu_unlock;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry_safe(peer_req, t, &connection->source_wait_ee, w.list) {
+ if (peer_req->peer_device->device == device)
+ list_move_tail(&peer_req->w.list, &work_list);
}
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ }
+ rcu_read_unlock();
- rcu_read_unlock();
+ list_for_each_entry_safe(peer_req, t, &work_list, w.list) {
+ list_del_init(&peer_req->w.list);
+ drbd_refuse_unsecured_write(peer_req);
}
+}
- if (integrity_alg[0]) {
- int hash_size;
+static void set_connection_dagtag(struct drbd_connection *connection, u64 dagtag)
+{
+ atomic64_set(&connection->last_dagtag_sector, dagtag);
+ set_bit(RECEIVED_DAGTAG, &connection->flags);
- /*
- * We can only change the peer data integrity algorithm
- * here. Changing our own data integrity algorithm
- * requires that we send a P_PROTOCOL_UPDATE packet at
- * the same time; otherwise, the peer has no way to
- * tell between which packets the algorithm should
- * change.
- */
+ release_dagtag_wait(connection->resource, connection->peer_node_id, dagtag);
+ answer_dagtag_wait_reqs(connection, dagtag, false);
+}
- peer_integrity_tfm = crypto_alloc_shash(integrity_alg, 0, 0);
- if (IS_ERR(peer_integrity_tfm)) {
- peer_integrity_tfm = NULL;
- drbd_err(connection, "peer data-integrity-alg %s not supported\n",
- integrity_alg);
- goto disconnect;
- }
+static void submit_peer_request_activity_log(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ int err;
+ int nr_al_extents = interval_to_al_extents(&peer_req->i);
- hash_size = crypto_shash_digestsize(peer_integrity_tfm);
- int_dig_in = kmalloc(hash_size, GFP_KERNEL);
- int_dig_vv = kmalloc(hash_size, GFP_KERNEL);
- if (!(int_dig_in && int_dig_vv)) {
- drbd_err(connection, "Allocation of buffers for data integrity checking failed\n");
- goto disconnect;
- }
+ if (nr_al_extents != 1 || !drbd_al_begin_io_fastpath(device, &peer_req->i)) {
+ drbd_queue_peer_request(device, peer_req);
+ return;
}
- new_net_conf = kmalloc_obj(struct net_conf);
- if (!new_net_conf)
- goto disconnect;
+ peer_req->flags |= EE_IN_ACTLOG;
+ atomic_sub(nr_al_extents, &device->wait_for_actlog_ecnt);
- mutex_lock(&connection->data.mutex);
- mutex_lock(&connection->resource->conf_update);
- old_net_conf = connection->net_conf;
- *new_net_conf = *old_net_conf;
+ err = drbd_submit_peer_request(peer_req);
+ if (err)
+ drbd_cleanup_after_failed_submit_peer_write(peer_req);
+}
- new_net_conf->wire_protocol = p_proto;
- new_net_conf->after_sb_0p = convert_after_sb(p_after_sb_0p);
- new_net_conf->after_sb_1p = convert_after_sb(p_after_sb_1p);
- new_net_conf->after_sb_2p = convert_after_sb(p_after_sb_2p);
- new_net_conf->two_primaries = p_two_primaries;
+/* Give up on a peer write that was held waiting for conflicts to resolve.
+ * Mirrors the cleanup done in receive_Data()'s out_del_list path, but also
+ * removes the interval (which was already inserted into the tree) and wakes
+ * conn_disconnect(), which may be waiting for active_ee_cnt to reach zero.
+ */
+static void cleanup_canceled_peer_write(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
- rcu_assign_pointer(connection->net_conf, new_net_conf);
- mutex_unlock(&connection->resource->conf_update);
- mutex_unlock(&connection->data.mutex);
+ drbd_remove_peer_req_interval(peer_req);
- crypto_free_shash(connection->peer_integrity_tfm);
- kfree(connection->int_dig_in);
- kfree(connection->int_dig_vv);
- connection->peer_integrity_tfm = peer_integrity_tfm;
- connection->int_dig_in = int_dig_in;
- connection->int_dig_vv = int_dig_vv;
+ spin_lock_irq(&connection->peer_reqs_lock);
+ peer_req->flags &= ~EE_ON_RECV_ORDER;
+ list_del(&peer_req->recv_order);
+ spin_unlock_irq(&connection->peer_reqs_lock);
- if (strcmp(old_net_conf->integrity_alg, integrity_alg))
- drbd_info(connection, "peer data-integrity-alg: %s\n",
- integrity_alg[0] ? integrity_alg : "(none)");
+ if (peer_req->flags & EE_IN_ACTLOG)
+ drbd_al_complete_io(device, &peer_req->i);
+ else
+ atomic_sub(interval_to_al_extents(&peer_req->i), &device->wait_for_actlog_ecnt);
- kvfree_rcu_mightsleep(old_net_conf);
- return 0;
+ if (peer_req->flags & EE_SEND_WRITE_ACK)
+ dec_unacked(peer_device);
+ drbd_may_finish_epoch(connection, peer_req->epoch, EV_PUT + EV_CLEANUP);
+ put_ldev(device);
+ drbd_free_peer_req(peer_req);
-disconnect_rcu_unlock:
- rcu_read_unlock();
-disconnect:
- crypto_free_shash(peer_integrity_tfm);
- kfree(int_dig_in);
- kfree(int_dig_vv);
- conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_HARD);
- return -EIO;
+ if (atomic_dec_and_test(&connection->active_ee_cnt))
+ wake_up(&connection->ee_wait);
}
-/* helper function
- * input: alg name, feature name
- * return: NULL (alg name was "")
- * ERR_PTR(error) if something goes wrong
- * or the crypto hash ptr, if it worked out ok. */
-static struct crypto_shash *drbd_crypto_alloc_digest_safe(
- const struct drbd_device *device,
- const char *alg, const char *name)
+void drbd_conflict_submit_peer_write(struct drbd_peer_request *peer_req)
{
- struct crypto_shash *tfm;
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ bool conflict = false;
+ bool canceled;
+
+ spin_lock_irq(&device->interval_lock);
+ clear_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &peer_req->i.flags);
+ canceled = test_bit(INTERVAL_CANCELED, &peer_req->i.flags);
+ conflict = drbd_find_conflict(device, &peer_req->i, 0);
+ if (conflict)
+ drbd_rs_depth_exempt(device, &peer_req->i);
+ else if (!canceled)
+ set_bit(INTERVAL_SUBMITTED, &peer_req->i.flags);
+ spin_unlock_irq(&device->interval_lock);
+
+ if (canceled)
+ cleanup_canceled_peer_write(peer_req);
+ else if (!conflict)
+ submit_peer_request_activity_log(peer_req);
+}
+
+/* mirrored write
+ *
+ * Request handling flow:
+ *
+ * conflict
+ * receive_Data -----------------------+
+ * | |
+ * | ...
+ * | |
+ * | v
+ * | drbd_do_submit_conflict
+ * | |
+ * | v
+ * +------------------ drbd_conflict_submit_peer_write
+ * |
+ * v wait for AL
+ * submit_peer_request_activity_log --------> drbd_queue_peer_request
+ * | |
+ * | ...
+ * | |
+ * | v AL extent active
+ * | do_submit ----------------+
+ * | | |
+ * | v v
+ * | send_and_submit_pending submit_fast_path
+ * | | |
+ * v v |
+ * drbd_submit_peer_request <-------- __drbd_submit_peer_request <------+
+ * |
+ * ... backing device
+ * |
+ * v
+ * drbd_peer_request_endio
+ * |
+ * v
+ * drbd_endio_write_sec_final
+ * |
+ * ... done_ee
+ * |
+ * v
+ * drbd_finish_peer_reqs
+ * |
+ * v
+ * e_end_block
+ * |
+ * ... via peer
+ * |
+ * v
+ * got_peer_ack
+ */
- if (!alg[0])
- return NULL;
+/* The state handling permits one incoming resync at a time. Return the peer
+ * device that runs it, with a reference on its connection, or NULL.
+ */
+static struct drbd_peer_device *get_sync_source_ref(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
- tfm = crypto_alloc_shash(alg, 0, 0);
- if (IS_ERR(tfm)) {
- drbd_err(device, "Can not allocate \"%s\" as %s (reason: %ld)\n",
- alg, name, PTR_ERR(tfm));
- return tfm;
+ read_lock_irq(&resource->state_rwlock);
+ for_each_peer_device(peer_device, device) {
+ if (!repl_is_sync_target(peer_device->repl_state[NOW]))
+ continue;
+ kref_get(&peer_device->connection->kref);
+ read_unlock_irq(&resource->state_rwlock);
+ return peer_device;
}
- return tfm;
+ read_unlock_irq(&resource->state_rwlock);
+
+ return NULL;
}
-static int ignore_remaining_packet(struct drbd_connection *connection, struct packet_info *pi)
+/* The wait request travels on the sync source's data stream. Sent from the
+ * writer's receiver, a full send buffer toward the source would stall that
+ * receiver; the source connection's sender sends it instead.
+ */
+struct dagtag_wait_send {
+ struct drbd_work w;
+ struct drbd_peer_device *sync_source;
+ sector_t sector;
+ unsigned int size;
+ u64 block_id;
+ unsigned int node_id;
+ u64 dagtag;
+};
+
+static int w_send_dagtag_wait_req(struct drbd_work *w, int cancel)
{
- void *buffer = connection->data.rbuf;
- int size = pi->size;
+ struct dagtag_wait_send *send = container_of(w, struct dagtag_wait_send, w);
+ struct drbd_connection *connection = send->sync_source->connection;
- while (size) {
- int s = min_t(int, size, DRBD_SOCKET_BUFFER_SIZE);
- s = drbd_recv(connection, buffer, s);
- if (s <= 0) {
- if (s < 0)
- return s;
- break;
- }
- size -= s;
- }
- if (size)
- return -EIO;
+ if (!cancel)
+ drbd_send_rs_request(send->sync_source, P_RS_DAGTAG_WAIT_REQ,
+ send->sector, send->size, send->block_id,
+ send->node_id, send->dagtag);
+ kref_put(&connection->kref, drbd_destroy_connection);
+ kfree(send);
return 0;
}
-/*
- * config_unknown_volume - device configuration command for unknown volume
- *
- * When a device is added to an existing connection, the node on which the
- * device is added first will send configuration commands to its peer but the
- * peer will not know about the device yet. It will warn and ignore these
- * commands. Once the device is added on the second node, the second node will
- * send the same device configuration commands, but in the other direction.
+/* A write that lands on a sync target in a range the sync source is about to
+ * overwrite must not be acknowledged before the sync source has it too.
+ * Otherwise the resync reply rolls it back, and the repair resync in the other
+ * direction carries the rolled back block to the source as well: the
+ * acknowledged write is gone everywhere and no bitmap records it.
*
- * (We can also end up here if drbd is misconfigured.)
+ * Ask the sync source to report when its own copy of the writer's stream
+ * reaches the position this write has in it: P_RS_DAGTAG_REACHED once it does,
+ * P_RS_DAGTAG_UNREACHABLE when it can not get there by itself. This only arms
+ * the write and sends the request. The write is submitted as usual; it is
+ * e_end_block() that withholds the acknowledgment until the answer arrives.
*/
-static int config_unknown_volume(struct drbd_connection *connection, struct packet_info *pi)
+static void drbd_withhold_ack_until_source_has_write(struct drbd_peer_request *peer_req)
{
- drbd_warn(connection, "%s packet received for volume %u, which is not configured locally\n",
- drbd_packet_name(pi->cmd), pi->vnr);
- return ignore_remaining_packet(connection, pi);
+ struct drbd_peer_device *writer = peer_req->peer_device;
+ struct drbd_device *device = writer->device;
+ struct drbd_peer_device *sync_source;
+ unsigned long first_bit, last_bit;
+ sector_t block_sector;
+ unsigned int block_size;
+
+ if (peer_req->i.size == 0)
+ return;
+
+ sync_source = get_sync_source_ref(device);
+ if (!sync_source)
+ return;
+
+ /* The sync source has its own writes. */
+ if (sync_source == writer)
+ goto out;
+
+ if (!(sync_source->connection->agreed_features & DRBD_FF_WRITE_POSTPONE))
+ goto out;
+
+ if (!get_ldev(device))
+ goto out;
+
+ unsecured_write_range(peer_req, &block_sector, &block_size);
+ first_bit = bm_sect_to_bit(device->bitmap, block_sector);
+ last_bit = first_bit + block_size / bm_block_size(device->bitmap) - 1;
+
+ if (drbd_bm_count_bits(device, sync_source->bitmap_index, first_bit, last_bit)) {
+ struct dagtag_wait_send *send;
+
+ peer_req->flags |= EE_WAIT_FOR_SOURCE;
+ peer_req->wait_source_node_id = sync_source->node_id;
+
+ send = kzalloc_obj(*send, GFP_NOIO);
+ if (!send) {
+ /* No request goes out; have e_end_block() refuse the write. */
+ peer_req->flags |= EE_SOURCE_UNREACHABLE;
+ goto out_ldev;
+ }
+ send->w.cb = w_send_dagtag_wait_req;
+ send->sync_source = sync_source;
+ send->sector = block_sector;
+ send->size = block_size;
+ send->block_id = (unsigned long)peer_req;
+ send->node_id = writer->connection->peer_node_id;
+ send->dagtag = peer_req->dagtag_sector;
+ put_ldev(device);
+ /* The connection reference travels with the work item. */
+ drbd_queue_work(&sync_source->connection->sender_work, &send->w);
+ return;
+ }
+
+out_ldev:
+ put_ldev(device);
+out:
+ kref_put(&sync_source->connection->kref, drbd_destroy_connection);
}
-static int receive_SyncParam(struct drbd_connection *connection, struct packet_info *pi)
+static int receive_Data(struct drbd_connection *connection, struct packet_info *pi)
{
struct drbd_peer_device *peer_device;
struct drbd_device *device;
- struct p_rs_param_95 *p;
- unsigned int header_size, data_size, exp_max_sz;
- struct crypto_shash *verify_tfm = NULL;
- struct crypto_shash *csums_tfm = NULL;
- struct net_conf *old_net_conf, *new_net_conf = NULL;
- struct disk_conf *old_disk_conf = NULL, *new_disk_conf = NULL;
- const int apv = connection->agreed_pro_version;
- struct fifo_buffer *old_plan = NULL, *new_plan = NULL;
- unsigned int fifo_size = 0;
- int err;
+ struct drbd_net_conf *nc;
+ struct drbd_peer_request *peer_req;
+ struct drbd_peer_request_details d;
+ int err, tp;
+ bool conflict = false;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
- return config_unknown_volume(connection, pi);
+ return -EIO;
device = peer_device->device;
- exp_max_sz = apv <= 87 ? sizeof(struct p_rs_param)
- : apv == 88 ? sizeof(struct p_rs_param)
- + SHARED_SECRET_MAX
- : apv <= 94 ? sizeof(struct p_rs_param_89)
- : /* apv >= 95 */ sizeof(struct p_rs_param_95);
-
- if (pi->size > exp_max_sz) {
- drbd_err(device, "SyncParam packet too long: received %u, expected <= %u bytes\n",
- pi->size, exp_max_sz);
- return -EIO;
- }
+ if (pi->cmd == P_TRIM)
+ D_ASSERT(peer_device, pi->size == 0);
- if (apv <= 88) {
- header_size = sizeof(struct p_rs_param);
- data_size = pi->size - header_size;
- } else if (apv <= 94) {
- header_size = sizeof(struct p_rs_param_89);
- data_size = pi->size - header_size;
- D_ASSERT(device, data_size == 0);
- } else {
- header_size = sizeof(struct p_rs_param_95);
- data_size = pi->size - header_size;
- D_ASSERT(device, data_size == 0);
- }
+ p_req_detail_from_pi(connection, &d, pi);
+ pi->data = NULL;
- /* initialize verify_alg and csums_alg */
- p = pi->data;
- BUILD_BUG_ON(sizeof(p->algs) != 2 * SHARED_SECRET_MAX);
- memset(&p->algs, 0, sizeof(p->algs));
+ if (!get_ldev(device)) {
+ int err2;
- err = drbd_recv_all(peer_device->connection, p, header_size);
- if (err)
+ err = wait_for_and_update_peer_seq(peer_device, d.peer_seq);
+ drbd_send_ack_dp(peer_device, P_NEG_ACK, &d);
+ atomic_inc(&connection->current_epoch->epoch_size);
+ err2 = ignore_remaining_packet(connection, pi->size);
+ if (!err)
+ err = err2;
return err;
+ }
- mutex_lock(&connection->resource->conf_update);
- old_net_conf = peer_device->connection->net_conf;
- if (get_ldev(device)) {
- new_disk_conf = kzalloc_obj(struct disk_conf);
- if (!new_disk_conf) {
- put_ldev(device);
- mutex_unlock(&connection->resource->conf_update);
- drbd_err(device, "Allocation of new disk_conf failed\n");
- return -ENOMEM;
- }
+ /*
+ * Corresponding put_ldev done either below (on various errors), or in
+ * drbd_peer_request_endio, if we successfully submit the data at the
+ * end of this function.
+ */
- old_disk_conf = device->ldev->disk_conf;
- *new_disk_conf = *old_disk_conf;
+ peer_req = drbd_alloc_peer_req(peer_device, GFP_TRY, d.bi_size,
+ wire_flags_to_bio(connection, d.dp_flags));
+ if (!peer_req) {
+ put_ldev(device);
+ return -EIO;
+ }
+ peer_req->i.size = d.bi_size; /* storage size */
+ peer_req->i.sector = d.sector;
+ peer_req->i.type = INTERVAL_PEER_WRITE;
- new_disk_conf->resync_rate = be32_to_cpu(p->resync_rate);
+ err = read_in_block(peer_req, &d);
+ if (err) {
+ drbd_free_peer_req(peer_req);
+ put_ldev(device);
+ return err;
}
- if (apv >= 88) {
- if (apv == 88) {
- if (data_size > SHARED_SECRET_MAX || data_size == 0) {
- drbd_err(device, "verify-alg of wrong size, "
- "peer wants %u, accepting only up to %u byte\n",
- data_size, SHARED_SECRET_MAX);
- goto reconnect;
- }
+ if (pi->cmd == P_TRIM)
+ peer_req->flags |= EE_TRIM;
+ else if (pi->cmd == P_ZEROES)
+ peer_req->flags |= EE_ZEROOUT;
- err = drbd_recv_all(peer_device->connection, p->verify_alg, data_size);
- if (err)
- goto reconnect;
- /* we expect NUL terminated string */
- /* but just in case someone tries to be evil */
- D_ASSERT(device, p->verify_alg[data_size-1] == 0);
- p->verify_alg[data_size-1] = 0;
+ peer_req->w.cb = e_end_block;
+ peer_req->submit_jif = jiffies;
- } else /* apv >= 89 */ {
- /* we still expect NUL terminated strings */
- /* but just in case someone tries to be evil */
- D_ASSERT(device, p->verify_alg[SHARED_SECRET_MAX-1] == 0);
- D_ASSERT(device, p->csums_alg[SHARED_SECRET_MAX-1] == 0);
- p->verify_alg[SHARED_SECRET_MAX-1] = 0;
- p->csums_alg[SHARED_SECRET_MAX-1] = 0;
- }
+ if (pi->cmd == P_TRIM) {
+ D_ASSERT(peer_device, peer_req->i.size > 0);
+ D_ASSERT(peer_device, d.dp_flags & DP_DISCARD);
+ D_ASSERT(peer_device, bio_op(peer_req->bios.head) == REQ_OP_DISCARD);
+ /* need to play safe: an older DRBD sender
+ * may mean zero-out while sending P_TRIM. */
+ if (0 == (connection->agreed_features & DRBD_FF_WZEROES))
+ peer_req->flags |= EE_ZEROOUT;
+ } else if (pi->cmd == P_ZEROES) {
+ D_ASSERT(peer_device, peer_req->i.size > 0);
+ D_ASSERT(peer_device, d.dp_flags & DP_ZEROES);
+ D_ASSERT(peer_device, bio_op(peer_req->bios.head) == REQ_OP_WRITE_ZEROES);
+ /* Do (not) pass down BLKDEV_ZERO_NOUNMAP? */
+ if (d.dp_flags & DP_DISCARD)
+ peer_req->flags |= EE_TRIM;
+ } else {
+ D_ASSERT(peer_device, peer_req->i.size > 0);
+ D_ASSERT(peer_device, bio_op(peer_req->bios.head) == REQ_OP_WRITE);
+ }
- if (strcmp(old_net_conf->verify_alg, p->verify_alg)) {
- if (device->state.conn == C_WF_REPORT_PARAMS) {
- drbd_err(device, "Different verify-alg settings. me=\"%s\" peer=\"%s\"\n",
- old_net_conf->verify_alg, p->verify_alg);
- goto disconnect;
- }
- verify_tfm = drbd_crypto_alloc_digest_safe(device,
- p->verify_alg, "verify-alg");
- if (IS_ERR(verify_tfm)) {
- verify_tfm = NULL;
- goto disconnect;
- }
- }
+ if (d.dp_flags & DP_MAY_SET_IN_SYNC)
+ peer_req->flags |= EE_MAY_SET_IN_SYNC;
- if (apv >= 89 && strcmp(old_net_conf->csums_alg, p->csums_alg)) {
- if (device->state.conn == C_WF_REPORT_PARAMS) {
- drbd_err(device, "Different csums-alg settings. me=\"%s\" peer=\"%s\"\n",
- old_net_conf->csums_alg, p->csums_alg);
- goto disconnect;
- }
- csums_tfm = drbd_crypto_alloc_digest_safe(device,
- p->csums_alg, "csums-alg");
- if (IS_ERR(csums_tfm)) {
- csums_tfm = NULL;
- goto disconnect;
+ spin_lock(&connection->epoch_lock);
+ peer_req->epoch = connection->current_epoch;
+ atomic_inc(&peer_req->epoch->epoch_size);
+ atomic_inc(&peer_req->epoch->active);
+ if (peer_req->epoch->oldest_unconfirmed_peer_req == NULL)
+ peer_req->epoch->oldest_unconfirmed_peer_req = peer_req;
+
+ if (connection->resource->write_ordering == WO_BIO_BARRIER &&
+ atomic_read(&peer_req->epoch->epoch_size) == 1) {
+ struct drbd_epoch *epoch;
+ /* Issue a barrier if we start a new epoch, and the previous epoch
+ was not a epoch containing a single request which already was
+ a Barrier. */
+ epoch = list_entry(peer_req->epoch->list.prev, struct drbd_epoch, list);
+ if (epoch == peer_req->epoch) {
+ set_bit(DE_CONTAINS_A_BARRIER, &peer_req->epoch->flags);
+ peer_req->bios.head->bi_opf |= REQ_PREFLUSH | REQ_FUA;
+ peer_req->flags |= EE_IS_BARRIER;
+ } else {
+ if (atomic_read(&epoch->epoch_size) > 1 ||
+ !test_bit(DE_CONTAINS_A_BARRIER, &epoch->flags)) {
+ set_bit(DE_BARRIER_IN_NEXT_EPOCH_ISSUED, &epoch->flags);
+ set_bit(DE_CONTAINS_A_BARRIER, &peer_req->epoch->flags);
+ peer_req->bios.head->bi_opf |= REQ_PREFLUSH | REQ_FUA;
+ peer_req->flags |= EE_IS_BARRIER;
}
}
+ }
+ spin_unlock(&connection->epoch_lock);
- if (apv > 94 && new_disk_conf) {
- new_disk_conf->c_plan_ahead = be32_to_cpu(p->c_plan_ahead);
- new_disk_conf->c_delay_target = be32_to_cpu(p->c_delay_target);
- new_disk_conf->c_fill_target = be32_to_cpu(p->c_fill_target);
- new_disk_conf->c_max_rate = be32_to_cpu(p->c_max_rate);
-
- fifo_size = (new_disk_conf->c_plan_ahead * 10 * SLEEP_TIME) / HZ;
- if (fifo_size != device->rs_plan_s->size) {
- new_plan = fifo_alloc(fifo_size);
- if (!new_plan) {
- drbd_err(device, "kmalloc of fifo_buffer failed");
- put_ldev(device);
- goto disconnect;
- }
- }
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ tp = nc->two_primaries;
+ if (connection->agreed_pro_version < 100) {
+ switch (nc->wire_protocol) {
+ case DRBD_PROT_C:
+ d.dp_flags |= DP_SEND_WRITE_ACK;
+ break;
+ case DRBD_PROT_B:
+ d.dp_flags |= DP_SEND_RECEIVE_ACK;
+ break;
}
+ }
+ rcu_read_unlock();
- if (verify_tfm || csums_tfm) {
- new_net_conf = kzalloc_obj(struct net_conf);
- if (!new_net_conf)
- goto disconnect;
+ if (d.dp_flags & DP_SEND_WRITE_ACK) {
+ peer_req->flags |= EE_SEND_WRITE_ACK;
+ inc_unacked(peer_device);
+ /* corresponding dec_unacked() in e_end_block()
+ * respective _drbd_clear_done_ee */
+ }
- *new_net_conf = *old_net_conf;
+ if (d.dp_flags & DP_SEND_RECEIVE_ACK) {
+ /* I really don't like it that the receiver thread
+ * sends on the msock, but anyways */
+ drbd_send_ack(peer_device, P_RECV_ACK, peer_req);
+ }
- if (verify_tfm) {
- strscpy(new_net_conf->verify_alg, p->verify_alg);
- new_net_conf->verify_alg_len = strlen(p->verify_alg) + 1;
- crypto_free_shash(peer_device->connection->verify_tfm);
- peer_device->connection->verify_tfm = verify_tfm;
- drbd_info(device, "using verify-alg: \"%s\"\n", p->verify_alg);
- }
- if (csums_tfm) {
- strscpy(new_net_conf->csums_alg, p->csums_alg);
- new_net_conf->csums_alg_len = strlen(p->csums_alg) + 1;
- crypto_free_shash(peer_device->connection->csums_tfm);
- peer_device->connection->csums_tfm = csums_tfm;
- drbd_info(device, "using csums-alg: \"%s\"\n", p->csums_alg);
- }
- rcu_assign_pointer(connection->net_conf, new_net_conf);
- }
+ if (tp) {
+ /* two primaries implies protocol C */
+ D_ASSERT(device, d.dp_flags & DP_SEND_WRITE_ACK);
+ err = wait_for_and_update_peer_seq(peer_device, d.peer_seq);
+ if (err)
+ goto out;
+ } else {
+ update_peer_seq(peer_device, d.peer_seq);
}
- if (new_disk_conf) {
- rcu_assign_pointer(device->ldev->disk_conf, new_disk_conf);
- put_ldev(device);
+ peer_req->dagtag_sector = atomic64_read(&connection->last_dagtag_sector) + (peer_req->i.size >> 9);
+
+ drbd_wait_for_activity_log_extents(peer_req);
+
+ atomic_inc(&connection->active_ee_cnt);
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_add_tail(&peer_req->recv_order, &connection->peer_requests);
+ peer_req->flags |= EE_ON_RECV_ORDER;
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ /* The answer is matched against peer_requests, so ask only once this
+ * request is on that list. Still before the interval goes into the
+ * tree: from there on the write may be submitted and completed.
+ */
+ drbd_withhold_ack_until_source_has_write(peer_req);
+
+ /* Note: this now may or may not be "hot" in the activity log.
+ * Still, it is the best time to record that we need to set the
+ * out-of-sync bit, if we delay that until drbd_submit_peer_request(),
+ * we may introduce a race with some re-attach on the peer.
+ * Unless we want to guarantee that we drain all in-flight IO
+ * whenever we receive a state change. Which I'm not sure about.
+ * Use the EE_SET_OUT_OF_SYNC flag, to be acted on just before
+ * the actual submit, when we can be sure it is "hot".
+ */
+ if (peer_device->disk_state[NOW] < D_INCONSISTENT) {
+ peer_req->flags &= ~EE_MAY_SET_IN_SYNC;
+ peer_req->flags |= EE_SET_OUT_OF_SYNC;
}
- if (new_plan) {
- old_plan = device->rs_plan_s;
- rcu_assign_pointer(device->rs_plan_s, new_plan);
+ spin_lock_irq(&device->interval_lock);
+ if (tp) {
+ err = drbd_peer_write_conflicts(peer_req);
+ if (err) {
+ spin_unlock_irq(&device->interval_lock);
+ goto out_del_list;
+ }
}
+ conflict = drbd_find_conflict(device, &peer_req->i, 0);
+ drbd_insert_interval(&device->requests, &peer_req->i);
+ if (conflict)
+ drbd_rs_depth_exempt(device, &peer_req->i);
+ else
+ set_bit(INTERVAL_SUBMITTED, &peer_req->i.flags);
+ spin_unlock_irq(&device->interval_lock);
- mutex_unlock(&connection->resource->conf_update);
- synchronize_rcu();
- if (new_net_conf)
- kfree(old_net_conf);
- kfree(old_disk_conf);
- kfree(old_plan);
+ /* The connection dagtag may only be set after inserting the interval
+ * into the tree, so that requests that were waiting for the dagtag
+ * enter the interval tree after the request with the dagtag itself. */
+ set_connection_dagtag(connection, peer_req->dagtag_sector);
+ if (!conflict)
+ submit_peer_request_activity_log(peer_req);
+ /* ldev_ref_transfer: put_ldev in peer_req endio */
return 0;
-reconnect:
- if (new_disk_conf) {
- put_ldev(device);
- kfree(new_disk_conf);
- }
- mutex_unlock(&connection->resource->conf_update);
- return -EIO;
+out_del_list:
+ spin_lock_irq(&connection->peer_reqs_lock);
+ peer_req->flags &= ~EE_ON_RECV_ORDER;
+ list_del(&peer_req->recv_order);
+ spin_unlock_irq(&connection->peer_reqs_lock);
-disconnect:
- kfree(new_plan);
- if (new_disk_conf) {
- put_ldev(device);
- kfree(new_disk_conf);
- }
- mutex_unlock(&connection->resource->conf_update);
- /* just for completeness: actually not needed,
- * as this is not reached if csums_tfm was ok. */
- crypto_free_shash(csums_tfm);
- /* but free the verify_tfm again, if csums_tfm did not work out */
- crypto_free_shash(verify_tfm);
- conn_request_state(peer_device->connection, NS(conn, C_DISCONNECTING), CS_HARD);
- return -EIO;
-}
+ if (atomic_dec_and_test(&connection->active_ee_cnt))
+ wake_up(&connection->ee_wait);
+ atomic_sub(interval_to_al_extents(&peer_req->i), &device->wait_for_actlog_ecnt);
-/* warn if the arguments differ by more than 12.5% */
-static void warn_if_differ_considerably(struct drbd_device *device,
- const char *s, sector_t a, sector_t b)
-{
- sector_t d;
- if (a == 0 || b == 0)
- return;
- d = (a > b) ? (a - b) : (b - a);
- if (d > (a>>3) || d > (b>>3))
- drbd_warn(device, "Considerable difference in %s: %llus vs. %llus\n", s,
- (unsigned long long)a, (unsigned long long)b);
+out:
+ if (peer_req->flags & EE_SEND_WRITE_ACK)
+ dec_unacked(peer_device);
+ drbd_may_finish_epoch(connection, peer_req->epoch, EV_PUT + EV_CLEANUP);
+ put_ldev(device);
+ drbd_free_peer_req(peer_req);
+ return err;
}
-static int receive_sizes(struct drbd_connection *connection, struct packet_info *pi)
+/*
+ * To be called when drbd_submit_peer_request() fails for a peer write request.
+ */
+void drbd_cleanup_after_failed_submit_peer_write(struct drbd_peer_request *peer_req)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_sizes *p = pi->data;
- struct o_qlim *o = (connection->agreed_features & DRBD_FF_WSAME) ? p->qlim : NULL;
- enum determine_dev_size dd = DS_UNCHANGED;
- sector_t p_size, p_usize, p_csize, my_usize;
- sector_t new_size, cur_size;
- int ldsc = 0; /* local disk size changed */
- enum dds_flags ddsf;
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return config_unknown_volume(connection, pi);
- device = peer_device->device;
- cur_size = get_capacity(device->vdisk);
+ drbd_err_ratelimit(peer_device, "submit failed, triggering re-connect\n");
- p_size = be64_to_cpu(p->d_size);
- p_usize = be64_to_cpu(p->u_size);
- p_csize = be64_to_cpu(p->c_size);
+ if (peer_req->flags & EE_IN_ACTLOG)
+ drbd_al_complete_io(device, &peer_req->i);
- /* just store the peer's disk size for now.
- * we still need to figure out whether we accept that. */
- device->p_size = p_size;
+ drbd_remove_peer_req_interval(peer_req);
- if (get_ldev(device)) {
- rcu_read_lock();
- my_usize = rcu_dereference(device->ldev->disk_conf)->disk_size;
- rcu_read_unlock();
+ drbd_may_finish_epoch(connection, peer_req->epoch, EV_PUT + EV_CLEANUP);
+ put_ldev(device);
+ drbd_free_peer_req(peer_req);
+ change_cstate(connection, C_PROTOCOL_ERROR, CS_HARD);
+}
- warn_if_differ_considerably(device, "lower level device sizes",
- p_size, drbd_get_max_capacity(device->ldev));
- warn_if_differ_considerably(device, "user requested size",
- p_usize, my_usize);
+/* Possibly "cancel" and forget about all peer_requests that had still been
+ * waiting for the activity log (wfa) when the connection to their peer failed,
+ * and pretend we never received them.
+ */
+void drbd_cleanup_peer_requests_wfa(struct drbd_device *device, struct list_head *cleanup)
+{
+ struct drbd_connection *connection;
+ struct drbd_peer_request *peer_req, *pr_tmp;
+
+ write_lock_irq(&device->resource->state_rwlock);
+ list_for_each_entry(peer_req, cleanup, w.list) {
+ atomic_dec(&peer_req->peer_device->connection->active_ee_cnt);
+ drbd_remove_peer_req_interval(peer_req);
+ }
+ write_unlock_irq(&device->resource->state_rwlock);
+
+ list_for_each_entry_safe(peer_req, pr_tmp, cleanup, w.list) {
+ atomic_sub(interval_to_al_extents(&peer_req->i), &device->wait_for_actlog_ecnt);
+ atomic_dec(&device->wait_for_actlog);
+ if (peer_req->flags & EE_SEND_WRITE_ACK)
+ dec_unacked(peer_req->peer_device);
+ list_del_init(&peer_req->w.list);
+ drbd_may_finish_epoch(peer_req->peer_device->connection, peer_req->epoch, EV_PUT | EV_CLEANUP);
+ drbd_free_peer_req(peer_req);
+ put_ldev(device);
+ }
+ /*
+ * We changed (likely: cleared) active_ee_cnt for "at least one" connection.
+ * We should wake potential waiters, just in case.
+ */
+ for_each_connection(connection, device->resource)
+ wake_up(&connection->ee_wait);
+}
- /* if this is the first connect, or an otherwise expected
- * param exchange, choose the minimum */
- if (device->state.conn == C_WF_REPORT_PARAMS)
- p_usize = min_not_zero(my_usize, p_usize);
+/* We may throttle resync, if the lower device seems to be busy,
+ * and current sync rate is above c_min_rate.
+ *
+ * To decide whether or not the lower device is busy, we use a scheme similar
+ * to MD RAID is_mddev_idle(): if the partition stats reveal "significant"
+ * (more than 64 sectors) of activity we cannot account for with our own resync
+ * activity, it obviously is "busy". Both terms count what the backing device
+ * has completed, so resync IO in flight does not hide application IO.
+ *
+ * The current sync rate used here uses only the most recent two step marks,
+ * to have a short time average so we can react faster.
+ */
+bool drbd_rs_c_min_rate_throttle(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct gendisk *disk = device->ldev->backing_bdev->bd_disk;
+ unsigned long dt, dbdt;
+ long db;
+ unsigned int c_min_rate;
+ int curr_events;
- /* Never shrink a device with usable data during connect,
- * or "attach" on the peer.
- * But allow online shrinking if we are connected. */
- new_size = drbd_new_dev_size(device, device->ldev, p_usize, 0);
- if (new_size < cur_size &&
- device->state.disk >= D_OUTDATED &&
- (device->state.conn < C_CONNECTED || device->state.pdsk == D_DISKLESS)) {
- drbd_err(device, "The peer's disk size is too small! (%llu < %llu sectors)\n",
- (unsigned long long)new_size, (unsigned long long)cur_size);
- conn_request_state(peer_device->connection, NS(conn, C_DISCONNECTING), CS_HARD);
- put_ldev(device);
- return -EIO;
- }
+ rcu_read_lock();
+ c_min_rate = rcu_dereference(peer_device->conf)->c_min_rate;
+ rcu_read_unlock();
- if (my_usize != p_usize) {
- struct disk_conf *old_disk_conf, *new_disk_conf = NULL;
+ /* feature disabled? */
+ if (c_min_rate == 0)
+ return false;
- new_disk_conf = kzalloc_obj(struct disk_conf);
- if (!new_disk_conf) {
- put_ldev(device);
- return -ENOMEM;
- }
+ curr_events = (int)part_stat_read_accum(disk->part0, sectors)
+ - atomic_read(&device->rs_sect_done);
- mutex_lock(&connection->resource->conf_update);
- old_disk_conf = device->ldev->disk_conf;
- *new_disk_conf = *old_disk_conf;
- new_disk_conf->disk_size = p_usize;
+ if (atomic_read(&device->ap_actlog_cnt) || curr_events - peer_device->rs_last_events > 64) {
+ unsigned long rs_left;
+ int i;
- rcu_assign_pointer(device->ldev->disk_conf, new_disk_conf);
- mutex_unlock(&connection->resource->conf_update);
- kvfree_rcu_mightsleep(old_disk_conf);
+ peer_device->rs_last_events = curr_events;
- drbd_info(device, "Peer sets u_size to %lu sectors (old: %lu)\n",
- (unsigned long)p_usize, (unsigned long)my_usize);
- }
+ /* sync speed average over the last 2*DRBD_SYNC_MARK_STEP,
+ * approx. */
+ i = (peer_device->rs_last_mark + DRBD_SYNC_MARKS-1) % DRBD_SYNC_MARKS;
- put_ldev(device);
- }
+ if (peer_device->repl_state[NOW] == L_VERIFY_S || peer_device->repl_state[NOW] == L_VERIFY_T)
+ rs_left = atomic64_read(&peer_device->ov_left);
+ else
+ rs_left = drbd_bm_total_weight(peer_device) - peer_device->rs_failed;
- device->peer_max_bio_size = be32_to_cpu(p->max_bio_size);
- /* Leave drbd_reconsider_queue_parameters() before drbd_determine_dev_size().
- In case we cleared the QUEUE_FLAG_DISCARD from our queue in
- drbd_reconsider_queue_parameters(), we can be sure that after
- drbd_determine_dev_size() no REQ_DISCARDs are in the queue. */
+ dt = ((long)jiffies - (long)peer_device->rs_mark_time[i]) / HZ;
+ if (!dt)
+ dt++;
- ddsf = be16_to_cpu(p->dds_flags);
- if (get_ldev(device)) {
- drbd_reconsider_queue_parameters(device, device->ldev, o);
- dd = drbd_determine_dev_size(device, ddsf, NULL);
- put_ldev(device);
- if (dd == DS_ERROR)
- return -EIO;
- drbd_md_sync(device);
- } else {
- /*
- * I am diskless, need to accept the peer's *current* size.
- * I must NOT accept the peers backing disk size,
- * it may have been larger than mine all along...
- *
- * At this point, the peer knows more about my disk, or at
- * least about what we last agreed upon, than myself.
- * So if his c_size is less than his d_size, the most likely
- * reason is that *my* d_size was smaller last time we checked.
- *
- * However, if he sends a zero current size,
- * take his (user-capped or) backing disk size anyways.
- *
- * Unless of course he does not have a disk himself.
- * In which case we ignore this completely.
+ /* db is signed: the out-of-sync count grows (db < 0) when the
+ * resync is losing ground to incoming application writes. Doing
+ * this subtraction unsigned would wrap to a huge value, making
+ * dbdt exceed c_min_rate and wrongly throttle the resync exactly
+ * when it is starved. If oos is not shrinking, the resync makes
+ * no forward progress, i.e. it is below any minimum rate, so we
+ * must not slow it down here.
*/
- sector_t new_size = p_csize ?: p_usize ?: p_size;
- drbd_reconsider_queue_parameters(device, NULL, o);
- if (new_size == 0) {
- /* Ignore, peer does not know nothing. */
- } else if (new_size == cur_size) {
- /* nothing to do */
- } else if (cur_size != 0 && p_size == 0) {
- drbd_warn(device, "Ignored diskless peer device size (peer:%llu != me:%llu sectors)!\n",
- (unsigned long long)new_size, (unsigned long long)cur_size);
- } else if (new_size < cur_size && device->state.role == R_PRIMARY) {
- drbd_err(device, "The peer's device size is too small! (%llu < %llu sectors); demote me first!\n",
- (unsigned long long)new_size, (unsigned long long)cur_size);
- conn_request_state(peer_device->connection, NS(conn, C_DISCONNECTING), CS_HARD);
- return -EIO;
+ db = (long)peer_device->rs_mark_left[i] - (long)rs_left;
+ if (db <= 0)
+ return false;
+ dbdt = device_bit_to_kb(device, db / dt);
+
+ if (dbdt > c_min_rate)
+ return true;
+ }
+ return false;
+}
+
+void drbd_verify_skipped_block(struct drbd_peer_device *peer_device,
+ const sector_t sector, const unsigned int size)
+{
+ ++peer_device->ov_skipped;
+ if (peer_device->ov_last_skipped_start + peer_device->ov_last_skipped_size == sector) {
+ peer_device->ov_last_skipped_size += size>>9;
+ } else {
+ ov_skipped_print(peer_device);
+ peer_device->ov_last_skipped_start = sector;
+ peer_device->ov_last_skipped_size = size>>9;
+ }
+}
+
+static void drbd_cleanup_peer_read(
+ struct drbd_peer_request *peer_req, bool in_interval_tree)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+
+ if (in_interval_tree)
+ drbd_remove_peer_req_interval(peer_req);
+
+ dec_unacked(peer_device);
+
+ drbd_free_peer_req(peer_req);
+ put_ldev(device);
+
+ if (atomic_dec_and_test(&connection->backing_ee_cnt))
+ wake_up(&connection->ee_wait);
+}
+
+void drbd_conflict_submit_peer_read(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ struct conflict_worker *submit_conflict = &device->submit_conflict;
+ bool submit = true;
+ bool interval_tree = false;
+ bool canceled = false;
+ bool too_deep = false;
+
+ /* Hold resync reads until conflicts have cleared so that we know which
+ * bitmap bits we can safely clear. Also add verify requests on the
+ * target to the interval tree so that conflicts can be detected.
+ * Verify requests on the source have already been added. */
+ if (drbd_interval_is_resync(&peer_req->i) || peer_req->i.type == INTERVAL_OV_READ_TARGET) {
+ bool conflict = false;
+
+ interval_tree = true;
+ spin_lock_irq(&device->interval_lock);
+ clear_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &peer_req->i.flags);
+ canceled = test_bit(INTERVAL_CANCELED, &peer_req->i.flags);
+ conflict = drbd_find_conflict(device, &peer_req->i, 0);
+ if (drbd_interval_empty(&peer_req->i)) {
+ if (conflict)
+ set_bit(INTERVAL_CONFLICT, &peer_req->i.flags);
+ drbd_insert_interval(&device->requests, &peer_req->i);
+ }
+ too_deep = !conflict && !canceled &&
+ drbd_interval_is_resync(&peer_req->i) &&
+ !test_bit(INTERVAL_APP_WAITS, &peer_req->i.flags) &&
+ drbd_rs_depth_exceeded(peer_device);
+ if (canceled) {
+ submit = false;
+ } else if (too_deep) {
+ set_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &peer_req->i.flags);
+ submit = false;
+ } else if (!conflict || drbd_interval_is_verify(&peer_req->i)) {
+ set_bit(INTERVAL_SUBMITTED, &peer_req->i.flags);
} else {
- /* I believe the peer, if
- * - I don't have a current size myself
- * - we agree on the size anyways
- * - I do have a current size, am Secondary,
- * and he has the only disk
- * - I do have a current size, am Primary,
- * and he has the only disk,
- * which is larger than my current size
- */
- drbd_set_my_capacity(device, new_size);
+ submit = false;
}
+ spin_unlock_irq(&device->interval_lock);
}
- if (get_ldev(device)) {
- if (device->ldev->known_size != drbd_get_capacity(device->ldev->backing_bdev)) {
- device->ldev->known_size = drbd_get_capacity(device->ldev->backing_bdev);
- ldsc = 1;
- }
+ if (too_deep) {
+ /* The read this node serves queues at its own backing device,
+ * where its application's IO queues behind it. Wait for that
+ * queue on the conflict submitter's list, not in the receiver:
+ * when the sync target is Primary, its replicated writes arrive
+ * on the same stream as these requests.
+ */
+ spin_lock_irq(&submit_conflict->lock);
+ list_add_tail(&peer_req->w.list, &submit_conflict->resync_reads);
+ spin_unlock_irq(&submit_conflict->lock);
- put_ldev(device);
+ smp_mb(); /* list_add vs. re-check; see drbd_rs_depth_completed() */
+ if (!drbd_rs_depth_exceeded(peer_device))
+ queue_work(submit_conflict->wq, &submit_conflict->worker);
+ return;
}
- if (device->state.conn > C_WF_REPORT_PARAMS) {
- if (be64_to_cpu(p->c_size) != get_capacity(device->vdisk) ||
- ldsc) {
- /* we have different sizes, probably peer
- * needs to know my new size... */
- drbd_send_sizes(peer_device, 0, ddsf);
- }
- if (test_and_clear_bit(RESIZE_PENDING, &device->flags) ||
- (dd == DS_GREW && device->state.conn == C_CONNECTED)) {
- if (device->state.pdsk >= D_INCONSISTENT &&
- device->state.disk >= D_INCONSISTENT) {
- if (ddsf & DDSF_NO_RESYNC)
- drbd_info(device, "Resync of new storage suppressed with --assume-clean\n");
- else
- resync_after_online_grow(device);
- } else
- set_bit(RESYNC_AFTER_NEG, &device->flags);
+ /* Wait if there are conflicts unless this is a verify request, in
+ * which case we submit it anyway but skip the block if it conflicted. */
+ if (submit) {
+ unsigned int size = drbd_interval_is_resync(&peer_req->i) ? peer_req->i.size : 0;
+ int err;
+
+ drbd_rs_depth_queued(device, size >> 9);
+ err = drbd_submit_peer_request(peer_req);
+ if (err) {
+ atomic_sub(size >> 9, &device->rs_sect_queued);
+ if (drbd_ratelimit())
+ drbd_err(peer_device, "submit failed, triggering re-connect\n");
+
+ drbd_cleanup_peer_read(peer_req, interval_tree);
+ change_cstate(peer_device->connection, C_PROTOCOL_ERROR, CS_HARD);
}
+ } else if (canceled) {
+ drbd_cleanup_peer_read(peer_req, interval_tree);
}
-
- return 0;
}
-static int receive_uuids(struct drbd_connection *connection, struct packet_info *pi)
+enum dagtag_reach {
+ DAGTAG_REACHED, /* the depended-on write stream got here */
+ DAGTAG_WAITING, /* not yet, but the peer that feeds it is connected */
+ DAGTAG_UNREACHABLE, /* that peer is gone; the position can not be reached */
+};
+
+static enum dagtag_reach dagtag_reach(struct drbd_resource *resource,
+ unsigned int node_id, u64 dagtag)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_uuids *p = pi->data;
- u64 *p_uuid;
- int i, updated_uuids = 0;
+ struct drbd_connection *connection;
+ enum dagtag_reach reach = DAGTAG_UNREACHABLE;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return config_unknown_volume(connection, pi);
- device = peer_device->device;
+ rcu_read_lock();
+ connection = drbd_connection_by_node_id(resource, node_id);
+ if (connection && connection->cstate[NOW] == C_CONNECTED &&
+ !test_bit(DAGTAG_STREAM_GONE, &connection->flags))
+ reach = atomic64_read(&connection->last_dagtag_sector) < dagtag ?
+ DAGTAG_WAITING : DAGTAG_REACHED;
+ rcu_read_unlock();
- p_uuid = kmalloc_objs(*p_uuid, UI_EXTENDED_SIZE, GFP_NOIO);
- if (!p_uuid)
- return false;
+ return reach;
+}
- for (i = UI_CURRENT; i < UI_EXTENDED_SIZE; i++)
- p_uuid[i] = be64_to_cpu(p->uuid[i]);
+static enum dagtag_reach dagtag_dependency_reach(struct drbd_peer_request *peer_req)
+{
+ struct drbd_device *device = peer_req->peer_device->device;
- kfree(device->p_uuid);
- device->p_uuid = p_uuid;
+ return dagtag_reach(device->resource, peer_req->depend_dagtag_node_id,
+ peer_req->depend_dagtag);
+}
- if ((device->state.conn < C_CONNECTED || device->state.pdsk == D_DISKLESS) &&
- device->state.disk < D_INCONSISTENT &&
- device->state.role == R_PRIMARY &&
- (device->ed_uuid & ~((u64)1)) != (p_uuid[UI_CURRENT] & ~((u64)1))) {
- drbd_err(device, "Can only connect to data with current UUID=%016llX\n",
- (unsigned long long)device->ed_uuid);
- conn_request_state(peer_device->connection, NS(conn, C_DISCONNECTING), CS_HARD);
- return -EIO;
+static void drbd_peer_resync_read_cancel(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ sector_t sector = peer_req->i.sector;
+ int size = peer_req->i.size;
+ u64 block_id = peer_req->block_id;
+
+ if (peer_req->i.type == INTERVAL_OV_READ_SOURCE) {
+ /* P_OV_REPLY.
+ * rs_pending was already decremented in
+ * receive_common_ov_reply() before the peer_req reached
+ * either of the two paths that call us; do not decrement
+ * again.
+ */
+ drbd_send_ov_result(peer_device, sector, size, block_id, OV_RESULT_SKIP);
+ } else if (peer_req->i.type == INTERVAL_OV_READ_TARGET) {
+ /* P_OV_REQUEST */
+ drbd_verify_skipped_block(peer_device, sector, size);
+ verify_progress(peer_device, sector, size);
+ drbd_send_ack_be(peer_device, P_RS_CANCEL, sector, size, block_id);
+ } else {
+ /* P_RS_DATA_REQUEST etc */
+ drbd_send_ack_be(peer_device, P_RS_CANCEL, sector, size, block_id);
}
-
- if (get_ldev(device)) {
- int skip_initial_sync =
- device->state.conn == C_CONNECTED &&
- peer_device->connection->agreed_pro_version >= 90 &&
- device->ldev->md.uuid[UI_CURRENT] == UUID_JUST_CREATED &&
- (p_uuid[UI_FLAGS] & 8);
- if (skip_initial_sync) {
- drbd_info(device, "Accepted new current UUID, preparing to skip initial sync\n");
- drbd_bitmap_io(device, &drbd_bmio_clear_n_write,
- "clear_n_write from receive_uuids",
- BM_LOCKED_TEST_ALLOWED, NULL);
- _drbd_uuid_set(device, UI_CURRENT, p_uuid[UI_CURRENT]);
- _drbd_uuid_set(device, UI_BITMAP, 0);
- spin_lock_irq(&device->resource->req_lock);
- _drbd_set_state(_NS2(device, disk, D_UP_TO_DATE, pdsk, D_UP_TO_DATE),
- CS_VERBOSE, NULL);
- spin_unlock_irq(&device->resource->req_lock);
- drbd_md_sync(device);
- updated_uuids = 1;
- }
- put_ldev(device);
- } else if (device->state.disk < D_INCONSISTENT &&
- device->state.role == R_PRIMARY) {
- /* I am a diskless primary, the peer just created a new current UUID
- for me. */
- updated_uuids = drbd_set_ed_uuid(device, p_uuid[UI_CURRENT]);
- }
-
- /* Before we test for the disk state, we should wait until an eventually
- ongoing cluster wide state change is finished. That is important if
- we are primary and are detaching from our disk. We need to see the
- new disk state... */
- mutex_lock(device->state_mutex);
- mutex_unlock(device->state_mutex);
- if (device->state.conn >= C_CONNECTED && device->state.disk < D_INCONSISTENT)
- updated_uuids |= drbd_set_ed_uuid(device, p_uuid[UI_CURRENT]);
-
- if (updated_uuids)
- drbd_print_uuids(device, "receiver updated UUIDs to");
-
- return 0;
}
-/**
- * convert_state() - Converts the peer's view of the cluster state to our point of view
- * @ps: The state as seen by the peer.
+/* The peer that feeds the depended-on write stream is gone, so the requested
+ * position can never be reached. Reading the local disk would answer with data
+ * that is older than what the requester holds itself.
+ *
+ * A verify request is cancelled the same way a request already parked for
+ * that dagtag is cancelled when the connection to that peer goes down: the
+ * requester skips the block and the verify goes on.
+ *
+ * A resync request from a target that agreed DRBD_FF_WRITE_POSTPONE is
+ * answered P_RS_DAGTAG_UNREACHABLE without data: the target knows which of
+ * its writes the requested range still has to secure, refuses them towards
+ * their writer, and asks again without the dependency, or ends the resync
+ * when it can not secure the range. Any other target gets the request
+ * answered as failed (P_NEG_RS_DREPLY): it records the block as failed and
+ * moves on, so it does not ask for the same bitmap bit again, and its
+ * resync ends with the disk left Inconsistent instead of a copy that lost an
+ * acknowledged write. This node's own state is not touched; the requester
+ * ends the exchange, as it does for a block this node could not read.
*/
-static union drbd_state convert_state(union drbd_state ps)
+static void drbd_cancel_unreachable_dagtag_request(struct drbd_peer_request *peer_req)
{
- union drbd_state ms;
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ unsigned int node_id = peer_req->depend_dagtag_node_id;
+
+ if (drbd_interval_is_verify(&peer_req->i)) {
+ drbd_peer_resync_read_cancel(peer_req);
+ } else if (peer_device->connection->agreed_features & DRBD_FF_WRITE_POSTPONE) {
+ dynamic_drbd_dbg(peer_device,
+ "Not connected to node %u, resync request at %llus+%u depends on it\n",
+ node_id, (unsigned long long)peer_req->i.sector,
+ peer_req->i.size);
+ drbd_send_ack_be(peer_device, P_RS_DAGTAG_UNREACHABLE, peer_req->i.sector,
+ peer_req->i.size, peer_req->block_id);
+ } else {
+ drbd_info_ratelimit(peer_device,
+ "Not connected to node %u, can not serve resync request at %llus+%u\n",
+ node_id, (unsigned long long)peer_req->i.sector,
+ peer_req->i.size);
+ drbd_send_ack_be(peer_device, P_NEG_RS_DREPLY, peer_req->i.sector,
+ peer_req->i.size, peer_req->block_id);
+ }
+ if (peer_req->i.type == INTERVAL_OV_READ_SOURCE)
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ dec_unacked(peer_device);
+ put_ldev(device);
+}
- static enum drbd_conns c_tab[] = {
- [C_WF_REPORT_PARAMS] = C_WF_REPORT_PARAMS,
- [C_CONNECTED] = C_CONNECTED,
+static void drbd_peer_resync_read(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ unsigned int size = peer_req->i.size;
- [C_STARTING_SYNC_S] = C_STARTING_SYNC_T,
- [C_STARTING_SYNC_T] = C_STARTING_SYNC_S,
- [C_DISCONNECTING] = C_TEAR_DOWN, /* C_NETWORK_FAILURE, */
- [C_VERIFY_S] = C_VERIFY_T,
- [C_MASK] = C_MASK,
- };
+ if (connection->peer_role[NOW] != R_PRIMARY &&
+ drbd_rs_c_min_rate_throttle(peer_device))
+ schedule_timeout_uninterruptible(HZ/10);
- ms.i = ps.i;
+ /* dagtag 0 means that there is no dependency to be fulfilled,
+ * so we can ignore it.
+ * If we are the dependent node, we can also ignore the dagtag
+ * dependency, because the request with this dagtag must already be in
+ * the interval tree, so the read will wait until the interval tree
+ * conflict is resolved before being submitted. */
+ if (peer_req->depend_dagtag &&
+ peer_req->depend_dagtag_node_id != device->resource->res_opts.node_id) {
+ enum dagtag_reach reach = dagtag_dependency_reach(peer_req);
+
+ if (reach == DAGTAG_WAITING) {
+ dynamic_drbd_dbg(peer_device,
+ "%s at %llus+%u: Waiting for dagtag %llus from peer %u\n",
+ drbd_interval_type_str(&peer_req->i),
+ (unsigned long long)peer_req->i.sector, size,
+ (unsigned long long)peer_req->depend_dagtag,
+ peer_req->depend_dagtag_node_id);
+ /* cancel_dagtag_dependent_requests() empties this list
+ * under this lock once the depended-on stream is torn
+ * down, and it marks that stream before it walks. Take
+ * the decision again here, so a request either lands
+ * before the walk or is never parked at all.
+ */
+ spin_lock_irq(&connection->peer_reqs_lock);
+ reach = dagtag_dependency_reach(peer_req);
+ if (reach == DAGTAG_WAITING)
+ list_add_tail(&peer_req->w.list, &connection->dagtag_wait_ee);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ if (reach == DAGTAG_WAITING)
+ return;
+ }
- ms.conn = c_tab[ps.conn];
- ms.peer = ps.role;
- ms.role = ps.peer;
- ms.pdsk = ps.disk;
- ms.disk = ps.pdsk;
- ms.peer_isp = (ps.aftr_isp | ps.user_isp);
+ if (reach == DAGTAG_UNREACHABLE) {
+ drbd_cancel_unreachable_dagtag_request(peer_req);
+ return;
+ }
+ }
- return ms;
+ atomic_inc(&connection->backing_ee_cnt);
+ drbd_conflict_submit_peer_read(peer_req);
}
-static int receive_req_state(struct drbd_connection *connection, struct packet_info *pi)
+/* A sync target has a write that this node, the sync source, may be about to
+ * overwrite from its own disk. Answer once this node has that write too,
+ * which it does as soon as the stream it belongs to has reached the position
+ * the requester names.
+ */
+static int receive_rs_dagtag_wait_req(struct drbd_connection *connection, struct packet_info *pi)
{
+ struct p_rs_req *p = pi->data;
struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_req_state *p = pi->data;
- union drbd_state mask, val;
- enum drbd_state_rv rv;
+ sector_t sector = be64_to_cpu(p->req_common.sector);
+ unsigned int size = be32_to_cpu(p->req_common.blksize);
+ unsigned int node_id = be32_to_cpu(p->dagtag_node_id);
+ u64 dagtag = be64_to_cpu(p->dagtag);
+ enum dagtag_reach reach;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
return -EIO;
- device = peer_device->device;
- mask.i = be32_to_cpu(p->mask);
- val.i = be32_to_cpu(p->val);
+ reach = dagtag_reach(connection->resource, node_id, dagtag);
+ if (reach == DAGTAG_WAITING)
+ return queue_dagtag_wait_req(peer_device, sector, size,
+ p->req_common.block_id, node_id, dagtag);
- if (test_bit(RESOLVE_CONFLICTS, &peer_device->connection->flags) &&
- mutex_is_locked(device->state_mutex)) {
- drbd_send_sr_reply(peer_device, SS_CONCURRENT_ST_CHG);
- return 0;
+ if (reach == DAGTAG_UNREACHABLE) {
+ dynamic_drbd_dbg(peer_device,
+ "Dagtag wait request at %llus+%u: can not reach dagtag %llus of node %u\n",
+ (unsigned long long)sector, size,
+ (unsigned long long)dagtag, node_id);
+ return drbd_send_ack_be(peer_device, P_RS_DAGTAG_UNREACHABLE,
+ sector, size, p->req_common.block_id);
}
- mask = convert_state(mask);
- val = convert_state(val);
-
- rv = drbd_change_state(device, CS_VERBOSE, mask, val);
- drbd_send_sr_reply(peer_device, rv);
-
- drbd_md_sync(device);
-
- return 0;
+ return drbd_send_ack_be(peer_device, P_RS_DAGTAG_REACHED, sector, size,
+ p->req_common.block_id);
}
-static int receive_req_conn_state(struct drbd_connection *connection, struct packet_info *pi)
+static int receive_digest(struct drbd_peer_request *peer_req, int digest_size)
{
- struct p_req_state *p = pi->data;
- union drbd_state mask, val;
- enum drbd_state_rv rv;
-
- mask.i = be32_to_cpu(p->mask);
- val.i = be32_to_cpu(p->val);
+ struct digest_info *di = NULL;
- if (test_bit(RESOLVE_CONFLICTS, &connection->flags) &&
- mutex_is_locked(&connection->cstate_mutex)) {
- conn_send_sr_reply(connection, SS_CONCURRENT_ST_CHG);
- return 0;
+ if (digest_size < 0) {
+ drbd_err(peer_req->peer_device, "Invalid digest size\n");
+ return -EIO;
}
- mask = convert_state(mask);
- val = convert_state(val);
+ di = kmalloc(sizeof(*di) + digest_size, GFP_NOIO);
+ if (!di)
+ return -ENOMEM;
- rv = conn_request_state(connection, mask, val, CS_VERBOSE | CS_LOCAL_ONLY | CS_IGN_OUTD_FAIL);
- conn_send_sr_reply(connection, rv);
+ di->digest_size = digest_size;
+ di->digest = (((char *)di)+sizeof(struct digest_info));
- return 0;
+ peer_req->digest = di;
+ peer_req->flags |= EE_HAS_DIGEST;
+
+ return drbd_recv_into(peer_req->peer_device->connection, di->digest, digest_size);
}
-static int receive_state(struct drbd_connection *connection, struct packet_info *pi)
+static int receive_common_data_request(struct drbd_connection *connection, struct packet_info *pi,
+ struct p_block_req_common *p,
+ unsigned int depend_dagtag_node_id, u64 depend_dagtag)
{
struct drbd_peer_device *peer_device;
struct drbd_device *device;
- struct p_state *p = pi->data;
- union drbd_state os, ns, peer_state;
- enum drbd_disk_state real_peer_disk;
- enum chg_state_flags cs_flags;
- int rv;
+ sector_t sector = be64_to_cpu(p->sector);
+ sector_t capacity;
+ struct drbd_peer_request *peer_req;
+ int size = be32_to_cpu(p->blksize);
+ enum drbd_disk_state min_d_state;
+ int err;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
- return config_unknown_volume(connection, pi);
+ return -EIO;
device = peer_device->device;
+ capacity = get_capacity(device->vdisk);
- peer_state.i = be32_to_cpu(p->state);
-
- real_peer_disk = peer_state.disk;
- if (peer_state.disk == D_NEGOTIATING) {
- real_peer_disk = device->p_uuid[UI_FLAGS] & 4 ? D_INCONSISTENT : D_CONSISTENT;
- drbd_info(device, "real peer disk state = %s\n", drbd_disk_str(real_peer_disk));
+ if (size <= 0 || !IS_ALIGNED(size, 512) || size > DRBD_MAX_BIO_SIZE) {
+ drbd_err(peer_device, "%s:%d: sector: %llus, size: %u\n", __FILE__, __LINE__,
+ (unsigned long long)sector, size);
+ return -EINVAL;
+ }
+ if (sector + (size>>9) > capacity &&
+ !in_range_after_size_change(device, sector, size)) {
+ drbd_err(peer_device, "%s:%d: sector: %llus, size: %u\n", __FILE__, __LINE__,
+ (unsigned long long)sector, size);
+ return -EINVAL;
}
- spin_lock_irq(&device->resource->req_lock);
- retry:
- os = ns = drbd_read_state(device);
- spin_unlock_irq(&device->resource->req_lock);
-
- /* If some other part of the code (ack_receiver thread, timeout)
- * already decided to close the connection again,
- * we must not "re-establish" it here. */
- if (os.conn <= C_TEAR_DOWN)
- return -ECONNRESET;
-
- /* If this is the "end of sync" confirmation, usually the peer disk
- * transitions from D_INCONSISTENT to D_UP_TO_DATE. For empty (0 bits
- * set) resync started in PausedSyncT, or if the timing of pause-/
- * unpause-sync events has been "just right", the peer disk may
- * transition from D_CONSISTENT to D_UP_TO_DATE as well.
+ /* Tell target to have a retry, waiting for the rescheduled
+ * drbd_start_resync to complete. Otherwise the concurrency
+ * of send oos and resync may lead to data loss.
*/
- if ((os.pdsk == D_INCONSISTENT || os.pdsk == D_CONSISTENT) &&
- real_peer_disk == D_UP_TO_DATE &&
- os.conn > C_CONNECTED && os.disk == D_UP_TO_DATE) {
- /* If we are (becoming) SyncSource, but peer is still in sync
- * preparation, ignore its uptodate-ness to avoid flapping, it
- * will change to inconsistent once the peer reaches active
- * syncing states.
- * It may have changed syncer-paused flags, however, so we
- * cannot ignore this completely. */
- if (peer_state.conn > C_CONNECTED &&
- peer_state.conn < C_SYNC_SOURCE)
- real_peer_disk = D_INCONSISTENT;
-
- /* if peer_state changes to connected at the same time,
- * it explicitly notifies us that it finished resync.
- * Maybe we should finish it up, too? */
- else if (os.conn >= C_SYNC_SOURCE &&
- peer_state.conn == C_CONNECTED) {
- if (drbd_bm_total_weight(device) <= device->rs_failed)
- drbd_resync_finished(peer_device);
- return 0;
+ if (peer_device->repl_state[NOW] == L_WF_BITMAP_S ||
+ peer_device->repl_state[NOW] == L_STARTING_SYNC_S) {
+ switch (pi->cmd) {
+ case P_RS_DATA_REQUEST:
+ case P_RS_DAGTAG_REQ:
+ case P_CSUM_RS_REQUEST:
+ case P_RS_CSUM_DAGTAG_REQ:
+ case P_RS_THIN_REQ:
+ case P_RS_THIN_DAGTAG_REQ:
+ drbd_send_ack_be(peer_device, P_RS_CANCEL, sector, size, p->block_id);
+ return ignore_remaining_packet(connection, pi->size);
+ default:
+ break;
}
}
- /* explicit verify finished notification, stop sector reached. */
- if (os.conn == C_VERIFY_T && os.disk == D_UP_TO_DATE &&
- peer_state.conn == C_CONNECTED && real_peer_disk == D_UP_TO_DATE) {
- ov_out_of_sync_print(peer_device);
- drbd_resync_finished(peer_device);
- return 0;
- }
+ min_d_state = pi->cmd == P_DATA_REQUEST ? D_UP_TO_DATE : D_OUTDATED;
+ if (!get_ldev_if_state(device, min_d_state)) {
+ switch (pi->cmd) {
+ case P_DATA_REQUEST:
+ drbd_send_ack_be(peer_device, P_NEG_DREPLY, sector, size, p->block_id);
+ break;
+ case P_OV_REQUEST:
+ case P_OV_DAGTAG_REQ:
+ drbd_verify_skipped_block(peer_device, sector, size);
+ verify_progress(peer_device, sector, size);
+ drbd_send_ack_be(peer_device, P_RS_CANCEL, sector, size, p->block_id);
+ break;
+ case P_RS_DATA_REQUEST:
+ case P_RS_DAGTAG_REQ:
+ case P_CSUM_RS_REQUEST:
+ case P_RS_CSUM_DAGTAG_REQ:
+ case P_RS_THIN_REQ:
+ case P_RS_THIN_DAGTAG_REQ:
+ if (peer_device->repl_state[NOW] == L_PAUSED_SYNC_S)
+ drbd_send_ack_be(peer_device, P_RS_CANCEL, sector, size, p->block_id);
+ else
+ drbd_send_ack_be(peer_device, P_NEG_RS_DREPLY, sector, size, p->block_id);
+ break;
+ default:
+ BUG();
+ }
- /* peer says his disk is inconsistent, while we think it is uptodate,
- * and this happens while the peer still thinks we have a sync going on,
- * but we think we are already done with the sync.
- * We ignore this to avoid flapping pdsk.
- * This should not happen, if the peer is a recent version of drbd. */
- if (os.pdsk == D_UP_TO_DATE && real_peer_disk == D_INCONSISTENT &&
- os.conn == C_CONNECTED && peer_state.conn > C_SYNC_SOURCE)
- real_peer_disk = D_UP_TO_DATE;
+ if (peer_device->repl_state[NOW] != L_PAUSED_SYNC_S)
+ drbd_err_ratelimit(device,
+ "Can not satisfy peer's read request, no local data.\n");
- if (ns.conn == C_WF_REPORT_PARAMS)
- ns.conn = C_CONNECTED;
+ /* drain possible payload */
+ return ignore_remaining_packet(connection, pi->size);
+ }
- if (peer_state.conn == C_AHEAD)
- ns.conn = C_BEHIND;
+ if (pi->cmd != P_DATA_REQUEST
+ && !IS_ALIGNED(size, bm_block_size(device->bitmap))
+ && (sector + (size >> 9) != device->bitmap->bm_dev_capacity)) {
+ drbd_warn_ratelimit(peer_device,
+ "Unaligned %s request (%u vs %u) at %llu; may lead to hung or repeating resync.\n",
+ drbd_packet_name(pi->cmd), size, bm_block_size(device->bitmap), sector);
+ /* For now, try to continue anyways */
+ }
- /* TODO:
- * if (primary and diskless and peer uuid != effective uuid)
- * abort attach on peer;
- *
- * If this node does not have good data, was already connected, but
- * the peer did a late attach only now, trying to "negotiate" with me,
- * AND I am currently Primary, possibly frozen, with some specific
- * "effective" uuid, this should never be reached, really, because
- * we first send the uuids, then the current state.
+ /* A sync source whose slot toward this peer still has
+ * MDF_PEER_DIVERGENCE_BITMAP set: Defer (retry) the resync request
+ * until the flag has been cleared.
*
- * In this scenario, we already dropped the connection hard
- * when we received the unsuitable uuids (receive_uuids().
- *
- * Should we want to change this, that is: not drop the connection in
- * receive_uuids() already, then we would need to add a branch here
- * that aborts the attach of "unsuitable uuids" on the peer in case
- * this node is currently Diskless Primary.
+ * Only toward a drbd-9 peer. A drbd-8.4 sync target treats P_RS_CANCEL
+ * as "this block is done" instead of rewinding to it, so the block would
+ * never be resynced.
*/
-
- if (device->p_uuid && peer_state.disk >= D_NEGOTIATING &&
- get_ldev_if_state(device, D_NEGOTIATING)) {
- int cr; /* consider resync */
-
- /* if we established a new connection */
- cr = (os.conn < C_CONNECTED);
- /* if we had an established connection
- * and one of the nodes newly attaches a disk */
- cr |= (os.conn == C_CONNECTED &&
- (peer_state.disk == D_NEGOTIATING ||
- os.disk == D_NEGOTIATING));
- /* if we have both been inconsistent, and the peer has been
- * forced to be UpToDate with --force */
- cr |= test_bit(CONSIDER_RESYNC, &device->flags);
- /* if we had been plain connected, and the admin requested to
- * start a sync by "invalidate" or "invalidate-remote" */
- cr |= (os.conn == C_CONNECTED &&
- (peer_state.conn >= C_STARTING_SYNC_S &&
- peer_state.conn <= C_WF_BITMAP_T));
-
- if (cr)
- ns.conn = drbd_sync_handshake(peer_device, peer_state.role, real_peer_disk);
-
- put_ldev(device);
- if (ns.conn == C_MASK) {
- ns.conn = C_CONNECTED;
- if (device->state.disk == D_NEGOTIATING) {
- drbd_force_state(device, NS(disk, D_FAILED));
- } else if (peer_state.disk == D_NEGOTIATING) {
- drbd_err(device, "Disk attach process on the peer node was aborted.\n");
- peer_state.disk = D_DISKLESS;
- real_peer_disk = D_DISKLESS;
- } else {
- if (test_and_clear_bit(CONN_DRY_RUN, &peer_device->connection->flags))
- return -EIO;
- D_ASSERT(device, os.conn == C_WF_REPORT_PARAMS);
- conn_request_state(peer_device->connection, NS(conn, C_DISCONNECTING), CS_HARD);
- return -EIO;
- }
+ if (repl_is_sync_source(peer_device->repl_state[NOW]) &&
+ connection->agreed_pro_version >= 110 &&
+ test_bit(__MDF_PEER_DIVERGENCE_BITMAP,
+ &device->ldev->md.peers[peer_device->node_id].flags)) {
+ switch (pi->cmd) {
+ case P_RS_DATA_REQUEST:
+ case P_RS_DAGTAG_REQ:
+ case P_CSUM_RS_REQUEST:
+ case P_RS_CSUM_DAGTAG_REQ:
+ case P_RS_THIN_REQ:
+ case P_RS_THIN_DAGTAG_REQ:
+ drbd_notice_ratelimit(peer_device,
+ "Deferring resync request %llus +%u, divergence bitmap not cleared yet\n",
+ (unsigned long long)sector, size);
+ drbd_send_ack_be(peer_device, P_RS_CANCEL, sector, size, p->block_id);
+ put_ldev(device);
+ return ignore_remaining_packet(connection, pi->size);
+ default:
+ break;
}
}
- spin_lock_irq(&device->resource->req_lock);
- if (os.i != drbd_read_state(device).i)
- goto retry;
- clear_bit(CONSIDER_RESYNC, &device->flags);
- ns.peer = peer_state.role;
- ns.pdsk = real_peer_disk;
- ns.peer_isp = (peer_state.aftr_isp | peer_state.user_isp);
- if ((ns.conn == C_CONNECTED || ns.conn == C_WF_BITMAP_S) && ns.disk == D_NEGOTIATING)
- ns.disk = device->new_state_tmp.disk;
- cs_flags = CS_VERBOSE + (os.conn < C_CONNECTED && ns.conn >= C_CONNECTED ? 0 : CS_HARD);
- if (ns.pdsk == D_CONSISTENT && drbd_suspended(device) && ns.conn == C_CONNECTED && os.conn < C_CONNECTED &&
- test_bit(NEW_CUR_UUID, &device->flags)) {
- /* Do not allow tl_restart(RESEND) for a rebooted peer. We can only allow this
- for temporal network outages! */
- spin_unlock_irq(&device->resource->req_lock);
- drbd_err(device, "Aborting Connect, can not thaw IO with an only Consistent peer\n");
- tl_clear(peer_device->connection);
- drbd_uuid_new_current(device);
- clear_bit(NEW_CUR_UUID, &device->flags);
- conn_request_state(peer_device->connection, NS2(conn, C_PROTOCOL_ERROR, susp, 0), CS_HARD);
- return -EIO;
- }
- rv = _drbd_set_state(device, ns, cs_flags, NULL);
- ns = drbd_read_state(device);
- spin_unlock_irq(&device->resource->req_lock);
+ inc_unacked(peer_device);
- if (rv < SS_SUCCESS) {
- conn_request_state(peer_device->connection, NS(conn, C_DISCONNECTING), CS_HARD);
- return -EIO;
+ peer_req = drbd_alloc_peer_req(peer_device, GFP_TRY, size, REQ_OP_READ);
+ err = -ENOMEM;
+ if (!peer_req)
+ goto fail;
+ peer_req->i.size = size;
+ peer_req->i.sector = sector;
+ peer_req->block_id = p->block_id;
+ peer_req->depend_dagtag_node_id = depend_dagtag_node_id;
+ peer_req->depend_dagtag = depend_dagtag;
+ /* no longer valid, about to call drbd_recv again for the digest... */
+ p = NULL;
+ pi->data = NULL;
+
+ if (peer_device->repl_state[NOW] == L_AHEAD) {
+ if (pi->cmd == P_DATA_REQUEST) {
+ /* P_DATA_REQUEST originates from a Primary,
+ * so if I am "Ahead", the Primary would be "Behind":
+ * Can not happen. */
+ drbd_err_ratelimit(peer_device, "received P_DATA_REQUEST while L_AHEAD\n");
+ err = -EINVAL;
+ goto fail2;
+ }
+ if (connection->agreed_pro_version >= 115) {
+ switch (pi->cmd) {
+ /* case P_DATA_REQUEST: see above, not based on protocol version */
+ case P_OV_REQUEST:
+ case P_OV_DAGTAG_REQ:
+ drbd_verify_skipped_block(peer_device, sector, size);
+ verify_progress(peer_device, sector, size);
+ fallthrough;
+ case P_RS_DATA_REQUEST:
+ case P_RS_DAGTAG_REQ:
+ case P_CSUM_RS_REQUEST:
+ case P_RS_CSUM_DAGTAG_REQ:
+ case P_RS_THIN_REQ:
+ case P_RS_THIN_DAGTAG_REQ:
+ err = drbd_send_ack(peer_device, P_RS_CANCEL_AHEAD, peer_req);
+ goto fail2;
+ default:
+ BUG();
+ }
+ }
}
- if (os.conn > C_WF_REPORT_PARAMS) {
- if (ns.conn > C_CONNECTED && peer_state.conn <= C_CONNECTED &&
- peer_state.disk != D_NEGOTIATING ) {
- /* we want resync, peer has not yet decided to sync... */
- /* Nowadays only used when forcing a node into primary role and
- setting its disk to UpToDate with that */
- drbd_send_uuids(peer_device);
- drbd_send_current_state(peer_device);
+ switch (pi->cmd) {
+ case P_DATA_REQUEST:
+ peer_req->w.cb = w_e_end_data_req;
+ peer_req->i.type = INTERVAL_PEER_READ;
+ goto submit;
+
+ case P_RS_THIN_REQ:
+ case P_RS_THIN_DAGTAG_REQ:
+ /* If at some point in the future we have a smart way to
+ find out if this data block is completely deallocated,
+ then we would do something smarter here than reading
+ the block... */
+ peer_req->flags |= EE_RS_THIN_REQ;
+ fallthrough;
+ case P_RS_DATA_REQUEST:
+ case P_RS_DAGTAG_REQ:
+ peer_req->i.type = INTERVAL_RESYNC_READ;
+ peer_req->w.cb = w_e_end_rsdata_req;
+ break;
+
+ case P_CSUM_RS_REQUEST:
+ case P_RS_CSUM_DAGTAG_REQ:
+ D_ASSERT(device, connection->agreed_pro_version >= 89);
+ peer_req->i.type = INTERVAL_RESYNC_READ;
+
+ err = receive_digest(peer_req, pi->size);
+ if (err)
+ goto fail2;
+
+ peer_req->w.cb = w_e_end_rsdata_req;
+ /* remember to report stats in drbd_resync_finished */
+ peer_device->use_csums = true;
+ break;
+
+ case P_OV_REQUEST:
+ case P_OV_DAGTAG_REQ:
+ peer_req->i.type = INTERVAL_OV_READ_TARGET;
+ peer_device->ov_position = sector;
+ if (peer_device->ov_start_sector == ~(sector_t)0 &&
+ connection->agreed_pro_version >= 90) {
+ unsigned long now = jiffies;
+ int i;
+ unsigned long ov_left = drbd_bm_bits(device)
+ - bm_sect_to_bit(device->bitmap, sector);
+ atomic64_set(&peer_device->ov_left, ov_left);
+ peer_device->ov_start_sector = sector;
+ peer_device->ov_skipped = 0;
+ peer_device->rs_total = ov_left;
+ peer_device->rs_last_writeout = now;
+ peer_device->rs_last_progress_report_ts = now;
+ for (i = 0; i < DRBD_SYNC_MARKS; i++) {
+ peer_device->rs_mark_left[i] = ov_left;
+ peer_device->rs_mark_time[i] = now;
+ }
+ drbd_info(device, "Online Verify start sector: %llu\n",
+ (unsigned long long)sector);
}
+ peer_req->w.cb = w_e_end_ov_req;
+ break;
+
+ default:
+ BUG();
}
- clear_bit(DISCARD_MY_DATA, &device->flags);
+submit:
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_add_tail(&peer_req->recv_order, &connection->peer_reads);
+ peer_req->flags |= EE_ON_RECV_ORDER;
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ if (pi->cmd == P_DATA_REQUEST) {
+ atomic_inc(&connection->backing_ee_cnt);
+ drbd_conflict_submit_peer_read(peer_req);
+ } else {
+ drbd_peer_resync_read(peer_req);
+ }
+ /* ldev_ref_transfer: put_ldev in peer_req endio */
+ return 0;
+fail2:
+ drbd_free_peer_req(peer_req);
+fail:
+ dec_unacked(peer_device);
+ put_ldev(device);
+ return err;
+}
- drbd_md_sync(device); /* update connected indicator, la_size_sect, ... */
+static int receive_data_request(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct p_block_req *p_block_req = pi->data;
- return 0;
+ return receive_common_data_request(connection, pi,
+ &p_block_req->req_common,
+ 0, 0);
}
-static int receive_sync_uuid(struct drbd_connection *connection, struct packet_info *pi)
+/* receive_dagtag_data_request() - handle a request for data with dagtag
+ * dependency initiated by the peer
+ *
+ * Request handling flow:
+ *
+ * receive_dagtag_data_request
+ * |
+ * V
+ * receive_common_data_request
+ * |
+ * v dagtag waiting
+ * drbd_peer_resync_read --------------+
+ * | |
+ * | ... dagtag_wait_ee
+ * | |
+ * | v
+ * +--------------- release_dagtag_wait
+ * |
+ * v conflict (resync only)
+ * drbd_conflict_submit_peer_read -----+
+ * | ^ |
+ * | | ...
+ * | | |
+ * | | v
+ * | +---- drbd_do_submit_conflict
+ * v
+ * drbd_submit_peer_request
+ * |
+ * ... backing device
+ * |
+ * v
+ * drbd_peer_request_endio
+ * |
+ * v online verify request
+ * drbd_endio_read_sec_final ------------------+
+ * | |
+ * ... sender_work ... sender_work
+ * | |
+ * v v
+ * w_e_end_rsdata_req w_e_end_ov_req
+ * | |
+ * ... via peer ... via peer
+ * | |
+ * v v
+ * got_RSWriteAck got_OVResult
+ */
+static int receive_dagtag_data_request(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct p_rs_req *p_rs_req = pi->data;
+
+ return receive_common_data_request(connection, pi,
+ &p_rs_req->req_common,
+ be32_to_cpu(p_rs_req->dagtag_node_id), be64_to_cpu(p_rs_req->dagtag));
+}
+
+static int receive_common_ov_reply(struct drbd_connection *connection, struct packet_info *pi,
+ struct p_block_req_common *p,
+ unsigned int depend_dagtag_node_id, u64 depend_dagtag)
{
struct drbd_peer_device *peer_device;
struct drbd_device *device;
- struct p_rs_uuid *p = pi->data;
+ sector_t sector = be64_to_cpu(p->sector);
+ struct drbd_peer_request *peer_req;
+ int size = be32_to_cpu(p->blksize);
+ int err;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
return -EIO;
device = peer_device->device;
- wait_event(device->misc_wait,
- device->state.conn == C_WF_SYNC_UUID ||
- device->state.conn == C_BEHIND ||
- device->state.conn < C_CONNECTED ||
- device->state.disk < D_NEGOTIATING);
-
- /* D_ASSERT(device, device->state.conn == C_WF_SYNC_UUID ); */
+ peer_req = find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_OV_READ_SOURCE),
+ sector, size, p->block_id);
+ if (!peer_req)
+ return -EIO;
- /* Here the _drbd_uuid_ functions are right, current should
- _not_ be rotated into the history */
- if (get_ldev_if_state(device, D_NEGOTIATING)) {
- _drbd_uuid_set(device, UI_CURRENT, be64_to_cpu(p->uuid));
- _drbd_uuid_set(device, UI_BITMAP, 0UL);
+ dec_rs_pending(peer_device);
- drbd_print_uuids(device, "updated sync uuid");
- drbd_start_resync(device, C_SYNC_TARGET);
+ if (!get_ldev_if_state(device, D_OUTDATED)) {
+ drbd_peer_resync_read_cancel(peer_req);
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
- put_ldev(device);
- } else
- drbd_err(device, "Ignoring SyncUUID packet!\n");
+ /* drain payload */
+ return ignore_remaining_packet(connection, pi->size);
+ }
- return 0;
-}
+ err = receive_digest(peer_req, pi->size);
+ if (err)
+ goto fail;
-/*
- * receive_bitmap_plain
- *
- * Return 0 when done, 1 when another iteration is needed, and a negative error
- * code upon failure.
- */
-static int
-receive_bitmap_plain(struct drbd_peer_device *peer_device, unsigned int size,
- unsigned long *p, struct bm_xfer_ctx *c)
-{
- unsigned int data_size = DRBD_SOCKET_BUFFER_SIZE -
- drbd_header_size(peer_device->connection);
- unsigned int num_words = min_t(size_t, data_size / sizeof(*p),
- c->bm_words - c->word_offset);
- unsigned int want = num_words * sizeof(*p);
- int err;
+ set_bit(INTERVAL_RECEIVED, &peer_req->i.flags);
- if (want != size) {
- drbd_err(peer_device, "%s:want (%u) != size (%u)\n", __func__, want, size);
- return -EIO;
- }
- if (want == 0)
- return 0;
- err = drbd_recv_all(peer_device->connection, p, want);
+ err = peer_req_alloc_bio(peer_req, size, GFP_NOIO, REQ_OP_READ);
if (err)
- return err;
+ goto fail;
- drbd_bm_merge_lel(peer_device->device, c->word_offset, num_words, p);
+ inc_unacked(peer_device);
- c->word_offset += num_words;
- c->bit_offset = c->word_offset * BITS_PER_LONG;
- if (c->bit_offset > c->bm_bits)
- c->bit_offset = c->bm_bits;
+ peer_req->depend_dagtag_node_id = depend_dagtag_node_id;
+ peer_req->depend_dagtag = depend_dagtag;
+ peer_req->w.cb = w_e_end_ov_reply;
- return 1;
+ /* track progress, we may need to throttle */
+ rs_sectors_came_in(peer_device, size);
+
+ drbd_peer_resync_read(peer_req);
+ /* ldev_ref_transfer: put_ldev in peer_req endio */
+ return 0;
+fail:
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ put_ldev(device);
+ return err;
}
-static enum drbd_bitmap_code dcbp_get_code(struct p_compressed_bm *p)
+static int receive_ov_reply(struct drbd_connection *connection, struct packet_info *pi)
{
- return (enum drbd_bitmap_code)(p->encoding & 0x0f);
+ struct p_block_req *p_block_req = pi->data;
+
+ return receive_common_ov_reply(connection, pi,
+ &p_block_req->req_common,
+ 0, 0);
}
-static int dcbp_get_start(struct p_compressed_bm *p)
+static int receive_dagtag_ov_reply(struct drbd_connection *connection, struct packet_info *pi)
{
- return (p->encoding & 0x80) != 0;
+ struct p_rs_req *p_rs_req = pi->data;
+
+ return receive_common_ov_reply(connection, pi,
+ &p_rs_req->req_common,
+ be32_to_cpu(p_rs_req->dagtag_node_id), be64_to_cpu(p_rs_req->dagtag));
}
-static int dcbp_get_pad_bits(struct p_compressed_bm *p)
+/* Answer a flush round of @initiator_connection on behalf of @primary_node_id */
+static void queue_flush_ack(struct drbd_connection *initiator_connection,
+ int primary_node_id, u64 flush_sequence)
{
- return (p->encoding >> 4) & 0x7;
+ unsigned long irq_flags;
+
+ spin_lock_irqsave(&initiator_connection->flush_ack_lock, irq_flags);
+ initiator_connection->flush_ack_sequence[primary_node_id] = flush_sequence;
+ drbd_queue_work_if_unqueued(&initiator_connection->sender_work,
+ &initiator_connection->flush_ack_work);
+ spin_unlock_irqrestore(&initiator_connection->flush_ack_lock, irq_flags);
}
/*
- * recv_bm_rle_bits
- *
- * Return 0 when done, 1 when another iteration is needed, and a negative error
- * code upon failure.
+ * No peer acks come from this peer any more, and the out-of-sync bits from
+ * its unacked writes are queued: a flush answer on its behalf may go out.
+ * Not while a connect to it is prepared: that connection may carry writes
+ * from before the Primary's flush point. Caller holds state_rwlock.
*/
-static int
-recv_bm_rle_bits(struct drbd_peer_device *peer_device,
- struct p_compressed_bm *p,
- struct bm_xfer_ctx *c,
- unsigned int len)
+static bool flush_ack_on_behalf_allowed(struct drbd_connection *connection)
{
- struct bitstream bs;
- u64 look_ahead;
- u64 rl;
- u64 tmp;
- unsigned long s = c->bit_offset;
- unsigned long e;
- int toggle = dcbp_get_start(p);
- int have;
- int bits;
+ struct drbd_resource *resource = connection->resource;
+ enum drbd_conn_state cstate = connection->cstate[NOW];
- bitstream_init(&bs, p->code, len, dcbp_get_pad_bits(p));
+ if (cstate != C_STANDALONE && cstate != C_UNCONNECTED && cstate != C_CONNECTING)
+ return false;
- bits = bitstream_get_bits(&bs, &look_ahead, 64);
- if (bits < 0)
- return -EIO;
+ return !(resource->remote_state_change && resource->twopc_reply.is_connect &&
+ drbd_twopc_between_peer_and_me(connection));
+}
- for (have = bits; have > 0; s += rl, toggle = !toggle) {
- bits = vli_decode_bits(&rl, look_ahead);
- if (bits <= 0)
- return -EIO;
+static int receive_flush_requests(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_connection *other_connection;
+ struct p_flush_requests *p_flush_requests = pi->data;
+ u64 flush_requests_dagtag;
+ u64 flush_sequence = be64_to_cpu(p_flush_requests->flush_sequence);
- if (toggle) {
- e = s + rl -1;
- if (e >= c->bm_bits) {
- drbd_err(peer_device, "bitmap overflow (e:%lu) while decoding bm RLE packet\n", e);
- return -EIO;
- }
- _drbd_bm_set_bits(peer_device->device, s, e);
- }
+ spin_lock_irq(&resource->tl_update_lock);
+ /*
+ * If the current dagtag was read from the metadata then there is no
+ * associated request. Hence there is nothing to flush. Flush up to the
+ * preceding dagtag instead.
+ */
+ if (resource->dagtag_sector == resource->dagtag_from_backing_dev)
+ flush_requests_dagtag = resource->dagtag_before_attach;
+ else
+ flush_requests_dagtag = resource->dagtag_sector;
+ spin_unlock_irq(&resource->tl_update_lock);
- if (have < bits) {
- drbd_err(peer_device, "bitmap decoding error: h:%d b:%d la:0x%08llx l:%u/%u\n",
- have, bits, look_ahead,
- (unsigned int)(bs.cur.b - p->code),
- (unsigned int)bs.buf_len);
- return -EIO;
- }
- /* if we consumed all 64 bits, assign 0; >> 64 is "undefined"; */
- if (likely(bits < 64))
- look_ahead >>= bits;
- else
- look_ahead = 0;
- have -= bits;
+ spin_lock_irq(&connection->primary_flush_lock);
+ connection->flush_requests_dagtag = flush_requests_dagtag;
+ connection->flush_sequence = flush_sequence;
+ connection->flush_forward_sent_mask = 0;
+ spin_unlock_irq(&connection->primary_flush_lock);
- bits = bitstream_get_bits(&bs, &tmp, 64 - have);
- if (bits < 0)
- return -EIO;
- look_ahead |= tmp << have;
- have += bits;
+ /* Queue any request waiting for peer ack to be sent */
+ drbd_flush_peer_acks(resource);
+
+ /* For each peer, check if peer ack for this dagtag has already been sent */
+ read_lock_irq(&resource->state_rwlock);
+ rcu_read_lock();
+ for_each_connection_rcu(other_connection, resource) {
+ if (other_connection->cstate[NOW] == C_CONNECTED)
+ queue_work(other_connection->ack_sender, &other_connection->peer_ack_work);
+ else if (flush_ack_on_behalf_allowed(other_connection))
+ queue_flush_ack(connection, other_connection->peer_node_id, flush_sequence);
}
+ rcu_read_unlock();
+ read_unlock_irq(&resource->state_rwlock);
- c->bit_offset = s;
- bm_xfer_ctx_bit_to_word_offset(c);
+ return 0;
+}
+
+static int receive_flush_requests_ack(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_connection *primary_connection;
+ struct p_flush_ack *p_flush_ack = pi->data;
+ u64 flush_sequence = be64_to_cpu(p_flush_ack->flush_sequence);
+ int primary_node_id = be32_to_cpu(p_flush_ack->primary_node_id);
+
+ spin_lock_irq(&resource->initiator_flush_lock);
+ if (flush_sequence < resource->current_flush_sequence) {
+ spin_unlock_irq(&resource->initiator_flush_lock);
+ return 0;
+ }
- return (s != c->bm_bits);
+ rcu_read_lock();
+ primary_connection = drbd_connection_by_node_id(resource, primary_node_id);
+ if (primary_connection)
+ primary_connection->pending_flush_mask &= ~NODE_MASK(connection->peer_node_id);
+ rcu_read_unlock();
+ spin_unlock_irq(&resource->initiator_flush_lock);
+ return 0;
}
/*
- * decode_bitmap_c
+ * config_unknown_volume - device configuration command for unknown volume
*
- * Return 0 when done, 1 when another iteration is needed, and a negative error
- * code upon failure.
+ * When a device is added to an existing connection, the node on which the
+ * device is added first will send configuration commands to its peer but the
+ * peer will not know about the device yet. It will warn and ignore these
+ * commands. Once the device is added on the second node, the second node will
+ * send the same device configuration commands, but in the other direction.
+ *
+ * (We can also end up here if drbd is misconfigured.)
*/
-static int
-decode_bitmap_c(struct drbd_peer_device *peer_device,
- struct p_compressed_bm *p,
- struct bm_xfer_ctx *c,
- unsigned int len)
+static int config_unknown_volume(struct drbd_connection *connection, struct packet_info *pi)
{
- if (dcbp_get_code(p) == RLE_VLI_Bits)
- return recv_bm_rle_bits(peer_device, p, c, len - sizeof(*p));
+ drbd_warn(connection, "%s packet received for volume %d, which is not configured locally\n",
+ drbd_packet_name(pi->cmd), pi->vnr);
+ return ignore_remaining_packet(connection, pi->size);
+}
- /* other variants had been implemented for evaluation,
- * but have been dropped as this one turned out to be "best"
- * during all our tests. */
+static int receive_enable_replication_next(struct drbd_connection *connection,
+ struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct p_enable_replication *p_enable_replication = pi->data;
- drbd_err(peer_device, "receive_bitmap_c: unknown encoding %u\n", p->encoding);
- conn_request_state(peer_device->connection, NS(conn, C_PROTOCOL_ERROR), CS_HARD);
- return -EIO;
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return config_unknown_volume(connection, pi);
+
+ assign_bit(REPLICATION_NEXT, peer_device->flags,
+ p_enable_replication->enable);
+
+ return 0;
}
-void INFO_bm_xfer_stats(struct drbd_peer_device *peer_device,
- const char *direction, struct bm_xfer_ctx *c)
+static int receive_enable_replication(struct drbd_connection *connection, struct packet_info *pi)
{
- /* what would it take to transfer it "plaintext" */
- unsigned int header_size = drbd_header_size(peer_device->connection);
- unsigned int data_size = DRBD_SOCKET_BUFFER_SIZE - header_size;
- unsigned int plain =
- header_size * (DIV_ROUND_UP(c->bm_words, data_size) + 1) +
- c->bm_words * sizeof(unsigned long);
- unsigned int total = c->bytes[0] + c->bytes[1];
- unsigned int r;
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ struct p_enable_replication *p_enable_replication = pi->data;
+ unsigned long irq_flags;
- /* total can not be zero. but just in case: */
- if (total == 0)
- return;
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
- /* don't report if not compressed */
- if (total >= plain)
- return;
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ peer_device->replication[NEW] = p_enable_replication->enable;
+ end_state_change(resource, &irq_flags, "enable-replication");
+ return 0;
+}
- /* total < plain. check for overflow, still */
- r = (total > UINT_MAX/1000) ? (total / (plain/1000))
- : (1000 * total / plain);
+/*
+ * drbd_asb_recover_0p - Recover after split-brain with no remaining primaries
+ */
+static enum sync_strategy drbd_asb_recover_0p(struct drbd_peer_device *peer_device)
+{
+ const int node_id = peer_device->device->resource->res_opts.node_id;
+ int self, peer;
+ enum sync_strategy rv = SPLIT_BRAIN_DISCONNECT;
+ unsigned long ch_self, ch_peer;
+ enum drbd_after_sb_p after_sb_0p;
- if (r > 1000)
- r = 1000;
+ self = drbd_bitmap_uuid(peer_device) & UUID_PRIMARY;
+ peer = peer_device->bitmap_uuids[node_id] & UUID_PRIMARY;
- r = 1000 - r;
- drbd_info(peer_device, "%s bitmap stats [Bytes(packets)]: plain %u(%u), RLE %u(%u), "
- "total %u; compression: %u.%u%%\n",
- direction,
- c->bytes[1], c->packets[1],
- c->bytes[0], c->packets[0],
- total, r/10, r % 10);
+ ch_peer = peer_device->dirty_bits;
+ ch_self = peer_device->comm_bm_set;
+
+ rcu_read_lock();
+ after_sb_0p = rcu_dereference(peer_device->connection->transport.net_conf)->after_sb_0p;
+ rcu_read_unlock();
+ switch (after_sb_0p) {
+ case ASB_CONSENSUS:
+ case ASB_DISCARD_SECONDARY:
+ case ASB_CALL_HELPER:
+ case ASB_VIOLENTLY:
+ case ASB_RETRY_CONNECT:
+ case ASB_AUTO_DISCARD:
+ drbd_err(peer_device, "Configuration error.\n");
+ break;
+ case ASB_DISCONNECT:
+ break;
+ case ASB_DISCARD_YOUNGER_PRI:
+ if (self == 0 && peer == 1) {
+ rv = SYNC_TARGET_USE_BITMAP;
+ break;
+ }
+ if (self == 1 && peer == 0) {
+ rv = SYNC_SOURCE_USE_BITMAP;
+ break;
+ }
+ fallthrough; /* to one of the other strategies */
+ case ASB_DISCARD_OLDER_PRI:
+ if (self == 0 && peer == 1) {
+ rv = SYNC_SOURCE_USE_BITMAP;
+ break;
+ }
+ if (self == 1 && peer == 0) {
+ rv = SYNC_TARGET_USE_BITMAP;
+ break;
+ }
+ drbd_warn(peer_device, "Discard younger/older primary did not find a decision\n"
+ "Using discard-least-changes instead\n");
+ fallthrough;
+ case ASB_DISCARD_ZERO_CHG:
+ if (ch_peer == 0 && ch_self == 0) {
+ rv = test_bit(RESOLVE_CONFLICTS, &peer_device->connection->transport.flags)
+ ? SYNC_TARGET_USE_BITMAP : SYNC_SOURCE_USE_BITMAP;
+ break;
+ } else {
+ if (ch_peer == 0) {
+ rv = SYNC_SOURCE_USE_BITMAP;
+ break;
+ }
+ if (ch_self == 0) {
+ rv = SYNC_TARGET_USE_BITMAP;
+ break;
+ }
+ }
+ if (after_sb_0p == ASB_DISCARD_ZERO_CHG)
+ break;
+ fallthrough;
+ case ASB_DISCARD_LEAST_CHG:
+ if (ch_self < ch_peer)
+ rv = SYNC_TARGET_USE_BITMAP;
+ else if (ch_self > ch_peer)
+ rv = SYNC_SOURCE_USE_BITMAP;
+ else /* ( ch_self == ch_peer ) */
+ /* Well, then use something else. */
+ rv = test_bit(RESOLVE_CONFLICTS, &peer_device->connection->transport.flags)
+ ? SYNC_TARGET_USE_BITMAP : SYNC_SOURCE_USE_BITMAP;
+ break;
+ case ASB_DISCARD_LOCAL:
+ rv = SYNC_TARGET_USE_BITMAP;
+ break;
+ case ASB_DISCARD_REMOTE:
+ rv = SYNC_SOURCE_USE_BITMAP;
+ }
+
+ return rv;
}
-/* Since we are processing the bitfield from lower addresses to higher,
- it does not matter if the process it in 32 bit chunks or 64 bit
- chunks as long as it is little endian. (Understand it as byte stream,
- beginning with the lowest byte...) If we would use big endian
- we would need to process it from the highest address to the lowest,
- in order to be agnostic to the 32 vs 64 bits issue.
+/*
+ * drbd_asb_recover_1p - Recover after split-brain with one remaining primary
+ */
+static enum sync_strategy drbd_asb_recover_1p(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_resource *resource = device->resource;
+ enum sync_strategy strategy, rv = SPLIT_BRAIN_DISCONNECT;
+ enum drbd_after_sb_p after_sb_1p;
- returns 0 on failure, 1 if we successfully received it. */
-static int receive_bitmap(struct drbd_connection *connection, struct packet_info *pi)
+ rcu_read_lock();
+ after_sb_1p = rcu_dereference(connection->transport.net_conf)->after_sb_1p;
+ rcu_read_unlock();
+ switch (after_sb_1p) {
+ case ASB_DISCARD_YOUNGER_PRI:
+ case ASB_DISCARD_OLDER_PRI:
+ case ASB_DISCARD_LEAST_CHG:
+ case ASB_DISCARD_LOCAL:
+ case ASB_DISCARD_REMOTE:
+ case ASB_DISCARD_ZERO_CHG:
+ case ASB_RETRY_CONNECT:
+ case ASB_AUTO_DISCARD:
+ drbd_err(device, "Configuration error.\n");
+ break;
+ case ASB_DISCONNECT:
+ break;
+ case ASB_CONSENSUS:
+ strategy = drbd_asb_recover_0p(peer_device);
+ if (strategy == SYNC_TARGET_USE_BITMAP && resource->role[NOW] == R_SECONDARY)
+ rv = strategy;
+ if (strategy == SYNC_SOURCE_USE_BITMAP && resource->role[NOW] == R_PRIMARY)
+ rv = strategy;
+ break;
+ case ASB_VIOLENTLY:
+ rv = drbd_asb_recover_0p(peer_device);
+ break;
+ case ASB_DISCARD_SECONDARY:
+ return resource->role[NOW] == R_PRIMARY ? SYNC_SOURCE_USE_BITMAP : SYNC_TARGET_USE_BITMAP;
+ case ASB_CALL_HELPER:
+ strategy = drbd_asb_recover_0p(peer_device);
+ if (strategy == SYNC_TARGET_USE_BITMAP && resource->role[NOW] == R_PRIMARY) {
+ /* Force-demote on the retry-connect path: a role change
+ * is a cluster-wide two-phase commit, barred here.
+ */
+ set_bit(CONN_HANDSHAKE_DEMOTE, &connection->flags);
+ rv = RETRY_CONNECT;
+ } else
+ rv = strategy;
+ }
+
+ return rv;
+}
+
+/*
+ * drbd_asb_recover_2p - Recover after split-brain with two remaining primaries
+ */
+static enum sync_strategy drbd_asb_recover_2p(struct drbd_peer_device *peer_device)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct bm_xfer_ctx c;
- int err;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ enum sync_strategy strategy, rv = SPLIT_BRAIN_DISCONNECT;
+ enum drbd_after_sb_p after_sb_2p;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ rcu_read_lock();
+ after_sb_2p = rcu_dereference(connection->transport.net_conf)->after_sb_2p;
+ rcu_read_unlock();
+ switch (after_sb_2p) {
+ case ASB_DISCARD_YOUNGER_PRI:
+ case ASB_DISCARD_OLDER_PRI:
+ case ASB_DISCARD_LEAST_CHG:
+ case ASB_DISCARD_LOCAL:
+ case ASB_DISCARD_REMOTE:
+ case ASB_CONSENSUS:
+ case ASB_DISCARD_SECONDARY:
+ case ASB_DISCARD_ZERO_CHG:
+ case ASB_RETRY_CONNECT:
+ case ASB_AUTO_DISCARD:
+ drbd_err(device, "Configuration error.\n");
+ break;
+ case ASB_VIOLENTLY:
+ rv = drbd_asb_recover_0p(peer_device);
+ break;
+ case ASB_DISCONNECT:
+ break;
+ case ASB_CALL_HELPER:
+ strategy = drbd_asb_recover_0p(peer_device);
+ if (strategy == SYNC_TARGET_USE_BITMAP) {
+ /* Force-demote on the retry-connect path: a role change
+ * is a cluster-wide two-phase commit, barred here.
+ */
+ set_bit(CONN_HANDSHAKE_DEMOTE, &connection->flags);
+ rv = RETRY_CONNECT;
+ } else
+ rv = strategy;
+ }
- drbd_bm_lock(device, "receive bitmap", BM_LOCKED_SET_ALLOWED);
- /* you are supposed to send additional out-of-sync information
- * if you actually set bits during this phase */
+ return rv;
+}
- c = (struct bm_xfer_ctx) {
- .bm_bits = drbd_bm_bits(device),
- .bm_words = drbd_bm_words(device),
- };
+static void drbd_uuid_dump_self(struct drbd_peer_device *peer_device, u64 bits, u64 flags)
+{
+ struct drbd_device *device = peer_device->device;
- for(;;) {
- if (pi->cmd == P_BITMAP)
- err = receive_bitmap_plain(peer_device, pi->size, pi->data, &c);
- else if (pi->cmd == P_COMPRESSED_BITMAP) {
- /* MAYBE: sanity check that we speak proto >= 90,
- * and the feature is enabled! */
- struct p_compressed_bm *p = pi->data;
+ drbd_info(peer_device, "self %016llX:%016llX:%016llX:%016llX bits:%llu flags:%llX\n",
+ (unsigned long long)drbd_resolved_uuid(peer_device, NULL),
+ (unsigned long long)drbd_bitmap_uuid(peer_device),
+ (unsigned long long)drbd_history_uuid(device, 0),
+ (unsigned long long)drbd_history_uuid(device, 1),
+ (unsigned long long)bits,
+ (unsigned long long)flags);
+}
- if (pi->size > DRBD_SOCKET_BUFFER_SIZE - drbd_header_size(connection)) {
- drbd_err(device, "ReportCBitmap packet too large\n");
- err = -EIO;
- goto out;
- }
- if (pi->size <= sizeof(*p)) {
- drbd_err(device, "ReportCBitmap packet too small (l:%u)\n", pi->size);
- err = -EIO;
- goto out;
- }
- err = drbd_recv_all(peer_device->connection, p, pi->size);
- if (err)
- goto out;
- err = decode_bitmap_c(peer_device, p, &c, pi->size);
+
+static void drbd_uuid_dump_peer(struct drbd_peer_device *peer_device, u64 bits, u64 flags)
+{
+ const int node_id = peer_device->device->resource->res_opts.node_id;
+
+ drbd_info(peer_device, "peer %016llX:%016llX:%016llX:%016llX bits:%llu flags:%llX\n",
+ (unsigned long long)peer_device->current_uuid,
+ (unsigned long long)peer_device->bitmap_uuids[node_id],
+ (unsigned long long)peer_device->history_uuids[0],
+ (unsigned long long)peer_device->history_uuids[1],
+ (unsigned long long)bits,
+ (unsigned long long)flags);
+}
+
+/* find the peer's bitmap slot for the given UUID, if they have one */
+static int drbd_find_peer_bitmap_by_uuid(struct drbd_peer_device *peer_device, u64 uuid)
+{
+ u64 peer;
+ int i;
+
+ for (i = 0; i < DRBD_PEERS_MAX; i++) {
+ peer = peer_device->bitmap_uuids[i] & ~UUID_PRIMARY;
+ if (uuid == peer)
+ return i;
+ }
+
+ return -1;
+}
+
+/* Find our bitmap slot for the given UUID, if we have one. Prefer a slot whose
+ * bitmap is safe to copy from -- a divergence bitmap -- over a convergence
+ * bitmap being cleared by an ongoing resync. Fall back to a convergence slot so
+ * that callers can still tell "UUID unknown" (-1) from "UUID known but only as
+ * a convergence bitmap"; callers that must copy re-check is_divergence_bitmap()
+ * on the result.
+ *
+ * Of the slots without a bitmap and without a known node, consider only the
+ * day0 slot _drbd_send_uuids110() last sent to this peer, if any. The peer
+ * sees only that one, so a match on another one would be a decision it does
+ * not share. Which slot that is, and whether one is sent at all, can change
+ * between sending and the handshake, so take what was sent.
+ */
+static int drbd_find_bitmap_by_uuid(struct drbd_peer_device *peer_device, u64 uuid)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int day0 = READ_ONCE(peer_device->comm_day0_node_id);
+ int i, any = -1;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ if (i == device->ldev->md.node_id)
+ continue;
+ if (connection->agreed_pro_version < 116 &&
+ peer_md[i].bitmap_index == -1)
+ continue;
+ if (i != day0 && i != peer_device->node_id &&
+ !test_bit(__MDF_HAVE_BITMAP, &peer_md[i].flags) &&
+ !test_bit(__MDF_NODE_EXISTS, &peer_md[i].flags))
+ continue;
+ if ((peer_md[i].bitmap_uuid & ~UUID_PRIMARY) != uuid)
+ continue;
+ if (is_divergence_bitmap(&peer_md[i]))
+ return i;
+ if (any == -1)
+ any = i;
+ }
+
+ return any;
+}
+
+static enum sync_strategy
+uuid_fixup_resync_end(struct drbd_peer_device *peer_device, enum sync_rule *rule)
+{
+ struct drbd_device *device = peer_device->device;
+ const int node_id = device->resource->res_opts.node_id;
+
+ if (peer_device->bitmap_uuids[node_id] == (u64)0 && drbd_bitmap_uuid(peer_device) != (u64)0) {
+
+ if (peer_device->connection->agreed_pro_version < 91)
+ return REQUIRES_PROTO_91;
+
+ if ((drbd_bitmap_uuid(peer_device) & ~UUID_PRIMARY) ==
+ (peer_device->history_uuids[0] & ~UUID_PRIMARY) &&
+ (drbd_history_uuid(device, 0) & ~UUID_PRIMARY) ==
+ (peer_device->history_uuids[0] & ~UUID_PRIMARY)) {
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[peer_device->node_id];
+ u64 previous_bitmap_uuid = peer_md->bitmap_uuid;
+
+ drbd_info(device, "was SyncSource, missed the resync finished event, corrected myself:\n");
+ drbd_set_peer_bitmap_uuid(peer_md, 0, 0);
+ _drbd_uuid_push_history(device, previous_bitmap_uuid);
+
+ drbd_uuid_dump_self(peer_device,
+ device->disk_state[NOW] >= D_NEGOTIATING ? drbd_bm_total_weight(peer_device) : 0, 0);
+ *rule = RULE_SYNC_SOURCE_MISSED_FINISH;
} else {
- drbd_warn(device, "receive_bitmap: cmd neither ReportBitMap nor ReportCBitMap (is 0x%x)", pi->cmd);
- err = -EIO;
- goto out;
+ drbd_info(device, "was SyncSource (peer failed to write sync_uuid)\n");
+ *rule = RULE_SYNC_SOURCE_PEER_MISSED_FINISH;
}
- c.packets[pi->cmd == P_BITMAP]++;
- c.bytes[pi->cmd == P_BITMAP] += drbd_header_size(connection) + pi->size;
+ return SYNC_SOURCE_USE_BITMAP;
+ }
- if (err <= 0) {
- if (err < 0)
- goto out;
- break;
+ if (drbd_bitmap_uuid(peer_device) == (u64)0 && peer_device->bitmap_uuids[node_id] != (u64)0) {
+
+ if (peer_device->connection->agreed_pro_version < 91)
+ return REQUIRES_PROTO_91;
+
+ if ((drbd_history_uuid(device, 0) & ~UUID_PRIMARY) ==
+ (peer_device->bitmap_uuids[node_id] & ~UUID_PRIMARY) &&
+ (drbd_history_uuid(device, 1) & ~UUID_PRIMARY) ==
+ (peer_device->history_uuids[0] & ~UUID_PRIMARY)) {
+ int i;
+
+ drbd_info(device, "was SyncTarget, peer missed the resync finished event, corrected peer:\n");
+
+ for (i = ARRAY_SIZE(peer_device->history_uuids) - 1; i > 0; i--)
+ peer_device->history_uuids[i] = peer_device->history_uuids[i - 1];
+ peer_device->history_uuids[i] = peer_device->bitmap_uuids[node_id];
+ peer_device->bitmap_uuids[node_id] = 0;
+
+ drbd_uuid_dump_peer(peer_device, peer_device->dirty_bits, peer_device->uuid_flags);
+ *rule = RULE_SYNC_TARGET_PEER_MISSED_FINISH;
+ } else {
+ drbd_info(device, "was SyncTarget (failed to write sync_uuid)\n");
+ *rule = RULE_SYNC_TARGET_MISSED_FINISH;
}
- err = drbd_recv_header(peer_device->connection, pi);
- if (err)
- goto out;
+
+ return SYNC_TARGET_USE_BITMAP;
+ }
+
+ return UNDETERMINED;
+}
+
+static enum sync_strategy
+uuid_fixup_resync_start1(struct drbd_peer_device *peer_device, enum sync_rule *rule)
+{
+ struct drbd_device *device = peer_device->device;
+ const int node_id = peer_device->device->resource->res_opts.node_id;
+ u64 self, peer;
+
+ self = drbd_current_uuid(device) & ~UUID_PRIMARY;
+ peer = peer_device->history_uuids[0] & ~UUID_PRIMARY;
+
+ if (self == peer) {
+ if (peer_device->connection->agreed_pro_version < 96 ?
+ (drbd_history_uuid(device, 0) & ~UUID_PRIMARY) ==
+ (peer_device->history_uuids[1] & ~UUID_PRIMARY) :
+ peer + UUID_NEW_BM_OFFSET == (peer_device->bitmap_uuids[node_id] & ~UUID_PRIMARY)) {
+ int i;
+
+ /* The last P_SYNC_UUID did not get though. Undo the last start of
+ resync as sync source modifications of the peer's UUIDs. */
+ *rule = RULE_SYNC_TARGET_MISSED_START;
+
+ if (peer_device->connection->agreed_pro_version < 91)
+ return REQUIRES_PROTO_91;
+
+ peer_device->bitmap_uuids[node_id] = peer_device->history_uuids[0];
+ for (i = 0; i < ARRAY_SIZE(peer_device->history_uuids) - 1; i++)
+ peer_device->history_uuids[i] = peer_device->history_uuids[i + 1];
+ peer_device->history_uuids[i] = 0;
+
+ drbd_info(device, "Lost last syncUUID packet, corrected:\n");
+ drbd_uuid_dump_peer(peer_device, peer_device->dirty_bits, peer_device->uuid_flags);
+
+ return SYNC_TARGET_USE_BITMAP;
+ }
+ }
+
+ return UNDETERMINED;
+}
+
+static enum sync_strategy
+uuid_fixup_resync_start2(struct drbd_peer_device *peer_device, enum sync_rule *rule)
+{
+ struct drbd_device *device = peer_device->device;
+ u64 self, peer;
+
+ self = drbd_history_uuid(device, 0) & ~UUID_PRIMARY;
+ peer = peer_device->current_uuid & ~UUID_PRIMARY;
+
+ if (self == peer) {
+ if (peer_device->connection->agreed_pro_version < 96 ?
+ (drbd_history_uuid(device, 1) & ~UUID_PRIMARY) ==
+ (peer_device->history_uuids[0] & ~UUID_PRIMARY) :
+ self + UUID_NEW_BM_OFFSET == (drbd_bitmap_uuid(peer_device) & ~UUID_PRIMARY)) {
+ u64 bitmap_uuid;
+
+ /* The last P_SYNC_UUID did not get though. Undo the last start of
+ resync as sync source modifications of our UUIDs. */
+ *rule = RULE_SYNC_SOURCE_MISSED_START;
+
+ if (peer_device->connection->agreed_pro_version < 91)
+ return REQUIRES_PROTO_91;
+
+ bitmap_uuid = _drbd_uuid_pull_history(peer_device);
+ __drbd_uuid_set_bitmap(peer_device, bitmap_uuid);
+
+ drbd_info(device, "Last syncUUID did not get through, corrected:\n");
+ drbd_uuid_dump_self(peer_device,
+ device->disk_state[NOW] >= D_NEGOTIATING ? drbd_bm_total_weight(peer_device) : 0, 0);
+
+ return SYNC_SOURCE_USE_BITMAP;
+ }
+ }
+
+ return UNDETERMINED;
+}
+
+/*
+ * Two diskful peers can carry the same current UUID yet sit at different
+ * positions in a (now absent) common primary's change stream. When both still
+ * hold out-of-sync bits toward each other, the bit counts do not order them;
+ * only the dagtag toward that common primary tells roll-forward from roll-back.
+ *
+ * Each of us kept its last_dagtag_sector toward the lost primary across the
+ * disconnect (see conn_disconnect()). Identify exactly one such common,
+ * currently-absent peer, so we can hand the other side our dagtag toward it
+ * during the handshake and let the node further along the stream become the
+ * resync source.
+ *
+ * Return -1 if there is no candidate, or more than one (ambiguous): we then
+ * fall back to a deterministic node-id choice, which still converges.
+ */
+static int find_common_lost_primary_node_id(struct drbd_connection *self)
+{
+ struct drbd_resource *resource = self->resource;
+ struct drbd_connection *connection;
+ int found = -1;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection == self)
+ continue;
+ if (connection->cstate[NOW] == C_CONNECTED)
+ continue;
+ /*
+ * Non-zero means we received writes (dagtags) from it: it was a
+ * source of data we may sit behind. Zero cannot order us.
+ */
+ if (atomic64_read(&connection->last_dagtag_sector) == 0)
+ continue;
+ if (found != -1) {
+ found = -1; /* ambiguous; caller uses the node-id fallback */
+ break;
+ }
+ found = connection->peer_node_id;
+ }
+ rcu_read_unlock();
+
+ return found;
+}
+
+static enum sync_strategy drbd_uuid_compare(struct drbd_peer_device *peer_device,
+ enum sync_rule *rule, int *peer_node_id)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ const int node_id = device->resource->res_opts.node_id;
+ bool my_current_in_peers_history, peers_current_in_my_history;
+ bool bitmap_matches, flags_matches, uuid_matches;
+ u64 resolved_uuid, bitmap_uuid;
+ u64 local_uuid_flags = 0;
+ u64 self, peer;
+ int i, j;
+
+ lockdep_assert_held(&device->ldev->md.uuid_lock);
+
+ resolved_uuid = drbd_resolved_uuid(peer_device, &local_uuid_flags) & ~UUID_PRIMARY;
+ bitmap_uuid = drbd_bitmap_uuid(peer_device);
+ local_uuid_flags |= drbd_collect_local_uuid_flags(peer_device, NULL);
+
+ uuid_matches = resolved_uuid == (peer_device->comm_current_uuid & ~UUID_PRIMARY);
+ bitmap_matches = bitmap_uuid == peer_device->comm_bitmap_uuid;
+ /* UUID_FLAG_INCONSISTENT is not relevant for the handshake, allow it to change.
+ * UUID_FLAG_BITMAP_AUTHORITATIVE moves with the first bit set or last bit
+ * cleared toward the peer; the handshake reads the value that was sent.
+ */
+ flags_matches = !((local_uuid_flags ^ peer_device->comm_uuid_flags) &
+ ~(UUID_FLAG_INCONSISTENT | UUID_FLAG_BITMAP_AUTHORITATIVE));
+ if (!test_bit(INITIAL_STATE_SENT, peer_device->flags)) {
+ drbd_warn(peer_device, "Initial UUIDs and state not sent yet. Not verifying\n");
+ } else if (!uuid_matches || !flags_matches || !bitmap_matches) {
+ if (!uuid_matches)
+ drbd_warn(peer_device, "My current UUID changed during handshake.\n");
+ if (!bitmap_matches)
+ drbd_warn(peer_device, "My bitmap UUID changed during "
+ "handshake. 0x%llX to 0x%llX\n",
+ (unsigned long long)peer_device->comm_bitmap_uuid,
+ (unsigned long long)bitmap_uuid);
+ if (!flags_matches)
+ drbd_warn(peer_device,
+ "My uuid_flags changed from 0x%llX to 0x%llX during handshake.\n",
+ (unsigned long long)peer_device->comm_uuid_flags,
+ (unsigned long long)local_uuid_flags);
+ if (connection->cstate[NOW] == C_CONNECTING) {
+ *rule = RULE_INITIAL_HANDSHAKE_CHANGED;
+ return RETRY_CONNECT;
+ }
+ }
+
+ self = resolved_uuid;
+ peer = peer_device->current_uuid & ~UUID_PRIMARY;
+
+ /* Before DRBD 8.0.2 (from 2007), the uuid on sync targets was set to
+ * zero during resyncs for no good reason. */
+ if (self == 0)
+ self = UUID_JUST_CREATED;
+ if (peer == 0)
+ peer = UUID_JUST_CREATED;
+
+ *rule = RULE_JUST_CREATED_BOTH;
+ if (self == UUID_JUST_CREATED && peer == UUID_JUST_CREATED)
+ return NO_SYNC;
+
+ *rule = RULE_JUST_CREATED_SELF;
+ if (self == UUID_JUST_CREATED)
+ return SYNC_TARGET_SET_BITMAP;
+
+ *rule = RULE_JUST_CREATED_PEER;
+ if (peer == UUID_JUST_CREATED)
+ return SYNC_SOURCE_SET_BITMAP;
+
+ if (self == peer) {
+ struct drbd_net_conf *nc;
+ int wire_protocol;
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ wire_protocol = nc->wire_protocol;
+ rcu_read_unlock();
+
+ if (connection->agreed_pro_version < 110) {
+ enum sync_strategy rv = uuid_fixup_resync_end(peer_device, rule);
+
+ if (rv != UNDETERMINED)
+ return rv;
+ }
+
+ if (test_bit(RS_SOURCE_MISSED_END, peer_device->flags)) {
+ *rule = RULE_SYNC_SOURCE_MISSED_FINISH;
+ return SYNC_SOURCE_USE_BITMAP;
+ }
+ if (test_bit(RS_PEER_MISSED_END, peer_device->flags)) {
+ *rule = RULE_SYNC_TARGET_PEER_MISSED_FINISH;
+ return SYNC_TARGET_USE_BITMAP;
+ }
+
+ /*
+ * Equal current UUIDs do not imply identical data: two peers can
+ * still hold out-of-sync bits toward each other. The rules below
+ * resolve the direction; where they would otherwise declare NO_SYNC
+ * and drop those bits, disk_states_to_strategy() reconciles via the
+ * bitmap afterwards (maybe_reconcile_equal_uuid_bitmap), so it never
+ * preempts a disk-state-based direction.
+ */
+
+ if (connection->agreed_pro_version >= 120) {
+ *rule = RULE_RECONNECTED;
+ if (peer_device->uuid_flags & UUID_FLAG_RECONNECT &&
+ local_uuid_flags & UUID_FLAG_RECONNECT)
+ return NO_SYNC;
+ }
+
+ if (connection->agreed_pro_version >= 121 &&
+ (wire_protocol == DRBD_PROT_A || wire_protocol == DRBD_PROT_B)) {
+ *rule = RULE_CRASHED_PRIMARY;
+ if (local_uuid_flags & UUID_FLAG_CRASHED_PRIMARY &&
+ !(peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY))
+ return SYNC_SOURCE_USE_BITMAP;
+
+ if (peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY &&
+ !(local_uuid_flags & UUID_FLAG_CRASHED_PRIMARY))
+ return SYNC_TARGET_USE_BITMAP;
+ }
+
+ *rule = RULE_LOST_QUORUM;
+ if (peer_device->uuid_flags & UUID_FLAG_PRIMARY_LOST_QUORUM &&
+ !test_bit(PRIMARY_LOST_QUORUM, &device->flags))
+ return SYNC_TARGET_IF_BOTH_FAILED;
+
+ if (!(peer_device->uuid_flags & UUID_FLAG_PRIMARY_LOST_QUORUM) &&
+ test_bit(PRIMARY_LOST_QUORUM, &device->flags))
+ return SYNC_SOURCE_IF_BOTH_FAILED;
+
+ if (peer_device->uuid_flags & UUID_FLAG_PRIMARY_LOST_QUORUM &&
+ test_bit(PRIMARY_LOST_QUORUM, &device->flags))
+ return test_bit(RESOLVE_CONFLICTS, &connection->transport.flags) ?
+ SYNC_SOURCE_IF_BOTH_FAILED :
+ SYNC_TARGET_IF_BOTH_FAILED;
+
+ *rule = RULE_RECONNECTED;
+ if (connection->agreed_pro_version < 120 &&
+ peer_device->uuid_flags & UUID_FLAG_RECONNECT &&
+ local_uuid_flags & UUID_FLAG_RECONNECT)
+ return NO_SYNC;
+
+ /* Peer crashed as primary, I survived, resync from me */
+ if (peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY &&
+ local_uuid_flags & UUID_FLAG_RECONNECT)
+ return SYNC_SOURCE_IF_BOTH_FAILED;
+
+ /* I am a crashed primary, peer survived, resync to me */
+ if (local_uuid_flags & UUID_FLAG_CRASHED_PRIMARY &&
+ peer_device->uuid_flags & UUID_FLAG_RECONNECT)
+ return SYNC_TARGET_IF_BOTH_FAILED;
+
+ /* One of us had a connection to the other node before.
+ i.e. this is not a common power failure. */
+ if (peer_device->uuid_flags & UUID_FLAG_RECONNECT ||
+ local_uuid_flags & UUID_FLAG_RECONNECT)
+ return NO_SYNC;
+
+ /* Common power [off|failure]? */
+ *rule = RULE_BOTH_OFF;
+ if (local_uuid_flags & UUID_FLAG_CRASHED_PRIMARY) {
+ if ((peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY) &&
+ test_bit(RESOLVE_CONFLICTS, &connection->transport.flags))
+ return SYNC_TARGET_IF_BOTH_FAILED;
+ return SYNC_SOURCE_IF_BOTH_FAILED;
+ } else if (peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY)
+ return SYNC_TARGET_IF_BOTH_FAILED;
+ else
+ return NO_SYNC;
+ }
+
+ *rule = RULE_BITMAP_PEER;
+ peer = peer_device->bitmap_uuids[node_id] & ~UUID_PRIMARY;
+ if (self == peer)
+ return SYNC_TARGET_USE_BITMAP;
+
+ *rule = RULE_BITMAP_PEER_OTHER;
+ i = drbd_find_peer_bitmap_by_uuid(peer_device, self);
+ if (i != -1) {
+ *peer_node_id = i;
+ return SYNC_TARGET_CLEAR_BITMAP;
+ }
+
+ if (connection->agreed_pro_version < 110) {
+ enum sync_strategy rv = uuid_fixup_resync_start1(peer_device, rule);
+
+ if (rv != UNDETERMINED)
+ return rv;
+ }
+
+ *rule = RULE_BITMAP_SELF;
+ self = bitmap_uuid & ~UUID_PRIMARY;
+ peer = peer_device->current_uuid & ~UUID_PRIMARY;
+ if (self == peer)
+ return SYNC_SOURCE_USE_BITMAP;
+
+ *rule = RULE_BITMAP_SELF_OTHER;
+ i = drbd_find_bitmap_by_uuid(peer_device, peer);
+ if (i != -1) {
+ if (is_divergence_bitmap(&device->ldev->md.peers[i])) {
+ *peer_node_id = i;
+ return SYNC_SOURCE_COPY_BITMAP;
+ }
+ /* A slot records the divergence from the peer's data, but its
+ * bitmap is being cleared by an ongoing resync and cannot be
+ * copied. The data is related, so resync the whole slot.
+ */
+ return SYNC_SOURCE_SET_BITMAP;
+ }
+
+ self = resolved_uuid;
+ my_current_in_peers_history = uuid_in_peer_history(peer_device, self);
+
+ if (connection->agreed_pro_version < 110) {
+ enum sync_strategy rv = uuid_fixup_resync_start2(peer_device, rule);
+
+ if (rv != UNDETERMINED)
+ return rv;
+ }
+
+ peer = peer_device->current_uuid & ~UUID_PRIMARY;
+ peers_current_in_my_history = uuid_in_my_history(device, peer);
+
+ if (my_current_in_peers_history && !peers_current_in_my_history) {
+ *rule = RULE_HISTORY_PEER;
+ return SYNC_TARGET_SET_BITMAP;
+ }
+ if (!my_current_in_peers_history && peers_current_in_my_history) {
+ *rule = RULE_HISTORY_SELF;
+ return SYNC_SOURCE_SET_BITMAP;
+ }
+
+ *rule = RULE_BITMAP_BOTH;
+ self = bitmap_uuid & ~UUID_PRIMARY;
+ peer = peer_device->bitmap_uuids[node_id] & ~UUID_PRIMARY;
+ if (self == peer && self != ((u64)0))
+ return SPLIT_BRAIN_AUTO_RECOVER;
+
+ *rule = RULE_HISTORY_BOTH;
+ for (i = 0; i < HISTORY_UUIDS; i++) {
+ self = drbd_history_uuid(device, i) & ~UUID_PRIMARY;
+ /* Don't conclude to have "data divergence" from a "common ancestor"
+ * if that common ancestor is just a not used yet slot in the history,
+ * which is still initialized to zero on both peers. */
+ if (self == 0)
+ break;
+ for (j = 0; j < ARRAY_SIZE(peer_device->history_uuids); j++) {
+ peer = peer_device->history_uuids[j] & ~UUID_PRIMARY;
+ if (peer == 0)
+ break;
+ if (self == peer)
+ return SPLIT_BRAIN_DISCONNECT;
+ }
+ }
+
+ return UNRELATED_DATA;
+}
+
+static void log_handshake(struct drbd_peer_device *peer_device)
+{
+ u64 uuid_flags = drbd_collect_local_uuid_flags(peer_device, NULL);
+
+ drbd_info(peer_device, "drbd_sync_handshake:\n");
+ drbd_uuid_dump_self(peer_device, peer_device->comm_bm_set, uuid_flags);
+ drbd_uuid_dump_peer(peer_device, peer_device->dirty_bits, peer_device->uuid_flags);
+}
+
+static enum sync_strategy drbd_handshake(struct drbd_peer_device *peer_device,
+ enum sync_rule *rule,
+ int *peer_node_id,
+ bool always_verbose)
+{
+ struct drbd_device *device = peer_device->device;
+ enum sync_strategy strategy;
+
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+ if (always_verbose)
+ log_handshake(peer_device);
+
+ strategy = drbd_uuid_compare(peer_device, rule, peer_node_id);
+ if (strategy != NO_SYNC && !always_verbose)
+ log_handshake(peer_device);
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+
+ if (strategy != NO_SYNC || always_verbose)
+ drbd_info(peer_device, "uuid_compare()=%s by rule=%s\n",
+ strategy_descriptor(strategy).name,
+ drbd_sync_rule_str(*rule));
+
+ return strategy;
+}
+
+static bool is_resync_running(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ bool rv = false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+
+ if (repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T) {
+ rv = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+static int bitmap_mod_after_handshake(struct drbd_peer_device *peer_device, enum sync_strategy strategy, int peer_node_id)
+{
+ struct drbd_device *device = peer_device->device;
+
+ if (strategy == SYNC_SOURCE_COPY_BITMAP) {
+ int from;
+
+ if (peer_device->bitmap_index == -1)
+ return 0;
+
+ drbd_suspend_io(device, WRITE_ONLY);
+ drbd_bm_slot_lock(peer_device, "copy_slot/set_many sync_handshake", BM_LOCK_BULK);
+ /* Pick the day-0 slot under the bitmap lock: cbt-start claims and
+ * clears a vacant slot under the same lock, so a slot found here
+ * keeps its day-0 bits until the copy is done.
+ */
+ from = device->ldev->md.peers[peer_node_id].bitmap_index;
+ if (from == -1)
+ from = drbd_unallocated_index(device->ldev);
+
+ if (from == -1) {
+ drbd_info(peer_device,
+ "Setting all bitmap bits, day0 bm not available node_id=%d\n",
+ peer_node_id);
+ drbd_bm_set_many_bits(peer_device, 0, -1UL);
+ } else {
+ drbd_info(peer_device,
+ "Copying bitmap of peer node_id=%d (bitmap_index=%d)\n",
+ peer_node_id, from);
+ drbd_bm_copy_slot(device, from, peer_device->bitmap_index);
+ }
+ drbd_bm_write(device, NULL);
+ drbd_bm_slot_unlock(peer_device);
+ drbd_resume_io(device);
+ } else if (strategy == SYNC_TARGET_CLEAR_BITMAP) {
+ drbd_info(peer_device, "Resync source provides bitmap (node_id=%d)\n", peer_node_id);
+ drbd_suspend_io(device, WRITE_ONLY);
+ drbd_bm_slot_lock(peer_device, "bm_clear_many_bits sync_handshake", BM_LOCK_BULK);
+ drbd_bm_clear_many_bits(peer_device, 0, -1UL);
+ drbd_bm_write(device, NULL);
+ drbd_bm_slot_unlock(peer_device);
+ drbd_resume_io(device);
+ } else if (strategy == SYNC_SOURCE_SET_BITMAP || strategy == SYNC_TARGET_SET_BITMAP) {
+ int err;
+
+ if (strategy == SYNC_TARGET_SET_BITMAP &&
+ drbd_current_uuid(device) == UUID_JUST_CREATED &&
+ is_resync_running(device))
+ return 0;
+
+ if (drbd_current_uuid(device) == UUID_JUST_CREATED) {
+ drbd_info(peer_device, "Setting and writing the whole bitmap, fresh node\n");
+ err = drbd_bitmap_io(device, &drbd_bmio_set_allocated_n_write,
+ "set_allocated_n_write sync_handshake",
+ BM_LOCK_CLEAR | BM_LOCK_BULK, NULL);
+ } else {
+ drbd_info(peer_device, "Setting and writing one bitmap slot, after drbd_sync_handshake\n");
+ err = drbd_bitmap_io(device, &drbd_bmio_set_n_write,
+ "set_n_write sync_handshake",
+ BM_LOCK_CLEAR | BM_LOCK_BULK | BM_LOCK_SINGLE_SLOT,
+ peer_device);
+ }
+ if (err)
+ return err;
+
+ if (drbd_current_uuid(device) != UUID_JUST_CREATED &&
+ peer_device->current_uuid != UUID_JUST_CREATED &&
+ strategy == SYNC_SOURCE_SET_BITMAP) {
+ /*
+ * We have just written the bitmap slot. Update the
+ * bitmap UUID so that the resync does not start from
+ * the beginning again if we disconnect and reconnect.
+ *
+ * Initial resync continuation is handled in
+ * drbd_start_resync() at comment:
+ * prepare to continue an interrupted initial resync later
+ */
+ drbd_uuid_set_bitmap(peer_device, peer_device->current_uuid);
+ drbd_print_uuids(peer_device, "updated bitmap UUID");
+ drbd_md_sync(device);
+ }
+ }
+ return 0;
+}
+
+static enum drbd_repl_state strategy_to_repl_state(struct drbd_peer_device *peer_device,
+ enum drbd_role peer_role,
+ enum sync_strategy strategy)
+{
+ enum drbd_role role = peer_device->device->resource->role[NOW];
+ enum drbd_repl_state rv;
+
+ if (strategy == SYNC_SOURCE_IF_BOTH_FAILED || strategy == SYNC_TARGET_IF_BOTH_FAILED) {
+ if (role == R_PRIMARY || peer_role == R_PRIMARY) {
+ /* We have at least one primary, follow that with the resync decision */
+ rv = peer_role == R_SECONDARY ? L_WF_BITMAP_S :
+ role == R_SECONDARY ? L_WF_BITMAP_T :
+ L_ESTABLISHED;
+ return rv;
+ }
+ /* No current primary. Handle it as a common power failure, consider the
+ roles at crash time */
+ }
+
+ if (strategy_descriptor(strategy).is_sync_source) {
+ rv = L_WF_BITMAP_S;
+ } else if (strategy_descriptor(strategy).is_sync_target) {
+ rv = L_WF_BITMAP_T;
+ } else {
+ rv = L_ESTABLISHED;
+ }
+
+ return rv;
+}
+
+static enum sync_strategy drbd_disk_states_source_strategy(
+ struct drbd_peer_device *peer_device,
+ int *peer_node_id)
+{
+ const int node_id = peer_device->device->resource->res_opts.node_id;
+ u64 bitmap_uuid;
+ int i = -1;
+
+ if (!(peer_device->uuid_flags & UUID_FLAG_SYNC_TARGET))
+ return SYNC_SOURCE_USE_BITMAP;
+
+ /* A resync with identical current-UUIDs -> USE_BITMAP */
+ bitmap_uuid = peer_device->bitmap_uuids[node_id];
+ if (bitmap_uuid == peer_device->current_uuid &&
+ bitmap_uuid == drbd_current_uuid(peer_device->device))
+ return SYNC_SOURCE_USE_BITMAP;
+
+ /* When the peer is already a sync target, we actually see its
+ * current UUID in the bitmap UUID slot towards us. We may need
+ * to pick a different bitmap as a result. */
+ if (bitmap_uuid)
+ i = drbd_find_bitmap_by_uuid(peer_device, bitmap_uuid);
+
+ if (i == peer_device->node_id)
+ return SYNC_SOURCE_USE_BITMAP;
+
+ if (i == -1 || !is_divergence_bitmap(&peer_device->device->ldev->md.peers[i]))
+ return SYNC_SOURCE_SET_BITMAP;
+
+ *peer_node_id = i;
+ return SYNC_SOURCE_COPY_BITMAP;
+}
+
+static enum sync_strategy drbd_disk_states_target_strategy(
+ struct drbd_peer_device *peer_device,
+ int *peer_node_id)
+{
+ const int node_id = peer_device->device->resource->res_opts.node_id;
+ u64 bitmap_uuid;
+ int i;
+
+ if (!(peer_device->comm_uuid_flags & UUID_FLAG_SYNC_TARGET))
+ return SYNC_TARGET_USE_BITMAP;
+
+ bitmap_uuid = drbd_bitmap_uuid(peer_device);
+ if (bitmap_uuid == peer_device->current_uuid &&
+ bitmap_uuid == drbd_current_uuid(peer_device->device))
+ return SYNC_TARGET_USE_BITMAP;
+
+ /* When we are already a sync target, we need to choose our
+ * strategy to mirror the peer's choice (see
+ * drbd_disk_states_source_strategy). */
+ i = drbd_find_peer_bitmap_by_uuid(peer_device, bitmap_uuid);
+
+ if (i == node_id)
+ return SYNC_TARGET_USE_BITMAP;
+
+ if (i == -1)
+ return SYNC_TARGET_SET_BITMAP;
+
+ *peer_node_id = i;
+ return SYNC_TARGET_CLEAR_BITMAP;
+}
+
+/*
+ * A node that has a primary at its side does not take an equal-UUID reconcile:
+ * the primary keeps its copy current and holds the record of what it is
+ * missing, while the node without one is about to outdate itself at the commit
+ * of this very connect (do_change_cstate(), PH_COMMIT). A resync in that
+ * direction hands that fencing verdict to a node it is not about, because
+ * drbd_resync_finished() gives the target the source's disk state.
+ *
+ * UUID_FLAG_STABLE is "no primary neighbor", and both sides read the same two
+ * values: the flag each of them sent with its UUIDs. At connect the connection
+ * being handshaked is not yet L_ESTABLISHED, so neither side counts the other
+ * as its neighbor -- which is why this is limited to the connect handshake.
+ *
+ * Returns false where both sides are placed alike, leaving the direction to the
+ * caller.
+ */
+static bool primary_neighbor_decides(struct drbd_peer_device *peer_device,
+ bool connecting, enum sync_strategy *strategy)
+{
+ bool we_stable = peer_device->comm_uuid_flags & UUID_FLAG_STABLE;
+ bool peer_stable = peer_device->uuid_flags & UUID_FLAG_STABLE;
+
+ /*
+ * Both sides must apply the same rule: a peer that does not know it
+ * orders by bit count, so fall back to that ordering toward it.
+ */
+ if (peer_device->connection->agreed_pro_version < 125)
+ return false;
+ if (!connecting || we_stable == peer_stable)
+ return false;
+
+ *strategy = we_stable ? SYNC_TARGET_USE_BITMAP : SYNC_SOURCE_USE_BITMAP;
+ return true;
+}
+
+/*
+ * Two equal-current-UUID peers can still hold out-of-sync bits toward each
+ * other: a write reached one of them and never the other, and no data
+ * generation recorded that -- the writer could not mint (a diskless Primary,
+ * a Primary that lost quorum under on-no-quorum=io-error, a crash before the
+ * bitmap was written out), or a common primary vanished while they sat at
+ * different positions in its change stream and the P_PEER_DAGTAG
+ * reconciliation never reached them. The equal-UUID rules would then declare
+ * NO_SYNC, which clears the bitmaps and silently drops the divergence.
+ * Reconcile via the bitmap instead.
+ *
+ * precondition (else leave the decision untouched):
+ * strategy == NO_SYNC (rules found no resync; bits would drop)
+ * both peers D_UP_TO_DATE (differing disk states are the
+ * disk-state decision's job, made above)
+ * out-of-sync bits remain (comm_bm_set || dirty_bits)
+ * peer supports the feature (DRBD_FF_RECONCILE_RECONNECT)
+ * the connect handshake (C_CONNECTING)
+ * from protocol version 125 on:
+ * a slot holding bits is authoritative on at least one side: its bits
+ * were set for blocks the other lacks (MDF_PEER_BITMAP_AUTHORITATIVE,
+ * exchanged as UUID_FLAG_BITMAP_AUTHORITATIVE); bits a sync target set
+ * toward an absent peer, or an invalidate left behind, do not count
+ * and two of the preconditions widen:
+ * the after-unstable re-handshake on an established connection
+ * qualifies as well (drbd_resync()), and
+ * with one side authoritative only that side must be D_UP_TO_DATE;
+ * the other may be D_OUTDATED or D_CONSISTENT (it lacks the blocks,
+ * and lifting its mark on NO_SYNC is what would lose the record)
+ * below protocol version 125:
+ * a common lost primary is named (see below)
+ * no stronger signal owns it (crashed primary / lost quorum)
+ *
+ * direction:
+ * one of the two has a primary neighbor -> it is the source
+ * (primary_neighbor_decides(), connect only,
+ * both sides D_UP_TO_DATE)
+ * one side authoritative -> it holds what the other lacks -> it is the
+ * source (below 125: the side holding bits)
+ * both sides -> dagtag toward the common lost primary,
+ * else the primary neighbor, else a
+ * deterministic node-id tie-break
+ *
+ * Placed here, after the disk-state resolution, so it never preempts a real
+ * resync direction; it only rescues the would-be NO_SYNC-drops-bits case.
+ */
+static bool reconcile_equal_uuid_bitmap_strategy(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state disk_state,
+ enum drbd_disk_state peer_disk_state,
+ enum sync_strategy *strategy,
+ enum sync_rule *rule)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ const int node_id = device->resource->res_opts.node_id;
+ bool by_provenance = connection->agreed_pro_version >= 125;
+ bool connecting = connection->cstate[NOW] == C_CONNECTING;
+ bool we_hold, peer_holds;
+ u64 local_uuid_flags;
+
+ if (*strategy != NO_SYNC)
+ return false;
+ if (!(peer_device->comm_bm_set || peer_device->dirty_bits))
+ return false;
+ if (!(connection->agreed_features & DRBD_FF_RECONCILE_RECONNECT))
+ return false;
+ if (!connecting && !(by_provenance && peer_device->repl_state[NOW] == L_ESTABLISHED))
+ return false;
+
+ if (by_provenance) {
+ /*
+ * Both sides see the same two flags: the one this node sent with
+ * its bit count, and the one the peer sent with its own. Bits
+ * that are authoritative on neither side are resync artifacts,
+ * and NO_SYNC (drop the bits) stays right for them; forcing a
+ * resync would pull a healthy UpToDate peer through Inconsistent
+ * for nothing.
+ */
+ we_hold = peer_device->comm_bm_set &&
+ (peer_device->comm_uuid_flags & UUID_FLAG_BITMAP_AUTHORITATIVE);
+ peer_holds = peer_device->dirty_bits &&
+ (peer_device->uuid_flags & UUID_FLAG_BITMAP_AUTHORITATIVE);
+ if (!we_hold && !peer_holds)
+ return false;
+
+ if (we_hold != peer_holds) {
+ enum drbd_disk_state holder = we_hold ? disk_state : peer_disk_state;
+ enum drbd_disk_state other = we_hold ? peer_disk_state : disk_state;
+
+ if (holder != D_UP_TO_DATE || other < D_OUTDATED)
+ return false;
+ } else if (disk_state != D_UP_TO_DATE || peer_disk_state != D_UP_TO_DATE) {
+ return false;
+ }
+ } else {
+ bool one_sided = !peer_device->comm_bm_set != !peer_device->dirty_bits;
+
+ if (disk_state != D_UP_TO_DATE || peer_disk_state != D_UP_TO_DATE)
+ return false;
+
+ /*
+ * Without the provenance record, equal current UUIDs with
+ * leftover bits reconcile only between secondaries of a *common
+ * lost primary*. Without one, the bits are taken for resync
+ * artifacts and NO_SYNC is kept.
+ *
+ * "Named a common lost primary" must be evaluated identically on
+ * both nodes, but each node only ever *hears* the other's naming
+ * (find_common_lost_primary_node_id excludes the peer itself, so
+ * a plain secondary of the bit-holder can never name one;
+ * P_PEER_DAGTAG stashed the peer's naming in reconcile_handshake).
+ * Gating on the stash alone therefore deadlocked one-sided
+ * reconciles: the side the naming reached went WFBitMapS while
+ * the namer stayed NO_SYNC/Established. Accept either direction
+ * of the exchange -- the peer named one, or we named one to the
+ * peer -- which is the same predicate on both nodes, so the
+ * one-sided case (ordered by the bit counts) derives mirrored
+ * decisions.
+ *
+ * The both-sided case cannot be ordered by bit counts; it
+ * additionally needs the peer's dagtag position toward the named
+ * lost primary, so it still requires the peer's naming.
+ */
+ if (connection->reconcile_handshake.lost_node_id == -1 &&
+ !(one_sided && connection->reconcile_handshake.sent_lost_node))
+ return false;
+
+ local_uuid_flags = drbd_collect_local_uuid_flags(peer_device, NULL);
+ if ((local_uuid_flags & UUID_FLAG_CRASHED_PRIMARY) ||
+ (peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY) ||
+ test_bit(PRIMARY_LOST_QUORUM, &device->flags) ||
+ (peer_device->uuid_flags & UUID_FLAG_PRIMARY_LOST_QUORUM))
+ return false;
+
+ we_hold = peer_device->comm_bm_set != 0;
+ peer_holds = peer_device->dirty_bits != 0;
+ }
+
+ *rule = RULE_RECONCILE_BITMAP;
+ if (we_hold != peer_holds) {
+ /*
+ * The primary neighbor overrides the holder only between two
+ * UpToDate copies. From protocol 125 on, the non-holder may be
+ * Outdated or Consistent here: it lacks the blocks the bits are
+ * about, and a primary neighbor that could not keep it current
+ * is no reason to make it the source.
+ */
+ bool both_up_to_date = disk_state == D_UP_TO_DATE &&
+ peer_disk_state == D_UP_TO_DATE;
+
+ if (!both_up_to_date ||
+ !primary_neighbor_decides(peer_device, connecting, strategy))
+ *strategy = we_hold ?
+ SYNC_SOURCE_USE_BITMAP : SYNC_TARGET_USE_BITMAP;
+ } else {
+ /*
+ * Both sides hold bits. Either direction restores identity, but the
+ * bit counts do not order us; the node further along the common lost
+ * primary's change stream (by dagtag) is the source. No usable dagtag
+ * offset (we are at the same position) -> deterministic node-id
+ * tie-break; either direction converges.
+ *
+ * The dagtag arm needs the naming in both directions: each node only
+ * holds the position the peer sent it. A node that named without
+ * hearing a name, or the reverse, would order by dagtag while its
+ * peer tie-breaks, and the two could pick the same role. The naming
+ * belongs to the connect handshake; on an established connection it
+ * is stale, so only the tie-break is used there.
+ */
+ int lost_id = connection->reconcile_handshake.lost_node_id;
+ struct drbd_connection *lost = NULL;
+ s64 offset = 0;
+
+ rcu_read_lock();
+ if (connecting && connection->reconcile_handshake.sent_lost_node)
+ lost = drbd_connection_by_node_id(device->resource, lost_id);
+ if (lost)
+ offset = atomic64_read(&lost->last_dagtag_sector) -
+ (s64)connection->reconcile_handshake.dagtag_sector;
+ rcu_read_unlock();
+
+ if (offset) {
+ *rule = RULE_RECONCILE_DAGTAG;
+ *strategy = offset > 0 ? SYNC_SOURCE_USE_BITMAP : SYNC_TARGET_USE_BITMAP;
+ } else if (!primary_neighbor_decides(peer_device, connecting, strategy)) {
+ *strategy = node_id < peer_device->node_id ?
+ SYNC_SOURCE_USE_BITMAP : SYNC_TARGET_USE_BITMAP;
+ }
+ }
+
+ return true;
+}
+
+static void maybe_reconcile_equal_uuid_bitmap(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state disk_state,
+ enum drbd_disk_state peer_disk_state,
+ enum sync_strategy *strategy,
+ enum sync_rule *rule)
+{
+ if (!reconcile_equal_uuid_bitmap_strategy(peer_device, disk_state, peer_disk_state,
+ strategy, rule))
+ return;
+
+ /* Mark this as a reconciliation resync, like the connected dagtag reconcile
+ * (receive_peer_dagtag). Its bitmap is pessimistic (set for tails that may
+ * have been delivered) between two full copies; the SyncTarget then forces
+ * checksum-based resync, clearing the matching majority without transfer and
+ * keeping its Inconsistent window short. Cleared when the resync finishes
+ * (sanitize_state) or on disconnect.
+ */
+ set_bit(RECONCILIATION_RESYNC, peer_device->flags);
+
+ drbd_info(peer_device,
+ "strategy = %s to reconcile equal-UUID peers holding out-of-sync bits\n",
+ strategy_descriptor(*strategy).name);
+}
+
+static void disk_states_to_strategy(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state peer_disk_state,
+ enum sync_strategy *strategy, enum sync_rule *rule,
+ int *peer_node_id)
+{
+ enum drbd_disk_state disk_state = peer_device->comm_state.disk;
+ struct drbd_device *device = peer_device->device;
+ bool decide_based_on_dstates = false;
+ bool prefer_local, either_inconsistent;
+
+ if (disk_state == D_NEGOTIATING)
+ disk_state = disk_state_from_md(device);
+
+ either_inconsistent =
+ (disk_state == D_INCONSISTENT && peer_disk_state > D_INCONSISTENT) ||
+ (peer_disk_state == D_INCONSISTENT && disk_state > D_INCONSISTENT);
+
+ if (peer_device->connection->agreed_pro_version >= 119) {
+ bool dstates_want_resync =
+ disk_state != peer_disk_state && disk_state >= D_INCONSISTENT &&
+ peer_disk_state >= D_INCONSISTENT && peer_disk_state != D_UNKNOWN;
+ bool resync_direction_arbitrary =
+ *strategy == SYNC_TARGET_IF_BOTH_FAILED ||
+ *strategy == SYNC_SOURCE_IF_BOTH_FAILED;
+
+ decide_based_on_dstates =
+ dstates_want_resync &&
+ (((*rule == RULE_RECONNECTED || *rule == RULE_LOST_QUORUM ||
+ *rule == RULE_BOTH_OFF) && resync_direction_arbitrary) ||
+ (*strategy == NO_SYNC && either_inconsistent));
+
+ prefer_local = disk_state > peer_disk_state;
+ /* RULE_BOTH_OFF means that the current UUIDs are equal. The decision
+ was found by looking at the crashed_primary bits.
+ The current disk states might give a better basis for decision-making! */
+
+ /* RULE_LOST_QUORUM means that the current UUIDs are equal. The resync direction
+ was found by looking if a node lost quorum while being primary */
+ } else {
+ decide_based_on_dstates =
+ (*rule == RULE_BOTH_OFF || *strategy == NO_SYNC) && either_inconsistent;
+
+ prefer_local = disk_state > D_INCONSISTENT;
+ }
+
+ if (decide_based_on_dstates) {
+ *strategy = prefer_local ?
+ drbd_disk_states_source_strategy(peer_device, peer_node_id) :
+ drbd_disk_states_target_strategy(peer_device, peer_node_id);
+ drbd_info(peer_device, "strategy = %s due to disk states. (%s/%s)\n",
+ strategy_descriptor(*strategy).name,
+ drbd_disk_str(disk_state), drbd_disk_str(peer_disk_state));
+ }
+
+ maybe_reconcile_equal_uuid_bitmap(peer_device, disk_state, peer_disk_state,
+ strategy, rule);
+}
+
+static enum sync_strategy drbd_attach_handshake(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state peer_disk_state)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ enum drbd_role peer_role = connection->peer_role[NOW];
+ enum sync_strategy strategy;
+ enum sync_rule rule;
+ int peer_node_id, err;
+ struct drbd_net_conf *nc;
+ int rr_conflict, always_asbp;
+ bool need_full_sync_after_split_brain;
+
+ strategy = drbd_handshake(peer_device, &rule, &peer_node_id, true);
+
+ if (!is_strategy_determined(strategy))
+ return strategy;
+
+ disk_states_to_strategy(peer_device, peer_disk_state, &strategy, &rule, &peer_node_id);
+
+ if (strategy == SPLIT_BRAIN_AUTO_RECOVER &&
+ (!drbd_device_stable(device, NULL) ||
+ !(peer_device->uuid_flags & UUID_FLAG_STABLE))) {
+ drbd_warn(peer_device,
+ "Ignoring split-brain during attach: at least one side unstable\n");
+ strategy = NO_SYNC;
+ }
+
+ if (strategy_descriptor(strategy).is_split_brain)
+ drbd_maybe_khelper(device, connection, "initial-split-brain");
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ always_asbp = nc->always_asbp;
+ rr_conflict = nc->rr_conflict;
+ rcu_read_unlock();
+
+ need_full_sync_after_split_brain = (strategy == SPLIT_BRAIN_DISCONNECT);
+
+ if (strategy == SPLIT_BRAIN_AUTO_RECOVER ||
+ (strategy == SPLIT_BRAIN_DISCONNECT && always_asbp)) {
+ int pcount = (device->resource->role[NOW] == R_PRIMARY)
+ + (peer_role == R_PRIMARY);
+
+ if (strategy_descriptor(strategy).is_split_brain) {
+ switch (pcount) {
+ case 0:
+ strategy = drbd_asb_recover_0p(peer_device);
+ break;
+ case 1:
+ strategy = drbd_asb_recover_1p(peer_device);
+ break;
+ case 2:
+ strategy = drbd_asb_recover_2p(peer_device);
+ break;
+ }
+ }
+ if (strategy == RETRY_CONNECT)
+ return strategy;
+ if (!strategy_descriptor(strategy).is_split_brain) {
+ drbd_warn(peer_device,
+ "Split-Brain detected during attach, %d primary(ies), automatically resolved. Sync from %s node\n",
+ pcount,
+ strategy_descriptor(strategy).is_sync_target ? "peer" : "this");
+ if (need_full_sync_after_split_brain) {
+ if (!strategy_descriptor(strategy).full_sync_equivalent) {
+ drbd_alert(peer_device,
+ "Want full sync but cannot decide direction, dropping connection!\n");
+ return SPLIT_BRAIN_DISCONNECT;
+ }
+ drbd_warn(peer_device,
+ "Doing a full sync since UUIDs were ambiguous.\n");
+ strategy = strategy_descriptor(strategy).full_sync_equivalent;
+ }
+ }
+ }
+
+ if (strategy_descriptor(strategy).is_split_brain) {
+ drbd_alert(peer_device,
+ "Split-Brain detected during attach but unresolved, dropping connection!\n");
+ drbd_maybe_khelper(device, connection, "split-brain");
+ return strategy;
+ }
+
+ if (!is_strategy_determined(strategy)) {
+ drbd_alert(peer_device,
+ "Failed to determine sync strategy during attach, dropping connection!\n");
+ return strategy;
+ }
+
+ if (strategy_descriptor(strategy).is_sync_target &&
+ strategy != SYNC_TARGET_IF_BOTH_FAILED &&
+ device->resource->role[NOW] == R_PRIMARY &&
+ device->disk_state[NOW] >= D_CONSISTENT &&
+ (peer_device->comm_bm_set > 0 || peer_device->dirty_bits > 0)) {
+ switch (rr_conflict) {
+ case ASB_CALL_HELPER:
+ drbd_maybe_khelper(device, connection, "pri-lost");
+ fallthrough;
+ case ASB_DISCONNECT:
+ case ASB_RETRY_CONNECT:
+ drbd_err(peer_device,
+ "I shall become SyncTarget during attach, but I am primary!\n");
+ strategy = rr_conflict == ASB_RETRY_CONNECT ?
+ SYNC_TARGET_PRIMARY_RECONNECT :
+ SYNC_TARGET_PRIMARY_DISCONNECT;
+ break;
+ case ASB_VIOLENTLY:
+ drbd_warn(peer_device,
+ "Becoming SyncTarget during attach, violating stable-data assumption\n");
+ break;
+ default:
+ break;
+ }
+ }
+
+ err = bitmap_mod_after_handshake(peer_device, strategy, peer_node_id);
+ if (err)
+ return RETRY_CONNECT;
+
+ return strategy;
+}
+
+static enum sync_strategy discard_my_data_to_strategy(struct drbd_peer_device *peer_device)
+{
+ enum sync_strategy strategy = UNDETERMINED;
+
+ if (test_bit(DISCARD_MY_DATA, peer_device->flags) &&
+ !(peer_device->uuid_flags & UUID_FLAG_DISCARD_MY_DATA))
+ strategy = SYNC_TARGET_USE_BITMAP;
+
+ if (!test_bit(DISCARD_MY_DATA, peer_device->flags) &&
+ (peer_device->uuid_flags & UUID_FLAG_DISCARD_MY_DATA))
+ strategy = SYNC_SOURCE_USE_BITMAP;
+
+ return strategy;
+}
+
+/* drbd_sync_handshake() returns the strategy for this connection; one with
+ * .reconnect or .disconnect set ends the connect attempt.
+ */
+static enum sync_strategy drbd_sync_handshake(struct drbd_peer_device *peer_device,
+ union drbd_state peer_state)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_net_conf *nc;
+ enum sync_strategy strategy;
+ enum sync_rule rule;
+ int rr_conflict, always_asbp, peer_node_id = 0, err;
+ enum drbd_role peer_role = peer_state.role;
+ enum drbd_disk_state peer_disk_state = peer_state.disk;
+ int required_protocol;
+ enum sync_strategy strategy_from_user = discard_my_data_to_strategy(peer_device);
+ bool need_full_sync_after_split_brain;
+
+ strategy = drbd_handshake(peer_device, &rule, &peer_node_id, true);
+
+ if (strategy == RETRY_CONNECT)
+ return strategy;
+
+ if (strategy == UNRELATED_DATA) {
+ drbd_alert(peer_device, "Unrelated data, aborting!\n");
+ return strategy;
+ }
+ required_protocol = strategy_descriptor(strategy).required_protocol;
+ if (required_protocol) {
+ drbd_alert(peer_device, "To resolve this both sides have to support at least protocol %d\n", required_protocol);
+ return strategy;
+ }
+
+ /* Protocol < 124 peers don't handle 0-bit missed-end-of-resync correctly.
+ * Retry the connection to let UUID cleanup resolve it, or keep retrying
+ * if the peer needs to be upgraded.
+ */
+ if (connection->agreed_pro_version < 124 &&
+ peer_device->comm_bm_set == 0 && peer_device->dirty_bits == 0) {
+ if (strategy == SYNC_SOURCE_USE_BITMAP &&
+ rule == RULE_SYNC_SOURCE_MISSED_FINISH) {
+ drbd_info(peer_device, "Missed end of resync as sync-source with 0 bits;"
+ " retrying to let UUID cleanup resolve it\n");
+ return RETRY_CONNECT;
+ }
+ if (strategy == SYNC_TARGET_USE_BITMAP &&
+ rule == RULE_SYNC_TARGET_PEER_MISSED_FINISH &&
+ device->resource->role[NOW] == R_PRIMARY) {
+ drbd_info(peer_device, "Missed end of resync as sync-target with 0 bits on Primary;"
+ " peer needs protocol 124+ to resolve, retrying\n");
+ return REQUIRES_PROTO_124;
+ }
+ }
+
+ disk_states_to_strategy(peer_device, peer_disk_state, &strategy, &rule, &peer_node_id);
+
+ if (strategy == SPLIT_BRAIN_AUTO_RECOVER && (!drbd_device_stable(device, NULL) || !(peer_device->uuid_flags & UUID_FLAG_STABLE))) {
+ drbd_warn(peer_device, "Ignore Split-Brain, for now, at least one side unstable\n");
+ strategy = NO_SYNC;
+ }
+
+ if (strategy_descriptor(strategy).is_split_brain)
+ drbd_maybe_khelper(device, connection, "initial-split-brain");
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ always_asbp = nc->always_asbp;
+ rr_conflict = nc->rr_conflict;
+ rcu_read_unlock();
+
+ /* Evaluate the original strategy,
+ * before it is re-mapped by additional configuration below.
+ */
+ need_full_sync_after_split_brain = (strategy == SPLIT_BRAIN_DISCONNECT);
+
+ if (strategy == SPLIT_BRAIN_AUTO_RECOVER || (strategy == SPLIT_BRAIN_DISCONNECT && always_asbp)) {
+ int pcount = (device->resource->role[NOW] == R_PRIMARY)
+ + (peer_role == R_PRIMARY);
+
+ if (device->resource->res_opts.quorum != QOU_OFF &&
+ connection->agreed_pro_version >= 113) {
+ if (device->have_quorum[NOW] && !peer_state.quorum)
+ strategy = SYNC_SOURCE_USE_BITMAP;
+ else if (!device->have_quorum[NOW] && peer_state.quorum)
+ strategy = SYNC_TARGET_USE_BITMAP;
+ }
+ if (strategy_descriptor(strategy).is_split_brain) {
+ switch (pcount) {
+ case 0:
+ strategy = drbd_asb_recover_0p(peer_device);
+ break;
+ case 1:
+ strategy = drbd_asb_recover_1p(peer_device);
+ break;
+ case 2:
+ strategy = drbd_asb_recover_2p(peer_device);
+ break;
+ }
+ }
+ if (strategy == RETRY_CONNECT) {
+ /* A dry run must not give up the Primary role. */
+ if (test_bit(CONN_DRY_RUN, &connection->flags)) {
+ clear_bit(CONN_HANDSHAKE_DEMOTE, &connection->flags);
+ drbd_info(peer_device,
+ "dry-run connect: Split-Brain, would give up Primary role.\n");
+ return SPLIT_BRAIN_DISCONNECT;
+ }
+ return strategy;
+ }
+ if (!strategy_descriptor(strategy).is_split_brain) {
+ drbd_warn(peer_device, "Split-Brain detected, %d primaries, "
+ "automatically solved. Sync from %s node\n",
+ pcount, strategy_descriptor(strategy).is_sync_target ? "peer" : "this");
+ if (need_full_sync_after_split_brain) {
+ if (!strategy_descriptor(strategy).full_sync_equivalent) {
+ drbd_alert(peer_device, "Want full sync but cannot decide direction, dropping connection!\n");
+ return SPLIT_BRAIN_DISCONNECT;
+ }
+ drbd_warn(peer_device, "Doing a full sync, since"
+ " UUIDs where ambiguous.\n");
+ strategy = strategy_descriptor(strategy).full_sync_equivalent;
+ }
+ }
+ }
+
+ if (strategy == SPLIT_BRAIN_DISCONNECT && strategy_from_user != UNDETERMINED) {
+ /* strategy_from_user via "--discard-my-data" is either
+ * SYNC_TARGET_USE_BITMAP or SYNC_SOURCE_USE_BITMAP.
+ * But here we do no longer have a relevant bitmap anymore.
+ * Map to their "full sync equivalent".
+ */
+ if (need_full_sync_after_split_brain)
+ strategy = strategy_descriptor(strategy_from_user).full_sync_equivalent;
+ else
+ strategy = strategy_from_user;
+ drbd_warn(peer_device, "Split-Brain detected, manually solved. %s from %s node\n",
+ need_full_sync_after_split_brain ? "Full sync" : "Sync",
+ strategy_descriptor(strategy).is_sync_target ? "peer" : "this");
+ }
+
+ if (strategy_descriptor(strategy).is_split_brain) {
+ drbd_alert(peer_device, "Split-Brain detected but unresolved, dropping connection!\n");
+ drbd_maybe_khelper(device, connection, "split-brain");
+ return strategy;
+ }
+
+ if (!is_strategy_determined(strategy)) {
+ drbd_alert(peer_device, "Failed to fully determine sync strategy, dropping connection!\n");
+ return strategy;
+ }
+
+ if (connection->agreed_pro_version >= 121 && strategy != NO_SYNC &&
+ strategy_from_user != UNDETERMINED &&
+ strategy_descriptor(strategy).is_sync_source != strategy_descriptor(strategy_from_user).is_sync_source) {
+ if (strategy_descriptor(strategy).reverse != UNDETERMINED) {
+ enum sync_strategy reversed = strategy_descriptor(strategy).reverse;
+ enum drbd_disk_state resync_source_disk_state =
+ strategy_descriptor(reversed).is_sync_source ? device->disk_state[NOW] : peer_disk_state;
+ if (resync_source_disk_state > D_INCONSISTENT) {
+ strategy = reversed;
+ drbd_warn(peer_device, "Resync direction reversed by --discard-my-data. Reverting to older data!\n");
+ } else {
+ drbd_warn(peer_device, "Ignoring --discard-my-data\n");
+ }
+ } else {
+ drbd_warn(peer_device, "Can not reverse resync direction (requested via --discard-my-data)\n");
+ }
+ }
+
+ if (strategy_descriptor(strategy).is_sync_target &&
+ strategy != SYNC_TARGET_IF_BOTH_FAILED &&
+ device->resource->role[NOW] == R_PRIMARY && device->disk_state[NOW] >= D_CONSISTENT &&
+ (peer_device->comm_bm_set > 0 || peer_device->dirty_bits > 0)) {
+ switch (rr_conflict) {
+ case ASB_CALL_HELPER:
+ drbd_maybe_khelper(device, connection, "pri-lost");
+ fallthrough;
+ case ASB_DISCONNECT:
+ case ASB_RETRY_CONNECT:
+ drbd_err(peer_device, "I shall become SyncTarget, but I am primary!\n");
+ strategy = rr_conflict == ASB_RETRY_CONNECT ?
+ SYNC_TARGET_PRIMARY_RECONNECT : SYNC_TARGET_PRIMARY_DISCONNECT;
+ break;
+ case ASB_VIOLENTLY:
+ drbd_warn(peer_device, "Becoming SyncTarget, violating the stable-data"
+ "assumption\n");
+ break;
+ case ASB_AUTO_DISCARD:
+ if (strategy == SYNC_TARGET_USE_BITMAP && rule == RULE_CRASHED_PRIMARY) {
+ drbd_warn(peer_device, "reversing resync by auto-discard\n");
+ strategy = SYNC_SOURCE_USE_BITMAP;
+ }
+ }
+ }
+ if (strategy == SYNC_SOURCE_USE_BITMAP && rule == RULE_CRASHED_PRIMARY &&
+ peer_role == R_PRIMARY && peer_disk_state >= D_CONSISTENT &&
+ rr_conflict == ASB_AUTO_DISCARD) {
+ drbd_warn(peer_device, "reversing resync by auto-discard\n");
+ strategy = SYNC_TARGET_USE_BITMAP;
+ }
+
+ if (rule == RULE_SYNC_SOURCE_MISSED_FINISH || rule == RULE_SYNC_SOURCE_PEER_MISSED_FINISH ||
+ rule == RULE_SYNC_TARGET_MISSED_FINISH || rule == RULE_SYNC_TARGET_PEER_MISSED_FINISH) {
+ if (strategy == SYNC_SOURCE_USE_BITMAP) {
+ enum drbd_disk_state disk_state = peer_device->comm_state.disk;
+
+ if (disk_state == D_NEGOTIATING)
+ disk_state = disk_state_from_md(device);
+ if (disk_state != D_UP_TO_DATE) {
+ drbd_info(peer_device,
+ "Resync (rule=%s) skipped: sync-source (%s)\n",
+ drbd_sync_rule_str(rule), drbd_disk_str(disk_state));
+ strategy = NO_SYNC;
+ }
+ } else if (strategy == SYNC_TARGET_USE_BITMAP) {
+ if (peer_disk_state != D_UP_TO_DATE) {
+ struct drbd_peer_md *peer_md =
+ &device->ldev->md.peers[peer_device->node_id];
+ u64 previous = peer_md->bitmap_uuid;
+
+ if (previous) {
+ drbd_set_peer_bitmap_uuid(peer_md, 0, 0);
+ _drbd_uuid_push_history(device, previous);
+ drbd_md_mark_dirty(device);
+ }
+ drbd_info(peer_device,
+ "Resync (rule=%s) skipped: peer sync-source (%s)\n",
+ drbd_sync_rule_str(rule), drbd_disk_str(peer_disk_state));
+ strategy = NO_SYNC;
+ }
+ }
+ }
+
+ if (test_bit(CONN_DRY_RUN, &connection->flags)) {
+ if (strategy == NO_SYNC)
+ drbd_info(peer_device, "dry-run connect: No resync, would become Connected immediately.\n");
+ else
+ drbd_info(peer_device, "dry-run connect: Would become %s, doing a %s resync.",
+ drbd_repl_str(strategy_descriptor(strategy).is_sync_target ? L_SYNC_TARGET : L_SYNC_SOURCE),
+ strategy_descriptor(strategy).name);
+ return DRY_RUN;
+ }
+
+ err = bitmap_mod_after_handshake(peer_device, strategy, peer_node_id);
+ if (err)
+ return RETRY_CONNECT;
+
+ return strategy;
+}
+
+static enum drbd_after_sb_p convert_after_sb(enum drbd_after_sb_p peer)
+{
+ /* ASB_DISCARD_REMOTE - ASB_DISCARD_LOCAL is valid */
+ if (peer == ASB_DISCARD_REMOTE)
+ return ASB_DISCARD_LOCAL;
+
+ /* any other things with ASB_DISCARD_REMOTE or ASB_DISCARD_LOCAL are invalid */
+ if (peer == ASB_DISCARD_LOCAL)
+ return ASB_DISCARD_REMOTE;
+
+ /* everything else is valid if they are equal on both sides. */
+ return peer;
+}
+
+static int receive_protocol(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct p_protocol *p = pi->data;
+ enum drbd_after_sb_p p_after_sb_0p, p_after_sb_1p, p_after_sb_2p;
+ int p_proto, p_discard_my_data, p_two_primaries, cf;
+ struct drbd_net_conf *nc, *old_net_conf, *new_net_conf = NULL;
+ char integrity_alg[SHARED_SECRET_MAX] = "";
+ struct crypto_shash *peer_integrity_tfm = NULL;
+ void *int_dig_in = NULL, *int_dig_vv = NULL;
+
+ p_proto = be32_to_cpu(p->protocol);
+ p_after_sb_0p = be32_to_cpu(p->after_sb_0p);
+ p_after_sb_1p = be32_to_cpu(p->after_sb_1p);
+ p_after_sb_2p = be32_to_cpu(p->after_sb_2p);
+ p_two_primaries = be32_to_cpu(p->two_primaries);
+ cf = be32_to_cpu(p->conn_flags);
+ p_discard_my_data = cf & CF_DISCARD_MY_DATA;
+
+ if (connection->agreed_pro_version >= 87) {
+ int err;
+
+ if (pi->size > sizeof(integrity_alg))
+ return -EIO;
+ err = drbd_recv_into(connection, integrity_alg, pi->size);
+ if (err)
+ return err;
+ integrity_alg[SHARED_SECRET_MAX - 1] = 0;
+ }
+
+ if (pi->cmd != P_PROTOCOL_UPDATE) {
+ if (cf & CF_DRY_RUN)
+ set_bit(CONN_DRY_RUN, &connection->flags);
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+
+ if (p_proto != nc->wire_protocol) {
+ drbd_err(connection, "incompatible %s settings\n", "protocol");
+ goto disconnect_rcu_unlock;
+ }
+
+ if (convert_after_sb(p_after_sb_0p) != nc->after_sb_0p) {
+ drbd_err(connection, "incompatible %s settings\n", "after-sb-0pri");
+ goto disconnect_rcu_unlock;
+ }
+
+ if (convert_after_sb(p_after_sb_1p) != nc->after_sb_1p) {
+ drbd_err(connection, "incompatible %s settings\n", "after-sb-1pri");
+ goto disconnect_rcu_unlock;
+ }
+
+ if (convert_after_sb(p_after_sb_2p) != nc->after_sb_2p) {
+ drbd_err(connection, "incompatible %s settings\n", "after-sb-2pri");
+ goto disconnect_rcu_unlock;
+ }
+
+ if (p_discard_my_data && test_bit(CONN_DISCARD_MY_DATA, &connection->flags)) {
+ drbd_err(connection, "incompatible %s settings\n", "discard-my-data");
+ goto disconnect_rcu_unlock;
+ }
+
+ if (p_two_primaries != nc->two_primaries) {
+ drbd_err(connection, "incompatible %s settings\n", "allow-two-primaries");
+ goto disconnect_rcu_unlock;
+ }
+
+ if (strcmp(integrity_alg, nc->integrity_alg)) {
+ drbd_err(connection, "incompatible %s settings\n", "data-integrity-alg");
+ goto disconnect_rcu_unlock;
+ }
+
+ rcu_read_unlock();
+ }
+
+ if (integrity_alg[0]) {
+ int hash_size;
+
+ /*
+ * We can only change the peer data integrity algorithm
+ * here. Changing our own data integrity algorithm
+ * requires that we send a P_PROTOCOL_UPDATE packet at
+ * the same time; otherwise, the peer has no way to
+ * tell between which packets the algorithm should
+ * change.
+ */
+
+ peer_integrity_tfm = crypto_alloc_shash(integrity_alg, 0, 0);
+ if (IS_ERR(peer_integrity_tfm)) {
+ peer_integrity_tfm = NULL;
+ drbd_err(connection, "peer data-integrity-alg %s not supported\n",
+ integrity_alg);
+ goto disconnect;
+ }
+
+ hash_size = crypto_shash_digestsize(peer_integrity_tfm);
+ int_dig_in = kmalloc(hash_size, GFP_KERNEL);
+ int_dig_vv = kmalloc(hash_size, GFP_KERNEL);
+ if (!(int_dig_in && int_dig_vv)) {
+ drbd_err(connection, "Allocation of buffers for data integrity checking failed\n");
+ goto disconnect;
+ }
+ }
+
+ new_net_conf = kmalloc_obj(struct drbd_net_conf);
+ if (!new_net_conf)
+ goto disconnect;
+
+ if (mutex_lock_interruptible(&connection->resource->conf_update)) {
+ drbd_err(connection, "Interrupted while waiting for conf_update\n");
+ goto disconnect;
+ }
+
+ mutex_lock(&connection->mutex[DATA_STREAM]);
+ old_net_conf = connection->transport.net_conf;
+ *new_net_conf = *old_net_conf;
+
+ new_net_conf->wire_protocol = p_proto;
+ new_net_conf->after_sb_0p = convert_after_sb(p_after_sb_0p);
+ new_net_conf->after_sb_1p = convert_after_sb(p_after_sb_1p);
+ new_net_conf->after_sb_2p = convert_after_sb(p_after_sb_2p);
+ new_net_conf->two_primaries = p_two_primaries;
+
+ rcu_assign_pointer(connection->transport.net_conf, new_net_conf);
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+ mutex_unlock(&connection->resource->conf_update);
+
+ crypto_free_shash(connection->peer_integrity_tfm);
+ kfree(connection->int_dig_in);
+ kfree(connection->int_dig_vv);
+ connection->peer_integrity_tfm = peer_integrity_tfm;
+ connection->int_dig_in = int_dig_in;
+ connection->int_dig_vv = int_dig_vv;
+
+ if (strcmp(old_net_conf->integrity_alg, integrity_alg))
+ drbd_info(connection, "peer data-integrity-alg: %s\n",
+ integrity_alg[0] ? integrity_alg : "(none)");
+
+ kvfree_rcu_mightsleep(old_net_conf);
+ return 0;
+
+disconnect_rcu_unlock:
+ rcu_read_unlock();
+disconnect:
+ kfree(new_net_conf);
+ crypto_free_shash(peer_integrity_tfm);
+ kfree(int_dig_in);
+ kfree(int_dig_vv);
+ change_cstate(connection, C_DISCONNECTING, CS_HARD);
+ return -EIO;
+}
+
+/* helper function
+ * input: alg name, feature name
+ * return: NULL (alg name was "")
+ * ERR_PTR(error) if something goes wrong
+ * or the crypto hash ptr, if it worked out ok. */
+static struct crypto_shash *drbd_crypto_alloc_digest_safe(const struct drbd_device *device,
+ const char *alg, const char *name)
+{
+ struct crypto_shash *tfm;
+
+ if (!alg[0])
+ return NULL;
+
+ tfm = crypto_alloc_shash(alg, 0, 0);
+ if (IS_ERR(tfm)) {
+ drbd_err(device, "Can not allocate \"%s\" as %s (reason: %ld)\n",
+ alg, name, PTR_ERR(tfm));
+ return tfm;
+ }
+ return tfm;
+}
+
+/* Receive P_SYNC_PARAM89 and the older P_SYNC_PARAM. The peer_device fields
+ * related to resync configuration are ignored. These include resync_rate,
+ * c_max_rate and the like. We ignore them because applying them to our own
+ * configuration would be confusing. It would cause us to swap configuration
+ * with our peer each time we connected. */
+static int receive_SyncParam(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+ struct p_rs_param_95 *p;
+ unsigned int header_size, data_size, exp_max_sz;
+ struct crypto_shash *verify_tfm = NULL;
+ struct crypto_shash *csums_tfm = NULL;
+ struct drbd_net_conf *old_net_conf, *new_net_conf = NULL;
+ struct drbd_peer_device_conf *old_peer_device_conf = NULL;
+ const int apv = connection->agreed_pro_version;
+ struct fifo_buffer *old_plan = NULL, *new_plan = NULL;
+ struct drbd_resource *resource = connection->resource;
+ int err;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return config_unknown_volume(connection, pi);
+ device = peer_device->device;
+
+ exp_max_sz = apv <= 87 ? sizeof(struct p_rs_param)
+ : apv == 88 ? sizeof(struct p_rs_param)
+ + SHARED_SECRET_MAX
+ : apv <= 94 ? sizeof(struct p_rs_param_89)
+ : /* apv >= 95 */ sizeof(struct p_rs_param_95);
+
+ if (pi->size > exp_max_sz) {
+ drbd_err(device, "SyncParam packet too long: received %u, expected <= %u bytes\n",
+ pi->size, exp_max_sz);
+ return -EIO;
+ }
+
+ if (apv <= 88) {
+ header_size = sizeof(struct p_rs_param);
+ data_size = pi->size - header_size;
+ } else if (apv <= 94) {
+ header_size = sizeof(struct p_rs_param_89);
+ data_size = pi->size - header_size;
+ D_ASSERT(device, data_size == 0);
+ } else {
+ header_size = sizeof(struct p_rs_param_95);
+ data_size = pi->size - header_size;
+ D_ASSERT(device, data_size == 0);
+ }
+
+ err = drbd_recv_all(connection, (void **)&p, header_size + data_size);
+ if (err)
+ return err;
+
+ err = mutex_lock_interruptible(&resource->conf_update);
+ if (err) {
+ drbd_err(connection, "Interrupted while waiting for conf_update\n");
+ return err;
+ }
+ old_net_conf = connection->transport.net_conf;
+
+ if (apv >= 88) {
+ if (apv == 88) {
+ if (data_size > SHARED_SECRET_MAX || data_size == 0) {
+ drbd_err(device, "verify-alg too long, "
+ "peer wants %u, accepting only %u byte\n",
+ data_size, SHARED_SECRET_MAX);
+ goto reconnect;
+ }
+ p->verify_alg[data_size] = 0;
+
+ } else /* apv >= 89 */ {
+ /* we still expect NUL terminated strings */
+ /* but just in case someone tries to be evil */
+ D_ASSERT(device, p->verify_alg[SHARED_SECRET_MAX-1] == 0);
+ D_ASSERT(device, p->csums_alg[SHARED_SECRET_MAX-1] == 0);
+ p->verify_alg[SHARED_SECRET_MAX-1] = 0;
+ p->csums_alg[SHARED_SECRET_MAX-1] = 0;
+ }
+
+ strscpy(connection->peer_verify_alg, p->verify_alg,
+ sizeof(connection->peer_verify_alg));
+
+ if (strcmp(old_net_conf->verify_alg, p->verify_alg)) {
+ if (peer_device->repl_state[NOW] == L_OFF) {
+ drbd_warn(device, "Different verify-alg settings. me=\"%s\" peer=\"%s\", online verify will be refused\n",
+ old_net_conf->verify_alg, p->verify_alg);
+ } else {
+ verify_tfm = drbd_crypto_alloc_digest_safe(device,
+ p->verify_alg, "verify-alg");
+ if (IS_ERR(verify_tfm)) {
+ verify_tfm = NULL;
+ goto disconnect;
+ }
+ }
+ }
+
+ if (apv >= 89 && strcmp(old_net_conf->csums_alg, p->csums_alg)) {
+ if (peer_device->repl_state[NOW] == L_OFF) {
+ drbd_err(device, "Different csums-alg settings. me=\"%s\" peer=\"%s\"\n",
+ old_net_conf->csums_alg, p->csums_alg);
+ goto disconnect;
+ }
+ csums_tfm = drbd_crypto_alloc_digest_safe(device,
+ p->csums_alg, "csums-alg");
+ if (IS_ERR(csums_tfm)) {
+ csums_tfm = NULL;
+ goto disconnect;
+ }
+ }
+
+ if (verify_tfm || csums_tfm) {
+ new_net_conf = kzalloc_obj(struct drbd_net_conf);
+ if (!new_net_conf)
+ goto disconnect;
+
+ *new_net_conf = *old_net_conf;
+
+ if (verify_tfm) {
+ strscpy(new_net_conf->verify_alg, p->verify_alg);
+ new_net_conf->verify_alg_len = strlen(p->verify_alg) + 1;
+ crypto_free_shash(connection->verify_tfm);
+ connection->verify_tfm = verify_tfm;
+ drbd_info(device, "using verify-alg: \"%s\"\n", p->verify_alg);
+ }
+ if (csums_tfm) {
+ strscpy(new_net_conf->csums_alg, p->csums_alg);
+ new_net_conf->csums_alg_len = strlen(p->csums_alg) + 1;
+ crypto_free_shash(connection->csums_tfm);
+ connection->csums_tfm = csums_tfm;
+ drbd_info(device, "using csums-alg: \"%s\"\n", p->csums_alg);
+ }
+ rcu_assign_pointer(connection->transport.net_conf, new_net_conf);
+ }
+ }
+
+ if (new_plan)
+ rcu_assign_pointer(peer_device->rs_plan_s, new_plan);
+
+ mutex_unlock(&resource->conf_update);
+ synchronize_rcu();
+ if (new_net_conf)
+ kfree(old_net_conf);
+ kfree(old_peer_device_conf);
+ if (new_plan)
+ kfree(old_plan);
+
+ return 0;
+
+reconnect:
+ mutex_unlock(&resource->conf_update);
+ return -EIO;
+
+disconnect:
+ kfree(new_plan);
+ mutex_unlock(&resource->conf_update);
+ /* just for completeness: actually not needed,
+ * as this is not reached if csums_tfm was ok. */
+ crypto_free_shash(csums_tfm);
+ /* but free the verify_tfm again, if csums_tfm did not work out */
+ crypto_free_shash(verify_tfm);
+ change_cstate(connection, C_DISCONNECTING, CS_HARD);
+ return -EIO;
+}
+
+static void drbd_setup_order_type(struct drbd_device *device, int peer)
+{
+ /* sorry, we currently have no working implementation
+ * of distributed TCQ */
+}
+
+/* warn if the arguments differ by more than 12.5% */
+static void warn_if_differ_considerably(struct drbd_peer_device *peer_device,
+ const char *s, sector_t a, sector_t b)
+{
+ sector_t d;
+
+ if (a == 0 || b == 0)
+ return;
+ d = (a > b) ? (a - b) : (b - a);
+ if (d > (a>>3) || d > (b>>3))
+ drbd_warn(peer_device, "Considerable difference in %s: %llus vs. %llus\n", s,
+ (unsigned long long)a, (unsigned long long)b);
+}
+
+static bool drbd_other_peer_smaller(struct drbd_peer_device *reference_peer_device, uint64_t new_size)
+{
+ struct drbd_device *device = reference_peer_device->device;
+ struct drbd_peer_device *peer_device;
+ bool smaller = false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device == reference_peer_device)
+ continue;
+
+ /* Ignore peers without an attached disk. */
+ if (peer_device->disk_state[NOW] < D_INCONSISTENT)
+ continue;
+
+ if (peer_device->d_size != 0 && peer_device->d_size < new_size)
+ smaller = true;
+ }
+ rcu_read_unlock();
+
+ return smaller;
+}
+
+static bool other_peer_has_usable_data(struct drbd_peer_device *reference_peer_device)
+{
+ struct drbd_device *device = reference_peer_device->device;
+ struct drbd_peer_device *peer_device;
+ bool usable = false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device == reference_peer_device)
+ continue;
+
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED &&
+ peer_device->disk_state[NOW] >= D_OUTDATED)
+ usable = true;
+ }
+ rcu_read_unlock();
+
+ return usable;
+}
+
+/* Maximum bio size that a protocol version supports. */
+static unsigned int conn_max_bio_size(struct drbd_connection *connection)
+{
+ if (connection->agreed_pro_version >= 100)
+ return DRBD_MAX_BIO_SIZE;
+ else if (connection->agreed_pro_version >= 95)
+ return DRBD_MAX_BIO_SIZE_P95;
+ else
+ return DRBD_MAX_SIZE_H80_PACKET;
+}
+
+static bool is_valid_block_size(unsigned int size)
+{
+ return size >= SECTOR_SIZE && size <= DRBD_MAX_BIO_SIZE && is_power_of_2(size);
+}
+
+static bool peer_block_sizes_are_sane(struct drbd_peer_device *peer_device, struct p_sizes *p)
+{
+ unsigned int logical_block_size, physical_block_size;
+
+ /* Older peers do not send queue limits at all. */
+ if (!(peer_device->connection->agreed_features & DRBD_FF_WSAME))
+ return true;
+
+ logical_block_size = be32_to_cpu(p->qlim->logical_block_size);
+ physical_block_size = be32_to_cpu(p->qlim->physical_block_size);
+
+ if (!is_valid_block_size(logical_block_size) ||
+ !is_valid_block_size(physical_block_size)) {
+ drbd_err(peer_device,
+ "Peer sent bogus block sizes (logical:%u physical:%u), disconnecting\n",
+ logical_block_size, physical_block_size);
+ return false;
+ }
+
+ return true;
+}
+
+static struct drbd_peer_device *get_neighbor_device(struct drbd_device *device,
+ enum drbd_neighbor neighbor)
+{
+ s32 self_id, peer_id, pivot;
+ struct drbd_peer_device *peer_device, *peer_device_ret = NULL;
+
+ if (!get_ldev(device))
+ return NULL;
+ self_id = device->ldev->md.node_id;
+ put_ldev(device);
+
+ pivot = neighbor == NEXT_LOWER ? 0 : neighbor == NEXT_HIGHER ? S32_MAX : -1;
+ if (pivot == -1)
+ return NULL;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ bool found_new = false;
+
+ peer_id = peer_device->node_id;
+
+ if (neighbor == NEXT_LOWER && peer_id < self_id && peer_id >= pivot)
+ found_new = true;
+ else if (neighbor == NEXT_HIGHER && peer_id > self_id && peer_id <= pivot)
+ found_new = true;
+
+ if (found_new && peer_device->disk_state[NOW] >= D_INCONSISTENT) {
+ pivot = peer_id;
+ peer_device_ret = peer_device;
+ }
+ }
+ rcu_read_unlock();
+
+ return peer_device_ret;
+}
+
+static void maybe_trigger_resync(struct drbd_device *device, struct drbd_peer_device *peer_device, bool grew, bool skip)
+{
+ if (!peer_device)
+ return;
+ if (peer_device->repl_state[NOW] <= L_OFF)
+ return;
+ if (test_and_clear_bit(RESIZE_PENDING, peer_device->flags) ||
+ (grew && peer_device->repl_state[NOW] == L_ESTABLISHED)) {
+ if (peer_device->disk_state[NOW] >= D_INCONSISTENT &&
+ device->disk_state[NOW] >= D_INCONSISTENT) {
+ if (skip)
+ drbd_info(peer_device, "Resync of new storage suppressed with --assume-clean\n");
+ else
+ resync_after_online_grow(peer_device);
+ } else
+ set_bit(RESYNC_AFTER_NEG, peer_device->flags);
+ }
+}
+
+static int receive_sizes(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device, *peer_device_it = NULL;
+ struct drbd_device *device;
+ struct p_sizes *p = pi->data;
+ uint64_t p_size, p_usize, p_csize;
+ uint64_t my_usize, my_max_size, cur_size;
+ enum determine_dev_size dd = DS_UNCHANGED;
+ bool should_send_sizes = false;
+ enum dds_flags ddsf;
+ unsigned int protocol_max_bio_size;
+ bool have_ldev = false;
+ bool have_mutex = false;
+ bool is_handshake;
+ int err;
+ u64 im;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return config_unknown_volume(connection, pi);
+ device = peer_device->device;
+
+ err = mutex_lock_interruptible(&connection->resource->conf_update);
+ if (err) {
+ drbd_err(connection, "Interrupted while waiting for conf_update\n");
+ goto out;
+ }
+ have_mutex = true;
+
+ if (!peer_block_sizes_are_sane(peer_device, p))
+ goto disconnect;
+
+ /* just store the peer's disk size for now.
+ * we still need to figure out whether we accept that. */
+ p_size = be64_to_cpu(p->d_size);
+ p_usize = be64_to_cpu(p->u_size);
+ p_csize = be64_to_cpu(p->c_size);
+
+ peer_device->d_size = p_size;
+ peer_device->u_size = p_usize;
+ peer_device->c_size = p_csize;
+
+ /* Ignore "current" size for calculating "max" size. */
+ /* If it used to have a disk, but now is detached, don't revert back to zero. */
+ if (p_size)
+ peer_device->max_size = p_size;
+
+ cur_size = get_capacity(device->vdisk);
+ dynamic_drbd_dbg(device, "current_size: %llu\n", (unsigned long long)cur_size);
+ dynamic_drbd_dbg(peer_device, "c_size: %llu u_size: %llu d_size: %llu max_size: %llu\n",
+ (unsigned long long)p_csize,
+ (unsigned long long)p_usize,
+ (unsigned long long)p_size,
+ (unsigned long long)peer_device->max_size);
+
+ if ((p_size && p_csize > p_size) || (p_usize && p_csize > p_usize)) {
+ drbd_warn(peer_device, "Peer sent bogus sizes, disconnecting\n");
+ goto disconnect;
+ }
+
+ /* The protocol version limits how big requests can be. In addition,
+ * peers before protocol version 94 cannot split large requests into
+ * multiple bios; their reported max_bio_size is a hard limit.
+ */
+ protocol_max_bio_size = conn_max_bio_size(connection);
+ peer_device->q_limits.max_bio_size = min(be32_to_cpu(p->max_bio_size),
+ protocol_max_bio_size);
+ ddsf = be16_to_cpu(p->dds_flags);
+ is_handshake = (peer_device->repl_state[NOW] == L_OFF);
+ set_bit(HAVE_SIZES, peer_device->flags);
+
+ if (get_ldev(device)) {
+ sector_t new_size;
+
+ have_ldev = true;
+
+ rcu_read_lock();
+ my_usize = rcu_dereference(device->ldev->disk_conf)->disk_size;
+ rcu_read_unlock();
+
+ my_max_size = drbd_get_max_capacity(device, device->ldev, false);
+ dynamic_drbd_dbg(peer_device, "la_size: %llu my_usize: %llu my_max_size: %llu\n",
+ (unsigned long long)device->ldev->md.effective_size,
+ (unsigned long long)my_usize,
+ (unsigned long long)my_max_size);
+
+ if (peer_device->disk_state[NOW] > D_DISKLESS)
+ warn_if_differ_considerably(peer_device, "lower level device sizes",
+ p_size, my_max_size);
+ warn_if_differ_considerably(peer_device, "user requested size",
+ p_usize, my_usize);
+
+ if (is_handshake)
+ p_usize = min_not_zero(my_usize, p_usize);
+
+ if (p_usize == 0) {
+ /* Peer may reset usize to zero only if it has a backend.
+ * Because a diskless node has no disk config,
+ * and always sends zero. */
+ if (p_size == 0)
+ p_usize = my_usize;
+ }
+
+ new_size = drbd_new_dev_size(device, 0, p_usize, ddsf);
+
+ /* Never shrink a device with usable data during connect,
+ * or "attach" on the peer. That includes usable data on
+ * another connected peer, our current size comes from it.
+ * But allow online shrinking if we are connected. */
+ if (new_size < cur_size &&
+ (device->disk_state[NOW] >= D_OUTDATED ||
+ other_peer_has_usable_data(peer_device)) &&
+ (peer_device->repl_state[NOW] < L_ESTABLISHED || peer_device->disk_state[NOW] == D_DISKLESS)) {
+ drbd_err(peer_device, "The peer's disk size is too small! (%llu < %llu sectors)\n",
+ (unsigned long long)new_size, (unsigned long long)cur_size);
+ goto disconnect;
+ }
+
+ /* Disconnect, if we cannot grow to the peer's current size */
+ if (my_max_size < p_csize && !is_handshake) {
+ drbd_err(peer_device, "Peer's size larger than my maximum capacity (%llu < %llu sectors)\n",
+ (unsigned long long)my_max_size, (unsigned long long)p_csize);
+ goto disconnect;
+ }
+
+ if (my_usize != p_usize) {
+ struct drbd_disk_conf *old_disk_conf, *new_disk_conf;
+
+ new_disk_conf = kzalloc_obj(struct drbd_disk_conf);
+ if (!new_disk_conf) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ old_disk_conf = device->ldev->disk_conf;
+ *new_disk_conf = *old_disk_conf;
+ new_disk_conf->disk_size = p_usize;
+
+ rcu_assign_pointer(device->ldev->disk_conf, new_disk_conf);
+ kvfree_rcu_mightsleep(old_disk_conf);
+
+ drbd_info(peer_device, "Peer sets u_size to %llu sectors (old: %llu)\n",
+ (unsigned long long)p_usize, (unsigned long long)my_usize);
+ /* Do not set should_send_sizes here. That might cause packet storms */
+ }
+ }
+
+ if (connection->agreed_features & DRBD_FF_WSAME) {
+ struct o_qlim *qlim = p->qlim;
+
+ peer_device->q_limits.physical_block_size = be32_to_cpu(qlim->physical_block_size);
+ peer_device->q_limits.logical_block_size = be32_to_cpu(qlim->logical_block_size);
+ peer_device->q_limits.alignment_offset = be32_to_cpu(qlim->alignment_offset);
+ peer_device->q_limits.io_min = be32_to_cpu(qlim->io_min);
+ peer_device->q_limits.io_opt = be32_to_cpu(qlim->io_opt);
+ }
+
+ if (connection->agreed_features & DRBD_FF_BM_BLOCK_SHIFT) {
+ peer_device->bm_block_shift =
+ p->qlim->bm_block_shift_minus_12 + BM_BLOCK_SHIFT_4k;
+ } else {
+ int bbs = have_ldev ? bm_block_size(device->bitmap) : BM_BLOCK_SIZE_4k;
+ /* May work as long as this one is SyncTarget. May result in
+ * funny never ending / repeating resyncs if the other guy is
+ * SyncTarget, but unaware of bitmap granularity issues.
+ */
+ if (bbs != BM_BLOCK_SIZE_4k)
+ drbd_warn(peer_device,
+ "My bitmap granularity is %u. Upgrade this peer to make it aware.\n",
+ bbs);
+ peer_device->bm_block_shift = BM_BLOCK_SHIFT_4k;
+ }
+
+ /* Leave drbd_reconsider_queue_parameters() before drbd_determine_dev_size().
+ In case we cleared the QUEUE_FLAG_DISCARD from our queue in
+ drbd_reconsider_queue_parameters(), we can be sure that after
+ drbd_determine_dev_size() no REQ_OP_DISCARDs are in the queue. */
+ if (have_ldev) {
+ enum dds_flags local_ddsf = ddsf;
+
+ drbd_reconsider_queue_parameters(device, device->ldev);
+
+ /* To support thinly provisioned nodes (partial resync) joining later,
+ clear all bitmap slots, including the unused ones. */
+ if (device->ldev->md.effective_size == 0)
+ local_ddsf |= DDSF_NO_RESYNC;
+
+ dd = drbd_determine_dev_size(device, p_csize, local_ddsf, NULL);
+
+ if (dd == DS_GREW || dd == DS_SHRUNK)
+ should_send_sizes = true;
+
+ if (dd == DS_ERROR) {
+ err = -EIO;
+ goto out;
+ }
+ drbd_md_sync_if_dirty(device);
+ } else {
+ uint64_t new_size = 0;
+
+ drbd_reconsider_queue_parameters(device, NULL);
+ /* In case I am diskless, need to accept the peer's *current* size.
+ *
+ * At this point, the peer knows more about my disk, or at
+ * least about what we last agreed upon, than myself.
+ * So if his c_size is less than his d_size, the most likely
+ * reason is that *my* d_size was smaller last time we checked,
+ * or some other peer does not (yet) have enough room.
+ *
+ * Unless of course he does not have a disk himself.
+ * In which case we ignore this completely.
+ */
+ new_size = p_csize;
+ new_size = min_not_zero(new_size, p_usize);
+ new_size = min_not_zero(new_size, p_size);
+
+ if (new_size == 0) {
+ /* Ignore, peer does not know nothing. */
+ } else if (new_size == cur_size) {
+ /* nothing to do */
+ } else if (cur_size != 0 && p_size == 0) {
+ dynamic_drbd_dbg(peer_device,
+ "Ignored diskless peer device size (peer:%llu != me:%llu sectors)!\n",
+ (unsigned long long)new_size, (unsigned long long)cur_size);
+ } else if (new_size < cur_size && device->resource->role[NOW] == R_PRIMARY) {
+ drbd_err(peer_device,
+ "The peer's device size is too small! (%llu < %llu sectors); demote me first!\n",
+ (unsigned long long)new_size, (unsigned long long)cur_size);
+ goto disconnect;
+ } else if (drbd_other_peer_smaller(peer_device, new_size)) {
+ dynamic_drbd_dbg(peer_device,
+ "Ignored peer device size (peer:%llu sectors); other peer smaller!\n",
+ (unsigned long long)new_size);
+ } else {
+ /* I believe the peer, if
+ * - I don't have a current size myself
+ * - we agree on the size anyways
+ * - I do have a current size, am Secondary,
+ * and he has the only disk
+ * - I do have a current size, am Primary,
+ * and he has the only disk,
+ * which is larger than my current size
+ */
+ should_send_sizes = true;
+ drbd_set_my_capacity(device, new_size);
+ }
+ }
+
+ if (device->device_conf.max_bio_size > protocol_max_bio_size ||
+ (connection->agreed_pro_version < 94 &&
+ device->device_conf.max_bio_size > peer_device->q_limits.max_bio_size)) {
+ drbd_err(device, "Peer cannot deal with requests bigger than %u. "
+ "Please reduce max_bio_size in the configuration.\n",
+ peer_device->q_limits.max_bio_size);
+ goto disconnect;
+ }
+
+ if (have_ldev) {
+ if (device->ldev->known_size != drbd_get_capacity(device->ldev->backing_bdev)) {
+ device->ldev->known_size = drbd_get_capacity(device->ldev->backing_bdev);
+ should_send_sizes = true;
+ }
+
+ drbd_setup_order_type(device, be16_to_cpu(p->queue_order_type));
+ }
+
+ cur_size = get_capacity(device->vdisk);
+
+ for_each_peer_device_ref(peer_device_it, im, device) {
+ struct drbd_connection *con_it = peer_device_it->connection;
+
+ /* drop cached max_size, if we already grew beyond it */
+ if (peer_device_it->max_size < cur_size)
+ peer_device_it->max_size = 0;
+
+ if (con_it->cstate[NOW] < C_CONNECTED)
+ continue;
+
+ /* Send size updates only if something relevant has changed.
+ * TODO: only tell the sender thread to do so,
+ * or we may end up in a distributed deadlock on congestion. */
+
+ if (should_send_sizes)
+ drbd_send_sizes(peer_device_it, p_usize, ddsf);
+ }
+
+ maybe_trigger_resync(device, get_neighbor_device(device, NEXT_HIGHER),
+ dd == DS_GREW, ddsf & DDSF_NO_RESYNC);
+ maybe_trigger_resync(device, get_neighbor_device(device, NEXT_LOWER),
+ dd == DS_GREW, ddsf & DDSF_NO_RESYNC);
+ err = 0;
+
+out:
+ if (have_ldev)
+ put_ldev(device);
+ if (have_mutex)
+ mutex_unlock(&connection->resource->conf_update);
+ return err;
+
+disconnect:
+ /* don't let a rejected peer confuse future handshakes with different peers. */
+ peer_device->max_size = 0;
+
+ if (connection->resource->remote_state_change)
+ set_bit(TWOPC_RECV_SIZES_ERR, &connection->resource->flags);
+ else
+ err = -EIO;
+ goto out;
+}
+
+static enum sync_strategy resolve_splitbrain_from_disk_states(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ enum drbd_disk_state peer_disk_state = peer_device->disk_state[NOW];
+ enum drbd_disk_state disk_state = device->disk_state[NOW];
+
+ return disk_state <= D_UP_TO_DATE && peer_disk_state == D_UP_TO_DATE ? SYNC_TARGET_USE_BITMAP :
+ disk_state == D_UP_TO_DATE && peer_disk_state <= D_UP_TO_DATE ? SYNC_SOURCE_USE_BITMAP :
+ SPLIT_BRAIN_AUTO_RECOVER;
+}
+
+static void drbd_resync(struct drbd_peer_device *peer_device,
+ enum resync_reason reason)
+{
+ enum drbd_role peer_role = peer_device->connection->peer_role[NOW];
+ enum drbd_repl_state new_repl_state;
+ enum drbd_disk_state peer_disk_state, announced_pdsk;
+ enum sync_strategy strategy;
+ enum sync_rule rule;
+ int peer_node_id;
+ enum drbd_state_rv rv;
+ const char *tag = reason == AFTER_UNSTABLE ? "after-unstable" : "diskless-primary";
+
+ strategy = drbd_handshake(peer_device, &rule, &peer_node_id, reason == DISKLESS_PRIMARY);
+ if (strategy == SPLIT_BRAIN_AUTO_RECOVER && reason == AFTER_UNSTABLE)
+ strategy = resolve_splitbrain_from_disk_states(peer_device);
+
+ if (!is_strategy_determined(strategy)) {
+ drbd_info(peer_device, "Unexpected result of handshake() %s!\n", strategy_descriptor(strategy).name);
+ return;
+ }
+
+ peer_disk_state = peer_device->disk_state[NOW];
+ /* the peer decides on the disk state it announced */
+ announced_pdsk = test_bit(PEER_UPGRADE_PREDICTED, peer_device->flags) ?
+ D_OUTDATED : peer_disk_state;
+ if (reason == DISKLESS_PRIMARY)
+ disk_states_to_strategy(peer_device, peer_disk_state, &strategy,
+ &rule, &peer_node_id);
+ else
+ maybe_reconcile_equal_uuid_bitmap(peer_device, peer_device->comm_state.disk,
+ announced_pdsk, &strategy, &rule);
+
+ new_repl_state = strategy_to_repl_state(peer_device, peer_role, strategy);
+ if (new_repl_state != L_ESTABLISHED) {
+ bitmap_mod_after_handshake(peer_device, strategy, peer_node_id);
+ drbd_info(peer_device, "Becoming %s %s\n", drbd_repl_str(new_repl_state),
+ reason == AFTER_UNSTABLE ? "after unstable" : "because primary is diskless");
+ }
+
+ /* No reconciliation resync with an UpToDate peer: we hold the current
+ * generation, so we owe no post-loss reconcile to anyone.
+ * Clear RECONCILE_PENDING and return to UpToDate (do not stay Consistent)
+ */
+ if (new_repl_state == L_ESTABLISHED && peer_disk_state == D_UP_TO_DATE &&
+ peer_device->device->disk_state[NOW] == D_CONSISTENT) {
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_device *pd;
+ bool cleared = false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(pd, device)
+ if (test_and_clear_bit(RECONCILE_PENDING, pd->flags))
+ cleared = true;
+ rcu_read_unlock();
+ if (cleared)
+ drbd_reconcile_settled_try_up_to_date(device->resource);
+ }
+
+ if (new_repl_state == L_ESTABLISHED && peer_disk_state >= D_CONSISTENT &&
+ peer_device->device->disk_state[NOW] == D_OUTDATED) {
+ /* No resync with up-to-date peer -> I should be consistent or up-to-date as well.
+ Note: Former unstable (but up-to-date) nodes become consistent for a short
+ time after loosing their primary peer. Therefore consider consistent here
+ as well. */
+ drbd_info(peer_device, "Upgrading local disk to %s after unstable/weak (and no resync).\n",
+ drbd_disk_str(peer_disk_state));
+ change_disk_state(peer_device->device, peer_disk_state, CS_VERBOSE, tag, NULL);
+ return;
+ }
+
+ rv = change_repl_state(peer_device, new_repl_state, CS_VERBOSE, tag);
+ if ((rv == SS_NOTHING_TO_DO || rv == SS_RESYNC_RUNNING) &&
+ (new_repl_state == L_WF_BITMAP_S || new_repl_state == L_WF_BITMAP_T)) {
+ /* Those events might happen very quickly. In case we are still processing
+ the previous resync we need to re-enter that state. Schedule sending of
+ the bitmap here explicitly */
+ peer_device->resync_again++;
+ drbd_info(peer_device, "...postponing this until current resync finished\n");
+ }
+}
+
+/* An Outdated peer upgrades itself when we trigger it (sanitize_state()); record
+ * that before its P_STATE, or a write in between is not replicated to it.
+ */
+void drbd_predict_peer_upgrade(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ enum drbd_disk_state disk_state = device->disk_state[NOW];
+ enum sync_strategy strategy;
+ unsigned long irq_flags;
+ enum sync_rule rule;
+ int peer_node_id;
+
+ /* An 8.4 peer does not upgrade itself on an established connection. */
+ if (peer_device->connection->agreed_pro_version < 110 ||
+ disk_state < D_CONSISTENT || !get_ldev(device))
+ return;
+ strategy = drbd_handshake(peer_device, &rule, &peer_node_id, false);
+ reconcile_equal_uuid_bitmap_strategy(peer_device, disk_state, D_OUTDATED,
+ &strategy, &rule);
+ put_ldev(device);
+ if (strategy != NO_SYNC)
+ return;
+
+ begin_state_change(device->resource, &irq_flags, CS_VERBOSE);
+ if (peer_device->repl_state[NOW] != L_ESTABLISHED ||
+ peer_device->disk_state[NOW] != D_OUTDATED ||
+ device->disk_state[NOW] != disk_state) {
+ abort_state_change(device->resource, &irq_flags);
+ return;
+ }
+ __change_peer_disk_state(peer_device, disk_state);
+ set_bit(PEER_UPGRADE_PREDICTED, peer_device->flags);
+ end_state_change(device->resource, &irq_flags, "peer-upgrade");
+}
+
+static void update_bitmap_slot_of_peer(struct drbd_peer_device *peer_device, int node_id, u64 bitmap_uuid)
+{
+ if (peer_device->bitmap_uuids[node_id] && bitmap_uuid == 0) {
+ /* If we learn from a neighbor that it no longer has a bitmap
+ against a third node, we need to deduce from that knowledge
+ that in the other direction the bitmap was cleared as well.
+ */
+ struct drbd_peer_device *peer_device2;
+
+ rcu_read_lock();
+ peer_device2 = peer_device_by_node_id(peer_device->device, node_id);
+ if (peer_device2) {
+ int node_id2 = peer_device->connection->peer_node_id;
+
+ peer_device2->bitmap_uuids[node_id2] = 0;
+ }
+ rcu_read_unlock();
+ }
+
+ peer_device->bitmap_uuids[node_id] = bitmap_uuid;
+}
+
+static void propagate_skip_initial_to_diskless(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ u64 im;
+
+ for_each_peer_device_ref(peer_device, im, device) {
+ if (peer_device->disk_state[NOW] == D_DISKLESS) {
+ drbd_send_uuids(peer_device, UUID_FLAG_SKIP_INITIAL_SYNC, 0);
+ peer_device->current_uuid = drbd_current_uuid(device);
+ }
+ }
+}
+
+static int __receive_uuids(struct drbd_peer_device *peer_device, u64 node_mask)
+{
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+ struct drbd_device *device = peer_device->device;
+ struct drbd_resource *resource = device->resource;
+ int updated_uuids = 0, err = 0;
+ bool bad_server, uuid_match;
+ struct drbd_net_conf *nc;
+ bool two_primaries_allowed;
+
+ uuid_match =
+ (device->exposed_data_uuid & ~UUID_PRIMARY) ==
+ (peer_device->current_uuid & ~UUID_PRIMARY);
+ bad_server =
+ repl_state < L_ESTABLISHED &&
+ device->disk_state[NOW] < D_INCONSISTENT &&
+ device->resource->role[NOW] == R_PRIMARY && !uuid_match;
+
+ if (peer_device->connection->agreed_pro_version < 110 && bad_server) {
+ drbd_err(device, "Can only connect to data with current UUID=%016llX\n",
+ (unsigned long long)device->exposed_data_uuid);
+ change_cstate(peer_device->connection, C_DISCONNECTING, CS_HARD);
+ return -EIO;
+ }
+
+ rcu_read_lock();
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ two_primaries_allowed = nc && nc->two_primaries;
+ rcu_read_unlock();
+
+ if (get_ldev(device)) {
+ bool skip_initial_sync =
+ repl_state == L_ESTABLISHED &&
+ peer_device->connection->agreed_pro_version >= 90 &&
+ drbd_current_uuid(device) == UUID_JUST_CREATED &&
+ (peer_device->uuid_flags & UUID_FLAG_SKIP_INITIAL_SYNC);
+ if (skip_initial_sync) {
+ unsigned long irq_flags;
+
+ drbd_info(device, "Accepted new current UUID, preparing to skip initial sync\n");
+ drbd_bitmap_io(device, &drbd_bmio_clear_all_n_write,
+ "clear_n_write from receive_uuids",
+ BM_LOCK_SET | BM_LOCK_CLEAR | BM_LOCK_BULK, NULL);
+ _drbd_uuid_set_current(device, peer_device->current_uuid);
+ peer_device->comm_current_uuid = peer_device->current_uuid;
+ peer_device->comm_uuid_flags = peer_device->uuid_flags;
+ peer_device->comm_bitmap_uuid = 0;
+ _drbd_uuid_set_bitmap(peer_device, 0);
+ begin_state_change(device->resource, &irq_flags, CS_VERBOSE);
+ __change_disk_state(device, D_UP_TO_DATE);
+ __change_peer_disk_state(peer_device, D_UP_TO_DATE);
+ end_state_change(device->resource, &irq_flags, "skip-initial-sync");
+ updated_uuids = 1;
+ propagate_skip_initial_to_diskless(device);
+ }
+
+ if (peer_device->uuid_flags & UUID_FLAG_NEW_DATAGEN)
+ drbd_uuid_received_new_current(peer_device, peer_device->current_uuid, node_mask);
+
+ drbd_uuid_detect_finished_resyncs(peer_device);
+
+ drbd_md_sync_if_dirty(device);
+ put_ldev(device);
+ } else if (device->disk_state[NOW] < D_INCONSISTENT && repl_state >= L_ESTABLISHED &&
+ peer_device->disk_state[NOW] == D_UP_TO_DATE && !uuid_match &&
+ (resource->role[NOW] == R_SECONDARY ||
+ (two_primaries_allowed &&
+ drbd_gen_obligation_discharge_by_adoption(device)))) {
+
+ write_lock_irq(&resource->state_rwlock);
+ if (resource->remote_state_change) {
+ drbd_info(peer_device, "Delaying update of exposed data uuid\n");
+ device->next_exposed_data_uuid = peer_device->current_uuid;
+ } else {
+ updated_uuids =
+ drbd_uuid_set_exposed(device, peer_device->current_uuid, false);
+ }
+ write_unlock_irq(&resource->state_rwlock);
+
+ }
+
+ if (device->disk_state[NOW] == D_DISKLESS && uuid_match &&
+ peer_device->disk_state[NOW] == D_CONSISTENT) {
+ drbd_info(peer_device, "Peer is on same UUID now\n");
+ change_peer_disk_state(peer_device, D_UP_TO_DATE, CS_VERBOSE, "receive-uuids");
+ }
+
+ if (updated_uuids)
+ drbd_print_uuids(peer_device, "receiver updated UUIDs to");
+
+ peer_device->uuid_node_mask = node_mask;
+
+ if ((repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T) &&
+ !(peer_device->uuid_flags & UUID_FLAG_STABLE) &&
+ !drbd_stable_sync_source_present(peer_device, NOW))
+ set_bit(UNSTABLE_RESYNC, peer_device->flags);
+
+ /* send notification in case UUID flags have changed */
+ drbd_broadcast_peer_device_state(peer_device);
+
+ return err;
+}
+
+/* drbd 8.4 compat */
+static int receive_uuids(struct drbd_connection *connection, struct packet_info *pi)
+{
+ const int node_id = connection->resource->res_opts.node_id;
+ struct drbd_peer_device *peer_device;
+ struct p_uuids *p = pi->data;
+ int history_uuids, i;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return config_unknown_volume(connection, pi);
+
+ history_uuids = min_t(int, HISTORY_UUIDS_V08,
+ ARRAY_SIZE(peer_device->history_uuids));
+
+ peer_device->current_uuid = be64_to_cpu(p->current_uuid);
+ peer_device->bitmap_uuids[node_id] = be64_to_cpu(p->bitmap_uuid);
+ for (i = 0; i < history_uuids; i++)
+ peer_device->history_uuids[i] = be64_to_cpu(p->history_uuids[i]);
+ for (; i < ARRAY_SIZE(peer_device->history_uuids); i++)
+ peer_device->history_uuids[i] = 0;
+ peer_device->dirty_bits = be64_to_cpu(p->dirty_bits);
+ peer_device->uuid_flags = be64_to_cpu(p->uuid_flags) | UUID_FLAG_STABLE;
+ set_bit(UUIDS_RECEIVED, peer_device->flags);
+
+ return __receive_uuids(peer_device, 0);
+}
+
+static int receive_uuids110(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct p_uuids110 *p = pi->data;
+ int bitmap_uuids, history_uuids, rest, i, pos, err;
+ u64 bitmap_uuids_mask, node_mask;
+ struct drbd_peer_md *peer_md = NULL;
+ struct drbd_device *device;
+ int not_allocated = -1;
+
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return config_unknown_volume(connection, pi);
+
+ device = peer_device->device;
+ bitmap_uuids_mask = be64_to_cpu(p->bitmap_uuids_mask);
+ if (bitmap_uuids_mask & ~(NODE_MASK(DRBD_PEERS_MAX) - 1))
+ return -EIO;
+ bitmap_uuids = hweight64(bitmap_uuids_mask);
+
+ if (pi->size / sizeof(p->other_uuids[0]) < bitmap_uuids)
+ return -EIO;
+ history_uuids = pi->size / sizeof(p->other_uuids[0]) - bitmap_uuids;
+ if (history_uuids > ARRAY_SIZE(peer_device->history_uuids))
+ history_uuids = ARRAY_SIZE(peer_device->history_uuids);
+
+ err = drbd_recv_into(connection, p->other_uuids,
+ (bitmap_uuids + history_uuids) *
+ sizeof(p->other_uuids[0]));
+ if (err)
+ return err;
+
+ rest = pi->size - (bitmap_uuids + history_uuids) * sizeof(p->other_uuids[0]);
+ if (rest) {
+ err = ignore_remaining_packet(connection, rest);
+ if (err)
+ return err;
+ }
+
+ if (get_ldev(device)) {
+ peer_md = device->ldev->md.peers;
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+ }
+
+ /* The condition below keeps a diskless primary's exposed current UUID
+ * from going backward when a stale/older value is reported (see commit
+ * 2f464b8e8 "drbd: fix exposed_uuid going backward"). But that
+ * protection must only apply to a CONFIRMED generation. If our view of
+ * this peer's current UUID was advanced only optimistically by our own
+ * UUID bump (CURRENT_UUID_UNCONFIRMED) and the peer never actually got
+ * it, the peer's handshake report is the truth -- take it, so we do not
+ * treat the peer as UpToDate at a generation it never persisted.
+ */
+ if (test_bit(CURRENT_UUID_UNCONFIRMED, peer_device->flags) ||
+ device->resource->role[NOW] != R_PRIMARY ||
+ device->disk_state[NOW] != D_DISKLESS ||
+ (peer_device->current_uuid & ~UUID_PRIMARY) !=
+ (device->exposed_data_uuid & ~UUID_PRIMARY) ||
+ (peer_device->comm_current_uuid & ~UUID_PRIMARY) !=
+ (device->exposed_data_uuid & ~UUID_PRIMARY))
+ peer_device->current_uuid = be64_to_cpu(p->current_uuid);
+
+ peer_device->dirty_bits = be64_to_cpu(p->dirty_bits);
+ peer_device->uuid_flags = be64_to_cpu(p->uuid_flags);
+ if (peer_device->uuid_flags & UUID_FLAG_HAS_UNALLOC) {
+ not_allocated = peer_device->uuid_flags >> UUID_FLAG_UNALLOC_SHIFT;
+ peer_device->uuid_flags &= ~UUID_FLAG_UNALLOC_MASK;
+ }
+
+ pos = 0;
+ for (i = 0; i < ARRAY_SIZE(peer_device->bitmap_uuids); i++) {
+ u64 bitmap_uuid;
+
+ if (bitmap_uuids_mask & NODE_MASK(i)) {
+ bitmap_uuid = be64_to_cpu(p->other_uuids[pos++]);
+
+ if (peer_md && !test_bit(__MDF_HAVE_BITMAP, &peer_md[i].flags) &&
+ i != not_allocated)
+ set_bit(__MDF_NODE_EXISTS, &peer_md[i].flags);
+ } else {
+ bitmap_uuid = -1;
+ }
+
+ update_bitmap_slot_of_peer(peer_device, i, bitmap_uuid);
+ }
+
+ for (i = 0; i < history_uuids; i++)
+ peer_device->history_uuids[i] = be64_to_cpu(p->other_uuids[pos++]);
+ while (i < ARRAY_SIZE(peer_device->history_uuids))
+ peer_device->history_uuids[i++] = 0;
+ set_bit(UUIDS_RECEIVED, peer_device->flags);
+ if (peer_md) {
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+ put_ldev(device);
+ }
+
+ node_mask = be64_to_cpu(p->node_mask);
+
+ if (peer_device->connection->peer_role[NOW] == R_PRIMARY &&
+ peer_device->uuid_flags & UUID_FLAG_STABLE)
+ check_resync_source(peer_device, node_mask);
+
+ err = __receive_uuids(peer_device, node_mask);
+
+ if (!test_bit(RECONCILIATION_RESYNC, peer_device->flags)) {
+ if (peer_device->uuid_flags & UUID_FLAG_GOT_STABLE) {
+ struct drbd_device *device = peer_device->device;
+
+ if (peer_device->repl_state[NOW] == L_ESTABLISHED &&
+ drbd_device_stable(device, NULL) && get_ldev(device)) {
+ drbd_send_uuids(peer_device, UUID_FLAG_RESYNC, 0);
+ drbd_resync(peer_device, AFTER_UNSTABLE);
+ put_ldev(device);
+ }
+ }
+
+ if (peer_device->uuid_flags & UUID_FLAG_RESYNC) {
+ if (get_ldev(device)) {
+ bool dp = peer_device->uuid_flags & UUID_FLAG_DISKLESS_PRIMARY;
+
+ drbd_resync(peer_device, dp ? DISKLESS_PRIMARY : AFTER_UNSTABLE);
+ put_ldev(device);
+ }
+ }
+ }
+
+ return err;
+}
+
+/*
+ * Nothing else restarts a resync that check_resync_source() aborted: with no
+ * other resync queued the target would stay Inconsistent next to an UpToDate
+ * Primary until a reconnect. Carry the blocks the weak source still owed over
+ * to the Primary's bitmap slot and resync from the Primary instead.
+ */
+static void resync_from_primary_after_abort(struct drbd_peer_device *primary_pd,
+ struct drbd_peer_device *weak_pd)
+{
+ struct drbd_device *device = primary_pd->device;
+ enum drbd_repl_state weak_rs = weak_pd->repl_state[NOW];
+
+ /* The weak peer started resyncing again while we drained. */
+ if (weak_rs == L_SYNC_TARGET || weak_rs == L_PAUSED_SYNC_T)
+ return;
+
+ if (primary_pd->repl_state[NOW] != L_ESTABLISHED ||
+ primary_pd->disk_state[NOW] != D_UP_TO_DATE ||
+ device->disk_state[NOW] >= D_UP_TO_DATE)
+ return;
+
+ if (!get_ldev(device))
+ return;
+
+ /* Without both slots the owed blocks cannot be carried over. */
+ if (weak_pd->bitmap_index == -1 || primary_pd->bitmap_index == -1) {
+ put_ldev(device);
+ return;
+ }
+
+ drbd_suspend_io(device, WRITE_ONLY);
+ drbd_bm_slot_lock(primary_pd, "or_slot after resync abort", BM_LOCK_BULK);
+ drbd_bm_or_slot(device, weak_pd->bitmap_index, primary_pd->bitmap_index);
+ drbd_bm_write(device, NULL);
+ drbd_bm_slot_unlock(primary_pd);
+ drbd_resume_io(device);
+
+ drbd_info(primary_pd, "Resyncing from the primary instead of the weak sync source\n");
+ drbd_send_uuids(primary_pd, UUID_FLAG_RESYNC, 0);
+ drbd_resync(primary_pd, AFTER_UNSTABLE);
+ put_ldev(device);
+}
+
+/**
+ * check_resync_source() - Abort resync if the source is weak
+ * @primary_pd: The peer device of the primary that reported @weak_nodes
+ * @weak_nodes: Mask of currently weak nodes in the cluster
+ *
+ * If a primary loses connection to a SYNC_SOURCE node from us, then we
+ * need to abort that resync. Why?
+ *
+ * When the primary sends a write, we get that and write that as well. With
+ * the peer_ack packet, we will set that as out-of-sync towards the sync
+ * source node.
+ * When the resync process finds such bits, we request outdated
+ * data from the sync source!
+ * We are stopping the resync from such an outdated source here and waiting
+ * until all the resync activity has drained (P_RS_DATA_REPLY packets).
+ */
+static void check_resync_source(struct drbd_peer_device *primary_pd, u64 weak_nodes)
+{
+ struct drbd_device *device = primary_pd->device;
+ struct drbd_peer_device *peer_device;
+ struct drbd_connection *connection;
+ int err;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+
+ if ((repl_state == L_SYNC_TARGET || repl_state == L_PAUSED_SYNC_T) &&
+ NODE_MASK(peer_device->node_id) & weak_nodes) {
+ rcu_read_unlock();
+ goto abort;
+ }
+ }
+ rcu_read_unlock();
+ return;
+abort:
+ connection = peer_device->connection;
+ drbd_info(peer_device, "My sync source became a weak node, aborting resync!\n");
+ if (change_repl_state(peer_device, L_ESTABLISHED, CS_VERBOSE, "abort-resync") < SS_SUCCESS)
+ return;
+ drbd_flush_workqueue(&connection->sender_work);
+ drbd_cancel_conflicting_resync_requests(peer_device);
+
+ /* Wait for the replies still in flight, then for their writes. */
+ err = wait_event_interruptible(device->misc_wait,
+ peer_device->repl_state[NOW] < L_ESTABLISHED ||
+ atomic_read(&peer_device->rs_pending_cnt) == 0);
+ if (!err)
+ err = wait_event_interruptible(connection->ee_wait,
+ peer_device->repl_state[NOW] < L_ESTABLISHED ||
+ atomic_read(&connection->backing_ee_cnt) == 0);
+
+ peer_device->rs_total = 0;
+ peer_device->rs_failed = 0;
+ peer_device->rs_paused = 0;
+
+ /* Interrupted, resync replies from the weak source may still arrive. */
+ if (err)
+ return;
+
+ resync_from_primary_after_abort(primary_pd, peer_device);
+}
+
+/**
+ * convert_state() - Converts the peer's view of the cluster state to our point of view
+ * @peer_state: The state as seen by the peer.
+ */
+static union drbd_state convert_state(union drbd_state peer_state)
+{
+ union drbd_state state;
+
+ static unsigned int c_tab[] = {
+ [L_OFF] = L_OFF,
+ [L_ESTABLISHED] = L_ESTABLISHED,
+
+ [L_STARTING_SYNC_S] = L_STARTING_SYNC_T,
+ [L_STARTING_SYNC_T] = L_STARTING_SYNC_S,
+ [L_WF_BITMAP_S] = L_WF_BITMAP_T,
+ [L_WF_BITMAP_T] = L_WF_BITMAP_S,
+ [C_DISCONNECTING] = C_TEAR_DOWN, /* C_NETWORK_FAILURE, */
+ [C_CONNECTING] = C_CONNECTING,
+ [L_VERIFY_S] = L_VERIFY_T,
+ [C_MASK] = C_MASK,
+ };
+
+ state.i = peer_state.i;
+
+ state.conn = c_tab[peer_state.conn];
+ state.peer = peer_state.role;
+ state.role = peer_state.peer;
+ state.pdsk = peer_state.disk;
+ state.disk = peer_state.pdsk;
+ state.peer_isp = (peer_state.aftr_isp | peer_state.user_isp);
+
+ return state;
+}
+
+static enum drbd_state_rv
+__change_connection_state(struct drbd_connection *connection,
+ union drbd_state mask, union drbd_state val,
+ enum chg_state_flags flags)
+{
+ struct drbd_resource *resource = connection->resource;
+
+ if (mask.role) {
+ /* not allowed */
+ }
+ if (mask.susp) {
+ mask.susp ^= -1;
+ /* Only resume-io lifts a suspend-io. */
+ if (val.susp)
+ __change_io_susp_user(resource, true);
+ }
+ if (mask.susp_nod) {
+ mask.susp_nod ^= -1;
+ __change_io_susp_no_data(resource, val.susp_nod);
+ }
+ if (mask.susp_fen) {
+ mask.susp_fen ^= -1;
+ __change_io_susp_fencing(connection, val.susp_fen);
+ }
+ if (mask.disk) {
+ /* Handled in __change_peer_device_state(). */
+ mask.disk ^= -1;
+ }
+ if (mask.conn) {
+ mask.conn ^= -1;
+ __change_cstate(connection,
+ min_t(enum drbd_conn_state, val.conn, C_CONNECTED));
+ }
+ if (mask.pdsk) {
+ /* Handled in __change_peer_device_state(). */
+ mask.pdsk ^= -1;
+ }
+ if (mask.peer) {
+ mask.peer ^= -1;
+ __change_peer_role(connection, val.peer);
+ }
+ if (mask.i) {
+ drbd_info(connection, "Remote state change: request %u/%u not "
+ "understood\n", mask.i, val.i & mask.i);
+ return SS_NOT_SUPPORTED;
+ }
+ return SS_SUCCESS;
+}
+
+static enum drbd_state_rv
+__change_peer_device_state(struct drbd_peer_device *peer_device,
+ union drbd_state mask, union drbd_state val)
+{
+ struct drbd_device *device = peer_device->device;
+
+ if (mask.peer) {
+ /* Handled in __change_connection_state(). */
+ mask.peer ^= -1;
+ }
+ if (mask.disk) {
+ mask.disk ^= -1;
+ __change_disk_state(device, val.disk);
+ }
+
+ if (mask.conn) {
+ mask.conn ^= -1;
+ __change_repl_state(peer_device,
+ max_t(enum drbd_repl_state, val.conn, L_OFF));
+ }
+ if (mask.pdsk) {
+ mask.pdsk ^= -1;
+ __change_peer_disk_state(peer_device, val.pdsk);
+ }
+ if (mask.user_isp) {
+ mask.user_isp ^= -1;
+ __change_resync_susp_user(peer_device, val.user_isp);
+ }
+ if (mask.peer_isp) {
+ mask.peer_isp ^= -1;
+ __change_resync_susp_peer(peer_device, val.peer_isp);
+ }
+ if (mask.aftr_isp) {
+ mask.aftr_isp ^= -1;
+ __change_resync_susp_dependency(peer_device, val.aftr_isp);
+ }
+ if (mask.i) {
+ drbd_info(peer_device, "Remote state change: request %u/%u not "
+ "understood\n", mask.i, val.i & mask.i);
+ return SS_NOT_SUPPORTED;
+ }
+ return SS_SUCCESS;
+}
+
+static union drbd_state
+sanitize_outdate(struct drbd_peer_device *peer_device,
+ union drbd_state mask,
+ union drbd_state val)
+{
+ struct drbd_device *device = peer_device->device;
+ union drbd_state result_mask = mask;
+
+ if (val.pdsk == D_OUTDATED && peer_device->disk_state[NEW] < D_OUTDATED)
+ result_mask.pdsk = 0;
+ if (val.disk == D_OUTDATED && device->disk_state[NEW] < D_OUTDATED)
+ result_mask.disk = 0;
+
+ return result_mask;
+}
+
+static void log_openers(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct opener *opener;
+
+ spin_lock(&device->openers_lock);
+ opener = list_first_entry_or_null(&device->openers, struct opener, list);
+ if (opener)
+ drbd_warn(device, "Held open by %s(%d)\n", opener->comm, opener->pid);
+ spin_unlock(&device->openers_lock);
+ }
+ rcu_read_unlock();
+}
+
+/**
+ * change_connection_state() - change state of a connection and all its peer devices
+ * @connection: DRBD connection to operate on
+ * @state_change: Prepared state change
+ * @reply: Two phase commit reply
+ * @flags: State change flags
+ *
+ * Also changes the state of the peer devices' devices and of the resource.
+ * Cluster-wide state changes are not supported.
+ */
+static enum drbd_state_rv
+change_connection_state(struct drbd_connection *connection,
+ struct twopc_state_change *state_change,
+ struct twopc_reply *reply,
+ enum chg_state_flags flags)
+{
+ struct drbd_resource *resource = connection->resource;
+ long t = resource->res_opts.auto_promote_timeout * HZ / 10;
+ union drbd_state mask = state_change->mask;
+ union drbd_state val = state_change->val;
+ bool is_disconnect = false;
+ bool is_connect = false;
+ bool abort = flags & CS_ABORT;
+ struct drbd_peer_device *peer_device;
+ unsigned long irq_flags;
+ enum drbd_state_rv rv;
+ int vnr;
+
+ if (reply) {
+ is_disconnect = reply->is_disconnect;
+ is_connect = reply->is_connect;
+ } else if (mask.conn == conn_MASK) {
+ is_connect = val.conn == C_CONNECTED;
+ is_disconnect = val.conn == C_DISCONNECTING;
+ }
+
+ mask = convert_state(mask);
+ val = convert_state(val);
+
+ if (is_connect && connection->agreed_pro_version >= 118) {
+ if (flags & CS_PREPARE)
+ conn_connect2(connection);
+ if (abort)
+ abort_connect(connection);
+ }
+retry:
+ begin_state_change(resource, &irq_flags, flags & ~CS_VERBOSE);
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ union drbd_state l_mask;
+
+ l_mask = is_disconnect ? sanitize_outdate(peer_device, mask, val) : mask;
+ rv = __change_peer_device_state(peer_device, l_mask, val);
+ if (rv < SS_SUCCESS)
+ goto fail;
+ }
+ rv = __change_connection_state(connection, mask, val, flags);
+ if (rv < SS_SUCCESS)
+ goto fail;
+
+ if (reply && !abort) {
+ u64 directly_reachable = directly_connected_nodes(resource, NEW) |
+ NODE_MASK(resource->res_opts.node_id);
+
+ if (reply->primary_nodes & ~directly_reachable)
+ __outdate_myself(resource);
+ }
+
+ if (is_connect && connection->agreed_pro_version >= 117)
+ apply_connect(connection, (flags & CS_PREPARED) && !abort);
+ rv = end_state_change(resource, &irq_flags, "remote");
+out:
+
+ if (((rv == SS_NO_UP_TO_DATE_DISK && resource->role[NOW] != R_PRIMARY) ||
+ rv == SS_PRIMARY_READER) && drbd_open_ro_count(resource)) {
+ /* Most probably udev opened it read-only. That might happen
+ if it was demoted very recently. Wait up to one second. */
+ t = wait_event_interruptible_timeout(resource->state_wait,
+ drbd_open_ro_count(resource) == 0,
+ t);
+ if (t > 0)
+ goto retry;
+ }
+
+ if (rv < SS_SUCCESS) {
+ drbd_err(resource, "State change failed: %s (%d)\n", drbd_set_st_err_str(rv), rv);
+ if (rv == SS_PRIMARY_READER)
+ log_openers(resource);
+ }
+
+ return rv;
+fail:
+ abort_state_change(resource, &irq_flags);
+ goto out;
+}
+
+/**
+ * change_peer_device_state() - change state of a peer and its connection
+ * @peer_device: DRBD peer device
+ * @state_change: Prepared state change
+ * @flags: State change flags
+ *
+ * Also changes the state of the peer device's device and of the resource.
+ * Cluster-wide state changes are not supported.
+ */
+static enum drbd_state_rv
+change_peer_device_state(struct drbd_peer_device *peer_device,
+ struct twopc_state_change *state_change,
+ enum chg_state_flags flags)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ union drbd_state mask = state_change->mask;
+ union drbd_state val = state_change->val;
+ unsigned long irq_flags;
+ enum drbd_state_rv rv;
+
+ mask = convert_state(mask);
+ val = convert_state(val);
+
+ begin_state_change(connection->resource, &irq_flags, flags);
+ rv = __change_peer_device_state(peer_device, mask, val);
+ if (rv < SS_SUCCESS)
+ goto fail;
+ rv = __change_connection_state(connection, mask, val, flags);
+ if (rv < SS_SUCCESS)
+ goto fail;
+ rv = end_state_change(connection->resource, &irq_flags, "remote");
+out:
+ return rv;
+fail:
+ abort_state_change(connection->resource, &irq_flags);
+ goto out;
+}
+
+static int receive_req_state(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct twopc_state_change *state_change = &resource->twopc.state_change;
+ struct drbd_peer_device *peer_device = NULL;
+ struct p_req_state *p = pi->data;
+ enum chg_state_flags flags = CS_VERBOSE | CS_LOCAL_ONLY | CS_TWOPC;
+ enum drbd_state_rv rv;
+ int vnr = -1;
+
+ if (!expect(connection, connection->agreed_pro_version < 110)) {
+ drbd_err(connection, "Packet %s not allowed in protocol version %d\n",
+ drbd_packet_name(pi->cmd),
+ connection->agreed_pro_version);
+ return -EIO;
+ }
+
+ state_change->mask.i = be32_to_cpu(p->mask);
+ state_change->val.i = be32_to_cpu(p->val);
+
+ /* Only val holds disk states; mask.disk and mask.pdsk are bit masks. */
+ drbd_disk_states_from_84(&state_change->val);
+
+ /* P_STATE_CHG_REQ packets must have a valid vnr. P_CONN_ST_CHG_REQ
+ * packets have an undefined vnr. */
+ if (pi->cmd == P_STATE_CHG_REQ) {
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device) {
+ const union drbd_state conn_mask = { .conn = conn_MASK };
+ const union drbd_state val_off = { .conn = L_OFF };
+
+ if (state_change->mask.i == conn_mask.i &&
+ state_change->val.i == val_off.i) {
+ /* The peer removed this volume, we do not have it... */
+ drbd_send_sr_reply(connection, vnr, SS_NOTHING_TO_DO);
+ return 0;
+ }
+
+ return -EIO;
+ }
+ vnr = peer_device->device->vnr;
+ }
+
+ rv = SS_SUCCESS;
+ write_lock_irq(&resource->state_rwlock);
+ if (resource->remote_state_change)
+ rv = SS_CONCURRENT_ST_CHG;
+ else
+ resource->remote_state_change = true;
+ write_unlock_irq(&resource->state_rwlock);
+
+ if (rv != SS_SUCCESS) {
+ drbd_info(connection, "Rejecting concurrent remote state change\n");
+ drbd_send_sr_reply(connection, vnr, rv);
+ return 0;
+ }
+
+ /* Send the reply before carrying out the state change: this is needed
+ * for connection state changes which close the network connection. */
+ if (peer_device) {
+ rv = change_peer_device_state(peer_device, state_change, flags | CS_PREPARE);
+ drbd_send_sr_reply(connection, vnr, rv);
+ rv = change_peer_device_state(peer_device, state_change, flags | CS_PREPARED);
+ if (rv >= SS_SUCCESS)
+ drbd_md_sync_if_dirty(peer_device->device);
+ } else {
+ flags |= CS_IGN_OUTD_FAIL;
+ rv = change_connection_state(connection, state_change, NULL, flags | CS_PREPARE);
+ drbd_send_sr_reply(connection, vnr, rv);
+ change_connection_state(connection, state_change, NULL, flags | CS_PREPARED);
+ }
+
+ write_lock_irq(&resource->state_rwlock);
+ resource->remote_state_change = false;
+ write_unlock_irq(&resource->state_rwlock);
+ wake_up_all(&resource->twopc_wait);
+
+ return 0;
+}
+
+static void drbd_abort_twopc(struct drbd_resource *resource)
+{
+ int initiator_node_id = resource->twopc_reply.initiator_node_id;
+ bool is_connect = resource->twopc_reply.is_connect;
+ struct drbd_connection *connection;
+
+ if (initiator_node_id == -1)
+ return;
+
+ connection = drbd_get_connection_by_node_id(resource, initiator_node_id);
+ resource->twopc_parent_nodes = 0;
+ __clear_remote_state_change(resource);
+
+ /* No commit or abort follows to end the connect. */
+ if (connection) {
+ if (is_connect)
+ abort_connect(connection);
+ kref_put(&connection->kref, drbd_destroy_connection);
+ }
+}
+
+void twopc_timer_fn(struct timer_list *t)
+{
+ struct drbd_resource *resource = timer_container_of(resource, t, twopc_timer);
+ unsigned long irq_flags;
+
+ write_lock_irqsave(&resource->state_rwlock, irq_flags);
+ if (!test_bit(TWOPC_WORK_PENDING, &resource->flags)) {
+ drbd_err(resource, "Two-phase commit %u timeout\n",
+ resource->twopc_reply.tid);
+ drbd_abort_twopc(resource);
+ } else {
+ mod_timer(&resource->twopc_timer, jiffies + HZ/10);
+ }
+ write_unlock_irqrestore(&resource->state_rwlock, irq_flags);
+}
+
+bool drbd_have_local_disk(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (device->disk_state[NOW] > D_DISKLESS) {
+ rcu_read_unlock();
+ return true;
+ }
+ }
+ rcu_read_unlock();
+ return false;
+}
+
+static enum drbd_state_rv
+far_away_change(struct drbd_connection *connection,
+ struct twopc_request *request,
+ struct twopc_reply *reply,
+ enum chg_state_flags flags)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct twopc_state_change *state_change = &resource->twopc.state_change;
+ u64 directly_reachable = directly_connected_nodes(resource, NOW) |
+ NODE_MASK(resource->res_opts.node_id);
+ union drbd_state mask = state_change->mask;
+ union drbd_state val = state_change->val;
+ int vnr = resource->twopc_reply.vnr;
+ struct drbd_device *device;
+ unsigned long irq_flags;
+ int iterate_vnr;
+
+
+ if (flags & CS_PREPARE && mask.role == role_MASK && val.role == R_PRIMARY &&
+ resource->role[NOW] == R_PRIMARY) {
+ struct drbd_net_conf *nc;
+ bool two_primaries_allowed = false;
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (nc)
+ two_primaries_allowed = nc->two_primaries;
+ rcu_read_unlock();
+ if (!two_primaries_allowed)
+ return SS_TWO_PRIMARIES;
+
+ /* A node further away wants to become primary. In case I am primary allow it only
+ * when I am diskless. See also check_primaries_distances() in drbd_state.c
+ */
+ if (drbd_have_local_disk(resource))
+ return SS_WEAKLY_CONNECTED;
+ }
+
+ begin_state_change(resource, &irq_flags, flags);
+ if (mask.i == 0 && val.i == 0 &&
+ resource->role[NOW] == R_PRIMARY && vnr == -1) {
+ /* A node far away test if there are primaries. I am the guy he is concerned
+ * about... He learned about me in the CS_PREPARE phase. Since he is committing it
+ * I know that he is outdated now...
+ */
+ struct drbd_connection *affected_connection;
+ int initiator_node_id = resource->twopc_reply.initiator_node_id;
+
+ affected_connection = drbd_get_connection_by_node_id(resource, initiator_node_id);
+ if (affected_connection) {
+ __downgrade_peer_disk_states(affected_connection, D_OUTDATED);
+ kref_put(&affected_connection->kref, drbd_destroy_connection);
+ } else if (flags & CS_PREPARED) {
+ idr_for_each_entry(&resource->devices, device, iterate_vnr) {
+ struct drbd_peer_md *peer_md;
+
+ if (!get_ldev(device))
+ continue;
+
+ peer_md = &device->ldev->md.peers[initiator_node_id];
+ set_bit(__MDF_PEER_OUTDATED, &peer_md->flags);
+ put_ldev(device);
+ drbd_md_mark_dirty(device);
+ }
+ }
+ }
+
+ if (state_change->primary_nodes & ~directly_reachable &&
+ !(request->flags & TWOPC_PRI_INCAPABLE))
+ __outdate_myself(resource);
+
+ idr_for_each_entry(&resource->devices, device, iterate_vnr) {
+ if (test_bit(OUTDATE_ON_2PC_COMMIT, &device->flags) &&
+ device->disk_state[NEW] > D_OUTDATED)
+ __change_disk_state(device, D_OUTDATED);
+ }
+
+ /* even if no outdate happens, CS_FORCE_RECALC might be set here */
+ return end_state_change(resource, &irq_flags, "far-away");
+}
+
+static void handle_neighbor_demotion(struct drbd_connection *connection,
+ struct twopc_state_change *state_change,
+ struct twopc_reply *reply)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_device *device;
+ int vnr;
+
+ if (reply->initiator_node_id != connection->peer_node_id ||
+ connection->peer_role[NOW] != R_PRIMARY ||
+ state_change->mask.role != role_MASK || state_change->val.role != R_SECONDARY)
+ return;
+
+ /* A directly connected neighbor that was primary demotes to secondary */
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ kref_get(&device->kref);
+ rcu_read_unlock();
+ if (get_ldev(device)) {
+ drbd_bitmap_io(device, &drbd_bm_write, "peer demote",
+ BM_LOCK_SET | BM_LOCK_CLEAR | BM_LOCK_BULK, NULL);
+ put_ldev(device);
+ }
+ rcu_read_lock();
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+ rcu_read_unlock();
+}
+
+static void peer_device_init_connect_state(struct drbd_peer_device *peer_device)
+{
+ clear_bit(INITIAL_STATE_SENT, peer_device->flags);
+ clear_bit(INITIAL_STATE_RECEIVED, peer_device->flags);
+ clear_bit(HAVE_SIZES, peer_device->flags);
+ clear_bit(UUIDS_RECEIVED, peer_device->flags);
+ clear_bit(CURRENT_UUID_RECEIVED, peer_device->flags);
+ clear_bit(PEER_QUORATE, peer_device->flags);
+ clear_bit(PEER_UPGRADE_PREDICTED, peer_device->flags);
+ peer_device->connect_state = (union drbd_state) {{ .disk = D_MASK }};
+}
+
+
+/**
+ * drbd_init_connect_state() - Prepare twopc that establishes the connection
+ * @connection: The connection this is about
+ *
+ * After a transport implementation has established the lower-level aspects
+ * of a connection, DRBD executes a two-phase commit so that the membership
+ * information changes in a cluster-wide, consistent way. During that
+ * two-phase commit, DRBD exchanges the UUIDs, size information, and the
+ * initial state. A two-phase commit might be aborted, which needs to be
+ * retried. This function re-initializes the struct members for this. The
+ * callsites are at the beginning of a two-phase connect commit, active and
+ * passive side.
+ */
+void drbd_init_connect_state(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ peer_device_init_connect_state(peer_device);
+ rcu_read_unlock();
+ clear_bit(CONN_HANDSHAKE_DISCONNECT, &connection->flags);
+ clear_bit(CONN_HANDSHAKE_RETRY, &connection->flags);
+ clear_bit(CONN_HANDSHAKE_READY, &connection->flags);
+ clear_bit(CONN_HANDSHAKE_DEMOTE, &connection->flags);
+}
+
+enum csc_rv {
+ CSC_CLEAR,
+ CSC_REJECT,
+ CSC_ABORT_LOCAL,
+ CSC_TID_MISS,
+ CSC_MATCH,
+};
+
+static enum csc_rv
+check_concurrent_transactions(struct drbd_resource *resource, struct twopc_reply *new_r)
+{
+ struct twopc_reply *ongoing = &resource->twopc_reply;
+
+ if (!resource->remote_state_change)
+ return CSC_CLEAR;
+
+ if (new_r->initiator_node_id < ongoing->initiator_node_id) {
+ if (ongoing->initiator_node_id == resource->res_opts.node_id)
+ return CSC_ABORT_LOCAL;
+ else
+ return CSC_REJECT;
+ } else if (new_r->initiator_node_id > ongoing->initiator_node_id) {
+ return CSC_REJECT;
+ }
+ if (new_r->tid != ongoing->tid)
+ return CSC_TID_MISS;
+
+ return CSC_MATCH;
+}
+
+
+enum alt_rv {
+ ALT_LOCKED,
+ ALT_MATCH,
+ ALT_TIMEOUT,
+};
+
+static enum alt_rv when_done_lock(struct drbd_resource *resource, unsigned int for_tid)
+{
+ write_lock_irq(&resource->state_rwlock);
+ if (!resource->remote_state_change)
+ return ALT_LOCKED;
+ write_unlock_irq(&resource->state_rwlock);
+ if (resource->twopc_reply.tid == for_tid)
+ return ALT_MATCH;
+
+ return ALT_TIMEOUT;
+}
+static enum alt_rv abort_local_transaction(struct drbd_connection *connection, unsigned int for_tid)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_net_conf *nc;
+ enum alt_rv rv;
+ long t;
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ t = nc->ping_timeo * HZ/10 * 3 / 2;
+ rcu_read_unlock();
+
+ set_bit(TWOPC_ABORT_LOCAL, &resource->flags);
+ write_unlock_irq(&resource->state_rwlock);
+ wake_up_all(&resource->state_wait);
+ wait_event_timeout(resource->twopc_wait,
+ (rv = when_done_lock(resource, for_tid)) != ALT_TIMEOUT, t);
+ clear_bit(TWOPC_ABORT_LOCAL, &resource->flags);
+ return rv;
+}
+
+static int receive_twopc(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct p_twopc_request *p = pi->data;
+ struct twopc_reply reply = {0};
+
+ reply.vnr = pi->vnr;
+ reply.tid = be32_to_cpu(p->tid);
+ if (connection->agreed_features & DRBD_FF_2PC_V2) {
+ reply.initiator_node_id = p->s8_initiator_node_id;
+ reply.target_node_id = p->s8_target_node_id;
+ } else {
+ reply.initiator_node_id = be32_to_cpu(p->u32_initiator_node_id);
+ reply.target_node_id = be32_to_cpu(p->u32_target_node_id);
+ }
+ reply.reachable_nodes = directly_connected_nodes(resource, NOW) |
+ NODE_MASK(resource->res_opts.node_id);
+
+ if (pi->cmd == P_TWOPC_PREPARE)
+ clear_bit(TWOPC_RECV_SIZES_ERR, &resource->flags);
+
+ process_twopc(connection, &reply, pi, jiffies);
+
+ return 0;
+}
+
+static void nested_twopc_abort(struct drbd_resource *resource, struct twopc_request *request)
+{
+ struct drbd_connection *connection;
+ u64 nodes_to_reach, reach_immediately, im;
+
+ read_lock_irq(&resource->state_rwlock);
+ nodes_to_reach = request->nodes_to_reach;
+ reach_immediately = directly_connected_nodes(resource, NOW) & nodes_to_reach;
+ nodes_to_reach &= ~(reach_immediately | NODE_MASK(resource->res_opts.node_id));
+ request->nodes_to_reach = nodes_to_reach;
+ read_unlock_irq(&resource->state_rwlock);
+
+ for_each_connection_ref(connection, im, resource) {
+ u64 mask = NODE_MASK(connection->peer_node_id);
+
+ if (reach_immediately & mask)
+ conn_send_twopc_request(connection, request);
+ }
+}
+
+static bool is_prepare(enum drbd_packet cmd)
+{
+ return cmd == P_TWOPC_PREP_RSZ || cmd == P_TWOPC_PREPARE;
+}
+
+
+enum determine_dev_size
+drbd_commit_size_change(struct drbd_device *device, struct drbd_resize_parms *rs,
+ u64 nodes_to_reach)
+{
+ struct twopc_resize *tr = &device->resource->twopc.resize;
+ struct drbd_peer_device *peer_device;
+ enum determine_dev_size dd;
+ uint64_t my_usize;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ /* update cached sizes, relevant for the next handshake */
+ peer_device->c_size = tr->new_size;
+ peer_device->u_size = tr->user_size;
+
+ if (peer_device->d_size)
+ peer_device->d_size = tr->new_size;
+ peer_device->max_size = tr->new_size;
+ }
+ rcu_read_unlock();
+
+ if (!get_ldev(device)) {
+ drbd_set_my_capacity(device, tr->new_size);
+ return DS_UNCHANGED; /* Not entirely true, but we are diskless... */
+ }
+
+ rcu_read_lock();
+ my_usize = rcu_dereference(device->ldev->disk_conf)->disk_size;
+ rcu_read_unlock();
+
+ if (my_usize != tr->user_size) {
+ struct drbd_disk_conf *old_disk_conf, *new_disk_conf;
+
+ drbd_info(device, "New u_size %llu sectors\n",
+ (unsigned long long)tr->user_size);
+
+ new_disk_conf = kzalloc_obj(struct drbd_disk_conf);
+ if (!new_disk_conf) {
+ device->ldev->disk_conf->disk_size = tr->user_size;
+ goto cont;
+ }
+
+ old_disk_conf = device->ldev->disk_conf;
+ *new_disk_conf = *old_disk_conf;
+ new_disk_conf->disk_size = tr->user_size;
+
+ rcu_assign_pointer(device->ldev->disk_conf, new_disk_conf);
+ kvfree_rcu_mightsleep(old_disk_conf);
+ }
+cont:
+ dd = drbd_determine_dev_size(device, tr->new_size, tr->dds_flags | DDSF_2PC, rs);
+
+ if (dd == DS_GREW && !(tr->dds_flags & DDSF_NO_RESYNC)) {
+ struct drbd_resource *resource = device->resource;
+ const int my_node_id = resource->res_opts.node_id;
+ struct drbd_peer_device *peer_device;
+ u64 im;
+
+ for_each_peer_device_ref(peer_device, im, device) {
+ if (peer_device->repl_state[NOW] != L_ESTABLISHED ||
+ peer_device->disk_state[NOW] < D_INCONSISTENT)
+ continue;
+
+ if (tr->diskful_primary_nodes) {
+ if (tr->diskful_primary_nodes & NODE_MASK(my_node_id)) {
+ enum drbd_repl_state resync;
+
+ if (tr->diskful_primary_nodes & NODE_MASK(peer_device->node_id)) {
+ /* peer is also primary */
+ resync = peer_device->node_id < my_node_id ?
+ L_SYNC_TARGET : L_SYNC_SOURCE;
+ } else {
+ /* peer is secondary */
+ resync = L_SYNC_SOURCE;
+ }
+ drbd_start_resync(peer_device, resync, "resize");
+ } else {
+ if (tr->diskful_primary_nodes & NODE_MASK(peer_device->node_id))
+ drbd_start_resync(peer_device, L_SYNC_TARGET,
+ "resize");
+ /* else no resync */
+ }
+ } else {
+ if (resource->twopc_parent_nodes & NODE_MASK(peer_device->node_id))
+ drbd_start_resync(peer_device, L_SYNC_TARGET, "resize");
+ else if (nodes_to_reach & NODE_MASK(peer_device->node_id))
+ drbd_start_resync(peer_device, L_SYNC_SOURCE, "resize");
+ /* else no resync */
+ }
+ }
+ }
+
+ put_ldev(device);
+ return dd;
+}
+
+enum drbd_state_rv drbd_support_2pc_resize(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ enum drbd_state_rv rv = SS_SUCCESS;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->cstate[NOW] == C_CONNECTED &&
+ connection->agreed_pro_version < 112) {
+ rv = SS_NOT_SUPPORTED;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+static bool any_neighbor_quorate(struct drbd_resource *resource)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_connection *connection;
+ bool peer_with_quorum = false;
+ int vnr;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ peer_with_quorum = true;
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (test_bit(PEER_QUORATE, peer_device->flags))
+ continue;
+ peer_with_quorum = false;
+ break;
+ }
+
+ if (peer_with_quorum)
+ break;
+ }
+ rcu_read_unlock();
+
+ return peer_with_quorum;
+}
+
+static void process_twopc(struct drbd_connection *connection,
+ struct twopc_reply *reply,
+ struct packet_info *pi,
+ unsigned long receive_jif)
+{
+ struct drbd_connection *affected_connection = connection;
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device = NULL;
+ struct p_twopc_request *p = pi->data;
+ struct twopc_state_change *state_change = &resource->twopc.state_change;
+ enum chg_state_flags flags = CS_VERBOSE | CS_LOCAL_ONLY;
+ enum drbd_state_rv rv = SS_SUCCESS;
+ struct twopc_request request;
+ bool waiting_allowed = true;
+ enum csc_rv csc_rv;
+
+ request.tid = be32_to_cpu(p->tid);
+ if (connection->agreed_features & DRBD_FF_2PC_V2) {
+ request.flags = be32_to_cpu(p->flags);
+ request.initiator_node_id = p->s8_initiator_node_id;
+ request.target_node_id = p->s8_target_node_id;
+ } else {
+ request.flags = 0;
+ request.initiator_node_id = be32_to_cpu(p->u32_initiator_node_id);
+ request.target_node_id = be32_to_cpu(p->u32_target_node_id);
+ }
+ request.nodes_to_reach = be64_to_cpu(p->nodes_to_reach);
+ request.cmd = pi->cmd;
+ request.vnr = pi->vnr;
+
+ /* Check for concurrent transactions and duplicate packets. */
+retry:
+ write_lock_irq(&resource->state_rwlock);
+
+ csc_rv = check_concurrent_transactions(resource, reply);
+
+ if (csc_rv == CSC_CLEAR && pi->cmd != P_TWOPC_ABORT) {
+ struct drbd_device *device;
+ int iterate_vnr;
+
+ if (!is_prepare(pi->cmd)) {
+ /* We have committed or aborted this transaction already. */
+ write_unlock_irq(&resource->state_rwlock);
+ dynamic_drbd_dbg(connection, "Ignoring %s packet %u\n",
+ drbd_packet_name(pi->cmd),
+ reply->tid);
+ return;
+ }
+ if (reply->is_aborted) {
+ write_unlock_irq(&resource->state_rwlock);
+ return;
+ }
+ resource->remote_state_change = true;
+ resource->twopc.type =
+ pi->cmd == P_TWOPC_PREPARE ? TWOPC_STATE_CHANGE : TWOPC_RESIZE;
+ resource->twopc_prepare_reply_cmd = 0;
+ resource->twopc_parent_nodes = NODE_MASK(connection->peer_node_id);
+ clear_bit(TWOPC_EXECUTED, &resource->flags);
+ set_bit(TWOPC_PREPARING, &resource->flags);
+ idr_for_each_entry(&resource->devices, device, iterate_vnr)
+ clear_bit(OUTDATE_ON_2PC_COMMIT, &device->flags);
+ } else if (csc_rv == CSC_MATCH && !is_prepare(pi->cmd)) {
+ flags |= CS_PREPARED;
+
+ if (test_and_set_bit(TWOPC_EXECUTED, &resource->flags)) {
+ write_unlock_irq(&resource->state_rwlock);
+ drbd_info(connection, "Ignoring redundant %s packet %u.\n",
+ drbd_packet_name(pi->cmd),
+ reply->tid);
+ return;
+ }
+ } else if (csc_rv == CSC_ABORT_LOCAL && is_prepare(pi->cmd)) {
+ enum alt_rv alt_rv;
+
+ drbd_info(connection, "Aborting local state change %u to yield to remote "
+ "state change %u.\n",
+ resource->twopc_reply.tid,
+ reply->tid);
+ alt_rv = abort_local_transaction(connection, reply->tid);
+ if (alt_rv == ALT_MATCH) {
+ /* abort_local_transaction() comes back unlocked in this case... */
+ goto match;
+ } else if (alt_rv == ALT_TIMEOUT) {
+ /* abort_local_transaction() comes back unlocked in this case... */
+ drbd_info(connection, "Aborting local state change %u "
+ "failed. Rejecting remote state change %u.\n",
+ resource->twopc_reply.tid,
+ reply->tid);
+ drbd_send_twopc_reply(connection, P_TWOPC_RETRY, reply);
+ return;
+ }
+ /* abort_local_transaction() returned with the state_rwlock write lock */
+ if (reply->is_aborted) {
+ write_unlock_irq(&resource->state_rwlock);
+ return;
+ }
+ resource->remote_state_change = true;
+ resource->twopc.type =
+ pi->cmd == P_TWOPC_PREPARE ? TWOPC_STATE_CHANGE : TWOPC_RESIZE;
+ resource->twopc_parent_nodes = NODE_MASK(connection->peer_node_id);
+ resource->twopc_prepare_reply_cmd = 0;
+ clear_bit(TWOPC_EXECUTED, &resource->flags);
+ set_bit(TWOPC_PREPARING, &resource->flags);
+ } else if (pi->cmd == P_TWOPC_ABORT) {
+ /* crc_rc != CRC_MATCH */
+ write_unlock_irq(&resource->state_rwlock);
+ nested_twopc_abort(resource, &request);
+ return;
+ } else {
+ write_unlock_irq(&resource->state_rwlock);
+
+ if (csc_rv == CSC_TID_MISS && is_prepare(pi->cmd) && waiting_allowed) {
+ /* CSC_TID_MISS implies the two transactions are from the same initiator */
+ if (!(resource->twopc_parent_nodes & NODE_MASK(connection->peer_node_id))) {
+ long timeout = twopc_timeout(resource) / 20; /* usually 1.5 sec */
+ /*
+ * We are expecting the P_TWOPC_COMMIT or P_TWOPC_ABORT through
+ * another connection. So we can wait without deadlocking.
+ */
+ wait_event_interruptible_timeout(resource->twopc_wait,
+ !resource->remote_state_change, timeout);
+ waiting_allowed = false; /* retry only once */
+ goto retry;
+ }
+ }
+
+ if (csc_rv == CSC_REJECT ||
+ (csc_rv == CSC_TID_MISS && is_prepare(pi->cmd))) {
+ drbd_info(connection, "Rejecting concurrent "
+ "remote state change %u because of "
+ "state change %u\n",
+ reply->tid,
+ resource->twopc_reply.tid);
+ drbd_send_twopc_reply(connection, P_TWOPC_RETRY, reply);
+ return;
+ }
+
+ if (is_prepare(pi->cmd)) {
+ if (csc_rv == CSC_MATCH) {
+ /* We have prepared this transaction already. */
+ enum drbd_packet reply_cmd;
+
+match:
+ drbd_info(connection,
+ "Duplicate prepare for remote state change %u\n",
+ reply->tid);
+ write_lock_irq(&resource->state_rwlock);
+ resource->twopc_parent_nodes |= NODE_MASK(connection->peer_node_id);
+ reply_cmd = resource->twopc_prepare_reply_cmd;
+ write_unlock_irq(&resource->state_rwlock);
+
+ if (reply_cmd) {
+ drbd_send_twopc_reply(connection, reply_cmd,
+ &resource->twopc_reply);
+ } else {
+ /* if a node sends us a prepare, that means he has
+ prepared this himsilf successfully. */
+ write_lock_irq(&resource->state_rwlock);
+ set_bit(TWOPC_YES, &connection->flags);
+ drbd_maybe_cluster_wide_reply(resource);
+ write_unlock_irq(&resource->state_rwlock);
+ }
+ }
+ } else {
+ drbd_info(connection, "Ignoring %s packet %u "
+ "current processing state change %u\n",
+ drbd_packet_name(pi->cmd),
+ reply->tid,
+ resource->twopc_reply.tid);
+ }
+ return;
+ }
+
+ if (reply->initiator_node_id != connection->peer_node_id) {
+ /*
+ * This is an indirect request. Unless we are directly
+ * connected to the initiator as well as indirectly, we don't
+ * have connection or peer device objects for this peer.
+ */
+ affected_connection = drbd_connection_by_node_id(resource, reply->initiator_node_id);
+ }
+
+ if (reply->target_node_id != -1 &&
+ reply->target_node_id != resource->res_opts.node_id) {
+ affected_connection = NULL;
+ }
+
+ switch (resource->twopc.type) {
+ case TWOPC_STATE_CHANGE:
+ if (pi->cmd == P_TWOPC_PREPARE) {
+ state_change->mask.i = be32_to_cpu(p->mask);
+ state_change->val.i = be32_to_cpu(p->val);
+ } else { /* P_TWOPC_COMMIT */
+ state_change->primary_nodes = be64_to_cpu(p->primary_nodes);
+ state_change->reachable_nodes = be64_to_cpu(p->reachable_nodes);
+ }
+ break;
+ case TWOPC_RESIZE:
+ if (request.cmd == P_TWOPC_PREP_RSZ) {
+ resource->twopc.resize.user_size = be64_to_cpu(p->user_size);
+ resource->twopc.resize.dds_flags = be16_to_cpu(p->dds_flags);
+ } else { /* P_TWOPC_COMMIT */
+ resource->twopc.resize.diskful_primary_nodes =
+ be64_to_cpu(p->diskful_primary_nodes);
+ resource->twopc.resize.new_size = be64_to_cpu(p->exposed_size);
+ }
+ }
+
+ if (affected_connection && affected_connection->cstate[NOW] < C_CONNECTED &&
+ state_change->mask.conn == 0)
+ affected_connection = NULL;
+
+ if (pi->vnr != -1 && affected_connection) {
+ peer_device = conn_peer_device(affected_connection, pi->vnr);
+ /* If we do not know the peer_device, then we are fine with
+ whatever is going on in the cluster. E.g. detach and del-minor
+ one each node, one after the other */
+
+ affected_connection = NULL; /* It is intended for a peer_device! */
+ }
+
+ if (state_change->mask.conn == conn_MASK) {
+ u64 m = NODE_MASK(reply->initiator_node_id);
+
+ if (state_change->val.conn == C_CONNECTED) {
+ reply->reachable_nodes |= m;
+ if (affected_connection) {
+ reply->is_connect = 1;
+
+ if (pi->cmd == P_TWOPC_PREPARE)
+ drbd_init_connect_state(connection);
+ }
+ }
+ if (state_change->val.conn == C_DISCONNECTING) {
+ reply->reachable_nodes &= ~m;
+ reply->is_disconnect = 1;
+ }
+ }
+
+ if (pi->cmd == P_TWOPC_PREPARE) {
+ reply->primary_nodes = be64_to_cpu(p->primary_nodes);
+ if (resource->role[NOW] == R_PRIMARY) {
+ reply->primary_nodes |= NODE_MASK(resource->res_opts.node_id);
+
+ if (drbd_res_data_accessible(resource))
+ reply->weak_nodes = ~reply->reachable_nodes;
+ }
+ }
+ if (pi->cmd == P_TWOPC_PREP_RSZ) {
+ struct drbd_device *device;
+
+ device = (peer_device ?: conn_peer_device(connection, pi->vnr))->device;
+ twopc_suspend_io(device);
+ if (get_ldev(device)) {
+ if (resource->role[NOW] == R_PRIMARY)
+ reply->diskful_primary_nodes = NODE_MASK(resource->res_opts.node_id);
+ reply->max_possible_size = drbd_local_max_size(device);
+ put_ldev(device);
+ } else {
+ reply->max_possible_size = DRBD_MAX_SECTORS;
+ reply->diskful_primary_nodes = 0;
+ }
+ }
+
+ resource->twopc_reply = *reply;
+ write_unlock_irq(&resource->state_rwlock);
+
+ if (affected_connection && affected_connection != connection &&
+ affected_connection->cstate[NOW] == C_CONNECTED) {
+ drbd_ping_peer(affected_connection);
+ if (affected_connection->cstate[NOW] < C_CONNECTED)
+ affected_connection = NULL;
+ }
+
+ switch (pi->cmd) {
+ case P_TWOPC_PREPARE:
+ drbd_print_cluster_wide_state_change(resource, "Preparing remote state change",
+ reply->tid, reply->initiator_node_id, reply->target_node_id,
+ state_change->mask, state_change->val);
+ flags |= CS_PREPARE;
+ break;
+ case P_TWOPC_PREP_RSZ:
+ dynamic_drbd_dbg(connection, "Preparing remote state change %u "
+ "(local_max_size = %llu KiB)\n",
+ reply->tid, (unsigned long long)reply->max_possible_size >> 1);
+ flags |= CS_PREPARE;
+ break;
+ case P_TWOPC_ABORT:
+ drbd_info(connection, "Aborting remote state change %u\n",
+ reply->tid);
+ flags |= CS_ABORT;
+ break;
+ case P_TWOPC_COMMIT:
+ drbd_info(connection, "Committing remote state change %u (primary_nodes=%llX)\n",
+ reply->tid, be64_to_cpu(p->primary_nodes));
+ break;
+ default:
+ BUG();
+ }
+
+ switch (resource->twopc.type) {
+ case TWOPC_STATE_CHANGE:
+ if (flags & CS_PREPARED && !(flags & CS_ABORT)) {
+ reply->primary_nodes = state_change->primary_nodes;
+ handle_neighbor_demotion(connection, state_change, reply);
+
+ if ((resource->cached_all_devices_have_quorum ||
+ any_neighbor_quorate(resource)) &&
+ request.flags & TWOPC_HAS_REACHABLE) {
+ resource->members = state_change->reachable_nodes;
+ if (!resource->cached_all_devices_have_quorum)
+ flags |= CS_FORCE_RECALC;
+ }
+ if (state_change->mask.conn == conn_MASK &&
+ state_change->val.conn == C_CONNECTED) {
+ /* Add nodes connecting "far away" to members */
+ u64 add_mask = NODE_MASK(reply->initiator_node_id) |
+ NODE_MASK(reply->target_node_id);
+
+ resource->members |= add_mask;
+ }
+ }
+
+ if (peer_device)
+ rv = change_peer_device_state(peer_device, state_change, flags);
+ else if (affected_connection)
+ rv = change_connection_state(affected_connection, state_change, reply,
+ flags | CS_IGN_OUTD_FAIL);
+ else
+ rv = far_away_change(connection, &request, reply, flags);
+
+ /* An abort is handled by clear_remote_state_change() */
+ if (!peer_device && !affected_connection && !(flags & CS_ABORT) &&
+ state_change->mask.conn == conn_MASK &&
+ state_change->val.conn == C_CONNECTED) {
+ if (flags & CS_PREPARE && rv >= SS_SUCCESS)
+ drbd_flush_connect_prepared(resource, reply->initiator_node_id,
+ reply->target_node_id);
+ else if (flags & CS_PREPARED)
+ drbd_flush_connect_committed(resource);
+ }
+ break;
+ case TWOPC_RESIZE:
+ if (flags & CS_PREPARE) {
+ rv = drbd_support_2pc_resize(resource);
+ /* The YES must mean "nothing in flight from here", see
+ * twopc_suspend_io(). Writes only: a read this node has
+ * outstanding on this connection completes through this
+ * receiver thread, while write acks arrive on the
+ * ack receiver.
+ */
+ if (rv >= SS_SUCCESS) {
+ struct drbd_peer_device *pd;
+
+ pd = peer_device ?: conn_peer_device(connection, pi->vnr);
+ twopc_drain_io(pd->device, WRITE_ONLY);
+ }
+ }
+ break;
+ }
+
+ if (flags & CS_PREPARE) {
+ mod_timer(&resource->twopc_timer, receive_jif + twopc_timeout(resource));
+
+ /* Retry replies can be sent immediately. Otherwise use the
+ * nested twopc path. This waits for the state handshake to
+ * complete in the case of a twopc for transitioning to
+ * C_CONNECTED. */
+ if (rv == SS_IN_TRANSIENT_STATE) {
+ write_lock_irq(&resource->state_rwlock);
+ clear_bit(TWOPC_PREPARING, &resource->flags);
+ resource->twopc_prepare_reply_cmd = P_TWOPC_RETRY;
+ write_unlock_irq(&resource->state_rwlock);
+ drbd_send_twopc_reply(connection, P_TWOPC_RETRY, reply);
+ } else {
+ resource->twopc_reply.state_change_failed = rv < SS_SUCCESS;
+ nested_twopc_request(resource, &request);
+ }
+ } else {
+ if (flags & CS_PREPARED) {
+ if (rv < SS_SUCCESS)
+ drbd_err(resource, "FATAL: Local commit of prepared %u failed!\n",
+ reply->tid);
+
+ timer_delete(&resource->twopc_timer);
+ }
+
+ nested_twopc_request(resource, &request);
+
+ if (resource->twopc.type == TWOPC_RESIZE && flags & CS_PREPARED &&
+ !(flags & CS_ABORT)) {
+ struct drbd_device *device;
+
+ device = (peer_device ?: conn_peer_device(connection, pi->vnr))->device;
+
+ drbd_commit_size_change(device, NULL, request.nodes_to_reach);
+ rv = SS_SUCCESS;
+ }
+
+ clear_remote_state_change(resource);
+
+ if (peer_device && rv >= SS_SUCCESS && !(flags & CS_ABORT))
+ drbd_md_sync_if_dirty(peer_device->device);
+
+ if (connection->agreed_pro_version < 117 &&
+ rv >= SS_SUCCESS && !(flags & CS_ABORT) &&
+ affected_connection &&
+ state_change->mask.conn == conn_MASK && state_change->val.conn == C_CONNECTED)
+ conn_connect2(connection);
+ }
+}
+
+static void drbd_try_to_get_resynced(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device, *best_peer_device = NULL;
+ enum sync_strategy best_strategy = UNDETERMINED;
+ int best_preference = 0;
+
+ if (!get_ldev(device))
+ return;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum sync_strategy strategy;
+ enum sync_rule rule;
+ int peer_node_id;
+
+ if (peer_device->disk_state[NOW] != D_UP_TO_DATE)
+ continue;
+
+ spin_lock_irq(&device->ldev->md.uuid_lock);
+ strategy = drbd_uuid_compare(peer_device, &rule, &peer_node_id);
+ spin_unlock_irq(&device->ldev->md.uuid_lock);
+ disk_states_to_strategy(peer_device, peer_device->disk_state[NOW], &strategy, &rule,
+ &peer_node_id);
+ drbd_info(peer_device, "strategy = %s\n", strategy_descriptor(strategy).name);
+ if (strategy_descriptor(strategy).resync_peer_preference > best_preference) {
+ best_preference = strategy_descriptor(strategy).resync_peer_preference;
+ best_peer_device = peer_device;
+ best_strategy = strategy;
+ }
+ }
+ rcu_read_unlock();
+ peer_device = best_peer_device;
+
+ if (best_strategy == NO_SYNC) {
+ change_disk_state(device, D_UP_TO_DATE, CS_VERBOSE, "get-resync", NULL);
+ } else if (peer_device &&
+ (!repl_is_sync_target(peer_device->repl_state[NOW]) ||
+ test_bit(UNSTABLE_RESYNC, peer_device->flags))) {
+ drbd_resync(peer_device, DISKLESS_PRIMARY);
+ drbd_send_uuids(peer_device, UUID_FLAG_RESYNC | UUID_FLAG_DISKLESS_PRIMARY, 0);
+ }
+ put_ldev(device);
+}
+
+/*
+ * drbd_try_to_get_resynced() must not run on the worker thread: the bitmap
+ * modification after the handshake performs blocking whole-bitmap IO, which
+ * the worker itself may need to make progress on (drbd_bitmap_io() asserts
+ * that it is not called from the worker). Triggered from
+ * w_after_state_change() via this work function instead.
+ */
+void drbd_try_get_resynced_work_fn(struct work_struct *ws)
+{
+ struct drbd_device *device =
+ container_of(ws, struct drbd_device, try_get_resynced_work);
+
+ drbd_try_to_get_resynced(device);
+ kref_put(&device->kref, drbd_destroy_device);
+}
+
+static void finish_nested_twopc(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ int vnr = 0;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (!test_bit(INITIAL_STATE_RECEIVED, peer_device->flags))
+ return;
+ }
+
+ set_bit(CONN_HANDSHAKE_READY, &connection->flags);
+
+ wake_up_all(&resource->state_wait);
+
+ write_lock_irq(&resource->state_rwlock);
+ drbd_maybe_cluster_wide_reply(resource);
+ write_unlock_irq(&resource->state_rwlock);
+}
+
+static bool uuid_in_peer_history(struct drbd_peer_device *peer_device, u64 uuid)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(peer_device->history_uuids); i++)
+ if ((peer_device->history_uuids[i] & ~UUID_PRIMARY) == uuid)
+ return true;
+
+ return false;
+}
+
+static bool uuid_in_my_history(struct drbd_device *device, u64 uuid)
+{
+ int i;
+
+ for (i = 0; i < HISTORY_UUIDS; i++) {
+ if ((drbd_history_uuid(device, i) & ~UUID_PRIMARY) == uuid)
+ return true;
+ }
+
+ return false;
+}
+
+static bool peer_data_is_successor_of_mine(struct drbd_peer_device *peer_device)
+{
+ u64 exposed = peer_device->device->exposed_data_uuid & ~UUID_PRIMARY;
+ int i;
+
+ i = drbd_find_peer_bitmap_by_uuid(peer_device, exposed);
+ if (i != -1)
+ return true;
+
+ return uuid_in_peer_history(peer_device, exposed);
+}
+
+bool peer_data_is_ancestor_of_mine(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ u64 peer_uuid = peer_device->current_uuid & ~UUID_PRIMARY;
+ struct drbd_peer_device *p2;
+ bool rv = false;
+ int i;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(p2, device) {
+ if (peer_device == p2)
+ continue;
+ i = drbd_find_peer_bitmap_by_uuid(p2, peer_uuid);
+ if (i != -1 || uuid_in_peer_history(p2, peer_uuid)) {
+ rv = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+static void propagate_exposed_uuid(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ u64 im;
+
+ for_each_peer_device_ref(peer_device, im, device) {
+ if (!test_bit(INITIAL_STATE_SENT, peer_device->flags))
+ continue;
+ drbd_send_current_uuid(peer_device,
+ diskless_primary_present_current_uuid(peer_device), 0);
+ }
+}
+
+static void maybe_force_secondary(struct drbd_peer_device *peer_device)
+{
+ struct drbd_resource *resource = peer_device->device->resource;
+ struct drbd_connection *connection = peer_device->connection;
+ unsigned long irq_flags;
+ bool force;
+
+ force = !resource->fail_io[NOW] && resource->cached_susp &&
+ resource->res_opts.on_susp_primary_outdated == SPO_FORCE_SECONDARY;
+ force |= test_and_clear_bit(CONN_HANDSHAKE_DEMOTE, &connection->flags);
+
+ if (force) {
+ drbd_warn(peer_device, "force secondary!\n");
+ begin_state_change(resource, &irq_flags,
+ CS_VERBOSE | CS_HARD | CS_FS_IGN_OPENERS);
+ resource->role[NEW] = R_SECONDARY;
+ /* resource->fail_io[NEW] gets set via CS_FS_IGN_OPENERS */
+ end_state_change(resource, &irq_flags, "peer-state");
+ }
+}
+
+/* A diskless primary that bumped its current UUID keeps the generation it
+ * bumped away from (exposed_data_uuid_predecessor) while the new one is still
+ * unconfirmed. A peer returning on that predecessor generation can be brought
+ * forward by resending the transfer log -- but only if the bridging writes are
+ * still there. They are kept exactly when the peer was lost while we were
+ * suspended (CONNECTION_LOST_WHILE_SUSPENDED keeps protocol B/C requests as
+ * RQ_COMPLETION_SUSP) and we have stayed suspended since, so the transfer log
+ * has not advanced. If we would instead have to resend a write we no longer
+ * hold (completed to the upper layers and freed), we cannot recover this peer
+ * here; it has to resync from a peer that holds good data.
+ *
+ * The resend cursor is conservative: writes this peer acked (RQ_NET_OK) are not
+ * resent, writes still pending to it are. A pending write may in fact have been
+ * processed by the peer already (its ack lost in the break); resending it is
+ * harmless. The only fatal case is a needed write that was given up on
+ * (RQ_NET_DONE without RQ_NET_OK), whose data may be gone.
+ */
+bool diskless_primary_can_replay_to(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_resource *resource = device->resource;
+ int idx = peer_device->node_id;
+ struct drbd_request *req;
+ bool have_resendable = false;
+
+ /* The transfer log must still be frozen; if it moved on, the bridging
+ * writes may have been completed and freed.
+ */
+ if (!resource->cached_susp)
+ return false;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(req, &resource->transfer_log, tl_requests) {
+ unsigned int local, net;
+
+ if (req->device != device)
+ continue;
+ local = READ_ONCE(req->local_rq_state);
+ if (!(local & RQ_WRITE))
+ continue;
+ net = READ_ONCE(req->net_rq_state[idx]);
+
+ /* A write we would need to resend but already gave up on towards
+ * this peer: its data may be gone -> cannot recover here.
+ */
+ if ((net & RQ_NET_DONE) && !(net & RQ_NET_OK)) {
+ rcu_read_unlock();
+ return false;
+ }
+ /* A write still pending to this peer, kept across the loss
+ * because completion was suspended: this is resendable.
+ */
+ if ((net & RQ_NET_PENDING) && (local & RQ_COMPLETION_SUSP))
+ have_resendable = true;
+ }
+ rcu_read_unlock();
+
+ return have_resendable;
+}
+
+/* Present our actual exposed current generation to a returning peer.
+ * Reverts 308cf12f2's present-predecessor (see commit message).
+ */
+u64 diskless_primary_present_current_uuid(struct drbd_peer_device *peer_device)
+{
+ return peer_device->device->exposed_data_uuid;
+}
+
+static void diskless_with_peers_different_current_uuids(struct drbd_peer_device *peer_device,
+ enum drbd_disk_state *peer_disk_state)
+{
+ bool data_successor = peer_data_is_successor_of_mine(peer_device);
+ bool data_ancestor = peer_data_is_ancestor_of_mine(peer_device);
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_device *device = peer_device->device;
+
+ if (data_successor && resource->role[NOW] == R_PRIMARY) {
+ drbd_warn(peer_device, "Remote node has more recent data\n");
+ maybe_force_secondary(peer_device);
+ set_bit(CONN_HANDSHAKE_RETRY, &connection->flags);
+ } else if (data_successor && resource->role[NOW] == R_SECONDARY) {
+ drbd_uuid_set_exposed(device, peer_device->current_uuid, true);
+ propagate_exposed_uuid(device);
+ } else if (data_ancestor) {
+ drbd_warn(peer_device, "Downgrading joining peer's disk as its data is older\n");
+ if (*peer_disk_state > D_OUTDATED)
+ *peer_disk_state = D_OUTDATED;
+ /* See "Do not trust this guy!" in sanitize_state() */
+ } else if (drbd_gen_obligation_state(device) == GEN_OBL_UNCONFIRMED &&
+ device->exposed_data_uuid_predecessor &&
+ (peer_device->current_uuid & ~UUID_PRIMARY) ==
+ (device->exposed_data_uuid_predecessor & ~UUID_PRIMARY) &&
+ diskless_primary_can_replay_to(peer_device)) {
+ /* The peer returned on the generation we bumped away from, and the
+ * writes bridging it to our (still unconfirmed) current generation are
+ * still replayable from our suspended transfer log. Its predecessor
+ * data is a complete, valid generation -- equivalent to our current
+ * minus the unacknowledged in-flight writes -- so assert it UpToDate
+ * now (a sole diskful peer cannot promote itself against a diskless
+ * primary; the "Do not trust this guy" exception in sanitize_state
+ * honors RECONCILE_INJECT_CUR_UUID and keeps it UpToDate despite the
+ * UUID mismatch). The post-handshake relabel
+ * (RECONCILE_INJECT_CUR_UUID -> SEND_RECONCILE_UUID) then advances it
+ * to our current generation, the replayed writes filling the gap.
+ */
+ drbd_info(peer_device,
+ "Peer is one generation behind; asserting UpToDate, will resend transfer log and relabel.\n");
+ set_bit(RECONCILE_INJECT_CUR_UUID, peer_device->flags);
+ *peer_disk_state = D_UP_TO_DATE;
+ } else if (drbd_gen_obligation_state(device) == GEN_OBL_UNCONFIRMED &&
+ resource->res_opts.on_no_quorum == ONQ_SUSPEND_IO) {
+ /* Park-until-resync. We are a diskless primary in an unconfirmed
+ * rotated generation, and this peer is on an older generation we can
+ * neither place as a recognised ancestor nor replay to (e.g. the
+ * first-lost peer, whose transfer log we discarded -- see
+ * diskless_primary_can_replay_to). Being diskless we are no resync
+ * source for it. Refusing it would strand the cluster; relabelling it
+ * forward would be silent divergence (it lacks the bridging writes).
+ * Instead accept the connection but keep it Outdated: it contributes no
+ * quorum and serves no data until it resyncs from a diskful peer that
+ * holds the current generation (the last-lost peer, once reconciled).
+ * We stay suspended (no quorum) until then -- correct, and loss-free
+ * because the rotated writes were never completed/acknowledged.
+ */
+ drbd_warn(peer_device,
+ "Peer is on an older generation we cannot source; keeping it Outdated until it resyncs from a peer that holds the current generation.\n");
+ if (*peer_disk_state > D_OUTDATED)
+ *peer_disk_state = D_OUTDATED;
+ } else {
+ drbd_warn(peer_device, "Current UUID of peer does not match my exposed UUID.");
+ set_bit(CONN_HANDSHAKE_DISCONNECT, &connection->flags);
+ }
+}
+
+static int receive_state(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device = NULL;
+ enum drbd_repl_state *repl_state;
+ struct drbd_device *device = NULL;
+ struct p_state *p = pi->data;
+ union drbd_state old_peer_state, peer_state;
+ enum drbd_disk_state peer_disk_state;
+ enum drbd_repl_state new_repl_state;
+ bool peer_was_resync_target, do_handshake = false;
+ bool consumed_resync;
+ enum chg_state_flags begin_state_chg_flags = CS_VERBOSE | CS_WAIT_COMPLETE;
+ unsigned long irq_flags;
+ int rv;
+
+ if (pi->vnr != -1) {
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return config_unknown_volume(connection, pi);
+ device = peer_device->device;
+ }
+
+ peer_state.i = be32_to_cpu(p->state);
+
+ if (connection->agreed_pro_version < 110)
+ drbd_disk_states_from_84(&peer_state);
+
+ if (peer_device)
+ clear_bit(PEER_UPGRADE_PREDICTED, peer_device->flags);
+
+ if (pi->vnr == -1) {
+ if (peer_state.role == R_SECONDARY) {
+ begin_state_change(resource, &irq_flags, CS_HARD | CS_VERBOSE);
+ __change_peer_role(connection, R_SECONDARY);
+ rv = end_state_change(resource, &irq_flags, "peer-state");
+ if (rv < SS_SUCCESS)
+ goto fail;
+ }
+ return 0;
+ }
+
+ peer_disk_state = peer_state.disk;
+
+ if (peer_disk_state > D_DISKLESS && !want_bitmap(peer_device)) {
+ drbd_warn(peer_device, "The peer is configured to be diskless but presents %s\n",
+ drbd_disk_str(peer_disk_state));
+ goto fail;
+ }
+
+ if (peer_state.disk == D_NEGOTIATING) {
+ peer_disk_state = peer_device->uuid_flags & UUID_FLAG_INCONSISTENT ?
+ D_INCONSISTENT : D_CONSISTENT;
+ drbd_info(peer_device, "real peer disk state = %s\n", drbd_disk_str(peer_disk_state));
+ }
+
+ read_lock_irq(&resource->state_rwlock);
+ old_peer_state = drbd_get_peer_device_state(peer_device, NOW);
+ read_unlock_irq(&resource->state_rwlock);
+ retry:
+ new_repl_state = max_t(enum drbd_repl_state, old_peer_state.conn, L_OFF);
+ consumed_resync = false;
+
+ /* If some other part of the code (ack_receiver thread, timeout)
+ * already decided to close the connection again,
+ * we must not "re-establish" it here. */
+ if (old_peer_state.conn <= C_TEAR_DOWN)
+ return -ECONNRESET;
+
+ if (!test_bit(INITIAL_STATE_RECEIVED, peer_device->flags) &&
+ peer_state.role == R_PRIMARY && peer_device->uuid_flags & UUID_FLAG_STABLE)
+ check_resync_source(peer_device, peer_device->uuid_node_mask);
+
+ peer_was_resync_target =
+ connection->agreed_pro_version >= 110 ?
+ peer_device->last_repl_state == L_SYNC_TARGET ||
+ peer_device->last_repl_state == L_PAUSED_SYNC_T
+ :
+ true;
+ /* If this is the "end of sync" confirmation, usually the peer disk
+ * was D_INCONSISTENT or D_CONSISTENT. (Since the peer might be
+ * weak we do not know anything about its new disk state)
+ */
+ if (peer_was_resync_target &&
+ (old_peer_state.pdsk == D_INCONSISTENT || old_peer_state.pdsk == D_CONSISTENT) &&
+ old_peer_state.conn > L_ESTABLISHED && old_peer_state.disk >= D_INCONSISTENT) {
+ /* If we are (becoming) SyncSource, but peer is still in sync
+ * preparation, ignore its uptodate-ness to avoid flapping, it
+ * will change to inconsistent once the peer reaches active
+ * syncing states.
+ * It may have changed syncer-paused flags, however, so we
+ * cannot ignore this completely. */
+ if (peer_state.conn > L_ESTABLISHED &&
+ peer_state.conn < L_SYNC_SOURCE)
+ peer_disk_state = D_INCONSISTENT;
+
+ /* if peer_state changes to connected at the same time,
+ * it explicitly notifies us that it finished resync.
+ * Maybe we should finish it up, too? */
+ else if (peer_state.conn == L_ESTABLISHED) {
+ bool finish_now = false;
+
+ if (old_peer_state.conn == L_WF_BITMAP_S) {
+ read_lock_irq(&resource->state_rwlock);
+ if (peer_device->repl_state[NOW] == L_WF_BITMAP_S)
+ peer_device->resync_finished_pdsk = peer_state.disk;
+ else if (peer_device->repl_state[NOW] == L_SYNC_SOURCE)
+ finish_now = true;
+ read_unlock_irq(&resource->state_rwlock);
+ }
+
+ if (finish_now || old_peer_state.conn == L_SYNC_SOURCE ||
+ old_peer_state.conn == L_PAUSED_SYNC_S) {
+ drbd_resync_finished(peer_device, peer_state.disk);
+ peer_device->last_repl_state = peer_state.conn;
+ }
+ return 0;
+ }
+ }
+
+ /* explicit verify finished notification, stop sector reached. */
+ if (old_peer_state.conn == L_VERIFY_T && old_peer_state.disk == D_UP_TO_DATE &&
+ peer_state.conn == L_ESTABLISHED && peer_disk_state == D_UP_TO_DATE) {
+ ov_out_of_sync_print(peer_device);
+ drbd_resync_finished(peer_device, D_MASK);
+ peer_device->last_repl_state = peer_state.conn;
+ return 0;
+ }
+
+ /* The peer pulled ahead and came back while we stayed SyncTarget
+ * (its P_RS_CANCEL_AHEAD outran the Ahead state change), so the
+ * SyncTarget -> Behind -> SyncTarget path that would clear the flag
+ * is never taken. Clear it here and resume issuing requests.
+ */
+ if (peer_state.conn == L_SYNC_SOURCE &&
+ peer_device->repl_state[NOW] == L_SYNC_TARGET &&
+ test_and_clear_bit(SYNC_TARGET_TO_BEHIND, peer_device->flags)) {
+ drbd_info(peer_device, "peer back from ahead, resuming resync requests\n");
+ mod_timer(&peer_device->resync_timer, jiffies + RS_MAKE_REQS_INTV);
+ }
+
+ /* Start resync after AHEAD/BEHIND */
+ if (connection->agreed_pro_version >= 110 &&
+ peer_state.conn == L_SYNC_SOURCE && old_peer_state.conn == L_BEHIND) {
+ /*
+ * Become Inconsistent immediately because we may now receive
+ * data. Delay the start of the resync itself until any
+ * previous resync is no longer active.
+ */
+ rv = change_disk_state(device, D_INCONSISTENT, CS_VERBOSE,
+ "resync-after-behind", NULL);
+ if (rv < SS_SUCCESS)
+ goto fail;
+
+ peer_device->start_resync_side = L_SYNC_TARGET;
+ drbd_peer_device_post_work(peer_device, RS_START);
+ return 0;
+ }
+
+ /* peer says his disk is inconsistent, while we think it is uptodate,
+ * and this happens while the peer still thinks we have a sync going on,
+ * but we think we are already done with the sync.
+ * We ignore this to avoid flapping pdsk.
+ * This should not happen, if the peer is a recent version of drbd. */
+ if (old_peer_state.pdsk == D_UP_TO_DATE && peer_disk_state == D_INCONSISTENT &&
+ old_peer_state.conn == L_ESTABLISHED && peer_state.conn > L_SYNC_SOURCE)
+ peer_disk_state = D_UP_TO_DATE;
+
+ if (new_repl_state == L_OFF)
+ new_repl_state = L_ESTABLISHED;
+
+ if (peer_state.conn == L_AHEAD)
+ new_repl_state = L_BEHIND;
+
+ /* with protocol >= 118 uuid & state packets come after the 2PC prepare packet */
+ do_handshake =
+ (test_bit(UUIDS_RECEIVED, peer_device->flags) ||
+ test_bit(CURRENT_UUID_RECEIVED, peer_device->flags)) &&
+ (connection->agreed_pro_version < 118 ||
+ drbd_twopc_between_peer_and_me(connection)) &&
+ old_peer_state.conn < L_ESTABLISHED;
+
+ if (test_bit(UUIDS_RECEIVED, peer_device->flags) &&
+ peer_state.disk >= D_NEGOTIATING &&
+ get_ldev_if_state(device, D_NEGOTIATING)) {
+ enum sync_strategy strategy = UNDETERMINED;
+ bool consider_resync;
+
+ /* clear CONN_DISCARD_MY_DATA so late, to not lose it if peer
+ gets aborted before we are able to do the resync handshake. */
+ clear_bit(CONN_DISCARD_MY_DATA, &connection->flags);
+
+ /* if we established a new connection */
+ consider_resync = do_handshake &&
+ !test_bit(INITIAL_STATE_RECEIVED, peer_device->flags);
+ /* if we have both been inconsistent, and the peer has been
+ * forced to be UpToDate with --force */
+ consumed_resync = test_and_clear_bit(CONSIDER_RESYNC, peer_device->flags);
+ consider_resync |= consumed_resync;
+ /* if we had been plain connected, and the admin requested to
+ * start a sync by "invalidate" or "invalidate-remote" */
+ consider_resync |= (old_peer_state.conn == L_ESTABLISHED &&
+ (peer_state.conn == L_STARTING_SYNC_S ||
+ peer_state.conn == L_STARTING_SYNC_T));
+
+ /* A crashed primary announces WFBitMapT when the connection is
+ * established; that P_STATE races with the bitmap we already send.
+ * Only run a handshake for it while no resync is under way.
+ */
+ consider_resync |= old_peer_state.conn == L_ESTABLISHED &&
+ peer_state.conn == L_WF_BITMAP_T &&
+ peer_device->uuid_flags & UUID_FLAG_CRASHED_PRIMARY;
+
+ /* The peer decides on the UUIDs we send after its attach; so do we,
+ * whichever handshake runs below. Once the peer has decided, it
+ * no longer reports D_NEGOTIATING; sending now would make us
+ * decide on UUIDs it never saw.
+ */
+ if (old_peer_state.conn == L_ESTABLISHED && peer_state.disk == D_NEGOTIATING)
+ drbd_send_uuids_peer_attach(peer_device);
+
+ if (consider_resync) {
+ strategy = drbd_sync_handshake(peer_device, peer_state);
+ new_repl_state = strategy_to_repl_state(peer_device, peer_state.role, strategy);
+ /* An Outdated peer upgrades itself in this commit when it
+ * connects to us as a stable UpToDate neighbor without a
+ * resync (see sanitize_state()). Record the same here, or
+ * a write before its next P_STATE is not replicated to it.
+ */
+ if (old_peer_state.conn < L_ESTABLISHED &&
+ new_repl_state == L_ESTABLISHED &&
+ peer_disk_state == D_OUTDATED &&
+ device->disk_state[NOW] >= D_CONSISTENT &&
+ peer_device->comm_uuid_flags & UUID_FLAG_STABLE)
+ peer_disk_state = device->disk_state[NOW];
+ } else if (old_peer_state.conn == L_ESTABLISHED &&
+ (peer_state.disk == D_NEGOTIATING ||
+ old_peer_state.disk == D_NEGOTIATING)) {
+ strategy = drbd_attach_handshake(peer_device, peer_disk_state);
+ new_repl_state = strategy_to_repl_state(peer_device, peer_state.role, strategy);
+ if (new_repl_state == L_ESTABLISHED && device->disk_state[NOW] == D_UP_TO_DATE)
+ peer_disk_state = D_UP_TO_DATE;
+ }
+
+ put_ldev(device);
+ if (strategy_descriptor(strategy).reconnect) { /* retry connect */
+ maybe_force_secondary(peer_device);
+ if (connection->agreed_pro_version >= 118)
+ set_bit(CONN_HANDSHAKE_RETRY, &connection->flags);
+ else
+ return -EIO; /* retry connect */
+ } else if (strategy_descriptor(strategy).disconnect) {
+ if (device->disk_state[NOW] == D_NEGOTIATING) {
+ new_repl_state = L_NEG_NO_RESULT;
+ } else if (peer_state.disk == D_NEGOTIATING) {
+ if (connection->agreed_pro_version < 110) {
+ drbd_err(device, "Disk attach process on the peer node was aborted.\n");
+ peer_state.disk = D_DISKLESS;
+ peer_disk_state = D_DISKLESS;
+ } else {
+ /* The peer will decide later and let us know... */
+ peer_disk_state = D_NEGOTIATING;
+ }
+ } else {
+ if (test_and_clear_bit(CONN_DRY_RUN, &connection->flags))
+ return -EIO;
+ if (connection->agreed_pro_version >= 118)
+ set_bit(CONN_HANDSHAKE_DISCONNECT, &connection->flags);
+ else
+ goto fail;
+ }
+ }
+
+ if (device->disk_state[NOW] == D_NEGOTIATING) {
+ begin_state_chg_flags |= CS_FORCE_RECALC;
+ peer_device->negotiation_result = new_repl_state;
+ }
+ }
+
+ if (test_bit(UUIDS_RECEIVED, peer_device->flags) &&
+ peer_device->repl_state[NOW] == L_OFF && device->disk_state[NOW] == D_DISKLESS) {
+ u64 exposed_data_uuid = device->exposed_data_uuid;
+ u64 peer_current_uuid = peer_device->current_uuid;
+
+ drbd_info(peer_device, "my exposed UUID: %016llX\n", exposed_data_uuid);
+ drbd_uuid_dump_peer(peer_device, peer_device->dirty_bits, peer_device->uuid_flags);
+
+ /* I am diskless connecting to a peer with disk, check that UUID match
+ We only check if the peer claims to have D_UP_TO_DATE data. Only then is the
+ peer a source for my data anyways. */
+ if (exposed_data_uuid && peer_state.disk == D_UP_TO_DATE &&
+ (exposed_data_uuid & ~UUID_PRIMARY) == (peer_current_uuid & ~UUID_PRIMARY)) {
+ /* The peer is already on our current generation's label, so
+ * the pending predecessor->current reconcile relabel is moot.
+ */
+ clear_bit(RECONCILE_INJECT_CUR_UUID, peer_device->flags);
+ } else if (exposed_data_uuid && peer_state.disk == D_UP_TO_DATE &&
+ (exposed_data_uuid & ~UUID_PRIMARY) !=
+ (peer_current_uuid & ~UUID_PRIMARY)) {
+ diskless_with_peers_different_current_uuids(peer_device, &peer_disk_state);
+ }
+ if (!exposed_data_uuid && peer_state.disk == D_UP_TO_DATE) {
+ drbd_uuid_set_exposed(device, peer_current_uuid, true);
+ propagate_exposed_uuid(device);
+ }
+ }
+
+ /*
+ * A peer that finished a resync sends its UUIDs ahead of its state, so
+ * receive_uuids() saw them while the peer was still Inconsistent and
+ * did not follow it. Follow it now that it is UpToDate. A peer that
+ * attaches with UpToDate data of another generation is not followed.
+ */
+ if (test_bit(UUIDS_RECEIVED, peer_device->flags) &&
+ peer_device->repl_state[NOW] >= L_ESTABLISHED &&
+ device->disk_state[NOW] == D_DISKLESS &&
+ resource->role[NOW] == R_SECONDARY &&
+ peer_device->disk_state[NOW] == D_INCONSISTENT &&
+ peer_state.disk == D_UP_TO_DATE)
+ drbd_uuid_set_exposed(device, peer_device->current_uuid, true);
+
+ if (peer_device->repl_state[NOW] == L_OFF && peer_state.disk == D_DISKLESS && get_ldev(device)) {
+ u64 uuid_flags = 0;
+
+ drbd_collect_local_uuid_flags(peer_device, NULL);
+ drbd_uuid_dump_self(peer_device, peer_device->comm_bm_set, uuid_flags);
+ drbd_info(peer_device, "peer's exposed UUID: %016llX\n", peer_device->current_uuid);
+
+ if (peer_state.role == R_PRIMARY &&
+ (peer_device->current_uuid & ~UUID_PRIMARY) ==
+ (drbd_current_uuid(device) & ~UUID_PRIMARY)) {
+ /* Connecting to diskless primary peer. When the state change is committed,
+ * sanitize_state might set me D_UP_TO_DATE. Make sure the
+ * effective_size is set. */
+ peer_device->max_size = peer_device->c_size;
+ drbd_determine_dev_size(device, peer_device->max_size, 0, NULL);
+ }
+
+ put_ldev(device);
+ }
+
+ if (test_bit(HOLDING_UUID_READ_LOCK, peer_device->flags) ||
+ connection->agreed_pro_version < 110) {
+ struct drbd_transport *transport = &connection->transport;
+ /* Last packet of handshake received, disarm receive timeout */
+ transport->class->ops.set_rcvtimeo(transport, DATA_STREAM, MAX_SCHEDULE_TIMEOUT);
+ }
+
+ if (new_repl_state == L_ESTABLISHED && peer_disk_state == D_CONSISTENT &&
+ drbd_suspended(device) && peer_device->repl_state[NOW] < L_ESTABLISHED &&
+ drbd_gen_obligation_state(device) == GEN_OBL_ARMED) {
+ /* Do not allow RESEND for a rebooted peer. We can only allow this
+ * for temporary network outages!
+ */
+ drbd_err(peer_device, "Aborting Connect, can not thaw IO with an only Consistent peer\n");
+ /* gen-rotate reason: DEGRADE (abort connect; only-Consistent peer,
+ * cannot thaw). The connection is torn down below; the writes
+ * held towards this peer are stranded either way, so mint before
+ * any resume. A suspend-io stays in place until resume-io lifts
+ * it.
+ */
+ drbd_gen_obligation_mint_before_resume(device, NODE_MASK(peer_device->node_id));
+ begin_state_change(resource, &irq_flags, CS_HARD);
+ __change_cstate(connection, C_PROTOCOL_ERROR);
+ end_state_change(resource, &irq_flags, "abort-connect");
+ return -EIO;
+ }
+
+ clear_bit(RS_SOURCE_MISSED_END, peer_device->flags);
+ clear_bit(RS_PEER_MISSED_END, peer_device->flags);
+
+ assign_bit(PEER_QUORATE, peer_device->flags, peer_state.quorum);
+
+ if (do_handshake) {
+ /* Ignoring state packets before the 2PC; they are from aborted 2PCs */
+ bool done = test_bit(INITIAL_STATE_RECEIVED, peer_device->flags);
+
+ set_bit(INITIAL_STATE_RECEIVED, peer_device->flags);
+ if (connection->cstate[NOW] == C_CONNECTING) {
+ peer_device->connect_state.peer_isp =
+ peer_state.aftr_isp | peer_state.user_isp;
+
+ if (!done) {
+ peer_device->connect_state.conn = new_repl_state;
+ peer_device->connect_state.peer = peer_state.role;
+ peer_device->connect_state.pdsk = peer_disk_state;
+ }
+ wake_up(&connection->ee_wait);
+ finish_nested_twopc(connection);
+ }
+ }
+
+ /* State change will be performed when the two-phase commit is committed. */
+ if (connection->cstate[NOW] == C_CONNECTING)
+ return 0;
+
+ if (peer_state.conn == L_OFF) {
+ /* device/minor hot add on the peer of a minor already locally known */
+ if (peer_device->repl_state[NOW] == L_NEGOTIATING) {
+ drbd_send_enable_replication_next(peer_device);
+ drbd_send_sizes(peer_device, 0, 0);
+ drbd_send_uuids(peer_device, 0, 0);
+ }
+ drbd_send_current_state(peer_device);
+ }
+
+ begin_state_change(resource, &irq_flags, begin_state_chg_flags);
+ if (old_peer_state.i != drbd_get_peer_device_state(peer_device, NOW).i) {
+ if (consumed_resync)
+ set_bit(CONSIDER_RESYNC, peer_device->flags);
+ old_peer_state = drbd_get_peer_device_state(peer_device, NOW);
+ abort_state_change_locked(resource);
+ write_unlock_irq(&resource->state_rwlock);
+ goto retry;
+ }
+ if ((repl_is_sync_source(old_peer_state.conn) && new_repl_state == L_WF_BITMAP_S) ||
+ (repl_is_sync_target(old_peer_state.conn) && new_repl_state == L_WF_BITMAP_T)) {
+ /* Re-entering the bitmap exchange is refused while that resync
+ * runs, and a refused state change costs the connection here.
+ * Postpone it as drbd_resync() does.
+ */
+ peer_device->resync_again++;
+ new_repl_state = old_peer_state.conn;
+ drbd_info(peer_device, "Postponing bitmap exchange until the running resync has finished\n");
+ }
+ if (device->disk_state[NOW] != D_NEGOTIATING)
+ __change_repl_state(peer_device, new_repl_state);
+ __change_peer_role(connection, peer_state.role);
+ if (peer_state.disk != D_NEGOTIATING)
+ __change_peer_disk_state(peer_device, peer_disk_state);
+ __change_resync_susp_peer(peer_device, peer_state.aftr_isp | peer_state.user_isp);
+ repl_state = peer_device->repl_state;
+ if (repl_state[OLD] < L_ESTABLISHED && repl_state[NEW] >= L_ESTABLISHED)
+ resource->state_change_flags |= CS_HARD;
+
+ rv = end_state_change(resource, &irq_flags, "peer-state");
+ new_repl_state = peer_device->repl_state[NOW];
+
+ if (rv < SS_SUCCESS)
+ goto fail;
+
+ if (old_peer_state.conn > L_OFF) {
+ if (new_repl_state > L_ESTABLISHED && peer_state.conn <= L_ESTABLISHED &&
+ peer_state.disk != D_NEGOTIATING) {
+ /* we want resync, peer has not yet decided to sync... */
+ /* Nowadays only used when forcing a node into primary role and
+ setting its disk to UpToDate with that */
+ drbd_send_uuids(peer_device, 0, 0);
+ drbd_send_current_state(peer_device);
+ }
+ }
+
+ clear_bit(DISCARD_MY_DATA, peer_device->flags); /* for agreed_pro_version < 117 */
+
+ drbd_md_sync(device); /* update connected indicator, effective_size, ... */
+
+ peer_device->last_repl_state = peer_state.conn;
+ return 0;
+fail:
+ change_cstate(connection, C_DISCONNECTING, CS_HARD);
+ return -EIO;
+}
+
+static int receive_sync_uuid(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+ struct p_uuid *p = pi->data;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
+ device = peer_device->device;
+
+ wait_event(device->misc_wait,
+ peer_device->repl_state[NOW] == L_WF_SYNC_UUID ||
+ peer_device->repl_state[NOW] == L_BEHIND ||
+ peer_device->repl_state[NOW] < L_ESTABLISHED ||
+ device->disk_state[NOW] < D_NEGOTIATING);
+
+ /* D_ASSERT(device, peer_device->repl_state[NOW] == L_WF_SYNC_UUID ); */
+
+ /* Here the _drbd_uuid_ functions are right, current should
+ _not_ be rotated into the history */
+ if (get_ldev_if_state(device, D_NEGOTIATING)) {
+ _drbd_uuid_set_current(device, be64_to_cpu(p->uuid));
+ _drbd_uuid_set_bitmap(peer_device, 0UL);
+
+ drbd_print_uuids(peer_device, "updated sync uuid");
+ drbd_start_resync(peer_device, L_SYNC_TARGET, "peer-sync-uuid");
+
+ put_ldev(device);
+ } else
+ drbd_err(device, "Ignoring SyncUUID packet!\n");
+
+ return 0;
+}
+
+static void scale_bits(unsigned long *base, unsigned int num_4k, unsigned int scale)
+{
+ unsigned int bits = num_4k * BITS_PER_LONG;
+ unsigned int sbit;
+
+ if (scale == 0)
+ return;
+
+ for (sbit = 0; sbit < bits; sbit++)
+ if (test_bit(sbit, base))
+ __set_bit(sbit >> scale, base);
+}
+
+/*
+ * receive_bitmap_plain
+ *
+ * Return 0 when done, 1 when another iteration is needed, and a negative error
+ * code upon failure.
+ *
+ * Received bitmap is 4k per bit, need to aggregate by c->scale.
+ */
+static int
+receive_bitmap_plain(struct drbd_peer_device *peer_device, unsigned int size,
+ struct bm_xfer_ctx *c)
+{
+ unsigned long *p;
+ unsigned int data_size = DRBD_SOCKET_BUFFER_SIZE -
+ drbd_header_size(peer_device->connection);
+ unsigned int num_words_4k = min_t(size_t, data_size / sizeof(*p),
+ (c->bm_words - c->word_offset) << c->scale);
+ unsigned int want = num_words_4k * sizeof(*p);
+ int err;
+
+ if (want != size) {
+ drbd_err(peer_device, "%s:want (%u) != size (%u)\n", __func__, want, size);
+ return -EIO;
+ }
+ if (want == 0)
+ return 0;
+ err = drbd_recv_all(peer_device->connection, (void **)&p, want);
+ if (err)
+ return err;
+
+ if ((num_words_4k & ((1 << c->scale)-1)) != 0) {
+ drbd_err(peer_device,
+ "number of words %u not aligned to scale %u while receiving bitmap\n",
+ num_words_4k, c->scale);
+ return -ERANGE;
+ }
+
+ if (get_ldev(peer_device->device)) {
+ scale_bits(p, num_words_4k, c->scale);
+ drbd_bm_merge_lel(peer_device, c->word_offset, num_words_4k >> c->scale, p);
+ put_ldev(peer_device->device);
+ } else {
+ drbd_err(peer_device, "lost backend device while receiving bitmap\n");
+ return -EIO;
+ }
+
+ c->word_offset += num_words_4k >> c->scale;
+ c->bit_offset = c->word_offset * BITS_PER_LONG;
+ c->bit_offset_4k = (c->word_offset << c->scale) * BITS_PER_LONG;
+ if (c->bit_offset > c->bm_bits)
+ c->bit_offset = c->bm_bits;
+
+ return 1;
+}
+
+static enum drbd_bitmap_code dcbp_get_code(struct p_compressed_bm *p)
+{
+ return (enum drbd_bitmap_code)(p->encoding & 0x0f);
+}
+
+static int dcbp_get_start(struct p_compressed_bm *p)
+{
+ return (p->encoding & 0x80) != 0;
+}
+
+static int dcbp_get_pad_bits(struct p_compressed_bm *p)
+{
+ return (p->encoding >> 4) & 0x7;
+}
+
+/*
+ * recv_bm_rle_bits
+ *
+ * Return 0 when done, 1 when another iteration is needed, and a negative error
+ * code upon failure.
+ */
+static int
+recv_bm_rle_bits(struct drbd_peer_device *peer_device,
+ struct p_compressed_bm *p,
+ struct bm_xfer_ctx *c,
+ unsigned int len)
+{
+ struct bitstream bs;
+ u64 look_ahead;
+ u64 rl_4k;
+ u64 tmp;
+ unsigned long s_4k = c->bit_offset_4k;
+ int toggle = dcbp_get_start(p);
+ int have;
+ int bits;
+
+ bitstream_init(&bs, p->code, len, dcbp_get_pad_bits(p));
+
+ bits = bitstream_get_bits(&bs, &look_ahead, 64);
+ if (bits < 0)
+ return -EIO;
+
+ for (have = bits; have > 0; s_4k += rl_4k, toggle = !toggle) {
+ bits = vli_decode_bits(&rl_4k, look_ahead);
+ if (bits <= 0)
+ return -EIO;
+
+ if (toggle) {
+ /* If peers bm_block_size is smaller than ours,
+ * this may be a "partially" set bit ;-)
+ * there is no such thing. Round down s, round up e.
+ */
+ unsigned long s = s_4k >> c->scale;
+ unsigned long e = ((s_4k + rl_4k + (1UL << c->scale)-1) >> c->scale) - 1;
+
+ if (e >= c->bm_bits) {
+ drbd_err(peer_device, "bitmap overflow (e:%lu) while decoding bm RLE packet\n", e);
+ return -EIO;
+ }
+ drbd_bm_set_many_bits(peer_device, s, e);
+ }
+
+ if (have < bits) {
+ drbd_err(peer_device, "bitmap decoding error: h:%d b:%d la:0x%08llx l:%u/%u\n",
+ have, bits, look_ahead,
+ (unsigned int)(bs.cur.b - p->code),
+ (unsigned int)bs.buf_len);
+ return -EIO;
+ }
+ /* if we consumed all 64 bits, assign 0; >> 64 is "undefined"; */
+ if (likely(bits < 64))
+ look_ahead >>= bits;
+ else
+ look_ahead = 0;
+ have -= bits;
+
+ bits = bitstream_get_bits(&bs, &tmp, 64 - have);
+ if (bits < 0)
+ return -EIO;
+ look_ahead |= tmp << have;
+ have += bits;
+ }
+
+ c->bit_offset_4k = s_4k;
+ c->bit_offset = s_4k >> c->scale;
+ bm_xfer_ctx_bit_to_word_offset(c);
+
+ return (c->bit_offset_4k != c->bm_bits_4k);
+}
+
+/*
+ * decode_bitmap_c
+ *
+ * Return 0 when done, 1 when another iteration is needed, and a negative error
+ * code upon failure.
+ */
+static int
+decode_bitmap_c(struct drbd_peer_device *peer_device,
+ struct p_compressed_bm *p,
+ struct bm_xfer_ctx *c,
+ unsigned int len)
+{
+ if (dcbp_get_code(p) == RLE_VLI_Bits) {
+ struct drbd_device *device = peer_device->device;
+ int res;
+
+ if (!get_ldev(device)) {
+ drbd_err(peer_device, "lost backend device while receiving bitmap\n");
+ return -EIO;
+ }
+
+ res = recv_bm_rle_bits(peer_device, p, c, len - sizeof(*p));
+ put_ldev(device);
+ return res;
+ }
+
+ /* other variants had been implemented for evaluation,
+ * but have been dropped as this one turned out to be "best"
+ * during all our tests.
+ */
+
+ drbd_err(peer_device, "receive_bitmap_c: unknown encoding %u\n", p->encoding);
+ change_cstate(peer_device->connection, C_PROTOCOL_ERROR, CS_HARD);
+ return -EIO;
+}
+
+void INFO_bm_xfer_stats(struct drbd_peer_device *peer_device,
+ const char *direction, struct bm_xfer_ctx *c)
+{
+ /* what would it take to transfer it "plaintext" */
+ unsigned int header_size = drbd_header_size(peer_device->connection);
+ unsigned int data_size = DRBD_SOCKET_BUFFER_SIZE - header_size;
+ unsigned int plain =
+ header_size * (DIV_ROUND_UP(c->bm_words, data_size) + 1) +
+ c->bm_words * sizeof(unsigned long);
+ unsigned int total = c->bytes[0] + c->bytes[1];
+ unsigned int r;
+
+ /* total can not be zero. but just in case: */
+ if (total == 0)
+ return;
+
+ /* don't report if not compressed */
+ if (total >= plain)
+ return;
+
+ /* total < plain. check for overflow, still */
+ r = (total > UINT_MAX/1000) ? (total / (plain/1000))
+ : (1000 * total / plain);
+
+ if (r > 1000)
+ r = 1000;
+
+ r = 1000 - r;
+ drbd_info(peer_device, "%s bitmap stats [Bytes(packets)]: plain %u(%u), RLE %u(%u), "
+ "total %u; compression: %u.%u%%\n",
+ direction,
+ c->bytes[1], c->packets[1],
+ c->bytes[0], c->packets[0],
+ total, r/10, r % 10);
+}
+
+static bool ready_for_bitmap(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
+ bool ready = true;
+
+ read_lock_irq(&resource->state_rwlock);
+ if (device->disk_state[NOW] == D_NEGOTIATING)
+ ready = false;
+ if (test_bit(TWOPC_STATE_CHANGE_PENDING, &resource->flags))
+ ready = false;
+ read_unlock_irq(&resource->state_rwlock);
+
+ return ready;
+}
+
+/* Since we are processing the bitfield from lower addresses to higher,
+ it does not matter if the process it in 32 bit chunks or 64 bit
+ chunks as long as it is little endian. (Understand it as byte stream,
+ beginning with the lowest byte...) If we would use big endian
+ we would need to process it from the highest address to the lowest,
+ in order to be agnostic to the 32 vs 64 bits issue.
+
+ returns 0 on failure, 1 if we successfully received it. */
+static int receive_bitmap(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ enum drbd_repl_state repl_state;
+ struct drbd_device *device;
+ struct bm_xfer_ctx c;
+ int err = -EIO;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
+ if (peer_device->bitmap_index == -1) {
+ drbd_err(peer_device, "No bitmap allocated in receive_bitmap()!\n");
+ return -EIO;
+ }
+ device = peer_device->device;
+
+ /* Final repl_states become visible when the disk leaves NEGOTIATING state */
+ wait_event_interruptible(device->resource->state_wait,
+ ready_for_bitmap(device));
+
+ if (!get_ldev(device)) {
+ drbd_err(device, "Cannot receive bitmap, local disk gone\n");
+ return -EIO;
+ }
+
+ drbd_bm_slot_lock(peer_device, "receive bitmap", BM_LOCK_CLEAR | BM_LOCK_BULK);
+ /* you are supposed to send additional out-of-sync information
+ * if you actually set bits during this phase */
+
+ if (!get_ldev(device))
+ goto out;
+
+ c = (struct bm_xfer_ctx) {
+ .bm_bits_4k = drbd_bm_bits_4k(device),
+ .bm_bits = drbd_bm_bits(device),
+ .bm_words = drbd_bm_words(device),
+ .scale = device->bitmap->bm_block_shift - BM_BLOCK_SHIFT_4k,
+ };
+ put_ldev(device);
+
+ for (;;) {
+ if (pi->cmd == P_BITMAP)
+ err = receive_bitmap_plain(peer_device, pi->size, &c);
+ else if (pi->cmd == P_COMPRESSED_BITMAP) {
+ /* MAYBE: sanity check that we speak proto >= 90,
+ * and the feature is enabled! */
+ struct p_compressed_bm *p;
+
+ if (pi->size > DRBD_SOCKET_BUFFER_SIZE - drbd_header_size(connection)) {
+ drbd_err(device, "ReportCBitmap packet too large\n");
+ err = -EIO;
+ goto out;
+ }
+ if (pi->size <= sizeof(*p)) {
+ drbd_err(device, "ReportCBitmap packet too small (l:%u)\n", pi->size);
+ err = -EIO;
+ goto out;
+ }
+ err = drbd_recv_all(connection, (void **)&p, pi->size);
+ if (err)
+ goto out;
+ err = decode_bitmap_c(peer_device, p, &c, pi->size);
+ } else {
+ drbd_warn(device, "receive_bitmap: cmd neither ReportBitMap nor ReportCBitMap (is 0x%x)", pi->cmd);
+ err = -EIO;
+ goto out;
+ }
+
+ c.packets[pi->cmd == P_BITMAP]++;
+ c.bytes[pi->cmd == P_BITMAP] += drbd_header_size(connection) + pi->size;
+
+ if (err <= 0) {
+ if (err < 0)
+ goto out;
+ break;
+ }
+ err = drbd_recv_header(connection, pi);
+ if (err)
+ goto out;
+ }
+
+ INFO_bm_xfer_stats(peer_device, "receive", &c);
+
+ repl_state = peer_device->repl_state[NOW];
+ if (repl_state == L_WF_BITMAP_T) {
+ err = drbd_send_bitmap(device, peer_device);
+ if (err)
+ goto out;
+ }
+
+ drbd_bm_slot_unlock(peer_device);
+ put_ldev(device);
+
+ if (test_bit(B_RS_H_DONE, peer_device->flags)) {
+ /* We have entered drbd_start_resync() since starting the bitmap exchange. */
+ drbd_warn(peer_device, "Received bitmap more than once; ignoring\n");
+ } else if (repl_state == L_WF_BITMAP_S) {
+ drbd_start_resync(peer_device, L_SYNC_SOURCE, "receive-bitmap");
+ } else if (repl_state == L_WF_BITMAP_T) {
+ if (connection->agreed_pro_version < 110) {
+ enum drbd_state_rv rv;
+
+ /* Omit CS_WAIT_COMPLETE and CS_SERIALIZE with this state
+ * transition to avoid deadlocks. */
+ rv = stable_change_repl_state(peer_device, L_WF_SYNC_UUID, CS_VERBOSE,
+ "receive-bitmap");
+ D_ASSERT(device, rv == SS_SUCCESS);
+ } else {
+ drbd_start_resync(peer_device, L_SYNC_TARGET, "receive-bitmap");
+ }
+ } else {
+ /* admin may have requested C_DISCONNECTING,
+ * other threads may have noticed network errors */
+ drbd_info(peer_device, "unexpected repl_state (%s) in receive_bitmap\n",
+ drbd_repl_str(repl_state));
+ }
+
+ return 0;
+ out:
+ drbd_bm_slot_unlock(peer_device);
+ put_ldev(device);
+ return err;
+}
+
+static int receive_skip(struct drbd_connection *connection, struct packet_info *pi)
+{
+ drbd_warn(connection, "skipping unknown optional packet type %d, l: %d!\n",
+ pi->cmd, pi->size);
+
+ return ignore_remaining_packet(connection, pi->size);
+}
+
+static int receive_UnplugRemote(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_transport *transport = &connection->transport;
+
+ /* Make sure we've acked all the data associated
+ * with the data requests being unplugged */
+ transport->class->ops.hint(transport, DATA_STREAM, QUICKACK);
+
+ /* just unplug all devices always, regardless which volume number */
+ drbd_unplug_all_devices(connection);
+
+ return 0;
+}
+
+static int receive_out_of_sync(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct p_block_desc *p = pi->data;
+ sector_t sector;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
+
+ sector = be64_to_cpu(p->sector);
+
+ /* see also process_one_request(), before drbd_send_out_of_sync().
+ * Make sure any pending write requests that potentially may
+ * set in-sync have drained, before setting it out-of-sync.
+ * That should be implicit, because of the "epoch" and P_BARRIER logic,
+ * But let's just double-check.
+ */
+ conn_wait_active_ee_empty_or_disconnect(connection);
+ conn_wait_done_ee_empty_or_disconnect(connection);
+
+ drbd_set_out_of_sync(peer_device, sector, be32_to_cpu(p->blksize));
+
+ return 0;
+}
+
+static int receive_dagtag(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct p_dagtag *p = pi->data;
+
+ set_connection_dagtag(connection, be64_to_cpu(p->dagtag));
+ return 0;
+}
+
+struct drbd_connection *drbd_connection_by_node_id(struct drbd_resource *resource, int node_id)
+{
+ /* Caller needs to hold rcu_read_lock(), conf_update */
+ struct drbd_connection *connection;
+
+ for_each_connection_rcu(connection, resource) {
+ if (connection->peer_node_id == node_id)
+ return connection;
+ }
+
+ return NULL;
+}
+
+struct drbd_connection *drbd_get_connection_by_node_id(struct drbd_resource *resource, int node_id)
+{
+ struct drbd_connection *connection;
+
+ rcu_read_lock();
+ connection = drbd_connection_by_node_id(resource, node_id);
+ if (connection)
+ kref_get(&connection->kref);
+ rcu_read_unlock();
+
+ return connection;
+}
+
+static int receive_peer_dagtag(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ enum drbd_repl_state new_repl_state;
+ struct p_peer_dagtag *p = pi->data;
+ struct drbd_connection *lost_peer;
+ enum sync_strategy strategy = NO_SYNC;
+ s64 dagtag_offset;
+ int vnr = 0;
+
+ /*
+ * During the connect handshake (DRBD_FF_RECONCILE_RECONNECT) the peer
+ * sends its position in a common lost primary's change stream before its
+ * state packet. Stash it for drbd_uuid_compare(), which uses it to roll
+ * an equal-UUID both-dirty reconcile forward, and do not run the
+ * established-connection reconciliation below: the connection is not up
+ * yet, and the resync direction is decided synchronously in the handshake.
+ */
+ if (connection->cstate[NOW] < C_CONNECTED) {
+ connection->reconcile_handshake.lost_node_id = be32_to_cpu(p->node_id);
+ connection->reconcile_handshake.dagtag_sector = be64_to_cpu(p->dagtag);
+ return 0;
+ }
+
+ lost_peer = drbd_get_connection_by_node_id(resource, be32_to_cpu(p->node_id));
+ if (!lost_peer)
+ return 0;
+
+ if (lost_peer->cstate[NOW] == C_CONNECTED) {
+ drbd_ping_peer(lost_peer);
+ if (lost_peer->cstate[NOW] == C_CONNECTED)
+ goto out;
+ }
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ enum sync_strategy ps;
+ enum sync_rule rule;
+ int unused;
+
+ if (peer_device->repl_state[NOW] > L_ESTABLISHED)
+ goto out;
+ if (peer_device->device->disk_state[NOW] != D_CONSISTENT &&
+ peer_device->device->disk_state[NOW] != D_UP_TO_DATE)
+ goto out;
+ if (!get_ldev(peer_device->device))
+ continue;
+ ps = drbd_uuid_compare(peer_device, &rule, &unused);
+ put_ldev(peer_device->device);
+
+ if (strategy == NO_SYNC) {
+ strategy = ps;
+ if (strategy != NO_SYNC &&
+ strategy != SYNC_SOURCE_USE_BITMAP &&
+ strategy != SYNC_TARGET_USE_BITMAP) {
+ drbd_info(peer_device,
+ "%s(): %s by rule=%s\n",
+ __func__,
+ strategy_descriptor(strategy).name,
+ drbd_sync_rule_str(rule));
+ goto out;
+ }
+ } else if (ps != strategy) {
+ drbd_err(peer_device,
+ "%s(): Inconsistent resync directions %s %s\n",
+ __func__,
+ strategy_descriptor(strategy).name, strategy_descriptor(ps).name);
+ goto out;
+ }
+ }
+
+ /* We must wait until the other receiver thread has called the
+ * cleanup_unacked_peer_requests() and drbd_notify_peers_lost_primary() functions. If we
+ * become a resync target, the peer would complain about being in the wrong state when he
+ * gets the bitmap before the P_PEER_DAGTAG packet.
+ */
+ wait_event(resource->state_wait,
+ !test_bit(NOTIFY_PEERS_LOST_PRIMARY, &lost_peer->flags));
+
+ dagtag_offset = atomic64_read(&lost_peer->last_dagtag_sector) - (s64)be64_to_cpu(p->dagtag);
+ if (strategy == SYNC_SOURCE_USE_BITMAP) {
+ new_repl_state = L_WF_BITMAP_S;
+ } else if (strategy == SYNC_TARGET_USE_BITMAP) {
+ new_repl_state = L_WF_BITMAP_T;
+ } else {
+ if (dagtag_offset > 0)
+ new_repl_state = L_WF_BITMAP_S;
+ else if (dagtag_offset < 0)
+ new_repl_state = L_WF_BITMAP_T;
+ else
+ new_repl_state = L_ESTABLISHED;
+ }
+
+ if (new_repl_state != L_ESTABLISHED) {
+ unsigned long irq_flags;
+ enum drbd_state_rv rv;
+
+ if (new_repl_state == L_WF_BITMAP_T) {
+ connection->after_reconciliation.dagtag_sector = be64_to_cpu(p->dagtag);
+ connection->after_reconciliation.lost_node_id = be32_to_cpu(p->node_id);
+ }
+
+ /* The peer drops our P_PEER_DAGTAG while it still has the lost node
+ * connected. Send ours again, ahead of the bitmap, so it decides too.
+ */
+ if (new_repl_state == L_WF_BITMAP_S)
+ drbd_send_peer_dagtag(connection, lost_peer);
+
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ __change_repl_state(peer_device, new_repl_state);
+ set_bit(RECONCILIATION_RESYNC, peer_device->flags);
+ }
+ rv = end_state_change(resource, &irq_flags, "receive-peer-dagtag");
+ if (rv == SS_SUCCESS) {
+ drbd_info(connection, "Reconciliation resync because \'%s\' disappeared. (o=%d)\n",
+ lost_peer->transport.net_conf->name, (int)dagtag_offset);
+ if (new_repl_state == L_WF_BITMAP_T)
+ drbd_send_peer_dagtag(connection, lost_peer);
+ } else if (rv == SS_NOTHING_TO_DO) {
+ drbd_info(connection, "\'%s\' disappeared (o=%d), no reconciliation since one diskless\n",
+ lost_peer->transport.net_conf->name, (int)dagtag_offset);
+ /* sanitize_state() silently removes the resync and the RECONCILIATION_RESYNC bit */
+ } else {
+ drbd_info(connection, "rv = %d", rv);
+ }
+ } else {
+ drbd_info(connection, "No reconciliation resync even though \'%s\' disappeared. (o=%d)\n",
+ lost_peer->transport.net_conf->name, (int)dagtag_offset);
+
+ /* Only bits this loss set are covered by the equal dagtags.
+ * Do not consume the flag: a second primary lost at the same
+ * time shares it, and the next loss rewrites it anyway.
+ */
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (test_bit(BITMAP_HELD_AT_PRIMARY_LOSS, peer_device->flags)) {
+ drbd_info(peer_device, "Keeping out-of-sync bits from before\n");
+ continue;
+ }
+ if (get_ldev(peer_device->device)) {
+ drbd_bm_clear_many_bits(peer_device, 0, -1UL);
+ put_ldev(peer_device->device);
+ }
+ }
+ }
+
+ /* hand the bridge over to RECONCILIATION_RESYNC, or discharge if none owed */
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ clear_bit(RECONCILE_PENDING, peer_device->flags);
+ drbd_reconcile_settled_try_up_to_date(resource);
+
+out:
+ kref_put(&lost_peer->kref, drbd_destroy_connection);
+ return 0;
+}
+
+/* Whether any of our peer slots holds the UUID. Unlike
+ * drbd_find_bitmap_by_uuid() this is a local question, so it looks at every
+ * slot, not only at those the peer knows about.
+ */
+static bool uuid_in_my_bitmap_slots(struct drbd_peer_device *peer_device, u64 uuid)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int i;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ if (i == device->ldev->md.node_id)
+ continue;
+ if (peer_device->connection->agreed_pro_version < 116 &&
+ peer_md[i].bitmap_index == -1)
+ continue;
+ if ((peer_md[i].bitmap_uuid & ~UUID_PRIMARY) == uuid)
+ return true;
+ }
+
+ return false;
+}
+
+static bool drbd_diskless_moved_on(struct drbd_peer_device *peer_device, u64 current_uuid)
+{
+ struct drbd_device *device = peer_device->device;
+ u64 previous = peer_device->current_uuid;
+ bool from_the_past = false;
+
+ /* No exposed UUID => did not move on. */
+ if (!current_uuid)
+ return false;
+
+ /* Same as last time => did not move on. */
+ if ((previous & ~UUID_PRIMARY) == (current_uuid & ~UUID_PRIMARY))
+ return false;
+
+ /* Only consider the peer to have moved on if we were on the same UUID. */
+ if ((previous & ~UUID_PRIMARY) != (drbd_current_uuid(device) & ~UUID_PRIMARY))
+ return false;
+
+ if (get_ldev(device)) {
+ from_the_past =
+ uuid_in_my_bitmap_slots(peer_device, current_uuid & ~UUID_PRIMARY);
+ if (!from_the_past)
+ from_the_past = uuid_in_my_history(device, current_uuid & ~UUID_PRIMARY);
+ put_ldev(device);
+ }
+
+ /* It is an old UUID => did not move on. */
+ if (from_the_past)
+ return false;
+
+ return true;
+}
+
+/* Accept a new current UUID generated on a diskless node, that just became primary
+ (or during handshake) */
+static int receive_current_uuid(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+ struct p_current_uuid *p = pi->data;
+ u64 current_uuid, weak_nodes;
+ bool moved_on;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return config_unknown_volume(connection, pi);
+ device = peer_device->device;
+
+ current_uuid = be64_to_cpu(p->uuid);
+ weak_nodes = be64_to_cpu(p->weak_nodes);
+ weak_nodes |= NODE_MASK(peer_device->node_id);
+ moved_on = drbd_diskless_moved_on(peer_device, current_uuid);
+
+ peer_device->current_uuid = current_uuid;
+
+ if (get_ldev(device)) {
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[peer_device->node_id];
+
+ set_bit(__MDF_NODE_EXISTS, &peer_md->flags);
+ put_ldev(device);
+ }
+ if (connection->peer_role[NOW] == R_PRIMARY)
+ check_resync_source(peer_device, weak_nodes);
+
+ if (connection->peer_role[NOW] == R_UNKNOWN) {
+ set_bit(CURRENT_UUID_RECEIVED, peer_device->flags);
+ if (moved_on && device->disk_state[NOW] > D_OUTDATED)
+ peer_device->connect_state.disk = D_OUTDATED;
+ return 0;
+ }
+
+ if (current_uuid == drbd_current_uuid(device))
+ return 0;
+
+ /* None of this is serialized against concurrent state changes: moved_on
+ * may be stale by now, and disk_state[NOW] can change between the branch
+ * conditions. Deciding it reliably would mean making it part of a state
+ * change transaction.
+ */
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED &&
+ connection->peer_role[NOW] == R_PRIMARY &&
+ get_ldev(device)) {
+ /* drbd_uuid_received_new_current() dispatches on our disk state:
+ * adopt if D_UP_TO_DATE, defer into the sync source's record if
+ * resync target (so resync-end adoption brings us to the
+ * primary's generation), drop otherwise.
+ */
+ drbd_uuid_received_new_current(peer_device, current_uuid, weak_nodes);
+ drbd_md_sync_if_dirty(device);
+ put_ldev(device);
+ } else if (peer_device->repl_state[NOW] >= L_ESTABLISHED &&
+ connection->peer_role[NOW] != R_PRIMARY && moved_on &&
+ get_ldev_if_state(device, D_UP_TO_DATE)) {
+ /* The peer is not primary but moved on to a new generation. */
+ if (resource->remote_state_change)
+ set_bit(OUTDATE_ON_2PC_COMMIT, &device->flags);
+ else
+ change_disk_state(device, D_OUTDATED, CS_VERBOSE,
+ "receive-current-uuid", NULL);
+ put_ldev(device);
+ } else if (device->disk_state[NOW] == D_DISKLESS && resource->role[NOW] == R_PRIMARY) {
+ drbd_uuid_set_exposed(device, peer_device->current_uuid, true);
+ } else if (connection->peer_role[NOW] == R_PRIMARY) {
+ /* A primary's generation we can not use: no local disk to
+ * label it on, or this volume is not established with it.
+ */
+ drbd_info(peer_device, "ignoring new current UUID %016llX (disk %s)\n",
+ current_uuid, drbd_disk_str(device->disk_state[NOW]));
+ }
+
+ return 0;
+}
+
+static bool interval_is_adjacent(const struct drbd_interval *i1, const struct drbd_interval *i2)
+{
+ return i1->sector + (i1->size >> SECTOR_SHIFT) == i2->sector;
+}
+
+/* Advance caching pointers received_last and discard_last. Return next discard to be submitted. */
+static struct drbd_peer_request *drbd_advance_to_next_rs_discard(
+ struct drbd_peer_device *peer_device, unsigned int align, bool submit_all)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_request *peer_req;
+ struct drbd_peer_request *discard_last = peer_device->discard_last;
+ bool discard_range_end = false;
+
+ /* Advance received_last. */
+ peer_req = list_prepare_entry(peer_device->received_last,
+ &peer_device->resync_requests, recv_order);
+ list_for_each_entry_continue(peer_req, &peer_device->resync_requests, recv_order) {
+ if (!test_bit(INTERVAL_RECEIVED, &peer_req->i.flags) && !submit_all)
+ break;
+
+ if (peer_req->flags & EE_TRIM)
+ break;
+
+ peer_device->received_last = peer_req;
+ }
+
+ /* Advance discard_last. */
+ peer_req = discard_last ? discard_last :
+ list_prepare_entry(peer_device->received_last,
+ &peer_device->resync_requests, recv_order);
+ list_for_each_entry_continue(peer_req, &peer_device->resync_requests, recv_order) {
+ /* Consider submitting previous discards. */
+ if (discard_last && !interval_is_adjacent(&discard_last->i, &peer_req->i)) {
+ discard_range_end = true;
+ break;
+ }
+
+ if (!(peer_req->flags & EE_TRIM)) {
+ discard_range_end =
+ test_bit(INTERVAL_RECEIVED, &peer_req->i.flags) || submit_all;
+ break;
+ }
+
+ discard_last = peer_req;
+
+ if (IS_ALIGNED(peer_req->i.sector + (peer_req->i.size >> SECTOR_SHIFT), align)) {
+ discard_range_end = true;
+ break;
+ }
+ }
+
+ /*
+ * If we haven't found a discard range, or that range is not
+ * finished, then there is nothing to submit.
+ */
+ if (!discard_last || !(discard_range_end || discard_last->flags & EE_LAST_RESYNC_REQUEST)) {
+ peer_device->discard_last = discard_last;
+ return NULL;
+ }
+
+ /* Find start of discard range. */
+ peer_req = list_next_entry(list_prepare_entry(peer_device->received_last,
+ &peer_device->resync_requests, recv_order), recv_order);
+
+ spin_lock(&device->interval_lock); /* irqs already disabled */
+ if (peer_req != discard_last) {
+ struct drbd_peer_request *peer_req_merged = peer_req;
+
+ list_for_each_entry_continue(peer_req_merged,
+ &peer_device->resync_requests, recv_order) {
+ drbd_remove_interval(&device->requests, &peer_req_merged->i);
+ drbd_clear_interval(&peer_req_merged->i);
+ peer_req_merged->w.cb = e_end_resync_block;
+ if (peer_req_merged == discard_last)
+ break;
+ }
+ }
+ drbd_update_interval_size(&peer_req->i,
+ discard_last->i.size +
+ ((discard_last->i.sector - peer_req->i.sector) << SECTOR_SHIFT));
+ spin_unlock(&device->interval_lock);
+
+ peer_device->received_last = discard_last;
+ peer_device->discard_last = NULL;
+
+ return peer_req;
+}
+
+static void drbd_submit_rs_discard(struct drbd_peer_request *peer_req)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+
+ if (get_ldev(device)) {
+ list_del(&peer_req->w.list);
+
+ peer_req->w.cb = e_end_resync_block;
+ peer_req->bios.head->bi_opf = REQ_OP_DISCARD;
+
+ atomic_inc(&connection->backing_ee_cnt);
+ drbd_conflict_submit_resync_request(peer_req);
+
+ /* No put_ldev() here. Gets called in drbd_endio_write_sec_final(). */
+ } else {
+ LIST_HEAD(free_list);
+ struct drbd_peer_request *t;
+
+ if (drbd_ratelimit())
+ drbd_err(device, "Cannot discard on local disk.\n");
+
+ drbd_send_ack(peer_device, P_RS_NEG_ACK, peer_req);
+
+ drbd_remove_peer_req_interval(peer_req);
+ list_move_tail(&peer_req->w.list, &free_list);
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ drbd_unmerge_discard(peer_req, &free_list);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ list_for_each_entry_safe(peer_req, t, &free_list, w.list)
+ drbd_free_peer_req(peer_req);
+ }
+}
+
+/* Find and submit discards in resync_requests which are ready. */
+void drbd_process_rs_discards(struct drbd_peer_device *peer_device, bool submit_all)
+{
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_request *peer_req;
+ struct drbd_peer_request *pr_tmp;
+ unsigned int align = DRBD_MAX_RS_DISCARD_SIZE;
+ LIST_HEAD(work_list);
+
+ if (get_ldev(device)) {
+ /*
+ * Limit the size of the merged requests. We want to allow the size to
+ * increase up to the backing discard granularity. If that is smaller
+ * than DRBD_MAX_RS_DISCARD_SIZE, then allow merging up to a size of
+ * DRBD_MAX_RS_DISCARD_SIZE.
+ */
+ align = max(DRBD_MAX_RS_DISCARD_SIZE, bdev_discard_granularity(
+ device->ldev->backing_bdev)) >> SECTOR_SHIFT;
+ put_ldev(device);
+ }
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ while (true) {
+ peer_req = drbd_advance_to_next_rs_discard(peer_device, align, submit_all);
+ if (!peer_req)
+ break;
+
+ list_add_tail(&peer_req->w.list, &work_list);
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ list_for_each_entry_safe(peer_req, pr_tmp, &work_list, w.list)
+ drbd_submit_rs_discard(peer_req); /* removes it from the work_list */
+}
+
+static int receive_rs_deallocated(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_device *device;
+ struct drbd_peer_request *peer_req;
+ sector_t sector;
+ int size, err = 0;
+ u64 block_id;
+ u64 im;
+
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
+ device = peer_device->device;
+
+ if (pi->cmd == P_RS_DEALLOCATED) {
+ struct p_block_desc *p = pi->data;
+
+ sector = be64_to_cpu(p->sector);
+ size = be32_to_cpu(p->blksize);
+ block_id = ID_SYNCER;
+ } else { /* P_RS_DEALLOCATED_ID */
+ struct p_block_ack *p = pi->data;
+
+ sector = be64_to_cpu(p->sector);
+ size = be32_to_cpu(p->blksize);
+ block_id = p->block_id;
+ }
+
+ peer_req = find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_RESYNC_WRITE),
+ sector, size, block_id);
+ if (!peer_req)
+ return -EIO;
+
+ dec_rs_pending(peer_device);
+ inc_unacked(peer_device);
+ peer_req->flags |= EE_TRIM;
+
+ /* Setting all peers out of sync here. The sync source peer will be
+ * set in sync when the discard completes. The sync source will soon
+ * set other peers in sync with a P_PEERS_IN_SYNC packet.
+ */
+ drbd_set_all_out_of_sync(device, peer_device->node_id, sector, size);
+ drbd_process_rs_discards(peer_device, false);
+ rs_sectors_came_in(peer_device, size);
+
+ for_each_peer_device_ref(peer_device, im, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+
+ if (repl_is_sync_source(repl_state) || repl_state == L_WF_BITMAP_S)
+ drbd_send_out_of_sync(peer_device, sector, size);
+ }
+
+ return err;
+}
+
+void drbd_last_resync_request(struct drbd_peer_device *peer_device, bool submit_all)
+{
+ struct drbd_connection *connection = peer_device->connection;
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ if (!list_empty(&peer_device->resync_requests)) {
+ struct drbd_peer_request *peer_req = list_last_entry(&peer_device->resync_requests,
+ struct drbd_peer_request, recv_order);
+ peer_req->flags |= EE_LAST_RESYNC_REQUEST;
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ drbd_process_rs_discards(peer_device, submit_all);
+}
+
+static int receive_disconnect(struct drbd_connection *connection, struct packet_info *pi)
+{
+ change_cstate_tag(connection, C_DISCONNECTING, CS_HARD, "receive-disconnect", NULL);
+ return 0;
+}
+
+struct data_cmd {
+ int expect_payload;
+ unsigned int pkt_size;
+ int (*fn)(struct drbd_connection *, struct packet_info *);
+};
+
+static struct data_cmd drbd_cmd_handler[] = {
+ [P_DATA] = { 1, sizeof(struct p_data), receive_Data },
+ [P_DATA_REPLY] = { 1, sizeof(struct p_data), receive_DataReply },
+ [P_RS_DATA_REPLY] = { 1, sizeof(struct p_data), receive_RSDataReply },
+ [P_BARRIER] = { 0, sizeof(struct p_barrier), receive_Barrier },
+ [P_BITMAP] = { 1, 0, receive_bitmap },
+ [P_COMPRESSED_BITMAP] = { 1, 0, receive_bitmap },
+ [P_UNPLUG_REMOTE] = { 0, 0, receive_UnplugRemote },
+ [P_DATA_REQUEST] = { 0, sizeof(struct p_block_req), receive_data_request },
+ [P_RS_DATA_REQUEST] = { 0, sizeof(struct p_block_req), receive_data_request },
+ [P_SYNC_PARAM] = { 1, 0, receive_SyncParam },
+ [P_SYNC_PARAM89] = { 1, 0, receive_SyncParam },
+ [P_PROTOCOL] = { 1, sizeof(struct p_protocol), receive_protocol },
+ [P_UUIDS] = { 0, sizeof(struct p_uuids), receive_uuids },
+ [P_SIZES] = { 0, sizeof(struct p_sizes), receive_sizes },
+ [P_STATE] = { 0, sizeof(struct p_state), receive_state },
+ [P_STATE_CHG_REQ] = { 0, sizeof(struct p_req_state), receive_req_state },
+ [P_SYNC_UUID] = { 0, sizeof(struct p_uuid), receive_sync_uuid },
+ [P_OV_REQUEST] = { 0, sizeof(struct p_block_req), receive_data_request },
+ [P_OV_REPLY] = { 1, sizeof(struct p_block_req), receive_ov_reply },
+ [P_CSUM_RS_REQUEST] = { 1, sizeof(struct p_block_req), receive_data_request },
+ [P_RS_THIN_REQ] = { 0, sizeof(struct p_block_req), receive_data_request },
+ [P_DELAY_PROBE] = { 0, sizeof(struct p_delay_probe93), receive_skip },
+ [P_OUT_OF_SYNC] = { 0, sizeof(struct p_block_desc), receive_out_of_sync },
+ [P_CONN_ST_CHG_REQ] = { 0, sizeof(struct p_req_state), receive_req_state },
+ [P_PROTOCOL_UPDATE] = { 1, sizeof(struct p_protocol), receive_protocol },
+ [P_TWOPC_PREPARE] = { 0, sizeof(struct p_twopc_request), receive_twopc },
+ [P_TWOPC_PREP_RSZ] = { 0, sizeof(struct p_twopc_request), receive_twopc },
+ [P_TWOPC_ABORT] = { 0, sizeof(struct p_twopc_request), receive_twopc },
+ [P_DAGTAG] = { 0, sizeof(struct p_dagtag), receive_dagtag },
+ [P_UUIDS110] = { 1, sizeof(struct p_uuids110), receive_uuids110 },
+ [P_PEER_DAGTAG] = { 0, sizeof(struct p_peer_dagtag), receive_peer_dagtag },
+ [P_CURRENT_UUID] = { 0, sizeof(struct p_current_uuid), receive_current_uuid },
+ [P_TWOPC_COMMIT] = { 0, sizeof(struct p_twopc_request), receive_twopc },
+ [P_TRIM] = { 0, sizeof(struct p_trim), receive_Data },
+ [P_ZEROES] = { 0, sizeof(struct p_trim), receive_Data },
+ [P_RS_DEALLOCATED] = { 0, sizeof(struct p_block_desc), receive_rs_deallocated },
+ [P_RS_DEALLOCATED_ID] = { 0, sizeof(struct p_block_ack), receive_rs_deallocated },
+ [P_DISCONNECT] = { 0, 0, receive_disconnect },
+ [P_RS_DAGTAG_REQ] = { 0, sizeof(struct p_rs_req), receive_dagtag_data_request },
+ [P_RS_CSUM_DAGTAG_REQ] = { 1, sizeof(struct p_rs_req), receive_dagtag_data_request },
+ [P_RS_THIN_DAGTAG_REQ] = { 0, sizeof(struct p_rs_req), receive_dagtag_data_request },
+ [P_OV_DAGTAG_REQ] = { 0, sizeof(struct p_rs_req), receive_dagtag_data_request },
+ [P_OV_DAGTAG_REPLY] = { 1, sizeof(struct p_rs_req), receive_dagtag_ov_reply },
+ [P_FLUSH_REQUESTS] = { 0, sizeof(struct p_flush_requests), receive_flush_requests },
+ [P_FLUSH_REQUESTS_ACK] = { 0, sizeof(struct p_flush_ack), receive_flush_requests_ack },
+ [P_ENABLE_REPLICATION_NEXT] = { 0, sizeof(struct p_enable_replication),
+ receive_enable_replication_next },
+ [P_ENABLE_REPLICATION] = { 0, sizeof(struct p_enable_replication),
+ receive_enable_replication },
+ [P_RS_DAGTAG_WAIT_REQ] = { 0, sizeof(struct p_rs_req), receive_rs_dagtag_wait_req },
+};
+
+static void drbdd(struct drbd_connection *connection)
+{
+ struct packet_info pi;
+ size_t shs; /* sub header size */
+ int err;
+
+ while (get_t_state(&connection->receiver) == RUNNING) {
+ struct data_cmd const *cmd;
+
+ drbd_thread_current_set_cpu(&connection->receiver);
+ update_receiver_timing_details(connection, drbd_recv_header_maybe_unplug);
+ if (drbd_recv_header_maybe_unplug(connection, &pi))
+ goto err_out;
+
+ cmd = &drbd_cmd_handler[pi.cmd];
+ if (unlikely(pi.cmd >= ARRAY_SIZE(drbd_cmd_handler) || !cmd->fn)) {
+ drbd_err(connection, "Unexpected data packet %s (0x%04x)",
+ drbd_packet_name(pi.cmd), pi.cmd);
+ goto err_out;
+ }
+
+ shs = cmd->pkt_size;
+ if (pi.cmd == P_SIZES && connection->agreed_features & DRBD_FF_WSAME)
+ shs += sizeof(struct o_qlim);
+ if (pi.size > shs && !cmd->expect_payload) {
+ drbd_err(connection, "No payload expected %s l:%d\n",
+ drbd_packet_name(pi.cmd), pi.size);
+ goto err_out;
+ }
+ if (pi.size < shs) {
+ drbd_err(connection, "%s: unexpected packet size, expected:%d received:%d\n",
+ drbd_packet_name(pi.cmd), (int)shs, pi.size);
+ goto err_out;
+ }
+
+ if (shs) {
+ update_receiver_timing_details(connection, drbd_recv_all_warn);
+ err = drbd_recv_all_warn(connection, &pi.data, shs);
+ if (err)
+ goto err_out;
+ pi.size -= shs;
+ }
+
+ update_receiver_timing_details(connection, cmd->fn);
+ err = cmd->fn(connection, &pi);
+ if (err) {
+ drbd_err(connection, "error receiving %s, e: %d l: %d!\n",
+ drbd_packet_name(pi.cmd), err, pi.size);
+ goto err_out;
+ }
+ }
+ return;
+
+err_out:
+ change_cstate(connection, C_PROTOCOL_ERROR, CS_HARD);
+}
+
+static void drbd_cancel_conflicting_resync_requests(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct conflict_worker *submit_conflict = &device->submit_conflict;
+ struct rb_node *node;
+ bool any_queued = false;
+
+ spin_lock_irq(&device->interval_lock);
+ for (node = rb_first(&device->requests); node; node = rb_next(node)) {
+ struct drbd_interval *i = rb_entry(node, struct drbd_interval, rb);
+ struct drbd_peer_request *peer_req;
+
+ if (!drbd_interval_is_resync(i))
+ continue;
+
+ peer_req = container_of(i, struct drbd_peer_request, i);
+
+ if (peer_req->peer_device != peer_device)
+ continue;
+
+ /* Only cancel requests which are waiting for conflicts to resolve. */
+ if (test_bit(INTERVAL_SUBMITTED, &i->flags) ||
+ (test_bit(INTERVAL_READY_TO_SEND, &i->flags) &&
+ !test_bit(INTERVAL_RECEIVED, &i->flags)) ||
+ test_bit(INTERVAL_CANCELED, &i->flags))
+ continue;
+
+ set_bit(INTERVAL_CANCELED, &i->flags);
+
+ dynamic_drbd_dbg(device,
+ "Cancel %s %s request at %llus+%u (sent=%d)\n",
+ test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags) ?
+ "already queued" : "unqueued",
+ drbd_interval_type_str(i),
+ (unsigned long long) i->sector, i->size,
+ test_bit(INTERVAL_READY_TO_SEND, &i->flags));
+
+ /* Parked at the depth bound, it would wait for a completion. */
+ if (test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags)) {
+ /* No longer outstanding, before a new resync resets the count. */
+ if (peer_req->flags & EE_RS_DEPTH_PARKED) {
+ peer_req->flags &= ~EE_RS_DEPTH_PARKED;
+ atomic_sub(i->size >> 9, &peer_device->rs_sect_parked);
+ }
+ any_queued = true;
+ continue;
+ }
+
+ set_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags);
+
+ spin_lock(&submit_conflict->lock);
+ switch (i->type) {
+ case INTERVAL_RESYNC_WRITE:
+ list_add_tail(&peer_req->w.list, &submit_conflict->resync_writes);
+ break;
+ case INTERVAL_RESYNC_READ:
+ list_add_tail(&peer_req->w.list, &submit_conflict->resync_reads);
+ break;
+ default:
+ drbd_err(peer_device, "Unexpected interval type in %s\n", __func__);
+ }
+ spin_unlock(&submit_conflict->lock);
+
+ any_queued = true;
+ }
+ spin_unlock_irq(&device->interval_lock);
+
+ if (any_queued)
+ queue_work(submit_conflict->wq, &submit_conflict->worker);
+}
+
+/* Peer writes that are held waiting for a conflicting interval to resolve keep
+ * active_ee_cnt above zero. If the conflict cannot resolve, conn_disconnect()
+ * would wait for active_ee_cnt forever. Cancelling such writes is okay, since
+ * they could have been lost in flight as well.
+ */
+static void cancel_conflicting_peer_writes(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct conflict_worker *submit_conflict = &device->submit_conflict;
+ struct rb_node *node;
+ bool any_queued = false;
+
+ spin_lock_irq(&device->interval_lock);
+ for (node = rb_first(&device->requests); node; node = rb_next(node)) {
+ struct drbd_interval *i = rb_entry(node, struct drbd_interval, rb);
+ struct drbd_peer_request *peer_req;
+
+ if (i->type != INTERVAL_PEER_WRITE)
+ continue;
+
+ peer_req = container_of(i, struct drbd_peer_request, i);
+
+ if (peer_req->peer_device != peer_device)
+ continue;
+
+ if (test_bit(INTERVAL_SUBMITTED, &i->flags) ||
+ test_bit(INTERVAL_CANCELED, &i->flags))
+ continue;
+
+ set_bit(INTERVAL_CANCELED, &i->flags);
+
+ if (test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags))
+ continue;
+
+ set_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags);
+
+ spin_lock(&submit_conflict->lock);
+ list_add_tail(&peer_req->w.list, &submit_conflict->peer_writes);
+ spin_unlock(&submit_conflict->lock);
+
+ any_queued = true;
+ }
+ spin_unlock_irq(&device->interval_lock);
+
+ if (any_queued)
+ queue_work(submit_conflict->wq, &submit_conflict->worker);
+}
+
+static void cancel_dagtag_dependent_requests(struct drbd_resource *resource, unsigned int node_id)
+{
+ struct drbd_connection *connection;
+ LIST_HEAD(work_list);
+ struct drbd_peer_request *peer_req, *t;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry_safe(peer_req, t, &connection->dagtag_wait_ee, w.list) {
+ if (peer_req->depend_dagtag_node_id != node_id)
+ continue;
+
+ dynamic_drbd_dbg(peer_req->peer_device, "%s at %llus+%u: Wait for dagtag %llus from peer %u cancelled\n",
+ drbd_interval_type_str(&peer_req->i),
+ (unsigned long long) peer_req->i.sector, peer_req->i.size,
+ (unsigned long long) peer_req->depend_dagtag,
+ node_id);
+
+ list_move_tail(&peer_req->w.list, &work_list);
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ }
+ rcu_read_unlock();
+
+ list_for_each_entry_safe(peer_req, t, &work_list, w.list) {
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+
+ drbd_peer_resync_read_cancel(peer_req);
+ /* Verify requests on the source side are placed in the
+ * interval tree when the request is made; they need to be
+ * removed if the reply was parked waiting for a dagtag.
+ * Mirrors free_dagtag_wait_requests().
+ */
+ if (peer_req->i.type == INTERVAL_OV_READ_SOURCE)
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ dec_unacked(peer_device);
+ put_ldev(peer_device->device);
+ }
+}
+
+static void cleanup_resync_leftovers(struct drbd_peer_device *peer_device)
+{
+ peer_device->rs_total = 0;
+ peer_device->rs_failed = 0;
+ D_ASSERT(peer_device, atomic_read(&peer_device->rs_pending_cnt) == 0);
+
+ timer_delete_sync(&peer_device->resync_timer);
+ resync_timer_fn(&peer_device->resync_timer);
+ timer_delete_sync(&peer_device->start_resync_timer);
+}
+
+static void free_waiting_resync_requests(struct drbd_connection *connection)
+{
+ LIST_HEAD(free_list);
+ struct drbd_peer_device *peer_device;
+ struct drbd_peer_request *peer_req, *t;
+ int vnr;
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ list_for_each_entry_safe(peer_req, t, &peer_device->resync_requests, recv_order) {
+ drbd_list_del_resync_request(peer_req);
+ list_add_tail(&peer_req->w.list, &free_list);
+ }
+ }
+ rcu_read_unlock();
+
+ list_for_each_entry_safe(peer_req, t, &connection->peer_reads, recv_order) {
+ if (peer_req->i.type == INTERVAL_PEER_READ)
+ continue;
+
+ peer_req->flags &= ~EE_ON_RECV_ORDER;
+ list_del(&peer_req->recv_order);
+
+ list_add_tail(&peer_req->w.list, &free_list);
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ list_for_each_entry_safe(peer_req, t, &free_list, w.list) {
+ /*
+ * Resync write requests waiting for peers-in-sync to be sent
+ * just need to be freed.
+ */
+ if (test_bit(INTERVAL_COMPLETED, &peer_req->i.flags)) {
+ drbd_free_peer_req(peer_req);
+ continue;
+ }
+
+ D_ASSERT(connection, test_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags));
+ D_ASSERT(connection, !test_bit(INTERVAL_RECEIVED, &peer_req->i.flags));
+ D_ASSERT(connection, !(peer_req->flags & EE_TRIM));
+
+ if (peer_req->i.type == INTERVAL_RESYNC_READ)
+ atomic_sub(peer_req->i.size >> 9, &connection->rs_in_flight);
+
+ dec_rs_pending(peer_req->peer_device);
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+ }
+}
+
+static void free_dagtag_wait_requests(struct drbd_connection *connection)
+{
+ LIST_HEAD(dagtag_wait_work_list);
+ struct drbd_peer_request *peer_req, *t;
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_splice_init(&connection->dagtag_wait_ee, &dagtag_wait_work_list);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ list_for_each_entry_safe(peer_req, t, &dagtag_wait_work_list, w.list) {
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+
+ /* Verify requests are placed in the interval tree when
+ * the request is made, so they need to be removed if
+ * the reply was waiting for a dagtag to be reached. */
+ if (peer_req->i.type == INTERVAL_OV_READ_SOURCE)
+ drbd_remove_peer_req_interval(peer_req);
+
+ drbd_free_peer_req(peer_req);
+ dec_unacked(peer_device);
+ put_ldev(peer_device->device);
+ }
+}
+
+/* Writes waiting for a sync source to have them as well. The connection to
+ * the writer is going down, so the acknowledgment they wait for has no
+ * receiver any more; end them as the cancelled writes they now are.
+ */
+static void free_source_wait_requests(struct drbd_connection *connection)
+{
+ LIST_HEAD(work_list);
+ struct drbd_peer_request *peer_req, *t;
+
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_splice_init(&connection->source_wait_ee, &work_list);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ list_for_each_entry_safe(peer_req, t, &work_list, w.list) {
+ /* Even a cancelled write differs from the source's content;
+ * record that, so a later resync reconciles it.
+ */
+ drbd_mark_unsecured_write_out_of_sync(peer_req);
+ peer_req->flags &= ~(EE_WAIT_FOR_SOURCE | EE_SOURCE_REACHED |
+ EE_SOURCE_UNREACHABLE);
+ e_end_block_tail(peer_req, 1);
+ }
+}
+
+static void drain_resync_activity(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ /*
+ * In order to understand this function, refer to the flow diagrams in
+ * the comments for make_resync_request(), make_ov_request() and
+ * receive_dagtag_data_request().
+ */
+
+ /* Refuse further dagtag waits on this stream before the walks below
+ * empty the wait lists: from here on a waiter takes the not-reachable
+ * branch instead of queueing behind a walk that already passed.
+ */
+ set_bit(DAGTAG_STREAM_GONE, &connection->flags);
+
+ /*
+ * We could receive data from a peer at any point. This might release a
+ * request that is waiting for a dagtag. That would cause it to
+ * progress to waiting for conflicts or the backing disk. So we need to
+ * remove these requests before flushing the other stages.
+ */
+ free_dagtag_wait_requests(connection);
+
+ /* Writes withheld from acknowledgment have lost their receiver. */
+ free_source_wait_requests(connection);
+
+ /* Wait for w_resync_timer/w_e_send_csum to finish, if running. */
+ drbd_flush_workqueue(&connection->sender_work);
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ kref_get(&device->kref);
+ rcu_read_unlock();
+
+ /* Cause remaining discards to be submitted. */
+ drbd_last_resync_request(peer_device, true);
+ /* Cause requests waiting due to conflicts to be canceled. */
+ drbd_cancel_conflicting_resync_requests(peer_device);
+
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+
+ /* Drain conflicting and backing requests. */
+ wait_event(connection->ee_wait, atomic_read(&connection->backing_ee_cnt) == 0);
+
+ /* Wait for work queued when backing requests finished. */
+ drbd_flush_workqueue(&connection->sender_work);
+
+ /* Clear up and remove requests that have progressed to done_ee. */
+ drbd_finish_peer_reqs(connection);
+
+ /*
+ * Requests that are waiting for a resync reply must be removed from
+ * the interval tree and then freed.
+ */
+ free_waiting_resync_requests(connection);
+
+ /* Requests that are waiting for a dagtag on this connection must be
+ * cancelled, because the dependency will never be fulfilled. */
+ cancel_dagtag_dependent_requests(connection->resource, connection->peer_node_id);
+
+ /* The same for dagtag wait requests parked on this connection's stream. */
+ answer_dagtag_wait_reqs(connection, 0, true);
+
+ /* And drop the dagtag wait requests this connection's peer asked
+ * elsewhere: their answers just lost their receiver.
+ */
+ drop_dagtag_wait_reqs_of(connection);
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ kref_get(&device->kref);
+ rcu_read_unlock();
+
+ cleanup_resync_leftovers(peer_device);
+
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+}
+
+/* True if the transfer log holds a write that completed to the application
+ * with success, or is about to, and that the lost peer may lack.
+ *
+ * The loss gives up every request of this peer that is not RQ_NET_DONE, and
+ * drbd_req_destroy() then marks the block out of sync towards it. Whether
+ * that records a divergence worth a generation depends on the completion:
+ *
+ * Completed with success. Under protocol C, RQ_NET_OK is the peer's write
+ * ack, so the peer holds the data; without it the peer refused the write
+ * (P_NEG_ACK) and lacks it. Under protocol A and B the ack preceded the
+ * peer's write, so the peer may lack it either way.
+ *
+ * Not completed yet. Under protocol A the write counts as successful
+ * towards the peer since it was handed to the network, so it completes with
+ * success once the local write ends -- unless the io-error policy is in
+ * effect, which errors every completion. Under protocol B and C the request
+ * still waits for the peer, and the loss itself decides its completion; that
+ * decision is the settle-round mint of peer_device_disconnected().
+ *
+ * Completed with an error. Nothing is owed, whatever the local disk holds:
+ * the application was told the write failed, so either resync direction is a
+ * correct recovery, and a generation here would turn a takeover by the peers
+ * into a split brain.
+ */
+static bool peer_may_lack_completed_write(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_request *req;
+ bool found = false;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(req, &device->resource->transfer_log, tl_requests) {
+ unsigned long s = req->net_rq_state[peer_device->node_id];
+ unsigned long l = req->local_rq_state;
+
+ if (req->device != device)
+ continue;
+ if (!(l & RQ_WRITE))
+ continue;
+ if (!(s & RQ_NET_MASK) || s & RQ_NET_DONE)
+ continue;
+ if (req->master_bio) {
+ if (s & (RQ_EXP_RECEIVE_ACK | RQ_EXP_WRITE_ACK))
+ continue;
+ if (!(s & RQ_NET_OK) || device->cached_err_io)
+ continue;
+ } else {
+ if (!(l & RQ_COMPLETED_OK))
+ continue;
+ if ((s & (RQ_EXP_WRITE_ACK | RQ_NET_OK)) == (RQ_EXP_WRITE_ACK | RQ_NET_OK))
+ continue;
+ }
+ found = true;
+ break;
+ }
+ rcu_read_unlock();
+ return found;
+}
+
+static void peer_device_disconnected(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+
+ if (test_and_clear_bit(HOLDING_UUID_READ_LOCK, peer_device->flags))
+ up_read_non_owner(&device->uuid_sem);
+
+ /* A pending reconcile relabel is only valid for the connection it was
+ * armed for; drop it so it cannot fire on a later connection.
+ */
+ clear_bit(RECONCILE_INJECT_CUR_UUID, peer_device->flags);
+ clear_bit(SEND_RECONCILE_UUID, peer_device->flags);
+ /* If a handshake reconcile armed RECONCILIATION_RESYNC but no resync
+ * followed, drop it so it cannot mislabel a later resync.
+ */
+ clear_bit(RECONCILIATION_RESYNC, peer_device->flags);
+
+ peer_device_init_connect_state(peer_device);
+
+ /* No need to start additional resyncs after reconnection. */
+ peer_device->resync_again = 0;
+
+ if (!drbd_suspended(device)) {
+ struct drbd_resource *resource = device->resource;
+
+ /* Create the new UUID when finishing requests that did not
+ * reach the lost peer -- but not while quorum is lost, where
+ * those requests are about to be errored instead.
+ * Entering MINTING is the consumption lock: it keeps
+ * inc_ap_bio_cond() blocking new writes until the rotate ran,
+ * without posting the work a second time. Refused, a write
+ * already claimed the obligation and the sender runs the same
+ * evaluation.
+ */
+ if (!list_empty(&resource->transfer_log) &&
+ drbd_data_accessible(device, NOW) &&
+ !test_bit(PRIMARY_LOST_QUORUM, &device->flags) &&
+ drbd_gen_obligation_mint_start(device)) {
+ drbd_gen_obligation_mint_done(device,
+ drbd_check_peers_new_current_uuid(device));
+ wake_up(&device->misc_wait);
+ }
+ }
+
+ /* A successful completion cannot be taken back, and a write this peer
+ * never wrote is not resent to it, so the divergence is a fact as soon
+ * as the loss is noticed. The obligation materializes and its mint runs
+ * here, with no quorum, data or suspension gate and no settle round:
+ * the completion decision was taken by the protocol, at ack time.
+ * While a mint of an earlier obligation runs, it only takes the
+ * attribute; that executor's exit then keeps it armed and mandatory.
+ * Test the state before walking the transfer log: without an obligation
+ * to materialize there is nothing to find.
+ */
+ if ((GEN_OBL_IN(drbd_gen_obligation_state(device)) &
+ (GEN_OBL_MATERIALIZE_FROM | GEN_OBL_IN(GEN_OBL_MINTING))) &&
+ peer_may_lack_completed_write(peer_device)) {
+ drbd_gen_obligation_materialize(device);
+ if (drbd_gen_obligation_mint_start(device)) {
+ drbd_gen_obligation_mint_done(device,
+ drbd_uuid_new_current(device, false));
+ wake_up(&device->misc_wait);
+ }
+ }
+
+ drbd_md_sync(device);
+
+ if (get_ldev(device)) {
+ drbd_bitmap_io(device, &drbd_bm_write_copy_pages, "write from disconnected",
+ BM_LOCK_BULK | BM_LOCK_SINGLE_SLOT, peer_device);
+ put_ldev(device);
+ }
+}
+
+static bool initiator_can_commit_or_abort(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ bool remote = resource->twopc_reply.initiator_node_id != resource->res_opts.node_id;
+
+ if (remote) {
+ u64 parents = resource->twopc_parent_nodes & ~NODE_MASK(connection->peer_node_id);
+
+ if (!parents)
+ return false;
+ resource->twopc_parent_nodes = parents;
+ /* A lost prepared peer becomes a RETRY reply, and the initiator aborts. */
+ return true;
+ }
+
+ if (test_bit(TWOPC_PREPARED, &connection->flags) &&
+ !(test_bit(TWOPC_YES, &connection->flags) ||
+ test_bit(TWOPC_NO, &connection->flags) ||
+ test_bit(TWOPC_RETRY, &connection->flags)))
+ return false;
+
+ return true;
+}
+
+static void cleanup_remote_state_change(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct twopc_reply *reply = &resource->twopc_reply;
+ struct twopc_request request;
+ bool remote = false;
+
+ write_lock_irq(&resource->state_rwlock);
+ if (resource->remote_state_change && !initiator_can_commit_or_abort(connection)) {
+ remote = reply->initiator_node_id != resource->res_opts.node_id;
+
+ if (remote)
+ request = (struct twopc_request) {
+ .nodes_to_reach = ~0,
+ .cmd = P_TWOPC_ABORT,
+ .tid = reply->tid,
+ .initiator_node_id = reply->initiator_node_id,
+ .target_node_id = reply->target_node_id,
+ .vnr = reply->vnr,
+ };
+
+ drbd_info(connection, "Aborting %s state change %u commit not possible\n",
+ remote ? "remote" : "local", reply->tid);
+ if (remote) {
+ timer_delete(&resource->twopc_timer);
+ __clear_remote_state_change(resource);
+ } else {
+ enum alt_rv alt_rv = abort_local_transaction(connection, 0);
+
+ if (alt_rv != ALT_LOCKED)
+ return;
+ }
+ }
+ write_unlock_irq(&resource->state_rwlock);
+
+ /* for a local transaction, change_cluster_wide_state() sends the P_TWOPC_ABORTs */
+ if (remote)
+ nested_twopc_abort(resource, &request);
+}
+
+/* Before cleanup_unacked_peer_requests() marks the lost primary's writes */
+static void note_bitmaps_held_at_primary_loss(struct drbd_connection *lost_peer)
+{
+ struct drbd_resource *resource = lost_peer->resource;
+ struct drbd_connection *connection;
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection == lost_peer)
+ continue;
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (drbd_bm_total_weight(peer_device))
+ set_bit(BITMAP_HELD_AT_PRIMARY_LOSS, peer_device->flags);
+ else
+ clear_bit(BITMAP_HELD_AT_PRIMARY_LOSS, peer_device->flags);
+ }
+ }
+ rcu_read_unlock();
+}
+
+static void drbd_notify_peers_lost_primary(struct drbd_connection *lost_peer)
+{
+ struct drbd_resource *resource = lost_peer->resource;
+ struct drbd_connection *connection;
+ u64 im;
+
+ for_each_connection_ref(connection, im, resource) {
+ struct drbd_peer_device *peer_device;
+ bool send_dagtag = false;
+ int vnr;
+
+ if (connection == lost_peer)
+ continue;
+ if (connection->cstate[NOW] != C_CONNECTED)
+ continue;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+ u64 current_uuid = drbd_current_uuid(device);
+ u64 weak_nodes = drbd_weak_nodes_device(device);
+
+ if (device->disk_state[NOW] < D_INCONSISTENT ||
+ peer_device->disk_state[NOW] < D_INCONSISTENT)
+ continue; /* Ignore if one side is diskless */
+
+ /* owe a reconcile to this peer; set before NOTIFY clears,
+ * which receive_peer_dagtag waits for (gap-free handover)
+ */
+ set_bit(RECONCILE_PENDING, peer_device->flags);
+
+ drbd_send_current_uuid(peer_device, current_uuid, weak_nodes);
+ send_dagtag = true;
+ }
+
+ if (send_dagtag)
+ drbd_send_peer_dagtag(connection, lost_peer);
+ }
+}
+
+/* Informed confirmation of an optimistically rotated data generation.
+ *
+ * Confirmed -- safe to complete the held writes (confirm-before-complete) --
+ * the first instant ALL of:
+ *
+ * quorate device->quorum[NOW] (only where completions
+ * are actually held: on-no-quorum=suspend-io)
+ * AND every still-connected protocol-C D_UP_TO_DATE peer:
+ * acked it (CURRENT_UUID_UNCONFIRMED clear), OR
+ * has been lost (requirement discharged: we are "informed")
+ *
+ * ALL surviving sync-UpToDate peers, not a quorum-sized count: a count could
+ * release while a future-quorum member still lacks the generation. Protocol
+ * A/B peers are excluded (that connection accepted losing in-flight writes).
+ * With io-error / quorum-off nothing is held, but we still clear once informed,
+ * to lift the bump deferral and the reconnect-handshake special-casing.
+ *
+ * Returns true (discharging the obligation) exactly when the generation just
+ * became confirmed; the caller then releases the held completions with a
+ * NEW_UUID_CONFIRMED transfer-log walk. Re-evaluated on:
+ *
+ * barrier ack got_BarrierAck
+ * state change central hook in ___end_state_change, post-commit
+ * (gained quorum / sync peer UpToDate / lost a copy)
+ */
+bool drbd_maybe_release_rotated_gen(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+
+ if (drbd_gen_obligation_state(device) != GEN_OBL_UNCONFIRMED)
+ return false;
+ /* Pair with the smp_wmb() in drbd_uuid_new_current(): having observed the
+ * gate, see the per-peer CURRENT_UUID_UNCONFIRMED marks the bump set before it.
+ * Otherwise a racing ack could read a survivor's mark as still clear and
+ * wrongly confirm a generation that peer never received.
+ */
+ smp_rmb();
+ if (device->resource->res_opts.on_no_quorum == ONQ_SUSPEND_IO &&
+ !device->quorum[NOW])
+ return false;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_net_conf *nc;
+
+ if (peer_device->repl_state[NOW] < L_ESTABLISHED)
+ continue; /* lost -> requirement discharged */
+ if (peer_device->disk_state[NOW] != D_UP_TO_DATE)
+ continue; /* not a full copy of a generation */
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (!nc || nc->wire_protocol != DRBD_PROT_C)
+ continue; /* A/B: in-flight loss was accepted */
+ if (test_bit(CURRENT_UUID_UNCONFIRMED, peer_device->flags)) {
+ rcu_read_unlock();
+ return false; /* a survivor still lacks it durably */
+ }
+ }
+ rcu_read_unlock();
+
+ /* The from-state check picks the single winner among concurrent
+ * evaluations, the way the test_and_clear_bit of the gate did.
+ */
+ if (!drbd_gen_obligation_transition(device, GEN_OBL_IN(GEN_OBL_UNCONFIRMED),
+ GEN_OBL_DISCHARGED, 0, 0))
+ return false;
+ drbd_info(device, "rotated data generation confirmed durable in a quorate partition (gen %016llX)\n",
+ device->exposed_data_uuid);
+ /* Both callers reach here outside the state change's own wake-ups. */
+ wake_up(&device->misc_wait);
+ return true;
+}
+
+/*
+ * A sync target waits for our answer to its flush round on behalf of each
+ * Primary, which that Primary can no longer send us. Give it now.
+ * Caller holds state_rwlock.
+ */
+void drbd_flush_ack_for_lost_peer(struct drbd_connection *lost_connection)
+{
+ struct drbd_resource *resource = lost_connection->resource;
+ struct drbd_connection *initiator_connection;
+ unsigned long irq_flags;
+
+ if (!flush_ack_on_behalf_allowed(lost_connection))
+ return;
+
+ rcu_read_lock();
+ for_each_connection_rcu(initiator_connection, resource) {
+ u64 flush_sequence;
+
+ if (initiator_connection == lost_connection)
+ continue;
+
+ spin_lock_irqsave(&initiator_connection->primary_flush_lock, irq_flags);
+ flush_sequence = initiator_connection->flush_sequence;
+ spin_unlock_irqrestore(&initiator_connection->primary_flush_lock, irq_flags);
+
+ if (flush_sequence) /* Active flushes use non-zero sequence numbers */
+ queue_flush_ack(initiator_connection, lost_connection->peer_node_id,
+ flush_sequence);
+ }
+ rcu_read_unlock();
+}
+
+static void conn_disconnect(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ enum drbd_conn_state oc;
+ unsigned long irq_flags;
+ bool reconsider = false;
+ bool kept_for_resend;
+ bool dry_run;
+ int vnr, i;
+
+ clear_bit(CONN_DRY_RUN, &connection->flags);
+ dry_run = test_and_clear_bit(CONN_DRY_RUN_LOCAL, &connection->flags);
+ clear_bit(CONN_CONGESTED, &connection->flags);
+ clear_bit(CONN_PRE_110, &connection->flags);
+
+ if (connection->cstate[NOW] == C_STANDALONE)
+ return;
+
+ /* We are about to start the cleanup after connection loss.
+ * Make sure drbd_submit_bio knows about that.
+ * Usually we should be in some network failure state already,
+ * but just in case we are not, we fix it up here.
+ */
+ change_cstate_tag(connection, C_NETWORK_FAILURE, CS_HARD, "disconnected", NULL);
+
+ del_connect_timer(connection);
+
+ /* ack_receiver does not clean up anything. it must not interfere, either */
+ if (connection->ack_sender) {
+ destroy_workqueue(connection->ack_sender);
+ connection->ack_sender = NULL;
+ }
+
+ /* restart sender thread,
+ * potentially get it out of blocking network operations */
+ drbd_thread_stop(&connection->sender);
+ drbd_thread_start(&connection->sender);
+
+ mutex_lock(&resource->conf_update);
+ drbd_transport_shutdown(connection, CLOSE_CONNECTION);
+ connection->peer_verify_alg[0] = 0;
+ mutex_unlock(&resource->conf_update);
+
+ cleanup_remote_state_change(connection);
+
+ drain_resync_activity(connection);
+
+ connection->after_reconciliation.lost_node_id = -1;
+ connection->reconcile_handshake.lost_node_id = -1;
+ connection->reconcile_handshake.sent_lost_node = false;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ cancel_conflicting_peer_writes(peer_device);
+ rcu_read_unlock();
+
+ /* Wait for current activity to cease. This includes waiting for
+ * peer_request queued to the submitter workqueue. */
+ wait_event(connection->ee_wait,
+ atomic_read(&connection->active_ee_cnt) == 0);
+
+ /* wait for all w_e_end_data_req, w_e_end_rsdata_req, w_send_barrier,
+ * etc. which may still be on the worker queue to be "canceled" */
+ drbd_flush_workqueue(&connection->sender_work);
+
+ drbd_finish_peer_reqs(connection);
+
+ /* This second workqueue flush is necessary, since drbd_finish_peer_reqs()
+ might have issued a work again. The one before drbd_finish_peer_reqs() is
+ necessary to reclaim net_ee in drbd_finish_peer_reqs(). */
+ drbd_flush_workqueue(&connection->sender_work);
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ kref_get(&device->kref);
+ rcu_read_unlock();
+
+ peer_device_disconnected(peer_device);
+ if (get_ldev(device)) {
+ drbd_reconsider_queue_parameters(device, device->ldev);
+ put_ldev(device);
+ } else {
+ drbd_reconsider_queue_parameters(device, NULL);
+ }
+
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+
+ /* Apply these changes after peer_device_disconnected() because that
+ * may cause the loss of other connections to be detected, which can
+ * change the suspended state. */
+ kept_for_resend = resource->cached_susp;
+ tl_walk(connection, &connection->req_not_net_done,
+ kept_for_resend ? CONNECTION_LOST_WHILE_SUSPENDED : CONNECTION_LOST);
+
+ /* Every request towards this peer is done now and has given its sectors
+ * back, except those the walk kept for a resend: those stay counted.
+ */
+ i = atomic_read(&connection->ap_in_flight);
+ if (kept_for_resend ? i < 0 : i != 0)
+ drbd_warn(connection, "ap_in_flight = %d, expected %s\n",
+ i, kept_for_resend ? "the sectors kept for resend" : "0");
+
+ /* Losing this peer may complete the informed-confirmation picture for a
+ * still-unconfirmed rotated generation (its durable-ack requirement is
+ * discharged), or drop us below quorum. That is handled centrally by the
+ * release hook in ___end_state_change, post-commit of the disconnect state
+ * change -- nothing to do here.
+ */
+
+ i = drbd_free_peer_reqs(connection, &connection->done_ee);
+ if (i)
+ drbd_info(connection, "done_ee not empty, killed %u entries\n", i);
+ i = drbd_free_peer_reqs(connection, &connection->dagtag_wait_ee);
+ if (i)
+ drbd_info(connection, "dagtag_wait_ee not empty, killed %u entries\n", i);
+
+ if (test_bit(NOTIFY_PEERS_LOST_PRIMARY, &connection->flags))
+ note_bitmaps_held_at_primary_loss(connection);
+ cleanup_unacked_peer_requests(connection);
+ cleanup_peer_ack_list(connection);
+
+ i = atomic_read(&connection->pp_in_use);
+ if (i)
+ drbd_info(connection, "pp_in_use = %d, expected 0\n", i);
+ i = atomic_read(&connection->pp_in_use_by_net);
+ if (i)
+ drbd_info(connection, "pp_in_use_by_net = %d, expected 0\n", i);
+
+ if (!list_empty(&connection->current_epoch->list))
+ drbd_err(connection, "ASSERTION FAILED: connection->current_epoch->list not empty\n");
+ /* ok, no more ee's on the fly, it is safe to reset the epoch_size */
+ atomic_set(&connection->current_epoch->epoch_size, 0);
+ atomic_set(&connection->current_epoch->confirmed, 0);
+ connection->send.seen_any_write_yet = false;
+ connection->send.current_dagtag_sector =
+ resource->dagtag_sector - ((BIO_MAX_VECS << PAGE_SHIFT) >> SECTOR_SHIFT) - 1;
+ connection->current_epoch->oldest_unconfirmed_peer_req = NULL;
+
+ /* Indicate that last_dagtag_sector may no longer be up-to-date. We
+ * need to keep last_dagtag_sector because we may still need it to
+ * resolve a reconciliation resync. However, we need to avoid issuing a
+ * resync request dependent on that dagtag because the resync source
+ * may not be aware of the dagtag, even though it has newer data. This
+ * can occur if the peer has been re-started since the request with the
+ * dagtag.
+ * */
+ clear_bit(RECEIVED_DAGTAG, &connection->flags);
+
+ /* Release any threads waiting for a barrier to be acked. */
+ WRITE_ONCE(connection->last_acked_epoch_nr,
+ READ_ONCE(connection->send.last_sent_epoch_nr));
+ wake_up(&resource->barrier_wait);
+
+ drbd_info(connection, "Connection closed\n");
+
+ if (resource->role[NOW] == R_PRIMARY &&
+ connection->fencing_policy != FP_DONT_CARE &&
+ conn_highest_pdsk(connection) >= D_UNKNOWN)
+ conn_try_outdate_peer_async(connection);
+
+ drbd_maybe_khelper(NULL, connection, "disconnected");
+
+ begin_state_change(resource, &irq_flags, CS_VERBOSE | CS_LOCAL_ONLY);
+ oc = connection->cstate[NOW];
+ if (oc >= C_UNCONNECTED && dry_run) {
+ /* The node that asked for the dry run stays down; its peer reconnects */
+ __change_cstate(connection, C_DISCONNECTING);
+ oc = C_DISCONNECTING;
+ } else if (oc >= C_UNCONNECTED) {
+ __change_cstate(connection, C_UNCONNECTED);
+ /* drbd_receiver() has to be restarted after it returns */
+ drbd_thread_restart_nowait(&connection->receiver);
+ }
+ end_state_change(resource, &irq_flags, "disconnected");
+
+ if (test_bit(NOTIFY_PEERS_LOST_PRIMARY, &connection->flags)) {
+ drbd_notify_peers_lost_primary(connection);
+ clear_bit(NOTIFY_PEERS_LOST_PRIMARY, &connection->flags);
+ reconsider = true;
+ }
+
+ /* a surviving peer we owe a reconcile to is itself leaving:
+ * discharge it, else we stay pinned at Consistent forever
+ */
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ if (test_and_clear_bit(RECONCILE_PENDING, peer_device->flags))
+ reconsider = true;
+
+ /* re-arm once nothing is owed (2-node case arms no RECONCILE_PENDING) */
+ if (reconsider)
+ drbd_reconcile_settled_try_up_to_date(resource);
+
+ if (oc == C_DISCONNECTING)
+ change_cstate_tag(connection, C_STANDALONE, CS_VERBOSE | CS_HARD | CS_LOCAL_ONLY,
+ "disconnected", NULL);
+
+ read_lock_irq(&resource->state_rwlock);
+ drbd_flush_ack_for_lost_peer(connection);
+ read_unlock_irq(&resource->state_rwlock);
+}
+
+/*
+ * We support PRO_VERSION_MIN to PRO_VERSION_MAX.
+ * But see also drbd_protocol_version_acceptable() and module parameter
+ * drbd_protocol_version_min.
+ * The protocol version we can agree on is stored in agreed_pro_version.
+ *
+ * feature flags and the reserved array should be enough room for future
+ * enhancements of the handshake protocol, and possible plugins...
+ * See also PRO_FEATURES.
+ *
+ */
+static int drbd_send_features(struct drbd_connection *connection)
+{
+ struct p_connection_features *p;
+
+ p = __conn_prepare_command(connection, sizeof(*p), DATA_STREAM);
+ if (!p)
+ return -EIO;
+ memset(p, 0, sizeof(*p));
+ p->protocol_min = cpu_to_be32(drbd_protocol_version_min);
+ p->protocol_max = cpu_to_be32(PRO_VERSION_MAX);
+ p->sender_node_id = cpu_to_be32(connection->resource->res_opts.node_id);
+ p->receiver_node_id = cpu_to_be32(connection->peer_node_id);
+ p->feature_flags = cpu_to_be32(PRO_FEATURES);
+ return __send_command(connection, -1, P_CONNECTION_FEATURES, DATA_STREAM);
+}
+
+/*
+ * return values:
+ * 1 yes, we have a valid connection
+ * 0 oops, did not work out, please try again
+ * -1 peer talks different language,
+ * no point in trying again, please go standalone.
+ */
+int drbd_do_features(struct drbd_connection *connection)
+{
+ /* ASSERT current == connection->receiver ... */
+ struct drbd_resource *resource = connection->resource;
+ struct p_connection_features *p;
+ const int expect = sizeof(struct p_connection_features);
+ struct packet_info pi;
+ int err;
+
+ err = drbd_send_features(connection);
+ if (err)
+ return 0;
+
+ err = drbd_recv_header(connection, &pi);
+ if (err) {
+ if (err == -EAGAIN)
+ drbd_err(connection, "timeout while waiting for feature packet\n");
+ return 0;
+ }
+
+ if (pi.cmd != P_CONNECTION_FEATURES) {
+ drbd_err(connection, "expected ConnectionFeatures packet, received: %s (0x%04x)\n",
+ drbd_packet_name(pi.cmd), pi.cmd);
+ return -1;
+ }
+
+ if (pi.size != expect) {
+ drbd_err(connection, "expected ConnectionFeatures length: %u, received: %u\n",
+ expect, pi.size);
+ return -1;
+ }
+
+ err = drbd_recv_all_warn(connection, (void **)&p, expect);
+ if (err)
+ return 0;
+
+ p->protocol_min = be32_to_cpu(p->protocol_min);
+ p->protocol_max = be32_to_cpu(p->protocol_max);
+ if (p->protocol_max == 0)
+ p->protocol_max = p->protocol_min;
+
+ if (PRO_VERSION_MAX < p->protocol_min ||
+ drbd_protocol_version_min > p->protocol_max) {
+ drbd_err(connection, "incompatible DRBD dialects: "
+ "I support %d-%d, peer supports %d-%d\n",
+ drbd_protocol_version_min, PRO_VERSION_MAX,
+ p->protocol_min, p->protocol_max);
+ return -1;
+ }
+ /* Older DRBD will always expect us to agree to their max,
+ * if it falls within our [min, max] range.
+ * But we have a gap in there that we do not support.
+ */
+ if (p->protocol_max > PRO_VERSION_8_MAX &&
+ p->protocol_max < PRO_VERSION_MIN) {
+ drbd_err(connection, "incompatible DRBD 9 dialects: I support %u-%u, peer supports %u-%u\n",
+ PRO_VERSION_MIN, PRO_VERSION_MAX,
+ p->protocol_min, p->protocol_max);
+ return -1;
+ }
+
+ connection->agreed_pro_version = min_t(int, PRO_VERSION_MAX, p->protocol_max);
+ connection->agreed_features = PRO_FEATURES & be32_to_cpu(p->feature_flags);
+
+ if (connection->agreed_pro_version == 121 &&
+ (connection->agreed_features & DRBD_FF_RESYNC_DAGTAG)) {
+ /*
+ * Releases drbd-9.2.0, drbd-9.2.1 and drbd-9.2.2 used an
+ * implementation of discard merging which caused one
+ * P_RS_WRITE_ACK to be sent for the whole merged interval.
+ * These are precisely the releases with PRO_VERSION_MAX == 121
+ * and feature DRBD_FF_RESYNC_DAGTAG.
+ *
+ * We do no support this case, so reject the connection.
+ */
+ drbd_err(connection, "incompatible DRBD 9 dialects: protocol 121 with feature RESYNC_DAGTAG; upgrade via DRBD 9.2.16\n");
+ return -1;
+ }
+
+ if (connection->agreed_pro_version < 110) {
+ struct drbd_connection *connection2;
+ bool multiple = false;
+
+ /* new-peer adds a connection under the write lock, and none beside CONN_PRE_110 */
+ read_lock_irq(&resource->state_rwlock);
+ for_each_connection(connection2, resource) {
+ if (connection == connection2)
+ continue;
+ multiple = true;
+ }
+ if (!multiple)
+ set_bit(CONN_PRE_110, &connection->flags);
+ read_unlock_irq(&resource->state_rwlock);
+
+ if (multiple) {
+ drbd_err(connection, "Peer supports protocols %d-%d, but "
+ "multiple connections are only supported in protocol "
+ "110 and above\n", p->protocol_min, p->protocol_max);
+ return -1;
+ }
+ }
+
+ if (connection->agreed_pro_version >= 110) {
+ if (be32_to_cpu(p->sender_node_id) != connection->peer_node_id) {
+ drbd_err(connection, "Peer presented a node_id of %d instead of %d\n",
+ be32_to_cpu(p->sender_node_id), connection->peer_node_id);
+ return 0;
+ }
+ if (be32_to_cpu(p->receiver_node_id) != resource->res_opts.node_id) {
+ drbd_err(connection, "Peer expects me to have a node_id of %d instead of %d\n",
+ be32_to_cpu(p->receiver_node_id), resource->res_opts.node_id);
+ return 0;
+ }
+ }
+
+ drbd_info(connection, "Handshake to peer %d successful: "
+ "Agreed network protocol version %d\n",
+ connection->peer_node_id,
+ connection->agreed_pro_version);
+
+ drbd_info(connection, "Feature flags enabled on protocol level: 0x%x%s%s%s%s%s%s\n",
+ connection->agreed_features,
+ connection->agreed_features & DRBD_FF_TRIM ? " TRIM" : "",
+ connection->agreed_features & DRBD_FF_THIN_RESYNC ? " THIN_RESYNC" : "",
+ connection->agreed_features & DRBD_FF_WSAME ? " WRITE_SAME" : "",
+ connection->agreed_features & DRBD_FF_WZEROES ? " WRITE_ZEROES" : "",
+ connection->agreed_features & DRBD_FF_RESYNC_DAGTAG ? " RESYNC_DAGTAG" : "",
+ connection->agreed_features & DRBD_FF_WRITE_POSTPONE ? " WRITE_POSTPONE" :
+ connection->agreed_features ? "" : " none");
+
+ return 1;
+}
+
+#if !defined(CONFIG_CRYPTO_HMAC) && !defined(CONFIG_CRYPTO_HMAC_MODULE)
+int drbd_do_auth(struct drbd_connection *connection)
+{
+ drbd_err(connection, "This kernel was build without CONFIG_CRYPTO_HMAC.\n");
+ drbd_err(connection, "You need to disable 'cram-hmac-alg' in drbd.conf.\n");
+ return -1;
+}
+#else
+#define CHALLENGE_LEN 64 /* must be multiple of 4 */
+
+/* Return value:
+ 1 - auth succeeded,
+ 0 - failed, try again (network error),
+ -1 - auth failed, don't try again.
+*/
+
+struct auth_challenge {
+ char d[CHALLENGE_LEN];
+ u32 i;
+} __packed;
+
+int drbd_do_auth(struct drbd_connection *connection)
+{
+ struct auth_challenge my_challenge, *peers_ch = NULL;
+ void *response;
+ char *right_response = NULL;
+ unsigned int key_len;
+ char secret[SHARED_SECRET_MAX]; /* 64 byte */
+ unsigned int resp_size;
+ struct shash_desc *desc;
+ struct packet_info pi;
+ struct drbd_net_conf *nc;
+ int err, rv, dig_size;
+ bool peer_is_drbd_9 = connection->agreed_pro_version >= 110;
+ void *packet_body;
+
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ key_len = strlen(nc->shared_secret);
+ memcpy(secret, nc->shared_secret, key_len);
+ rcu_read_unlock();
+
+ desc = kmalloc(sizeof(struct shash_desc) +
+ crypto_shash_descsize(connection->cram_hmac_tfm),
+ GFP_KERNEL);
+ if (!desc) {
+ rv = -1;
+ goto fail;
+ }
+ desc->tfm = connection->cram_hmac_tfm;
+
+ rv = crypto_shash_setkey(connection->cram_hmac_tfm, (u8 *)secret, key_len);
+ if (rv) {
+ drbd_err(connection, "crypto_shash_setkey() failed with %d\n", rv);
+ rv = -1;
+ goto fail;
+ }
+
+ get_random_bytes(my_challenge.d, sizeof(my_challenge.d));
+
+ packet_body = __conn_prepare_command(connection, sizeof(my_challenge.d), DATA_STREAM);
+ if (!packet_body) {
+ rv = 0;
+ goto fail;
+ }
+ memcpy(packet_body, my_challenge.d, sizeof(my_challenge.d));
+
+ rv = !__send_command(connection, -1, P_AUTH_CHALLENGE, DATA_STREAM);
+ if (!rv)
+ goto fail;
+
+ err = drbd_recv_header(connection, &pi);
+ if (err) {
+ rv = 0;
+ goto fail;
+ }
+
+ if (pi.cmd != P_AUTH_CHALLENGE) {
+ drbd_err(connection, "expected AuthChallenge packet, received: %s (0x%04x)\n",
+ drbd_packet_name(pi.cmd), pi.cmd);
+ rv = -1;
+ goto fail;
+ }
+
+ if (pi.size != sizeof(peers_ch->d)) {
+ drbd_err(connection, "unexpected AuthChallenge payload.\n");
+ rv = -1;
+ goto fail;
+ }
+
+ peers_ch = kmalloc_obj(*peers_ch, GFP_NOIO);
+ if (!peers_ch) {
+ rv = -1;
+ goto fail;
+ }
+
+ err = drbd_recv_into(connection, peers_ch->d, sizeof(peers_ch->d));
+ if (err) {
+ rv = 0;
+ goto fail;
+ }
+
+ if (!memcmp(my_challenge.d, peers_ch->d, sizeof(my_challenge.d))) {
+ drbd_err(connection, "Peer presented the same challenge!\n");
+ rv = -1;
+ goto fail;
+ }
+
+ resp_size = crypto_shash_digestsize(connection->cram_hmac_tfm);
+ response = __conn_prepare_command(connection, resp_size, DATA_STREAM);
+ if (!response) {
+ rv = 0;
+ goto fail;
+ }
+
+ dig_size = pi.size;
+ if (peer_is_drbd_9) {
+ peers_ch->i = cpu_to_be32(connection->resource->res_opts.node_id);
+ dig_size += sizeof(peers_ch->i);
+ }
+
+ rv = crypto_shash_digest(desc, peers_ch->d, dig_size, response);
+ if (rv) {
+ drbd_err(connection, "crypto_shash_digest() failed with %d\n", rv);
+ rv = -1;
+ goto fail;
+ }
+
+ rv = !__send_command(connection, -1, P_AUTH_RESPONSE, DATA_STREAM);
+ if (!rv)
+ goto fail;
+
+ err = drbd_recv_header(connection, &pi);
+ if (err) {
+ rv = 0;
+ goto fail;
+ }
+
+ if (pi.cmd != P_AUTH_RESPONSE) {
+ drbd_err(connection, "expected AuthResponse packet, received: %s (0x%04x)\n",
+ drbd_packet_name(pi.cmd), pi.cmd);
+ rv = 0;
+ goto fail;
+ }
+
+ if (pi.size != resp_size) {
+ drbd_err(connection, "expected AuthResponse payload of %u bytes, received %u\n",
+ resp_size, pi.size);
+ rv = 0;
+ goto fail;
+ }
+
+ err = drbd_recv_all(connection, &response, resp_size);
+ if (err) {
+ rv = 0;
+ goto fail;
+ }
+
+ right_response = kmalloc(resp_size, GFP_NOIO);
+ if (!right_response) {
+ rv = -1;
+ goto fail;
+ }
+
+ dig_size = sizeof(my_challenge.d);
+ if (peer_is_drbd_9) {
+ my_challenge.i = cpu_to_be32(connection->peer_node_id);
+ dig_size += sizeof(my_challenge.i);
+ }
+
+ rv = crypto_shash_digest(desc, my_challenge.d, dig_size, right_response);
+ if (rv) {
+ drbd_err(connection, "crypto_shash_digest() failed with %d\n", rv);
+ rv = -1;
+ goto fail;
}
- INFO_bm_xfer_stats(peer_device, "receive", &c);
+ rv = !memcmp(response, right_response, resp_size);
- if (device->state.conn == C_WF_BITMAP_T) {
- enum drbd_state_rv rv;
+ if (rv)
+ drbd_info(connection, "Peer authenticated using %d bytes HMAC\n",
+ resp_size);
+ else
+ rv = -1;
- err = drbd_send_bitmap(device, peer_device);
- if (err)
- goto out;
- /* Omit CS_ORDERED with this state transition to avoid deadlocks. */
- rv = _drbd_request_state(device, NS(conn, C_WF_SYNC_UUID), CS_VERBOSE);
- D_ASSERT(device, rv == SS_SUCCESS);
- } else if (device->state.conn != C_WF_BITMAP_S) {
- /* admin may have requested C_DISCONNECTING,
- * other threads may have noticed network errors */
- drbd_info(device, "unexpected cstate (%s) in receive_bitmap\n",
- drbd_conn_str(device->state.conn));
+ fail:
+ kfree(peers_ch);
+ kfree(right_response);
+ if (desc) {
+ shash_desc_zero(desc);
+ kfree(desc);
}
- err = 0;
- out:
- drbd_bm_unlock(device);
- if (!err && device->state.conn == C_WF_BITMAP_S)
- drbd_start_resync(device, C_SYNC_SOURCE);
- return err;
+ return rv;
}
+#endif
-static int receive_skip(struct drbd_connection *connection, struct packet_info *pi)
+int drbd_receiver(struct drbd_thread *thi)
{
- drbd_warn(connection, "skipping unknown optional packet type %d, l: %d!\n",
- pi->cmd, pi->size);
+ struct drbd_connection *connection = thi->connection;
+
+ if (conn_connect(connection)) {
+ blk_start_plug(&connection->receiver_plug);
+ drbdd(connection);
+ blk_finish_plug(&connection->receiver_plug);
+ }
- return ignore_remaining_packet(connection, pi);
+ conn_disconnect(connection);
+ return 0;
}
-static int receive_UnplugRemote(struct drbd_connection *connection, struct packet_info *pi)
+/* ********* acknowledge sender ******** */
+
+static void drbd_check_flush_dagtag_reached(struct drbd_connection *peer_ack_connection)
{
- /* Make sure we've acked all the TCP data associated
- * with the data requests being unplugged */
- tcp_sock_set_quickack(connection->data.socket->sk, 2);
- return 0;
+ struct drbd_resource *resource = peer_ack_connection->resource;
+ struct drbd_connection *flush_requests_connection;
+ u64 peer_ack_node_mask = NODE_MASK(peer_ack_connection->peer_node_id);
+ u64 last_peer_ack_dagtag_seen = peer_ack_connection->last_peer_ack_dagtag_seen;
+ u64 im;
+
+ for_each_connection_ref(flush_requests_connection, im, resource) {
+ u64 flush_sequence;
+ u64 *sent_mask;
+ u64 flush_requests_dagtag;
+
+ spin_lock_irq(&flush_requests_connection->primary_flush_lock);
+ flush_requests_dagtag = flush_requests_connection->flush_requests_dagtag;
+ flush_sequence = flush_requests_connection->flush_sequence;
+ sent_mask = &flush_requests_connection->flush_forward_sent_mask;
+
+ if (!flush_sequence || /* Active flushes use non-zero sequence numbers */
+ *sent_mask & peer_ack_node_mask ||
+ last_peer_ack_dagtag_seen < flush_requests_dagtag) {
+ spin_unlock_irq(&flush_requests_connection->primary_flush_lock);
+ continue;
+ }
+
+ *sent_mask |= peer_ack_node_mask;
+ spin_unlock_irq(&flush_requests_connection->primary_flush_lock);
+
+ if (peer_ack_connection == flush_requests_connection)
+ drbd_send_flush_requests_ack(peer_ack_connection,
+ flush_sequence,
+ resource->res_opts.node_id);
+ else
+ drbd_send_flush_forward(peer_ack_connection,
+ flush_sequence,
+ flush_requests_connection->peer_node_id);
+ }
}
-static int receive_out_of_sync(struct drbd_connection *connection, struct packet_info *pi)
+static int process_peer_ack_list(struct drbd_connection *connection)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_block_desc *p = pi->data;
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_ack *peer_ack, *tmp;
+ u64 node_id_mask;
+ int err = 0;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ node_id_mask = NODE_MASK(connection->peer_node_id);
+
+ spin_lock_irq(&resource->peer_ack_lock);
+ peer_ack = list_first_entry(&resource->peer_ack_list, struct drbd_peer_ack, list);
+ while (&peer_ack->list != &resource->peer_ack_list) {
+ u64 pending_mask = peer_ack->pending_mask;
+ u64 mask = peer_ack->mask;
+ u64 dagtag_sector = peer_ack->dagtag_sector;
+
+ tmp = list_next_entry(peer_ack, list);
+
+ if (!(peer_ack->queued_mask & node_id_mask)) {
+ peer_ack = tmp;
+ continue;
+ }
+
+ /*
+ * After disconnecting, queue_peer_ack_send() sets
+ * last_peer_ack_dagtag_seen directly. Do not jump back if we
+ * process a peer ack with a lower dagtag here shortly after.
+ */
+ connection->last_peer_ack_dagtag_seen =
+ max(connection->last_peer_ack_dagtag_seen, dagtag_sector);
+
+ peer_ack->queued_mask &= ~node_id_mask;
+ drbd_destroy_peer_ack_if_done(peer_ack);
+ peer_ack = tmp;
+
+ if (!(pending_mask & node_id_mask))
+ continue;
+ spin_unlock_irq(&resource->peer_ack_lock);
- switch (device->state.conn) {
- case C_WF_SYNC_UUID:
- case C_WF_BITMAP_T:
- case C_BEHIND:
+ err = drbd_send_peer_ack(connection, mask, dagtag_sector);
+
+ spin_lock_irq(&resource->peer_ack_lock);
+ if (err)
break;
- default:
- drbd_err(device, "ASSERT FAILED cstate = %s, expected: WFSyncUUID|WFBitMapT|Behind\n",
- drbd_conn_str(device->state.conn));
}
+ spin_unlock_irq(&resource->peer_ack_lock);
- drbd_set_out_of_sync(peer_device, be64_to_cpu(p->sector), be32_to_cpu(p->blksize));
+ if (!err && connection->agreed_pro_version >= 123)
+ drbd_check_flush_dagtag_reached(connection);
- return 0;
+ return err;
}
-static int receive_rs_deallocated(struct drbd_connection *connection, struct packet_info *pi)
+static int got_peers_in_sync(struct drbd_connection *connection, struct packet_info *pi)
{
struct drbd_peer_device *peer_device;
- struct p_block_desc *p = pi->data;
struct drbd_device *device;
+ struct p_peer_block_desc *p = pi->data;
sector_t sector;
- int size, err = 0;
+ u64 in_sync_b;
+ int size;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
return -EIO;
- device = peer_device->device;
-
- sector = be64_to_cpu(p->sector);
- size = be32_to_cpu(p->blksize);
- dec_rs_pending(peer_device);
+ device = peer_device->device;
if (get_ldev(device)) {
- struct drbd_peer_request *peer_req;
+ unsigned long modified;
- peer_req = drbd_alloc_peer_req(peer_device, ID_SYNCER, sector,
- size, 0, GFP_NOIO);
- if (!peer_req) {
- put_ldev(device);
- return -ENOMEM;
- }
-
- peer_req->i.type = INTERVAL_RESYNC_WRITE;
- peer_req->w.cb = e_end_resync_block;
- peer_req->opf = REQ_OP_DISCARD;
- peer_req->submit_jif = jiffies;
- peer_req->flags |= EE_TRIM;
+ sector = be64_to_cpu(p->sector);
+ size = be32_to_cpu(p->size);
+ in_sync_b = node_ids_to_bitmap(device, be64_to_cpu(p->mask));
- spin_lock_irq(&device->resource->req_lock);
- list_add_tail(&peer_req->w.list, &device->sync_ee);
- spin_unlock_irq(&device->resource->req_lock);
+ modified = drbd_set_sync(device, sector, size, 0, in_sync_b);
- atomic_add(pi->size >> 9, &device->rs_sect_ev);
- err = drbd_submit_peer_request(peer_req);
+ /* If we are SyncSource then we rely on P_PEERS_IN_SYNC from
+ * the peer to inform us of sync progress. Otherwise only send
+ * peers-in-sync when we have actually cleared some bits.
+ * This prevents an infinite loop with the peer. */
+ if (modified || peer_device->repl_state[NOW] == L_SYNC_SOURCE)
+ drbd_queue_update_peers(peer_device, sector, sector + (size >> SECTOR_SHIFT));
- if (err) {
- spin_lock_irq(&device->resource->req_lock);
- list_del(&peer_req->w.list);
- spin_unlock_irq(&device->resource->req_lock);
+ put_ldev(device);
+ }
- drbd_free_peer_req(device, peer_req);
- put_ldev(device);
- err = 0;
- goto fail;
- }
+ return 0;
+}
- inc_unacked(device);
+static int got_RqSReply(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct p_req_state_reply *p = pi->data;
+ int retcode = be32_to_cpu(p->retcode);
- /* No put_ldev() here. Gets called in drbd_endio_write_sec_final(),
- as well as drbd_rs_complete_io() */
- } else {
- fail:
- drbd_rs_complete_io(device, sector);
- drbd_send_ack_ex(peer_device, P_NEG_ACK, sector, size, ID_SYNCER);
+ if (retcode >= SS_SUCCESS)
+ set_bit(TWOPC_YES, &connection->flags);
+ else {
+ set_bit(TWOPC_NO, &connection->flags);
+ dynamic_drbd_dbg(connection, "Requested state change failed by peer: %s (%d)\n",
+ drbd_set_st_err_str(retcode), retcode);
}
- atomic_add(size >> 9, &device->rs_sect_in);
+ wake_up_all(&connection->resource->state_wait);
- return err;
+ return 0;
}
-struct data_cmd {
- int expect_payload;
- unsigned int pkt_size;
- int (*fn)(struct drbd_connection *, struct packet_info *);
-};
-
-static struct data_cmd drbd_cmd_handler[] = {
- [P_DATA] = { 1, sizeof(struct p_data), receive_Data },
- [P_DATA_REPLY] = { 1, sizeof(struct p_data), receive_DataReply },
- [P_RS_DATA_REPLY] = { 1, sizeof(struct p_data), receive_RSDataReply } ,
- [P_BARRIER] = { 0, sizeof(struct p_barrier), receive_Barrier } ,
- [P_BITMAP] = { 1, 0, receive_bitmap } ,
- [P_COMPRESSED_BITMAP] = { 1, 0, receive_bitmap } ,
- [P_UNPLUG_REMOTE] = { 0, 0, receive_UnplugRemote },
- [P_DATA_REQUEST] = { 0, sizeof(struct p_block_req), receive_DataRequest },
- [P_RS_DATA_REQUEST] = { 0, sizeof(struct p_block_req), receive_DataRequest },
- [P_SYNC_PARAM] = { 1, 0, receive_SyncParam },
- [P_SYNC_PARAM89] = { 1, 0, receive_SyncParam },
- [P_PROTOCOL] = { 1, sizeof(struct p_protocol), receive_protocol },
- [P_UUIDS] = { 0, sizeof(struct p_uuids), receive_uuids },
- [P_SIZES] = { 0, sizeof(struct p_sizes), receive_sizes },
- [P_STATE] = { 0, sizeof(struct p_state), receive_state },
- [P_STATE_CHG_REQ] = { 0, sizeof(struct p_req_state), receive_req_state },
- [P_SYNC_UUID] = { 0, sizeof(struct p_rs_uuid), receive_sync_uuid },
- [P_OV_REQUEST] = { 0, sizeof(struct p_block_req), receive_DataRequest },
- [P_OV_REPLY] = { 1, sizeof(struct p_block_req), receive_DataRequest },
- [P_CSUM_RS_REQUEST] = { 1, sizeof(struct p_block_req), receive_DataRequest },
- [P_RS_THIN_REQ] = { 0, sizeof(struct p_block_req), receive_DataRequest },
- [P_DELAY_PROBE] = { 0, sizeof(struct p_delay_probe93), receive_skip },
- [P_OUT_OF_SYNC] = { 0, sizeof(struct p_block_desc), receive_out_of_sync },
- [P_CONN_ST_CHG_REQ] = { 0, sizeof(struct p_req_state), receive_req_conn_state },
- [P_PROTOCOL_UPDATE] = { 1, sizeof(struct p_protocol), receive_protocol },
- [P_TRIM] = { 0, sizeof(struct p_trim), receive_Data },
- [P_ZEROES] = { 0, sizeof(struct p_trim), receive_Data },
- [P_RS_DEALLOCATED] = { 0, sizeof(struct p_block_desc), receive_rs_deallocated },
-};
-
-static void drbdd(struct drbd_connection *connection)
-{
- struct packet_info pi;
- size_t shs; /* sub header size */
- int err;
-
- while (get_t_state(&connection->receiver) == RUNNING) {
- struct data_cmd const *cmd;
+static int got_twopc_reply(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct p_twopc_reply *p = pi->data;
+
+ write_lock_irq(&resource->state_rwlock);
+ if (resource->twopc_reply.initiator_node_id == be32_to_cpu(p->initiator_node_id) &&
+ resource->twopc_reply.tid == be32_to_cpu(p->tid)) {
+ dynamic_drbd_dbg(connection, "Got a %s reply for state change %u\n",
+ drbd_packet_name(pi->cmd),
+ resource->twopc_reply.tid);
+
+ if (pi->cmd == P_TWOPC_YES) {
+ struct drbd_peer_device *peer_device;
+ u64 reachable_nodes;
+ u64 max_size;
+
+ reachable_nodes = be64_to_cpu(p->reachable_nodes);
+
+ switch (resource->twopc.type) {
+ case TWOPC_STATE_CHANGE:
+ if (resource->res_opts.node_id ==
+ resource->twopc_reply.initiator_node_id &&
+ connection->peer_node_id ==
+ resource->twopc_reply.target_node_id) {
+ resource->twopc_reply.target_reachable_nodes |=
+ reachable_nodes;
+ } else {
+ resource->twopc_reply.reachable_nodes |=
+ reachable_nodes;
+ }
+ resource->twopc_reply.primary_nodes |=
+ be64_to_cpu(p->primary_nodes);
+ resource->twopc_reply.weak_nodes |=
+ be64_to_cpu(p->weak_nodes);
+ break;
+ case TWOPC_RESIZE:
+ resource->twopc_reply.reachable_nodes |= reachable_nodes;
+ resource->twopc_reply.common_reachable_nodes &=
+ reachable_nodes;
+ resource->twopc_reply.diskful_primary_nodes |=
+ be64_to_cpu(p->diskful_primary_nodes);
+ max_size = be64_to_cpu(p->max_possible_size);
+ resource->twopc_reply.max_possible_size =
+ min_t(sector_t, resource->twopc_reply.max_possible_size,
+ max_size);
+ peer_device = conn_peer_device(connection, resource->twopc_reply.vnr);
+ if (peer_device)
+ peer_device->max_size = max_size;
+ break;
+ }
+ }
- drbd_thread_current_set_cpu(&connection->receiver);
- update_receiver_timing_details(connection, drbd_recv_header_maybe_unplug);
- if (drbd_recv_header_maybe_unplug(connection, &pi))
- goto err_out;
+ if (pi->cmd == P_TWOPC_YES)
+ set_bit(TWOPC_YES, &connection->flags);
+ else if (pi->cmd == P_TWOPC_NO)
+ set_bit(TWOPC_NO, &connection->flags);
+ else if (pi->cmd == P_TWOPC_RETRY)
+ set_bit(TWOPC_RETRY, &connection->flags);
+ drbd_maybe_cluster_wide_reply(resource);
+ } else {
+ dynamic_drbd_dbg(connection, "Ignoring %s reply for state change %u\n",
+ drbd_packet_name(pi->cmd),
+ be32_to_cpu(p->tid));
+ }
+ write_unlock_irq(&resource->state_rwlock);
- cmd = &drbd_cmd_handler[pi.cmd];
- if (unlikely(pi.cmd >= ARRAY_SIZE(drbd_cmd_handler) || !cmd->fn)) {
- drbd_err(connection, "Unexpected data packet %s (0x%04x)",
- drbd_packet_name(pi.cmd), pi.cmd);
- goto err_out;
- }
+ return 0;
+}
- shs = cmd->pkt_size;
- if (pi.cmd == P_SIZES && connection->agreed_features & DRBD_FF_WSAME)
- shs += sizeof(struct o_qlim);
- if (pi.size > shs && !cmd->expect_payload) {
- drbd_err(connection, "No payload expected %s l:%d\n",
- drbd_packet_name(pi.cmd), pi.size);
- goto err_out;
- }
- if (pi.size < shs) {
- drbd_err(connection, "%s: unexpected packet size, expected:%d received:%d\n",
- drbd_packet_name(pi.cmd), (int)shs, pi.size);
- goto err_out;
- }
+static int got_Ping(struct drbd_connection *connection, struct packet_info *pi)
+{
+ drbd_queue_ping_ack(connection);
+ return 0;
+}
- if (shs) {
- update_receiver_timing_details(connection, drbd_recv_all_warn);
- err = drbd_recv_all_warn(connection, pi.data, shs);
- if (err)
- goto err_out;
- pi.size -= shs;
- }
+static int got_PingAck(struct drbd_connection *connection, struct packet_info *pi)
+{
+ clear_bit(PING_TIMEOUT_ACTIVE, &connection->flags);
+ set_rcvtimeo(connection, REGULAR_TIMEOUT);
- update_receiver_timing_details(connection, cmd->fn);
- err = cmd->fn(connection, &pi);
- if (err) {
- drbd_err(connection, "error receiving %s, e: %d l: %d!\n",
- drbd_packet_name(pi.cmd), err, pi.size);
- goto err_out;
- }
+ if (test_bit(PING_PENDING, &connection->flags)) {
+ clear_bit(PING_PENDING, &connection->flags);
+ wake_up_all(&connection->resource->state_wait);
}
- return;
- err_out:
- conn_request_state(connection, NS(conn, C_PROTOCOL_ERROR), CS_HARD);
+ return 0;
}
-static void conn_disconnect(struct drbd_connection *connection)
+static int got_IsInSync(struct drbd_connection *connection, struct packet_info *pi)
{
struct drbd_peer_device *peer_device;
- enum drbd_conns oc;
- int vnr;
+ struct drbd_device *device;
+ struct drbd_peer_request *peer_req;
+ struct p_block_ack *p = pi->data;
+ sector_t sector = be64_to_cpu(p->sector);
+ int blksize = be32_to_cpu(p->blksize);
- if (connection->cstate == C_STANDALONE)
- return;
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
+ device = peer_device->device;
- /* We are about to start the cleanup after connection loss.
- * Make sure drbd_make_request knows about that.
- * Usually we should be in some network failure state already,
- * but just in case we are not, we fix it up here.
- */
- conn_request_state(connection, NS(conn, C_NETWORK_FAILURE), CS_HARD);
+ D_ASSERT(device, connection->agreed_pro_version >= 89);
- /* ack_receiver does not clean up anything. it must not interfere, either */
- drbd_thread_stop(&connection->ack_receiver);
- if (connection->ack_sender) {
- destroy_workqueue(connection->ack_sender);
- connection->ack_sender = NULL;
- }
- drbd_free_sock(connection);
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- kref_get(&device->kref);
- rcu_read_unlock();
- drbd_disconnected(peer_device);
- kref_put(&device->kref, drbd_destroy_device);
- rcu_read_lock();
- }
- rcu_read_unlock();
+ /* Do not rely on the block_id from older peers. */
+ if (connection->agreed_pro_version < 122)
+ p->block_id = ID_SYNCER;
- if (!list_empty(&connection->current_epoch->list))
- drbd_err(connection, "ASSERTION FAILED: connection->current_epoch->list not empty\n");
- /* ok, no more ee's on the fly, it is safe to reset the epoch_size */
- atomic_set(&connection->current_epoch->epoch_size, 0);
- connection->send.seen_any_write_yet = false;
+ peer_req = find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_RESYNC_WRITE),
+ sector, blksize, p->block_id);
+ if (!peer_req)
+ return -EIO;
- drbd_info(connection, "Connection closed\n");
+ dec_rs_pending(peer_device);
- if (conn_highest_role(connection) == R_PRIMARY && conn_highest_pdsk(connection) >= D_UNKNOWN)
- conn_try_outdate_peer_async(connection);
+ set_bit(INTERVAL_RECEIVED, &peer_req->i.flags);
- spin_lock_irq(&connection->resource->req_lock);
- oc = connection->cstate;
- if (oc >= C_UNCONNECTED)
- _conn_request_state(connection, NS(conn, C_UNCONNECTED), CS_VERBOSE);
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_del(&peer_req->w.list);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+
+ if (get_ldev(device)) {
+ drbd_set_in_sync(peer_device, sector, blksize);
+ /* rs_same_csums is supposed to count in units of BM_BLOCK_SIZE */
+ peer_device->rs_same_csum += (blksize >> device->ldev->md.bm_block_shift);
+ put_ldev(device);
+ }
+ rs_sectors_came_in(peer_device, blksize);
+
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_resync_request_complete(peer_req);
+ return 0;
+}
+
+static int
+validate_req_change_req_state(struct drbd_peer_device *peer_device, u64 id, sector_t sector,
+ enum drbd_interval_type type, const char *func,
+ enum drbd_req_event what, bool missing_ok)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_request *req;
- spin_unlock_irq(&connection->resource->req_lock);
+ spin_lock_irq(&device->interval_lock);
+ req = find_request(device, type, id, sector, missing_ok, func);
+ spin_unlock_irq(&device->interval_lock);
+ if (unlikely(!req))
+ return -EIO;
+ req_mod(req, what, peer_device);
- if (oc == C_DISCONNECTING)
- conn_request_state(connection, NS(conn, C_STANDALONE), CS_VERBOSE | CS_HARD);
+ return 0;
}
-static int drbd_disconnected(struct drbd_peer_device *peer_device)
+static int got_BlockAck(struct drbd_connection *connection, struct packet_info *pi)
{
- struct drbd_device *device = peer_device->device;
- unsigned int i;
-
- /* wait for current activity to cease. */
- spin_lock_irq(&device->resource->req_lock);
- _drbd_wait_ee_list_empty(device, &device->active_ee);
- _drbd_wait_ee_list_empty(device, &device->sync_ee);
- _drbd_wait_ee_list_empty(device, &device->read_ee);
- spin_unlock_irq(&device->resource->req_lock);
-
- /* We do not have data structures that would allow us to
- * get the rs_pending_cnt down to 0 again.
- * * On C_SYNC_TARGET we do not have any data structures describing
- * the pending RSDataRequest's we have sent.
- * * On C_SYNC_SOURCE there is no data structure that tracks
- * the P_RS_DATA_REPLY blocks that we sent to the SyncTarget.
- * And no, it is not the sum of the reference counts in the
- * resync_LRU. The resync_LRU tracks the whole operation including
- * the disk-IO, while the rs_pending_cnt only tracks the blocks
- * on the fly. */
- drbd_rs_cancel_all(device);
- device->rs_total = 0;
- device->rs_failed = 0;
- atomic_set(&device->rs_pending_cnt, 0);
- wake_up(&device->misc_wait);
+ struct drbd_peer_device *peer_device;
+ struct p_block_ack *p = pi->data;
+ sector_t sector = be64_to_cpu(p->sector);
+ enum drbd_req_event what;
- timer_delete_sync(&device->resync_timer);
- resync_timer_fn(&device->resync_timer);
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
- /* wait for all w_e_end_data_req, w_e_end_rsdata_req, w_send_barrier,
- * w_make_resync_request etc. which may still be on the worker queue
- * to be "canceled" */
- drbd_flush_workqueue(&peer_device->connection->sender_work);
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- drbd_finish_peer_reqs(device);
+ switch (pi->cmd) {
+ case P_RS_WRITE_ACK: /* agreed_pro_version < 122 */
+ case P_WRITE_ACK_IN_SYNC:
+ what = WRITE_ACKED_BY_PEER_AND_SIS;
+ break;
+ case P_WRITE_ACK:
+ what = WRITE_ACKED_BY_PEER;
+ break;
+ case P_RECV_ACK:
+ what = RECV_ACKED_BY_PEER;
+ break;
+ default:
+ BUG();
+ }
- /* This second workqueue flush is necessary, since drbd_finish_peer_reqs()
- might have issued a work again. The one before drbd_finish_peer_reqs() is
- necessary to reclain net_ee in drbd_finish_peer_reqs(). */
- drbd_flush_workqueue(&peer_device->connection->sender_work);
+ return validate_req_change_req_state(peer_device, p->block_id, sector,
+ INTERVAL_LOCAL_WRITE, __func__,
+ what, false);
+}
- /* need to do it again, drbd_finish_peer_reqs() may have populated it
- * again via drbd_try_clear_on_disk_bm(). */
- drbd_rs_cancel_all(device);
+/* Process acks for resync writes. */
+static int got_RSWriteAck(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct p_block_ack *p = pi->data;
+ bool is_neg_ack = pi->cmd == P_NEG_ACK || pi->cmd == P_RS_NEG_ACK;
+ sector_t sector = be64_to_cpu(p->sector);
+ int size = be32_to_cpu(p->blksize);
+ struct drbd_peer_request *peer_req;
- kfree(device->p_uuid);
- device->p_uuid = NULL;
+ /* P_RS_WRITE_ACK used to be used instead of P_WRITE_ACK_IN_SYNC. */
+ if (connection->agreed_pro_version < 122 && p->block_id != ID_SYNCER)
+ return got_BlockAck(connection, pi);
- if (!drbd_suspended(device))
- tl_clear(peer_device->connection);
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
- drbd_md_sync(device);
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- if (get_ldev(device)) {
- drbd_bitmap_io(device, &drbd_bm_write_copy_pages,
- "write from disconnected", BM_LOCKED_CHANGE_ALLOWED, NULL);
- put_ldev(device);
- }
+ if (is_neg_ack && peer_device->disk_state[NOW] == D_UP_TO_DATE)
+ set_bit(GOT_NEG_ACK, peer_device->flags);
- i = atomic_read(&device->pp_in_use_by_net);
- if (i)
- drbd_info(device, "pp_in_use_by_net = %d, expected 0\n", i);
- i = atomic_read(&device->pp_in_use);
- if (i)
- drbd_info(device, "pp_in_use = %d, expected 0\n", i);
+ peer_req = find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_RESYNC_READ),
+ sector, size, p->block_id);
+ if (!peer_req)
+ return -EIO;
- D_ASSERT(device, list_empty(&device->read_ee));
- D_ASSERT(device, list_empty(&device->active_ee));
- D_ASSERT(device, list_empty(&device->sync_ee));
- D_ASSERT(device, list_empty(&device->done_ee));
+ if (is_neg_ack)
+ drbd_rs_failed_io(peer_device, sector, size);
+ else
+ drbd_set_in_sync(peer_device, sector, size);
- return 0;
-}
+ atomic_sub(size >> 9, &connection->rs_in_flight);
-/*
- * We support PRO_VERSION_MIN to PRO_VERSION_MAX. The protocol version
- * we can agree on is stored in agreed_pro_version.
- *
- * feature flags and the reserved array should be enough room for future
- * enhancements of the handshake protocol, and possible plugins...
- *
- * for now, they are expected to be zero, but ignored.
- */
-static int drbd_send_features(struct drbd_connection *connection)
-{
- struct drbd_socket *sock;
- struct p_connection_features *p;
+ dec_rs_pending(peer_device);
- sock = &connection->data;
- p = conn_prepare_command(connection, sock);
- if (!p)
- return -EIO;
- memset(p, 0, sizeof(*p));
- p->protocol_min = cpu_to_be32(PRO_VERSION_MIN);
- p->protocol_max = cpu_to_be32(PRO_VERSION_MAX);
- p->feature_flags = cpu_to_be32(PRO_FEATURES);
- return conn_send_command(connection, sock, P_CONNECTION_FEATURES, sizeof(*p), NULL, 0);
+ /*
+ * Remove from the interval tree now so that
+ * find_resync_request() cannot find this request again
+ * if we get another ack for this interval.
+ */
+ drbd_remove_peer_req_interval(peer_req);
+
+ drbd_resync_read_req_mod(peer_req, INTERVAL_RECEIVED);
+ return 0;
}
-/*
- * return values:
- * 1 yes, we have a valid connection
- * 0 oops, did not work out, please try again
- * -1 peer talks different language,
- * no point in trying again, please go standalone.
- */
-static int drbd_do_features(struct drbd_connection *connection)
+static int got_NegAck(struct drbd_connection *connection, struct packet_info *pi)
{
- /* ASSERT current == connection->receiver ... */
- struct p_connection_features *p;
- const int expect = sizeof(struct p_connection_features);
- struct packet_info pi;
+ struct drbd_peer_device *peer_device;
+ struct p_block_ack *p = pi->data;
+ sector_t sector = be64_to_cpu(p->sector);
+ int size = be32_to_cpu(p->blksize);
int err;
- err = drbd_send_features(connection);
- if (err)
- return 0;
-
- err = drbd_recv_header(connection, &pi);
- if (err)
- return 0;
-
- if (pi.cmd != P_CONNECTION_FEATURES) {
- drbd_err(connection, "expected ConnectionFeatures packet, received: %s (0x%04x)\n",
- drbd_packet_name(pi.cmd), pi.cmd);
- return -1;
- }
-
- if (pi.size != expect) {
- drbd_err(connection, "expected ConnectionFeatures length: %u, received: %u\n",
- expect, pi.size);
- return -1;
- }
+ /* P_NEG_ACK used to be used instead of P_RS_NEG_ACK. */
+ if (p->block_id == ID_SYNCER)
+ return got_RSWriteAck(connection, pi);
- p = pi.data;
- err = drbd_recv_all_warn(connection, p, expect);
- if (err)
- return 0;
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
- p->protocol_min = be32_to_cpu(p->protocol_min);
- p->protocol_max = be32_to_cpu(p->protocol_max);
- if (p->protocol_max == 0)
- p->protocol_max = p->protocol_min;
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- if (PRO_VERSION_MAX < p->protocol_min ||
- PRO_VERSION_MIN > p->protocol_max)
- goto incompat;
+ if (peer_device->disk_state[NOW] == D_UP_TO_DATE)
+ set_bit(GOT_NEG_ACK, peer_device->flags);
- connection->agreed_pro_version = min_t(int, PRO_VERSION_MAX, p->protocol_max);
- connection->agreed_features = PRO_FEATURES & be32_to_cpu(p->feature_flags);
+ err = validate_req_change_req_state(peer_device, p->block_id, sector,
+ INTERVAL_LOCAL_WRITE, __func__, NEG_ACKED, true);
+ if (err) {
+ /* Protocol A has no P_WRITE_ACKs, but has P_NEG_ACKs.
+ The master bio might already be completed, therefore the
+ request is no longer in the collision hash. */
+ /* In Protocol B we might already have got a P_RECV_ACK
+ but then get a P_NEG_ACK afterwards. */
+ drbd_set_out_of_sync(peer_device, sector, size);
+ }
+ return 0;
+}
- drbd_info(connection, "Handshake successful: "
- "Agreed network protocol version %d\n", connection->agreed_pro_version);
+/* The peer, a sync target, can not secure this write toward its sync source
+ * and counts it as not processed. Mark the request to be retried: once its
+ * references drain, drbd_restart_request() re-submits the original bio as a
+ * brand-new request; under IO suspension only after the resume.
+ */
+static int got_RetryWrite(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct p_block_ack *p = pi->data;
+ sector_t sector = be64_to_cpu(p->sector);
- drbd_info(connection, "Feature flags enabled on protocol level: 0x%x%s%s%s%s.\n",
- connection->agreed_features,
- connection->agreed_features & DRBD_FF_TRIM ? " TRIM" : "",
- connection->agreed_features & DRBD_FF_THIN_RESYNC ? " THIN_RESYNC" : "",
- connection->agreed_features & DRBD_FF_WSAME ? " WRITE_SAME" : "",
- connection->agreed_features & DRBD_FF_WZEROES ? " WRITE_ZEROES" :
- connection->agreed_features ? "" : " none");
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
- return 1;
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- incompat:
- drbd_err(connection, "incompatible DRBD dialects: "
- "I support %d-%d, peer supports %d-%d\n",
- PRO_VERSION_MIN, PRO_VERSION_MAX,
- p->protocol_min, p->protocol_max);
- return -1;
+ /* A request that is gone was already resolved by connection loss. */
+ validate_req_change_req_state(peer_device, p->block_id, sector,
+ INTERVAL_LOCAL_WRITE, __func__,
+ POSTPONED_BY_PEER, true);
+ return 0;
}
-#if !defined(CONFIG_CRYPTO_HMAC) && !defined(CONFIG_CRYPTO_HMAC_MODULE)
-static int drbd_do_auth(struct drbd_connection *connection)
+static int got_NegDReply(struct drbd_connection *connection, struct packet_info *pi)
{
- drbd_err(connection, "This kernel was build without CONFIG_CRYPTO_HMAC.\n");
- drbd_err(connection, "You need to disable 'cram-hmac-alg' in drbd.conf.\n");
- return -1;
-}
-#else
-#define CHALLENGE_LEN 64
-
-/* Return value:
- 1 - auth succeeded,
- 0 - failed, try again (network error),
- -1 - auth failed, don't try again.
-*/
+ struct drbd_peer_device *peer_device;
+ struct p_block_ack *p = pi->data;
+ sector_t sector = be64_to_cpu(p->sector);
-static int drbd_do_auth(struct drbd_connection *connection)
-{
- struct drbd_socket *sock;
- char my_challenge[CHALLENGE_LEN]; /* 64 Bytes... */
- char *response = NULL;
- char *right_response = NULL;
- char *peers_ch = NULL;
- unsigned int key_len;
- char secret[SHARED_SECRET_MAX]; /* 64 byte */
- unsigned int resp_size;
- struct shash_desc *desc;
- struct packet_info pi;
- struct net_conf *nc;
- int err, rv;
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
- /* FIXME: Put the challenge/response into the preallocated socket buffer. */
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- key_len = strlen(nc->shared_secret);
- memcpy(secret, nc->shared_secret, key_len);
- rcu_read_unlock();
+ drbd_warn_ratelimit(peer_device, "Got NegDReply; Sector %llus, len %u.\n",
+ (unsigned long long)sector, be32_to_cpu(p->blksize));
- desc = kmalloc(sizeof(struct shash_desc) +
- crypto_shash_descsize(connection->cram_hmac_tfm),
- GFP_KERNEL);
- if (!desc) {
- rv = -1;
- goto fail;
- }
- desc->tfm = connection->cram_hmac_tfm;
+ return validate_req_change_req_state(peer_device, p->block_id, sector,
+ INTERVAL_LOCAL_READ, __func__,
+ NEG_ACKED, false);
+}
- rv = crypto_shash_setkey(connection->cram_hmac_tfm, (u8 *)secret, key_len);
- if (rv) {
- drbd_err(connection, "crypto_shash_setkey() failed with %d\n", rv);
- rv = -1;
- goto fail;
- }
+void drbd_unsuccessful_resync_request(struct drbd_peer_request *peer_req, bool failed)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
- get_random_bytes(my_challenge, CHALLENGE_LEN);
+ if (get_ldev_if_state(device, D_DETACHING)) {
+ if (failed) {
+ drbd_rs_failed_io(peer_device, peer_req->i.sector, peer_req->i.size);
+ } else {
+ if (drbd_interval_is_verify(&peer_req->i)) {
+ drbd_verify_skipped_block(peer_device, peer_req->i.sector, peer_req->i.size);
+ verify_progress(peer_device, peer_req->i.sector, peer_req->i.size);
+ } else {
+ set_bit(RS_REQUEST_UNSUCCESSFUL, peer_device->flags);
+ }
+ }
- sock = &connection->data;
- if (!conn_prepare_command(connection, sock)) {
- rv = 0;
- goto fail;
+ rs_sectors_came_in(peer_device, peer_req->i.size);
+ mod_timer(&peer_device->resync_timer, jiffies + RS_MAKE_REQS_INTV);
+ put_ldev(device);
}
- rv = !conn_send_command(connection, sock, P_AUTH_CHALLENGE, 0,
- my_challenge, CHALLENGE_LEN);
- if (!rv)
- goto fail;
- err = drbd_recv_header(connection, &pi);
- if (err) {
- rv = 0;
- goto fail;
- }
+ drbd_remove_peer_req_interval(peer_req);
+ drbd_free_peer_req(peer_req);
+}
- if (pi.cmd != P_AUTH_CHALLENGE) {
- drbd_err(connection, "expected AuthChallenge packet, received: %s (0x%04x)\n",
- drbd_packet_name(pi.cmd), pi.cmd);
- rv = -1;
- goto fail;
- }
+static int got_NegRSDReply(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_peer_request *peer_req;
+ sector_t sector;
+ int size;
+ u64 block_id;
+ struct p_block_ack *p = pi->data;
- if (pi.size > CHALLENGE_LEN * 2) {
- drbd_err(connection, "expected AuthChallenge payload too big.\n");
- rv = -1;
- goto fail;
- }
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
- if (pi.size < CHALLENGE_LEN) {
- drbd_err(connection, "AuthChallenge payload too small.\n");
- rv = -1;
- goto fail;
- }
+ sector = be64_to_cpu(p->sector);
+ size = be32_to_cpu(p->blksize);
- peers_ch = kmalloc(pi.size, GFP_NOIO);
- if (!peers_ch) {
- rv = -1;
- goto fail;
- }
+ /* Prior to protocol version 122, block_id may be meaningless. */
+ block_id = peer_device->connection->agreed_pro_version >= 122 ? p->block_id : ID_SYNCER;
- err = drbd_recv_all_warn(connection, peers_ch, pi.size);
- if (err) {
- rv = 0;
- goto fail;
- }
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- if (!memcmp(my_challenge, peers_ch, CHALLENGE_LEN)) {
- drbd_err(connection, "Peer presented the same challenge!\n");
- rv = -1;
- goto fail;
- }
+ peer_req = find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_RESYNC_WRITE) |
+ INTERVAL_TYPE_MASK(INTERVAL_OV_READ_SOURCE),
+ sector, size, block_id);
+ if (!peer_req)
+ return -EIO;
- resp_size = crypto_shash_digestsize(connection->cram_hmac_tfm);
- response = kmalloc(resp_size, GFP_NOIO);
- if (!response) {
- rv = -1;
- goto fail;
- }
+ dec_rs_pending(peer_device);
- rv = crypto_shash_digest(desc, peers_ch, pi.size, response);
- if (rv) {
- drbd_err(connection, "crypto_hash_digest() failed with %d\n", rv);
- rv = -1;
- goto fail;
- }
+ if (pi->cmd == P_RS_CANCEL_AHEAD &&
+ !test_and_set_bit(SYNC_TARGET_TO_BEHIND, peer_device->flags) &&
+ peer_device->repl_state[NOW] == L_SYNC_TARGET)
+ drbd_info(peer_device, "peer pulled ahead during resync\n");
- if (!conn_prepare_command(connection, sock)) {
- rv = 0;
- goto fail;
- }
- rv = !conn_send_command(connection, sock, P_AUTH_RESPONSE, 0,
- response, resp_size);
- if (!rv)
- goto fail;
+ drbd_unsuccessful_resync_request(peer_req, pi->cmd == P_NEG_RS_DREPLY);
+ return 0;
+}
- err = drbd_recv_header(connection, &pi);
- if (err) {
- rv = 0;
- goto fail;
- }
+/* Find the write a dagtag wait answer is for. The answer arrives on the connection
+ * to the sync source, the write itself belongs to the connection to its
+ * writer: either parked on that connection's source_wait_ee (taken off and
+ * returned), or still on its peer_requests because the answer overtook the
+ * local completion -- then the answer is recorded on the request, and
+ * e_end_block() acts on it.
+ */
+static bool dagtag_wait_answer_matches(struct drbd_peer_request *peer_req,
+ unsigned int source_node_id, sector_t sector,
+ u64 block_id)
+{
+ sector_t range_sector;
+ unsigned int range_size;
- if (pi.cmd != P_AUTH_RESPONSE) {
- drbd_err(connection, "expected AuthResponse packet, received: %s (0x%04x)\n",
- drbd_packet_name(pi.cmd), pi.cmd);
- rv = 0;
- goto fail;
- }
+ if ((u64)(unsigned long)peer_req != block_id ||
+ !(peer_req->flags & EE_WAIT_FOR_SOURCE) ||
+ peer_req->wait_source_node_id != source_node_id)
+ return false;
- if (pi.size != resp_size) {
- drbd_err(connection, "expected AuthResponse payload of wrong size\n");
- rv = 0;
- goto fail;
- }
+ /* The answer echoes the range the request named. A request that
+ * is gone may have left its address to a new one; do not let a
+ * late answer settle that one.
+ */
+ unsecured_write_range(peer_req, &range_sector, &range_size);
+ return range_sector == sector;
+}
- err = drbd_recv_all_warn(connection, response , resp_size);
- if (err) {
- rv = 0;
- goto fail;
- }
+static struct drbd_peer_request *dagtag_wait_answer_request(struct drbd_resource *resource,
+ unsigned int source_node_id,
+ sector_t sector, u64 block_id,
+ unsigned long answer_flag)
+{
+ struct drbd_peer_request *peer_req, *found = NULL;
+ struct drbd_connection *connection;
+ bool recorded = false;
- right_response = kmalloc(resp_size, GFP_NOIO);
- if (!right_response) {
- rv = -1;
- goto fail;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry(peer_req, &connection->source_wait_ee, w.list) {
+ if (!dagtag_wait_answer_matches(peer_req, source_node_id, sector, block_id))
+ continue;
+ list_del(&peer_req->w.list);
+ /* Reached: acknowledged from here on, nothing can
+ * refuse it any more. Unreachable: refused next.
+ */
+ if (answer_flag == EE_SOURCE_REACHED)
+ peer_req->flags |= EE_ACK_DECIDED;
+ found = peer_req;
+ break;
+ }
+ if (!found) {
+ list_for_each_entry(peer_req, &connection->peer_requests, recv_order) {
+ if (!dagtag_wait_answer_matches(peer_req, source_node_id,
+ sector, block_id))
+ continue;
+ peer_req->flags |= answer_flag;
+ recorded = true;
+ break;
+ }
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ if (found || recorded)
+ break;
}
+ rcu_read_unlock();
- rv = crypto_shash_digest(desc, my_challenge, CHALLENGE_LEN,
- right_response);
- if (rv) {
- drbd_err(connection, "crypto_hash_digest() failed with %d\n", rv);
- rv = -1;
- goto fail;
- }
+ return found;
+}
- rv = !memcmp(response, right_response, resp_size);
+static int got_RSDagtagReached(struct drbd_connection *connection, struct packet_info *pi)
+{
+ struct p_block_ack *p = pi->data;
+ struct drbd_peer_device *peer_device;
+ struct drbd_peer_request *peer_req;
- if (rv)
- drbd_info(connection, "Peer authenticated using %d bytes HMAC\n",
- resp_size);
- else
- rv = -1;
+ peer_device = conn_peer_device(connection, pi->vnr);
+ if (!peer_device)
+ return -EIO;
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- fail:
- kfree(peers_ch);
- kfree(response);
- kfree(right_response);
- if (desc) {
- shash_desc_zero(desc);
- kfree(desc);
- }
+ peer_req = dagtag_wait_answer_request(connection->resource, connection->peer_node_id,
+ be64_to_cpu(p->sector), p->block_id,
+ EE_SOURCE_REACHED);
+ if (!peer_req)
+ return 0;
- return rv;
+ peer_req->flags &= ~(EE_WAIT_FOR_SOURCE | EE_SOURCE_REACHED);
+ drbd_queue_work(&peer_req->peer_device->connection->sender_work, &peer_req->w);
+
+ return 0;
}
-#endif
-int drbd_receiver(struct drbd_thread *thi)
+static bool peer_req_overlaps(struct drbd_peer_request *peer_req, sector_t sector,
+ unsigned int size)
{
- struct drbd_connection *connection = thi->connection;
- int h;
+ return peer_req->i.size &&
+ peer_req->i.sector < sector + (size >> SECTOR_SHIFT) &&
+ sector < peer_req->i.sector + (peer_req->i.size >> SECTOR_SHIFT);
+}
+
+/* The sync source can not serve a resync request of ours with its dependency
+ * met: it has lost the node whose write stream the request depends on, so its
+ * copy of the range is older than every write of that stream this node
+ * received since. It may still overwrite the range once no such write can be
+ * lost: every protocol C write in the range whose writer may still hold it
+ * pending is taken back from the writer the way drbd_refuse_unsecured_write()
+ * does it, retried or held across a severed connection, and reaches this node
+ * again once the resync brought it up to date. A write whose acknowledgment
+ * is already decided, or a protocol A or B write, can not be secured.
+ *
+ * Runs on the ack receiver of the connection to the sync source. Returns
+ * true when the range is secured.
+ */
+static bool drbd_secure_resync_range(struct drbd_peer_device *source,
+ sector_t sector, unsigned int size)
+{
+ struct drbd_device *device = source->device;
+ struct drbd_resource *resource = device->resource;
+ struct drbd_connection *connection;
+ struct drbd_peer_request *peer_req, *t;
+ LIST_HEAD(refuse_list);
+ bool secured = true;
- drbd_info(connection, "receiver (re)started\n");
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection == source->connection)
+ continue;
- do {
- h = conn_connect(connection);
- if (h == 0) {
- conn_disconnect(connection);
- schedule_timeout_interruptible(HZ);
+ spin_lock_irq(&connection->peer_reqs_lock);
+ /* Withheld, waiting for the source's answer: this is it. */
+ list_for_each_entry_safe(peer_req, t, &connection->source_wait_ee, w.list) {
+ if (peer_req->peer_device->device != device ||
+ !peer_req_overlaps(peer_req, sector, size))
+ continue;
+ if (!unsecured_write_refusable(peer_req)) {
+ secured = false;
+ break;
+ }
+ list_move_tail(&peer_req->w.list, &refuse_list);
}
- if (h == -1) {
- drbd_warn(connection, "Discarding network configuration.\n");
- conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_HARD);
+ /* Not completed yet, or completed and not peer-acked yet. */
+ list_for_each_entry(peer_req, &connection->peer_requests, recv_order) {
+ if (!secured)
+ break;
+ if (peer_req->peer_device->device != device ||
+ !peer_req_overlaps(peer_req, sector, size))
+ continue;
+ if (peer_req->flags & EE_POSTPONE)
+ continue; /* refused already */
+ if (peer_req->flags & EE_ACK_DECIDED ||
+ !unsecured_write_refusable(peer_req)) {
+ secured = false;
+ break;
+ }
+ /* Still to complete: have e_end_block() refuse it. */
+ peer_req->wait_source_node_id = source->node_id;
+ peer_req->flags |= EE_WAIT_FOR_SOURCE | EE_SOURCE_UNREACHABLE;
}
- } while (h == 0);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+ if (!secured)
+ break;
+ }
+ rcu_read_unlock();
- if (h > 0) {
- blk_start_plug(&connection->receiver_plug);
- drbdd(connection);
- blk_finish_plug(&connection->receiver_plug);
+ list_for_each_entry_safe(peer_req, t, &refuse_list, w.list) {
+ list_del_init(&peer_req->w.list);
+ drbd_refuse_unsecured_write(peer_req);
}
- conn_disconnect(connection);
+ return secured;
+}
+
+/* Ask the sync source again for a range drbd_secure_resync_range() secured,
+ * this time without the dependency the source can not meet. On cancel the
+ * teardown frees the request.
+ */
+static int w_send_unsecured_resync_request(struct drbd_work *w, int cancel)
+{
+ struct drbd_peer_request *peer_req =
+ container_of(w, struct drbd_peer_request, w);
+
+ if (cancel)
+ return 0;
- drbd_info(connection, "receiver terminated\n");
+ drbd_send_rs_request(peer_req->peer_device, P_RS_DATA_REQUEST,
+ peer_req->i.sector, peer_req->i.size,
+ peer_req->block_id, 0, 0);
return 0;
}
-/* ********* acknowledge sender ******** */
-
-static int got_conn_RqSReply(struct drbd_connection *connection, struct packet_info *pi)
+static void drbd_resync_request_unreachable(struct drbd_peer_request *peer_req)
{
- struct p_req_state_reply *p = pi->data;
- int retcode = be32_to_cpu(p->retcode);
+ struct drbd_peer_device *source = peer_req->peer_device;
- if (retcode >= SS_SUCCESS) {
- set_bit(CONN_WD_ST_CHG_OKAY, &connection->flags);
- } else {
- set_bit(CONN_WD_ST_CHG_FAIL, &connection->flags);
- drbd_err(connection, "Requested state change failed by peer: %s (%d)\n",
- drbd_set_st_err_str(retcode), retcode);
+ if (repl_is_sync_target(source->repl_state[NOW]) &&
+ drbd_secure_resync_range(source, peer_req->i.sector, peer_req->i.size)) {
+ dynamic_drbd_dbg(source,
+ "Resync request at %llus+%u: source lost the stream it depends on, asking without\n",
+ (unsigned long long)peer_req->i.sector, peer_req->i.size);
+ peer_req->w.cb = w_send_unsecured_resync_request;
+ drbd_queue_work(&source->connection->sender_work, &peer_req->w);
+ return;
}
- wake_up(&connection->ping_wait);
- return 0;
+ drbd_info_ratelimit(source,
+ "Can not secure resync request at %llus+%u against a source that lost the writer, ending the resync\n",
+ (unsigned long long)peer_req->i.sector, peer_req->i.size);
+ if (repl_is_sync_target(source->repl_state[NOW]))
+ change_repl_state(source, L_ESTABLISHED, CS_VERBOSE, "unsecured-resync");
+ dec_rs_pending(source);
+ drbd_unsuccessful_resync_request(peer_req, false);
}
-static int got_RqSReply(struct drbd_connection *connection, struct packet_info *pi)
+/* The sync source can not reach the position in the writer's stream by
+ * itself: for a resync request of ours, secure the range and ask again
+ * without the dependency; for a write withheld from acknowledgment, refuse
+ * it.
+ */
+static int got_RSDagtagUnreachable(struct drbd_connection *connection, struct packet_info *pi)
{
+ struct p_block_ack *p = pi->data;
struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_req_state_reply *p = pi->data;
- int retcode = be32_to_cpu(p->retcode);
+ struct drbd_peer_request *peer_req;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
return -EIO;
- device = peer_device->device;
+ update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
- if (test_bit(CONN_WD_ST_CHG_REQ, &connection->flags)) {
- D_ASSERT(device, connection->agreed_pro_version < 100);
- return got_conn_RqSReply(connection, pi);
+ peer_req = __find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_RESYNC_WRITE),
+ be64_to_cpu(p->sector), be32_to_cpu(p->blksize),
+ p->block_id, false);
+ if (peer_req) {
+ drbd_resync_request_unreachable(peer_req);
+ return 0;
}
- if (retcode >= SS_SUCCESS) {
- set_bit(CL_ST_CHG_SUCCESS, &device->flags);
- } else {
- set_bit(CL_ST_CHG_FAIL, &device->flags);
- drbd_err(device, "Requested state change failed by peer: %s (%d)\n",
- drbd_set_st_err_str(retcode), retcode);
- }
- wake_up(&device->state_wait);
+ peer_req = dagtag_wait_answer_request(connection->resource, connection->peer_node_id,
+ be64_to_cpu(p->sector), p->block_id,
+ EE_SOURCE_UNREACHABLE);
+ if (!peer_req)
+ return 0;
+
+ drbd_refuse_unsecured_write(peer_req);
return 0;
}
-static int got_Ping(struct drbd_connection *connection, struct packet_info *pi)
+static int got_BarrierAck(struct drbd_connection *connection, struct packet_info *pi)
{
- return drbd_send_ping_ack(connection);
+ struct p_barrier_ack *p = pi->data;
+ struct drbd_peer_device *peer_device;
+ bool release_gen = false;
+ int vnr, err;
+
+ /* tl_release flushes this epoch (RQ_NET_DONE) first, so the confirm below
+ * sees the just-flushed held writes.
+ */
+ err = tl_release(connection, 0, 0, p->barrier, be32_to_cpu(p->set_size));
+ /* confirm any peer that has now durably persisted all the generation's
+ * held writes, then re-evaluate the release gate.
+ */
+ if (!err) {
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ drbd_peer_maybe_confirm_rotated_gen(peer_device, p->barrier);
+ if (drbd_maybe_release_rotated_gen(peer_device->device))
+ release_gen = true;
+ }
+ rcu_read_unlock();
+ if (release_gen)
+ tl_walk(connection, NULL, NEW_UUID_CONFIRMED);
+ }
+
+ return err;
}
-static int got_PingAck(struct drbd_connection *connection, struct packet_info *pi)
+static int got_confirm_stable(struct drbd_connection *connection, struct packet_info *pi)
{
- /* restore idle timeout */
- connection->meta.socket->sk->sk_rcvtimeo = connection->net_conf->ping_int*HZ;
- if (!test_and_set_bit(GOT_PING_ACK, &connection->flags))
- wake_up(&connection->ping_wait);
+ struct p_confirm_stable *p = pi->data;
- return 0;
+ return tl_release(connection, p->oldest_block_id, p->youngest_block_id, 0,
+ be32_to_cpu(p->set_size));
}
-static int got_IsInSync(struct drbd_connection *connection, struct packet_info *pi)
+static int got_OVResult(struct drbd_connection *connection, struct packet_info *pi)
{
struct drbd_peer_device *peer_device;
struct drbd_device *device;
- struct p_block_ack *p = pi->data;
- sector_t sector = be64_to_cpu(p->sector);
- int blksize = be32_to_cpu(p->blksize);
+ struct drbd_peer_request *peer_req;
+ sector_t sector;
+ int size;
+ u64 block_id;
+ u32 seq_num;
+ enum ov_result result;
peer_device = conn_peer_device(connection, pi->vnr);
if (!peer_device)
return -EIO;
device = peer_device->device;
- D_ASSERT(device, peer_device->connection->agreed_pro_version >= 89);
+ if (pi->cmd == P_OV_RESULT) {
+ struct p_block_ack *p = pi->data;
- update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
+ sector = be64_to_cpu(p->sector);
+ size = be32_to_cpu(p->blksize);
+ block_id = ID_SYNCER;
+ seq_num = be32_to_cpu(p->seq_num);
+ result = drbd_block_id_to_ov_result(be64_to_cpu(p->block_id));
+ } else { /* P_OV_RESULT_ID */
+ struct p_ov_result *p = pi->data;
- if (get_ldev(device)) {
- drbd_rs_complete_io(device, sector);
- drbd_set_in_sync(peer_device, sector, blksize);
- /* rs_same_csums is supposed to count in units of BM_BLOCK_SIZE */
- device->rs_same_csum += (blksize >> BM_BLOCK_SHIFT);
- put_ldev(device);
+ sector = be64_to_cpu(p->sector);
+ size = be32_to_cpu(p->blksize);
+ block_id = p->block_id;
+ seq_num = be32_to_cpu(p->seq_num);
+ result = be32_to_cpu(p->result);
}
+
+ update_peer_seq(peer_device, seq_num);
+
+ peer_req = find_resync_request(peer_device, INTERVAL_TYPE_MASK(INTERVAL_OV_READ_TARGET),
+ sector, size, block_id);
+ if (!peer_req)
+ return -EIO;
+
+ drbd_remove_peer_req_interval(peer_req);
+
+ /* This may be a request that we could not cancel because the peer does
+ * not understand P_RS_CANCEL. Treat it as a skipped block. */
+ if (connection->agreed_pro_version < 110 && test_bit(INTERVAL_CONFLICT, &peer_req->i.flags))
+ result = OV_RESULT_SKIP;
+
+ drbd_free_peer_req(peer_req);
+ peer_req = NULL;
+
+ if (result == OV_RESULT_SKIP)
+ drbd_verify_skipped_block(peer_device, sector, size);
+ if (result == OV_RESULT_OUT_OF_SYNC)
+ drbd_ov_out_of_sync_found(peer_device, sector, size);
+ else
+ ov_out_of_sync_print(peer_device);
+
dec_rs_pending(peer_device);
- atomic_add(blksize >> 9, &device->rs_sect_in);
+ if (!get_ldev(device))
+ return 0;
+
+ verify_progress(peer_device, sector, size);
+
+ put_ldev(device);
+ return 0;
+}
+
+static int got_skip(struct drbd_connection *connection, struct packet_info *pi)
+{
return 0;
}
-static int
-validate_req_change_req_state(struct drbd_peer_device *peer_device, u64 id, sector_t sector,
- struct rb_root *root, const char *func,
- enum drbd_req_event what, bool missing_ok)
+static u64 node_id_to_mask(struct drbd_peer_md *peer_md, int node_id)
{
- struct drbd_device *device = peer_device->device;
- struct drbd_request *req;
- struct bio_and_error m;
+ int bitmap_bit = peer_md[node_id].bitmap_index;
- spin_lock_irq(&device->resource->req_lock);
- req = find_request(device, root, id, sector, missing_ok, func);
- if (unlikely(!req)) {
- spin_unlock_irq(&device->resource->req_lock);
- return -EIO;
+ return (bitmap_bit >= 0) ? NODE_MASK(bitmap_bit) : 0;
+}
+
+static u64 node_ids_to_bitmap(struct drbd_device *device, u64 node_ids)
+{
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ u64 bitmap_bits = 0;
+ int node_id;
+
+ for_each_set_bit(node_id, (unsigned long *)&node_ids, DRBD_NODE_ID_MAX)
+ bitmap_bits |= node_id_to_mask(peer_md, node_id);
+ return bitmap_bits;
+}
+
+static struct drbd_peer_request *drbd_send_oos_next_req(struct drbd_resource *resource,
+ int oos_node_id, struct drbd_peer_request *peer_req)
+{
+ lockdep_assert_held(&resource->send_oos_lock);
+
+ if (peer_req == NULL)
+ peer_req = list_entry(&resource->send_oos,
+ struct drbd_peer_request, recv_order);
+
+ list_for_each_entry_continue(peer_req, &resource->send_oos, recv_order) {
+ if (NODE_MASK(oos_node_id) & peer_req->send_oos_pending)
+ return peer_req;
}
- __req_mod(req, what, peer_device, &m);
- spin_unlock_irq(&device->resource->req_lock);
- if (m.bio)
- complete_master_bio(device, &m);
- return 0;
+ return NULL;
}
-static int got_BlockAck(struct drbd_connection *connection, struct packet_info *pi)
+static void free_send_oos_peer_req(struct drbd_peer_request *peer_req)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_block_ack *p = pi->data;
- sector_t sector = be64_to_cpu(p->sector);
- int blksize = be32_to_cpu(p->blksize);
- enum drbd_req_event what;
+ struct drbd_connection *peer_ack_connection = peer_req->peer_device->connection;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ peer_req->flags &= ~EE_ON_SEND_OOS;
+ drbd_free_peer_req(peer_req);
+ kref_put(&peer_ack_connection->kref, drbd_destroy_connection);
+}
- update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
+/*
+ * Only this work clears the bit of its peer in send_oos_pending, so the
+ * next entry stays on the list while send_oos_lock is dropped.
+ *
+ * @cancel: the stream to the peer is dead. The work was queued while the
+ * peer was connected and still owns its bits on the send_oos list. Clear
+ * them without sending: our bitmap covers the writes.
+ */
+int drbd_send_out_of_sync_wf(struct drbd_work *w, int cancel)
+{
+ struct drbd_connection *oos_connection =
+ container_of(w, struct drbd_connection, send_oos_work);
+ struct drbd_resource *resource = oos_connection->resource;
+ int oos_node_id = oos_connection->peer_node_id;
+ struct drbd_peer_request *peer_req;
- if (p->block_id == ID_SYNCER) {
- drbd_set_in_sync(peer_device, sector, blksize);
- dec_rs_pending(peer_device);
- return 0;
- }
- switch (pi->cmd) {
- case P_RS_WRITE_ACK:
- what = WRITE_ACKED_BY_PEER_AND_SIS;
- break;
- case P_WRITE_ACK:
- what = WRITE_ACKED_BY_PEER;
- break;
- case P_RECV_ACK:
- what = RECV_ACKED_BY_PEER;
- break;
- case P_SUPERSEDED:
- what = CONFLICT_RESOLVED;
- break;
- case P_RETRY_WRITE:
- what = POSTPONE_WRITE;
- break;
- default:
- BUG();
+ spin_lock_irq(&resource->send_oos_lock);
+ peer_req = drbd_send_oos_next_req(resource, oos_node_id, NULL);
+ spin_unlock_irq(&resource->send_oos_lock);
+
+ while (peer_req) {
+ struct drbd_peer_device *peer_device =
+ conn_peer_device(oos_connection, peer_req->peer_device->device->vnr);
+ struct drbd_peer_request *next_peer_req;
+ bool free_it;
+
+ /* Ignore errors and keep iterating to clear up list */
+ if (!cancel)
+ drbd_send_out_of_sync(peer_device, peer_req->i.sector, peer_req->i.size);
+
+ spin_lock_irq(&resource->send_oos_lock);
+ peer_req->send_oos_pending &= ~NODE_MASK(oos_node_id);
+ next_peer_req = drbd_send_oos_next_req(resource, oos_node_id, peer_req);
+ free_it = !peer_req->send_oos_pending;
+ if (free_it)
+ list_del(&peer_req->recv_order);
+ spin_unlock_irq(&resource->send_oos_lock);
+
+ if (free_it)
+ free_send_oos_peer_req(peer_req);
+
+ peer_req = next_peer_req;
}
- return validate_req_change_req_state(peer_device, p->block_id, sector,
- &device->write_requests, __func__,
- what, false);
+ return 0;
+}
+
+static bool is_sync_source(struct drbd_peer_device *peer_device)
+{
+ return is_sync_source_state(peer_device, NOW) ||
+ peer_device->repl_state[NOW] == L_WF_BITMAP_S;
}
-static int got_NegAck(struct drbd_connection *connection, struct packet_info *pi)
+static u64 drbd_calculate_send_oos_pending(struct drbd_device *device, u64 in_sync)
{
struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_block_ack *p = pi->data;
- sector_t sector = be64_to_cpu(p->sector);
- int size = be32_to_cpu(p->blksize);
- int err;
+ u64 send_oos_pending = 0;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (!(NODE_MASK(peer_device->node_id) & in_sync) &&
+ is_sync_source(peer_device))
+ send_oos_pending |= NODE_MASK(peer_device->node_id);
+ }
+ rcu_read_unlock();
- update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
+ return send_oos_pending;
+}
- if (p->block_id == ID_SYNCER) {
- dec_rs_pending(peer_device);
- drbd_rs_failed_io(peer_device, sector, size);
- return 0;
+static void drbd_queue_send_out_of_sync(struct drbd_connection *peer_ack_connection,
+ struct list_head *send_oos_peer_req_list, u64 any_send_oos_pending)
+{
+ struct drbd_resource *resource = peer_ack_connection->resource;
+ struct drbd_connection *oos_connection;
+ struct drbd_peer_request *peer_req, *tmp;
+ u64 queue_mask = 0;
+ LIST_HEAD(free_list);
+
+ if (!any_send_oos_pending)
+ return;
+
+ list_for_each_entry(peer_req, send_oos_peer_req_list, recv_order) {
+ kref_get(&peer_ack_connection->kref);
}
- err = validate_req_change_req_state(peer_device, p->block_id, sector,
- &device->write_requests, __func__,
- NEG_ACKED, true);
- if (err) {
- /* Protocol A has no P_WRITE_ACKs, but has P_NEG_ACKs.
- The master bio might already be completed, therefore the
- request is no longer in the collision hash. */
- /* In Protocol B we might already have got a P_RECV_ACK
- but then get a P_NEG_ACK afterwards. */
- drbd_set_out_of_sync(peer_device, sector, size);
+ /*
+ * Hold state_rwlock from choosing the peers until their work is queued,
+ * so that each of them is still connected and its sender still runs.
+ */
+ read_lock_irq(&resource->state_rwlock);
+ for_each_connection(oos_connection, resource) {
+ u64 m = NODE_MASK(oos_connection->peer_node_id);
+
+ if (m & any_send_oos_pending && oos_connection->cstate[NOW] >= C_CONNECTED)
+ queue_mask |= m;
+ }
+
+ /*
+ * A peer that lost its connection since drbd_calculate_send_oos_pending()
+ * gets no P_OUT_OF_SYNC. Our bitmap bits for it cover these writes.
+ */
+ if (any_send_oos_pending & ~queue_mask) {
+ list_for_each_entry_safe(peer_req, tmp, send_oos_peer_req_list, recv_order) {
+ peer_req->send_oos_pending &= queue_mask;
+ if (!peer_req->send_oos_pending)
+ list_move_tail(&peer_req->recv_order, &free_list);
+ }
+ }
+
+ spin_lock(&resource->send_oos_lock);
+ list_splice_tail(send_oos_peer_req_list, &resource->send_oos);
+ spin_unlock(&resource->send_oos_lock);
+
+ for_each_connection(oos_connection, resource) {
+ if (NODE_MASK(oos_connection->peer_node_id) & queue_mask)
+ drbd_queue_work_if_unqueued(&oos_connection->sender_work,
+ &oos_connection->send_oos_work);
+ }
+ read_unlock_irq(&resource->state_rwlock);
+
+ /* Not the last krefs: the caller runs on peer_ack_connection */
+ list_for_each_entry_safe(peer_req, tmp, &free_list, recv_order) {
+ list_del(&peer_req->recv_order);
+ free_send_oos_peer_req(peer_req);
}
- return 0;
}
-static int got_NegDReply(struct drbd_connection *connection, struct packet_info *pi)
+/*
+ * All senders have stopped, and each ran its work. Anything left on the
+ * send_oos list has bits that no work clears.
+ */
+void drbd_free_send_oos_leftovers(struct drbd_resource *resource)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_block_ack *p = pi->data;
- sector_t sector = be64_to_cpu(p->sector);
-
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ struct drbd_peer_request *peer_req, *tmp;
+ LIST_HEAD(free_list);
- update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
+ spin_lock_irq(&resource->send_oos_lock);
+ list_splice_init(&resource->send_oos, &free_list);
+ spin_unlock_irq(&resource->send_oos_lock);
- drbd_err(device, "Got NegDReply; Sector %llus, len %u.\n",
- (unsigned long long)sector, be32_to_cpu(p->blksize));
+ if (list_empty(&free_list))
+ return;
- return validate_req_change_req_state(peer_device, p->block_id, sector,
- &device->read_requests, __func__,
- NEG_ACKED, false);
+ drbd_err(resource, "peer requests left on send_oos list\n");
+ list_for_each_entry_safe(peer_req, tmp, &free_list, recv_order) {
+ list_del(&peer_req->recv_order);
+ free_send_oos_peer_req(peer_req);
+ }
}
-static int got_NegRSDReply(struct drbd_connection *connection, struct packet_info *pi)
+static int got_peer_ack(struct drbd_connection *connection, struct packet_info *pi)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- sector_t sector;
- int size;
- struct p_block_ack *p = pi->data;
+ struct p_peer_ack *p = pi->data;
+ u64 dagtag, in_sync;
+ struct drbd_peer_request *peer_req, *tmp;
+ struct list_head work_list;
+ u64 any_send_oos_pending = 0;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ dagtag = be64_to_cpu(p->dagtag);
+ in_sync = be64_to_cpu(p->mask);
- sector = be64_to_cpu(p->sector);
- size = be32_to_cpu(p->blksize);
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_for_each_entry(peer_req, &connection->peer_requests, recv_order) {
+ if (dagtag == peer_req->dagtag_sector)
+ goto found;
+ }
+ spin_unlock_irq(&connection->peer_reqs_lock);
- update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
+ drbd_err(connection, "peer request with dagtag %llu not found\n", dagtag);
+ return -EIO;
- dec_rs_pending(peer_device);
+found:
+ list_cut_position(&work_list, &connection->peer_requests, &peer_req->recv_order);
+ spin_unlock_irq(&connection->peer_reqs_lock);
- if (get_ldev_if_state(device, D_FAILED)) {
- drbd_rs_complete_io(device, sector);
- switch (pi->cmd) {
- case P_NEG_RS_DREPLY:
- drbd_rs_failed_io(peer_device, sector, size);
- break;
- case P_RS_CANCEL:
- break;
- default:
- BUG();
+ list_for_each_entry_safe(peer_req, tmp, &work_list, recv_order) {
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ u64 in_sync_b, mask;
+
+ D_ASSERT(peer_device, peer_req->flags & EE_IN_ACTLOG);
+
+ if (get_ldev(device)) {
+ if ((peer_req->flags & EE_WAS_ERROR) == 0)
+ in_sync_b = node_ids_to_bitmap(device, in_sync);
+ else
+ in_sync_b = 0;
+ mask = ~node_id_to_mask(device->ldev->md.peers,
+ connection->peer_node_id);
+
+ drbd_set_sync(device, peer_req->i.sector,
+ peer_req->i.size, ~in_sync_b, mask);
+ drbd_al_complete_io(device, &peer_req->i);
+ put_ldev(device);
+ }
+
+ peer_req->send_oos_pending = drbd_calculate_send_oos_pending(device, in_sync);
+ any_send_oos_pending |= peer_req->send_oos_pending;
+ if (peer_req->send_oos_pending) {
+ peer_req->flags &= ~EE_ON_RECV_ORDER;
+ peer_req->flags |= EE_ON_SEND_OOS;
+ } else {
+ drbd_free_peer_req(peer_req); /* list_del() because EE_ON_RECV_ORDER */
}
- put_ldev(device);
}
+ drbd_queue_send_out_of_sync(connection, &work_list, any_send_oos_pending);
return 0;
}
-static int got_BarrierAck(struct drbd_connection *connection, struct packet_info *pi)
+void apply_unacked_peer_requests(struct drbd_connection *connection)
{
- struct p_barrier_ack *p = pi->data;
- struct drbd_peer_device *peer_device;
- int vnr;
-
- tl_release(connection, p->barrier, be32_to_cpu(p->set_size));
+ struct drbd_peer_request *peer_req;
+ unsigned long flags;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ spin_lock_irqsave(&connection->peer_reqs_lock, flags);
+ list_for_each_entry(peer_req, &connection->peer_requests, recv_order) {
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
struct drbd_device *device = peer_device->device;
+ int bitmap_index = peer_device->bitmap_index;
+ u64 mask = ~(bitmap_index != -1 ? 1UL << bitmap_index : 0UL);
- if (device->state.conn == C_AHEAD &&
- atomic_read(&device->ap_in_flight) == 0 &&
- !test_and_set_bit(AHEAD_TO_SYNC_SOURCE, &device->flags)) {
- device->start_resync_timer.expires = jiffies + HZ;
- add_timer(&device->start_resync_timer);
- }
+ drbd_set_sync(device, peer_req->i.sector, peer_req->i.size,
+ mask, mask);
}
- rcu_read_unlock();
-
- return 0;
+ spin_unlock_irqrestore(&connection->peer_reqs_lock, flags);
}
-static int got_OVResult(struct drbd_connection *connection, struct packet_info *pi)
+static void cleanup_unacked_peer_requests(struct drbd_connection *connection)
{
- struct drbd_peer_device *peer_device;
- struct drbd_device *device;
- struct p_block_ack *p = pi->data;
- struct drbd_device_work *dw;
- sector_t sector;
- int size;
+ struct drbd_peer_request *peer_req, *tmp;
+ LIST_HEAD(work_list);
+ u64 any_send_oos_pending = 0;
- peer_device = conn_peer_device(connection, pi->vnr);
- if (!peer_device)
- return -EIO;
- device = peer_device->device;
+ spin_lock_irq(&connection->peer_reqs_lock);
+ list_splice_init(&connection->peer_requests, &work_list);
+ spin_unlock_irq(&connection->peer_reqs_lock);
- sector = be64_to_cpu(p->sector);
- size = be32_to_cpu(p->blksize);
+ list_for_each_entry_safe(peer_req, tmp, &work_list, recv_order) {
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device->device;
+ int bitmap_index = peer_device->bitmap_index;
+ u64 mask = ~(bitmap_index != -1 ? 1UL << bitmap_index : 0UL);
- update_peer_seq(peer_device, be32_to_cpu(p->seq_num));
+ if (get_ldev(device)) {
+ drbd_set_sync(device, peer_req->i.sector, peer_req->i.size,
+ mask, mask);
+ drbd_al_complete_io(device, &peer_req->i);
+ put_ldev(device);
+ }
- if (be64_to_cpu(p->block_id) == ID_OUT_OF_SYNC)
- drbd_ov_out_of_sync_found(peer_device, sector, size);
- else
- ov_out_of_sync_print(peer_device);
+ peer_req->send_oos_pending = drbd_calculate_send_oos_pending(device, 0);
+ any_send_oos_pending |= peer_req->send_oos_pending;
+ if (peer_req->send_oos_pending) {
+ peer_req->flags &= ~EE_ON_RECV_ORDER;
+ peer_req->flags |= EE_ON_SEND_OOS;
+ } else {
+ drbd_free_peer_req(peer_req); /* list_del() because EE_ON_RECV_ORDER */
+ }
+ }
- if (!get_ldev(device))
- return 0;
+ drbd_queue_send_out_of_sync(connection, &work_list, any_send_oos_pending);
+}
- drbd_rs_complete_io(device, sector);
- dec_rs_pending(peer_device);
+static void cleanup_peer_ack_list(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_ack *peer_ack, *tmp;
+ struct drbd_request *req;
+ int idx = connection->peer_node_id;
+ u64 node_id_mask = NODE_MASK(idx);
+
+ spin_lock_irq(&resource->peer_ack_lock);
+ list_for_each_entry_safe(peer_ack, tmp, &resource->peer_ack_list, list) {
+ if (!(peer_ack->queued_mask & node_id_mask))
+ continue;
+ peer_ack->queued_mask &= ~node_id_mask;
+ drbd_destroy_peer_ack_if_done(peer_ack);
+ }
+ req = resource->peer_ack_req;
+ if (req)
+ req->net_rq_state[idx] &= ~RQ_NET_SENT;
+ spin_unlock_irq(&resource->peer_ack_lock);
+}
- --device->ov_left;
+int drbd_flush_ack_wf(struct drbd_work *w, int unused)
+{
+ struct drbd_connection *connection =
+ container_of(w, struct drbd_connection, flush_ack_work);
+ int primary_node_id;
- /* let's advance progress step marks only for every other megabyte */
- if ((device->ov_left & 0x200) == 0x200)
- drbd_advance_rs_marks(peer_device, device->ov_left);
+ for (primary_node_id = 0; primary_node_id < DRBD_PEERS_MAX; primary_node_id++) {
+ u64 flush_sequence;
- if (device->ov_left == 0) {
- dw = kmalloc_obj(*dw, GFP_NOIO);
- if (dw) {
- dw->w.cb = w_ov_finished;
- dw->device = device;
- drbd_queue_work(&peer_device->connection->sender_work, &dw->w);
- } else {
- drbd_err(device, "kmalloc(dw) failed.");
- ov_out_of_sync_print(peer_device);
- drbd_resync_finished(peer_device);
- }
+ spin_lock_irq(&connection->flush_ack_lock);
+ flush_sequence = connection->flush_ack_sequence[primary_node_id];
+ connection->flush_ack_sequence[primary_node_id] = 0;
+ spin_unlock_irq(&connection->flush_ack_lock);
+
+ if (flush_sequence) /* Active flushes use non-zero sequence numbers */
+ drbd_send_flush_requests_ack(connection, flush_sequence, primary_node_id);
}
- put_ldev(device);
+
return 0;
}
-static int got_skip(struct drbd_connection *connection, struct packet_info *pi)
+static int got_flush_forward(struct drbd_connection *connection, struct packet_info *pi)
{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_connection *initiator_connection;
+ struct p_flush_forward *p = pi->data;
+ u64 flush_sequence = be64_to_cpu(p->flush_sequence);
+ int initiator_node_id = be32_to_cpu(p->initiator_node_id);
+
+ rcu_read_lock();
+ initiator_connection = drbd_connection_by_node_id(resource, initiator_node_id);
+ if (!initiator_connection) {
+ rcu_read_unlock();
+ return 0;
+ }
+
+ queue_flush_ack(initiator_connection, connection->peer_node_id, flush_sequence);
+ rcu_read_unlock();
return 0;
}
-struct meta_sock_cmd {
- size_t pkt_size;
- int (*fn)(struct drbd_connection *connection, struct packet_info *);
-};
-
-static void set_rcvtimeo(struct drbd_connection *connection, bool ping_timeout)
+static void set_rcvtimeo(struct drbd_connection *connection, enum rcv_timeou_kind kind)
{
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
+ bool ping_timeout = kind == PING_TIMEOUT;
+ struct drbd_net_conf *nc;
long t;
- struct net_conf *nc;
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
+ nc = rcu_dereference(transport->net_conf);
t = ping_timeout ? nc->ping_timeo : nc->ping_int;
rcu_read_unlock();
@@ -5741,202 +14628,267 @@ static void set_rcvtimeo(struct drbd_connection *connection, bool ping_timeout)
if (ping_timeout)
t /= 10;
- connection->meta.socket->sk->sk_rcvtimeo = t;
+ tr_ops->set_rcvtimeo(transport, CONTROL_STREAM, t);
}
-static void set_ping_timeout(struct drbd_connection *connection)
+void drbd_send_ping_wf(struct work_struct *ws)
{
- set_rcvtimeo(connection, 1);
-}
+ struct drbd_connection *connection =
+ container_of(ws, struct drbd_connection, send_ping_work);
+ int err;
-static void set_idle_timeout(struct drbd_connection *connection)
-{
- set_rcvtimeo(connection, 0);
+ set_rcvtimeo(connection, PING_TIMEOUT);
+ set_bit(PING_TIMEOUT_ACTIVE, &connection->flags);
+ err = drbd_send_ping(connection);
+ if (err)
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ kref_put(&connection->kref, drbd_destroy_connection);
}
+struct meta_sock_cmd {
+ size_t pkt_size;
+ int (*fn)(struct drbd_connection *connection, struct packet_info *);
+};
+
static struct meta_sock_cmd ack_receiver_tbl[] = {
- [P_PING] = { 0, got_Ping },
- [P_PING_ACK] = { 0, got_PingAck },
- [P_RECV_ACK] = { sizeof(struct p_block_ack), got_BlockAck },
- [P_WRITE_ACK] = { sizeof(struct p_block_ack), got_BlockAck },
- [P_RS_WRITE_ACK] = { sizeof(struct p_block_ack), got_BlockAck },
- [P_SUPERSEDED] = { sizeof(struct p_block_ack), got_BlockAck },
- [P_NEG_ACK] = { sizeof(struct p_block_ack), got_NegAck },
- [P_NEG_DREPLY] = { sizeof(struct p_block_ack), got_NegDReply },
- [P_NEG_RS_DREPLY] = { sizeof(struct p_block_ack), got_NegRSDReply },
- [P_OV_RESULT] = { sizeof(struct p_block_ack), got_OVResult },
- [P_BARRIER_ACK] = { sizeof(struct p_barrier_ack), got_BarrierAck },
- [P_STATE_CHG_REPLY] = { sizeof(struct p_req_state_reply), got_RqSReply },
- [P_RS_IS_IN_SYNC] = { sizeof(struct p_block_ack), got_IsInSync },
- [P_DELAY_PROBE] = { sizeof(struct p_delay_probe93), got_skip },
- [P_RS_CANCEL] = { sizeof(struct p_block_ack), got_NegRSDReply },
- [P_CONN_ST_CHG_REPLY]={ sizeof(struct p_req_state_reply), got_conn_RqSReply },
- [P_RETRY_WRITE] = { sizeof(struct p_block_ack), got_BlockAck },
+ [P_PING] = { 0, got_Ping },
+ [P_PING_ACK] = { 0, got_PingAck },
+ [P_RECV_ACK] = { sizeof(struct p_block_ack), got_BlockAck },
+ [P_WRITE_ACK] = { sizeof(struct p_block_ack), got_BlockAck },
+ [P_WRITE_ACK_IN_SYNC] = { sizeof(struct p_block_ack), got_BlockAck },
+ [P_NEG_ACK] = { sizeof(struct p_block_ack), got_NegAck },
+ [P_NEG_DREPLY] = { sizeof(struct p_block_ack), got_NegDReply },
+ [P_NEG_RS_DREPLY] = { sizeof(struct p_block_ack), got_NegRSDReply },
+ [P_RS_WRITE_ACK] = { sizeof(struct p_block_ack), got_RSWriteAck },
+ [P_RS_NEG_ACK] = { sizeof(struct p_block_ack), got_RSWriteAck },
+ [P_OV_RESULT] = { sizeof(struct p_block_ack), got_OVResult },
+ [P_OV_RESULT_ID] = { sizeof(struct p_ov_result), got_OVResult },
+ [P_BARRIER_ACK] = { sizeof(struct p_barrier_ack), got_BarrierAck },
+ [P_CONFIRM_STABLE] = { sizeof(struct p_confirm_stable), got_confirm_stable },
+ [P_STATE_CHG_REPLY] = { sizeof(struct p_req_state_reply), got_RqSReply },
+ [P_RS_IS_IN_SYNC] = { sizeof(struct p_block_ack), got_IsInSync },
+ [P_DELAY_PROBE] = { sizeof(struct p_delay_probe93), got_skip },
+ [P_RS_CANCEL] = { sizeof(struct p_block_ack), got_NegRSDReply },
+ [P_RS_CANCEL_AHEAD] = { sizeof(struct p_block_ack), got_NegRSDReply },
+ [P_CONN_ST_CHG_REPLY] = { sizeof(struct p_req_state_reply), got_RqSReply },
+ [P_PEER_ACK] = { sizeof(struct p_peer_ack), got_peer_ack },
+ [P_PEERS_IN_SYNC] = { sizeof(struct p_peer_block_desc), got_peers_in_sync },
+ [P_TWOPC_YES] = { sizeof(struct p_twopc_reply), got_twopc_reply },
+ [P_TWOPC_NO] = { sizeof(struct p_twopc_reply), got_twopc_reply },
+ [P_TWOPC_RETRY] = { sizeof(struct p_twopc_reply), got_twopc_reply },
+ [P_FLUSH_FORWARD] = { sizeof(struct p_flush_forward), got_flush_forward },
+ [P_RS_DAGTAG_REACHED] = { sizeof(struct p_block_ack), got_RSDagtagReached },
+ [P_RS_DAGTAG_UNREACHABLE] = { sizeof(struct p_block_ack), got_RSDagtagUnreachable },
+ [P_RETRY_WRITE] = { sizeof(struct p_block_ack), got_RetryWrite },
};
-int drbd_ack_receiver(struct drbd_thread *thi)
+static void fillup_buffer_from(struct drbd_mutable_buffer *to_fill, unsigned int need, struct drbd_const_buffer *pool)
{
- struct drbd_connection *connection = thi->connection;
- struct meta_sock_cmd *cmd = NULL;
+ if (to_fill->avail < need) {
+ unsigned int missing = min(need - to_fill->avail, pool->avail);
+
+ memcpy(to_fill->buffer + to_fill->avail, pool->buffer, missing);
+ pool->buffer += missing;
+ pool->avail -= missing;
+ to_fill->avail += missing;
+ }
+}
+
+static int decode_meta_cmd(struct drbd_connection *connection, const u8 *pos, struct packet_info *pi)
+{
+ int header_version, payload_size;
+ struct meta_sock_cmd *cmd;
+
+ /*
+ * A ping packet (via the control stream) can overtake the
+ * feature packet. We might get it with a different header version
+ * than expected since we will agree on the protocol version
+ * by receiving the feature packet.
+ */
+ header_version = __decode_header(pos, pi);
+ if (header_version < 0) {
+ drbd_err(connection, "Wrong magic value 0x%08x in protocol version %d [control]\n",
+ be32_to_cpu(*(__be32 *)pos), header_version);
+ return -EINVAL;
+ }
+
+ if (pi->cmd >= ARRAY_SIZE(ack_receiver_tbl)) {
+ drbd_err(connection, "Unexpected meta packet %s (0x%04x)\n",
+ drbd_packet_name(pi->cmd), pi->cmd);
+ return -ENOENT;
+ }
+
+ cmd = &ack_receiver_tbl[pi->cmd];
+ payload_size = cmd->pkt_size;
+ if (pi->size != payload_size) {
+ drbd_err(connection, "Wrong packet size on meta (c: %d, l: %d)\n",
+ pi->cmd, pi->size);
+ return -EINVAL;
+ }
+
+ return payload_size;
+}
+
+static int process_previous_part(struct drbd_connection *connection, struct drbd_const_buffer *pool)
+{
+ struct drbd_mutable_buffer *buffer = &connection->reassemble_buffer;
+ int payload_size, packet_size;
+ unsigned int header_size;
struct packet_info pi;
- unsigned long pre_recv_jif;
- int rv;
- void *buf = connection->meta.rbuf;
- int received = 0;
- unsigned int header_size = drbd_header_size(connection);
- int expect = header_size;
- bool ping_timeout_active = false;
+ int err;
+
+ fillup_buffer_from(buffer, sizeof(u32), pool);
+ if (buffer->avail < sizeof(u32))
+ return 0;
- sched_set_fifo_low(current);
+ header_size = decode_header_size(buffer->buffer);
+ fillup_buffer_from(buffer, header_size, pool);
+ if (buffer->avail < header_size)
+ return 0;
- while (get_t_state(thi) == RUNNING) {
- drbd_thread_current_set_cpu(thi);
+ payload_size = decode_meta_cmd(connection, buffer->buffer, &pi);
+ if (payload_size < 0)
+ return payload_size;
- if (test_and_clear_bit(SEND_PING, &connection->flags)) {
- if (drbd_send_ping(connection)) {
- drbd_err(connection, "drbd_send_ping has failed\n");
- goto reconnect;
- }
- set_ping_timeout(connection);
- ping_timeout_active = true;
- }
-
- pre_recv_jif = jiffies;
- rv = drbd_recv_short(connection->meta.socket, buf, expect-received, 0);
-
- /* Note:
- * -EINTR (on meta) we got a signal
- * -EAGAIN (on meta) rcvtimeo expired
- * -ECONNRESET other side closed the connection
- * -ERESTARTSYS (on data) we got a signal
- * rv < 0 other than above: unexpected error!
- * rv == expected: full header or command
- * rv < expected: "woken" by signal during receive
- * rv == 0 : "connection shut down by peer"
- */
- if (likely(rv > 0)) {
- received += rv;
- buf += rv;
- } else if (rv == 0) {
- if (test_bit(DISCONNECT_SENT, &connection->flags)) {
- long t;
- rcu_read_lock();
- t = rcu_dereference(connection->net_conf)->ping_timeo * HZ/10;
- rcu_read_unlock();
-
- t = wait_event_timeout(connection->ping_wait,
- connection->cstate < C_WF_REPORT_PARAMS,
- t);
- if (t)
- break;
- }
- drbd_err(connection, "meta connection shut down by peer.\n");
+ packet_size = header_size + payload_size;
+ fillup_buffer_from(buffer, packet_size, pool);
+ if (buffer->avail < packet_size)
+ return 0;
+
+ err = ack_receiver_tbl[pi.cmd].fn(connection, &pi);
+ connection->reassemble_buffer.avail = 0;
+ return err;
+}
+
+void drbd_control_data_ready(struct drbd_transport *transport, struct drbd_const_buffer *pool)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
+ unsigned int header_size;
+ int err;
+
+ if (connection->cstate[NOW] < C_TEAR_DOWN)
+ return;
+
+ if (connection->reassemble_buffer.avail) {
+ err = process_previous_part(connection, pool);
+ if (err < 0)
goto reconnect;
- } else if (rv == -EAGAIN) {
- /* If the data socket received something meanwhile,
- * that is good enough: peer is still alive. */
- if (time_after(connection->last_received, pre_recv_jif))
- continue;
- if (ping_timeout_active) {
- drbd_err(connection, "PingAck did not arrive in time.\n");
- goto reconnect;
- }
- set_bit(SEND_PING, &connection->flags);
- continue;
- } else if (rv == -EINTR) {
- /* maybe drbd_thread_stop(): the while condition will notice.
- * maybe woken for send_ping: we'll send a ping above,
- * and change the rcvtimeo */
- flush_signals(current);
- continue;
- } else {
- drbd_err(connection, "sock_recvmsg returned %d\n", rv);
+ }
+
+ while (pool->avail >= sizeof(u32)) {
+ int payload_size, packet_size;
+ struct packet_info pi;
+
+ header_size = decode_header_size(pool->buffer);
+ if (header_size > pool->avail)
+ goto keep_part;
+
+ payload_size = decode_meta_cmd(connection, pool->buffer, &pi);
+ if (payload_size < 0) {
+ err = payload_size;
goto reconnect;
}
- if (received == expect && cmd == NULL) {
- if (decode_header(connection, connection->meta.rbuf, &pi))
- goto reconnect;
- cmd = &ack_receiver_tbl[pi.cmd];
- if (pi.cmd >= ARRAY_SIZE(ack_receiver_tbl) || !cmd->fn) {
- drbd_err(connection, "Unexpected meta packet %s (0x%04x)\n",
- drbd_packet_name(pi.cmd), pi.cmd);
- goto disconnect;
- }
- expect = header_size + cmd->pkt_size;
- if (pi.size != expect - header_size) {
- drbd_err(connection, "Wrong packet size on meta (c: %d, l: %d)\n",
- pi.cmd, pi.size);
- goto reconnect;
- }
- }
- if (received == expect) {
- bool err;
+ packet_size = header_size + payload_size;
+ if (packet_size > pool->avail)
+ goto keep_part;
- err = cmd->fn(connection, &pi);
- if (err) {
- drbd_err(connection, "%ps failed\n", cmd->fn);
- goto reconnect;
- }
+ err = ack_receiver_tbl[pi.cmd].fn(connection, &pi);
+ if (err)
+ goto reconnect;
- connection->last_received = jiffies;
+ pool->buffer += packet_size;
+ pool->avail -= packet_size;
+ }
+ if (pool->avail > 0) {
+keep_part:
+ memcpy(connection->reassemble_buffer.buffer, pool->buffer, pool->avail);
+ connection->reassemble_buffer.avail = pool->avail;
+ pool->avail = 0;
+ }
+ return;
- if (cmd == &ack_receiver_tbl[P_PING_ACK]) {
- set_idle_timeout(connection);
- ping_timeout_active = false;
- }
+reconnect:
+ change_cstate(connection, err == -EPROTO ? C_PROTOCOL_ERROR : C_NETWORK_FAILURE, CS_HARD);
+}
+EXPORT_SYMBOL(drbd_control_data_ready);
+
+void drbd_control_event(struct drbd_transport *transport, enum drbd_tr_event event)
+{
+ struct drbd_connection *connection =
+ container_of(transport, struct drbd_connection, transport);
- buf = connection->meta.rbuf;
- received = 0;
- expect = header_size;
- cmd = NULL;
+ if (event == TIMEOUT) {
+ if (!test_bit(PING_TIMEOUT_ACTIVE, &connection->flags)) {
+ drbd_queue_ping(connection);
+ return;
+ } else {
+ if (connection->cstate[NOW] == C_CONNECTED)
+ drbd_warn(connection, "PingAck did not arrive in time.\n");
}
+ } else /* event == CLOSED_BY_PEER */ {
+ if (connection->cstate[NOW] == C_CONNECTED && disconnect_expected(connection))
+ return;
+ drbd_warn(connection, "meta connection shut down by peer.\n");
}
- if (0) {
-reconnect:
- conn_request_state(connection, NS(conn, C_NETWORK_FAILURE), CS_HARD);
- conn_md_sync(connection);
- }
- if (0) {
-disconnect:
- conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_HARD);
- }
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+}
+EXPORT_SYMBOL(drbd_control_event);
- drbd_info(connection, "ack_receiver terminated\n");
+static bool disconnect_expected(struct drbd_connection *connection)
+{
+ struct drbd_resource *resource = connection->resource;
+ bool expect_disconnect;
- return 0;
+ /* We are reacting to a not-committed state change! The disconnect might
+ get aborted. This is not a problem worth much more complex code.
+ In the unlikely case, it happens that a two-phase-commit of a graceful
+ disconnect gets aborted and the control connection breaks in exactly
+ this time window, we will notice it as soon as sending something on the
+ control stream. */
+ read_lock_irq(&resource->state_rwlock);
+ expect_disconnect = resource->remote_state_change &&
+ drbd_twopc_between_peer_and_me(connection) &&
+ resource->twopc_reply.is_disconnect;
+ read_unlock_irq(&resource->state_rwlock);
+ return expect_disconnect;
}
void drbd_send_acks_wf(struct work_struct *ws)
{
- struct drbd_peer_device *peer_device =
- container_of(ws, struct drbd_peer_device, send_acks_work);
- struct drbd_connection *connection = peer_device->connection;
- struct drbd_device *device = peer_device->device;
- struct net_conf *nc;
+ struct drbd_connection *connection =
+ container_of(ws, struct drbd_connection, send_acks_work);
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_net_conf *nc;
int tcp_cork, err;
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
+ nc = rcu_dereference(transport->net_conf);
tcp_cork = nc->tcp_cork;
rcu_read_unlock();
+ /* TODO: conditionally cork; it may hurt latency if we cork without
+ much to send */
if (tcp_cork)
- tcp_sock_set_cork(connection->meta.socket->sk, true);
+ drbd_cork(connection, CONTROL_STREAM);
+ err = drbd_finish_peer_reqs(connection);
- err = drbd_finish_peer_reqs(device);
- kref_put(&device->kref, drbd_destroy_device);
- /* get is in drbd_endio_write_sec_final(). That is necessary to keep the
- struct work_struct send_acks_work alive, which is in the peer_device object */
+ /* but unconditionally uncork unless disabled */
+ if (err)
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
+ else if (tcp_cork)
+ drbd_uncork(connection, CONTROL_STREAM);
- if (err) {
- conn_request_state(connection, NS(conn, C_NETWORK_FAILURE), CS_HARD);
- return;
- }
+}
- if (tcp_cork)
- tcp_sock_set_cork(connection->meta.socket->sk, false);
+void drbd_send_peer_ack_wf(struct work_struct *ws)
+{
+ struct drbd_connection *connection =
+ container_of(ws, struct drbd_connection, peer_ack_work);
- return;
+ if (process_peer_ack_list(connection))
+ change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
}
+
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 16/20] drbd: rework the netlink core for DRBD 9
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (14 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 15/20] drbd: rework receiver for DRBD 9 transport and multi-peer protocol Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 17/20] drbd: serve the "drbd" genl family at version 1 for existing userspace Christoph Böhmwalder
` (3 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Andreas Gruenbacher,
Joel Colledge, Moritz Tanner, Roland Kammerer
Rework the generic netlink administration core to DRBD 9's multi-peer
topology model, and detach it from any one wire format: drbd_nl.c
works on the neutral config, info and statistics structs declared in
drbd_nl_types.h and asks a struct drbd_nl_dialect for everything that
touches the wire (parsing attributes into those structs, emitting
dump entries and notifications, reporting outcomes). The dialects
follow in the next two commits; this one is the core they plug into.
Connections are identified by peer node ID rather than address pairs,
and the admin API gains operations for creating and removing peer
connections and managing network paths within each connection. Add
per-peer-device configuration, metadata slot reclamation and resource
renaming as new administrative commands.
Lift role promotion to resource scope and use quorum-aware logic with
auto-promote timeout, replacing the per-device state machine. Disk
attach and detach gain support for per-peer bitmap slot allocation,
DAX/PMEM-backed metadata and variable bitmap block sizes. Resize and
other multi-peer operations use the new transactional state change
API to coordinate across all peers atomically.
Administrative commands now additionally require CAP_SYS_ADMIN on top
of the CAP_NET_ADMIN that generic netlink already enforces for
GENL_ADMIN_PERM commands, and the global genl_lock() serialization is
replaced by parallel_ops with fine-grained locking. Notifications
cover path-level state and detailed per-peer resync progress.
The default-value setters for the neutral structs are generated into
drbd_nl_defaults.c so that every dialect links them. drbd.h and
drbd_limits.h in the UAPI gain the DRBD 9 constants and limits; the
existing "drbd" family header, drbd_genl.h, is left exactly as it is,
because that family keeps serving version 1.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Moritz Tanner <moritz.tanner@linbit.com>
Signed-off-by: Moritz Tanner <moritz.tanner@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Makefile | 1 +
drivers/block/drbd/drbd_nl.c | 10232 ++++++++++++++++--------
drivers/block/drbd/drbd_nl.h | 400 +
drivers/block/drbd/drbd_nl_defaults.c | 168 +
drivers/block/drbd/drbd_nl_types.h | 402 +
include/uapi/linux/drbd.h | 305 +-
include/uapi/linux/drbd_limits.h | 138 +-
7 files changed, 8207 insertions(+), 3439 deletions(-)
create mode 100644 drivers/block/drbd/drbd_nl.h
create mode 100644 drivers/block/drbd/drbd_nl_defaults.c
create mode 100644 drivers/block/drbd/drbd_nl_types.h
diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index f184b99d6a93..f430536636c2 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -4,6 +4,7 @@ drbd-y += drbd_sender.o drbd_receiver.o drbd_req.o drbd_actlog.o
drbd-y += drbd_main.o drbd_strings.o drbd_nl.o
drbd-y += drbd_interval.o drbd_state.o
drbd-y += drbd_nl_gen.o
+drbd-y += drbd_nl_defaults.o
drbd-y += drbd_transport.o
drbd-$(CONFIG_DEV_DAX_PMEM) += drbd_dax_pmem.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
diff --git a/drivers/block/drbd/drbd_nl.c b/drivers/block/drbd/drbd_nl.c
index 3511a943e4b1..773345f08c1e 100644
--- a/drivers/block/drbd/drbd_nl.c
+++ b/drivers/block/drbd/drbd_nl.c
@@ -15,555 +15,635 @@
#include <linux/fs.h>
#include <linux/file.h>
#include <linux/slab.h>
-#include <linux/blkpg.h>
#include <linux/cpumask.h>
+#include <linux/random.h>
#include "drbd_int.h"
#include "drbd_protocol.h"
#include "drbd_req.h"
#include "drbd_state_change.h"
-#include <linux/unaligned.h>
+#include "drbd_debugfs.h"
+#include "drbd_transport.h"
+#include "drbd_dax_pmem.h"
#include <linux/drbd_limits.h>
#include <linux/kthread.h>
+#include <linux/security.h>
+#include <linux/netlink.h>
+#include <net/net_namespace.h>
-#include <net/genetlink.h>
+#include "drbd_meta_data.h"
+#include "drbd_legacy_84.h"
-#include "drbd_nl_gen.h"
-
-static int drbd_genl_multicast_events(struct sk_buff *skb, gfp_t flags)
-{
- return genlmsg_multicast(&drbd_nl_family, skb, 0,
- DRBD_NLGRP_EVENTS, flags);
-}
-
-static atomic_t drbd_genl_seq = ATOMIC_INIT(2); /* two. */
-static atomic_t notify_genl_seq = ATOMIC_INIT(2); /* two. */
+atomic_t drbd_genl_seq = ATOMIC_INIT(2); /* two. */
DEFINE_MUTEX(notification_mutex);
/* used bdev_open_by_path, to claim our meta data device(s) */
static char *drbd_m_holder = "Hands off! this is DRBD's meta data device.";
-static void drbd_adm_send_reply(struct sk_buff *skb, struct genl_info *info)
-{
- genlmsg_end(skb, genlmsg_data(nlmsg_data(nlmsg_hdr(skb))));
- if (genlmsg_reply(skb, info))
- pr_err("error sending genl reply\n");
-}
-
-/* Used on a fresh "drbd_adm_prepare"d reply_skb, this cannot fail: The only
- * reason it could fail was no space in skb, and there are 4k available. */
-static int drbd_msg_put_info(struct sk_buff *skb, const char *info)
-{
- struct nlattr *nla;
- int err = -EMSGSIZE;
-
- if (!info || !info[0])
- return 0;
-
- nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_REPLY);
- if (!nla)
- return err;
-
- err = nla_put_string(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, info);
- if (err) {
- nla_nest_cancel(skb, nla);
- return err;
- } else
- nla_nest_end(skb, nla);
- return 0;
-}
-
__printf(2, 3)
-static int drbd_msg_sprintf_info(struct sk_buff *skb, const char *fmt, ...)
+void drbd_adm_msg(struct drbd_adm_ctx *ctx, const char *fmt, ...)
{
+ unsigned int room = sizeof(ctx->msg) - ctx->msg_len;
va_list args;
- struct nlattr *nla, *txt;
- int err = -EMSGSIZE;
int len;
- nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_REPLY);
- if (!nla)
- return err;
-
- txt = nla_reserve(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, 256);
- if (!txt) {
- nla_nest_cancel(skb, nla);
- return err;
- }
+ /* not even room for an empty string plus its NUL */
+ if (room < 2)
+ return;
va_start(args, fmt);
- len = vscnprintf(nla_data(txt), 256, fmt, args);
+ len = vsnprintf(ctx->msg + ctx->msg_len,
+ min_t(unsigned int, room, DRBD_ADM_MSG_MAX), fmt, args);
va_end(args);
+ /* empty text is ignored, as the legacy drbd_msg_put_info did */
+ if (len <= 0)
+ return;
+ /* per-message cap, as the legacy 256-byte reserve */
+ if (len >= DRBD_ADM_MSG_MAX)
+ len = DRBD_ADM_MSG_MAX - 1;
+ if (len + 1 > (int)room) {
+ /*
+ * Does not fit into what is left of the buffer: drop the
+ * whole message, just like the -EMSGSIZE of an over-full
+ * reply skb dropped it before.
+ */
+ ctx->msg[ctx->msg_len] = '\0';
+ return;
+ }
+ ctx->msg_len += len + 1;
+}
- /* maybe: retry with larger reserve, if truncated */
- txt->nla_len = nla_attr_size(len+1);
- nlmsg_trim(skb, (char*)txt + NLA_ALIGN(txt->nla_len));
- nla_nest_end(skb, nla);
+static struct drbd_path *first_path(struct drbd_connection *connection)
+{
+ /* Ideally this function is removed at a later point in time.
+ It was introduced when replacing the single address pair
+ with a list of address pairs (or paths). */
- return 0;
+ return list_first_or_null_rcu(&connection->transport.paths, struct drbd_path, list);
}
-/* Flags for drbd_adm_prepare() */
-#define DRBD_ADM_NEED_MINOR (1 << 0)
-#define DRBD_ADM_NEED_RESOURCE (1 << 1)
-#define DRBD_ADM_NEED_CONNECTION (1 << 2)
-
-/* Per-command flags for drbd_pre_doit() */
-static const unsigned int drbd_genl_cmd_flags[] = {
- [DRBD_ADM_GET_STATUS] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_NEW_MINOR] = DRBD_ADM_NEED_RESOURCE,
- [DRBD_ADM_DEL_MINOR] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_NEW_RESOURCE] = 0,
- [DRBD_ADM_DEL_RESOURCE] = DRBD_ADM_NEED_RESOURCE,
- [DRBD_ADM_RESOURCE_OPTS] = DRBD_ADM_NEED_RESOURCE,
- [DRBD_ADM_CONNECT] = DRBD_ADM_NEED_RESOURCE,
- [DRBD_ADM_CHG_NET_OPTS] = DRBD_ADM_NEED_CONNECTION,
- [DRBD_ADM_DISCONNECT] = DRBD_ADM_NEED_CONNECTION,
- [DRBD_ADM_ATTACH] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_CHG_DISK_OPTS] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_RESIZE] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_PRIMARY] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_SECONDARY] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_NEW_C_UUID] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_START_OV] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_DETACH] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_INVALIDATE] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_INVAL_PEER] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_PAUSE_SYNC] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_RESUME_SYNC] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_SUSPEND_IO] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_RESUME_IO] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_OUTDATE] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_GET_TIMEOUT_TYPE] = DRBD_ADM_NEED_MINOR,
- [DRBD_ADM_DOWN] = DRBD_ADM_NEED_RESOURCE,
-};
+/*
+ * The netlink dialects this module serves. Registered at module init
+ * and never removed afterwards, so no locking is needed to walk them.
+ */
+static const struct drbd_nl_dialect *drbd_nl_dialects[4];
+static unsigned int drbd_nl_n_dialects;
-/* Detect attempts to change invariant attributes in a _change_ handler. */
-#define has_invariant(ntb, attr) \
-({ \
- bool __found = !!(ntb)[attr]; \
- if (__found) \
- pr_info("must not change invariant attr: %s\n", #attr); \
- __found; \
-})
+int drbd_nl_register_dialect(const struct drbd_nl_dialect *dialect)
+{
+ if (drbd_nl_n_dialects >= ARRAY_SIZE(drbd_nl_dialects))
+ return -ENOSPC;
+ drbd_nl_dialects[drbd_nl_n_dialects++] = dialect;
+ return 0;
+}
/*
- * At this point, we still rely on the global genl_lock().
- * If we want to avoid that, and allow "genl_family.parallel_ops", we may need
- * to add additional synchronization against object destruction/modification.
+ * Resolve the objects the command refers to. On success the members of
+ * adm_ctx are valid and NO_ERROR is returned; otherwise the failure is
+ * also recorded in adm_ctx->result.
*/
-static int drbd_adm_prepare(struct drbd_config_context *adm_ctx,
- struct sk_buff *skb, struct genl_info *info, unsigned flags)
+int drbd_adm_ctx_resolve(struct drbd_adm_ctx *adm_ctx, unsigned int flags)
{
- struct drbd_genlmsghdr *d_in = genl_info_userhdr(info);
- const u8 cmd = info->genlhdr->cmd;
int err;
- /* genl_rcv_msg only checks for CAP_NET_ADMIN on "GENL_ADMIN_PERM" :( */
- if (cmd != DRBD_ADM_GET_STATUS && !capable(CAP_NET_ADMIN))
- return -EPERM;
-
- adm_ctx->reply_skb = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL);
- if (!adm_ctx->reply_skb) {
- err = -ENOMEM;
- goto fail;
- }
-
- adm_ctx->reply_dh = genlmsg_put_reply(adm_ctx->reply_skb,
- info, &drbd_nl_family, 0, cmd);
- /* put of a few bytes into a fresh skb of >= 4k will always succeed.
- * but anyways */
- if (!adm_ctx->reply_dh) {
- err = -ENOMEM;
- goto fail;
- }
-
- adm_ctx->reply_dh->minor = d_in->minor;
- adm_ctx->reply_dh->ret_code = NO_ERROR;
-
- adm_ctx->volume = VOLUME_UNSPECIFIED;
- if (info->attrs[DRBD_NLA_CFG_CONTEXT]) {
- struct nlattr **ntb;
- struct nlattr *nla;
-
- /* parse and validate, get nested attribute table */
- err = drbd_cfg_context_ntb_from_attrs(&ntb, info);
- if (err)
- goto fail;
-
- /* It was present, and valid,
- * copy it over to the reply skb. */
- err = nla_put_nohdr(adm_ctx->reply_skb,
- info->attrs[DRBD_NLA_CFG_CONTEXT]->nla_len,
- info->attrs[DRBD_NLA_CFG_CONTEXT]);
- if (err) {
- kfree(ntb);
- goto fail;
- }
+ if (flags & DRBD_ADM_NEED_PEER_DEVICE)
+ flags |= DRBD_ADM_NEED_CONNECTION;
+ if (flags & DRBD_ADM_NEED_CONNECTION)
+ flags |= DRBD_ADM_NEED_PEER_NODE;
+ if (flags & DRBD_ADM_NEED_PEER_NODE)
+ flags |= DRBD_ADM_NEED_RESOURCE;
- /* and assign stuff to the adm_ctx */
- nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME];
- if (nla)
- adm_ctx->volume = nla_get_u32(nla);
- nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME];
- if (nla)
- adm_ctx->resource_name = nla_data(nla);
- adm_ctx->my_addr = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR];
- adm_ctx->peer_addr = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR];
- kfree(ntb);
- if ((adm_ctx->my_addr &&
- nla_len(adm_ctx->my_addr) > sizeof(adm_ctx->connection->my_addr)) ||
- (adm_ctx->peer_addr &&
- nla_len(adm_ctx->peer_addr) > sizeof(adm_ctx->connection->peer_addr))) {
- err = -EINVAL;
- goto fail;
- }
+ if (adm_ctx->resource_name) {
+ adm_ctx->resource = drbd_find_resource(adm_ctx->resource_name);
}
- adm_ctx->minor = d_in->minor;
- adm_ctx->device = minor_to_device(d_in->minor);
-
- /* We are protected by the global genl_lock().
- * But we may explicitly drop it/retake it in drbd_nl_set_role(),
- * so make sure this object stays around. */
- if (adm_ctx->device)
+ rcu_read_lock();
+ adm_ctx->device = minor_to_device(adm_ctx->minor);
+ if (adm_ctx->device) {
kref_get(&adm_ctx->device->kref);
-
- if (adm_ctx->resource_name) {
- adm_ctx->resource = drbd_find_resource(adm_ctx->resource_name);
}
+ rcu_read_unlock();
if (!adm_ctx->device && (flags & DRBD_ADM_NEED_MINOR)) {
- drbd_msg_put_info(adm_ctx->reply_skb, "unknown minor");
- return ERR_MINOR_INVALID;
+ drbd_adm_msg(adm_ctx, "%s", "unknown minor");
+ err = ERR_MINOR_INVALID;
+ goto finish;
}
if (!adm_ctx->resource && (flags & DRBD_ADM_NEED_RESOURCE)) {
- drbd_msg_put_info(adm_ctx->reply_skb, "unknown resource");
+ drbd_adm_msg(adm_ctx, "%s", "unknown resource");
+ err = ERR_INVALID_REQUEST;
if (adm_ctx->resource_name)
- return ERR_RES_NOT_KNOWN;
- return ERR_INVALID_REQUEST;
+ err = ERR_RES_NOT_KNOWN;
+ goto finish;
}
-
- if (flags & DRBD_ADM_NEED_CONNECTION) {
- if (adm_ctx->resource) {
- drbd_msg_put_info(adm_ctx->reply_skb, "no resource name expected");
- return ERR_INVALID_REQUEST;
+ if (adm_ctx->peer_node_id != PEER_NODE_ID_UNSPECIFIED) {
+ /* peer_node_id is unsigned int */
+ if (adm_ctx->peer_node_id >= DRBD_NODE_ID_MAX) {
+ drbd_adm_msg(adm_ctx, "%s", "peer node id out of range");
+ err = ERR_INVALID_REQUEST;
+ goto finish;
}
- if (adm_ctx->device) {
- drbd_msg_put_info(adm_ctx->reply_skb, "no minor number expected");
- return ERR_INVALID_REQUEST;
+ if (!adm_ctx->resource) {
+ drbd_adm_msg(adm_ctx, "%s", "peer node id given without a resource");
+ err = ERR_INVALID_REQUEST;
+ goto finish;
}
- if (adm_ctx->my_addr && adm_ctx->peer_addr)
- adm_ctx->connection = conn_get_by_addrs(nla_data(adm_ctx->my_addr),
- nla_len(adm_ctx->my_addr),
- nla_data(adm_ctx->peer_addr),
- nla_len(adm_ctx->peer_addr));
+ if (adm_ctx->peer_node_id == adm_ctx->resource->res_opts.node_id) {
+ drbd_adm_msg(adm_ctx, "%s", "peer node id cannot be my own node id");
+ err = ERR_INVALID_REQUEST;
+ goto finish;
+ }
+ adm_ctx->connection = drbd_get_connection_by_node_id(adm_ctx->resource, adm_ctx->peer_node_id);
+ } else if (flags & DRBD_ADM_NEED_PEER_NODE) {
+ drbd_adm_msg(adm_ctx, "%s", "peer node id missing");
+ err = ERR_INVALID_REQUEST;
+ goto finish;
+ }
+ if (flags & DRBD_ADM_NEED_CONNECTION) {
if (!adm_ctx->connection) {
- drbd_msg_put_info(adm_ctx->reply_skb, "unknown connection");
- return ERR_INVALID_REQUEST;
+ drbd_adm_msg(adm_ctx, "%s", "unknown connection");
+ err = ERR_INVALID_REQUEST;
+ goto finish;
+ }
+ }
+ if (flags & DRBD_ADM_NEED_PEER_DEVICE) {
+ rcu_read_lock();
+ if (adm_ctx->volume != VOLUME_UNSPECIFIED)
+ adm_ctx->peer_device =
+ idr_find(&adm_ctx->connection->peer_devices,
+ adm_ctx->volume);
+ if (!adm_ctx->peer_device) {
+ drbd_adm_msg(adm_ctx, "%s", "unknown volume");
+ err = ERR_INVALID_REQUEST;
+ rcu_read_unlock();
+ goto finish;
+ }
+ if (!adm_ctx->device) {
+ adm_ctx->device = adm_ctx->peer_device->device;
+ kref_get(&adm_ctx->device->kref);
}
+ rcu_read_unlock();
}
/* some more paranoia, if the request was over-determined */
if (adm_ctx->device && adm_ctx->resource &&
adm_ctx->device->resource != adm_ctx->resource) {
pr_warn("request: minor=%u, resource=%s; but that minor belongs to resource %s\n",
- adm_ctx->minor, adm_ctx->resource->name,
- adm_ctx->device->resource->name);
- drbd_msg_put_info(adm_ctx->reply_skb, "minor exists in different resource");
- return ERR_INVALID_REQUEST;
+ adm_ctx->minor, adm_ctx->resource->name,
+ adm_ctx->device->resource->name);
+ drbd_adm_msg(adm_ctx, "%s", "minor exists in different resource");
+ err = ERR_INVALID_REQUEST;
+ goto finish;
}
if (adm_ctx->device &&
adm_ctx->volume != VOLUME_UNSPECIFIED &&
adm_ctx->volume != adm_ctx->device->vnr) {
pr_warn("request: minor=%u, volume=%u; but that minor is volume %u in %s\n",
- adm_ctx->minor, adm_ctx->volume,
- adm_ctx->device->vnr, adm_ctx->device->resource->name);
- drbd_msg_put_info(adm_ctx->reply_skb, "minor exists as different volume");
- return ERR_INVALID_REQUEST;
+ adm_ctx->minor, adm_ctx->volume,
+ adm_ctx->device->vnr,
+ adm_ctx->device->resource->name);
+ drbd_adm_msg(adm_ctx, "%s", "minor exists as different volume");
+ err = ERR_INVALID_REQUEST;
+ goto finish;
+ }
+ if (adm_ctx->device && adm_ctx->peer_device &&
+ adm_ctx->resource && adm_ctx->resource->name &&
+ adm_ctx->peer_device->device != adm_ctx->device) {
+ drbd_adm_msg(adm_ctx, "%s", "peer_device->device != device");
+ pr_warn("request: minor=%u, resource=%s, volume=%u, peer_node=%u; device != peer_device->device\n",
+ adm_ctx->minor, adm_ctx->resource->name,
+ adm_ctx->device->vnr, adm_ctx->peer_node_id);
+ err = ERR_INVALID_REQUEST;
+ goto finish;
}
/* still, provide adm_ctx->resource always, if possible. */
if (!adm_ctx->resource) {
adm_ctx->resource = adm_ctx->device ? adm_ctx->device->resource
: adm_ctx->connection ? adm_ctx->connection->resource : NULL;
- if (adm_ctx->resource)
+ if (adm_ctx->resource) {
kref_get(&adm_ctx->resource->kref);
+ }
}
-
return NO_ERROR;
-fail:
- nlmsg_free(adm_ctx->reply_skb);
- adm_ctx->reply_skb = NULL;
+finish:
+ adm_ctx->result = err;
return err;
}
-int drbd_pre_doit(const struct genl_split_ops *ops,
- struct sk_buff *skb, struct genl_info *info)
+/* Drop the references drbd_adm_ctx_resolve() acquired. */
+void drbd_adm_ctx_release(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx;
- u8 cmd = info->genlhdr->cmd;
- unsigned int flags;
- int err;
+ if (adm_ctx->device) {
+ kref_put(&adm_ctx->device->kref, drbd_destroy_device);
+ }
+ if (adm_ctx->connection) {
+ kref_put(&adm_ctx->connection->kref, drbd_destroy_connection);
+ }
+ if (adm_ctx->resource) {
+ kref_put(&adm_ctx->resource->kref, drbd_destroy_resource);
+ }
+}
- adm_ctx = kzalloc_obj(*adm_ctx);
- if (!adm_ctx)
- return -ENOMEM;
+static void conn_md_sync(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ int vnr;
- flags = (cmd < ARRAY_SIZE(drbd_genl_cmd_flags))
- ? drbd_genl_cmd_flags[cmd] : 0;
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
- err = drbd_adm_prepare(adm_ctx, skb, info, flags);
- if (err && !adm_ctx->reply_skb) {
- /* Fatal error before reply_skb was allocated. */
- kfree(adm_ctx);
- return err;
+ kref_get(&device->kref);
+ rcu_read_unlock();
+ drbd_md_sync_if_dirty(device);
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
}
- if (err)
- adm_ctx->reply_dh->ret_code = err;
-
- info->user_ptr[0] = adm_ctx;
- return 0;
+ rcu_read_unlock();
}
-void drbd_post_doit(const struct genl_split_ops *ops,
- struct sk_buff *skb, struct genl_info *info)
+/* Try to figure out where we are happy to become primary.
+ This is unsed by the crm-fence-peer mechanism
+*/
+static u64 up_to_date_nodes(struct drbd_device *device, bool op_is_fence)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
+ struct drbd_resource *resource = device->resource;
+ const int my_node_id = resource->res_opts.node_id;
+ u64 mask = NODE_MASK(my_node_id);
- if (!adm_ctx)
- return;
+ if (resource->role[NOW] == R_PRIMARY || op_is_fence) {
+ struct drbd_peer_device *peer_device;
- if (adm_ctx->reply_skb)
- drbd_adm_send_reply(adm_ctx->reply_skb, info);
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_disk_state pdsk = peer_device->disk_state[NOW];
- if (adm_ctx->device) {
- kref_put(&adm_ctx->device->kref, drbd_destroy_device);
- adm_ctx->device = NULL;
- }
- if (adm_ctx->connection) {
- kref_put(&adm_ctx->connection->kref, &drbd_destroy_connection);
- adm_ctx->connection = NULL;
- }
- if (adm_ctx->resource) {
- kref_put(&adm_ctx->resource->kref, drbd_destroy_resource);
- adm_ctx->resource = NULL;
+ if (pdsk == D_UP_TO_DATE)
+ mask |= NODE_MASK(peer_device->node_id);
+ }
+ rcu_read_unlock();
+ } else if (device->disk_state[NOW] == D_UP_TO_DATE) {
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ int node_id;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_device *peer_device;
+
+ if (node_id == my_node_id)
+ continue;
+
+ peer_device = peer_device_by_node_id(device, node_id);
+
+ if ((peer_device && peer_device->disk_state[NOW] == D_UP_TO_DATE) ||
+ (test_bit(__MDF_NODE_EXISTS, &peer_md[node_id].flags) &&
+ peer_md[node_id].bitmap_uuid == 0))
+ mask |= NODE_MASK(node_id);
+ }
+ } else {
+ mask = 0;
}
- kfree(adm_ctx);
+ return mask;
}
-static void setup_khelper_env(struct drbd_connection *connection, char **envp)
+/* Buffer to construct the environment of a user-space helper in. */
+struct env {
+ char *buffer;
+ int size, pos;
+};
+
+/* Print into an env buffer. */
+static __printf(2, 3) int env_print(struct env *env, const char *fmt, ...)
{
- char *afs;
+ va_list args;
+ int pos, ret;
- /* FIXME: A future version will not allow this case. */
- if (connection->my_addr_len == 0 || connection->peer_addr_len == 0)
- return;
+ pos = env->pos;
+ if (pos < 0)
+ return pos;
+ va_start(args, fmt);
+ ret = vsnprintf(env->buffer + pos, env->size - pos, fmt, args);
+ va_end(args);
+ if (ret < 0) {
+ env->pos = ret;
+ goto out;
+ }
+ if (ret >= env->size - pos) {
+ ret = env->pos = -ENOMEM;
+ goto out;
+ }
+ env->pos += ret + 1;
+out:
+ return ret;
+}
+
+/* Put env variables for an address into an env buffer. */
+static void env_print_address(struct env *env, const char *prefix,
+ struct sockaddr_storage *storage)
+{
+ const char *afs;
- switch (((struct sockaddr *)&connection->peer_addr)->sa_family) {
+ switch (storage->ss_family) {
case AF_INET6:
afs = "ipv6";
- snprintf(envp[4], 60, "DRBD_PEER_ADDRESS=%pI6",
- &((struct sockaddr_in6 *)&connection->peer_addr)->sin6_addr);
+ env_print(env, "%sADDRESS=%pI6", prefix,
+ &((struct sockaddr_in6 *)storage)->sin6_addr);
break;
case AF_INET:
afs = "ipv4";
- snprintf(envp[4], 60, "DRBD_PEER_ADDRESS=%pI4",
- &((struct sockaddr_in *)&connection->peer_addr)->sin_addr);
+ env_print(env, "%sADDRESS=%pI4", prefix,
+ &((struct sockaddr_in *)storage)->sin_addr);
break;
default:
afs = "ssocks";
- snprintf(envp[4], 60, "DRBD_PEER_ADDRESS=%pI4",
- &((struct sockaddr_in *)&connection->peer_addr)->sin_addr);
+ env_print(env, "%sADDRESS=%pI4", prefix,
+ &((struct sockaddr_in *)storage)->sin_addr);
}
- snprintf(envp[3], 20, "DRBD_PEER_AF=%s", afs);
+ env_print(env, "%sAF=%s", prefix, afs);
+}
+
+/* Construct char **envp inside an env buffer. */
+static char **make_envp(struct env *env)
+{
+ char **envp, *b;
+ unsigned int n;
+
+ if (env->pos < 0)
+ return NULL;
+ if (env->pos >= env->size)
+ goto out_nomem;
+ env->buffer[env->pos++] = 0;
+ for (b = env->buffer, n = 1; *b; n++)
+ b = strchr(b, 0) + 1;
+ if (env->size - env->pos < sizeof(envp) * n)
+ goto out_nomem;
+ envp = (char **)(env->buffer + env->size) - n;
+
+ for (b = env->buffer; *b; ) {
+ *envp++ = b;
+ b = strchr(b, 0) + 1;
+ }
+ *envp++ = NULL;
+ return envp - n;
+
+out_nomem:
+ env->pos = -ENOMEM;
+ return NULL;
}
-int drbd_khelper(struct drbd_device *device, char *cmd)
+/* Macro refers to local variables peer_device, device and connection! */
+#define magic_printk(level, fmt, args...) \
+ do { \
+ if (peer_device) \
+ drbd_printk(NOLIMIT, level, peer_device, fmt, args); \
+ else if (device) \
+ drbd_printk(NOLIMIT, level, device, fmt, args); \
+ else \
+ drbd_printk(NOLIMIT, level, connection, fmt, args); \
+ } while (0)
+
+static int drbd_khelper(struct drbd_device *device, struct drbd_connection *connection, char *cmd)
{
- char *envp[] = { "HOME=/",
- "TERM=linux",
- "PATH=/sbin:/usr/sbin:/bin:/usr/bin",
- (char[20]) { }, /* address family */
- (char[60]) { }, /* address */
- NULL };
- char mb[14];
- char *argv[] = {drbd_usermode_helper, cmd, mb, NULL };
- struct drbd_connection *connection = first_peer_device(device)->connection;
- struct sib_info sib;
+ struct drbd_resource *resource = device ? device->resource : connection->resource;
+ char *argv[] = { drbd_usermode_helper, cmd, resource->name, NULL };
+ struct drbd_peer_device *peer_device = NULL;
+ struct env env = { .size = PAGE_SIZE };
+ char **envp;
int ret;
- if (current == connection->worker.task)
- set_bit(CALLBACK_PENDING, &connection->flags);
+enlarge_buffer:
+ env.buffer = (char *)__get_free_pages(GFP_NOIO, get_order(env.size));
+ if (!env.buffer) {
+ ret = -ENOMEM;
+ goto out_err;
+ }
+ env.pos = 0;
+
+ rcu_read_lock();
+ env_print(&env, "HOME=/");
+ env_print(&env, "TERM=linux");
+ env_print(&env, "PATH=/sbin:/usr/sbin:/bin:/usr/bin");
+ if (device) {
+ env_print(&env, "DRBD_MINOR=%u", device->minor);
+ env_print(&env, "DRBD_VOLUME=%u", device->vnr);
+ if (get_ldev(device)) {
+ struct drbd_disk_conf *disk_conf =
+ rcu_dereference(device->ldev->disk_conf);
+ env_print(&env, "DRBD_BACKING_DEV=%s",
+ disk_conf->backing_dev);
+ put_ldev(device);
+ }
+ }
+ if (connection) {
+ struct drbd_path *path;
+
+ rcu_read_lock();
+ path = first_path(connection);
+ if (path) {
+ /* TO BE DELETED */
+ env_print_address(&env, "DRBD_MY_", &path->my_addr);
+ env_print_address(&env, "DRBD_PEER_", &path->peer_addr);
+ }
+ rcu_read_unlock();
+
+ env_print(&env, "DRBD_PEER_NODE_ID=%u", connection->peer_node_id);
+ env_print(&env, "DRBD_CSTATE=%s", drbd_conn_str(connection->cstate[NOW]));
+ }
+ if (connection && !device) {
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ env_print(&env, "DRBD_MINOR_%u=%u",
+ vnr, peer_device->device->minor);
+ if (get_ldev(device)) {
+ struct drbd_disk_conf *disk_conf =
+ rcu_dereference(device->ldev->disk_conf);
+ env_print(&env, "DRBD_BACKING_DEV_%u=%s",
+ vnr, disk_conf->backing_dev);
+ put_ldev(device);
+ }
+ }
+ }
+ rcu_read_unlock();
+
+ if (strstr(cmd, "fence")) {
+ bool op_is_fence = strcmp(cmd, "fence-peer") == 0;
+ struct drbd_peer_device *peer_device;
+ u64 mask = -1ULL;
+ int vnr;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ if (get_ldev(device)) {
+ u64 m = up_to_date_nodes(device, op_is_fence);
+
+ if (m)
+ mask &= m;
+ put_ldev(device);
+ /* Yes we outright ignore volumes that are not up-to-date
+ on a single node. */
+ }
+ }
+ env_print(&env, "UP_TO_DATE_NODES=0x%08llX", mask);
+ }
+
+ envp = make_envp(&env);
+ if (!envp) {
+ if (env.pos == -ENOMEM) {
+ free_pages((unsigned long)env.buffer, get_order(env.size));
+ env.size += PAGE_SIZE;
+ goto enlarge_buffer;
+ }
+ ret = env.pos;
+ goto out_err;
+ }
- snprintf(mb, 14, "minor-%d", device_to_minor(device));
- setup_khelper_env(connection, envp);
+ if (current == resource->worker.task)
+ set_bit(CALLBACK_PENDING, &resource->flags);
/* The helper may take some time.
* write out any unsynced meta data changes now */
- drbd_md_sync(device);
+ if (device)
+ drbd_md_sync_if_dirty(device);
+ else if (connection)
+ conn_md_sync(connection);
- drbd_info(device, "helper command: %s %s %s\n", drbd_usermode_helper, cmd, mb);
- sib.sib_reason = SIB_HELPER_PRE;
- sib.helper_name = cmd;
- drbd_bcast_event(device, &sib);
+ if (connection && device)
+ peer_device = conn_peer_device(connection, device->vnr);
+
+ magic_printk(KERN_INFO, "helper command: %s %s\n", drbd_usermode_helper, cmd);
notify_helper(NOTIFY_CALL, device, connection, cmd, 0);
ret = call_usermodehelper(drbd_usermode_helper, argv, envp, UMH_WAIT_PROC);
if (ret)
- drbd_warn(device, "helper command: %s %s %s exit code %u (0x%x)\n",
- drbd_usermode_helper, cmd, mb,
- (ret >> 8) & 0xff, ret);
+ magic_printk(KERN_WARNING,
+ "helper command: %s %s exit code %u (0x%x)\n",
+ drbd_usermode_helper, cmd,
+ (ret >> 8) & 0xff, ret);
else
- drbd_info(device, "helper command: %s %s %s exit code %u (0x%x)\n",
- drbd_usermode_helper, cmd, mb,
- (ret >> 8) & 0xff, ret);
- sib.sib_reason = SIB_HELPER_POST;
- sib.helper_exit_code = ret;
- drbd_bcast_event(device, &sib);
+ magic_printk(KERN_INFO,
+ "helper command: %s %s exit code 0\n",
+ drbd_usermode_helper, cmd);
notify_helper(NOTIFY_RESPONSE, device, connection, cmd, ret);
- if (current == connection->worker.task)
- clear_bit(CALLBACK_PENDING, &connection->flags);
+ if (current == resource->worker.task)
+ clear_bit(CALLBACK_PENDING, &resource->flags);
if (ret < 0) /* Ignore any ERRNOs we got. */
ret = 0;
+ free_pages((unsigned long)env.buffer, get_order(env.size));
return ret;
-}
-enum drbd_peer_state conn_khelper(struct drbd_connection *connection, char *cmd)
-{
- char *envp[] = { "HOME=/",
- "TERM=linux",
- "PATH=/sbin:/usr/sbin:/bin:/usr/bin",
- (char[20]) { }, /* address family */
- (char[60]) { }, /* address */
- NULL };
- char *resource_name = connection->resource->name;
- char *argv[] = {drbd_usermode_helper, cmd, resource_name, NULL };
- int ret;
-
- setup_khelper_env(connection, envp);
- conn_md_sync(connection);
-
- drbd_info(connection, "helper command: %s %s %s\n", drbd_usermode_helper, cmd, resource_name);
- /* TODO: conn_bcast_event() ?? */
- notify_helper(NOTIFY_CALL, NULL, connection, cmd, 0);
+out_err:
+ drbd_err(resource, "Could not call %s user-space helper: error %d"
+ "out of memory\n", cmd, ret);
+ return 0;
+}
- ret = call_usermodehelper(drbd_usermode_helper, argv, envp, UMH_WAIT_PROC);
- if (ret)
- drbd_warn(connection, "helper command: %s %s %s exit code %u (0x%x)\n",
- drbd_usermode_helper, cmd, resource_name,
- (ret >> 8) & 0xff, ret);
- else
- drbd_info(connection, "helper command: %s %s %s exit code %u (0x%x)\n",
- drbd_usermode_helper, cmd, resource_name,
- (ret >> 8) & 0xff, ret);
- /* TODO: conn_bcast_event() ?? */
- notify_helper(NOTIFY_RESPONSE, NULL, connection, cmd, ret);
+#undef magic_printk
- if (ret < 0) /* Ignore any ERRNOs we got. */
- ret = 0;
+int drbd_maybe_khelper(struct drbd_device *device, struct drbd_connection *connection, char *cmd)
+{
+ if (strcmp(drbd_usermode_helper, "disabled") == 0)
+ return DRBD_UMH_DISABLED;
- return ret;
+ return drbd_khelper(device, connection, cmd);
}
-static enum drbd_fencing_p highest_fencing_policy(struct drbd_connection *connection)
+static bool initial_states_pending(struct drbd_connection *connection)
{
- enum drbd_fencing_p fp = FP_NOT_AVAIL;
struct drbd_peer_device *peer_device;
int vnr;
+ bool pending = false;
rcu_read_lock();
idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- if (get_ldev_if_state(device, D_CONSISTENT)) {
- struct disk_conf *disk_conf =
- rcu_dereference(peer_device->device->ldev->disk_conf);
- fp = max_t(enum drbd_fencing_p, fp, disk_conf->fencing);
- put_ldev(device);
+ if (test_bit(INITIAL_STATE_SENT, peer_device->flags) &&
+ peer_device->repl_state[NOW] == L_OFF) {
+ pending = true;
+ break;
}
}
rcu_read_unlock();
-
- return fp;
+ return pending;
}
-static bool resource_is_supended(struct drbd_resource *resource)
+static bool intentional_diskless(struct drbd_resource *resource)
{
- return resource->susp || resource->susp_fen || resource->susp_nod;
+ bool intentional_diskless = true;
+ struct drbd_device *device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (!device->device_conf.intentional_diskless) {
+ intentional_diskless = false;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return intentional_diskless;
}
-bool conn_try_outdate_peer(struct drbd_connection *connection)
+static bool conn_try_outdate_peer(struct drbd_connection *connection, const char *tag)
{
- struct drbd_resource * const resource = connection->resource;
- unsigned int connect_cnt;
- union drbd_state mask = { };
- union drbd_state val = { };
- enum drbd_fencing_p fp;
+ struct drbd_resource *resource = connection->resource;
+ unsigned long last_reconnect_jif;
+ enum drbd_fencing_policy fencing_policy;
+ enum drbd_disk_state disk_state;
char *ex_to_string;
int r;
+ unsigned long irq_flags;
- spin_lock_irq(&resource->req_lock);
- if (connection->cstate >= C_WF_REPORT_PARAMS) {
- drbd_err(connection, "Expected cstate < C_WF_REPORT_PARAMS\n");
- spin_unlock_irq(&resource->req_lock);
+ read_lock_irq(&resource->state_rwlock);
+ if (connection->cstate[NOW] >= C_CONNECTED) {
+ drbd_err(connection, "Expected cstate < C_CONNECTED\n");
+ read_unlock_irq(&resource->state_rwlock);
return false;
}
- connect_cnt = connection->connect_cnt;
- spin_unlock_irq(&resource->req_lock);
-
- fp = highest_fencing_policy(connection);
- switch (fp) {
- case FP_NOT_AVAIL:
- drbd_warn(connection, "Not fencing peer, I'm not even Consistent myself.\n");
- spin_lock_irq(&resource->req_lock);
- if (connection->cstate < C_WF_REPORT_PARAMS) {
- _conn_request_state(connection,
- (union drbd_state) { { .susp_fen = 1 } },
- (union drbd_state) { { .susp_fen = 0 } },
- CS_VERBOSE | CS_HARD | CS_DC_SUSP);
- /* We are no longer suspended due to the fencing policy.
- * We may still be suspended due to the on-no-data-accessible policy.
- * If that was OND_IO_ERROR, fail pending requests. */
- if (!resource_is_supended(resource))
- _tl_restart(connection, CONNECTION_LOST_WHILE_PENDING);
- }
- /* Else: in case we raced with a connection handshake,
- * let the handshake figure out if we maybe can RESEND,
- * and do not resume/fail pending requests here.
- * Worst case is we stay suspended for now, which may be
- * resolved by either re-establishing the replication link, or
- * the next link failure, or eventually the administrator. */
- spin_unlock_irq(&resource->req_lock);
+ last_reconnect_jif = connection->last_reconnect_jif;
+
+ disk_state = conn_highest_disk(connection);
+ if (disk_state < D_CONSISTENT &&
+ !(disk_state == D_DISKLESS && intentional_diskless(resource))) {
+ begin_state_change_locked(resource, CS_VERBOSE | CS_HARD);
+ __change_io_susp_fencing(connection, false);
+ end_state_change_locked(resource, tag);
+ read_unlock_irq(&resource->state_rwlock);
return false;
+ }
+ read_unlock_irq(&resource->state_rwlock);
- case FP_DONT_CARE:
+ fencing_policy = connection->fencing_policy;
+ if (fencing_policy == FP_DONT_CARE)
return true;
- default: ;
- }
- r = conn_khelper(connection, "fence-peer");
+ r = drbd_maybe_khelper(NULL, connection, "fence-peer");
+ if (r == DRBD_UMH_DISABLED)
+ return true;
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
switch ((r>>8) & 0xff) {
case P_INCONSISTENT: /* peer is inconsistent */
ex_to_string = "peer is inconsistent or worse";
- mask.pdsk = D_MASK;
- val.pdsk = D_INCONSISTENT;
+ __downgrade_peer_disk_states(connection, D_INCONSISTENT);
break;
case P_OUTDATED: /* peer got outdated, or was already outdated */
ex_to_string = "peer was fenced";
- mask.pdsk = D_MASK;
- val.pdsk = D_OUTDATED;
+ __downgrade_peer_disk_states(connection, D_OUTDATED);
break;
case P_DOWN: /* peer was down */
if (conn_highest_disk(connection) == D_UP_TO_DATE) {
/* we will(have) create(d) a new UUID anyways... */
ex_to_string = "peer is unreachable, assumed to be dead";
- mask.pdsk = D_MASK;
- val.pdsk = D_OUTDATED;
+ __downgrade_peer_disk_states(connection, D_OUTDATED);
} else {
ex_to_string = "peer unreachable, doing nothing since disk != UpToDate";
}
@@ -573,42 +653,44 @@ bool conn_try_outdate_peer(struct drbd_connection *connection)
* become R_PRIMARY, but finds the other peer being active. */
ex_to_string = "peer is active";
drbd_warn(connection, "Peer is primary, outdating myself.\n");
- mask.disk = D_MASK;
- val.disk = D_OUTDATED;
+ __downgrade_disk_states(resource, D_OUTDATED);
break;
case P_FENCING:
/* THINK: do we need to handle this
- * like case 4, or more like case 5? */
- if (fp != FP_STONITH)
+ * like case 4 P_OUTDATED, or more like case 5 P_DOWN? */
+ if (fencing_policy != FP_STONITH)
drbd_err(connection, "fence-peer() = 7 && fencing != Stonith !!!\n");
ex_to_string = "peer was stonithed";
- mask.pdsk = D_MASK;
- val.pdsk = D_OUTDATED;
+ __downgrade_peer_disk_states(connection, D_OUTDATED);
break;
default:
/* The script is broken ... */
drbd_err(connection, "fence-peer helper broken, returned %d\n", (r>>8)&0xff);
+ abort_state_change(resource, &irq_flags);
return false; /* Eventually leave IO frozen */
}
drbd_info(connection, "fence-peer helper returned %d (%s)\n",
(r>>8) & 0xff, ex_to_string);
- /* Not using
- conn_request_state(connection, mask, val, CS_VERBOSE);
- here, because we might were able to re-establish the connection in the
- meantime. */
- spin_lock_irq(&resource->req_lock);
- if (connection->cstate < C_WF_REPORT_PARAMS && !test_bit(STATE_SENT, &connection->flags)) {
- if (connection->connect_cnt != connect_cnt)
- /* In case the connection was established and droped
- while the fence-peer handler was running, ignore it */
- drbd_info(connection, "Ignoring fence-peer exit code\n");
- else
- _conn_request_state(connection, mask, val, CS_VERBOSE);
+ if (connection->cstate[NOW] >= C_CONNECTED ||
+ initial_states_pending(connection)) {
+ /* connection re-established; do not fence */
+ goto abort;
+ }
+ if (connection->last_reconnect_jif != last_reconnect_jif) {
+ /* In case the connection was established and dropped
+ while the fence-peer handler was running, ignore it */
+ drbd_info(connection, "Ignoring fence-peer exit code\n");
+ goto abort;
}
- spin_unlock_irq(&resource->req_lock);
+ end_state_change(resource, &irq_flags, tag);
+
+ goto out;
+ abort:
+ abort_state_change(resource, &irq_flags);
+ out:
return conn_highest_pdsk(connection) <= D_OUTDATED;
}
@@ -616,8 +698,7 @@ static int _try_outdate_peer_async(void *data)
{
struct drbd_connection *connection = (struct drbd_connection *)data;
- conn_try_outdate_peer(connection);
-
+ conn_try_outdate_peer(connection, "outdate-async");
kref_put(&connection->kref, drbd_destroy_connection);
return 0;
}
@@ -640,206 +721,564 @@ void conn_try_outdate_peer_async(struct drbd_connection *connection)
}
}
-enum drbd_state_rv
-drbd_set_role(struct drbd_device *const device, enum drbd_role new_role, int force)
+bool barrier_pending(struct drbd_resource *resource)
{
- struct drbd_peer_device *const peer_device = first_peer_device(device);
- struct drbd_connection *const connection = peer_device ? peer_device->connection : NULL;
- const int max_tries = 4;
- enum drbd_state_rv rv = SS_UNKNOWN_ERROR;
- struct net_conf *nc;
- int try = 0;
- int forced = 0;
- union drbd_state mask, val;
-
- if (new_role == R_PRIMARY) {
- struct drbd_connection *connection;
-
- /* Detect dead peers as soon as possible. */
+ struct drbd_connection *connection;
+ bool rv = false;
- rcu_read_lock();
- for_each_connection(connection, device->resource)
- request_ping(connection);
- rcu_read_unlock();
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (READ_ONCE(connection->last_acked_epoch_nr) !=
+ READ_ONCE(connection->send.last_sent_epoch_nr)) {
+ rv = true;
+ break;
+ }
}
+ rcu_read_unlock();
- mutex_lock(device->state_mutex);
+ return rv;
+}
- mask.i = 0; mask.role = R_MASK;
- val.i = 0; val.role = new_role;
+static int count_up_to_date(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr, nr_up_to_date = 0;
- while (try++ < max_tries) {
- rv = _drbd_request_state_holding_state_mutex(device, mask, val, CS_WAIT_COMPLETE);
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ enum drbd_disk_state disk_state = device->disk_state[NOW];
+
+ if (disk_state == D_UP_TO_DATE)
+ nr_up_to_date++;
+ }
+ rcu_read_unlock();
+ return nr_up_to_date;
+}
+
+static bool reconciliation_ongoing(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device_rcu(peer_device, device) {
+ if (test_bit(RECONCILIATION_RESYNC, peer_device->flags))
+ return true;
+ }
+ return false;
+}
+
+static bool any_peer_is_consistent(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->disk_state[NOW] == D_CONSISTENT)
+ return true;
+ }
+ return false;
+}
+/* reconciliation resyncs finished and I know if I am D_UP_TO_DATE or D_OUTDATED */
+static bool after_primary_lost_events_settled(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr;
+
+ if (test_bit(TRY_BECOME_UP_TO_DATE_PENDING, &resource->flags))
+ return false;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ enum drbd_disk_state disk_state = device->disk_state[NOW];
+
+ if (disk_state == D_CONSISTENT ||
+ any_peer_is_consistent(device) ||
+ (reconciliation_ongoing(device) &&
+ (disk_state == D_OUTDATED || disk_state == D_INCONSISTENT))) {
+ rcu_read_unlock();
+ return false;
+ }
+ }
+ rcu_read_unlock();
+ return true;
+}
+
+static long drbd_max_ping_timeout(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ long ping_timeout = 0;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource)
+ ping_timeout = max(ping_timeout, (long) connection->transport.net_conf->ping_timeo);
+ rcu_read_unlock();
+
+ return ping_timeout;
+}
+
+static bool wait_up_to_date(struct drbd_resource *resource)
+{
+ /*
+ * Adding ping-timeout is necessary to ensure that we do not proceed
+ * while the loss of some connection has not yet been detected. Ideally
+ * we would use the maximum ping timeout from the entire cluster. Since
+ * we do not have that, use the maximum from our connections on a
+ * best-effort basis.
+ */
+ long timeout = (resource->res_opts.auto_promote_timeout +
+ drbd_max_ping_timeout(resource)) * HZ / 10;
+ int initial_up_to_date, up_to_date;
+
+ initial_up_to_date = count_up_to_date(resource);
+ wait_event_interruptible_timeout(resource->state_wait,
+ after_primary_lost_events_settled(resource),
+ timeout);
+ up_to_date = count_up_to_date(resource);
+ return up_to_date > initial_up_to_date;
+}
+
+static void set_role_backoff(enum chg_state_flags flags, long timeout)
+{
+ __set_current_state(flags & CS_NO_SIGNALS ? TASK_UNINTERRUPTIBLE : TASK_INTERRUPTIBLE);
+ schedule_timeout(timeout);
+}
+
+enum drbd_state_rv
+drbd_set_role(struct drbd_resource *resource, enum drbd_role role,
+ enum chg_state_flags flags, const char *tag, struct drbd_adm_ctx *ctx)
+{
+ struct drbd_device *device;
+ int vnr, try = 0;
+ const int max_tries = 4;
+ enum drbd_state_rv rv = SS_UNKNOWN_ERROR;
+ bool retried_ss_two_primaries = false, retried_ss_primary_nop = false;
+ const char *err_str = NULL;
+ bool force = flags & CS_FORCE_ROLE;
+ bool fenced_peers = false;
+
+ flags |= CS_ALREADY_SERIALIZED | CS_DONT_RETRY | CS_WAIT_COMPLETE;
+
+retry:
+
+ if (role == R_PRIMARY) {
+ drbd_check_peers(resource);
+ wait_up_to_date(resource);
+ }
+ down(&resource->state_sem);
+ while (try++ < max_tries) {
+ if (try == max_tries - 1)
+ flags |= CS_VERBOSE;
+
+ kfree(err_str);
+ err_str = NULL;
+ if (flags & CS_NO_SIGNALS)
+ rv = stable_state_change_no_signals(resource,
+ change_role(resource, role, flags, tag, &err_str));
+ else
+ rv = stable_state_change(resource,
+ change_role(resource, role, flags, tag, &err_str));
+
+ if (rv == SS_TIMEOUT || rv == SS_CONCURRENT_ST_CHG) {
+ long timeout = twopc_retry_timeout(resource, try);
+ /* It might be that the receiver tries to start resync, and
+ sleeps on state_sem. Give it up, and retry in a short
+ while */
+ up(&resource->state_sem);
+ set_role_backoff(flags, timeout);
+ goto retry;
+ }
/* in case we first succeeded to outdate,
* but now suddenly could establish a connection */
- if (rv == SS_CW_FAILED_BY_PEER && mask.pdsk != 0) {
- val.pdsk = 0;
- mask.pdsk = 0;
+ if (rv == SS_CW_FAILED_BY_PEER && fenced_peers) {
+ flags &= ~CS_FP_LOCAL_UP_TO_DATE;
continue;
}
- if (rv == SS_NO_UP_TO_DATE_DISK && force &&
- (device->state.disk < D_UP_TO_DATE &&
- device->state.disk >= D_INCONSISTENT)) {
- mask.disk = D_MASK;
- val.disk = D_UP_TO_DATE;
- forced = 1;
+ if (rv == SS_NO_UP_TO_DATE_DISK && force && !(flags & CS_FP_LOCAL_UP_TO_DATE)) {
+ flags |= CS_FP_LOCAL_UP_TO_DATE;
continue;
}
- if (rv == SS_NO_UP_TO_DATE_DISK &&
- device->state.disk == D_CONSISTENT && mask.pdsk == 0) {
- D_ASSERT(device, device->state.pdsk == D_UNKNOWN);
+ if (rv == SS_DEVICE_IN_USE && force && !(flags & CS_FS_IGN_OPENERS)) {
+ drbd_warn(resource, "forced demotion\n");
+ flags |= CS_FS_IGN_OPENERS; /* this sets resource->fail_io[NOW] */
+ continue;
+ }
+
+ if (rv == SS_NO_UP_TO_DATE_DISK) {
+ bool a_disk_became_up_to_date;
+
+ /* need to give up state_sem, see try_become_up_to_date(); */
+ up(&resource->state_sem);
+ drbd_flush_workqueue(&resource->work);
+ a_disk_became_up_to_date = wait_up_to_date(resource);
+ down(&resource->state_sem);
+ if (a_disk_became_up_to_date)
+ continue;
+ /* fall through into possible fence-peer or even force cases */
+ }
+
+ if (rv == SS_NO_UP_TO_DATE_DISK && !(flags & CS_FP_LOCAL_UP_TO_DATE)) {
+ struct drbd_connection *connection;
+ bool any_fencing_failed = false;
+ u64 im;
+
+ fenced_peers = false;
+ up(&resource->state_sem); /* Allow connect while fencing */
+ for_each_connection_ref(connection, im, resource) {
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ if (conn_highest_pdsk(connection) != D_UNKNOWN)
+ continue;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *device = peer_device->device;
+
+ if (device->disk_state[NOW] != D_CONSISTENT)
+ continue;
- if (conn_try_outdate_peer(connection)) {
- val.disk = D_UP_TO_DATE;
- mask.disk = D_MASK;
+ if (conn_try_outdate_peer(connection, tag))
+ fenced_peers = true;
+ else
+ any_fencing_failed = true;
+ }
}
+ down(&resource->state_sem);
+ if (fenced_peers && !any_fencing_failed) {
+ flags |= CS_FP_LOCAL_UP_TO_DATE;
+ continue;
+ }
+ }
+
+ /* In case the disk is Consistent and fencing is enabled, and fencing did not work
+ * but the user forces promote..., try it pretending we fenced the peers */
+ if (rv == SS_PRIMARY_NOP && force &&
+ (flags & CS_FP_LOCAL_UP_TO_DATE) && !(flags & CS_FP_OUTDATE_PEERS)) {
+ flags |= CS_FP_OUTDATE_PEERS;
+ continue;
+ }
+
+ if (rv == SS_NO_QUORUM && force && !(flags & CS_FP_OUTDATE_PEERS)) {
+ flags |= CS_FP_OUTDATE_PEERS;
continue;
}
if (rv == SS_NOTHING_TO_DO)
goto out;
- if (rv == SS_PRIMARY_NOP && mask.pdsk == 0) {
- if (!conn_try_outdate_peer(connection) && force) {
- drbd_warn(device, "Forced into split brain situation!\n");
- mask.pdsk = D_MASK;
- val.pdsk = D_OUTDATED;
+ if (rv == SS_PRIMARY_NOP && !retried_ss_primary_nop) {
+ struct drbd_connection *connection;
+ u64 im;
+ retried_ss_primary_nop = true;
+
+ up(&resource->state_sem); /* Allow connect while fencing */
+ for_each_connection_ref(connection, im, resource) {
+ bool outdated_peer = conn_try_outdate_peer(connection, tag);
+
+ if (!outdated_peer && force) {
+ drbd_warn(connection, "Forced into split brain situation!\n");
+ flags |= CS_FP_LOCAL_UP_TO_DATE;
+ }
}
+ down(&resource->state_sem);
continue;
}
- if (rv == SS_TWO_PRIMARIES) {
- /* Maybe the peer is detected as dead very soon...
- retry at most once more in this case. */
- if (try < max_tries) {
- int timeo;
- try = max_tries - 1;
- rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- timeo = nc ? (nc->ping_timeo + 1) * HZ / 10 : 1;
- rcu_read_unlock();
- schedule_timeout_interruptible(timeo);
+
+ if (rv == SS_TWO_PRIMARIES && !retried_ss_two_primaries) {
+ struct drbd_connection *connection;
+ struct drbd_net_conf *nc;
+ int timeout = 0;
+
+ retried_ss_two_primaries = true;
+
+ /*
+ * Catch the case where we discover that the other
+ * primary has died soon after the state change
+ * failure: retry once after a short timeout.
+ */
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (nc && nc->ping_timeo > timeout)
+ timeout = nc->ping_timeo;
}
- continue;
- }
- if (rv < SS_SUCCESS) {
- rv = _drbd_request_state(device, mask, val,
- CS_VERBOSE + CS_WAIT_COMPLETE);
- if (rv < SS_SUCCESS)
- goto out;
+ rcu_read_unlock();
+ timeout = timeout * HZ / 10;
+ if (timeout == 0)
+ timeout = 1;
+
+ up(&resource->state_sem);
+ set_role_backoff(flags, timeout);
+ goto retry;
}
+
break;
}
if (rv < SS_SUCCESS)
goto out;
- if (forced)
- drbd_warn(device, "Forced to consider local data as UpToDate!\n");
-
- /* Wait until nothing is on the fly :) */
- wait_event(device->misc_wait, atomic_read(&device->ap_pending_cnt) == 0);
-
- /* FIXME also wait for all pending P_BARRIER_ACK? */
+ if (force) {
+ if (flags & CS_FP_LOCAL_UP_TO_DATE)
+ drbd_warn(resource, "Forced to consider local data as UpToDate!\n");
+ if (flags & CS_FP_OUTDATE_PEERS)
+ drbd_warn(resource, "Forced to consider peers as Outdated!\n");
+ }
- if (new_role == R_SECONDARY) {
- if (get_ldev(device)) {
- device->ldev->md.uuid[UI_CURRENT] &= ~(u64)1;
- put_ldev(device);
+ if (role == R_SECONDARY) {
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (get_ldev(device)) {
+ device->ldev->md.current_uuid &= ~UUID_PRIMARY;
+ put_ldev(device);
+ }
}
} else {
- mutex_lock(&device->resource->conf_update);
- nc = connection->net_conf;
- if (nc)
- nc->discard_my_data = 0; /* without copy; single bit op is atomic */
- mutex_unlock(&device->resource->conf_update);
+ struct drbd_connection *connection;
- if (get_ldev(device)) {
- if (((device->state.conn < C_CONNECTED ||
- device->state.pdsk <= D_FAILED)
- && device->ldev->md.uuid[UI_BITMAP] == 0) || forced)
- drbd_uuid_new_current(device);
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource)
+ clear_bit(CONN_DISCARD_MY_DATA, &connection->flags);
+ rcu_read_unlock();
- device->ldev->md.uuid[UI_CURRENT] |= (u64)1;
- put_ldev(device);
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (flags & CS_FP_LOCAL_UP_TO_DATE) {
+ enum drbd_mint_outcome outcome;
+ bool through_executor;
+
+ /* gen-rotate reason: OTHER (admin force-primary).
+ * This generation is the promotion's own and is
+ * owed whether or not an obligation is armed;
+ * where it is also the obligation's mint, its
+ * outcome decides.
+ */
+ through_executor = drbd_gen_obligation_mint_start(device);
+ outcome = drbd_uuid_new_current(device, true);
+ if (through_executor) {
+ drbd_gen_obligation_mint_done(device, outcome);
+ wake_up(&device->misc_wait);
+ }
+ }
}
}
- /* writeout of activity log covered areas of the bitmap
- * to stable storage done in after state change already */
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ struct drbd_peer_device *peer_device;
+ u64 im;
+
+ for_each_peer_device_ref(peer_device, im, device) {
+ /* writeout of activity log covered areas of the bitmap
+ * to stable storage done in after state change already */
+
+ if (peer_device->connection->cstate[NOW] == C_CONNECTED) {
+ /* if this was forced, we should consider sync */
+ if (flags & CS_FP_LOCAL_UP_TO_DATE) {
+ drbd_send_uuids(peer_device, 0, 0);
+ set_bit(CONSIDER_RESYNC, peer_device->flags);
+ }
+ drbd_send_current_state(peer_device);
+ }
+ }
+ }
- if (device->state.conn >= C_WF_REPORT_PARAMS) {
- /* if this was forced, we should consider sync */
- if (forced)
- drbd_send_uuids(peer_device);
- drbd_send_current_state(peer_device);
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ drbd_md_sync_if_dirty(device);
+ if (!resource->res_opts.auto_promote && role == R_PRIMARY)
+ kobject_uevent(&disk_to_dev(device->vdisk)->kobj, KOBJ_CHANGE);
}
- drbd_md_sync(device);
- set_disk_ro(device->vdisk, new_role == R_SECONDARY);
- kobject_uevent(&disk_to_dev(device->vdisk)->kobj, KOBJ_CHANGE);
out:
- mutex_unlock(device->state_mutex);
+ up(&resource->state_sem);
+ if (err_str) {
+ drbd_err(resource, "%s", err_str);
+ if (ctx)
+ drbd_adm_msg(ctx, "%s", err_str);
+ kfree(err_str);
+ }
return rv;
}
-static const char *from_attrs_err_to_txt(int err)
+/* suggested buffer size: 128 byte */
+void youngest_and_oldest_opener_to_str(struct drbd_device *device, char *buf, size_t len)
+{
+ struct timespec64 ts;
+ struct tm tm;
+ struct opener *first;
+ struct opener *last;
+ int cnt;
+
+ buf[0] = '\0';
+ /* Do we have opener information? */
+ if (!device->open_cnt)
+ return;
+ cnt = snprintf(buf, len, " open_cnt:%d", device->open_cnt);
+ if (cnt > 0 && cnt < len) {
+ buf += cnt;
+ len -= cnt;
+ } else
+ return;
+ spin_lock(&device->openers_lock);
+ if (!list_empty(&device->openers)) {
+ first = list_first_entry(&device->openers, struct opener, list);
+ ts = ktime_to_timespec64(first->opened);
+ time64_to_tm(ts.tv_sec, -sys_tz.tz_minuteswest * 60, &tm);
+ cnt = snprintf(buf, len, " [%s:%d:%04ld-%02d-%02d_%02d:%02d:%02d.%03ld]",
+ first->comm, first->pid,
+ tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,
+ tm.tm_hour, tm.tm_min, tm.tm_sec, ts.tv_nsec / NSEC_PER_MSEC);
+ last = list_last_entry(&device->openers, struct opener, list);
+ if (cnt > 0 && cnt < len && last != first) {
+ /* append, overwriting the previously added ']' */
+ buf += cnt-1;
+ len -= cnt-1;
+ ts = ktime_to_timespec64(last->opened);
+ time64_to_tm(ts.tv_sec, -sys_tz.tz_minuteswest * 60, &tm);
+ snprintf(buf, len, "%s%s:%d:%04ld-%02d-%02d_%02d:%02d:%02d.%03ld]",
+ device->open_cnt > 2 ? ", ..., " : ", ",
+ last->comm, last->pid,
+ tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,
+ tm.tm_hour, tm.tm_min, tm.tm_sec, ts.tv_nsec / NSEC_PER_MSEC);
+ }
+ }
+ spin_unlock(&device->openers_lock);
+}
+
+static int put_device_opener_info(struct drbd_device *device, struct drbd_adm_ctx *ctx)
+{
+ struct timespec64 ts;
+ struct opener *o;
+ struct tm tm;
+ int cnt = 0;
+ char *dotdotdot = "";
+
+ spin_lock(&device->openers_lock);
+ if (!device->open_cnt) {
+ spin_unlock(&device->openers_lock);
+ return cnt;
+ }
+ drbd_adm_msg(ctx,
+ "/dev/drbd%d open_cnt:%d, writable:%d; list of openers follows",
+ device->minor, device->open_cnt, device->writable);
+ list_for_each_entry(o, &device->openers, list) {
+ ts = ktime_to_timespec64(o->opened);
+ time64_to_tm(ts.tv_sec, -sys_tz.tz_minuteswest * 60, &tm);
+
+ if (++cnt >= 10 && !list_is_last(&o->list, &device->openers)) {
+ o = list_last_entry(&device->openers, struct opener, list);
+ dotdotdot = "[...]\n";
+ }
+ drbd_adm_msg(ctx,
+ "%sdrbd%d opened by %s (pid %d) at %04ld-%02d-%02d %02d:%02d:%02d.%03ld",
+ dotdotdot,
+ device->minor, o->comm, o->pid,
+ tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,
+ tm.tm_hour, tm.tm_min, tm.tm_sec,
+ ts.tv_nsec / NSEC_PER_MSEC);
+ }
+ spin_unlock(&device->openers_lock);
+ return cnt;
+}
+
+static void opener_info(struct drbd_resource *resource,
+ struct drbd_adm_ctx *ctx,
+ enum drbd_state_rv rv)
{
- return err == -ENOMSG ? "required attribute missing" :
+ struct drbd_device *device;
+ int i;
+
+ if (rv != SS_DEVICE_IN_USE && rv != SS_NO_UP_TO_DATE_DISK)
+ return;
+
+ idr_for_each_entry(&resource->devices, device, i)
+ put_device_opener_info(device, ctx);
+}
+
+/* Report the failure of a dialect overlay callback. */
+static void drbd_adm_msg_overlay_error(struct drbd_adm_ctx *ctx, int err)
+{
+ const char *txt =
+ err == -ENOMSG ? "required attribute missing" :
err == -EEXIST ? "can not change invariant setting" :
"invalid attribute value";
+
+ drbd_adm_msg(ctx, "%s", txt);
}
-static int drbd_nl_set_role(struct sk_buff *skb, struct genl_info *info)
+static int drbd_adm_set_role(struct drbd_adm_ctx *adm_ctx, enum drbd_role new_role)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct set_role_parms parms;
- int err;
- enum drbd_ret_code retcode;
+ struct drbd_resource *resource;
+ struct drbd_set_role_parms parms;
enum drbd_state_rv rv;
+ enum drbd_ret_code retcode = NO_ERROR;
+ int err;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
-
+ resource = adm_ctx->resource;
memset(&parms, 0, sizeof(parms));
- if (info->attrs[DRBD_NLA_SET_ROLE_PARMS]) {
- err = set_role_parms_from_attrs(&parms, info);
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_SET_ROLE_PARMS)) {
+ err = drbd_adm_overlay_set_role_parms(adm_ctx, &parms);
if (err) {
retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
+ drbd_adm_msg_overlay_error(adm_ctx, err);
goto out;
}
}
- genl_unlock();
- mutex_lock(&adm_ctx->resource->adm_mutex);
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
+ }
- if (info->genlhdr->cmd == DRBD_ADM_PRIMARY)
- rv = drbd_set_role(adm_ctx->device, R_PRIMARY, parms.assume_uptodate);
- else
- rv = drbd_set_role(adm_ctx->device, R_SECONDARY, 0);
+ if (new_role == R_PRIMARY)
+ set_bit(EXPLICIT_PRIMARY, &resource->flags);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
- genl_lock();
- adm_ctx->reply_dh->ret_code = rv;
- return 0;
+ rv = drbd_set_role(resource,
+ new_role,
+ parms.force ? CS_FORCE_ROLE : 0,
+ new_role == R_PRIMARY ? "primary" : "secondary",
+ adm_ctx);
+
+ if (resource->role[NOW] != R_PRIMARY)
+ clear_bit(EXPLICIT_PRIMARY, &resource->flags);
+
+ if (rv == SS_DEVICE_IN_USE)
+ opener_info(resource, adm_ctx, rv);
+
+ mutex_unlock(&resource->adm_mutex);
+ retcode = (enum drbd_ret_code)rv;
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
-int drbd_nl_primary_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_primary(struct drbd_adm_ctx *adm_ctx)
+{
+ return drbd_adm_set_role(adm_ctx, R_PRIMARY);
+}
+
+int drbd_adm_secondary(struct drbd_adm_ctx *adm_ctx)
{
- return drbd_nl_set_role(skb, info);
+ return drbd_adm_set_role(adm_ctx, R_SECONDARY);
}
-int drbd_nl_secondary_doit(struct sk_buff *skb, struct genl_info *info)
+u64 drbd_capacity_to_on_disk_bm_sect(u64 capacity_sect, const struct drbd_md *md)
{
- return drbd_nl_set_role(skb, info);
+ u64 bits, bytes;
+
+ /* round up storage sectors to full "bitmap sectors per bit", then
+ * convert to number of bits needed, and round that up to 64bit words
+ * to ease interoperability between 32bit and 64bit architectures.
+ */
+ bits = ALIGN(sect_to_bit(
+ ALIGN(capacity_sect, sect_per_bit(md->bm_block_shift)),
+ md->bm_block_shift), 64);
+
+ /* convert to bytes, multiply by number of peers,
+ * and, because we do all our meta data IO in 4k blocks,
+ * round up to full 4k
+ */
+ bytes = ALIGN(bits / 8 * md->max_peers, 4096);
+
+ /* convert to number of sectors */
+ return bytes >> 9;
}
/* Initializes the md.*_offset members, so we are able to find
@@ -861,10 +1300,9 @@ int drbd_nl_secondary_doit(struct sk_buff *skb, struct genl_info *info)
* ==> bitmap sectors = Y = al_offset - bm_offset
*
* Activity log size used to be fixed 32kB,
- * but is about to become configurable.
+ * but is actually al_stripes * al_stripe_size_4k.
*/
-static void drbd_md_set_sector_offsets(struct drbd_device *device,
- struct drbd_backing_dev *bdev)
+void drbd_md_set_sector_offsets(struct drbd_backing_dev *bdev)
{
sector_t md_size_sect = 0;
unsigned int al_size_sect = bdev->md.al_size_4k * 8;
@@ -874,33 +1312,32 @@ static void drbd_md_set_sector_offsets(struct drbd_device *device,
switch (bdev->md.meta_dev_idx) {
default:
/* v07 style fixed size indexed meta data */
- bdev->md.md_size_sect = MD_128MB_SECT;
- bdev->md.al_offset = MD_4kB_SECT;
- bdev->md.bm_offset = MD_4kB_SECT + al_size_sect;
+ /* FIXME we should drop support for this! */
+ bdev->md.md_size_sect = (128 << 20 >> 9);
+ bdev->md.al_offset = (4096 >> 9);
+ bdev->md.bm_offset = (4096 >> 9) + al_size_sect;
break;
case DRBD_MD_INDEX_FLEX_EXT:
/* just occupy the full device; unit: sectors */
bdev->md.md_size_sect = drbd_get_capacity(bdev->md_bdev);
- bdev->md.al_offset = MD_4kB_SECT;
- bdev->md.bm_offset = MD_4kB_SECT + al_size_sect;
+ bdev->md.al_offset = (4096 >> 9);
+ bdev->md.bm_offset = (4096 >> 9) + al_size_sect;
break;
case DRBD_MD_INDEX_INTERNAL:
case DRBD_MD_INDEX_FLEX_INT:
- /* al size is still fixed */
bdev->md.al_offset = -al_size_sect;
- /* we need (slightly less than) ~ this much bitmap sectors: */
- md_size_sect = drbd_get_capacity(bdev->backing_bdev);
- md_size_sect = ALIGN(md_size_sect, BM_SECT_PER_EXT);
- md_size_sect = BM_SECT_TO_EXT(md_size_sect);
- md_size_sect = ALIGN(md_size_sect, 8);
- /* plus the "drbd meta data super block",
+ /* enough bitmap to cover the storage,
+ * plus the "drbd meta data super block",
* and the activity log; */
- md_size_sect += MD_4kB_SECT + al_size_sect;
+ md_size_sect = drbd_capacity_to_on_disk_bm_sect(
+ drbd_get_capacity(bdev->backing_bdev),
+ &bdev->md)
+ + (4096 >> 9) + al_size_sect;
bdev->md.md_size_sect = md_size_sect;
/* bitmap offset is adjusted by 'super' block size */
- bdev->md.bm_offset = -md_size_sect + MD_4kB_SECT;
+ bdev->md.bm_offset = -md_size_sect + (4096 >> 9);
break;
}
}
@@ -912,28 +1349,22 @@ char *ppsize(char *buf, unsigned long long size)
* -1ULL ==> "16384 EB" */
static char units[] = { 'K', 'M', 'G', 'T', 'P', 'E' };
int base = 0;
+
while (size >= 10000 && base < sizeof(units)-1) {
/* shift + round */
size = (size >> 10) + !!(size & (1<<9));
base++;
}
- sprintf(buf, "%u %cB", (unsigned)size, units[base]);
+ sprintf(buf, "%u %cB", (unsigned int)size, units[base]);
return buf;
}
-/* there is still a theoretical deadlock when called from receiver
- * on an D_INCONSISTENT R_PRIMARY:
- * remote READ does inc_ap_bio, receiver would need to receive answer
- * packet from remote to dec_ap_bio again.
- * receiver receive_sizes(), comes here,
- * waits for ap_bio_cnt == 0. -> deadlock.
- * but this cannot happen, actually, because:
- * R_PRIMARY D_INCONSISTENT, and peer's disk is unreachable
- * (not connected, or bad/no disk on peer):
- * see drbd_fail_request_early, ap_bio_cnt is zero.
- * R_PRIMARY D_INCONSISTENT, and C_SYNC_TARGET:
- * peer may not initiate a resize.
+/* The receiver may call drbd_suspend_io(device, WRITE_ONLY).
+ * It should not call drbd_suspend_io(device, READ_AND_WRITE) since
+ * if the node is an D_INCONSISTENT R_PRIMARY (L_SYNC_TARGET) it
+ * may need to issue remote READs. Those is turn need the receiver
+ * to complete. -> calling drbd_suspend_io(device, READ_AND_WRITE) deadlocks.
*/
/* Note these are not to be confused with
* drbd_nl_suspend_io_doit/drbd_nl_resume_io_doit,
@@ -943,12 +1374,37 @@ char *ppsize(char *buf, unsigned long long size)
* and should be short-lived. */
/* It needs to be a counter, since multiple threads might
independently suspend and resume IO. */
-void drbd_suspend_io(struct drbd_device *device)
+static bool ap_bio_drained(struct drbd_device *device, enum suspend_scope ss)
+{
+ return drbd_suspended(device) ||
+ atomic_read(&device->ap_bio_cnt[WRITE]) +
+ (ss == READ_AND_WRITE ? atomic_read(&device->ap_bio_cnt[READ]) : 0) == 0;
+}
+
+void __drbd_suspend_io(struct drbd_device *device)
{
atomic_inc(&device->suspend_cnt);
- if (drbd_suspended(device))
- return;
- wait_event(device->misc_wait, !atomic_read(&device->ap_bio_cnt));
+ /* Order the suspend_cnt store before the ap_bio_cnt load in the
+ * caller's wait condition. Pairs with the cmpxchg + suspend_cnt
+ * re-check in inc_ap_bio_cond(): a submitter that increments
+ * ap_bio_cnt after we raised suspend_cnt is guaranteed to observe
+ * suspend_cnt and roll back, or we observe its increment and wait.
+ */
+ smp_mb__after_atomic();
+}
+
+void drbd_suspend_io(struct drbd_device *device, enum suspend_scope ss)
+{
+ __drbd_suspend_io(device);
+ wait_event(device->misc_wait, ap_bio_drained(device, ss));
+}
+
+/* For a suspend taken with __drbd_suspend_io(); the caller still owns it
+ * when this returns, drained or not.
+ */
+bool drbd_wait_io_drained(struct drbd_device *device, enum suspend_scope ss, long timeout)
+{
+ return wait_event_timeout(device->misc_wait, ap_bio_drained(device, ss), timeout) > 0;
}
void drbd_resume_io(struct drbd_device *device)
@@ -957,18 +1413,75 @@ void drbd_resume_io(struct drbd_device *device)
wake_up(&device->misc_wait);
}
+int drbd_suspend_io_interruptible(struct drbd_device *device, enum suspend_scope ss)
+{
+ int ret;
+
+ __drbd_suspend_io(device);
+ ret = wait_event_interruptible(device->misc_wait, ap_bio_drained(device, ss));
+ if (ret)
+ drbd_resume_io(device);
+ return ret;
+}
+
+/**
+ * effective_disk_size_determined() - is the effective disk size "fixed" already?
+ * @device: DRBD device.
+ *
+ * When a device is configured in a cluster, the size of the replicated disk is
+ * determined by the minimum size of the disks on all nodes. Additional nodes
+ * can be added, and this can still change the effective size of the replicated
+ * disk.
+ *
+ * When the disk on any node becomes D_UP_TO_DATE, the effective disk size
+ * becomes "fixed". It is written to the metadata so that it will not be
+ * forgotten across node restarts. Further nodes can only be added if their
+ * disks are big enough.
+ */
+static bool effective_disk_size_determined(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ bool rv = false;
+
+ if (device->ldev->md.effective_size != 0)
+ return true;
+ if (device->disk_state[NOW] == D_UP_TO_DATE)
+ return true;
+
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (peer_device->disk_state[NOW] == D_UP_TO_DATE) {
+ rv = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+void drbd_set_my_capacity(struct drbd_device *device, sector_t size)
+{
+ char ppb[10];
+
+ set_capacity_and_notify(device->vdisk, size);
+
+ drbd_info(device, "size = %s (%llu KB)\n",
+ ppsize(ppb, size>>1), (unsigned long long)size>>1);
+}
+
/*
* drbd_determine_dev_size() - Sets the right device size obeying all constraints
* @device: DRBD device.
*
- * Returns 0 on success, negative return values indicate errors.
* You should call drbd_md_sync() after calling this function.
*/
enum determine_dev_size
-drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct resize_parms *rs) __must_hold(local)
+drbd_determine_dev_size(struct drbd_device *device, sector_t peer_current_size,
+ enum dds_flags flags, struct drbd_resize_parms *rs)
{
struct md_offsets_and_sizes {
- u64 last_agreed_sect;
+ u64 effective_size;
u64 md_offset;
s32 al_offset;
s32 bm_offset;
@@ -977,7 +1490,7 @@ drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct
u32 al_stripes;
u32 al_stripe_size_4k;
} prev;
- sector_t u_size, size;
+ sector_t u_size, size, prev_size;
struct drbd_md *md = &device->ldev->md;
void *buffer;
@@ -992,21 +1505,45 @@ drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct
* Move is not exactly correct, btw, currently we have all our meta
* data in core memory, to "move" it we just write it all out, there
* are no reads. */
- drbd_suspend_io(device);
+ drbd_suspend_io(device, READ_AND_WRITE);
+
+ /* Take the AL transaction lock before the md_buffer to avoid an
+ * AB-BA deadlock against al_write_transaction().
+ */
+ wait_event(device->al_wait, drbd_al_try_lock_for_transaction(device));
+
+ /* Take the bitmap lock before md_buffer. Whole-bitmap operations
+ * (drbd_bitmap_io()/w_bitmap_io()) hold the bitmap lock while their
+ * io_fn acquires md_buffer via drbd_md_sync().
+ */
+ if (device->bitmap)
+ drbd_bm_lock(device, __func__, BM_LOCK_ALL);
+
buffer = drbd_md_get_buffer(device, __func__); /* Lock meta-data IO */
if (!buffer) {
+ if (device->bitmap)
+ drbd_bm_unlock(device);
+ lc_unlock(device->act_log);
+ wake_up(&device->al_wait);
drbd_resume_io(device);
return DS_ERROR;
}
/* remember current offset and sizes */
- prev.last_agreed_sect = md->la_size_sect;
+ prev.effective_size = md->effective_size;
prev.md_offset = md->md_offset;
prev.al_offset = md->al_offset;
prev.bm_offset = md->bm_offset;
prev.md_size_sect = md->md_size_sect;
prev.al_stripes = md->al_stripes;
prev.al_stripe_size_4k = md->al_stripe_size_4k;
+ prev_size = get_capacity(device->vdisk);
+
+ /* We do some synchronous IO below, which may take some time.
+ * Clear the timer, to avoid scary "timer expired!" messages,
+ * "Superblock" is written out at least twice below, anyways.
+ */
+ timer_delete(&device->md_sync_timer);
if (rs) {
/* rs is non NULL if we should change the AL layout only */
@@ -1015,14 +1552,23 @@ drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct
md->al_size_4k = (u64)rs->al_stripes * rs->al_stripe_size / 4;
}
- drbd_md_set_sector_offsets(device, device->ldev);
+ drbd_md_set_sector_offsets(device->ldev);
rcu_read_lock();
u_size = rcu_dereference(device->ldev->disk_conf)->disk_size;
rcu_read_unlock();
- size = drbd_new_dev_size(device, device->ldev, u_size, flags & DDSF_FORCED);
+ if (flags & DDSF_2PC) {
+ /* Take the size the transaction agreed on. Deriving it again
+ * here would use this node's own view of who takes part, which
+ * is short of the initiator's, so a node that can not see the
+ * whole cluster would refuse a size everyone agreed to.
+ */
+ size = peer_current_size;
+ } else {
+ size = drbd_new_dev_size(device, 0, u_size, flags);
+ }
- if (size < prev.last_agreed_sect) {
+ if (size < prev.effective_size) {
if (rs && u_size == 0) {
/* Remove "rs &&" later. This check should always be active, but
right now the receiver expects the permissive behavior */
@@ -1038,9 +1584,25 @@ drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct
}
if (get_capacity(device->vdisk) != size ||
- drbd_bm_capacity(device) != size) {
- int err;
- err = drbd_bm_resize(device, size, !(flags & DDSF_NO_RESYNC));
+ (device->bitmap && drbd_bm_capacity(device) != size)) {
+ sector_t bm_capacity = device->bitmap ? drbd_bm_capacity(device) : 0;
+ int err = 0;
+
+ if (device->bitmap)
+ err = drbd_bm_resize(device, device->bitmap, size,
+ !(flags & DDSF_NO_RESYNC));
+ /* Without DDSF_NO_RESYNC the resize has set the new area in
+ * every slot. Otherwise the new area is unknown data to a
+ * backup made before.
+ */
+ if (!err && device->bitmap && size > bm_capacity && (flags & DDSF_NO_RESYNC)) {
+ struct drbd_bitmap *bm = device->bitmap;
+ int cbt_index = drbd_cbt_bitmap_index(device->ldev);
+
+ if (cbt_index != -1)
+ _drbd_bm_set_many_bits(device, cbt_index,
+ bm_sect_to_bit(bm, bm_capacity), -1UL);
+ }
if (unlikely(err)) {
/* currently there is only one error: ENOMEM! */
size = drbd_bm_capacity(device);
@@ -1052,36 +1614,47 @@ drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct
"Leaving size unchanged\n");
}
rv = DS_ERROR;
+ } else {
+ /* racy, see comments above. */
+ drbd_set_my_capacity(device, size);
+ if (effective_disk_size_determined(device)
+ && md->effective_size != size) {
+ char ppb[10];
+
+ drbd_info(device, "persisting effective size = %s (%llu KB)\n",
+ ppsize(ppb, size >> 1),
+ (unsigned long long)size >> 1);
+ md->effective_size = size;
+ }
}
- /* racy, see comments above. */
- drbd_set_my_capacity(device, size);
- md->la_size_sect = size;
}
if (rv <= DS_ERROR)
goto err_out;
- la_size_changed = (prev.last_agreed_sect != md->la_size_sect);
+ la_size_changed = (prev.effective_size != md->effective_size);
md_moved = prev.md_offset != md->md_offset
|| prev.md_size_sect != md->md_size_sect;
if (la_size_changed || md_moved || rs) {
- u32 prev_flags;
-
- /* We do some synchronous IO below, which may take some time.
- * Clear the timer, to avoid scary "timer expired!" messages,
- * "Superblock" is written out at least twice below, anyways. */
- timer_delete(&device->md_sync_timer);
-
- /* We won't change the "al-extents" setting, we just may need
- * to move the on-disk location of the activity log ringbuffer.
- * Lock for transaction is good enough, it may well be "dirty"
- * or even "starving". */
- wait_event(device->al_wait, lc_try_lock_for_transaction(device->act_log));
+ int i;
+ bool prev_al_disabled = 0;
+ u32 prev_peer_full_sync = 0;
+
+ if (drbd_md_dax_active(device->ldev)) {
+ if (drbd_dax_map(device->ldev)) {
+ drbd_err(device, "Could not remap DAX; aborting resize\n");
+ goto err_out;
+ }
+ }
/* mark current on-disk bitmap and activity log as unreliable */
- prev_flags = md->flags;
- md->flags |= MDF_FULL_SYNC | MDF_AL_DISABLED;
+ prev_al_disabled = !!(md->flags & MDF_AL_DISABLED);
+ md->flags |= MDF_AL_DISABLED;
+ for (i = 0; i < DRBD_PEERS_MAX; i++) {
+ if (test_and_set_bit(__MDF_PEER_FULL_SYNC, &md->peers[i].flags))
+ prev_peer_full_sync |= 1 << i;
+ }
drbd_md_write(device, buffer);
drbd_al_initialize(device, buffer);
@@ -1089,29 +1662,46 @@ drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct
drbd_info(device, "Writing the whole bitmap, %s\n",
la_size_changed && md_moved ? "size changed and md moved" :
la_size_changed ? "size changed" : "md moved");
- /* next line implicitly does drbd_suspend_io()+drbd_resume_io() */
- drbd_bitmap_io(device, md_moved ? &drbd_bm_write_all : &drbd_bm_write,
- "size changed", BM_LOCKED_MASK, NULL);
+ /* The in-memory bitmap is correct at this point: callers load it
+ * from disk before invoking drbd_determine_dev_size(), and
+ * drbd_bm_resize() above has marked the grown region dirty when
+ * set_new_bits was true. Write it to disk to update la_size and
+ * persist any resync markers for the newly grown region.
+ *
+ * The bitmap lock is already held and IO is suspended, so call
+ * the io_fn directly instead of going through drbd_bitmap_io().
+ */
+ if (device->bitmap) {
+ if (md_moved)
+ drbd_bm_write_all(device, NULL);
+ else
+ drbd_bm_write(device, NULL);
+ }
/* on-disk bitmap and activity log is authoritative again
* (unless there was an IO error meanwhile...) */
- md->flags = prev_flags;
+ if (!prev_al_disabled)
+ md->flags &= ~MDF_AL_DISABLED;
+ for (i = 0; i < DRBD_PEERS_MAX; i++) {
+ if (0 == (prev_peer_full_sync & (1 << i)))
+ clear_bit(__MDF_PEER_FULL_SYNC, &md->peers[i].flags);
+ }
drbd_md_write(device, buffer);
if (rs)
drbd_info(device, "Changed AL layout to al-stripes = %d, al-stripe-size-kB = %d\n",
- md->al_stripes, md->al_stripe_size_4k * 4);
+ md->al_stripes, md->al_stripe_size_4k * 4);
}
- if (size > prev.last_agreed_sect)
- rv = prev.last_agreed_sect ? DS_GREW : DS_GREW_FROM_ZERO;
- if (size < prev.last_agreed_sect)
+ if (size > prev_size)
+ rv = prev_size ? DS_GREW : DS_GREW_FROM_ZERO;
+ if (size < prev_size)
rv = DS_SHRUNK;
if (0) {
- err_out:
+err_out:
/* restore previous offset and sizes */
- md->la_size_sect = prev.last_agreed_sect;
+ md->effective_size = prev.effective_size;
md->md_offset = prev.md_offset;
md->al_offset = prev.al_offset;
md->bm_offset = prev.bm_offset;
@@ -1120,57 +1710,195 @@ drbd_determine_dev_size(struct drbd_device *device, enum dds_flags flags, struct
md->al_stripe_size_4k = prev.al_stripe_size_4k;
md->al_size_4k = (u64)prev.al_stripes * prev.al_stripe_size_4k;
}
+ drbd_md_put_buffer(device);
+ if (device->bitmap)
+ drbd_bm_unlock(device);
lc_unlock(device->act_log);
wake_up(&device->al_wait);
- drbd_md_put_buffer(device);
drbd_resume_io(device);
return rv;
}
-sector_t
-drbd_new_dev_size(struct drbd_device *device, struct drbd_backing_dev *bdev,
- sector_t u_size, int assume_peer_has_space)
+/**
+ * get_max_agreeable_size()
+ * @device: DRBD device
+ * @max: Pointer to store the maximum agreeable size in
+ * @twopc_reachable_nodes: Bitmap of reachable nodes from two-phase-commit reply
+ *
+ * Check if all peer devices that have bitmap slots assigned in the metadata
+ * are connected.
+ */
+static bool get_max_agreeable_size(struct drbd_device *device, uint64_t *max,
+ uint64_t twopc_reachable_nodes)
{
- sector_t p_size = device->p_size; /* partner's disk size. */
- sector_t la_size_sect = bdev->md.la_size_sect; /* last agreed size. */
- sector_t m_size; /* my size */
- sector_t size = 0;
-
- m_size = drbd_get_max_capacity(bdev);
+ int node_id;
+ bool all_known;
- if (device->state.conn < C_CONNECTED && assume_peer_has_space) {
- drbd_warn(device, "Resize while not connected was forced by the user!\n");
- p_size = m_size;
- }
+ all_known = true;
+ rcu_read_lock();
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &device->ldev->md.peers[node_id];
+ struct drbd_peer_device *peer_device;
- if (p_size && m_size) {
- size = min_t(sector_t, p_size, m_size);
- } else {
- if (la_size_sect) {
- size = la_size_sect;
- if (m_size && m_size < size)
- size = m_size;
- if (p_size && p_size < size)
- size = p_size;
+ if (device->ldev->md.node_id == node_id) {
+ dynamic_drbd_dbg(device, "my node_id: %u\n", node_id);
+ continue; /* skip myself... */
+ }
+ /* peer_device may be NULL if we don't have a connection to that node. */
+ peer_device = peer_device_by_node_id(device, node_id);
+ if (twopc_reachable_nodes & NODE_MASK(node_id)) {
+ uint64_t size = device->resource->twopc_reply.max_possible_size;
+
+ /* That is the minimum over all of them. Prefer the
+ * answer this peer gave us itself, where TWOPC_YES says
+ * it belongs to this transaction; a relay answers with
+ * the minimum over the nodes behind it, so neither is
+ * above what the peer can do. A cache pinned below a
+ * peer's own maximum keeps the cluster from growing
+ * into it: P_SIZES advertises the minimum over these.
+ */
+ if (peer_device &&
+ test_bit(TWOPC_YES, &peer_device->connection->flags) &&
+ peer_device->max_size > size)
+ size = peer_device->max_size;
+
+ dynamic_drbd_dbg(device, "node_id: %u, twopc YES for max_size: %llu\n",
+ node_id, (unsigned long long)size);
+
+ /* Update our cached information, they said "yes".
+ * Note:
+ * d_size == 0 indicates diskless peer, or not directly
+ * connected. It will be ignored by the min_not_zero()
+ * aggregation elsewhere. Only reset if size > d_size
+ * here. Once we really commit the change, this will
+ * also be assigned if it was a shrinkage.
+ */
+ if (peer_device) {
+ if (peer_device->d_size && size > peer_device->d_size)
+ peer_device->d_size = size;
+ if (size > peer_device->max_size)
+ peer_device->max_size = size;
+ }
+ continue;
+ }
+ if (peer_device) {
+ enum drbd_disk_state pdsk = peer_device->disk_state[NOW];
+
+ dynamic_drbd_dbg(peer_device, "node_id: %u idx: %u bm-uuid: 0x%llx flags: 0x%lx max_size: %llu (%s)\n",
+ node_id,
+ peer_md->bitmap_index,
+ peer_md->bitmap_uuid,
+ peer_md->flags,
+ peer_device->max_size,
+ drbd_disk_str(pdsk));
+
+ if (test_bit(HAVE_SIZES, peer_device->flags)) {
+ /* If we still can see it, consider its last
+ * known size, even if it may have meanwhile
+ * detached from its disk.
+ * If we no longer see it, we may want to
+ * ignore the size we last knew, and
+ * "assume_peer_has_space". */
+ *max = min_not_zero(*max, peer_device->max_size);
+ continue;
+ }
} else {
- if (m_size)
- size = m_size;
- if (p_size)
- size = p_size;
+ dynamic_drbd_dbg(device, "node_id: %u idx: %u bm-uuid: 0x%llx flags: 0x%lx (not currently reachable)\n",
+ node_id,
+ peer_md->bitmap_index,
+ peer_md->bitmap_uuid,
+ peer_md->flags);
}
- }
+ /* Even the currently diskless peer does not really know if it
+ * is diskless on purpose (a "DRBD client") or if it just was
+ * not possible to attach (backend device gone for some
+ * reason). But we remember in our meta data if we have ever
+ * seen a peer disk for this peer. If we did not ever see a
+ * peer disk, and it is not configured with a bitmap either,
+ * assume that's intentional.
+ */
+ if (!test_bit(__MDF_PEER_DEVICE_SEEN, &peer_md->flags) &&
+ !(peer_device && want_bitmap(peer_device)))
+ continue;
- if (size == 0)
- drbd_err(device, "Both nodes diskless!\n");
-
- if (u_size) {
- if (u_size > size)
- drbd_err(device, "Requested disk size is too big (%lu > %lu)\n",
- (unsigned long)u_size>>1, (unsigned long)size>>1);
- else
- size = u_size;
+ all_known = false;
+ /* don't break yet, min aggregation may still find a peer */
}
+ rcu_read_unlock();
+ return all_known;
+}
+
+#define DDUMP_LLU(d, x) do { dynamic_drbd_dbg(d, "%u: " #x ": %llu\n", __LINE__, (unsigned long long)x); } while (0)
+
+/* MUST hold a reference on ldev. */
+sector_t
+drbd_new_dev_size(struct drbd_device *device,
+ sector_t agreed_max_size, /* with DDSF_2PC: what the reachable nodes agreed to */
+ sector_t user_capped_size, /* want (at most) this much */
+ enum dds_flags flags)
+{
+ struct drbd_resource *resource = device->resource;
+ uint64_t p_size = 0;
+ uint64_t la_size = device->ldev->md.effective_size; /* last agreed size */
+ uint64_t m_size; /* my size */
+ uint64_t size = 0;
+ bool all_known_connected;
+
+ /* If there are reachable_nodes, get_max_agreeable_size() will
+ * also aggregate the twopc.resize.new_size into their d_size
+ * and max_size. Do that first, so drbd_partition_data_capacity()
+ * can use that new knowledge.
+ */
+
+ all_known_connected = get_max_agreeable_size(device, &p_size,
+ flags & DDSF_2PC ? resource->twopc_reply.reachable_nodes : 0);
+ /* A node the transaction reached through a relay may have no peer
+ * device here, or a diskless one, so its answer can be missing from
+ * every cache above. Take the aggregate as a term of its own, or the
+ * result could exceed what that node can do.
+ */
+ if (flags & DDSF_2PC)
+ p_size = min_not_zero(p_size, (uint64_t)agreed_max_size);
+ m_size = drbd_partition_data_capacity(device);
+
+ if (all_known_connected) {
+ /* If we currently can see all peer devices,
+ * and p_size is still 0, apparently all our peers have been
+ * diskless, always. If we have the only persistent backend,
+ * only our size counts. */
+ DDUMP_LLU(device, p_size);
+ DDUMP_LLU(device, m_size);
+ p_size = min_not_zero(p_size, m_size);
+ } else if (flags & DDSF_ASSUME_UNCONNECTED_PEER_HAS_SPACE) {
+ DDUMP_LLU(device, p_size);
+ DDUMP_LLU(device, m_size);
+ DDUMP_LLU(device, la_size);
+ p_size = min_not_zero(p_size, m_size);
+ if (p_size > la_size)
+ drbd_warn(device, "Resize forced while not fully connected!\n");
+ } else {
+ DDUMP_LLU(device, p_size);
+ DDUMP_LLU(device, m_size);
+ DDUMP_LLU(device, la_size);
+ /* We currently cannot see all peer devices,
+ * fall back to what we last agreed upon. */
+ p_size = min_not_zero(p_size, la_size);
+ }
+
+ DDUMP_LLU(device, p_size);
+ DDUMP_LLU(device, m_size);
+ size = min_not_zero(p_size, m_size);
+ DDUMP_LLU(device, size);
+ if (size == 0)
+ drbd_err(device, "All nodes diskless!\n");
+
+ if (user_capped_size > size)
+ drbd_err(device, "Requested disk size is too big (%llu > %llu)kiB\n",
+ (unsigned long long)user_capped_size>>1,
+ (unsigned long long)size>>1);
+ else if (user_capped_size)
+ size = user_capped_size;
return size;
}
@@ -1183,7 +1911,7 @@ drbd_new_dev_size(struct drbd_device *device, struct drbd_backing_dev *bdev,
* failed, and 0 on success. You should call drbd_md_sync() after you called
* this function.
*/
-static int drbd_check_al_size(struct drbd_device *device, struct disk_conf *dc)
+static int drbd_check_al_size(struct drbd_device *device, struct drbd_disk_conf *dc)
{
struct lru_cache *n, *t;
struct lc_element *e;
@@ -1222,57 +1950,58 @@ static int drbd_check_al_size(struct drbd_device *device, struct disk_conf *dc)
return -EBUSY;
} else {
lc_destroy(t);
+ device->al_writ_cnt = 0;
+ memset(device->al_histogram, 0, sizeof(device->al_histogram));
}
drbd_md_mark_dirty(device); /* we changed device->act_log->nr_elemens */
return 0;
}
-static unsigned int drbd_max_peer_bio_size(struct drbd_device *device)
+static u32 common_connection_features(struct drbd_resource *resource)
{
- /*
- * We may ignore peer limits if the peer is modern enough. From 8.3.8
- * onwards the peer can use multiple BIOs for a single peer_request.
- */
- if (device->state.conn < C_WF_REPORT_PARAMS)
- return device->peer_max_bio_size;
-
- if (first_peer_device(device)->connection->agreed_pro_version < 94)
- return min(device->peer_max_bio_size, DRBD_MAX_SIZE_H80_PACKET);
+ struct drbd_connection *connection;
+ u32 features = -1;
- /*
- * Correct old drbd (up to 8.3.7) if it believes it can do more than
- * 32KiB.
- */
- if (first_peer_device(device)->connection->agreed_pro_version == 94)
- return DRBD_MAX_SIZE_H80_PACKET;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->cstate[NOW] < C_CONNECTED)
+ continue;
+ features &= connection->agreed_features;
+ }
+ rcu_read_unlock();
- /*
- * drbd 8.3.8 onwards, before 8.4.0
- */
- if (first_peer_device(device)->connection->agreed_pro_version < 100)
- return DRBD_MAX_BIO_SIZE_P95;
- return DRBD_MAX_BIO_SIZE;
+ return features;
}
-static unsigned int drbd_max_discard_sectors(struct drbd_connection *connection)
+static unsigned int drbd_max_discard_sectors(struct drbd_resource *resource)
{
- /* when we introduced REQ_WRITE_SAME support, we also bumped
+ struct drbd_connection *connection;
+ unsigned int s = DRBD_MAX_BBIO_SECTORS;
+
+ /* when we introduced WRITE_SAME support, we also bumped
* our maximum supported batch bio size used for discards. */
- if (connection->agreed_features & DRBD_FF_WSAME)
- return DRBD_MAX_BBIO_SECTORS;
- /* before, with DRBD <= 8.4.6, we only allowed up to one AL_EXTENT_SIZE. */
- return AL_EXTENT_SIZE >> 9;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->cstate[NOW] == C_CONNECTED &&
+ !(connection->agreed_features & DRBD_FF_WSAME)) {
+ /* before, with DRBD <= 8.4.6, we only allowed up to one AL_EXTENT_SIZE. */
+ s = AL_EXTENT_SIZE >> SECTOR_SHIFT;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return s;
}
-static bool drbd_discard_supported(struct drbd_connection *connection,
+static bool drbd_discard_supported(struct drbd_device *device,
struct drbd_backing_dev *bdev)
{
if (bdev && !bdev_max_discard_sectors(bdev->backing_bdev))
return false;
- if (connection->cstate >= C_CONNECTED &&
- !(connection->agreed_features & DRBD_FF_TRIM)) {
- drbd_info(connection,
+ if (!(common_connection_features(device->resource) & DRBD_FF_TRIM)) {
+ drbd_info(device,
"peer DRBD too old, does not support TRIM: disabling discards\n");
return false;
}
@@ -1280,85 +2009,75 @@ static bool drbd_discard_supported(struct drbd_connection *connection,
return true;
}
-/* This is the workaround for "bio would need to, but cannot, be split" */
-static unsigned int drbd_backing_dev_max_segments(struct drbd_device *device)
+static void get_common_queue_limits(struct queue_limits *common_limits,
+ struct drbd_device *device)
{
- unsigned int max_segments;
+ struct drbd_peer_device *peer_device;
+ struct queue_limits peer_limits = { 0 };
+
+ blk_set_stacking_limits(common_limits);
+ common_limits->max_hw_sectors = device->device_conf.max_bio_size >> SECTOR_SHIFT;
+ common_limits->max_sectors = device->device_conf.max_bio_size >> SECTOR_SHIFT;
+ common_limits->physical_block_size = device->device_conf.block_size;
+ common_limits->logical_block_size = device->device_conf.block_size;
+ common_limits->io_min = device->device_conf.block_size;
+ common_limits->max_hw_zone_append_sectors = 0;
rcu_read_lock();
- max_segments = rcu_dereference(device->ldev->disk_conf)->max_bio_bvecs;
+ for_each_peer_device_rcu(peer_device, device) {
+ if (!test_bit(HAVE_SIZES, peer_device->flags) &&
+ peer_device->repl_state[NOW] < L_ESTABLISHED)
+ continue;
+ blk_set_stacking_limits(&peer_limits);
+ peer_limits.logical_block_size = peer_device->q_limits.logical_block_size;
+ peer_limits.physical_block_size = peer_device->q_limits.physical_block_size;
+ peer_limits.alignment_offset = peer_device->q_limits.alignment_offset;
+ peer_limits.io_min = peer_device->q_limits.io_min;
+ peer_limits.io_opt = peer_device->q_limits.io_opt;
+ peer_limits.max_hw_sectors = peer_device->q_limits.max_bio_size >> SECTOR_SHIFT;
+ peer_limits.max_sectors = peer_device->q_limits.max_bio_size >> SECTOR_SHIFT;
+ blk_stack_limits(common_limits, &peer_limits, 0);
+ }
rcu_read_unlock();
-
- if (!max_segments)
- return BLK_MAX_SEGMENTS;
- return max_segments;
}
-void drbd_reconsider_queue_parameters(struct drbd_device *device,
- struct drbd_backing_dev *bdev, struct o_qlim *o)
+void drbd_reconsider_queue_parameters(struct drbd_device *device, struct drbd_backing_dev *bdev)
{
- struct drbd_connection *connection =
- first_peer_device(device)->connection;
struct request_queue * const q = device->rq_queue;
- unsigned int now = queue_max_hw_sectors(q) << 9;
struct queue_limits lim;
struct request_queue *b = NULL;
- unsigned int new;
-
- if (bdev) {
- b = bdev->backing_bdev->bd_disk->queue;
-
- device->local_max_bio_size =
- queue_max_hw_sectors(b) << SECTOR_SHIFT;
- }
-
- /*
- * We may later detach and re-attach on a disconnected Primary. Avoid
- * decreasing the value in this case.
- *
- * We want to store what we know the peer DRBD can handle, not what the
- * peer IO backend can handle.
- */
- new = min3(DRBD_MAX_BIO_SIZE, device->local_max_bio_size,
- max(drbd_max_peer_bio_size(device), device->peer_max_bio_size));
- if (new != now) {
- if (device->state.role == R_PRIMARY && new < now)
- drbd_err(device, "ASSERT FAILED new < now; (%u < %u)\n",
- new, now);
- drbd_info(device, "max BIO size = %u\n", new);
- }
lim = queue_limits_start_update(q);
- if (bdev) {
- blk_set_stacking_limits(&lim);
- lim.max_segments = drbd_backing_dev_max_segments(device);
- } else {
- lim.max_segments = BLK_MAX_SEGMENTS;
- lim.features = BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA |
- BLK_FEAT_ROTATIONAL | BLK_FEAT_STABLE_WRITES;
- }
-
- lim.max_hw_sectors = new >> SECTOR_SHIFT;
- lim.seg_boundary_mask = PAGE_SIZE - 1;
+ get_common_queue_limits(&lim, device);
/*
- * We don't care for the granularity, really.
- *
- * Stacking limits below should fix it for the local device. Whether or
- * not it is a suitable granularity on the remote device is not our
- * problem, really. If you care, you need to use devices with similar
- * topology on all peers.
+ * discard_granularity == DRBD_DISCARD_GRANULARITY_DEF (sentinel):
+ * not explicitly configured; use the legacy heuristic
+ * (drbd_discard_supported decides, granularity=512).
+ * discard_granularity == 0: explicitly disable discards.
+ * discard_granularity > 0: use the configured value and enable discards
+ * unconditionally (e.g. LINSTOR knows the real granularity from
+ * storage pool info and configures it for diskless primaries or to
+ * advertise a larger granularity than strictly required).
*/
- if (drbd_discard_supported(connection, bdev)) {
- lim.discard_granularity = 512;
- lim.max_hw_discard_sectors =
- drbd_max_discard_sectors(connection);
+ if (device->device_conf.discard_granularity == DRBD_DISCARD_GRANULARITY_DEF) {
+ if (drbd_discard_supported(device, bdev)) {
+ lim.discard_granularity = 512;
+ lim.max_hw_discard_sectors = drbd_max_discard_sectors(device->resource);
+ } else {
+ lim.discard_granularity = 0;
+ lim.max_hw_discard_sectors = 0;
+ }
+ } else if (device->device_conf.discard_granularity) {
+ lim.discard_granularity = device->device_conf.discard_granularity;
+ lim.max_hw_discard_sectors = drbd_max_discard_sectors(device->resource);
} else {
lim.discard_granularity = 0;
lim.max_hw_discard_sectors = 0;
}
if (bdev) {
+ b = bdev->backing_bdev->bd_disk->queue;
blk_stack_limits(&lim, &b->limits, 0);
/*
* blk_set_stacking_limits() cleared the features, and
@@ -1375,14 +2094,28 @@ void drbd_reconsider_queue_parameters(struct drbd_device *device,
* receiver will detect a checksum mismatch.
*/
lim.features |= BLK_FEAT_STABLE_WRITES;
+
+ /*
+ * blk_stack_limits() uses max() for discard_granularity and
+ * min_not_zero() for max_hw_discard_sectors, both of which can
+ * re-enable discards from the backing device even when the user
+ * explicitly disabled them (discard_granularity == 0).
+ */
+ if (device->device_conf.discard_granularity == 0) {
+ lim.discard_granularity = 0;
+ lim.max_hw_discard_sectors = 0;
+ }
+ } else {
+ lim.features = BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA |
+ BLK_FEAT_ROTATIONAL | BLK_FEAT_STABLE_WRITES;
}
/*
- * If we can handle "zeroes" efficiently on the protocol, we want to do
- * that, even if our backend does not announce max_write_zeroes_sectors
- * itself.
+ * If we can handle "zeroes" efficiently on the protocol,
+ * we want to do that, even if our backend does not announce
+ * max_write_zeroes_sectors itself.
*/
- if (connection->agreed_features & DRBD_FF_WZEROES)
+ if (common_connection_features(device->resource) & DRBD_FF_WZEROES)
lim.max_write_zeroes_sectors = DRBD_MAX_BBIO_SECTORS;
else
lim.max_write_zeroes_sectors = 0;
@@ -1390,6 +2123,11 @@ void drbd_reconsider_queue_parameters(struct drbd_device *device,
if ((lim.discard_granularity >> SECTOR_SHIFT) >
lim.max_hw_discard_sectors) {
+ /*
+ * discard_granularity is the smallest supported unit of a
+ * discard. If that is larger than the maximum supported discard
+ * size, we need to disable discards altogether.
+ */
lim.discard_granularity = 0;
lim.max_hw_discard_sectors = 0;
}
@@ -1398,58 +2136,44 @@ void drbd_reconsider_queue_parameters(struct drbd_device *device,
drbd_err(device, "setting new queue limits failed\n");
}
-/* Starts the worker thread */
-static void conn_reconfig_start(struct drbd_connection *connection)
+/* Make sure IO is suspended before calling this function(). */
+static void drbd_try_suspend_al(struct drbd_device *device)
{
- drbd_thread_start(&connection->worker);
- drbd_flush_workqueue(&connection->sender_work);
-}
+ struct drbd_peer_device *peer_device;
+ bool suspend = true;
+ int max_peers = device->ldev->md.max_peers, bitmap_index;
-/* if still unconfigured, stops worker again. */
-static void conn_reconfig_done(struct drbd_connection *connection)
-{
- bool stop_threads;
- spin_lock_irq(&connection->resource->req_lock);
- stop_threads = conn_all_vols_unconf(connection) &&
- connection->cstate == C_STANDALONE;
- spin_unlock_irq(&connection->resource->req_lock);
- if (stop_threads) {
- /* ack_receiver thread and ack_sender workqueue are implicitly
- * stopped by receiver in conn_disconnect() */
- drbd_thread_stop(&connection->receiver);
- drbd_thread_stop(&connection->worker);
+ if (device->bitmap) {
+ for (bitmap_index = 0; bitmap_index < max_peers; bitmap_index++) {
+ if (_drbd_bm_total_weight(device, bitmap_index) != drbd_bm_bits(device))
+ return;
+ }
}
-}
-
-/* Make sure IO is suspended before calling this function(). */
-static void drbd_suspend_al(struct drbd_device *device)
-{
- int s = 0;
- if (!lc_try_lock(device->act_log)) {
- drbd_warn(device, "Failed to lock al in drbd_suspend_al()\n");
+ if (!drbd_al_try_lock(device)) {
+ drbd_warn(device, "Failed to lock al in %s()", __func__);
return;
}
drbd_al_shrink(device);
- spin_lock_irq(&device->resource->req_lock);
- if (device->state.conn < C_CONNECTED)
- s = !test_and_set_bit(AL_SUSPENDED, &device->flags);
- spin_unlock_irq(&device->resource->req_lock);
+ read_lock_irq(&device->resource->state_rwlock);
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED) {
+ suspend = false;
+ break;
+ }
+ }
+ if (suspend)
+ suspend = !test_and_set_bit(AL_SUSPENDED, &device->flags);
+ read_unlock_irq(&device->resource->state_rwlock);
lc_unlock(device->act_log);
+ wake_up(&device->al_wait);
- if (s)
+ if (suspend)
drbd_info(device, "Suspended AL updates\n");
}
-static bool should_set_defaults(struct genl_info *info)
-{
- struct drbd_genlmsghdr *dh = genl_info_userhdr(info);
-
- return 0 != (dh->flags & DRBD_GENL_F_SET_DEFAULTS);
-}
-
static unsigned int drbd_al_extents_max(struct drbd_backing_dev *bdev)
{
/* This is limited by 16 bit "slot" numbers,
@@ -1467,7 +2191,7 @@ static unsigned int drbd_al_extents_max(struct drbd_backing_dev *bdev)
*/
const unsigned int max_al_nr = DRBD_AL_EXTENTS_MAX;
const unsigned int sufficient_on_disk =
- (max_al_nr + AL_CONTEXT_PER_TRANSACTION -1)
+ (max_al_nr + AL_CONTEXT_PER_TRANSACTION - 1)
/AL_CONTEXT_PER_TRANSACTION;
unsigned int al_size_4k = bdev->md.al_size_4k;
@@ -1478,14 +2202,14 @@ static unsigned int drbd_al_extents_max(struct drbd_backing_dev *bdev)
return (al_size_4k - 1) * AL_CONTEXT_PER_TRANSACTION;
}
-static bool write_ordering_changed(struct disk_conf *a, struct disk_conf *b)
+static bool write_ordering_changed(struct drbd_disk_conf *a, struct drbd_disk_conf *b)
{
return a->disk_barrier != b->disk_barrier ||
a->disk_flushes != b->disk_flushes ||
a->disk_drain != b->disk_drain;
}
-static void sanitize_disk_conf(struct drbd_device *device, struct disk_conf *disk_conf,
+static void sanitize_disk_conf(struct drbd_device *device, struct drbd_disk_conf *disk_conf,
struct drbd_backing_dev *nbc)
{
struct block_device *bdev = nbc->backing_bdev;
@@ -1502,29 +2226,51 @@ static void sanitize_disk_conf(struct drbd_device *device, struct disk_conf *dis
}
}
+ /* To be effective, rs_discard_granularity must not be larger than the
+ * maximum resync request size, and multiple of 4k
+ * (preferably a power-of-two multiple 4k).
+ * See also make_resync_request().
+ * That also means that if q->limits.discard_granularity or
+ * q->limits.discard_alignment are "odd", rs_discard_granularity won't
+ * be particularly effective, or not effective at all.
+ */
if (disk_conf->rs_discard_granularity) {
- int orig_value = disk_conf->rs_discard_granularity;
- sector_t discard_size = bdev_max_discard_sectors(bdev) << 9;
+ unsigned int new_discard_granularity =
+ disk_conf->rs_discard_granularity;
+ unsigned int discard_sectors = bdev_max_discard_sectors(bdev);
unsigned int discard_granularity = bdev_discard_granularity(bdev);
- int remainder;
-
- if (discard_granularity > disk_conf->rs_discard_granularity)
- disk_conf->rs_discard_granularity = discard_granularity;
-
- remainder = disk_conf->rs_discard_granularity %
- discard_granularity;
- disk_conf->rs_discard_granularity += remainder;
-
- if (disk_conf->rs_discard_granularity > discard_size)
- disk_conf->rs_discard_granularity = discard_size;
- if (disk_conf->rs_discard_granularity != orig_value)
+ /* should be at least the discard_granularity of the bdev,
+ * and preferably a multiple (or the backend won't be able to
+ * discard some of the "cuttings").
+ * This also sanitizes nonsensical settings like "77 byte".
+ */
+ new_discard_granularity = roundup(new_discard_granularity,
+ discard_granularity);
+
+ /* more than the max resync request size won't work anyways */
+ discard_sectors = min(discard_sectors,
+ DRBD_RS_DISCARD_GRANULARITY_MAX >> SECTOR_SHIFT);
+ /* Avoid compiler warning about truncated integer.
+ * The min() above made sure the result fits even after left shift. */
+ new_discard_granularity = min(
+ new_discard_granularity >> SECTOR_SHIFT,
+ discard_sectors) << SECTOR_SHIFT;
+ /* less than the backend discard granularity is allowed if
+ the backend granularity is a multiple of the configured value */
+ if (new_discard_granularity < discard_granularity &&
+ discard_granularity % new_discard_granularity != 0)
+ new_discard_granularity = 0;
+
+ if (disk_conf->rs_discard_granularity != new_discard_granularity) {
drbd_info(device, "rs_discard_granularity changed to %d\n",
- disk_conf->rs_discard_granularity);
+ new_discard_granularity);
+ disk_conf->rs_discard_granularity = new_discard_granularity;
+ }
}
}
-static int disk_opts_check_al_size(struct drbd_device *device, struct disk_conf *dc)
+static int disk_opts_check_al_size(struct drbd_device *device, struct drbd_disk_conf *dc)
{
int err = -EBUSY;
@@ -1532,13 +2278,13 @@ static int disk_opts_check_al_size(struct drbd_device *device, struct disk_conf
device->act_log->nr_elements == dc->al_extents)
return 0;
- drbd_suspend_io(device);
+ drbd_suspend_io(device, READ_AND_WRITE);
/* If IO completion is currently blocked, we would likely wait
* "forever" for the activity log to become unused. So we don't. */
- if (atomic_read(&device->ap_bio_cnt))
+ if (atomic_read(&device->ap_bio_cnt[WRITE]) || atomic_read(&device->ap_bio_cnt[READ]))
goto out;
- wait_event(device->al_wait, lc_try_lock(device->act_log));
+ wait_event(device->al_wait, drbd_al_try_lock(device));
drbd_al_shrink(device);
err = drbd_check_al_size(device, dc);
lc_unlock(device->act_log);
@@ -1548,25 +2294,122 @@ static int disk_opts_check_al_size(struct drbd_device *device, struct disk_conf
return err;
}
-int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info)
+static struct drbd_connection *the_only_peer_with_disk(struct drbd_device *device,
+ enum which_state which)
+{
+ const int my_node_id = device->resource->res_opts.node_id;
+ struct drbd_peer_md *peer_md = device->ldev->md.peers;
+ struct drbd_connection *connection = NULL;
+ struct drbd_peer_device *peer_device;
+ int node_id, peer_disks = 0;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (node_id == my_node_id)
+ continue;
+
+ if (test_bit(__MDF_PEER_DEVICE_SEEN, &peer_md[node_id].flags))
+ peer_disks++;
+
+ if (peer_disks > 1)
+ return NULL;
+
+ peer_device = peer_device_by_node_id(device, node_id);
+ if (peer_device) {
+ enum drbd_disk_state pdsk = peer_device->disk_state[which];
+
+ if (pdsk >= D_INCONSISTENT && pdsk != D_UNKNOWN)
+ connection = peer_device->connection;
+ }
+ }
+ return connection;
+}
+
+static void __update_mdf_al_disabled(struct drbd_device *device, bool al_updates,
+ enum which_state which)
+{
+ struct drbd_md *md = &device->ldev->md;
+ struct drbd_connection *peer = NULL;
+ bool al_updates_old = !(md->flags & MDF_AL_DISABLED) ||
+ test_bit(AL_ENABLE_PENDING, &device->flags);
+ bool optimized = false;
+
+ if (al_updates)
+ peer = the_only_peer_with_disk(device, which);
+
+ /* With CRASHED_PRIMARY set, the next attach after a crash reads a
+ * disabled AL as "crashed as Primary without AL" and sets all bits.
+ */
+ if (device->bitmap == NULL ||
+ (al_updates && device->ldev->md.max_peers == 1 &&
+ peer && peer->peer_role[which] == R_PRIMARY &&
+ device->resource->role[which] == R_SECONDARY &&
+ !test_bit(CRASHED_PRIMARY, &device->flags))) {
+ al_updates = false;
+ optimized = true;
+ }
+
+ if (al_updates_old == al_updates)
+ return;
+
+ if (al_updates) {
+ drbd_info(device, "Enabling local AL-updates\n");
+ /* Extents that became active meanwhile have no transaction;
+ * drbd_al_begin_io_commit() writes them, then clears the flag.
+ */
+ if (!drbd_md_dax_active(device->ldev)) {
+ set_bit(AL_ENABLE_PENDING, &device->flags);
+ return;
+ }
+ md->flags &= ~MDF_AL_DISABLED;
+ } else {
+ drbd_info(device, "Disabling local AL-updates %s\n",
+ optimized ? "(optimization)" : "(config)");
+ clear_bit(AL_ENABLE_PENDING, &device->flags);
+ md->flags |= MDF_AL_DISABLED;
+ }
+ drbd_md_mark_dirty(device);
+}
+
+/**
+ * drbd_update_mdf_al_disabled() - update the MDF_AL_DISABLED bit in md.flags
+ * @device: DRBD device
+ * @which: OLD or NEW
+ *
+ * This function also optimizes performance by turning off al-updates when:
+ * - the cluster has only two nodes with backing disk
+ * - the other node with a backing disk is the primary
+ * - this node is not a crashed primary
+ */
+void drbd_update_mdf_al_disabled(struct drbd_device *device, enum which_state which)
+{
+ bool al_updates;
+
+ if (!get_ldev(device))
+ return;
+
+ rcu_read_lock();
+ al_updates = rcu_dereference(device->ldev->disk_conf)->al_updates;
+ rcu_read_unlock();
+ __update_mdf_al_disabled(device, al_updates, which);
+
+ put_ldev(device);
+}
+
+int drbd_adm_disk_opts(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
+ enum drbd_ret_code retcode = NO_ERROR;
struct drbd_device *device;
- struct disk_conf *new_disk_conf, *old_disk_conf;
- struct fifo_buffer *old_plan = NULL, *new_plan = NULL;
- struct nlattr **ntb;
+ struct drbd_resource *resource;
+ struct drbd_disk_conf *new_disk_conf, *old_disk_conf;
+ struct drbd_peer_device *peer_device;
int err;
- unsigned int fifo_size;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto finish;
device = adm_ctx->device;
- mutex_lock(&adm_ctx->resource->adm_mutex);
+ resource = device->resource;
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
/* we also need a disk
* to change the options on */
@@ -1575,69 +2418,83 @@ int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info)
goto out;
}
- new_disk_conf = kmalloc_obj(struct disk_conf);
+ new_disk_conf = kmalloc_obj(struct drbd_disk_conf);
if (!new_disk_conf) {
retcode = ERR_NOMEM;
goto fail;
}
- mutex_lock(&device->resource->conf_update);
+ mutex_lock(&resource->conf_update);
old_disk_conf = device->ldev->disk_conf;
*new_disk_conf = *old_disk_conf;
- if (should_set_defaults(info))
- set_disk_conf_defaults(new_disk_conf);
+ if (adm_ctx->set_defaults)
+ drbd_set_disk_conf_defaults(new_disk_conf);
- err = disk_conf_from_attrs(new_disk_conf, info);
+ err = drbd_adm_overlay_disk_conf(adm_ctx, new_disk_conf);
if (err && err != -ENOMSG) {
retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
+ drbd_adm_msg_overlay_error(adm_ctx, err);
goto fail_unlock;
}
- err = disk_conf_ntb_from_attrs(&ntb, info);
- if (!err) {
- if (has_invariant(ntb, DRBD_A_DISK_CONF_BACKING_DEV) ||
- has_invariant(ntb, DRBD_A_DISK_CONF_META_DEV) ||
- has_invariant(ntb, DRBD_A_DISK_CONF_META_DEV_IDX) ||
- has_invariant(ntb, DRBD_A_DISK_CONF_DISK_SIZE) ||
- has_invariant(ntb, DRBD_A_DISK_CONF_MAX_BIO_BVECS)) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb,
- "cannot change invariant setting");
- kfree(ntb);
- goto fail_unlock;
- }
- kfree(ntb);
+ if (adm_ctx->d->attr_present(adm_ctx, DRBD_ADM_F_DISK_BACKING_DEV) ||
+ adm_ctx->d->attr_present(adm_ctx, DRBD_ADM_F_DISK_META_DEV) ||
+ adm_ctx->d->attr_present(adm_ctx, DRBD_ADM_F_DISK_META_DEV_IDX) ||
+ adm_ctx->d->attr_present(adm_ctx, DRBD_ADM_F_DISK_SIZE)) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg(adm_ctx, "%s", "cannot change invariant setting");
+ goto fail_unlock;
}
- if (!expect(device, new_disk_conf->resync_rate >= 1))
- new_disk_conf->resync_rate = 1;
-
sanitize_disk_conf(device, new_disk_conf, device->ldev);
- if (new_disk_conf->c_plan_ahead > DRBD_C_PLAN_AHEAD_MAX)
- new_disk_conf->c_plan_ahead = DRBD_C_PLAN_AHEAD_MAX;
-
- fifo_size = (new_disk_conf->c_plan_ahead * 10 * SLEEP_TIME) / HZ;
- if (fifo_size != device->rs_plan_s->size) {
- new_plan = fifo_alloc(fifo_size);
- if (!new_plan) {
- drbd_err(device, "kmalloc of fifo_buffer failed");
- retcode = ERR_NOMEM;
- goto fail_unlock;
- }
- }
-
err = disk_opts_check_al_size(device, new_disk_conf);
if (err) {
/* Could be just "busy". Ignore?
* Introduce dedicated error code? */
- drbd_msg_put_info(adm_ctx->reply_skb,
+ drbd_adm_msg(adm_ctx, "%s",
"Try again without changing current al-extents setting");
retcode = ERR_NOMEM;
goto fail_unlock;
}
+ if (!old_disk_conf->d_bitmap && new_disk_conf->d_bitmap) {
+ struct drbd_md *md = &device->ldev->md;
+ struct drbd_bitmap *bitmap;
+
+ bitmap = drbd_bm_alloc(md->max_peers, md->bm_block_shift);
+ if (!bitmap) {
+ drbd_adm_msg(adm_ctx, "%s", "Failed to allocate bitmap");
+ retcode = ERR_NOMEM;
+ goto fail_unlock;
+ }
+ _drbd_bm_lock(device, bitmap, NULL, __func__, BM_LOCK_ALL);
+ err = drbd_bm_resize(device, bitmap, get_capacity(device->vdisk), true);
+ _drbd_bm_unlock(device, bitmap);
+ if (err) {
+ kfree(bitmap);
+ drbd_adm_msg(adm_ctx, "%s", "Failed to allocate bitmap pages");
+ retcode = ERR_NOMEM;
+ goto fail_unlock;
+ }
+
+ /* Publish only after bm_pages is populated, otherwise readers
+ * in __bm_op() can observe device->bitmap != NULL with
+ * bm_pages == NULL and trip the assertion. No
+ * smp_load_acquire() needed: bm_pages is always read under
+ * bitmap->bm_lock, and lockless NULL-checks are ordered by
+ * address dependency.
+ */
+ smp_store_release(&device->bitmap, bitmap);
+
+ drbd_bitmap_io(device, &drbd_bm_write, "write from disk_opts", BM_LOCK_ALL, NULL);
+ } else if (old_disk_conf->d_bitmap && !new_disk_conf->d_bitmap) {
+ /* That would be quite some effort, and there is no use case for this */
+ drbd_adm_msg(adm_ctx, "%s", "Online freeing of the bitmap not supported");
+ retcode = ERR_INVALID_REQUEST;
+ goto fail_unlock;
+ }
+
lock_all_resources();
retcode = drbd_resync_after_valid(device, new_disk_conf->resync_after);
if (retcode == NO_ERROR) {
@@ -1649,1704 +2506,4412 @@ int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info)
if (retcode != NO_ERROR)
goto fail_unlock;
- if (new_plan) {
- old_plan = device->rs_plan_s;
- rcu_assign_pointer(device->rs_plan_s, new_plan);
- }
-
- mutex_unlock(&device->resource->conf_update);
+ mutex_unlock(&resource->conf_update);
- if (new_disk_conf->al_updates)
- device->ldev->md.flags &= ~MDF_AL_DISABLED;
- else
- device->ldev->md.flags |= MDF_AL_DISABLED;
-
- if (new_disk_conf->md_flushes)
- clear_bit(MD_NO_FUA, &device->flags);
- else
- set_bit(MD_NO_FUA, &device->flags);
+ assign_bit(MD_NO_FUA, &device->flags, !new_disk_conf->md_flushes);
if (write_ordering_changed(old_disk_conf, new_disk_conf))
- drbd_bump_write_ordering(device->resource, NULL, WO_BDEV_FLUSH);
+ drbd_bump_write_ordering(device->resource, NULL, WO_BIO_BARRIER);
if (old_disk_conf->discard_zeroes_if_aligned !=
new_disk_conf->discard_zeroes_if_aligned)
- drbd_reconsider_queue_parameters(device, device->ldev, NULL);
-
- drbd_md_sync(device);
+ drbd_reconsider_queue_parameters(device, device->ldev);
- if (device->state.conn >= C_CONNECTED) {
- struct drbd_peer_device *peer_device;
+ /* No extent becomes active without a transaction before the superblock
+ * says AL updates are off.
+ */
+ wait_event(device->al_wait, drbd_al_try_lock_for_transaction(device));
+ __update_mdf_al_disabled(device, new_disk_conf->al_updates, NOW);
+ drbd_md_sync_if_dirty(device);
+ lc_unlock(device->act_log);
+ wake_up(&device->al_wait);
- for_each_peer_device(peer_device, device)
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED)
drbd_send_sync_param(peer_device);
}
kvfree_rcu_mightsleep(old_disk_conf);
- kfree(old_plan);
mod_timer(&device->request_timer, jiffies + HZ);
goto success;
fail_unlock:
- mutex_unlock(&device->resource->conf_update);
+ mutex_unlock(&resource->conf_update);
fail:
kfree(new_disk_conf);
- kfree(new_plan);
success:
+ if (retcode != NO_ERROR)
+ synchronize_rcu();
put_ldev(device);
out:
mutex_unlock(&adm_ctx->resource->adm_mutex);
- finish:
- adm_ctx->reply_dh->ret_code = retcode;
+out_no_adm_mutex:
+ adm_ctx->result = retcode;
return 0;
}
-static struct file *open_backing_dev(struct drbd_device *device,
- const char *bdev_path, void *claim_ptr, bool do_bd_link)
+static void mutex_unlock_cond(struct mutex *mutex, bool *have_mutex)
{
- struct file *file;
- int err = 0;
-
- file = bdev_file_open_by_path(bdev_path, BLK_OPEN_READ | BLK_OPEN_WRITE,
- claim_ptr, NULL);
- if (IS_ERR(file)) {
- drbd_err(device, "open(\"%s\") failed with %ld\n",
- bdev_path, PTR_ERR(file));
- return file;
+ if (*have_mutex) {
+ mutex_unlock(mutex);
+ *have_mutex = false;
}
+}
- if (!do_bd_link)
- return file;
+static void update_resource_dagtag(struct drbd_resource *resource, struct drbd_backing_dev *bdev)
+{
+ u64 dagtag = 0;
+ int node_id;
- err = bd_link_disk_holder(file_bdev(file), device->vdisk);
- if (err) {
- fput(file);
- drbd_err(device, "bd_link_disk_holder(\"%s\", ...) failed with %d\n",
- bdev_path, err);
- file = ERR_PTR(err);
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md;
+
+ if (bdev->md.node_id == node_id)
+ continue;
+
+ peer_md = &bdev->md.peers[node_id];
+
+ if (peer_md->bitmap_uuid)
+ dagtag = max(peer_md->bitmap_dagtag, dagtag);
}
- return file;
+
+ spin_lock_irq(&resource->tl_update_lock);
+ if (dagtag > resource->dagtag_sector) {
+ if (resource->dagtag_sector != resource->dagtag_from_backing_dev)
+ resource->dagtag_before_attach = resource->dagtag_sector;
+ resource->dagtag_from_backing_dev = dagtag;
+ WRITE_ONCE(resource->dagtag_sector, dagtag);
+ }
+ spin_unlock_irq(&resource->tl_update_lock);
}
-static int open_backing_devices(struct drbd_device *device,
- struct disk_conf *new_disk_conf,
- struct drbd_backing_dev *nbc)
+static int used_bitmap_slots(struct drbd_backing_dev *bdev)
{
- struct file *file;
+ int node_id;
+ int used = 0;
- file = open_backing_dev(device, new_disk_conf->backing_dev, device,
- true);
- if (IS_ERR(file))
- return ERR_OPEN_DISK;
- nbc->backing_bdev = file_bdev(file);
- nbc->backing_bdev_file = file;
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &bdev->md.peers[node_id];
- /*
- * meta_dev_idx >= 0: external fixed size, possibly multiple
- * drbd sharing one meta device. TODO in that case, paranoia
- * check that [md_bdev, meta_dev_idx] is not yet used by some
- * other drbd minor! (if you use drbd.conf + drbdadm, that
- * should check it for you already; but if you don't, or
- * someone fooled it, we need to double check here)
- */
- file = open_backing_dev(device, new_disk_conf->meta_dev,
- /* claim ptr: device, if claimed exclusively; shared drbd_m_holder,
- * if potentially shared with other drbd minors */
- (new_disk_conf->meta_dev_idx < 0) ? (void*)device : (void*)drbd_m_holder,
- /* avoid double bd_claim_by_disk() for the same (source,target) tuple,
- * as would happen with internal metadata. */
- (new_disk_conf->meta_dev_idx != DRBD_MD_INDEX_FLEX_INT &&
- new_disk_conf->meta_dev_idx != DRBD_MD_INDEX_INTERNAL));
- if (IS_ERR(file))
- return ERR_OPEN_MD_DISK;
- nbc->md_bdev = file_bdev(file);
- nbc->f_md_bdev = file;
- return NO_ERROR;
+ if (test_bit(__MDF_HAVE_BITMAP, &peer_md->flags))
+ used++;
+ }
+ if (drbd_cbt_bitmap_index(bdev) != -1)
+ used++;
+
+ return used;
}
-static void close_backing_dev(struct drbd_device *device,
- struct file *bdev_file, bool do_bd_unlink)
+static bool bitmap_index_vacant(struct drbd_backing_dev *bdev, int bitmap_index)
{
- if (!bdev_file)
- return;
- if (do_bd_unlink)
- bd_unlink_disk_holder(file_bdev(bdev_file), device->vdisk);
- fput(bdev_file);
+ int node_id;
+
+ if (drbd_cbt_bitmap_index(bdev) == bitmap_index)
+ return false;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &bdev->md.peers[node_id];
+
+ if (peer_md->bitmap_index == bitmap_index)
+ return false;
+ }
+ return true;
}
-void drbd_backing_dev_free(struct drbd_device *device, struct drbd_backing_dev *ldev)
+int drbd_unallocated_index(struct drbd_backing_dev *bdev)
{
- if (ldev == NULL)
- return;
+ int bitmap_index;
+ int bm_max_peers = bdev->md.max_peers;
- close_backing_dev(device, ldev->f_md_bdev,
- ldev->md_bdev != ldev->backing_bdev);
- close_backing_dev(device, ldev->backing_bdev_file, true);
+ for (bitmap_index = 0; bitmap_index < bm_max_peers; bitmap_index++) {
+ if (bitmap_index_vacant(bdev, bitmap_index))
+ return bitmap_index;
+ }
- kfree(ldev->disk_conf);
- kfree(ldev);
+ return -1;
}
-int drbd_nl_attach_doit(struct sk_buff *skb, struct genl_info *info)
+static int
+allocate_bitmap_index(struct drbd_peer_device *peer_device,
+ struct drbd_backing_dev *nbc)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct drbd_device *device;
- struct drbd_peer_device *peer_device;
- struct drbd_connection *connection;
- int err;
- enum drbd_ret_code retcode;
- enum determine_dev_size dd;
- sector_t max_possible_sectors;
+ const int peer_node_id = peer_device->connection->peer_node_id;
+ struct drbd_peer_md *peer_md = &nbc->md.peers[peer_node_id];
+ int bitmap_index;
+
+ bitmap_index = drbd_unallocated_index(nbc);
+ if (bitmap_index == -1) {
+ drbd_err(peer_device, "Not enough free bitmap slots\n");
+ return -ENOSPC;
+ }
+
+ peer_md->bitmap_index = bitmap_index;
+ peer_device->bitmap_index = bitmap_index;
+ set_bit(__MDF_HAVE_BITMAP, &peer_md->flags);
+ /* The slot comes with the day-0 tracking bits of an unallocated slot,
+ * or with whatever is on disk; neither says who set them. A record
+ * this node id kept from an earlier peer does not describe them.
+ */
+ peer_md->placeholder_src = 0;
+ peer_md->placeholder_src_complete = false;
+
+ return 0;
+}
+
+static struct drbd_peer_md *day0_peer_md(struct drbd_device *device, int exclude)
+{
+ int node_id = drbd_day0_node_id(&device->ldev->md, exclude);
+
+ return node_id == -1 ? NULL : &device->ldev->md.peers[node_id];
+}
+
+/* Clear the flags in "mask", one bit at a time, so that a concurrent set_bit()
+ * on a flag outside the mask is not lost.
+ */
+static void clear_peer_md_flags(struct drbd_peer_md *peer_md, u32 mask)
+{
+ unsigned long bits = mask;
+ int bit;
+
+ for_each_set_bit(bit, &bits, 32)
+ clear_bit(bit, &peer_md->flags);
+}
+
+/*
+ * Clear the slot for this peer in the metadata. If md_flags is empty, clear
+ * the slot completely. Otherwise make it a slot for a diskless peer. Also
+ * clear any bitmap associated with this peer.
+ */
+static int clear_peer_slot(struct drbd_device *device, int peer_node_id, u32 md_flags)
+{
+ struct drbd_peer_md *peer_md, *day0_md;
+ struct meta_data_on_disk_9 *buffer;
+ int from_index, freed_index;
+ bool free_bitmap_slot;
+
+ if (!get_ldev(device))
+ return -ENODEV;
+
+ peer_md = &device->ldev->md.peers[peer_node_id];
+ free_bitmap_slot = test_bit(__MDF_HAVE_BITMAP, &peer_md->flags);
+ if (free_bitmap_slot) {
+ drbd_suspend_io(device, WRITE_ONLY);
+
+ /*
+ * Unallocated slots are considered to track writes to the
+ * device since day 0. In order to keep that promise, copy the
+ * bitmap from an unallocated slot to this one, or set it to
+ * all out-of-sync.
+ */
+
+ from_index = drbd_unallocated_index(device->ldev);
+ freed_index = peer_md->bitmap_index;
+
+ /* Take the bitmap lock before md_buffer. Correct order. */
+ drbd_bm_lock(device, __func__, BM_LOCK_BULK);
+ }
+ buffer = drbd_md_get_buffer(device, __func__); /* lock meta-data IO to superblock */
+ if (buffer == NULL)
+ goto out_no_buffer;
+
+ /* Look for day0 UUID before changing this peer slot to a day0 slot. */
+ day0_md = day0_peer_md(device, peer_node_id);
+
+ clear_peer_md_flags(peer_md, ~md_flags | MDF_HAVE_BITMAP);
+ peer_md->bitmap_index = -1;
+
+ if (free_bitmap_slot) {
+ /*
+ * Regular bitmap OPs (calling into bm_op()) can run in parallel to
+ * drbd_bm_copy_slot() and interleave with it as drbd_bm_copy_slot()
+ * gives up its locks when it moves on to the next source page.
+ * The bitmap->bm_all_slots_lock ensures that drbd_set_sync()
+ * (which iterates over multiple slots) does not interleave with
+ * drbd_bm_copy_slot() while it copies data from one slot to another
+ * one.
+ */
+ if (from_index != -1)
+ drbd_bm_copy_slot(device, from_index, freed_index);
+ else
+ _drbd_bm_set_many_bits(device, freed_index, 0, -1UL);
+
+ drbd_bm_write(device, NULL);
+ }
+
+ /*
+ * When we forget a peer, we clear the flags. In this case, reset the
+ * bitmap UUID to the day0 UUID. Peer slots without any bitmap index or
+ * any flags set should always contain the day0 UUID.
+ */
+ if (!peer_md->flags && day0_md) {
+ /* Assign directly, not via drbd_set_peer_bitmap_uuid(): a day0 slot must
+ * keep flags == 0, and is implicitly a divergence bitmap through the
+ * !MDF_HAVE_BITMAP path, so it must not gain MDF_PEER_DIVERGENCE_BITMAP.
+ */
+ peer_md->bitmap_uuid = day0_md->bitmap_uuid;
+ peer_md->bitmap_dagtag = day0_md->bitmap_dagtag;
+ } else {
+ drbd_set_peer_bitmap_uuid(peer_md, 0, 0);
+ }
+
+ clear_bit(MD_DIRTY, &device->flags);
+ drbd_md_write(device, buffer);
+ drbd_md_put_buffer(device);
+
+ out_no_buffer:
+ if (free_bitmap_slot) {
+ drbd_bm_unlock(device);
+ drbd_resume_io(device);
+ }
+
+ put_ldev(device);
+
+ return 0;
+}
+
+bool want_bitmap(struct drbd_peer_device *peer_device)
+{
+ struct drbd_peer_device_conf *pdc;
+ bool want_bitmap = false;
+
+ rcu_read_lock();
+ pdc = rcu_dereference(peer_device->conf);
+ if (pdc)
+ want_bitmap |= pdc->bitmap;
+ rcu_read_unlock();
+
+ return want_bitmap;
+}
+
+static void close_backing_dev(struct drbd_device *device,
+ struct file *bdev_file, bool do_bd_unlink)
+{
+ if (!bdev_file)
+ return;
+ if (do_bd_unlink)
+ bd_unlink_disk_holder(file_bdev(bdev_file), device->vdisk);
+ fput(bdev_file);
+}
+
+void drbd_backing_dev_free(struct drbd_device *device, struct drbd_backing_dev *ldev)
+{
+ if (ldev == NULL)
+ return;
+
+ drbd_dax_close(ldev);
+
+ close_backing_dev(device,
+ ldev->f_md_bdev,
+ ldev->md_bdev != ldev->backing_bdev);
+ close_backing_dev(device, ldev->backing_bdev_file, true);
+
+ kfree(ldev->disk_conf);
+ kfree(ldev);
+}
+
+static struct file *open_backing_dev(struct drbd_device *device,
+ const char *bdev_path, void *claim_ptr)
+{
+ struct file *file = bdev_file_open_by_path(bdev_path,
+ BLK_OPEN_READ | BLK_OPEN_WRITE,
+ claim_ptr, NULL);
+ if (IS_ERR(file)) {
+ drbd_err(device, "open(\"%s\") failed with %ld\n",
+ bdev_path, PTR_ERR(file));
+ }
+ return file;
+}
+
+static int link_backing_dev(struct drbd_device *device,
+ const char *bdev_path, struct file *file)
+{
+ int err = bd_link_disk_holder(file_bdev(file), device->vdisk);
+
+ if (err) {
+ drbd_err(device, "bd_link_disk_holder(\"%s\", ...) failed with %d\n",
+ bdev_path, err);
+ }
+ return err;
+}
+
+static int open_backing_devices(struct drbd_device *device,
+ struct drbd_disk_conf *new_disk_conf,
+ struct drbd_backing_dev *nbc)
+{
+ struct file *file;
+ void *meta_claim_ptr;
+ int err;
+
+ file = open_backing_dev(device, new_disk_conf->backing_dev, device);
+ if (IS_ERR(file))
+ return ERR_OPEN_DISK;
+
+ err = link_backing_dev(device, new_disk_conf->backing_dev, file);
+ if (err) {
+ /* close without unlinking; otherwise error path will try to unlink */
+ close_backing_dev(device, file, false);
+ return ERR_OPEN_DISK;
+ }
+ nbc->backing_bdev = file_bdev(file);
+ nbc->backing_bdev_file = file;
+
+ /* meta_claim_ptr: device, if claimed exclusively; shared drbd_m_holder,
+ * if potentially shared with other drbd minors
+ */
+ meta_claim_ptr = (new_disk_conf->meta_dev_idx < 0) ?
+ (void *)device : (void *)drbd_m_holder;
+ /*
+ * meta_dev_idx >= 0: external fixed size, possibly multiple
+ * drbd sharing one meta device. TODO in that case, paranoia
+ * check that [md_bdev, meta_dev_idx] is not yet used by some
+ * other drbd minor! (if you use drbd.conf + drbdadm, that
+ * should check it for you already; but if you don't, or
+ * someone fooled it, we need to double check here)
+ */
+ file = open_backing_dev(device, new_disk_conf->meta_dev, meta_claim_ptr);
+ if (IS_ERR(file))
+ return ERR_OPEN_MD_DISK;
+
+ /* avoid double bd_claim_by_disk() for the same (source,target) tuple,
+ * as would happen with internal metadata. */
+ if (file_bdev(file) != nbc->backing_bdev) {
+ err = link_backing_dev(device, new_disk_conf->meta_dev, file);
+ if (err) {
+ /* close without unlinking; otherwise error path will try to unlink */
+ close_backing_dev(device, file, false);
+ return ERR_OPEN_MD_DISK;
+ }
+ }
+
+ nbc->md_bdev = file_bdev(file);
+ nbc->f_md_bdev = file;
+ return NO_ERROR;
+}
+
+static int check_activity_log_stripe_size(struct drbd_device *device, struct drbd_md *md)
+{
+ u32 al_stripes = md->al_stripes;
+ u32 al_stripe_size_4k = md->al_stripe_size_4k;
+ u64 al_size_4k;
+
+ /* both not set: default to old fixed size activity log */
+ if (al_stripes == 0 && al_stripe_size_4k == 0) {
+ al_stripes = 1;
+ al_stripe_size_4k = (32768 >> 9)/8;
+ }
+
+ /* some paranoia plausibility checks */
+
+ /* we need both values to be set */
+ if (al_stripes == 0 || al_stripe_size_4k == 0)
+ goto err;
+
+ al_size_4k = (u64)al_stripes * al_stripe_size_4k;
+
+ /* Upper limit of activity log area, to avoid potential overflow
+ * problems in al_tr_number_to_on_disk_sector(). As right now, more
+ * than 72 * 4k blocks total only increases the amount of history,
+ * limiting this arbitrarily to 16 GB is not a real limitation ;-) */
+ if (al_size_4k > (16 * 1024 * 1024/4))
+ goto err;
+
+ /* Lower limit: we need at least 8 transaction slots (32kB)
+ * to not break existing setups */
+ if (al_size_4k < (32768 >> 9)/8)
+ goto err;
+
+ md->al_size_4k = al_size_4k;
+
+ return 0;
+err:
+ drbd_err(device, "invalid activity log striping: al_stripes=%u, al_stripe_size_4k=%u\n",
+ al_stripes, al_stripe_size_4k);
+ return -EINVAL;
+}
+
+static int check_offsets_and_sizes(struct drbd_device *device, struct drbd_backing_dev *bdev)
+{
+ sector_t capacity = drbd_get_capacity(bdev->md_bdev);
+ struct drbd_md *md = &bdev->md;
+ s32 on_disk_al_sect;
+ s32 on_disk_bm_sect;
+
+ if (md->max_peers > DRBD_PEERS_MAX) {
+ drbd_err(device, "bm_max_peers too high\n");
+ goto err;
+ }
+
+ /* The on-disk size of the activity log, calculated from offsets, and
+ * the size of the activity log calculated from the stripe settings,
+ * should match.
+ * Though we could relax this a bit: it is ok, if the striped activity log
+ * fits in the available on-disk activity log size.
+ * Right now, that would break how resize is implemented.
+ * TODO: make drbd_determine_dev_size() (and the drbdmeta tool) aware
+ * of possible unused padding space in the on disk layout. */
+ if (md->al_offset < 0) {
+ if (md->bm_offset > md->al_offset)
+ goto err;
+ on_disk_al_sect = -md->al_offset;
+ on_disk_bm_sect = md->al_offset - md->bm_offset;
+ } else {
+ if (md->al_offset != (4096 >> 9))
+ goto err;
+ if (md->bm_offset < md->al_offset + md->al_size_4k * (4096 >> 9))
+ goto err;
+
+ on_disk_al_sect = md->bm_offset - (4096 >> 9);
+ on_disk_bm_sect = md->md_size_sect - md->bm_offset;
+ }
+
+ /* old fixed size meta data is exactly that: fixed. */
+ if (md->meta_dev_idx >= 0) {
+ if (md->bm_block_size != BM_BLOCK_SIZE_4k
+ || md->md_size_sect != (128 << 20 >> 9)
+ || md->al_offset != (4096 >> 9)
+ || md->bm_offset != (4096 >> 9) + (32768 >> 9)
+ || md->al_stripes != 1
+ || md->al_stripe_size_4k != (32768 >> 12))
+ goto err;
+ }
+
+ if (capacity < md->md_size_sect)
+ goto err;
+ if (capacity - md->md_size_sect < drbd_md_first_sector(bdev))
+ goto err;
+
+ /* should be aligned, and at least 32k */
+ if ((on_disk_al_sect & 7) || (on_disk_al_sect < (32768 >> 9)))
+ goto err;
+
+ /* should fit (for now: exactly) into the available on-disk space;
+ * overflow prevention is in check_activity_log_stripe_size() above. */
+ if (on_disk_al_sect != md->al_size_4k * (4096 >> 9))
+ goto err;
+
+ /* again, should be aligned */
+ if (md->bm_offset & 7)
+ goto err;
+
+ /* FIXME check for device grow with flex external meta data? */
+
+ /* can the available bitmap space cover the last agreed device size? */
+ if (on_disk_bm_sect < drbd_capacity_to_on_disk_bm_sect(
+ md->effective_size, md))
+ goto err;
+
+ return 0;
+
+err:
+ drbd_err(device, "meta data offsets don't make sense: idx=%d bm_block_size=%d al_s=%u, al_sz4k=%u, al_offset=%d, bm_offset=%d, md_size_sect=%u, la_size=%llu, md_capacity=%llu\n",
+ md->meta_dev_idx, md->bm_block_size,
+ md->al_stripes, md->al_stripe_size_4k,
+ md->al_offset, md->bm_offset, md->md_size_sect,
+ (unsigned long long)md->effective_size,
+ (unsigned long long)capacity);
+
+ return -EINVAL;
+}
+
+__printf(2, 3)
+static void drbd_err_and_skb_info(struct drbd_adm_ctx *adm_ctx, const char *format, ...)
+{
+ struct drbd_device *device = adm_ctx->device;
+ va_list args;
+ char *text;
+
+ va_start(args, format);
+ text = kvasprintf(GFP_ATOMIC, format, args);
+ va_end(args);
+
+ if (!text)
+ return;
+
+ drbd_err(device, "%s", text);
+ drbd_adm_msg(adm_ctx, "%s", text);
+
+ kfree(text);
+}
+
+static void decode_md_9(struct meta_data_on_disk_9 *on_disk, struct drbd_md *md)
+{
+ int i;
+
+ md->effective_size = be64_to_cpu(on_disk->effective_size);
+ md->current_uuid = be64_to_cpu(on_disk->current_uuid);
+ md->prev_members = be64_to_cpu(on_disk->members);
+ md->prev_features = be64_to_cpu(on_disk->features);
+ md->device_uuid = be64_to_cpu(on_disk->device_uuid);
+ md->md_size_sect = be32_to_cpu(on_disk->md_size_sect);
+ md->al_offset = be32_to_cpu(on_disk->al_offset);
+
+ md->bm_offset = be32_to_cpu(on_disk->bm_offset);
+
+ md->flags = be32_to_cpu(on_disk->flags);
+
+ md->max_peers = be32_to_cpu(on_disk->bm_max_peers);
+ md->bm_block_size = be32_to_cpu(on_disk->bm_bytes_per_bit);
+ md->node_id = be32_to_cpu(on_disk->node_id);
+ md->al_stripes = be32_to_cpu(on_disk->al_stripes);
+ md->al_stripe_size_4k = be32_to_cpu(on_disk->al_stripe_size_4k);
+
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ struct drbd_peer_md *peer_md = &md->peers[i];
+ unsigned long flags = be32_to_cpu(on_disk->peers[i].flags);
+ s32 bitmap_index = be32_to_cpu(on_disk->peers[i].bitmap_index);
+
+ if (bitmap_index != -1)
+ flags |= MDF_HAVE_BITMAP;
+
+ peer_md->bitmap_uuid = be64_to_cpu(on_disk->peers[i].bitmap_uuid);
+ peer_md->bitmap_dagtag = be64_to_cpu(on_disk->peers[i].bitmap_dagtag);
+ peer_md->flags = flags;
+ peer_md->bitmap_index = bitmap_index;
+ /* The origin of the bits on disk is not on disk. Reading the
+ * bitmap calls drbd_md_slot_emptied() for every empty slot,
+ * which is where this becomes true again.
+ */
+ peer_md->placeholder_src = 0;
+ peer_md->placeholder_src_complete = false;
+ }
+ for (i = 0; i < ARRAY_SIZE(on_disk->history_uuids); i++)
+ md->history_uuids[i] = be64_to_cpu(on_disk->history_uuids[i]);
+
+ BUILD_BUG_ON(ARRAY_SIZE(md->history_uuids) != ARRAY_SIZE(on_disk->history_uuids));
+
+ memset(&md->cbt, 0, sizeof(md->cbt));
+ md->cbt.bitmap_index = -1;
+ if (md->prev_features & DRBD_MDFF_CBT) {
+ md->cbt.id = be64_to_cpu(on_disk->cbt_bookmark.id);
+ md->cbt.bitmap_uuid = be64_to_cpu(on_disk->cbt_bookmark.bitmap_uuid);
+ md->cbt.bitmap_dagtag = be64_to_cpu(on_disk->cbt_bookmark.bitmap_dagtag);
+ md->cbt.since = be64_to_cpu(on_disk->cbt_bookmark.since);
+ /* only true while I/O stays suspended after the flush */
+ md->cbt.flags = be32_to_cpu(on_disk->cbt_bookmark.flags) & ~CBT_BOOKMARK_FLUSHED;
+ md->cbt.bitmap_index = be32_to_cpu(on_disk->cbt_bookmark.bitmap_index);
+ }
+}
+
+/* Drop the peer flags which the DRBD that last wrote this meta data did not
+ * understand. Such a DRBD preserves a flag without updating its value, so the
+ * value we read is not necessarily current. This protects against downgrade -
+ * upgrade sequences.
+ *
+ * MDF_PEER_BITMAP_AUTHORITATIVE goes the other way: a writer that did not
+ * maintain it may have set bits for blocks the peer lacks without recording
+ * that, so every standing bit counts as such.
+ */
+static void distrust_unsupported_peer_flags(struct drbd_device *device, struct drbd_md *md)
+{
+ u64 nodes = 0;
+ int i;
+
+ if (!(md->prev_features & DRBD_MDFF_BITMAP_AUTHORITATIVE)) {
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ struct drbd_peer_md *peer_md = &md->peers[i];
+
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md->flags))
+ continue;
+ if (!test_and_set_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md->flags))
+ nodes |= NODE_MASK(i);
+ }
+ if (nodes)
+ drbd_info(device, "Meta data written without the bitmap provenance feature; "
+ "all out-of-sync bits toward node(s) 0x%llX count as set with reason\n",
+ nodes);
+ nodes = 0;
+ }
+
+ if (md->prev_features & DRBD_MDFF_DIVERGENCE_BITMAP)
+ return;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ struct drbd_peer_md *peer_md = &md->peers[i];
+
+ if (!test_and_clear_bit(__MDF_PEER_DIVERGENCE_BITMAP, &peer_md->flags))
+ continue;
+
+ /* A slot without a bitmap of its own is a divergence bitmap in
+ * any case, so clearing the flag changes nothing there. Report
+ * only the slots where it makes a difference.
+ */
+ if (test_bit(__MDF_HAVE_BITMAP, &peer_md->flags))
+ nodes |= NODE_MASK(i);
+ }
+
+ if (nodes)
+ drbd_info(device, "Meta data written without the divergence bitmap feature; "
+ "not trusting MDF_PEER_DIVERGENCE_BITMAP of node(s) 0x%llX\n", nodes);
+}
+
+/* A slot without a bitmap needs no divergence flag: is_divergence_bitmap() is
+ * true for it regardless, and a day0 slot must keep flags == 0.
+ */
+static void drop_divergence_flag_without_bitmap(struct drbd_md *md)
+{
+ int i;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ struct drbd_peer_md *peer_md = &md->peers[i];
+
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md->flags))
+ clear_bit(__MDF_PEER_DIVERGENCE_BITMAP, &peer_md->flags);
+ }
+}
+
+/* Make all non-zero day0 slots agree with the one drbd_day0_node_id() picks,
+ * so that forget-peer cannot copy a disagreeing value into a forgotten slot.
+ *
+ * Slots at 0 are left alone: before the first UUID bump all of them are, and
+ * the bump fills them all with the same UUID.
+ *
+ * A day0 slot has no bitmap of its own, so this changes no bitmap. A returning
+ * node whose current UUID is one of the overwritten values no longer finds it
+ * here; it can still match by our history, otherwise the handshake no longer
+ * relates it.
+ */
+static void repair_day0_slots(struct drbd_device *device, struct drbd_md *md)
+{
+ int day0 = drbd_day0_node_id(md, -1);
+ u64 day0_uuid, nodes = 0;
+ int node_id;
+
+ if (day0 == -1 || !md->peers[day0].bitmap_uuid)
+ return;
+ day0_uuid = md->peers[day0].bitmap_uuid;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &md->peers[node_id];
+
+ if (!drbd_is_day0_slot(md, node_id) || !peer_md->bitmap_uuid ||
+ (peer_md->bitmap_uuid & ~UUID_PRIMARY) == (day0_uuid & ~UUID_PRIMARY))
+ continue;
+ peer_md->bitmap_uuid = day0_uuid;
+ peer_md->bitmap_dagtag = md->peers[day0].bitmap_dagtag;
+ nodes |= NODE_MASK(node_id);
+ }
+
+ if (nodes)
+ drbd_warn(device, "Unallocated peer slot(s) of node(s) 0x%llX did not hold the day0 UUID %016llX; reset them to it\n",
+ nodes, (unsigned long long)day0_uuid);
+}
+
+static void decode_magic(struct meta_data_on_disk_9 *on_disk, u32 *magic, u32 *flags)
+{
+ /* magic and flags are in at the same offsets in 8.4 and 9 */
+ *magic = be32_to_cpu(on_disk->magic);
+ *flags = be32_to_cpu(on_disk->flags);
+}
+
+static
+int drbd_md_decode(struct drbd_adm_ctx *adm_ctx,
+ struct drbd_backing_dev *bdev,
+ void *buffer)
+{
+ struct drbd_device *device = adm_ctx->device;
+ u32 magic, flags;
+ int i, rv = NO_ERROR;
+ int my_node_id = device->resource->res_opts.node_id;
+
+ decode_magic(buffer, &magic, &flags);
+ if ((magic == DRBD_MD_MAGIC_09 && !(flags & MDF_AL_CLEAN)) ||
+ magic == DRBD_MD_MAGIC_84_UNCLEAN ||
+ (magic == DRBD_MD_MAGIC_08 && !(flags & MDF_AL_CLEAN))) {
+ /* btw: that's Activity Log clean, not "all" clean. */
+ drbd_err_and_skb_info(adm_ctx,
+ "Found unclean meta data. Did you \"drbdadm apply-al\"?\n");
+ rv = ERR_MD_UNCLEAN;
+ goto err;
+ }
+ rv = ERR_MD_INVALID;
+ if (magic != DRBD_MD_MAGIC_09 && magic !=
+ DRBD_MD_MAGIC_84_UNCLEAN && magic != DRBD_MD_MAGIC_08) {
+ if (magic == DRBD_MD_MAGIC_07)
+ drbd_err_and_skb_info(adm_ctx,
+ "Found old meta data magic. Did you \"drbdadm create-md\"?\n");
+ else
+ drbd_err_and_skb_info(adm_ctx,
+ "Meta data magic not found. Did you \"drbdadm create-md\"?\n");
+ goto err;
+ }
+
+ if (magic == DRBD_MD_MAGIC_09) {
+ clear_bit(LEGACY_84_MD, &device->flags);
+ decode_md_9(buffer, &bdev->md);
+ distrust_unsupported_peer_flags(device, &bdev->md);
+ drop_divergence_flag_without_bitmap(&bdev->md);
+ repair_day0_slots(device, &bdev->md);
+ } else {
+ if (!device->resource->res_opts.drbd8_compat_mode) {
+ drbd_err_and_skb_info(adm_ctx,
+ "Found old meta data magic. Did you \"drbdadm create-md\"?\n");
+ goto err;
+ }
+ set_bit(LEGACY_84_MD, &device->flags);
+ drbd_md_decode_84(buffer, &bdev->md);
+ if (bdev->md.bm_block_size != BM_BLOCK_SIZE_4k) {
+ drbd_err_and_skb_info(adm_ctx,
+ "unexpected bm_bytes_per_bit: %u (expected %u)\n",
+ bdev->md.bm_block_size, BM_BLOCK_SIZE_4k);
+ goto err;
+ }
+ }
+
+ if (!is_power_of_2(bdev->md.bm_block_size)
+ || bdev->md.bm_block_size < BM_BLOCK_SIZE_MIN
+ || bdev->md.bm_block_size > BM_BLOCK_SIZE_MAX) {
+ drbd_err_and_skb_info(adm_ctx,
+ "unexpected bm_bytes_per_bit: %u (expected power of 2 in [%u..%u])\n",
+ bdev->md.bm_block_size, BM_BLOCK_SIZE_MIN, BM_BLOCK_SIZE_MAX);
+ goto err;
+ }
+ bdev->md.bm_block_shift = ilog2(bdev->md.bm_block_size);
+
+ if (check_activity_log_stripe_size(device, &bdev->md))
+ goto err;
+ if (check_offsets_and_sizes(device, bdev))
+ goto err;
+
+ if (bdev->md.node_id != -1 && bdev->md.node_id != my_node_id) {
+ drbd_err_and_skb_info(adm_ctx, "ambiguous node id: meta-data: %d, config: %d\n",
+ bdev->md.node_id, my_node_id);
+ goto err;
+ }
+
+ for (i = 0; i < DRBD_NODE_ID_MAX; i++) {
+ struct drbd_peer_md *peer_md = &bdev->md.peers[i];
+
+ if (peer_md->bitmap_index == -1)
+ continue;
+ if (i == my_node_id) {
+ drbd_err_and_skb_info(adm_ctx, "my own node id (%d) should not have a bitmap index (%d)\n",
+ my_node_id, peer_md->bitmap_index);
+ goto err;
+ }
+ if (peer_md->bitmap_index < -1 || peer_md->bitmap_index >= bdev->md.max_peers) {
+ drbd_err_and_skb_info(adm_ctx, "peer node id %d: bitmap index (%d) exceeds allocated bitmap slots (%d)\n",
+ i, peer_md->bitmap_index, bdev->md.max_peers);
+ goto err;
+ }
+ /* maybe: for each bitmap_index != -1, create a connection object
+ * with peer_node_id = i, unless already present. */
+ }
+
+ if (bdev->md.cbt.flags & CBT_BOOKMARK_IN_USE) {
+ int cbt_index = bdev->md.cbt.bitmap_index;
+ bool taken = cbt_index < 0 || cbt_index >= bdev->md.max_peers;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX && !taken; i++)
+ taken = bdev->md.peers[i].bitmap_index == cbt_index;
+ if (taken) {
+ drbd_err_and_skb_info(adm_ctx,
+ "changed block tracking: bitmap index (%d) is invalid or owned by a peer\n",
+ cbt_index);
+ goto err;
+ }
+ }
+
+ rv = NO_ERROR;
+
+err:
+ return rv;
+}
+
+/**
+ * drbd_md_read() - Reads in the meta data super block
+ * @adm_ctx: DRBD config context.
+ * @bdev: Device from which the meta data should be read in.
+ *
+ * Return NO_ERROR on success, and an enum drbd_ret_code in case
+ * something goes wrong.
+ *
+ * Called exactly once during drbd_adm_attach(), while still being D_DISKLESS,
+ * even before @bdev is assigned to @device->ldev.
+ */
+static int drbd_md_read(struct drbd_adm_ctx *adm_ctx, struct drbd_backing_dev *bdev)
+{
+ struct drbd_device *device = adm_ctx->device;
+ void *buffer;
+ int rv;
+
+ if (device->disk_state[NOW] != D_DISKLESS)
+ return ERR_DISK_CONFIGURED;
+
+ /* First, figure out where our meta data superblock is located,
+ * and read it. */
+ bdev->md.meta_dev_idx = bdev->disk_conf->meta_dev_idx;
+ bdev->md.md_offset = drbd_md_ss(bdev);
+ /* Even for (flexible or indexed) external meta data,
+ * initially restrict us to the 4k superblock for now.
+ * Affects the paranoia out-of-range access check in drbd_md_sync_page_io(). */
+ bdev->md.md_size_sect = 8;
+
+ drbd_dax_open(bdev);
+ if (drbd_md_dax_active(bdev)) {
+ drbd_info(device, "meta-data IO uses: dax-pmem\n");
+ rv = drbd_md_decode(adm_ctx, bdev, drbd_dax_md_addr(bdev));
+ if (rv != NO_ERROR)
+ return rv;
+ if (drbd_dax_map(bdev))
+ return ERR_IO_MD_DISK;
+ return NO_ERROR;
+ }
+ drbd_info(device, "meta-data IO uses: blk-bio\n");
+
+ buffer = drbd_md_get_buffer(device, __func__);
+ if (!buffer)
+ return ERR_NOMEM;
+
+ if (drbd_md_sync_page_io(device, bdev, bdev->md.md_offset,
+ REQ_OP_READ)) {
+ /* NOTE: can't do normal error processing here as this is
+ called BEFORE disk is attached */
+ drbd_err_and_skb_info(adm_ctx, "Error while reading metadata.\n");
+ rv = ERR_IO_MD_DISK;
+ goto err;
+ }
+
+ rv = drbd_md_decode(adm_ctx, bdev, buffer);
+ err:
+ drbd_md_put_buffer(device);
+
+ return rv;
+}
+
+/* May this node restore the quorum it had when it last wrote its meta data?
+ *
+ * Only if every other node of that membership is an intentionally diskless
+ * node. Those never vote in calc_quorum(), they only act as tie-breakers, and
+ * a tie-breaker can not create quorum, it can only preserve an existing one.
+ * So this node was quorate on its own, which is what RESTORE_QUORUM restores.
+ *
+ * A peer we hold a bitmap slot for, or that we have ever seen with a disk, is
+ * a voter. Unknown nodes count as voters as well.
+ */
+static bool may_restore_quorum(struct drbd_device *device)
+{
+ const u64 me = NODE_MASK(device->resource->res_opts.node_id);
+ u64 others = device->ldev->md.prev_members & ~me;
+ int node_id;
+
+ if (!(device->ldev->md.prev_members & me))
+ return false;
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ if (!(others & NODE_MASK(node_id)))
+ continue;
+ if (test_bit(__MDF_HAVE_BITMAP, &device->ldev->md.peers[node_id].flags) ||
+ test_bit(__MDF_PEER_DEVICE_SEEN, &device->ldev->md.peers[node_id].flags))
+ return false;
+ }
+
+ return true;
+}
+
+int drbd_adm_attach(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_device *device;
+ struct drbd_resource *resource;
+ int err, retcode = NO_ERROR;
+ enum determine_dev_size dd;
+ enum drbd_disk_state ds;
sector_t min_md_device_sectors;
- struct drbd_backing_dev *nbc = NULL; /* new_backing_conf */
- struct disk_conf *new_disk_conf = NULL;
- struct lru_cache *resync_lru = NULL;
- struct fifo_buffer *new_plan = NULL;
- union drbd_state ns, os;
+ struct drbd_backing_dev *nbc; /* new_backing_conf */
+ struct drbd_bitmap *bitmap = NULL; /* unpublished until bm_pages is wired up */
+ bool published_bitmap = false;
+ sector_t backing_disk_max_sectors;
+ struct drbd_disk_conf *new_disk_conf = NULL;
enum drbd_state_rv rv;
- struct net_conf *nc;
+ struct drbd_peer_device *peer_device;
+ unsigned int slots_needed = 0;
+ bool have_conf_update = false;
+ bool al_updates;
+
+ device = adm_ctx->device;
+ resource = device->resource;
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
+
+ /* allocation not in the IO path, drbdsetup context */
+ nbc = kzalloc_obj(struct drbd_backing_dev);
+ if (!nbc) {
+ retcode = ERR_NOMEM;
+ goto fail;
+ }
+ spin_lock_init(&nbc->md.uuid_lock);
+
+ new_disk_conf = kzalloc_obj(struct drbd_disk_conf);
+ if (!new_disk_conf) {
+ retcode = ERR_NOMEM;
+ goto fail;
+ }
+ nbc->disk_conf = new_disk_conf;
+
+ drbd_set_disk_conf_defaults(new_disk_conf);
+ err = drbd_adm_overlay_disk_conf(adm_ctx, new_disk_conf);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto fail;
+ }
+
+ if (new_disk_conf->meta_dev_idx < DRBD_MD_INDEX_FLEX_INT) {
+ retcode = ERR_MD_IDX_INVALID;
+ goto fail;
+ }
+
+ lock_all_resources();
+ retcode = drbd_resync_after_valid(device, new_disk_conf->resync_after);
+ unlock_all_resources();
+ if (retcode != NO_ERROR)
+ goto fail;
+
+ retcode = open_backing_devices(device, new_disk_conf, nbc);
+ if (retcode != NO_ERROR)
+ goto fail;
+
+ if ((nbc->backing_bdev == nbc->md_bdev) !=
+ (new_disk_conf->meta_dev_idx == DRBD_MD_INDEX_INTERNAL ||
+ new_disk_conf->meta_dev_idx == DRBD_MD_INDEX_FLEX_INT)) {
+ retcode = ERR_MD_IDX_INVALID;
+ goto fail;
+ }
+
+ /* if you want to reconfigure, please tear down first */
+ if (device->disk_state[NOW] > D_DISKLESS) {
+ retcode = ERR_DISK_CONFIGURED;
+ goto fail;
+ }
+ /* It may just now have detached because of IO error. Make sure
+ * drbd_ldev_destroy is done already, we may end up here very fast,
+ * e.g. if someone calls attach from the on-io-error handler,
+ * to realize a "hot spare" feature (not that I'd recommend that) */
+ wait_event(device->misc_wait, !test_bit(GOING_DISKLESS, &device->flags));
+
+ /* make sure there is no leftover from previous force-detach attempts */
+ clear_bit(FORCE_DETACH, &device->flags);
+
+ /* and no leftover from previously aborted resync or verify, either */
+ for_each_peer_device(peer_device, device) {
+ while (atomic_read(&peer_device->rs_pending_cnt)) {
+ drbd_info_ratelimit(peer_device, "wait for rs_pending_cnt to clear\n");
+ if (schedule_timeout_interruptible(HZ / 10)) {
+ retcode = ERR_INTR;
+ goto fail;
+ }
+ }
+
+ peer_device->rs_total = 0;
+ peer_device->rs_failed = 0;
+ }
+
+ /* Read our meta data super block early.
+ * This also sets other on-disk offsets.
+ */
+ retcode = drbd_md_read(adm_ctx, nbc);
+ if (retcode != NO_ERROR)
+ goto fail;
+
+ if (device->bitmap) {
+ drbd_err_and_skb_info(adm_ctx, "already has a bitmap, this should not happen\n");
+ retcode = ERR_INVALID_REQUEST;
+ goto fail;
+ }
+
+ if (new_disk_conf->d_bitmap) {
+ /* ldev_safe: attach path, allocating bitmap. */
+ bitmap = drbd_bm_alloc(nbc->md.max_peers, nbc->md.bm_block_shift);
+ if (!bitmap) {
+ retcode = ERR_NOMEM;
+ goto fail;
+ }
+ } else {
+ if (!list_empty(&resource->connections)) {
+ drbd_err_and_skb_info(adm_ctx,
+ "Disabling bitmap allocation with peers defined is not allowed");
+ retcode = ERR_INVALID_REQUEST;
+ goto fail;
+ }
+ /* Without a bitmap no write gets tracked. */
+ if (nbc->md.cbt.flags & CBT_BOOKMARK_IN_USE)
+ nbc->md.cbt.flags |= CBT_BOOKMARK_INVALID;
+ }
+ device->last_bm_block_shift = nbc->md.bm_block_shift;
+
+ sanitize_disk_conf(device, new_disk_conf, nbc);
+
+ backing_disk_max_sectors = drbd_get_max_capacity(device, nbc, true);
+ if (backing_disk_max_sectors < new_disk_conf->disk_size) {
+ drbd_err_and_skb_info(adm_ctx, "max capacity %llu smaller than disk size %llu\n",
+ (unsigned long long) backing_disk_max_sectors,
+ (unsigned long long) new_disk_conf->disk_size);
+ retcode = ERR_DISK_TOO_SMALL;
+ goto fail;
+ }
+
+ if (new_disk_conf->meta_dev_idx < 0) {
+ /* at least one MB, otherwise it does not make sense */
+ min_md_device_sectors = (2<<10);
+ } else {
+ min_md_device_sectors = (128 << 20 >> 9) * (new_disk_conf->meta_dev_idx + 1);
+ }
+
+ if (drbd_get_capacity(nbc->md_bdev) < min_md_device_sectors) {
+ retcode = ERR_MD_DISK_TOO_SMALL;
+ drbd_warn(device, "refusing attach: md-device too small, "
+ "at least %llu sectors needed for this meta-disk type\n",
+ (unsigned long long) min_md_device_sectors);
+ goto fail;
+ }
+
+ /* Make sure the new disk is big enough
+ * (we may currently be R_PRIMARY with no local disk...) */
+ if (backing_disk_max_sectors <
+ get_capacity(device->vdisk)) {
+ drbd_err_and_skb_info(adm_ctx,
+ "Current (diskless) capacity %llu, cannot attach smaller (%llu) disk\n",
+ (unsigned long long)get_capacity(device->vdisk),
+ (unsigned long long)backing_disk_max_sectors);
+ retcode = ERR_DISK_TOO_SMALL;
+ goto fail;
+ }
+
+ nbc->known_size = drbd_get_capacity(nbc->backing_bdev);
+
+ drbd_suspend_io(device, READ_AND_WRITE);
+ wait_event(resource->barrier_wait, !barrier_pending(resource));
+ for_each_peer_device(peer_device, device)
+ wait_event(device->misc_wait,
+ (!atomic_read(&peer_device->ap_pending_cnt) ||
+ drbd_suspended(device)));
+ /* and for other previously queued resource work */
+ drbd_flush_workqueue(&resource->work);
+
+ rv = stable_state_change(resource,
+ change_disk_state(device, D_ATTACHING, CS_VERBOSE | CS_SERIALIZE, "attach", NULL));
+ retcode = (enum drbd_ret_code)rv;
+ if (rv >= SS_SUCCESS)
+ update_resource_dagtag(resource, nbc);
+ drbd_resume_io(device);
+ if (rv < SS_SUCCESS)
+ goto fail;
+
+ if (!get_ldev_if_state(device, D_ATTACHING))
+ goto force_diskless;
+
+ drbd_info(device, "Maximum number of peer devices = %u\n", nbc->md.max_peers);
+
+ mutex_lock(&resource->conf_update);
+ have_conf_update = true;
+
+ /* Make sure the local node id matches or is unassigned */
+ if (nbc->md.node_id != -1 && nbc->md.node_id != resource->res_opts.node_id) {
+ drbd_err_and_skb_info(adm_ctx, "Local node id %d differs from local "
+ "node id %d on device\n",
+ resource->res_opts.node_id,
+ nbc->md.node_id);
+ retcode = ERR_INVALID_REQUEST;
+ goto force_diskless_dec;
+ }
+
+ /* Make sure no bitmap slot has our own node id.
+ * If we are operating in "drbd 8 compatibility mode", the node ID is
+ * not yet initialized at this point, so just ignore this check.
+ */
+ if (resource->res_opts.node_id != -1 &&
+ nbc->md.peers[resource->res_opts.node_id].bitmap_index != -1) {
+ drbd_err_and_skb_info(adm_ctx, "There is a bitmap for my own node id (%d)\n",
+ resource->res_opts.node_id);
+ retcode = ERR_INVALID_REQUEST;
+ goto force_diskless_dec;
+ }
+
+ /* Make sure we have a bitmap slot for each peer id */
+ for_each_peer_device(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
+ int bitmap_index;
+
+ if (peer_device->bitmap_index != -1) {
+ drbd_err_and_skb_info(adm_ctx,
+ "ASSERTION FAILED bitmap_index %d during attach, expected -1\n",
+ peer_device->bitmap_index);
+ }
+
+ bitmap_index = nbc->md.peers[connection->peer_node_id].bitmap_index;
+ if (want_bitmap(peer_device)) {
+ if (bitmap_index != -1)
+ peer_device->bitmap_index = bitmap_index;
+ else
+ slots_needed++;
+ } else if (bitmap_index != -1) {
+ /* Pretend in core that there is not bitmap for that peer,
+ in the on disk meta-data we keep it until it is de-allocated
+ with forget-peer */
+ clear_bit(__MDF_HAVE_BITMAP,
+ &nbc->md.peers[connection->peer_node_id].flags);
+ }
+ }
+ if (slots_needed) {
+ int slots_available = nbc->md.max_peers - used_bitmap_slots(nbc);
+
+ if (slots_needed > slots_available) {
+ drbd_err_and_skb_info(adm_ctx, "Not enough free bitmap "
+ "slots (available=%d, needed=%d)\n",
+ slots_available,
+ slots_needed);
+ retcode = ERR_INVALID_REQUEST;
+ goto force_diskless_dec;
+ }
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->bitmap_index != -1 || !want_bitmap(peer_device))
+ continue;
+
+ err = allocate_bitmap_index(peer_device, nbc);
+ if (err) {
+ retcode = ERR_INVALID_REQUEST;
+ goto force_diskless_dec;
+ }
+ }
+ }
+
+ /* Assign the local node id (if not assigned already) */
+ nbc->md.node_id = resource->res_opts.node_id;
+
+ if (resource->role[NOW] == R_PRIMARY && device->exposed_data_uuid &&
+ (device->exposed_data_uuid & ~UUID_PRIMARY) !=
+ (nbc->md.current_uuid & ~UUID_PRIMARY)) {
+ int data_present = false;
+
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->disk_state[NOW] == D_UP_TO_DATE)
+ data_present = true;
+ }
+ if (!data_present) {
+ drbd_err_and_skb_info(adm_ctx,
+ "Can only attach to data with current UUID=%016llX\n",
+ (unsigned long long)device->exposed_data_uuid);
+ retcode = ERR_DATA_NOT_CURRENT;
+ goto force_diskless_dec;
+ }
+ }
+
+ /* Since we are diskless, fix the activity log first... */
+ if (drbd_check_al_size(device, new_disk_conf)) {
+ retcode = ERR_NOMEM;
+ goto force_diskless_dec;
+ }
+
+ /* Point of no return reached.
+ * Devices and memory are no longer released by error cleanup below.
+ * now device takes over responsibility, and the state engine should
+ * clean it up somewhere. */
+ D_ASSERT(device, device->ldev == NULL);
+ device->ldev = nbc;
+ nbc = NULL;
+ /* A cbt-flush of the previous disk says nothing about this one. */
+ clear_bit(CBT_BITMAPS_FLUSHED, &device->flags);
+ new_disk_conf = NULL;
+ drbd_resource_update_rx_alignment(device->resource);
+
+ if (drbd_md_dax_active(device->ldev)) {
+ /* The on-disk activity log is always initialized with the
+ * non-pmem format. We have now decided to access it using
+ * dax, so re-initialize it appropriately. */
+ if (drbd_dax_al_initialize(device)) {
+ retcode = ERR_IO_MD_DISK;
+ goto force_diskless_dec;
+ }
+ }
+
+ mutex_unlock(&resource->conf_update);
+ have_conf_update = false;
+
+ lock_all_resources();
+ retcode = drbd_resync_after_valid(device, device->ldev->disk_conf->resync_after);
+ if (retcode != NO_ERROR) {
+ unlock_all_resources();
+ goto force_diskless_dec;
+ }
+
+ /* Reset the "barriers don't work" bits here, then force meta data to
+ * be written, to ensure we determine if barriers are supported. */
+ assign_bit(MD_NO_FUA, &device->flags,
+ !device->ldev->disk_conf->md_flushes);
+
+ drbd_resync_after_changed(device);
+ drbd_bump_write_ordering(resource, device->ldev, WO_BIO_BARRIER);
+ unlock_all_resources();
+
+ /* Prevent shrinking of consistent devices ! */
+ {
+ unsigned long long nsz = drbd_new_dev_size(device, 0, device->ldev->disk_conf->disk_size, 0);
+ unsigned long long eff = device->ldev->md.effective_size;
+
+ if (drbd_md_test_flag(device->ldev, MDF_CONSISTENT) && nsz < eff) {
+ if (nsz == device->ldev->disk_conf->disk_size) {
+ drbd_warn(device, "truncating a consistent device during attach (%llu < %llu)\n", nsz, eff);
+ } else {
+ drbd_warn(device, "refusing to truncate a consistent device (%llu < %llu)\n", nsz, eff);
+ drbd_adm_msg(adm_ctx,
+ "To-be-attached device has last effective > current size, and is consistent\n"
+ "(%llu > %llu sectors). Refusing to attach.", eff, nsz);
+ retcode = ERR_IMPLICIT_SHRINK;
+ goto force_diskless_dec;
+ }
+ }
+ }
+
+ if (drbd_md_test_flag(device->ldev, MDF_HAVE_QUORUM) &&
+ drbd_md_test_flag(device->ldev, MDF_WAS_UP_TO_DATE) &&
+ may_restore_quorum(device))
+ set_bit(RESTORE_QUORUM, &device->flags);
+
+ assign_bit(CRASHED_PRIMARY, &device->flags,
+ drbd_md_test_flag(device->ldev, MDF_CRASHED_PRIMARY) &&
+ !(resource->role[NOW] == R_PRIMARY && resource->susp_nod[NOW]) &&
+ !device->exposed_data_uuid && !drbd_gen_obligation_outstanding(device));
+
+ if (drbd_md_test_flag(device->ldev, MDF_PRIMARY_LOST_QUORUM) &&
+ !device->have_quorum[NOW])
+ set_bit(PRIMARY_LOST_QUORUM, &device->flags);
+
+ device->read_cnt = 0;
+ device->writ_cnt = 0;
+
+ drbd_reconsider_queue_parameters(device, device->ldev);
+
+ /* If I am currently not R_PRIMARY,
+ * but meta data primary indicator is set,
+ * I just now recover from a hard crash,
+ * and have been R_PRIMARY before that crash.
+ *
+ * Now, if I had no connection before that crash
+ * (have been degraded R_PRIMARY), chances are that
+ * I won't find my peer now either.
+ *
+ * In that case, and _only_ in that case,
+ * we use the degr-wfc-timeout instead of the default,
+ * so we can automatically recover from a crash of a
+ * degraded but active "cluster" after a certain timeout.
+ */
+ for_each_peer_device(peer_device, device) {
+ clear_bit(USE_DEGR_WFC_T, peer_device->flags);
+ if (resource->role[NOW] != R_PRIMARY &&
+ drbd_md_test_flag(device->ldev, MDF_PRIMARY_IND) &&
+ !drbd_md_test_peer_flag(peer_device, __MDF_PEER_CONNECTED))
+ set_bit(USE_DEGR_WFC_T, peer_device->flags);
+ }
+
+ /* Load on-disk tracking bits before peers' advertised sizes can
+ * influence the computed device size.
+ *
+ * old_size is the number of sectors for which the on-disk bitmap
+ * carries meaningful content. Use min_not_zero() so that:
+ * - day0 attach (md->effective_size == 0, device already live at
+ * dev_size via a diskless connection): old_size = dev_size, reading
+ * whatever bitmap the metadata preparer wrote (all-set for full
+ * sync, or a day0 tracking bitmap for inception-based resync).
+ * - normal re-attach: old_size = min(md_size, dev_size), so any
+ * gap between md_size and dev_size (e.g. a peer ran resize while
+ * this node was diskless) is covered by drbd_determine_dev_size()
+ * below with bits SET, rather than by whatever happens to be on
+ * disk at that offset.
+ */
+ {
+ sector_t md_size = device->ldev->md.effective_size;
+ sector_t dev_size = get_capacity(device->vdisk);
+ sector_t old_size = min_not_zero(md_size, dev_size);
+
+ /* old_size == 0 means no prior history exists (first-ever attach
+ * before any connection): skip the read and let
+ * drbd_determine_dev_size() below allocate and initialise the
+ * bitmap fresh.
+ */
+ if (old_size > 0 && bitmap) {
+ _drbd_bm_lock(device, bitmap, NULL, __func__, BM_LOCK_ALL);
+ err = drbd_bm_resize(device, bitmap, old_size, false);
+ _drbd_bm_unlock(device, bitmap);
+ if (err) {
+ retcode = ERR_NOMEM_BITMAP;
+ goto force_diskless_dec;
+ }
+
+ /* Publish only after bm_pages is populated. */
+ smp_store_release(&device->bitmap, bitmap);
+ bitmap = NULL;
+ published_bitmap = true;
+
+ err = drbd_bitmap_io(device, &drbd_bm_read,
+ "read from attaching", BM_LOCK_ALL,
+ NULL);
+ if (err) {
+ retcode = ERR_IO_MD_DISK;
+ goto force_diskless_dec;
+ }
+ } else if (bitmap) {
+ /* old_size == 0: no prior bitmap to read.
+ * drbd_determine_dev_size() below resizes via
+ * device->bitmap, so we have to publish here. The
+ * window during which device->bitmap is non-NULL with
+ * bm_pages == NULL is quiescent (D_ATTACHING, no peer
+ * connected, no IO touching the bitmap).
+ */
+ smp_store_release(&device->bitmap, bitmap);
+ bitmap = NULL;
+ published_bitmap = true;
+ }
+ }
+
+ /* Determine the final device size, growing [old_size, new_size].
+ * Normally new-region bits are SET (new territory, needs resync).
+ * Exception: if the backing device guarantees zeroes for all new
+ * allocations (discard_zeroes_if_aligned) and the device has a real
+ * UUID (not just-created), the new region may be known-clean:
+ *
+ * Path A -- device was UpToDate before detach: no dirty data, new
+ * space on a zero-guarantee backend is safe to skip.
+ *
+ * Path B -- device was NOT UpToDate, but all per-peer bitmap UUIDs
+ * and all history UUIDs are zero: this is a day0 volume
+ * that has never diverged from any peer; new space is
+ * guaranteed zeroed by the backend. Requires
+ * rs_discard_granularity to be explicitly configured
+ * (non-zero), because discard_zeroes_if_aligned defaults
+ * to 1 on all devices -- without an explicit thin-
+ * provisioning opt-in the new-region content is not
+ * guaranteed clean.
+ *
+ * In both cases pass DDSF_NO_RESYNC to leave the new-region bits
+ * clear rather than marking them dirty.
+ *
+ * The on-disk bitmap is written here if la_size_changed; the
+ * in-memory bitmap is already correct:
+ * [0, old_size] loaded from disk above
+ * [old_size, new_size] SET or clear depending on ddsf
+ */
+ {
+ enum dds_flags ddsf = 0;
+ bool dzoia;
+ u32 rs_discard_granularity;
+
+ rcu_read_lock();
+ dzoia = rcu_dereference(device->ldev->disk_conf)->discard_zeroes_if_aligned;
+ rs_discard_granularity =
+ rcu_dereference(device->ldev->disk_conf)->rs_discard_granularity;
+ rcu_read_unlock();
+
+ if (dzoia &&
+ (device->ldev->md.current_uuid & ~UUID_PRIMARY) != UUID_JUST_CREATED) {
+ const char *reason = NULL;
+
+ if (drbd_md_test_flag(device->ldev, MDF_WAS_UP_TO_DATE)) {
+ reason = "was UpToDate";
+ } else if (rs_discard_granularity) {
+ /* Path B: check that all bitmap and history UUIDs are zero */
+ bool all_zero = true;
+ int i;
+
+ for (i = 0; i < DRBD_NODE_ID_MAX && all_zero; i++)
+ if (device->ldev->md.peers[i].bitmap_uuid)
+ all_zero = false;
+ for (i = 0; i < HISTORY_UUIDS && all_zero; i++)
+ if (device->ldev->md.history_uuids[i])
+ all_zero = false;
+
+ if (all_zero)
+ reason = "day0 volume (all bitmap and history UUIDs zero)";
+ }
+
+ if (reason) {
+ drbd_info(device, "%s, new region assumed zeroed\n", reason);
+ ddsf = DDSF_NO_RESYNC;
+ }
+ }
+
+ dd = drbd_determine_dev_size(device, 0, ddsf, NULL);
+ }
+ if (dd == DS_ERROR) {
+ retcode = ERR_NOMEM_BITMAP;
+ goto force_diskless_dec;
+ } else if (dd == DS_GREW) {
+ for_each_peer_device(peer_device, device)
+ set_bit(RESYNC_AFTER_NEG, peer_device->flags);
+ }
+
+ /* No activity log record of the crash window: every region may differ
+ * from any peer, in every slot -- an unallocated one tracks day 0.
+ */
+ /* MDF_PRIMARY_IND: a clean shutdown clears it, MDF_CRASHED_PRIMARY may
+ * stay set, so this runs on the first attach after the crash only.
+ */
+ /* Without a bitmap there is nothing to set; skip rather than claim a
+ * FullSync in the log that drbd_bitmap_io() then does not do.
+ */
+ if (device->bitmap &&
+ test_bit(CRASHED_PRIMARY, &device->flags) &&
+ drbd_md_test_flag(device->ldev, MDF_PRIMARY_IND) &&
+ drbd_md_test_flag(device->ldev, MDF_AL_DISABLED)) {
+ drbd_info(device, "AL disabled at crash: all blocks out of sync (aka FullSync)\n");
+ if (drbd_bitmap_io(device, &drbd_bmio_set_all_n_write,
+ "set_all_n_write from attaching", BM_LOCK_ALL, NULL)) {
+ retcode = ERR_IO_MD_DISK;
+ goto force_diskless_dec;
+ }
+ }
+
+ for_each_peer_device(peer_device, device) {
+ if (drbd_md_test_peer_flag(peer_device, __MDF_PEER_FULL_SYNC)) {
+ drbd_info(peer_device, "Assuming that all blocks are out of sync "
+ "(aka FullSync)\n");
+ if (drbd_bitmap_io(device, &drbd_bmio_set_n_write,
+ "set_n_write from attaching", BM_LOCK_ALL | BM_LOCK_SINGLE_SLOT,
+ peer_device)) {
+ retcode = ERR_IO_MD_DISK;
+ goto force_diskless_dec;
+ }
+ }
+ }
+
+ /* The activity log replay (drbdmeta apply-al) marked the crash window
+ * out of sync toward every peer; any of them may lack any of it.
+ */
+ if (test_bit(CRASHED_PRIMARY, &device->flags) &&
+ drbd_md_test_flag(device->ldev, MDF_PRIMARY_IND))
+ drbd_md_set_bitmaps_authoritative(device);
+
+ drbd_try_suspend_al(device); /* IO is still suspended here... */
+
+ /* Not drbd_update_mdf_al_disabled(): its get_ldev() fails at
+ * D_ATTACHING, leaving the flag at the previous session's value.
+ */
+ rcu_read_lock();
+ al_updates = rcu_dereference(device->ldev->disk_conf)->al_updates;
+ rcu_read_unlock();
+ __update_mdf_al_disabled(device, al_updates, NOW);
+
+ /* change_disk_state uses disk_state_from_md(device); in case D_NEGOTIATING not
+ necessary, and falls back to a local state change */
+ rv = stable_state_change(resource, change_disk_state(device,
+ D_NEGOTIATING, CS_VERBOSE | CS_SERIALIZE, "attach", NULL));
+
+ if (rv < SS_SUCCESS) {
+ if (rv == SS_CW_FAILED_BY_PEER)
+ drbd_adm_msg(adm_ctx, "%s",
+ "Probably this node is marked as intentional diskless on a peer");
+ retcode = rv;
+ goto force_diskless_dec;
+ }
+
+ device->device_conf.intentional_diskless = false; /* just in case... */
+
+ mod_timer(&device->request_timer, jiffies + HZ);
+
+ if (resource->role[NOW] == R_PRIMARY
+ && device->ldev->md.current_uuid != UUID_JUST_CREATED)
+ device->ldev->md.current_uuid |= UUID_PRIMARY;
+ else
+ device->ldev->md.current_uuid &= ~UUID_PRIMARY;
+
+ drbd_md_sync(device);
+
+ kobject_uevent(&disk_to_dev(device->vdisk)->kobj, KOBJ_CHANGE);
+ put_ldev(device);
+ mutex_unlock(&resource->adm_mutex);
+
+ /*
+ * Wait for UUID negotiation with connected peers to settle before
+ * reporting the outcome to the caller. In the no-peers case
+ * disk_state_from_md() resolves D_NEGOTIATING immediately inside the
+ * state change, so wait_event returns without blocking.
+ *
+ * D_DETACHING is set by sanitize_state() when all connected peers
+ * return L_NEG_NO_RESULT (stale re-attach with diverged data). It
+ * will transition to D_DISKLESS via go_diskless(), so wait for that
+ * transition too before reporting the error.
+ */
+ wait_event(device->misc_wait,
+ (ds = device->disk_state[NOW]) != D_ATTACHING &&
+ ds != D_NEGOTIATING &&
+ ds != D_DETACHING);
+
+ if (ds == D_DISKLESS || ds == D_FAILED) {
+ drbd_adm_msg(adm_ctx,
+ "attach negotiation ended in unexpected state %s",
+ drbd_disk_str(ds));
+ retcode = ds == D_FAILED ? ERR_IO_MD_DISK : ERR_DATA_NOT_CURRENT;
+ }
+ goto out_no_adm_mutex;
+
+ force_diskless_dec:
+ put_ldev(device);
+ force_diskless:
+ change_disk_state(device, D_DISKLESS, CS_HARD, "attach", NULL);
+ fail:
+ kfree(bitmap); /* free unpublished local; NULL after publication */
+ /* A bitmap this attach did not publish belongs to a backing device
+ * whose drbd_ldev_destroy() has not run yet.
+ */
+ if (published_bitmap)
+ drbd_bm_free(device);
+ mutex_unlock_cond(&resource->conf_update, &have_conf_update);
+ drbd_backing_dev_free(device, nbc);
+ mutex_unlock(&resource->adm_mutex);
+ out_no_adm_mutex:
+ adm_ctx->result = retcode;
+ return 0;
+}
+
+static enum drbd_disk_state get_disk_state(struct drbd_device *device)
+{
+ struct drbd_resource *resource = device->resource;
+ enum drbd_disk_state disk_state;
+
+ read_lock_irq(&resource->state_rwlock);
+ disk_state = device->disk_state[NOW];
+ read_unlock_irq(&resource->state_rwlock);
+ return disk_state;
+}
+
+static int adm_detach(struct drbd_device *device, bool force, bool intentional_diskless,
+ const char *tag, struct drbd_adm_ctx *ctx)
+{
+ const char *err_str = NULL;
+ int ret, retcode;
+
+ device->device_conf.intentional_diskless = intentional_diskless;
+ if (force) {
+ set_bit(FORCE_DETACH, &device->flags);
+ change_disk_state(device, D_DETACHING, CS_HARD, tag, NULL);
+ retcode = SS_SUCCESS;
+ goto out;
+ }
+
+ /* so no-one is stuck in drbd_al_begin_io */
+ ret = drbd_suspend_io_interruptible(device, READ_AND_WRITE);
+ if (ret) {
+ device->device_conf.intentional_diskless = false;
+ retcode = ERR_INTR;
+ goto out;
+ }
+ retcode = stable_state_change(device->resource,
+ change_disk_state(device, D_DETACHING,
+ CS_VERBOSE | CS_SERIALIZE, tag, &err_str));
+ /*
+ * D_DETACHING will transition to DISKLESS.
+ * I did not use CS_WAIT_COMPLETE above since that would deadlock on a backing device that
+ * does not finish the I/O requests from writing to internal meta-data. Instead, I
+ * explicitly flush the worker queue here to ensure w_after_state_change() is completed.
+ */
+ drbd_flush_workqueue_interruptible(device);
+
+ drbd_resume_io(device);
+ ret = wait_event_interruptible(device->misc_wait,
+ get_disk_state(device) != D_DETACHING);
+ if (retcode >= SS_SUCCESS) {
+ wait_event_interruptible(device->misc_wait, !test_bit(GOING_DISKLESS, &device->flags));
+
+ device->al_writ_cnt = 0;
+ device->bm_writ_cnt = 0;
+ device->read_cnt = 0;
+ device->writ_cnt = 0;
+ clear_bit(AL_SUSPENDED, &device->flags);
+ } else {
+ device->device_conf.intentional_diskless = false;
+ }
+ if (retcode == SS_IS_DISKLESS)
+ retcode = SS_NOTHING_TO_DO;
+ if (ret)
+ retcode = ERR_INTR;
+out:
+ if (err_str) {
+ drbd_adm_msg(ctx, "%s", err_str);
+ kfree(err_str);
+ } else if (retcode == SS_NO_UP_TO_DATE_DISK)
+ put_device_opener_info(device, ctx);
+ return retcode;
+}
+
+/* Detaching the disk is a process in multiple stages. First we need to lock
+ * out application IO, in-flight IO, IO stuck in drbd_al_begin_io.
+ * Then we transition to D_DISKLESS, and wait for put_ldev() to return all
+ * internal references as well.
+ * Only then we have finally detached. */
+int drbd_adm_detach(struct drbd_adm_ctx *adm_ctx)
+{
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_detach_parms parms = { };
+ int err;
+
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_DETACH_PARMS)) {
+ err = drbd_adm_overlay_detach_parms(adm_ctx, &parms);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto out;
+ }
+ }
+
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
+ }
+ retcode = (enum drbd_ret_code)adm_detach(adm_ctx->device, parms.force_detach,
+ parms.intentional_diskless_detach, "detach", adm_ctx);
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+
+out:
+ adm_ctx->result = retcode;
+ return 0;
+}
+
+static bool conn_resync_running(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ bool rv = false;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (peer_device->repl_state[NOW] == L_SYNC_SOURCE ||
+ peer_device->repl_state[NOW] == L_SYNC_TARGET ||
+ peer_device->repl_state[NOW] == L_PAUSED_SYNC_S ||
+ peer_device->repl_state[NOW] == L_PAUSED_SYNC_T) {
+ rv = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+static bool conn_ov_running(struct drbd_connection *connection)
+{
+ struct drbd_peer_device *peer_device;
+ bool rv = false;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (peer_device->repl_state[NOW] == L_VERIFY_S ||
+ peer_device->repl_state[NOW] == L_VERIFY_T) {
+ rv = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return rv;
+}
+
+static enum drbd_ret_code
+_check_net_options(struct drbd_connection *connection, struct drbd_net_conf *old_net_conf,
+ struct drbd_net_conf *new_net_conf)
+{
+ if (old_net_conf && connection->cstate[NOW] == C_CONNECTED && connection->agreed_pro_version < 100) {
+ if (new_net_conf->wire_protocol != old_net_conf->wire_protocol)
+ return ERR_NEED_APV_100;
+
+ if (new_net_conf->two_primaries != old_net_conf->two_primaries)
+ return ERR_NEED_APV_100;
+
+ if (strcmp(new_net_conf->integrity_alg, old_net_conf->integrity_alg))
+ return ERR_NEED_APV_100;
+ }
+
+ if (!new_net_conf->two_primaries &&
+ connection->resource->role[NOW] == R_PRIMARY &&
+ connection->peer_role[NOW] == R_PRIMARY)
+ return ERR_NEED_ALLOW_TWO_PRI;
+
+ if (new_net_conf->two_primaries &&
+ (new_net_conf->wire_protocol != DRBD_PROT_C))
+ return ERR_NOT_PROTO_C;
+
+ if (new_net_conf->wire_protocol == DRBD_PROT_A &&
+ new_net_conf->fencing_policy == FP_STONITH)
+ return ERR_STONITH_AND_PROT_A;
+
+ if (new_net_conf->on_congestion != OC_BLOCK &&
+ new_net_conf->wire_protocol != DRBD_PROT_A)
+ return ERR_CONG_NOT_PROTO_A;
+
+ return NO_ERROR;
+}
+
+static enum drbd_ret_code
+check_net_options(struct drbd_connection *connection, struct drbd_net_conf *new_net_conf)
+{
+ enum drbd_ret_code rv;
+
+ rcu_read_lock();
+ rv = _check_net_options(connection, rcu_dereference(connection->transport.net_conf), new_net_conf);
+ rcu_read_unlock();
+
+ return rv;
+}
+
+struct crypto {
+ struct crypto_shash *verify_tfm;
+ struct crypto_shash *csums_tfm;
+ struct crypto_shash *cram_hmac_tfm;
+ struct crypto_shash *integrity_tfm;
+};
+
+static bool needs_key(struct crypto_shash *h)
+{
+ return h && (crypto_shash_get_flags(h) & CRYPTO_TFM_NEED_KEY);
+}
+
+/**
+ * alloc_shash() - Allocate a keyed or unkeyed shash algorithm
+ * @tfm: Destination crypto_shash
+ * @tfm_name: Which algorithm to use
+ * @type: The functionality that the hash is used for
+ * @must_unkeyed: If set, a check is included which ensures that the algorithm
+ * does not require a key
+ * @ctx: for sending detailed error description to user-space
+ */
+static int
+alloc_shash(struct crypto_shash **tfm, char *tfm_name, const char *type, bool must_unkeyed,
+ struct drbd_adm_ctx *ctx)
+{
+ if (!tfm_name[0])
+ return 0;
+
+ *tfm = crypto_alloc_shash(tfm_name, 0, 0);
+ if (IS_ERR(*tfm)) {
+ drbd_adm_msg(ctx, "failed to allocate %s for %s\n", tfm_name, type);
+ *tfm = NULL;
+ return -EINVAL;
+ }
+
+ if (must_unkeyed && needs_key(*tfm)) {
+ drbd_adm_msg(ctx,
+ "may not use %s for %s. It requires an unkeyed algorithm\n",
+ tfm_name, type);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static enum drbd_ret_code
+alloc_crypto(struct crypto *crypto, struct drbd_net_conf *new_net_conf, struct drbd_adm_ctx *ctx)
+{
+ char hmac_name[CRYPTO_MAX_ALG_NAME];
+ int digest_size = 0;
+ int err;
+
+ err = alloc_shash(&crypto->csums_tfm, new_net_conf->csums_alg,
+ "csums", true, ctx);
+ if (err)
+ return ERR_CSUMS_ALG;
+
+ err = alloc_shash(&crypto->verify_tfm, new_net_conf->verify_alg,
+ "verify", true, ctx);
+ if (err)
+ return ERR_VERIFY_ALG;
+
+ err = alloc_shash(&crypto->integrity_tfm, new_net_conf->integrity_alg,
+ "integrity", true, ctx);
+ if (err)
+ return ERR_INTEGRITY_ALG;
+
+ if (crypto->integrity_tfm) {
+ const int max_digest_size = sizeof(((struct drbd_connection *)0)->scratch_buffer.d.before);
+
+ digest_size = crypto_shash_digestsize(crypto->integrity_tfm);
+ if (digest_size > max_digest_size) {
+ drbd_adm_msg(ctx,
+ "we currently support only digest sizes <= %d bits, but digest size of %s is %d bits\n",
+ max_digest_size * 8, new_net_conf->integrity_alg, digest_size * 8);
+ return ERR_INTEGRITY_ALG;
+ }
+ }
+
+ if (new_net_conf->cram_hmac_alg[0] != 0) {
+ snprintf(hmac_name, CRYPTO_MAX_ALG_NAME, "hmac(%s)",
+ new_net_conf->cram_hmac_alg);
+
+ err = alloc_shash(&crypto->cram_hmac_tfm, hmac_name,
+ "hmac", false, ctx);
+ if (err)
+ return ERR_AUTH_ALG;
+ }
+
+ return NO_ERROR;
+}
+
+static void free_crypto(struct crypto *crypto)
+{
+ crypto_free_shash(crypto->cram_hmac_tfm);
+ crypto_free_shash(crypto->integrity_tfm);
+ crypto_free_shash(crypto->csums_tfm);
+ crypto_free_shash(crypto->verify_tfm);
+}
+
+int drbd_adm_net_opts(struct drbd_adm_ctx *adm_ctx)
+{
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_connection *connection;
+ struct drbd_peer_device *peer_device;
+ struct drbd_transport *transport;
+ struct drbd_net_conf *old_net_conf, *new_net_conf = NULL;
+ int err, vnr;
+ int ovr; /* online verify running */
+ int rsr; /* re-sync running */
+ struct crypto crypto = { };
+
+ connection = adm_ctx->connection;
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
+
+ new_net_conf = kzalloc_obj(struct drbd_net_conf);
+ if (!new_net_conf) {
+ retcode = ERR_NOMEM;
+ goto out;
+ }
+
+ drbd_flush_workqueue(&connection->sender_work);
+
+ mutex_lock(&connection->resource->conf_update);
+ mutex_lock(&connection->mutex[DATA_STREAM]);
+ transport = &connection->transport;
+ old_net_conf = transport->net_conf;
+
+ if (!old_net_conf) {
+ drbd_adm_msg(adm_ctx, "%s", "net conf missing, try connect");
+ retcode = ERR_INVALID_REQUEST;
+ goto fail;
+ }
+
+ *new_net_conf = *old_net_conf;
+ if (adm_ctx->set_defaults)
+ drbd_set_net_conf_defaults(new_net_conf);
+
+ /* The transport_name is immutable taking precedence over drbd_set_net_conf_defaults() */
+ memcpy(new_net_conf->transport_name, old_net_conf->transport_name,
+ old_net_conf->transport_name_len);
+ new_net_conf->transport_name_len = old_net_conf->transport_name_len;
+ new_net_conf->load_balance_paths = old_net_conf->load_balance_paths;
+
+ err = drbd_adm_overlay_net_conf(adm_ctx, new_net_conf);
+ if (err && err != -ENOMSG) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto fail;
+ }
+
+ if (adm_ctx->d->attr_present(adm_ctx, DRBD_ADM_F_NET_TRANSPORT_NAME) ||
+ adm_ctx->d->attr_present(adm_ctx, DRBD_ADM_F_NET_LOAD_BALANCE_PATHS)) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg(adm_ctx, "%s", "cannot change invariant setting");
+ goto fail;
+ }
+
+ retcode = check_net_options(connection, new_net_conf);
+ if (retcode != NO_ERROR)
+ goto fail;
+
+ /* re-sync running */
+ rsr = conn_resync_running(connection);
+ if (rsr && strcmp(new_net_conf->csums_alg, old_net_conf->csums_alg)) {
+ retcode = ERR_CSUMS_RESYNC_RUNNING;
+ goto fail;
+ }
+
+ /* online verify running */
+ ovr = conn_ov_running(connection);
+ if (ovr && strcmp(new_net_conf->verify_alg, old_net_conf->verify_alg)) {
+ retcode = ERR_VERIFY_RUNNING;
+ goto fail;
+ }
+
+ retcode = alloc_crypto(&crypto, new_net_conf, adm_ctx);
+ if (retcode != NO_ERROR)
+ goto fail;
+
+ /* Call before updating net_conf in case the transport needs to compare
+ * old and new configurations. */
+ err = transport->class->ops.net_conf_change(transport, new_net_conf);
+ if (err) {
+ drbd_adm_msg(adm_ctx, "transport net_conf_change failed: %d", err);
+ retcode = ERR_INVALID_REQUEST;
+ goto fail;
+ }
+
+ rcu_assign_pointer(transport->net_conf, new_net_conf);
+ connection->fencing_policy = new_net_conf->fencing_policy;
+
+ /* max-buffers may have changed: let the hint be logged again */
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ clear_bit(RS_MXB_LIMITED_LOGGED, peer_device->flags);
+
+ if (!rsr) {
+ crypto_free_shash(connection->csums_tfm);
+ connection->csums_tfm = crypto.csums_tfm;
+ crypto.csums_tfm = NULL;
+ }
+ if (!ovr) {
+ crypto_free_shash(connection->verify_tfm);
+ connection->verify_tfm = crypto.verify_tfm;
+ crypto.verify_tfm = NULL;
+ }
+
+ crypto_free_shash(connection->integrity_tfm);
+ connection->integrity_tfm = crypto.integrity_tfm;
+ if (connection->cstate[NOW] >= C_CONNECTED && connection->agreed_pro_version >= 100)
+ /* Do this without trying to take connection->data.mutex again. */
+ __drbd_send_protocol(connection, P_PROTOCOL_UPDATE);
+
+ crypto_free_shash(connection->cram_hmac_tfm);
+ connection->cram_hmac_tfm = crypto.cram_hmac_tfm;
+
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+ mutex_unlock(&connection->resource->conf_update);
+ kvfree_rcu_mightsleep(old_net_conf);
+
+ if (connection->cstate[NOW] >= C_CONNECTED) {
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ drbd_send_sync_param(peer_device);
+ }
+
+ goto out;
+
+ fail:
+ mutex_unlock(&connection->mutex[DATA_STREAM]);
+ mutex_unlock(&connection->resource->conf_update);
+ free_crypto(&crypto);
+ kfree(new_net_conf);
+ out:
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ out_no_adm_mutex:
+ adm_ctx->result = retcode;
+ return 0;
+}
+
+static int adjust_resync_fifo(struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_conf *conf,
+ struct fifo_buffer **pp_old_plan)
+{
+ struct fifo_buffer *old_plan, *new_plan = NULL;
+ unsigned int fifo_size;
+
+ fifo_size = (conf->c_plan_ahead * 10 * RS_MAKE_REQS_INTV) / HZ;
+
+ old_plan = rcu_dereference_protected(peer_device->rs_plan_s,
+ lockdep_is_held(&peer_device->connection->resource->conf_update));
+ if (!old_plan || fifo_size != old_plan->size) {
+ new_plan = fifo_alloc(fifo_size);
+ if (!new_plan) {
+ drbd_err(peer_device, "kmalloc of fifo_buffer failed");
+ return -ENOMEM;
+ }
+ rcu_assign_pointer(peer_device->rs_plan_s, new_plan);
+ if (pp_old_plan)
+ *pp_old_plan = old_plan;
+ }
+
+ return 0;
+}
+
+int drbd_adm_peer_device_opts(struct drbd_adm_ctx *adm_ctx)
+{
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_peer_device *peer_device;
+ struct drbd_peer_device_conf *old_peer_device_conf, *new_peer_device_conf = NULL;
+ struct fifo_buffer *old_plan = NULL;
+ struct drbd_device *device;
+ bool notify = false;
+ int err;
+
+ peer_device = adm_ctx->peer_device;
+ device = peer_device->device;
+
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
+ mutex_lock(&adm_ctx->resource->conf_update);
+
+ new_peer_device_conf = kzalloc_obj(struct drbd_peer_device_conf);
+ if (!new_peer_device_conf)
+ goto fail;
+
+ old_peer_device_conf = peer_device->conf;
+ *new_peer_device_conf = *old_peer_device_conf;
+ if (adm_ctx->set_defaults)
+ drbd_set_peer_device_conf_defaults(new_peer_device_conf);
+
+ err = drbd_adm_overlay_peer_device_conf(adm_ctx, new_peer_device_conf);
+ if (err && err != -ENOMSG) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto fail_ret_set;
+ }
+
+ if (!old_peer_device_conf->bitmap && new_peer_device_conf->bitmap &&
+ peer_device->bitmap_index == -1) {
+ if (get_ldev(device)) {
+ err = allocate_bitmap_index(peer_device, device->ldev);
+ put_ldev(device);
+ if (err) {
+ drbd_adm_msg(adm_ctx, "%s",
+ "No bitmap slot available in meta-data");
+ retcode = ERR_INVALID_REQUEST;
+ goto fail_ret_set;
+ }
+ drbd_info(peer_device,
+ "Former intentional diskless peer got bitmap slot %d\n",
+ peer_device->bitmap_index);
+ drbd_md_sync(device);
+ notify = true;
+ }
+ }
+
+ if (old_peer_device_conf->bitmap && !new_peer_device_conf->bitmap) {
+ enum drbd_disk_state pdsk = peer_device->disk_state[NOW];
+ enum drbd_disk_state disk = device->disk_state[NOW];
+
+ if (!(disk == D_DISKLESS || pdsk == D_DISKLESS || pdsk == D_UNKNOWN)) {
+ drbd_adm_msg(adm_ctx, "%s",
+ "Can not drop the bitmap when both sides have a disk");
+ retcode = ERR_INVALID_REQUEST;
+ goto fail_ret_set;
+ }
+ err = clear_peer_slot(device, peer_device->node_id, MDF_NODE_EXISTS);
+ if (!err) {
+ peer_device->bitmap_index = -1;
+ notify = true;
+ }
+ }
+
+ if (!expect(peer_device, new_peer_device_conf->resync_rate >= 1))
+ new_peer_device_conf->resync_rate = 1;
+
+ if (new_peer_device_conf->c_plan_ahead > DRBD_C_PLAN_AHEAD_MAX)
+ new_peer_device_conf->c_plan_ahead = DRBD_C_PLAN_AHEAD_MAX;
+
+ /* c-plan-ahead 0 selects the fixed resync-rate, which the sender paces
+ * against c-max-rate as well. Say so, but leave the configured value
+ * alone: clamping it here would make every later "drbdadm adjust" see a
+ * setting that differs from the configuration file and set it again.
+ */
+ if (new_peer_device_conf->c_plan_ahead == 0 &&
+ new_peer_device_conf->c_max_rate &&
+ new_peer_device_conf->c_max_rate < new_peer_device_conf->resync_rate)
+ drbd_adm_msg(adm_ctx,
+ "c-plan-ahead 0: resync-rate %u KiB/s is above c-max-rate %u KiB/s, the resync will run at c-max-rate",
+ new_peer_device_conf->resync_rate,
+ new_peer_device_conf->c_max_rate);
+
+ err = adjust_resync_fifo(peer_device, new_peer_device_conf, &old_plan);
+ if (err)
+ goto fail;
+
+ rcu_assign_pointer(peer_device->conf, new_peer_device_conf);
+
+ kvfree_rcu_mightsleep(old_peer_device_conf);
+ kfree(old_plan);
+
+ /* No need to call drbd_send_sync_param() here. The values in
+ * peer_device->conf that we send are ignored by recent peers anyway. */
+
+ if (0) {
+fail:
+ retcode = ERR_NOMEM;
+fail_ret_set:
+ kfree(new_peer_device_conf);
+ }
+
+ mutex_unlock(&adm_ctx->resource->conf_update);
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+out_no_adm_mutex:
+ if (notify)
+ drbd_broadcast_peer_device_state(peer_device);
+ adm_ctx->result = retcode;
+ return 0;
+
+}
+
+int drbd_create_peer_device_default_config(struct drbd_peer_device *peer_device)
+{
+ struct drbd_peer_device_conf *conf;
+ int err;
+
+ conf = kzalloc_obj(*conf);
+ if (!conf)
+ return -ENOMEM;
+
+ drbd_set_peer_device_conf_defaults(conf);
+ err = adjust_resync_fifo(peer_device, conf, NULL);
+ if (err)
+ return err;
+
+ peer_device->conf = conf;
+
+ return 0;
+}
+
+static void connection_to_info(struct drbd_connection_info *info,
+ struct drbd_connection *connection)
+{
+ info->conn_connection_state = connection->cstate[NOW];
+ info->conn_role = connection->peer_role[NOW];
+#ifdef CONFIG_DRBD_COMPAT_84
+ /* No "before" state of its own; see resource_to_info() above. */
+ info->old_conn_connection_state = info->conn_connection_state;
+ info->old_conn_role = info->conn_role;
+#endif
+}
+
+#define str_to_info(info, field, str) ({ \
+ strscpy(info->field, str, sizeof(info->field)); \
+ info->field ## _len = min(strlen(str), sizeof(info->field)); \
+})
+
+/* shared logic between peer_device_to_info and peer_device_state_change_to_info */
+static void __peer_device_to_info(struct drbd_peer_device_info *info,
+ struct drbd_peer_device *peer_device,
+ enum which_state which)
+{
+ info->peer_resync_susp_dependency = resync_susp_comb_dep(peer_device, which);
+ info->peer_is_intentional_diskless = !want_bitmap(peer_device);
+}
+
+static void peer_device_to_info(struct drbd_peer_device_info *info,
+ struct drbd_peer_device *peer_device)
+{
+ info->peer_repl_state = peer_device->repl_state[NOW];
+ info->peer_disk_state = peer_device->disk_state[NOW];
+ info->peer_resync_susp_user = peer_device->resync_susp_user[NOW];
+ info->peer_resync_susp_peer = peer_device->resync_susp_peer[NOW];
+ info->peer_resync_susp_max_parallel = peer_device->resync_susp_max_parallel[NOW];
+ __peer_device_to_info(info, peer_device, NOW);
+#ifdef CONFIG_DRBD_COMPAT_84
+ /*
+ * No "before" state of its own; see resource_to_info() above. This
+ * matters in particular for drbd_broadcast_peer_device_state()'s
+ * RS_PROGRESS-driven NOTIFY_CHANGE call (drbd_nl.c): it fires far
+ * more often than the resync bitmap-writeout path's own throttled
+ * SIB_SYNC_PROGRESS emission, and without old == new here it would
+ * feed compat84_notify_peer_device_state()'s fused-event code a
+ * fabricated repl/disk-state transition from uninitialized stack
+ * data instead of correctly reporting no change.
+ */
+ info->old_peer_repl_state = info->peer_repl_state;
+ info->old_peer_disk_state = info->peer_disk_state;
+ info->old_peer_resync_susp_user = info->peer_resync_susp_user;
+ info->old_peer_resync_susp_peer = info->peer_resync_susp_peer;
+ info->old_peer_resync_susp_dependency = info->peer_resync_susp_dependency;
+#endif
+}
+
+void peer_device_state_change_to_info(struct drbd_peer_device_info *info,
+ struct drbd_peer_device_state_change *state_change)
+{
+ info->peer_repl_state = state_change->repl_state[NEW];
+ info->peer_disk_state = state_change->disk_state[NEW];
+ info->peer_resync_susp_user = state_change->resync_susp_user[NEW];
+ info->peer_resync_susp_peer = state_change->resync_susp_peer[NEW];
+ info->peer_resync_susp_max_parallel = state_change->resync_susp_max_parallel[NEW];
+ __peer_device_to_info(info, state_change->peer_device, NEW);
+#ifdef CONFIG_DRBD_COMPAT_84
+ info->old_peer_repl_state = state_change->repl_state[OLD];
+ info->old_peer_disk_state = state_change->disk_state[OLD];
+ /*
+ * Exact pre-transition values, straight from the frozen snapshot
+ * (unlike the live peer_device object, where which_state's OLD
+ * aliases NOW and would already read the post-transition value by
+ * the time a notification runs). old_peer_resync_susp_dependency
+ * combines resync_susp_dependency[OLD] and resync_susp_other_c[OLD],
+ * but not resync_susp_comb_dep()'s third term (sync source with an
+ * inconsistent local disk): that needs the sibling device's
+ * disk_state[OLD], which this callback cannot reach. The v1 dialect
+ * allows a single peer only, and with one peer that term is always
+ * false, so the result equals resync_susp_comb_dep().
+ */
+ info->old_peer_resync_susp_user = state_change->resync_susp_user[OLD];
+ info->old_peer_resync_susp_peer = state_change->resync_susp_peer[OLD];
+ info->old_peer_resync_susp_dependency =
+ state_change->resync_susp_dependency[OLD] || state_change->resync_susp_other_c[OLD];
+#endif
+}
+
+/* shared logic between device_to_info and device_state_change_to_info */
+static void __device_to_info(struct drbd_device_info *info,
+ struct drbd_device *device)
+{
+ info->is_intentional_diskless = device->device_conf.intentional_diskless;
+ info->dev_is_open = device->open_cnt != 0;
+
+ rcu_read_lock();
+ if (get_ldev_if_state(device, D_FAILED)) {
+ struct drbd_disk_conf *disk_conf =
+ rcu_dereference(device->ldev->disk_conf);
+ str_to_info(info, backing_dev_path, disk_conf->backing_dev);
+ put_ldev(device);
+ } else {
+ info->backing_dev_path[0] = '\0';
+ info->backing_dev_path_len = 0;
+ }
+ rcu_read_unlock();
+}
+
+void device_to_info(struct drbd_device_info *info,
+ struct drbd_device *device)
+{
+ info->dev_disk_state = device->disk_state[NOW];
+ info->dev_has_quorum = device->have_quorum[NOW];
+ __device_to_info(info, device);
+#ifdef CONFIG_DRBD_COMPAT_84
+ /* No "before" state of its own; see resource_to_info() above. */
+ info->old_dev_disk_state = info->dev_disk_state;
+#endif
+}
+
+void device_state_change_to_info(struct drbd_device_info *info,
+ struct drbd_device_state_change *state_change)
+{
+ info->dev_disk_state = state_change->disk_state[NEW];
+ info->dev_has_quorum = state_change->have_quorum[NEW];
+ __device_to_info(info, state_change->device);
+#ifdef CONFIG_DRBD_COMPAT_84
+ info->old_dev_disk_state = state_change->disk_state[OLD];
+#endif
+}
+
+static bool is_resync_target_in_other_connection(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_peer_device *p;
+
+ for_each_peer_device(p, device) {
+ if (p == peer_device)
+ continue;
+
+ if (p->repl_state[NOW] == L_SYNC_TARGET)
+ return true;
+ }
+
+ return false;
+}
+
+static enum drbd_ret_code drbd_check_name_str(const char *name, const bool strict);
+static void drbd_msg_put_name_error(struct drbd_adm_ctx *ctx, enum drbd_ret_code ret_code);
+
+static enum drbd_ret_code drbd_check_conn_name(struct drbd_resource *resource, const char *new_name)
+{
+ struct drbd_connection *connection;
+ enum drbd_ret_code retcode = NO_ERROR;
+ const char *tmp_name;
+
+ retcode = drbd_check_name_str(new_name, drbd_strict_names);
+ if (retcode != NO_ERROR)
+ return retcode;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ /* is this even possible? */
+ if (!connection->transport.net_conf)
+ continue;
+ tmp_name = connection->transport.net_conf->name;
+ if (!tmp_name)
+ continue;
+ if (strcmp(tmp_name, new_name))
+ continue;
+ retcode = ERR_ALREADY_EXISTS;
+ break;
+ }
+ rcu_read_unlock();
+ return retcode;
+}
+
+static int adm_new_connection(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_connection_info connection_info;
+ enum drbd_notification_type flags;
+ unsigned int peer_devices = 0;
+ struct drbd_device *device;
+ struct drbd_peer_device *peer_device;
+ struct drbd_net_conf *old_net_conf, *new_net_conf = NULL;
+ struct crypto crypto = { NULL, };
+ struct drbd_connection *connection;
+ enum drbd_ret_code retcode = NO_ERROR;
+ int i, err;
+ char *transport_name;
+ struct drbd_transport_class *tr_class;
+ struct drbd_transport *transport;
+ struct drbd_connection *connection2;
+ int pre_110_node_id = -1, pre_110_pro = 0;
+
+ /* allocation not in the IO path, drbdsetup / netlink process context */
+ new_net_conf = kzalloc_obj(*new_net_conf);
+ if (!new_net_conf)
+ return ERR_NOMEM;
+
+ drbd_set_net_conf_defaults(new_net_conf);
+
+ err = drbd_adm_overlay_net_conf(adm_ctx, new_net_conf);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto fail;
+ }
+
+ retcode = drbd_check_conn_name(adm_ctx->resource, new_net_conf->name);
+ if (retcode != NO_ERROR) {
+ drbd_msg_put_name_error(adm_ctx, retcode);
+ goto fail;
+ }
+
+ transport_name = new_net_conf->transport_name_len ? new_net_conf->transport_name :
+ new_net_conf->load_balance_paths ? "lb-tcp" : "tcp";
+ tr_class = drbd_get_transport_class(transport_name);
+ if (!tr_class) {
+ retcode = ERR_CREATE_TRANSPORT;
+ goto fail;
+ }
+
+ connection = drbd_create_connection(adm_ctx->resource, tr_class);
+ if (!connection) {
+ retcode = ERR_NOMEM;
+ goto fail_put_transport;
+ }
+ connection->peer_node_id = adm_ctx->peer_node_id;
+ /* transport class reference now owned by connection,
+ * prevent double cleanup. */
+ tr_class = NULL;
+
+ mutex_lock(&adm_ctx->resource->conf_update);
+ retcode = check_net_options(connection, new_net_conf);
+ if (retcode != NO_ERROR)
+ goto unlock_fail_free_connection;
+
+ retcode = alloc_crypto(&crypto, new_net_conf, adm_ctx);
+ if (retcode != NO_ERROR)
+ goto unlock_fail_free_connection;
+
+ ((char *)new_net_conf->shared_secret)[SHARED_SECRET_MAX-1] = 0;
+
+ idr_for_each_entry(&adm_ctx->resource->devices, device, i) {
+ int id;
+
+ retcode = ERR_NOMEM;
+ peer_device = create_peer_device(device, connection);
+ if (!peer_device)
+ goto unlock_fail_free_connection;
+ id = idr_alloc(&connection->peer_devices, peer_device,
+ device->vnr, device->vnr + 1, GFP_KERNEL);
+ if (id < 0)
+ goto unlock_fail_free_connection;
+
+ if (get_ldev(device)) {
+ struct drbd_peer_md *peer_md =
+ &device->ldev->md.peers[adm_ctx->peer_node_id];
+ if (test_bit(__MDF_PEER_OUTDATED, &peer_md->flags))
+ peer_device->disk_state[NOW] = D_OUTDATED;
+ put_ldev(device);
+ }
+ }
+
+ /* Set bitmap_index if it was allocated previously */
+ idr_for_each_entry(&connection->peer_devices, peer_device, i) {
+ unsigned int bitmap_index;
+
+ device = peer_device->device;
+ if (!get_ldev(device))
+ continue;
+
+ bitmap_index = device->ldev->md.peers[adm_ctx->peer_node_id].bitmap_index;
+ if (bitmap_index != -1) {
+ if (want_bitmap(peer_device))
+ peer_device->bitmap_index = bitmap_index;
+ else
+ clear_bit(__MDF_HAVE_BITMAP,
+ &device->ldev->md.peers[adm_ctx->peer_node_id].flags);
+ }
+ put_ldev(device);
+ }
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, i) {
+ peer_device->send_cnt = 0;
+ peer_device->recv_cnt = 0;
+ }
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, i) {
+ struct drbd_device *device = peer_device->device;
+
+ peer_device->resync_susp_other_c[NOW] =
+ is_resync_target_in_other_connection(peer_device);
+ list_add_rcu(&peer_device->peer_devices, &device->peer_devices);
+ kref_get(&connection->kref);
+ kref_get(&device->kref);
+ peer_devices++;
+ peer_device->node_id = connection->peer_node_id;
+ }
+
+ write_lock_irq(&adm_ctx->resource->state_rwlock);
+
+ /*
+ * Initialize to the current dagtag so that flushes can be acked even
+ * if no further writes occur.
+ */
+ connection->last_peer_ack_dagtag_seen = READ_ONCE(adm_ctx->resource->dagtag_sector);
+
+ /* A peer below protocol 110 must stay the only one (drbd_do_features()) */
+ for_each_connection(connection2, adm_ctx->resource) {
+ if (test_bit(CONN_PRE_110, &connection2->flags)) {
+ pre_110_node_id = connection2->peer_node_id;
+ pre_110_pro = connection2->agreed_pro_version;
+ break;
+ }
+ }
+ if (pre_110_node_id >= 0) {
+ write_unlock_irq(&adm_ctx->resource->state_rwlock);
+ drbd_adm_msg(adm_ctx,
+ "peer %d is connected with protocol %d, which allows no other connection",
+ pre_110_node_id, pre_110_pro);
+ retcode = ERR_INVALID_REQUEST;
+ goto unlock_fail_free_connection;
+ }
+
+ list_add_tail_rcu(&connection->connections, &adm_ctx->resource->connections);
+ write_unlock_irq(&adm_ctx->resource->state_rwlock);
+
+ transport = &connection->transport;
+ old_net_conf = transport->net_conf;
+ if (old_net_conf) {
+ retcode = ERR_NET_CONFIGURED;
+ goto unlock_fail_free_connection;
+ }
+
+ err = transport->class->ops.net_conf_change(transport, new_net_conf);
+ if (err) {
+ drbd_adm_msg(adm_ctx, "transport net_conf_change failed: %d", err);
+ retcode = ERR_INVALID_REQUEST;
+ goto unlock_fail_free_connection;
+ }
+
+ rcu_assign_pointer(transport->net_conf, new_net_conf);
+ connection->fencing_policy = new_net_conf->fencing_policy;
+
+ connection->cram_hmac_tfm = crypto.cram_hmac_tfm;
+ connection->integrity_tfm = crypto.integrity_tfm;
+ connection->csums_tfm = crypto.csums_tfm;
+ connection->verify_tfm = crypto.verify_tfm;
+
+ /* transferred ownership. prevent double cleanup. */
+ new_net_conf = NULL;
+ memset(&crypto, 0, sizeof(crypto));
+
+ if (connection->peer_node_id > adm_ctx->resource->max_node_id)
+ adm_ctx->resource->max_node_id = connection->peer_node_id;
+
+ connection_to_info(&connection_info, connection);
+ flags = (peer_devices--) ? NOTIFY_CONTINUES : 0;
+ mutex_lock(¬ification_mutex);
+ notify_connection_state(NULL, 0, connection, &connection_info, NOTIFY_CREATE | flags);
+ idr_for_each_entry(&connection->peer_devices, peer_device, i) {
+ struct drbd_peer_device_info peer_device_info;
+
+ peer_device_to_info(&peer_device_info, peer_device);
+ flags = (peer_devices--) ? NOTIFY_CONTINUES : 0;
+ notify_peer_device_state(NULL, 0, peer_device, &peer_device_info, NOTIFY_CREATE | flags);
+ }
+ mutex_unlock(¬ification_mutex);
+
+ mutex_unlock(&adm_ctx->resource->conf_update);
+
+ drbd_debugfs_connection_add(connection); /* after ->net_conf was assigned */
+ drbd_thread_start(&connection->sender);
+ return NO_ERROR;
+
+unlock_fail_free_connection:
+ drbd_unregister_connection(connection);
+ mutex_unlock(&adm_ctx->resource->conf_update);
+ synchronize_rcu();
+ drbd_reclaim_connection(&connection->rcu);
+fail_put_transport:
+ drbd_put_transport_class(tr_class);
+fail:
+ free_crypto(&crypto);
+ kfree(new_net_conf);
+
+ return retcode;
+}
+
+static bool path_my_addr_eq(const struct drbd_path *path, const struct drbd_path_parms *pp)
+{
+ return path->my_addr_len == pp->my_addr_len &&
+ memcmp(&path->my_addr, pp->my_addr, pp->my_addr_len) == 0;
+}
+
+static bool path_peer_addr_eq(const struct drbd_path *path, const struct drbd_path_parms *pp)
+{
+ return path->peer_addr_len == pp->peer_addr_len &&
+ memcmp(&path->peer_addr, pp->peer_addr, pp->peer_addr_len) == 0;
+}
+
+static enum drbd_ret_code
+check_path_against_parms(const struct drbd_path *path, const struct drbd_path_parms *pp)
+{
+ enum drbd_ret_code ret = NO_ERROR;
+
+ if (path_my_addr_eq(path, pp))
+ ret = ERR_LOCAL_ADDR;
+ if (path_peer_addr_eq(path, pp))
+ ret = (ret == ERR_LOCAL_ADDR ? ERR_LOCAL_AND_PEER_ADDR : ERR_PEER_ADDR);
+ return ret;
+}
+
+static enum drbd_ret_code
+check_path_usable(struct drbd_adm_ctx *adm_ctx, const struct drbd_path_parms *pp)
+{
+ struct drbd_resource *resource;
+ struct drbd_connection *connection;
+ enum drbd_ret_code retcode = NO_ERROR;
+
+ if (!(pp->my_addr_len && pp->peer_addr_len)) {
+ drbd_adm_msg(adm_ctx, "%s", "connection endpoint(s) missing");
+ return ERR_INVALID_REQUEST;
+ }
+
+ for_each_resource_rcu(resource, &drbd_resources) {
+ for_each_connection_rcu(connection, resource) {
+ struct drbd_path *path;
+
+ list_for_each_entry_rcu(path, &connection->transport.paths, list) {
+ retcode = check_path_against_parms(path, pp);
+ if (retcode == NO_ERROR)
+ continue;
+ /* Within the same resource, it is ok to use
+ * the same endpoint several times */
+ if (retcode != ERR_LOCAL_AND_PEER_ADDR &&
+ resource == adm_ctx->resource)
+ continue;
+ return retcode;
+ }
+ }
+ }
+ return NO_ERROR;
+}
+
+/* Check that adding the candidate path does not make incoming connections
+ * ambiguous between paths of distinct connections of the resource. Same-
+ * connection ambiguity is harmless because the assignment is irrelevant.
+ */
+static enum drbd_ret_code
+check_path_not_ambiguous(struct drbd_adm_ctx *adm_ctx,
+ struct drbd_path *candidate)
+{
+ struct drbd_resource *resource = adm_ctx->resource;
+ struct drbd_connection *connection;
+
+ for_each_connection_rcu(connection, resource) {
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_path *path;
+
+ if (connection == adm_ctx->connection)
+ continue;
+ if (transport->class != candidate->transport->class)
+ continue;
+
+ list_for_each_entry_rcu(path, &transport->paths, list) {
+ if (drbd_path_conflicts_by_listener(path, candidate)) {
+ drbd_adm_msg(adm_ctx, "%s",
+ "path indistinguishable from path in another connection");
+ return ERR_PATH_COLLISION;
+ }
+ }
+ }
+ return NO_ERROR;
+}
+
+
+static enum drbd_ret_code
+adm_add_path(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_transport *transport = &adm_ctx->connection->transport;
+ struct drbd_resource *resource = adm_ctx->resource;
+ struct drbd_connection *connection = adm_ctx->connection;
+ struct drbd_path_parms pp = { };
+ struct drbd_path *path;
+ struct net *existing_net;
+ enum drbd_ret_code retcode = NO_ERROR;
+ int err;
+
+ /* parse and validate only */
+ existing_net = drbd_net_assigned_to_connection(adm_ctx->connection);
+ if (existing_net && !net_eq(adm_ctx->net, existing_net)) {
+ drbd_adm_msg(adm_ctx, "%s",
+ "connection already assigned to a different network namespace");
+ return ERR_INVALID_REQUEST;
+ }
+
+ err = drbd_adm_overlay_path_parms(adm_ctx, &pp);
+ if (err) {
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ return ERR_MANDATORY_TAG;
+ }
+
+ path = kzalloc(transport->class->path_instance_size, GFP_KERNEL);
+ if (!path)
+ return ERR_NOMEM;
+
+ path->net = adm_ctx->net;
+ path->my_addr_len = pp.my_addr_len;
+ memcpy(&path->my_addr, pp.my_addr, path->my_addr_len);
+ path->peer_addr_len = pp.peer_addr_len;
+ memcpy(&path->peer_addr, pp.peer_addr, path->peer_addr_len);
+ path->transport = transport;
+ kref_init(&path->kref);
+ kref_get(&adm_ctx->connection->kref);
+
+ rcu_read_lock();
+ retcode = check_path_usable(adm_ctx, &pp);
+ if (retcode == NO_ERROR)
+ retcode = check_path_not_ambiguous(adm_ctx, path);
+ rcu_read_unlock();
+ if (retcode != NO_ERROR) {
+ kref_put(&path->kref, drbd_destroy_path);
+ return retcode;
+ }
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto finish;
+ if (connection->resource->res_opts.drbd8_compat_mode && resource->res_opts.node_id == -1) {
+ err = drbd_setup_node_ids_84(connection, path, adm_ctx->peer_node_id);
+ if (err) {
+ drbd_adm_msg(adm_ctx, "%s",
+ err == -ENOTUNIQ ? "node-id from drbdsetup and meta-data differ" :
+ "error setting up node IDs");
+ kref_put(&path->kref, drbd_destroy_path);
+ return ERR_INVALID_REQUEST;
+ }
+ }
- device = adm_ctx->device;
- mutex_lock(&adm_ctx->resource->adm_mutex);
- peer_device = first_peer_device(device);
- connection = peer_device->connection;
- conn_reconfig_start(connection);
+ /* Exclusive with transport op "prepare_connect()" */
+ mutex_lock(&resource->conf_update);
- /* if you want to reconfigure, please tear down first */
- if (device->state.disk > D_DISKLESS) {
- retcode = ERR_DISK_CONFIGURED;
- goto fail;
+ err = transport->class->ops.add_path(path);
+
+ if (err) {
+ kref_put(&path->kref, drbd_destroy_path);
+ drbd_err(connection, "add_path() failed with %d\n", err);
+ drbd_adm_msg(adm_ctx, "%s", "add_path on transport failed");
+ mutex_unlock(&resource->conf_update);
+ return ERR_INVALID_REQUEST;
}
- /* It may just now have detached because of IO error. Make sure
- * drbd_ldev_destroy is done already, we may end up here very fast,
- * e.g. if someone calls attach from the on-io-error handler,
- * to realize a "hot spare" feature (not that I'd recommend that) */
- wait_event(device->misc_wait, !test_bit(GOING_DISKLESS, &device->flags));
- /* make sure there is no leftover from previous force-detach attempts */
- clear_bit(FORCE_DETACH, &device->flags);
- clear_bit(WAS_IO_ERROR, &device->flags);
- clear_bit(WAS_READ_ERROR, &device->flags);
+ /* Exclusive with reading state, in particular remember_state_change() */
+ write_lock_irq(&resource->state_rwlock);
+ list_add_tail_rcu(&path->list, &transport->paths);
+ write_unlock_irq(&resource->state_rwlock);
- /* and no leftover from previously aborted resync or verify, either */
- device->rs_total = 0;
- device->rs_failed = 0;
- atomic_set(&device->rs_pending_cnt, 0);
+ mutex_unlock(&resource->conf_update);
- /* allocation not in the IO path, drbdsetup context */
- nbc = kzalloc_obj(struct drbd_backing_dev);
- if (!nbc) {
- retcode = ERR_NOMEM;
- goto fail;
+ notify_path(adm_ctx->connection, path, NOTIFY_CREATE);
+ return NO_ERROR;
+}
+
+int drbd_adm_connect(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_connect_parms parms = { 0, };
+ struct drbd_peer_device *peer_device;
+ struct drbd_connection *connection;
+ enum drbd_ret_code retcode = NO_ERROR;
+ enum drbd_state_rv rv;
+ enum drbd_conn_state cstate;
+ int i, err;
+
+ connection = adm_ctx->connection;
+ cstate = connection->cstate[NOW];
+ if (cstate != C_STANDALONE) {
+ retcode = ERR_NET_CONFIGURED;
+ goto out;
}
- spin_lock_init(&nbc->md.uuid_lock);
- new_disk_conf = kzalloc_obj(struct disk_conf);
- if (!new_disk_conf) {
- retcode = ERR_NOMEM;
- goto fail;
+ if (first_path(connection) == NULL) {
+ drbd_adm_msg(adm_ctx, "%s", "connection endpoint(s) missing");
+ retcode = ERR_INVALID_REQUEST;
+ goto out;
}
- nbc->disk_conf = new_disk_conf;
- set_disk_conf_defaults(new_disk_conf);
- err = disk_conf_from_attrs(new_disk_conf, info);
- if (err) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
- goto fail;
+ if (!net_eq(adm_ctx->net, drbd_net_assigned_to_connection(connection))) {
+ drbd_adm_msg(adm_ctx, "%s", "connection assigned to a different network namespace");
+ retcode = ERR_INVALID_REQUEST;
+ goto out;
}
- if (new_disk_conf->c_plan_ahead > DRBD_C_PLAN_AHEAD_MAX)
- new_disk_conf->c_plan_ahead = DRBD_C_PLAN_AHEAD_MAX;
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_CONNECT_PARMS)) {
+ err = drbd_adm_overlay_connect_parms(adm_ctx, &parms);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto out;
+ }
+ }
+ if (parms.discard_my_data) {
+ if (adm_ctx->resource->role[NOW] == R_PRIMARY) {
+ retcode = ERR_DISCARD_IMPOSSIBLE;
+ goto out;
+ }
+ set_bit(CONN_DISCARD_MY_DATA, &connection->flags);
+ }
+ if (parms.tentative) {
+ set_bit(CONN_DRY_RUN, &connection->flags);
+ set_bit(CONN_DRY_RUN_LOCAL, &connection->flags);
+ }
- new_plan = fifo_alloc((new_disk_conf->c_plan_ahead * 10 * SLEEP_TIME) / HZ);
- if (!new_plan) {
- retcode = ERR_NOMEM;
- goto fail;
+ /* Eventually allocate bitmap indexes for the peer_devices here */
+ idr_for_each_entry(&connection->peer_devices, peer_device, i) {
+ struct drbd_device *device;
+
+ if (peer_device->bitmap_index != -1 || !want_bitmap(peer_device))
+ continue;
+
+ device = peer_device->device;
+ if (!get_ldev(device))
+ continue;
+
+ err = allocate_bitmap_index(peer_device, device->ldev);
+ put_ldev(device);
+ if (err) {
+ retcode = ERR_INVALID_REQUEST;
+ goto out;
+ }
+ drbd_md_mark_dirty(device);
}
- if (new_disk_conf->meta_dev_idx < DRBD_MD_INDEX_FLEX_INT) {
- retcode = ERR_MD_IDX_INVALID;
- goto fail;
+ rv = change_cstate_tag(connection, C_UNCONNECTED, CS_VERBOSE, "connect", NULL);
+ adm_ctx->result = rv;
+ return 0;
+out:
+ adm_ctx->result = retcode;
+ return 0;
+}
+
+int drbd_adm_new_peer(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_connection *connection;
+ struct drbd_resource *resource;
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_device *device;
+ int vnr, n_connections = 0;
+
+ resource = adm_ctx->resource;
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
}
rcu_read_lock();
- nc = rcu_dereference(connection->net_conf);
- if (nc) {
- if (new_disk_conf->fencing == FP_STONITH && nc->wire_protocol == DRBD_PROT_A) {
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ bool fail = false;
+
+ if (get_ldev_if_state(device, D_FAILED)) {
+ fail = !device->ldev->disk_conf->d_bitmap;
+ put_ldev(device);
+ }
+ if (fail) {
rcu_read_unlock();
- retcode = ERR_STONITH_AND_PROT_A;
- goto fail;
+ retcode = ERR_INVALID_REQUEST;
+ drbd_adm_msg(adm_ctx,
+ "Cannot add a peer while having a disk without an allocated bitmap");
+ goto out_unlock;
}
}
rcu_read_unlock();
- retcode = open_backing_devices(device, new_disk_conf, nbc);
- if (retcode != NO_ERROR)
- goto fail;
-
- if ((nbc->backing_bdev == nbc->md_bdev) !=
- (new_disk_conf->meta_dev_idx == DRBD_MD_INDEX_INTERNAL ||
- new_disk_conf->meta_dev_idx == DRBD_MD_INDEX_FLEX_INT)) {
- retcode = ERR_MD_IDX_INVALID;
- goto fail;
+ for_each_connection(connection, resource)
+ n_connections++;
+ if (resource->res_opts.drbd8_compat_mode && n_connections >= 1) {
+ retcode = ERR_INVALID_REQUEST;
+ drbd_adm_msg(adm_ctx, "drbd8 compat mode allows one peer at max");
+ goto out_unlock;
}
- resync_lru = lc_create("resync", drbd_bm_ext_cache,
- 1, 61, sizeof(struct bm_extent),
- offsetof(struct bm_extent, lce));
- if (!resync_lru) {
- retcode = ERR_NOMEM;
- goto fail;
+ /* ensure uniqueness of peer_node_id by checking with adm_mutex */
+ connection = drbd_connection_by_node_id(resource, adm_ctx->peer_node_id);
+ if (adm_ctx->connection || connection) {
+ retcode = ERR_INVALID_REQUEST;
+ drbd_adm_msg(adm_ctx,
+ "Connection for peer node id %d already exists",
+ adm_ctx->peer_node_id);
+ } else {
+ retcode = adm_new_connection(adm_ctx);
}
- /* Read our meta data super block early.
- * This also sets other on-disk offsets. */
- retcode = drbd_md_read(device, nbc);
- if (retcode != NO_ERROR)
- goto fail;
-
- sanitize_disk_conf(device, new_disk_conf, nbc);
+out_unlock:
+ mutex_unlock(&resource->adm_mutex);
+out:
+ adm_ctx->result = retcode;
+ return 0;
+}
- if (drbd_get_max_capacity(nbc) < new_disk_conf->disk_size) {
- drbd_err(device, "max capacity %llu smaller than disk size %llu\n",
- (unsigned long long) drbd_get_max_capacity(nbc),
- (unsigned long long) new_disk_conf->disk_size);
- retcode = ERR_DISK_TOO_SMALL;
- goto fail;
- }
+int drbd_adm_new_path(struct drbd_adm_ctx *adm_ctx)
+{
+ enum drbd_ret_code retcode = NO_ERROR;
- if (new_disk_conf->meta_dev_idx < 0) {
- max_possible_sectors = DRBD_MAX_SECTORS_FLEX;
- /* at least one MB, otherwise it does not make sense */
- min_md_device_sectors = (2<<10);
+ /* remote transport endpoints need to be globally unique */
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
} else {
- max_possible_sectors = DRBD_MAX_SECTORS;
- min_md_device_sectors = MD_128MB_SECT * (new_disk_conf->meta_dev_idx + 1);
+ retcode = adm_add_path(adm_ctx);
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
}
+ adm_ctx->result = retcode;
+ return 0;
+}
- if (drbd_get_capacity(nbc->md_bdev) < min_md_device_sectors) {
- retcode = ERR_MD_DISK_TOO_SMALL;
- drbd_warn(device, "refusing attach: md-device too small, "
- "at least %llu sectors needed for this meta-disk type\n",
- (unsigned long long) min_md_device_sectors);
- goto fail;
+static enum drbd_ret_code
+adm_del_path(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_resource *resource = adm_ctx->resource;
+ struct drbd_connection *connection = adm_ctx->connection;
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_path_parms pp = { };
+ struct drbd_path *path;
+ int nr_paths = 0;
+ int err;
+
+ /* parse and validate only */
+ if (!net_eq(adm_ctx->net, drbd_net_assigned_to_connection(connection))) {
+ drbd_adm_msg(adm_ctx, "%s", "connection assigned to a different network namespace");
+ return ERR_INVALID_REQUEST;
}
- /* Make sure the new disk is big enough
- * (we may currently be R_PRIMARY with no local disk...) */
- if (drbd_get_max_capacity(nbc) < get_capacity(device->vdisk)) {
- retcode = ERR_DISK_TOO_SMALL;
- goto fail;
+ err = drbd_adm_overlay_path_parms(adm_ctx, &pp);
+ if (err) {
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ return ERR_MANDATORY_TAG;
}
- nbc->known_size = drbd_get_capacity(nbc->backing_bdev);
+ list_for_each_entry(path, &transport->paths, list)
+ nr_paths++;
- if (nbc->known_size > max_possible_sectors) {
- drbd_warn(device, "==> truncating very big lower level device "
- "to currently maximum possible %llu sectors <==\n",
- (unsigned long long) max_possible_sectors);
- if (new_disk_conf->meta_dev_idx >= 0)
- drbd_warn(device, "==>> using internal or flexible "
- "meta data may help <<==\n");
- }
-
- drbd_suspend_io(device);
- /* also wait for the last barrier ack. */
- /* FIXME see also https://daiquiri.linbit/cgi-bin/bugzilla/show_bug.cgi?id=171
- * We need a way to either ignore barrier acks for barriers sent before a device
- * was attached, or a way to wait for all pending barrier acks to come in.
- * As barriers are counted per resource,
- * we'd need to suspend io on all devices of a resource.
- */
- wait_event(device->misc_wait, !atomic_read(&device->ap_pending_cnt) || drbd_suspended(device));
- /* and for any other previously queued work */
- drbd_flush_workqueue(&connection->sender_work);
+ if (nr_paths == 1 && connection->cstate[NOW] >= C_CONNECTING) {
+ drbd_adm_msg(adm_ctx, "%s", "Can not delete last path, use disconnect first!");
+ return ERR_INVALID_REQUEST;
+ }
- rv = _drbd_request_state(device, NS(disk, D_ATTACHING), CS_VERBOSE);
- retcode = (enum drbd_ret_code)rv;
- drbd_resume_io(device);
- if (rv < SS_SUCCESS)
- goto fail;
+ err = -ENOENT;
+ list_for_each_entry(path, &transport->paths, list) {
+ if (!path_my_addr_eq(path, &pp))
+ continue;
+ if (!path_peer_addr_eq(path, &pp))
+ continue;
- if (!get_ldev_if_state(device, D_ATTACHING))
- goto force_diskless;
+ /* Exclusive with transport op "prepare_connect()" */
+ mutex_lock(&resource->conf_update);
- if (!device->bitmap) {
- if (drbd_bm_init(device)) {
- retcode = ERR_NOMEM;
- goto force_diskless_dec;
+ if (!transport->class->ops.may_remove_path(path)) {
+ err = -EBUSY;
+ mutex_unlock(&resource->conf_update);
+ break;
}
- }
- if (device->state.pdsk != D_UP_TO_DATE && device->ed_uuid &&
- (device->state.role == R_PRIMARY || device->state.peer == R_PRIMARY) &&
- (device->ed_uuid & ~((u64)1)) != (nbc->md.uuid[UI_CURRENT] & ~((u64)1))) {
- drbd_err(device, "Can only attach to data with current UUID=%016llX\n",
- (unsigned long long)device->ed_uuid);
- retcode = ERR_DATA_NOT_CURRENT;
- goto force_diskless_dec;
- }
+ transport->class->ops.remove_path(path);
- /* Since we are diskless, fix the activity log first... */
- if (drbd_check_al_size(device, new_disk_conf)) {
- retcode = ERR_NOMEM;
- goto force_diskless_dec;
- }
+ set_bit(TR_UNREGISTERED, &path->flags);
+ /* Ensure flag visible before list manipulation. */
+ smp_wmb();
- /* Prevent shrinking of consistent devices ! */
- {
- unsigned long long nsz = drbd_new_dev_size(device, nbc, nbc->disk_conf->disk_size, 0);
- unsigned long long eff = nbc->md.la_size_sect;
- if (drbd_md_test_flag(nbc, MDF_CONSISTENT) && nsz < eff) {
- if (nsz == nbc->disk_conf->disk_size) {
- drbd_warn(device, "truncating a consistent device during attach (%llu < %llu)\n", nsz, eff);
- } else {
- drbd_warn(device, "refusing to truncate a consistent device (%llu < %llu)\n", nsz, eff);
- drbd_msg_sprintf_info(adm_ctx->reply_skb,
- "To-be-attached device has last effective > current size, and is consistent\n"
- "(%llu > %llu sectors). Refusing to attach.", eff, nsz);
- retcode = ERR_IMPLICIT_SHRINK;
- goto force_diskless_dec;
- }
- }
- }
+ /* Exclusive with reading state, in particular remember_state_change() */
+ write_lock_irq(&resource->state_rwlock);
+ list_del_rcu(&path->list);
+ write_unlock_irq(&resource->state_rwlock);
- lock_all_resources();
- retcode = drbd_resync_after_valid(device, new_disk_conf->resync_after);
- if (retcode != NO_ERROR) {
- unlock_all_resources();
- goto force_diskless_dec;
+ mutex_unlock(&resource->conf_update);
+
+ notify_path(connection, path, NOTIFY_DESTROY);
+ /* Transport modules might use RCU on the path list. */
+ call_rcu(&path->rcu, drbd_reclaim_path);
+
+ return NO_ERROR;
}
- /* Reset the "barriers don't work" bits here, then force meta data to
- * be written, to ensure we determine if barriers are supported. */
- if (new_disk_conf->md_flushes)
- clear_bit(MD_NO_FUA, &device->flags);
- else
- set_bit(MD_NO_FUA, &device->flags);
+ drbd_err(connection, "del_path() failed with %d\n", err);
+ drbd_adm_msg(adm_ctx, "%s",
+ err == -ENOENT ? "no such path" : "del_path on transport failed");
+ return ERR_INVALID_REQUEST;
+}
- /* Point of no return reached.
- * Devices and memory are no longer released by error cleanup below.
- * now device takes over responsibility, and the state engine should
- * clean it up somewhere. */
- D_ASSERT(device, device->ldev == NULL);
- device->ldev = nbc;
- device->resync = resync_lru;
- device->rs_plan_s = new_plan;
- nbc = NULL;
- resync_lru = NULL;
- new_disk_conf = NULL;
- new_plan = NULL;
+int drbd_adm_del_path(struct drbd_adm_ctx *adm_ctx)
+{
+ enum drbd_ret_code retcode = NO_ERROR;
- drbd_resync_after_changed(device);
- drbd_bump_write_ordering(device->resource, device->ldev, WO_BDEV_FLUSH);
- unlock_all_resources();
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ } else {
+ retcode = adm_del_path(adm_ctx);
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ }
+ adm_ctx->result = retcode;
+ return 0;
+}
- if (drbd_md_test_flag(device->ldev, MDF_CRASHED_PRIMARY))
- set_bit(CRASHED_PRIMARY, &device->flags);
- else
- clear_bit(CRASHED_PRIMARY, &device->flags);
+int drbd_open_ro_count(struct drbd_resource *resource)
+{
+ struct drbd_device *device;
+ int vnr, open_ro_cnt = 0;
- if (drbd_md_test_flag(device->ldev, MDF_PRIMARY_IND) &&
- !(device->state.role == R_PRIMARY && device->resource->susp_nod))
- set_bit(CRASHED_PRIMARY, &device->flags);
+ read_lock_irq(&resource->state_rwlock);
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (!device->writable)
+ open_ro_cnt += device->open_cnt;
+ }
+ read_unlock_irq(&resource->state_rwlock);
- device->send_cnt = 0;
- device->recv_cnt = 0;
- device->read_cnt = 0;
- device->writ_cnt = 0;
+ return open_ro_cnt;
+}
- drbd_reconsider_queue_parameters(device, device->ldev, NULL);
+/* How often to repeat a disconnect that did not conclude. */
+#define DISCONNECT_RETRIES 5
- /* If I am currently not R_PRIMARY,
- * but meta data primary indicator is set,
- * I just now recover from a hard crash,
- * and have been R_PRIMARY before that crash.
- *
- * Now, if I had no connection before that crash
- * (have been degraded R_PRIMARY), chances are that
- * I won't find my peer now either.
- *
- * In that case, and _only_ in that case,
- * we use the degr-wfc-timeout instead of the default,
- * so we can automatically recover from a crash of a
- * degraded but active "cluster" after a certain timeout.
- */
- clear_bit(USE_DEGR_WFC_T, &device->flags);
- if (device->state.role != R_PRIMARY &&
- drbd_md_test_flag(device->ldev, MDF_PRIMARY_IND) &&
- !drbd_md_test_flag(device->ldev, MDF_CONNECTED_IND))
- set_bit(USE_DEGR_WFC_T, &device->flags);
-
- dd = drbd_determine_dev_size(device, 0, NULL);
- if (dd <= DS_ERROR) {
- retcode = ERR_NOMEM_BITMAP;
- goto force_diskless_dec;
- } else if (dd == DS_GREW)
- set_bit(RESYNC_AFTER_NEG, &device->flags);
-
- if (drbd_md_test_flag(device->ldev, MDF_FULL_SYNC) ||
- (test_bit(CRASHED_PRIMARY, &device->flags) &&
- drbd_md_test_flag(device->ldev, MDF_AL_DISABLED))) {
- drbd_info(device, "Assuming that all blocks are out of sync "
- "(aka FullSync)\n");
- if (drbd_bitmap_io(device, &drbd_bmio_set_n_write,
- "set_n_write from attaching", BM_LOCKED_MASK,
- NULL)) {
- retcode = ERR_IO_MD_DISK;
- goto force_diskless_dec;
- }
- } else {
- if (drbd_bitmap_io(device, &drbd_bm_read,
- "read from attaching", BM_LOCKED_MASK,
- NULL)) {
- retcode = ERR_IO_MD_DISK;
- goto force_diskless_dec;
+static enum drbd_state_rv conn_try_disconnect(struct drbd_connection *connection, bool force,
+ const char *tag, struct drbd_adm_ctx *ctx)
+{
+ struct drbd_resource *resource = connection->resource;
+ enum drbd_conn_state cstate;
+ enum drbd_state_rv rv;
+ enum chg_state_flags flags = (force ? CS_HARD : 0) | CS_VERBOSE;
+ const char *err_str = NULL;
+ int retries = 0;
+ long t;
+
+repeat:
+ rv = change_cstate_tag(connection, C_DISCONNECTING, flags, tag, &err_str);
+ switch (rv) {
+ case SS_CW_FAILED_BY_PEER:
+ case SS_NEED_CONNECTION:
+ read_lock_irq(&resource->state_rwlock);
+ cstate = connection->cstate[NOW];
+ read_unlock_irq(&resource->state_rwlock);
+ /* A peer that refused while the connection is up refuses again;
+ * that is an answer, not something to repeat.
+ */
+ if (rv == SS_CW_FAILED_BY_PEER && cstate >= C_CONNECTED)
+ break;
+ /* Below C_CONNECTED the repeat needs no cluster-wide agreement;
+ * connected again, it can be disconnected properly now.
+ * Still: bound the number of retries.
+ */
+ if (++retries > DISCONNECT_RETRIES)
+ break;
+ goto repeat;
+ case SS_NO_UP_TO_DATE_DISK:
+ if (resource->role[NOW] == R_PRIMARY)
+ break;
+ /* Most probably udev opened it read-only. That might happen
+ if it was demoted very recently. Wait up to one second. */
+ t = wait_event_interruptible_timeout(resource->state_wait,
+ drbd_open_ro_count(resource) == 0,
+ HZ);
+ if (t <= 0)
+ break;
+ goto repeat;
+ case SS_ALREADY_STANDALONE:
+ rv = SS_SUCCESS;
+ break;
+ case SS_IS_DISKLESS:
+ case SS_LOWER_THAN_OUTDATED:
+ rv = change_cstate_tag(connection, C_DISCONNECTING, CS_HARD, tag, NULL);
+ break;
+ case SS_NO_QUORUM:
+ if (!(flags & CS_VERBOSE)) {
+ flags |= CS_VERBOSE;
+ goto repeat;
}
+ break;
+ default:
+ break;
+ /* no special handling necessary */
}
- if (_drbd_bm_total_weight(device) == drbd_bm_bits(device))
- drbd_suspend_al(device); /* IO is still suspended here... */
-
- spin_lock_irq(&device->resource->req_lock);
- os = drbd_read_state(device);
- ns = os;
- /* If MDF_CONSISTENT is not set go into inconsistent state,
- otherwise investigate MDF_WasUpToDate...
- If MDF_WAS_UP_TO_DATE is not set go into D_OUTDATED disk state,
- otherwise into D_CONSISTENT state.
- */
- if (drbd_md_test_flag(device->ldev, MDF_CONSISTENT)) {
- if (drbd_md_test_flag(device->ldev, MDF_WAS_UP_TO_DATE))
- ns.disk = D_CONSISTENT;
- else
- ns.disk = D_OUTDATED;
- } else {
- ns.disk = D_INCONSISTENT;
+ if (rv >= SS_SUCCESS)
+ wait_event_interruptible_timeout(resource->state_wait,
+ connection->cstate[NOW] == C_STANDALONE,
+ HZ);
+ if (err_str) {
+ drbd_adm_msg(ctx, "%s", err_str);
+ kfree(err_str);
}
- if (drbd_md_test_flag(device->ldev, MDF_PEER_OUT_DATED))
- ns.pdsk = D_OUTDATED;
-
- rcu_read_lock();
- if (ns.disk == D_CONSISTENT &&
- (ns.pdsk == D_OUTDATED || rcu_dereference(device->ldev->disk_conf)->fencing == FP_DONT_CARE))
- ns.disk = D_UP_TO_DATE;
+ return rv;
+}
- /* All tests on MDF_PRIMARY_IND, MDF_CONNECTED_IND,
- MDF_CONSISTENT and MDF_WAS_UP_TO_DATE must happen before
- this point, because drbd_request_state() modifies these
- flags. */
+/* this can only be called immediately after a successful
+ * peer_try_disconnect, within the same resource->adm_mutex */
+static void del_connection(struct drbd_connection *connection, const char *tag)
+{
+ struct drbd_resource *resource = connection->resource;
+ struct drbd_peer_device *peer_device;
+ enum drbd_state_rv rv2;
+ int vnr;
- if (rcu_dereference(device->ldev->disk_conf)->al_updates)
- device->ldev->md.flags &= ~MDF_AL_DISABLED;
- else
- device->ldev->md.flags |= MDF_AL_DISABLED;
+ if (test_bit(C_UNREGISTERED, &connection->flags))
+ return;
- rcu_read_unlock();
+ /* No one else can reconfigure the network while I am here.
+ * The state handling only uses drbd_thread_stop_nowait(),
+ * we want to really wait here until the receiver is no more.
+ */
+ drbd_thread_stop(&connection->receiver);
- /* In case we are C_CONNECTED postpone any decision on the new disk
- state after the negotiation phase. */
- if (device->state.conn == C_CONNECTED) {
- device->new_state_tmp.i = ns.i;
- ns.i = os.i;
- ns.disk = D_NEGOTIATING;
+ /* Race breaker. This additional state change request may be
+ * necessary, if this was a forced disconnect during a receiver
+ * restart. We may have "killed" the receiver thread just
+ * after drbd_receiver() returned. Typically, we should be
+ * C_STANDALONE already, now, and this becomes a no-op.
+ */
+ rv2 = change_cstate_tag(connection, C_STANDALONE, CS_VERBOSE | CS_HARD, tag, NULL);
+ if (rv2 < SS_SUCCESS)
+ drbd_err(connection,
+ "unexpected rv2=%d in del_connection()\n",
+ rv2);
+ /* Make sure the sender thread has actually stopped: state
+ * handling only does drbd_thread_stop_nowait().
+ */
+ drbd_thread_stop(&connection->sender);
- /* We expect to receive up-to-date UUIDs soon.
- To avoid a race in receive_state, free p_uuid while
- holding req_lock. I.e. atomic with the state change */
- kfree(device->p_uuid);
- device->p_uuid = NULL;
- }
+ mutex_lock(&resource->conf_update);
+ drbd_unregister_connection(connection);
+ mutex_unlock(&resource->conf_update);
- rv = _drbd_set_state(device, ns, CS_VERBOSE, NULL);
- spin_unlock_irq(&device->resource->req_lock);
+ /*
+ * Flush the resource work queue to make sure that no more
+ * events like state change notifications for this connection
+ * are queued: we want the "destroy" event to come last.
+ */
+ drbd_flush_workqueue(&resource->work);
- if (rv < SS_SUCCESS)
- goto force_diskless_dec;
+ mutex_lock(¬ification_mutex);
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
+ notify_peer_device_state(NULL, 0, peer_device, NULL,
+ NOTIFY_DESTROY | NOTIFY_CONTINUES);
+ notify_connection_state(NULL, 0, connection, NULL, NOTIFY_DESTROY);
+ mutex_unlock(¬ification_mutex);
+ call_rcu(&connection->rcu, drbd_reclaim_connection);
+}
- mod_timer(&device->request_timer, jiffies + HZ);
+static int adm_disconnect(struct drbd_adm_ctx *adm_ctx, bool destroy)
+{
+ struct drbd_disconnect_parms parms;
+ struct drbd_connection *connection;
+ struct net *existing_net;
+ enum drbd_state_rv rv;
+ enum drbd_ret_code retcode = NO_ERROR;
+ const char *tag = destroy ? "del-peer" : "disconnect";
- if (device->state.role == R_PRIMARY)
- device->ldev->md.uuid[UI_CURRENT] |= (u64)1;
- else
- device->ldev->md.uuid[UI_CURRENT] &= ~(u64)1;
+ memset(&parms, 0, sizeof(parms));
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_DISCONNECT_PARMS)) {
+ int err = drbd_adm_overlay_disconnect_parms(adm_ctx, &parms);
- drbd_md_mark_dirty(device);
- drbd_md_sync(device);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto fail;
+ }
+ }
- kobject_uevent(&disk_to_dev(device->vdisk)->kobj, KOBJ_CHANGE);
- put_ldev(device);
- conn_reconfig_done(connection);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
- adm_ctx->reply_dh->ret_code = retcode;
- return 0;
+ existing_net = drbd_net_assigned_to_connection(adm_ctx->connection);
+ if (existing_net && !net_eq(adm_ctx->net, existing_net)) {
+ drbd_adm_msg(adm_ctx, "%s", "connection assigned to a different network namespace");
+ retcode = ERR_INVALID_REQUEST;
+ goto fail;
+ }
- force_diskless_dec:
- put_ldev(device);
- force_diskless:
- drbd_force_state(device, NS(disk, D_DISKLESS));
- drbd_md_sync(device);
- fail:
- conn_reconfig_done(connection);
- if (nbc) {
- close_backing_dev(device, nbc->f_md_bdev,
- nbc->md_bdev != nbc->backing_bdev);
- close_backing_dev(device, nbc->backing_bdev_file, true);
- kfree(nbc);
+ connection = adm_ctx->connection;
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto fail;
}
- kfree(new_disk_conf);
- lc_destroy(resync_lru);
- kfree(new_plan);
+ rv = conn_try_disconnect(connection, parms.force_disconnect, tag, adm_ctx);
+ if (rv >= SS_SUCCESS && destroy)
+ del_connection(connection, tag);
+ if (rv < SS_SUCCESS)
+ retcode = (enum drbd_ret_code)rv;
+ else
+ retcode = NO_ERROR;
mutex_unlock(&adm_ctx->resource->adm_mutex);
- finish:
- adm_ctx->reply_dh->ret_code = retcode;
+ fail:
+ adm_ctx->result = retcode;
return 0;
}
-static int adm_detach(struct drbd_device *device, int force)
+int drbd_adm_disconnect(struct drbd_adm_ctx *adm_ctx)
{
- if (force) {
- set_bit(FORCE_DETACH, &device->flags);
- drbd_force_state(device, NS(disk, D_FAILED));
- return SS_SUCCESS;
- }
+ return adm_disconnect(adm_ctx, 0);
+}
- return drbd_request_detach_interruptible(device);
+int drbd_adm_del_peer(struct drbd_adm_ctx *adm_ctx)
+{
+ return adm_disconnect(adm_ctx, 1);
}
-/* Detaching the disk is a process in multiple stages. First we need to lock
- * out application IO, in-flight IO, IO stuck in drbd_al_begin_io.
- * Then we transition to D_DISKLESS, and wait for put_ldev() to return all
- * internal references as well.
- * Only then we have finally detached. */
-int drbd_nl_detach_doit(struct sk_buff *skb, struct genl_info *info)
+void resync_after_online_grow(struct drbd_peer_device *peer_device)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
- struct detach_parms parms = { };
- int err;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_device *device = peer_device->device;
+ bool sync_source = false;
+ s32 peer_id;
+
+ drbd_info(peer_device, "Resync of new storage after online grow\n");
+ if (device->resource->role[NOW] != connection->peer_role[NOW])
+ sync_source = (device->resource->role[NOW] == R_PRIMARY);
+ else if (connection->agreed_pro_version < 111)
+ sync_source = test_bit(RESOLVE_CONFLICTS,
+ &peer_device->connection->transport.flags);
+ else if (get_ldev(device)) {
+ /* multiple or no primaries, proto new enough, resolve by node-id */
+ s32 self_id = device->ldev->md.node_id;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ put_ldev(device);
+ peer_id = peer_device->node_id;
- if (info->attrs[DRBD_NLA_DETACH_PARMS]) {
- err = detach_parms_from_attrs(&parms, info);
- if (err) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
- goto out;
- }
+ sync_source = self_id < peer_id ? 1 : 0;
}
- mutex_lock(&adm_ctx->resource->adm_mutex);
- retcode = adm_detach(adm_ctx->device, parms.force_detach);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
-out:
- adm_ctx->reply_dh->ret_code = retcode;
- return 0;
+ if (!sync_source && connection->agreed_pro_version < 110) {
+ stable_change_repl_state(peer_device, L_WF_SYNC_UUID,
+ CS_VERBOSE | CS_SERIALIZE, "online-grow");
+ return;
+ }
+ drbd_start_resync(peer_device, sync_source ? L_SYNC_SOURCE : L_SYNC_TARGET, "online-grow");
}
-static bool conn_resync_running(struct drbd_connection *connection)
+sector_t drbd_local_max_size(struct drbd_device *device)
{
- struct drbd_peer_device *peer_device;
- bool rv = false;
- int vnr;
+ struct drbd_backing_dev *tmp_bdev;
+ sector_t s;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- if (device->state.conn == C_SYNC_SOURCE ||
- device->state.conn == C_SYNC_TARGET ||
- device->state.conn == C_PAUSED_SYNC_S ||
- device->state.conn == C_PAUSED_SYNC_T) {
- rv = true;
- break;
- }
- }
- rcu_read_unlock();
+ tmp_bdev = kmalloc_obj(struct drbd_backing_dev, GFP_ATOMIC);
+ if (!tmp_bdev)
+ return 0;
- return rv;
+ *tmp_bdev = *device->ldev;
+ drbd_md_set_sector_offsets(tmp_bdev);
+ s = drbd_get_max_capacity(device, tmp_bdev, false);
+ kfree(tmp_bdev);
+
+ return s;
}
-static bool conn_ov_running(struct drbd_connection *connection)
+/* Only from a cluster whose diskful members are all connected and quiet, and
+ * only on the lowest node id among them.
+ *
+ * Connected, because with dds_flags = 0 the transaction holds the size at the
+ * last agreed one while a diskful peer is absent anyway. A peer configured
+ * with "bitmap no" is a client by intent: it agrees to no size and applies
+ * whatever is agreed, so it is not waited for. A client the grow would leave
+ * without an UpToDate peer still stops it: it answers the prepare through the
+ * one node it is connected to, and the view it reports is short. Quiet,
+ * because growing rewrites the bitmap a running resync reads, which is why
+ * drbd_adm_resize() refuses as well; that also keeps two grows from
+ * overlapping. Growing needs no hurry: the next connection, or the end of the
+ * resync, arms this again.
+ *
+ * Lowest node id, so exactly one node starts the transaction: every node that
+ * noticed arms itself, and the others would find the size unchanged and abort.
+ */
+static bool may_start_auto_grow(struct drbd_device *device)
{
+ struct drbd_resource *resource = device->resource;
struct drbd_peer_device *peer_device;
- bool rv = false;
- int vnr;
+ bool may = true;
rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
- struct drbd_device *device = peer_device->device;
- if (device->state.conn == C_VERIFY_S ||
- device->state.conn == C_VERIFY_T) {
- rv = true;
+ for_each_peer_device_rcu(peer_device, device) {
+ /* a client by intent, see above */
+ if (!want_bitmap(peer_device))
+ continue;
+
+ if (peer_device->connection->cstate[NOW] != C_CONNECTED ||
+ peer_device->repl_state[NOW] != L_ESTABLISHED ||
+ peer_device->node_id < resource->res_opts.node_id) {
+ may = false;
break;
}
}
rcu_read_unlock();
- return rv;
+ return may;
}
-static enum drbd_ret_code
-_check_net_options(struct drbd_connection *connection, struct net_conf *old_net_conf, struct net_conf *new_net_conf)
+/* What a node advertises in P_SIZES is the minimum over the sizes it has
+ * cached for its peers, and what it receives feeds that minimum again, so the
+ * exchange can only ratchet down. With three or more nodes the peers that
+ * stayed connected keep each other at the size the cluster last agreed on, and
+ * a backing device that grew under DRBD never reaches the cluster. A size
+ * transaction is immune to it: every participant answers the prepare with its
+ * own drbd_local_max_size(), so no cache takes part in the decision.
+ *
+ * Runs in the worker, armed from after_state_change();
+ * change_cluster_wide_device_size() sleeps.
+ */
+void drbd_auto_grow(struct drbd_device *device)
{
- struct drbd_peer_device *peer_device;
- int i;
-
- if (old_net_conf && connection->cstate == C_WF_REPORT_PARAMS && connection->agreed_pro_version < 100) {
- if (new_net_conf->wire_protocol != old_net_conf->wire_protocol)
- return ERR_NEED_APV_100;
-
- if (new_net_conf->two_primaries != old_net_conf->two_primaries)
- return ERR_NEED_APV_100;
-
- if (strcmp(new_net_conf->integrity_alg, old_net_conf->integrity_alg))
- return ERR_NEED_APV_100;
- }
+ sector_t local_max_size, u_size;
+ enum determine_dev_size dd;
- if (!new_net_conf->two_primaries &&
- conn_highest_role(connection) == R_PRIMARY &&
- conn_highest_peer(connection) == R_PRIMARY)
- return ERR_NEED_ALLOW_TWO_PRI;
+ if (!get_ldev(device))
+ return;
- if (new_net_conf->two_primaries &&
- (new_net_conf->wire_protocol != DRBD_PROT_C))
- return ERR_NOT_PROTO_C;
+ if (!may_start_auto_grow(device))
+ goto out;
- idr_for_each_entry(&connection->peer_devices, peer_device, i) {
- struct drbd_device *device = peer_device->device;
- if (get_ldev(device)) {
- enum drbd_fencing_p fp = rcu_dereference(device->ldev->disk_conf)->fencing;
- put_ldev(device);
- if (new_net_conf->wire_protocol == DRBD_PROT_A && fp == FP_STONITH)
- return ERR_STONITH_AND_PROT_A;
- }
- if (device->state.role == R_PRIMARY && new_net_conf->discard_my_data)
- return ERR_DISCARD_IMPOSSIBLE;
- }
+ rcu_read_lock();
+ u_size = rcu_dereference(device->ldev->disk_conf)->disk_size;
+ rcu_read_unlock();
- if (new_net_conf->on_congestion != OC_BLOCK && new_net_conf->wire_protocol != DRBD_PROT_A)
- return ERR_CONG_NOT_PROTO_A;
+ local_max_size = drbd_local_max_size(device);
+ if (u_size)
+ local_max_size = min(local_max_size, u_size);
+ if (local_max_size <= get_capacity(device->vdisk))
+ goto out;
+ if (local_max_size == device->auto_grow_asked)
+ goto out;
- return NO_ERROR;
+ /* dds_flags = 0: a peer this node can not see keeps whatever it
+ * allowed when the cluster last agreed on a size, so this can never
+ * settle above what an administrative resize would. Growing past an
+ * absent peer stays the explicit --assume-peer-has-space promise,
+ * which comes with the duty to grow that peer's backend.
+ *
+ * automatic: commit only what every participant applies, and leave
+ * the log to what the transaction changes.
+ */
+ device->auto_grow_asked = local_max_size;
+ dd = change_cluster_wide_device_size(device, local_max_size, u_size, 0, true, NULL);
+ if (dd == DS_2PC_ERR)
+ /* A timeout or a competing state change is no answer at all,
+ * so forget what was asked and let the next arming edge ask
+ * again. Every other outcome answers what this cluster can
+ * do, and stands until a peer connects.
+ */
+ device->auto_grow_asked = 0;
+ if (dd == DS_2PC_NOT_SUPPORTED)
+ dynamic_drbd_dbg(device, "Not growing: a peer is too old for cluster-wide size changes\n");
+ drbd_md_sync_if_dirty(device);
+out:
+ put_ldev(device);
}
-static enum drbd_ret_code
-check_net_options(struct drbd_connection *connection, struct net_conf *new_net_conf)
+int drbd_adm_resize(struct drbd_adm_ctx *adm_ctx)
{
- enum drbd_ret_code rv;
+ struct drbd_disk_conf *old_disk_conf, *new_disk_conf = NULL;
+ struct drbd_resize_parms rs;
+ struct drbd_device *device;
+ enum determine_dev_size dd;
+ bool change_al_layout = false;
+ enum dds_flags ddsf;
+ sector_t u_size;
+ int err, retcode = NO_ERROR;
struct drbd_peer_device *peer_device;
- int i;
+ bool resolve_by_node_id = true;
+ bool has_up_to_date_primary;
+ bool traditional_resize = false;
+ sector_t local_max_size, wanted_size;
- rcu_read_lock();
- rv = _check_net_options(connection, rcu_dereference(connection->net_conf), new_net_conf);
- rcu_read_unlock();
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
+ device = adm_ctx->device;
+ if (!get_ldev(device)) {
+ retcode = ERR_NO_DISK;
+ goto fail;
+ }
- /* connection->peer_devices protected by genl_lock() here */
- idr_for_each_entry(&connection->peer_devices, peer_device, i) {
- struct drbd_device *device = peer_device->device;
- if (!device->bitmap) {
- if (drbd_bm_init(device))
- return ERR_NOMEM;
+ memset(&rs, 0, sizeof(struct drbd_resize_parms));
+ rs.al_stripes = device->ldev->md.al_stripes;
+ rs.al_stripe_size = device->ldev->md.al_stripe_size_4k * 4;
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_RESIZE_PARMS)) {
+ err = drbd_adm_overlay_resize_parms(adm_ctx, &rs);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto fail_ldev;
}
}
- return rv;
-}
-
-struct crypto {
- struct crypto_shash *verify_tfm;
- struct crypto_shash *csums_tfm;
- struct crypto_shash *cram_hmac_tfm;
- struct crypto_shash *integrity_tfm;
-};
+ device = adm_ctx->device;
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->repl_state[NOW] > L_ESTABLISHED) {
+ retcode = ERR_RESIZE_RESYNC;
+ goto fail_ldev;
+ }
+ }
-static int
-alloc_shash(struct crypto_shash **tfm, char *tfm_name, int err_alg)
-{
- if (!tfm_name[0])
- return NO_ERROR;
- *tfm = crypto_alloc_shash(tfm_name, 0, 0);
- if (IS_ERR(*tfm)) {
- *tfm = NULL;
- return err_alg;
+ /* Without --size, check the size the device already presents: a backing
+ * device that shrank below it can not hold the data plus the meta data.
+ */
+ local_max_size = drbd_local_max_size(device);
+ wanted_size = rs.resize_size ? (sector_t)rs.resize_size :
+ get_capacity(device->vdisk);
+ if (local_max_size < wanted_size) {
+ drbd_err(device, "%s %llu sectors, backend seems only able to support %llu\n",
+ rs.resize_size ? "requested" : "device presents",
+ (unsigned long long)wanted_size,
+ (unsigned long long)local_max_size);
+ retcode = ERR_DISK_TOO_SMALL;
+ goto fail_ldev;
}
- return NO_ERROR;
-}
+ /* Maybe I could serve as sync source myself? */
+ has_up_to_date_primary =
+ device->resource->role[NOW] == R_PRIMARY &&
+ device->disk_state[NOW] == D_UP_TO_DATE;
-static enum drbd_ret_code
-alloc_crypto(struct crypto *crypto, struct net_conf *new_net_conf)
-{
- char hmac_name[CRYPTO_MAX_ALG_NAME];
- enum drbd_ret_code rv;
+ if (!has_up_to_date_primary) {
+ for_each_peer_device(peer_device, device) {
+ /* ignore unless connection is fully established */
+ if (peer_device->repl_state[NOW] < L_ESTABLISHED)
+ continue;
+ if (peer_device->connection->agreed_pro_version < 111) {
+ resolve_by_node_id = false;
+ if (peer_device->connection->peer_role[NOW] == R_PRIMARY
+ && peer_device->disk_state[NOW] == D_UP_TO_DATE) {
+ has_up_to_date_primary = true;
+ break;
+ }
+ }
+ }
+ }
- rv = alloc_shash(&crypto->csums_tfm, new_net_conf->csums_alg,
- ERR_CSUMS_ALG);
- if (rv != NO_ERROR)
- return rv;
- rv = alloc_shash(&crypto->verify_tfm, new_net_conf->verify_alg,
- ERR_VERIFY_ALG);
- if (rv != NO_ERROR)
- return rv;
- rv = alloc_shash(&crypto->integrity_tfm, new_net_conf->integrity_alg,
- ERR_INTEGRITY_ALG);
- if (rv != NO_ERROR)
- return rv;
- if (new_net_conf->cram_hmac_alg[0] != 0) {
- snprintf(hmac_name, CRYPTO_MAX_ALG_NAME, "hmac(%s)",
- new_net_conf->cram_hmac_alg);
+ if (!has_up_to_date_primary && !resolve_by_node_id) {
+ retcode = ERR_NO_PRIMARY;
+ goto fail_ldev;
+ }
- rv = alloc_shash(&crypto->cram_hmac_tfm, hmac_name,
- ERR_AUTH_ALG);
+ for_each_peer_device(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
+
+ if (rs.no_resync &&
+ connection->cstate[NOW] == C_CONNECTED &&
+ connection->agreed_pro_version < 93) {
+ retcode = ERR_NEED_APV_93;
+ goto fail_ldev;
+ }
}
- return rv;
-}
+ rcu_read_lock();
+ u_size = rcu_dereference(device->ldev->disk_conf)->disk_size;
+ rcu_read_unlock();
+ if (u_size != (sector_t)rs.resize_size) {
+ new_disk_conf = kmalloc_obj(struct drbd_disk_conf);
+ if (!new_disk_conf) {
+ retcode = ERR_NOMEM;
+ goto fail_ldev;
+ }
+ }
-static void free_crypto(struct crypto *crypto)
-{
- crypto_free_shash(crypto->cram_hmac_tfm);
- crypto_free_shash(crypto->integrity_tfm);
- crypto_free_shash(crypto->csums_tfm);
- crypto_free_shash(crypto->verify_tfm);
-}
+ if (device->ldev->md.al_stripes != rs.al_stripes ||
+ device->ldev->md.al_stripe_size_4k != rs.al_stripe_size / 4) {
+ u32 al_size_k = rs.al_stripes * rs.al_stripe_size;
-int drbd_nl_chg_net_opts_doit(struct sk_buff *skb, struct genl_info *info)
-{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
- struct drbd_connection *connection;
- struct net_conf *old_net_conf, *new_net_conf = NULL;
- struct nlattr **ntb;
- int err;
- int ovr; /* online verify running */
- int rsr; /* re-sync running */
- struct crypto crypto = { };
+ if (al_size_k > (16 * 1024 * 1024)) {
+ retcode = ERR_MD_LAYOUT_TOO_BIG;
+ goto fail_ldev;
+ }
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto finish;
+ if (al_size_k < (32768 >> 10)) {
+ retcode = ERR_MD_LAYOUT_TOO_SMALL;
+ goto fail_ldev;
+ }
- connection = adm_ctx->connection;
- mutex_lock(&adm_ctx->resource->adm_mutex);
+ /* Removed this pre-condition while merging from 8.4 to 9.0
+ if (device->state.conn != C_CONNECTED && !rs.resize_force) {
+ retcode = ERR_MD_LAYOUT_CONNECTED;
+ goto fail_ldev;
+ } */
- new_net_conf = kzalloc_obj(struct net_conf);
- if (!new_net_conf) {
- retcode = ERR_NOMEM;
- goto out;
+ change_al_layout = true;
}
- conn_reconfig_start(connection);
-
- mutex_lock(&connection->data.mutex);
- mutex_lock(&connection->resource->conf_update);
- old_net_conf = connection->net_conf;
+ device->ldev->known_size = drbd_get_capacity(device->ldev->backing_bdev);
- if (!old_net_conf) {
- drbd_msg_put_info(adm_ctx->reply_skb, "net conf missing, try connect");
- retcode = ERR_INVALID_REQUEST;
- goto fail;
+ if (new_disk_conf) {
+ mutex_lock(&device->resource->conf_update);
+ old_disk_conf = device->ldev->disk_conf;
+ *new_disk_conf = *old_disk_conf;
+ new_disk_conf->disk_size = (sector_t)rs.resize_size;
+ rcu_assign_pointer(device->ldev->disk_conf, new_disk_conf);
+ mutex_unlock(&device->resource->conf_update);
+ kvfree_rcu_mightsleep(old_disk_conf);
+ new_disk_conf = NULL;
}
- *new_net_conf = *old_net_conf;
- if (should_set_defaults(info))
- set_net_conf_defaults(new_net_conf);
+ ddsf = (rs.resize_force ? DDSF_ASSUME_UNCONNECTED_PEER_HAS_SPACE : 0)
+ | (rs.no_resync ? DDSF_NO_RESYNC : 0);
+
+ dd = change_cluster_wide_device_size(device, local_max_size, rs.resize_size, ddsf,
+ false, change_al_layout ? &rs : NULL);
+ if (dd == DS_2PC_NOT_SUPPORTED) {
+ traditional_resize = true;
+ dd = drbd_determine_dev_size(device, 0, ddsf, change_al_layout ? &rs : NULL);
+ } else if (dd == DS_UNCHANGED &&
+ drbd_md_ss(device->ldev) != device->ldev->md.md_offset) {
+ /* The device keeps its size, but this node's backing device is
+ * not the size it was, which moves internal meta data. Only
+ * drbd_determine_dev_size() recomputes the layout, and the size
+ * change calls it only when it commits: apply the unchanged size
+ * here so the meta data is written where it now belongs.
+ */
+ dd = drbd_determine_dev_size(device, get_capacity(device->vdisk),
+ ddsf | DDSF_2PC, NULL);
+ }
- err = net_conf_from_attrs(new_net_conf, info);
- if (err && err != -ENOMSG) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
+ drbd_md_sync_if_dirty(device);
+ put_ldev(device);
+ if (dd == DS_ERROR) {
+ retcode = ERR_NOMEM_BITMAP;
+ goto fail;
+ } else if (dd == DS_ERROR_SPACE_MD) {
+ retcode = ERR_MD_LAYOUT_NO_FIT;
+ goto fail;
+ } else if (dd == DS_ERROR_SHRINK) {
+ retcode = ERR_IMPLICIT_SHRINK;
+ goto fail;
+ } else if (dd == DS_2PC_ERR) {
+ retcode = SS_INTERRUPTED;
goto fail;
}
- err = net_conf_ntb_from_attrs(&ntb, info);
- if (!err) {
- if (has_invariant(ntb, DRBD_A_NET_CONF_DISCARD_MY_DATA) ||
- has_invariant(ntb, DRBD_A_NET_CONF_TENTATIVE)) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb,
- "cannot change invariant setting");
- kfree(ntb);
- goto fail;
+ if (traditional_resize) {
+ for_each_peer_device(peer_device, device) {
+ if (peer_device->repl_state[NOW] == L_ESTABLISHED) {
+ if (dd == DS_GREW)
+ set_bit(RESIZE_PENDING, peer_device->flags);
+ drbd_send_uuids(peer_device, 0, 0);
+ drbd_send_sizes(peer_device, rs.resize_size, ddsf);
+ }
}
- kfree(ntb);
}
- retcode = check_net_options(connection, new_net_conf);
- if (retcode != NO_ERROR)
- goto fail;
+ fail:
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ out_no_adm_mutex:
+ adm_ctx->result = retcode;
+ return 0;
- /* re-sync running */
- rsr = conn_resync_running(connection);
- if (rsr && strcmp(new_net_conf->csums_alg, old_net_conf->csums_alg)) {
- retcode = ERR_CSUMS_RESYNC_RUNNING;
- goto fail;
+ fail_ldev:
+ put_ldev(device);
+ kfree(new_disk_conf);
+ goto fail;
+}
+
+int drbd_adm_resource_opts(struct drbd_adm_ctx *adm_ctx)
+{
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_res_opts res_opts;
+ int err;
+
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
}
+ res_opts = adm_ctx->resource->res_opts;
+ if (adm_ctx->set_defaults)
+ drbd_set_res_opts_defaults(&res_opts);
- /* online verify running */
- ovr = conn_ov_running(connection);
- if (ovr && strcmp(new_net_conf->verify_alg, old_net_conf->verify_alg)) {
- retcode = ERR_VERIFY_RUNNING;
+ err = drbd_adm_overlay_res_opts(adm_ctx, &res_opts);
+ if (err && err != -ENOMSG) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
goto fail;
}
- retcode = alloc_crypto(&crypto, new_net_conf);
- if (retcode != NO_ERROR)
+ if (adm_ctx->d->attr_present(adm_ctx, DRBD_ADM_F_RES_NODE_ID)) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg(adm_ctx, "%s", "cannot change invariant setting");
goto fail;
-
- rcu_assign_pointer(connection->net_conf, new_net_conf);
-
- if (!rsr) {
- crypto_free_shash(connection->csums_tfm);
- connection->csums_tfm = crypto.csums_tfm;
- crypto.csums_tfm = NULL;
}
- if (!ovr) {
- crypto_free_shash(connection->verify_tfm);
- connection->verify_tfm = crypto.verify_tfm;
- crypto.verify_tfm = NULL;
- }
-
- crypto_free_shash(connection->integrity_tfm);
- connection->integrity_tfm = crypto.integrity_tfm;
- if (connection->cstate >= C_WF_REPORT_PARAMS && connection->agreed_pro_version >= 100)
- /* Do this without trying to take connection->data.mutex again. */
- __drbd_send_protocol(connection, P_PROTOCOL_UPDATE);
- crypto_free_shash(connection->cram_hmac_tfm);
- connection->cram_hmac_tfm = crypto.cram_hmac_tfm;
+ if (res_opts.explicit_drbd8_compat) {
+ struct drbd_connection *connection;
+ int n_connections = 0;
- mutex_unlock(&connection->resource->conf_update);
- mutex_unlock(&connection->data.mutex);
- kvfree_rcu_mightsleep(old_net_conf);
+ for_each_connection(connection, adm_ctx->resource)
+ n_connections++;
- if (connection->cstate >= C_WF_REPORT_PARAMS) {
- struct drbd_peer_device *peer_device;
- int vnr;
+ if (n_connections > 1) {
+ drbd_adm_msg(adm_ctx, "drbd8 compat mode allows one peer at max");
+ goto fail;
+ }
+ }
- idr_for_each_entry(&connection->peer_devices, peer_device, vnr)
- drbd_send_sync_param(peer_device);
+ if (res_opts.node_id != -1) {
+#ifdef CONFIG_DRBD_COMPAT_84
+ if (!res_opts.drbd8_compat_mode && res_opts.explicit_drbd8_compat)
+ atomic_inc(&nr_drbd8_devices);
+ else if (res_opts.drbd8_compat_mode && !res_opts.explicit_drbd8_compat)
+ atomic_dec(&nr_drbd8_devices);
+#endif
+ res_opts.drbd8_compat_mode = res_opts.explicit_drbd8_compat;
}
- goto done;
+ err = set_resource_options(adm_ctx->resource, &res_opts, "resource-options");
+ if (err) {
+ retcode = ERR_INVALID_REQUEST;
+ if (err == -ENOMEM)
+ retcode = ERR_NOMEM;
+ }
- fail:
- mutex_unlock(&connection->resource->conf_update);
- mutex_unlock(&connection->data.mutex);
- free_crypto(&crypto);
- kfree(new_net_conf);
- done:
- conn_reconfig_done(connection);
- out:
+fail:
mutex_unlock(&adm_ctx->resource->adm_mutex);
- finish:
- adm_ctx->reply_dh->ret_code = retcode;
+out:
+ adm_ctx->result = retcode;
return 0;
}
-static void connection_to_info(struct connection_info *info,
- struct drbd_connection *connection)
+static enum drbd_state_rv invalidate_resync(struct drbd_peer_device *peer_device)
{
- info->conn_connection_state = connection->cstate;
- info->conn_role = conn_highest_peer(connection);
-}
+ struct drbd_resource *resource = peer_device->connection->resource;
+ enum drbd_state_rv rv;
-static void peer_device_to_info(struct peer_device_info *info,
- struct drbd_peer_device *peer_device)
-{
- struct drbd_device *device = peer_device->device;
+ drbd_flush_workqueue(&peer_device->connection->sender_work);
- info->peer_repl_state =
- max_t(enum drbd_conns, C_WF_REPORT_PARAMS, device->state.conn);
- info->peer_disk_state = device->state.pdsk;
- info->peer_resync_susp_user = device->state.user_isp;
- info->peer_resync_susp_peer = device->state.peer_isp;
- info->peer_resync_susp_dependency = device->state.aftr_isp;
+ rv = change_repl_state(peer_device, L_STARTING_SYNC_T, CS_SERIALIZE, "invalidate");
+
+ if (rv < SS_SUCCESS && rv != SS_NEED_CONNECTION)
+ rv = stable_change_repl_state(peer_device, L_STARTING_SYNC_T,
+ CS_VERBOSE | CS_SERIALIZE, "invalidate");
+
+ wait_event_interruptible(resource->state_wait,
+ peer_device->repl_state[NOW] != L_STARTING_SYNC_T);
+
+ return rv;
}
-int drbd_nl_connect_doit(struct sk_buff *skb, struct genl_info *info)
+static enum drbd_state_rv invalidate_no_resync(struct drbd_device *device)
{
- struct connection_info connection_info;
- enum drbd_notification_type flags;
- unsigned int peer_devices = 0;
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
+ struct drbd_resource *resource = device->resource;
struct drbd_peer_device *peer_device;
- struct net_conf *old_net_conf, *new_net_conf = NULL;
- struct crypto crypto = { };
- struct drbd_resource *resource;
struct drbd_connection *connection;
- enum drbd_ret_code retcode;
+ unsigned long irq_flags;
enum drbd_state_rv rv;
- int i;
- int err;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
- if (!(adm_ctx->my_addr && adm_ctx->peer_addr)) {
- drbd_msg_put_info(adm_ctx->reply_skb, "connection endpoint(s) missing");
- retcode = ERR_INVALID_REQUEST;
- goto out;
+ begin_state_change(resource, &irq_flags, CS_VERBOSE);
+ for_each_connection(connection, resource) {
+ peer_device = conn_peer_device(connection, device->vnr);
+ if (peer_device->repl_state[NOW] >= L_ESTABLISHED) {
+ abort_state_change(resource, &irq_flags);
+ return SS_UNKNOWN_ERROR;
+ }
}
+ __change_disk_state(device, D_INCONSISTENT);
+ rv = end_state_change(resource, &irq_flags, "invalidate");
- /* No need for _rcu here. All reconfiguration is
- * strictly serialized on genl_lock(). We are protected against
- * concurrent reconfiguration/addition/deletion */
- for_each_resource(resource, &drbd_resources) {
- for_each_connection(connection, resource) {
- if (nla_len(adm_ctx->my_addr) == connection->my_addr_len &&
- !memcmp(nla_data(adm_ctx->my_addr), &connection->my_addr,
- connection->my_addr_len)) {
- retcode = ERR_LOCAL_ADDR;
- goto out;
- }
-
- if (nla_len(adm_ctx->peer_addr) == connection->peer_addr_len &&
- !memcmp(nla_data(adm_ctx->peer_addr), &connection->peer_addr,
- connection->peer_addr_len)) {
- retcode = ERR_PEER_ADDR;
- goto out;
- }
- }
+ if (rv >= SS_SUCCESS) {
+ drbd_bitmap_io(device, &drbd_bmio_set_all_n_write,
+ "set_n_write from invalidate",
+ BM_LOCK_CLEAR | BM_LOCK_BULK,
+ NULL);
}
- mutex_lock(&adm_ctx->resource->adm_mutex);
- connection = first_connection(adm_ctx->resource);
- conn_reconfig_start(connection);
+ return rv;
+}
- if (connection->cstate > C_STANDALONE) {
- retcode = ERR_NET_CONFIGURED;
- goto fail;
- }
+int drbd_adm_invalidate(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_peer_device *sync_from_peer_device = NULL;
+ struct drbd_resource *resource;
+ struct drbd_device *device;
+ int retcode = 0; /* enum drbd_ret_code rsp. enum drbd_state_rv */
+ struct drbd_invalidate_parms inv = {
+ .sync_from_peer_node_id = -1,
+ .reset_bitmap = DRBD_INVALIDATE_RESET_BITMAP_DEF,
+ };
- /* allocation not in the IO path, drbdsetup / netlink process context */
- new_net_conf = kzalloc_obj(*new_net_conf);
- if (!new_net_conf) {
- retcode = ERR_NOMEM;
- goto fail;
+ device = adm_ctx->device;
+
+ if (!get_ldev(device)) {
+ retcode = ERR_NO_DISK;
+ goto out_no_ldev;
}
- set_net_conf_defaults(new_net_conf);
+ resource = device->resource;
- err = net_conf_from_attrs(new_net_conf, info);
- if (err && err != -ENOMSG) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
- goto fail;
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
}
- retcode = check_net_options(connection, new_net_conf);
- if (retcode != NO_ERROR)
- goto fail;
-
- retcode = alloc_crypto(&crypto, new_net_conf);
- if (retcode != NO_ERROR)
- goto fail;
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_INVALIDATE_PARMS)) {
+ int err;
- ((char *)new_net_conf->shared_secret)[SHARED_SECRET_MAX-1] = 0;
+ err = drbd_adm_overlay_invalidate_parms(adm_ctx, &inv);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto out_no_resume;
+ }
- drbd_flush_workqueue(&connection->sender_work);
+ if (inv.sync_from_peer_node_id != -1) {
+ struct drbd_connection *connection =
+ drbd_connection_by_node_id(resource, inv.sync_from_peer_node_id);
+ sync_from_peer_device = conn_peer_device(connection, device->vnr);
+ }
- mutex_lock(&adm_ctx->resource->conf_update);
- old_net_conf = connection->net_conf;
- if (old_net_conf) {
- retcode = ERR_NET_CONFIGURED;
- mutex_unlock(&adm_ctx->resource->conf_update);
- goto fail;
+ if (!inv.reset_bitmap && sync_from_peer_device &&
+ sync_from_peer_device->connection->agreed_pro_version < 120) {
+ retcode = ERR_APV_TOO_LOW;
+ drbd_adm_msg(adm_ctx, "%s",
+ "Need protocol level 120 to initiate bitmap based resync");
+ goto out_no_resume;
+ }
}
- rcu_assign_pointer(connection->net_conf, new_net_conf);
-
- conn_free_crypto(connection);
- connection->cram_hmac_tfm = crypto.cram_hmac_tfm;
- connection->integrity_tfm = crypto.integrity_tfm;
- connection->csums_tfm = crypto.csums_tfm;
- connection->verify_tfm = crypto.verify_tfm;
- connection->my_addr_len = nla_len(adm_ctx->my_addr);
- memcpy(&connection->my_addr, nla_data(adm_ctx->my_addr), connection->my_addr_len);
- connection->peer_addr_len = nla_len(adm_ctx->peer_addr);
- memcpy(&connection->peer_addr, nla_data(adm_ctx->peer_addr), connection->peer_addr_len);
+ /* If there is still bitmap IO pending, probably because of a previous
+ * resync just being finished, wait for it before requesting a new resync.
+ * Also wait for its after_state_ch(). */
+ drbd_suspend_io(device, READ_AND_WRITE);
+ wait_event(device->misc_wait, !atomic_read(&device->pending_bitmap_work.n));
- idr_for_each_entry(&connection->peer_devices, peer_device, i) {
- peer_devices++;
- }
+ if (sync_from_peer_device) {
+ if (inv.reset_bitmap) {
+ retcode = invalidate_resync(sync_from_peer_device);
+ } else {
+ retcode = change_repl_state(sync_from_peer_device, L_WF_BITMAP_T,
+ CS_VERBOSE | CS_CLUSTER_WIDE | CS_WAIT_COMPLETE |
+ CS_SERIALIZE, "invalidate");
+ }
+ } else {
+ int retry = 3;
- connection_to_info(&connection_info, connection);
- flags = (peer_devices--) ? NOTIFY_CONTINUES : 0;
- mutex_lock(¬ification_mutex);
- notify_connection_state(NULL, 0, connection, &connection_info, NOTIFY_CREATE | flags);
- idr_for_each_entry(&connection->peer_devices, peer_device, i) {
- struct peer_device_info peer_device_info;
+ do {
+ struct drbd_connection *connection;
- peer_device_to_info(&peer_device_info, peer_device);
- flags = (peer_devices--) ? NOTIFY_CONTINUES : 0;
- notify_peer_device_state(NULL, 0, peer_device, &peer_device_info, NOTIFY_CREATE | flags);
- }
- mutex_unlock(¬ification_mutex);
- mutex_unlock(&adm_ctx->resource->conf_update);
+ for_each_connection(connection, resource) {
+ struct drbd_peer_device *peer_device;
+
+ peer_device = conn_peer_device(connection, device->vnr);
+ if (!peer_device)
+ continue;
+
+ if (inv.reset_bitmap) {
+ retcode = invalidate_resync(peer_device);
+ } else {
+ if (connection->agreed_pro_version < 120) {
+ retcode = ERR_APV_TOO_LOW;
+ continue;
+ }
+ retcode = change_repl_state(peer_device, L_WF_BITMAP_T,
+ CS_VERBOSE | CS_CLUSTER_WIDE |
+ CS_WAIT_COMPLETE | CS_SERIALIZE,
+ "invalidate");
+ }
+ if (retcode >= SS_SUCCESS)
+ goto out;
+ }
+ if (retcode != SS_NEED_CONNECTION)
+ break;
- rcu_read_lock();
- idr_for_each_entry(&connection->peer_devices, peer_device, i) {
- struct drbd_device *device = peer_device->device;
- device->send_cnt = 0;
- device->recv_cnt = 0;
+ retcode = invalidate_no_resync(device);
+ } while (retcode == SS_UNKNOWN_ERROR && retry--);
}
- rcu_read_unlock();
-
- rv = conn_request_state(connection, NS(conn, C_UNCONNECTED), CS_VERBOSE);
-
- conn_reconfig_done(connection);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
- adm_ctx->reply_dh->ret_code = rv;
- return 0;
-
-fail:
- free_crypto(&crypto);
- kfree(new_net_conf);
- conn_reconfig_done(connection);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ drbd_resume_io(device);
+out_no_resume:
+ mutex_unlock(&resource->adm_mutex);
+out_no_adm_mutex:
+ put_ldev(device);
+out_no_ldev:
+ adm_ctx->result = retcode;
return 0;
}
-static enum drbd_state_rv conn_try_disconnect(struct drbd_connection *connection, bool force)
+static int drbd_bmio_set_susp_al(struct drbd_device *device, struct drbd_peer_device *peer_device)
{
- enum drbd_conns cstate;
- enum drbd_state_rv rv;
-
-repeat:
- rv = conn_request_state(connection, NS(conn, C_DISCONNECTING),
- force ? CS_HARD : 0);
+ int rv;
- switch (rv) {
- case SS_NOTHING_TO_DO:
- break;
- case SS_ALREADY_STANDALONE:
- return SS_SUCCESS;
- case SS_PRIMARY_NOP:
- /* Our state checking code wants to see the peer outdated. */
- rv = conn_request_state(connection, NS2(conn, C_DISCONNECTING, pdsk, D_OUTDATED), 0);
+ rv = drbd_bmio_set_n_write(device, peer_device);
+ drbd_try_suspend_al(device);
+ return rv;
+}
- if (rv == SS_OUTDATE_WO_CONN) /* lost connection before graceful disconnect succeeded */
- rv = conn_request_state(connection, NS(conn, C_DISCONNECTING), CS_VERBOSE);
+static int full_sync_from_peer(struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_resource *resource = device->resource;
+ int retcode; /* enum drbd_ret_code rsp. enum drbd_state_rv */
- break;
- case SS_CW_FAILED_BY_PEER:
- spin_lock_irq(&connection->resource->req_lock);
- cstate = connection->cstate;
- spin_unlock_irq(&connection->resource->req_lock);
- if (cstate <= C_WF_CONNECTION)
- goto repeat;
- /* The peer probably wants to see us outdated. */
- rv = conn_request_state(connection, NS2(conn, C_DISCONNECTING,
- disk, D_OUTDATED), 0);
- if (rv == SS_IS_DISKLESS || rv == SS_LOWER_THAN_OUTDATED) {
- rv = conn_request_state(connection, NS(conn, C_DISCONNECTING),
- CS_HARD);
+ retcode = stable_change_repl_state(peer_device, L_STARTING_SYNC_S, CS_SERIALIZE,
+ "invalidate-remote");
+ if (retcode < SS_SUCCESS) {
+ if (retcode == SS_NEED_CONNECTION && resource->role[NOW] == R_PRIMARY) {
+ /* The peer will get a resync upon connect anyways.
+ * Just make that into a full resync. */
+ retcode = change_peer_disk_state(peer_device, D_INCONSISTENT,
+ CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE,
+ "invalidate-remote");
+ if (retcode >= SS_SUCCESS) {
+ if (drbd_bitmap_io(device, &drbd_bmio_set_susp_al,
+ "set_n_write from invalidate_peer",
+ BM_LOCK_CLEAR | BM_LOCK_BULK |
+ BM_LOCK_SINGLE_SLOT,
+ peer_device))
+ retcode = ERR_IO_MD_DISK;
+ }
+ } else {
+ retcode = stable_change_repl_state(peer_device, L_STARTING_SYNC_S,
+ CS_VERBOSE | CS_SERIALIZE, "invalidate-remote");
}
- break;
- default:;
- /* no special handling necessary */
}
- if (rv >= SS_SUCCESS) {
- enum drbd_state_rv rv2;
- /* No one else can reconfigure the network while I am here.
- * The state handling only uses drbd_thread_stop_nowait(),
- * we want to really wait here until the receiver is no more.
- */
- drbd_thread_stop(&connection->receiver);
+ return retcode;
+}
+
+
+int drbd_adm_invalidate_peer(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_peer_device *peer_device;
+ struct drbd_resource *resource;
+ struct drbd_device *device;
+ int retcode; /* enum drbd_ret_code rsp. enum drbd_state_rv */
+ struct drbd_invalidate_peer_parms inv = {
+ .p_reset_bitmap = DRBD_INVALIDATE_RESET_BITMAP_DEF,
+ };
+
+ peer_device = adm_ctx->peer_device;
+ device = peer_device->device;
+ resource = device->resource;
- /* Race breaker. This additional state change request may be
- * necessary, if this was a forced disconnect during a receiver
- * restart. We may have "killed" the receiver thread just
- * after drbd_receiver() returned. Typically, we should be
- * C_STANDALONE already, now, and this becomes a no-op.
- */
- rv2 = conn_request_state(connection, NS(conn, C_STANDALONE),
- CS_VERBOSE | CS_HARD);
- if (rv2 < SS_SUCCESS)
- drbd_err(connection,
- "unexpected rv2=%d in conn_try_disconnect()\n",
- rv2);
- /* Unlike in DRBD 9, the state engine has generated
- * NOTIFY_DESTROY events before clearing connection->net_conf. */
+ if (!get_ldev(device)) {
+ retcode = ERR_NO_DISK;
+ goto out;
}
- return rv;
-}
-int drbd_nl_disconnect_doit(struct sk_buff *skb, struct genl_info *info)
-{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct disconnect_parms parms;
- struct drbd_connection *connection;
- enum drbd_state_rv rv;
- enum drbd_ret_code retcode;
- int err;
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto fail;
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_INVALIDATE_PEER_PARMS)) {
+ int err;
- connection = adm_ctx->connection;
- memset(&parms, 0, sizeof(parms));
- if (info->attrs[DRBD_NLA_DISCONNECT_PARMS]) {
- err = disconnect_parms_from_attrs(&parms, info);
+ err = drbd_adm_overlay_invalidate_peer_parms(adm_ctx, &inv);
if (err) {
retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
- goto fail;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto out_unlock;
+ }
+ if (!inv.p_reset_bitmap && peer_device->connection->agreed_pro_version < 120) {
+ retcode = ERR_APV_TOO_LOW;
+ drbd_adm_msg(adm_ctx, "%s",
+ "Need protocol level 120 to initiate bitmap based resync");
+ goto out_unlock;
}
}
- mutex_lock(&adm_ctx->resource->adm_mutex);
- rv = conn_try_disconnect(connection, parms.force_disconnect);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
- if (rv < SS_SUCCESS) {
- adm_ctx->reply_dh->ret_code = rv;
- return 0;
+ drbd_suspend_io(device, READ_AND_WRITE);
+ wait_event(device->misc_wait, !atomic_read(&device->pending_bitmap_work.n));
+ drbd_flush_workqueue(&peer_device->connection->sender_work);
+
+ if (inv.p_reset_bitmap) {
+ retcode = full_sync_from_peer(peer_device);
+ } else {
+ retcode = change_repl_state(peer_device, L_WF_BITMAP_S,
+ CS_VERBOSE | CS_CLUSTER_WIDE | CS_WAIT_COMPLETE | CS_SERIALIZE,
+ "invalidate-remote");
}
- retcode = NO_ERROR;
- fail:
- adm_ctx->reply_dh->ret_code = retcode;
+ drbd_resume_io(device);
+
+out_unlock:
+ mutex_unlock(&resource->adm_mutex);
+out_no_adm_mutex:
+ put_ldev(device);
+out:
+ adm_ctx->result = retcode;
return 0;
}
-void resync_after_online_grow(struct drbd_device *device)
+int drbd_adm_pause_sync(struct drbd_adm_ctx *adm_ctx)
{
- int iass; /* I am sync source */
+ struct drbd_peer_device *peer_device;
+ enum drbd_ret_code retcode = NO_ERROR;
- drbd_info(device, "Resync of new storage after online grow\n");
- if (device->state.role != device->state.peer)
- iass = (device->state.role == R_PRIMARY);
- else
- iass = test_bit(RESOLVE_CONFLICTS, &first_peer_device(device)->connection->flags);
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
+ }
- if (iass)
- drbd_start_resync(device, C_SYNC_SOURCE);
- else
- _drbd_request_state(device, NS(conn, C_WF_SYNC_UUID), CS_VERBOSE + CS_SERIALIZE);
+ peer_device = adm_ctx->peer_device;
+ if (change_resync_susp_user(peer_device, true,
+ CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE) == SS_NOTHING_TO_DO)
+ retcode = ERR_PAUSE_IS_SET;
+
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ out:
+ adm_ctx->result = retcode;
+ return 0;
}
-int drbd_nl_resize_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_resume_sync(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct disk_conf *old_disk_conf, *new_disk_conf = NULL;
- struct resize_parms rs;
- struct drbd_device *device;
- enum drbd_ret_code retcode;
- enum determine_dev_size dd;
- bool change_al_layout = false;
- enum dds_flags ddsf;
- sector_t u_size;
- int err;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto finish;
+ struct drbd_peer_device *peer_device;
+ enum drbd_ret_code retcode = NO_ERROR;
- mutex_lock(&adm_ctx->resource->adm_mutex);
- device = adm_ctx->device;
- if (!get_ldev(device)) {
- retcode = ERR_NO_DISK;
- goto fail;
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
}
- memset(&rs, 0, sizeof(struct resize_parms));
- rs.al_stripes = device->ldev->md.al_stripes;
- rs.al_stripe_size = device->ldev->md.al_stripe_size_4k * 4;
- if (info->attrs[DRBD_NLA_RESIZE_PARMS]) {
- err = resize_parms_from_attrs(&rs, info);
- if (err) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
- goto fail_ldev;
+ peer_device = adm_ctx->peer_device;
+ if (change_resync_susp_user(peer_device, false,
+ CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE) == SS_NOTHING_TO_DO) {
+
+ if (peer_device->repl_state[NOW] == L_PAUSED_SYNC_S ||
+ peer_device->repl_state[NOW] == L_PAUSED_SYNC_T) {
+ if (peer_device->resync_susp_dependency[NOW])
+ retcode = ERR_PIC_AFTER_DEP;
+ else if (peer_device->resync_susp_peer[NOW])
+ retcode = ERR_PIC_PEER_DEP;
+ else
+ retcode = ERR_PAUSE_IS_CLEAR;
+ } else {
+ retcode = ERR_PAUSE_IS_CLEAR;
}
}
- if (device->state.conn > C_CONNECTED) {
- retcode = ERR_RESIZE_RESYNC;
- goto fail_ldev;
- }
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ out:
+ adm_ctx->result = retcode;
+ return 0;
+}
- if (device->state.role == R_SECONDARY &&
- device->state.peer == R_SECONDARY) {
- retcode = ERR_NO_PRIMARY;
- goto fail_ldev;
- }
+static bool io_drained(struct drbd_device *device)
+{
+ struct drbd_peer_device *peer_device;
+ bool drained = true;
- if (rs.no_resync && first_peer_device(device)->connection->agreed_pro_version < 93) {
- retcode = ERR_NEED_APV_93;
- goto fail_ldev;
- }
+ if (atomic_read(&device->local_cnt))
+ return false;
rcu_read_lock();
- u_size = rcu_dereference(device->ldev->disk_conf)->disk_size;
- rcu_read_unlock();
- if (u_size != (sector_t)rs.resize_size) {
- new_disk_conf = kmalloc_obj(struct disk_conf);
- if (!new_disk_conf) {
- retcode = ERR_NOMEM;
- goto fail_ldev;
+ for_each_peer_device_rcu(peer_device, device) {
+ if (atomic_read(&peer_device->ap_pending_cnt)) {
+ drained = false;
+ break;
}
}
+ rcu_read_unlock();
- if (device->ldev->md.al_stripes != rs.al_stripes ||
- device->ldev->md.al_stripe_size_4k != rs.al_stripe_size / 4) {
- u32 al_size_k = rs.al_stripes * rs.al_stripe_size;
+ return drained;
+}
- if (al_size_k > (16 * 1024 * 1024)) {
- retcode = ERR_MD_LAYOUT_TOO_BIG;
- goto fail_ldev;
- }
+int drbd_adm_suspend_io(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_resource *resource;
+ struct drbd_device *device;
+ int retcode = NO_ERROR, vnr, err = 0;
+ struct drbd_suspend_io_parms params = {
+ .bdev_freeze = true,
+ .flush_to_peers = true,
+ };
- if (al_size_k < MD_32kB_SECT/2) {
- retcode = ERR_MD_LAYOUT_TOO_SMALL;
- goto fail_ldev;
- }
+ resource = adm_ctx->device->resource;
- if (device->state.conn != C_CONNECTED && !rs.resize_force) {
- retcode = ERR_MD_LAYOUT_CONNECTED;
- goto fail_ldev;
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_SUSPEND_IO_PARMS)) {
+ err = drbd_adm_overlay_suspend_io_parms(adm_ctx, ¶ms);
+ if (err) {
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ return err;
}
+ }
- change_al_layout = true;
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
}
- if (device->ldev->known_size != drbd_get_capacity(device->ldev->backing_bdev))
- device->ldev->known_size = drbd_get_capacity(device->ldev->backing_bdev);
+ idr_for_each_entry(&resource->devices, device, vnr)
+ clear_bit(CBT_BITMAPS_FLUSHED, &device->flags);
- if (new_disk_conf) {
- mutex_lock(&device->resource->conf_update);
- old_disk_conf = device->ldev->disk_conf;
- *new_disk_conf = *old_disk_conf;
- new_disk_conf->disk_size = (sector_t)rs.resize_size;
- rcu_assign_pointer(device->ldev->disk_conf, new_disk_conf);
- mutex_unlock(&device->resource->conf_update);
- kvfree_rcu_mightsleep(old_disk_conf);
- new_disk_conf = NULL;
- }
+ idr_for_each_entry(&resource->devices, device, vnr)
+ if (params.bdev_freeze && !test_bit(BDEV_FROZEN, &device->flags)) {
+ err = bdev_freeze(device->vdisk->part0);
+ if (err)
+ goto out_thaw;
- ddsf = (rs.resize_force ? DDSF_FORCED : 0) | (rs.no_resync ? DDSF_NO_RESYNC : 0);
- dd = drbd_determine_dev_size(device, ddsf, change_al_layout ? &rs : NULL);
- drbd_md_sync(device);
- put_ldev(device);
- if (dd == DS_ERROR) {
- retcode = ERR_NOMEM_BITMAP;
- goto fail;
- } else if (dd == DS_ERROR_SPACE_MD) {
- retcode = ERR_MD_LAYOUT_NO_FIT;
- goto fail;
- } else if (dd == DS_ERROR_SHRINK) {
- retcode = ERR_IMPLICIT_SHRINK;
- goto fail;
- }
+ set_bit(BDEV_FROZEN, &device->flags);
+ }
- if (device->state.conn == C_CONNECTED) {
- if (dd == DS_GREW)
- set_bit(RESIZE_PENDING, &device->flags);
+ retcode = stable_state_change(resource, change_io_susp_user(resource, true,
+ CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE));
+ mutex_unlock(&resource->adm_mutex);
+ if (retcode < SS_SUCCESS)
+ goto out;
- drbd_send_uuids(first_peer_device(device));
- drbd_send_sizes(first_peer_device(device), 1, ddsf);
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ if (wait_event_interruptible(device->misc_wait, io_drained(device))) {
+ retcode = ERR_INTR;
+ goto out;
+ }
}
- fail:
- mutex_unlock(&adm_ctx->resource->adm_mutex);
- finish:
- adm_ctx->reply_dh->ret_code = retcode;
+ if (!params.flush_to_peers)
+ goto out;
+
+ if (wait_for_barrier_acks_interruptible(resource))
+ retcode = ERR_INTR;
+out:
+ adm_ctx->result = retcode;
return 0;
+out_thaw:
+ idr_for_each_entry(&resource->devices, device, vnr)
+ if (test_and_clear_bit(BDEV_FROZEN, &device->flags))
+ bdev_thaw(device->vdisk->part0);
- fail_ldev:
- put_ldev(device);
- kfree(new_disk_conf);
- goto fail;
+ mutex_unlock(&resource->adm_mutex);
+ adm_ctx->result = retcode;
+ return err;
}
-int drbd_nl_resource_opts_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_resume_io(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
- struct res_opts res_opts;
- int err;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto fail;
-
- res_opts = adm_ctx->resource->res_opts;
- if (should_set_defaults(info))
- set_res_opts_defaults(&res_opts);
+ struct drbd_connection *connection;
+ struct drbd_resource *resource;
+ struct drbd_device *device, *d;
+ unsigned long irq_flags;
+ int vnr, retcode; /* enum drbd_ret_code rsp. enum drbd_state_rv */
- err = res_opts_from_attrs(&res_opts, info);
- if (err && err != -ENOMSG) {
- retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
- goto fail;
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out;
}
+ device = adm_ctx->device;
+ resource = device->resource;
+ /* gen-rotate reason: DEGRADE (deferred bump flushed on admin resume-io).
+ * The state change below clears every suspension reason of the resource,
+ * so it finalizes writes held on any of its volumes; the obligation is
+ * per volume, so ask each of them.
+ */
+ idr_for_each_entry(&resource->devices, d, vnr)
+ drbd_gen_obligation_mint_before_resume(d, 0);
+ idr_for_each_entry(&resource->devices, d, vnr)
+ drbd_cbt_clear_flushed(d);
+ drbd_suspend_io(device, READ_AND_WRITE);
+ begin_state_change(resource, &irq_flags, CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE);
+ __change_io_susp_user(resource, false);
+ __change_io_susp_no_data(resource, false);
+ for_each_connection(connection, resource)
+ __change_io_susp_fencing(connection, false);
+
+ __change_io_susp_quorum(resource, false);
+ retcode = end_state_change(resource, &irq_flags, "resume-io");
+ drbd_resume_io(device);
- mutex_lock(&adm_ctx->resource->adm_mutex);
- err = set_resource_options(adm_ctx->resource, &res_opts);
- if (err) {
- retcode = ERR_INVALID_REQUEST;
- if (err == -ENOMEM)
- retcode = ERR_NOMEM;
- }
- mutex_unlock(&adm_ctx->resource->adm_mutex);
+ idr_for_each_entry(&resource->devices, device, vnr)
+ if (test_and_clear_bit(BDEV_FROZEN, &device->flags))
+ bdev_thaw(device->vdisk->part0);
-fail:
- adm_ctx->reply_dh->ret_code = retcode;
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ out:
+ adm_ctx->result = retcode;
return 0;
}
-int drbd_nl_invalidate_doit(struct sk_buff *skb, struct genl_info *info)
+/* Like io_drained(), but for a caller that holds an ldev reference itself */
+static bool writes_drained(struct drbd_device *device)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct drbd_device *device;
- int retcode; /* enum drbd_ret_code rsp. enum drbd_state_rv */
+ struct drbd_peer_device *peer_device;
+ bool drained = true;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ if (atomic_read(&device->ap_bio_cnt[WRITE]))
+ return false;
- device = adm_ctx->device;
- if (!get_ldev(device)) {
- retcode = ERR_NO_DISK;
- goto out;
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ if (atomic_read(&peer_device->ap_pending_cnt)) {
+ drained = false;
+ break;
+ }
}
+ rcu_read_unlock();
- mutex_lock(&adm_ctx->resource->adm_mutex);
-
- /* If there is still bitmap IO pending, probably because of a previous
- * resync just being finished, wait for it before requesting a new resync.
- * Also wait for it's after_state_ch(). */
- drbd_suspend_io(device);
- wait_event(device->misc_wait, !test_bit(BITMAP_IO, &device->flags));
- drbd_flush_workqueue(&first_peer_device(device)->connection->sender_work);
-
- /* If we happen to be C_STANDALONE R_SECONDARY, just change to
- * D_INCONSISTENT, and set all bits in the bitmap. Otherwise,
- * try to start a resync handshake as sync target for full sync.
- */
- if (device->state.conn == C_STANDALONE && device->state.role == R_SECONDARY) {
- retcode = drbd_request_state(device, NS(disk, D_INCONSISTENT));
- if (retcode >= SS_SUCCESS) {
- if (drbd_bitmap_io(device, &drbd_bmio_set_n_write,
- "set_n_write from invalidate", BM_LOCKED_MASK, NULL))
- retcode = ERR_IO_MD_DISK;
- }
- } else
- retcode = drbd_request_state(device, NS(conn, C_STARTING_SYNC_T));
- drbd_resume_io(device);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
- put_ldev(device);
-out:
- adm_ctx->reply_dh->ret_code = retcode;
- return 0;
+ return drained;
}
-static int drbd_adm_simple_request_state(struct sk_buff *skb, struct genl_info *info,
- union drbd_state mask, union drbd_state val)
+static bool peer_writes_executing(struct drbd_device *device)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
+ struct drbd_peer_device *peer_device;
+ bool executing = false;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
- mutex_lock(&adm_ctx->resource->adm_mutex);
- retcode = drbd_request_state(adm_ctx->device, mask, val);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
-out:
- adm_ctx->reply_dh->ret_code = retcode;
- return 0;
+ if (atomic_read(&connection->active_ee_cnt) ||
+ atomic_read(&connection->backing_ee_cnt)) {
+ executing = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return executing;
}
-static int drbd_bmio_set_susp_al(struct drbd_device *device,
- struct drbd_peer_device *peer_device) __must_hold(local)
+static bool resync_running(struct drbd_device *device)
{
- int rv;
+ struct drbd_peer_device *peer_device;
+ bool running = false;
- rv = drbd_bmio_set_n_write(device, peer_device);
- drbd_suspend_al(device);
- return rv;
+ rcu_read_lock();
+ for_each_peer_device_rcu(peer_device, device) {
+ enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
+
+ if ((repl_state >= L_STARTING_SYNC_S && repl_state <= L_WF_SYNC_UUID) ||
+ repl_is_sync(repl_state)) {
+ running = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return running;
}
-int drbd_nl_inval_peer_doit(struct sk_buff *skb, struct genl_info *info)
+/* Write the bitmap and the superblock out, so that a snapshot taken while I/O
+ * stays suspended contains every write up to now in the tracking slot. Issued
+ * on every node with a disk after suspend-io succeeded on all nodes. After
+ * --flush-to-peers=no a write may still reach a Secondary; cbt-start sees it.
+ */
+int drbd_adm_cbt_flush(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- int retcode; /* drbd_ret_code, drbd_state_rv */
- struct drbd_device *device;
+ struct drbd_device *device = adm_ctx->device;
+ struct drbd_resource *resource = device->resource;
+ enum drbd_ret_code retcode = NO_ERROR;
+ int err;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
goto out;
-
- device = adm_ctx->device;
+ }
if (!get_ldev(device)) {
retcode = ERR_NO_DISK;
- goto out;
+ goto out_unlock;
+ }
+
+ retcode = ERR_INVALID_REQUEST;
+ if (test_bit(LEGACY_84_MD, &device->flags) || !device->bitmap) {
+ drbd_adm_msg(adm_ctx, "changed block tracking needs v09 meta data and a bitmap");
+ goto out_put;
+ }
+ if (!resource->susp_user[NOW] || !writes_drained(device)) {
+ drbd_adm_msg(adm_ctx, "I/O is not suspended and drained by suspend-io");
+ goto out_put;
+ }
+ if (peer_writes_executing(device)) {
+ drbd_adm_msg(adm_ctx, "peer writes are still executing on this node");
+ goto out_put;
+ }
+ if (resync_running(device)) {
+ drbd_adm_msg(adm_ctx, "a resync is running");
+ goto out_put;
+ }
+ if (device->disk_state[NOW] < D_OUTDATED) {
+ drbd_adm_msg(adm_ctx, "the local disk is %s",
+ drbd_disk_str(device->disk_state[NOW]));
+ goto out_put;
}
- mutex_lock(&adm_ctx->resource->adm_mutex);
+ set_bit(CBT_BITMAPS_FLUSHED, &device->flags);
+ device->cbt_flush_writ_cnt = READ_ONCE(device->writ_cnt);
- /* If there is still bitmap IO pending, probably because of a previous
- * resync just being finished, wait for it before requesting a new resync.
- * Also wait for it's after_state_ch(). */
- drbd_suspend_io(device);
- wait_event(device->misc_wait, !test_bit(BITMAP_IO, &device->flags));
- drbd_flush_workqueue(&first_peer_device(device)->connection->sender_work);
-
- /* If we happen to be C_STANDALONE R_PRIMARY, just set all bits
- * in the bitmap. Otherwise, try to start a resync handshake
- * as sync source for full sync.
- */
- if (device->state.conn == C_STANDALONE && device->state.role == R_PRIMARY) {
- /* The peer will get a resync upon connect anyways. Just make that
- into a full resync. */
- retcode = drbd_request_state(device, NS(pdsk, D_INCONSISTENT));
- if (retcode >= SS_SUCCESS) {
- if (drbd_bitmap_io(device, &drbd_bmio_set_susp_al,
- "set_n_write from invalidate_peer",
- BM_LOCKED_SET_ALLOWED, NULL))
- retcode = ERR_IO_MD_DISK;
- }
- } else
- retcode = drbd_request_state(device, NS(conn, C_STARTING_SYNC_S));
- drbd_resume_io(device);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
+ err = drbd_bitmap_io(device, &drbd_bm_write, "cbt-flush", BM_LOCK_BULK, NULL);
+ if (!err) {
+ /* Tell a reader of a snapshot of the meta data that its bitmap
+ * is complete although the activity log is not clean.
+ */
+ if (drbd_cbt_bitmap_index(device->ldev) != -1)
+ WRITE_ONCE(device->ldev->md.cbt.flags,
+ device->ldev->md.cbt.flags | CBT_BOOKMARK_FLUSHED);
+ drbd_md_mark_dirty(device);
+ err = drbd_md_sync_if_dirty(device);
+ }
+ if (err)
+ clear_bit(CBT_BITMAPS_FLUSHED, &device->flags);
+ retcode = err ? ERR_IO_MD_DISK : NO_ERROR;
+out_put:
put_ldev(device);
+out_unlock:
+ mutex_unlock(&resource->adm_mutex);
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
-int drbd_nl_pause_sync_doit(struct sk_buff *skb, struct genl_info *info)
+/* On a Secondary the position in the write stream is the Primary's dagtag
+ * as last received from it, not the dagtag of this node.
+ */
+static u64 cbt_bookmark_dagtag(struct drbd_resource *resource)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
+ struct drbd_connection *connection;
+ u64 dagtag = resource->dagtag_sector;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ if (resource->role[NOW] == R_PRIMARY)
+ return dagtag;
- mutex_lock(&adm_ctx->resource->adm_mutex);
- if (drbd_request_state(adm_ctx->device, NS(user_isp, 1)) == SS_NOTHING_TO_DO)
- retcode = ERR_PAUSE_IS_SET;
- mutex_unlock(&adm_ctx->resource->adm_mutex);
-out:
- adm_ctx->reply_dh->ret_code = retcode;
- return 0;
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (connection->peer_role[NOW] == R_PRIMARY) {
+ dagtag = atomic64_read(&connection->last_dagtag_sector);
+ break;
+ }
+ }
+ rcu_read_unlock();
+
+ return dagtag;
}
-int drbd_nl_resume_sync_doit(struct sk_buff *skb, struct genl_info *info)
+/* A write is in ap_bio_cnt or in the peer request counts until it completes,
+ * and in writ_cnt after. Read them in that order, so that one of them has it.
+ */
+static bool written_since_cbt_flush(struct drbd_device *device)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- union drbd_dev_state s;
- enum drbd_ret_code retcode;
+ if (atomic_read(&device->ap_bio_cnt[WRITE]) || peer_writes_executing(device))
+ return true;
+ /* pairs with the ordered decrement of those counts after writ_cnt += */
+ smp_rmb();
+ return READ_ONCE(device->writ_cnt) != device->cbt_flush_writ_cnt;
+}
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
+/* Start a new changed block tracking interval at the bookmark of the snapshot
+ * that was just taken. The snapshot holds the previous interval: the slot was
+ * written out by cbt-flush, and nothing was written since.
+ */
+int drbd_adm_cbt_start(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_device *device = adm_ctx->device;
+ struct drbd_cbt_start_parms parms = { };
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_md *md;
+ int err, cbt_index;
+
+ err = drbd_adm_overlay_cbt_start_parms(adm_ctx, &parms);
+ if (err) {
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ return err;
+ }
+
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
goto out;
+ }
+ if (!get_ldev(device)) {
+ retcode = ERR_NO_DISK;
+ goto out_unlock;
+ }
+ md = &device->ldev->md;
- mutex_lock(&adm_ctx->resource->adm_mutex);
- if (drbd_request_state(adm_ctx->device, NS(user_isp, 0)) == SS_NOTHING_TO_DO) {
- s = adm_ctx->device->state;
- if (s.conn == C_PAUSED_SYNC_S || s.conn == C_PAUSED_SYNC_T) {
- retcode = s.aftr_isp ? ERR_PIC_AFTER_DEP :
- s.peer_isp ? ERR_PIC_PEER_DEP : ERR_PAUSE_IS_CLEAR;
- } else {
- retcode = ERR_PAUSE_IS_CLEAR;
- }
+ retcode = ERR_INVALID_REQUEST;
+ if (test_bit(LEGACY_84_MD, &device->flags) || !device->bitmap) {
+ drbd_adm_msg(adm_ctx, "changed block tracking needs v09 meta data and a bitmap");
+ goto out_put;
}
+ if (device->disk_state[NOW] < D_OUTDATED) {
+ drbd_adm_msg(adm_ctx, "the local disk is %s",
+ drbd_disk_str(device->disk_state[NOW]));
+ goto out_put;
+ }
+ if (!device->resource->susp_user[NOW] || !test_bit(CBT_BITMAPS_FLUSHED, &device->flags)) {
+ drbd_adm_msg(adm_ctx, "no cbt-flush since suspend-io");
+ goto out_put;
+ }
+ /* A resync that started since cbt-flush merged the peer's slot into
+ * ours and wrote it out as its crash safety; clearing the slot now
+ * would drop that.
+ */
+ if (resync_running(device)) {
+ drbd_adm_msg(adm_ctx, "a resync is running");
+ goto out_put;
+ }
+ if (written_since_cbt_flush(device)) {
+ drbd_adm_msg(adm_ctx, "the backing device was written to after the bitmap flush");
+ goto out_put;
+ }
+ cbt_index = drbd_cbt_bitmap_index(device->ldev);
+ if (cbt_index == -1)
+ cbt_index = drbd_unallocated_index(device->ldev);
+ if (cbt_index == -1) {
+ drbd_adm_msg(adm_ctx, "no free bitmap slot");
+ goto out_put;
+ }
+
+ drbd_bm_lock(device, "cbt-start", BM_LOCK_BULK);
+ /* Claim the slot before clearing it. A handshake looks for a vacant
+ * slot without adm_mutex and copies it as the peer's day-0 bitmap;
+ * an empty slot would declare the peer in sync.
+ */
+ WRITE_ONCE(md->cbt.bitmap_index, cbt_index);
+ /* publishes bitmap_index; pairs with the acquire in drbd_cbt_bitmap_index() */
+ smp_store_release(&md->cbt.flags, CBT_BOOKMARK_IN_USE);
+ _drbd_bm_clear_many_bits(device, cbt_index, 0, -1UL);
+ md->cbt.id = parms.cbt_id;
+ md->cbt.bitmap_uuid = drbd_current_uuid(device);
+ md->cbt.bitmap_dagtag = cbt_bookmark_dagtag(device->resource);
+ md->cbt.since = ktime_divns(ktime_get_real(), NSEC_PER_SEC);
+ WRITE_ONCE(md->cbt.flags, CBT_BOOKMARK_IN_USE | CBT_BOOKMARK_FLUSHED);
+ /* A write between the check above and the clear lost its bits. */
+ smp_mb();
+ if (written_since_cbt_flush(device)) {
+ WRITE_ONCE(md->cbt.flags, md->cbt.flags | CBT_BOOKMARK_INVALID);
+ drbd_warn(device, "changed block tracking: the backing device was written to during cbt-start\n");
+ }
+ err = drbd_bm_write(device, NULL);
+ drbd_bm_unlock(device);
+ drbd_md_mark_dirty(device);
+ if (!err)
+ err = drbd_md_sync_if_dirty(device);
+ retcode = err ? ERR_IO_MD_DISK : NO_ERROR;
+ if (!err)
+ drbd_info(device, "changed block tracking started at bookmark %016llX (bitmap slot %d)\n",
+ (unsigned long long)parms.cbt_id, cbt_index);
+out_put:
+ put_ldev(device);
+out_unlock:
mutex_unlock(&adm_ctx->resource->adm_mutex);
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
-int drbd_nl_suspend_io_doit(struct sk_buff *skb, struct genl_info *info)
-{
- return drbd_adm_simple_request_state(skb, info, NS(susp, 1));
-}
-
-int drbd_nl_resume_io_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_cbt_stop(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct drbd_device *device;
- int retcode; /* enum drbd_ret_code rsp. enum drbd_state_rv */
+ struct drbd_device *device = adm_ctx->device;
+ enum drbd_ret_code retcode = NO_ERROR;
+ int err, cbt_index, from_index;
+ struct drbd_md *md;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
goto out;
-
- mutex_lock(&adm_ctx->resource->adm_mutex);
- device = adm_ctx->device;
- if (test_bit(NEW_CUR_UUID, &device->flags)) {
- if (get_ldev_if_state(device, D_ATTACHING)) {
- drbd_uuid_new_current(device);
- put_ldev(device);
- } else {
- /* This is effectively a multi-stage "forced down".
- * The NEW_CUR_UUID bit is supposedly only set, if we
- * lost the replication connection, and are configured
- * to freeze IO and wait for some fence-peer handler.
- * So we still don't have a replication connection.
- * And now we don't have a local disk either. After
- * resume, we will fail all pending and new IO, because
- * we don't have any data anymore. Which means we will
- * eventually be able to terminate all users of this
- * device, and then take it down. By bumping the
- * "effective" data uuid, we make sure that you really
- * need to tear down before you reconfigure, we will
- * the refuse to re-connect or re-attach (because no
- * matching real data uuid exists).
- */
- u64 val;
- val = get_random_u64();
- drbd_set_ed_uuid(device, val);
- drbd_warn(device, "Resumed without access to data; please tear down before attempting to re-configure.\n");
- }
- clear_bit(NEW_CUR_UUID, &device->flags);
}
- drbd_suspend_io(device);
- retcode = drbd_request_state(device, NS3(susp, 0, susp_nod, 0, susp_fen, 0));
- if (retcode == SS_SUCCESS) {
- if (device->state.conn < C_CONNECTED)
- tl_clear(first_peer_device(device)->connection);
- if (device->state.disk == D_DISKLESS || device->state.disk == D_FAILED)
- tl_restart(first_peer_device(device)->connection, FAIL_FROZEN_DISK_IO);
+ if (!get_ldev(device)) {
+ retcode = ERR_NO_DISK;
+ goto out_unlock;
+ }
+ md = &device->ldev->md;
+ cbt_index = drbd_cbt_bitmap_index(device->ldev);
+ if (cbt_index == -1)
+ goto out_put;
+
+ drbd_suspend_io(device, WRITE_ONLY);
+ if (device->bitmap) {
+ drbd_bm_lock(device, "cbt-stop", BM_LOCK_BULK);
+ /* An unallocated slot tracks writes since day 0, as
+ * clear_peer_slot() leaves a freed peer slot. Look for one
+ * while ours is still taken.
+ */
+ from_index = drbd_unallocated_index(device->ldev);
+ WRITE_ONCE(md->cbt.flags, 0);
+ if (from_index != -1)
+ drbd_bm_copy_slot(device, from_index, cbt_index);
+ else
+ _drbd_bm_set_many_bits(device, cbt_index, 0, -1UL);
+ err = drbd_bm_write(device, NULL);
+ drbd_bm_unlock(device);
+ } else {
+ /* attached without a bitmap: nothing was tracked, no slot to free */
+ WRITE_ONCE(md->cbt.flags, 0);
+ err = 0;
}
+ /* bitmap_index stays: a writer that saw IN_USE before the store above
+ * reads it afterwards. The slot is a day-0 slot now, a bit more in it
+ * does no harm.
+ */
+ md->cbt.id = 0;
+ md->cbt.bitmap_uuid = 0;
+ md->cbt.bitmap_dagtag = 0;
+ md->cbt.since = 0;
drbd_resume_io(device);
+ drbd_md_mark_dirty(device);
+ if (!err)
+ err = drbd_md_sync_if_dirty(device);
+ retcode = err ? ERR_IO_MD_DISK : NO_ERROR;
+ if (!err)
+ drbd_info(device, "changed block tracking stopped, freed bitmap slot %d\n",
+ cbt_index);
+out_put:
+ put_ldev(device);
+out_unlock:
mutex_unlock(&adm_ctx->resource->adm_mutex);
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
-int drbd_nl_outdate_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_cbt_status(struct drbd_adm_ctx *adm_ctx)
{
- return drbd_adm_simple_request_state(skb, info, NS(disk, D_OUTDATED));
-}
+ struct drbd_device *device = adm_ctx->device;
+ struct drbd_cbt_info info = { };
+ struct drbd_cbt_md *cbt;
-static int nla_put_drbd_cfg_context(struct sk_buff *skb,
- struct drbd_resource *resource,
- struct drbd_connection *connection,
- struct drbd_device *device)
-{
- struct nlattr *nla;
- nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_CONTEXT);
- if (!nla)
- goto nla_put_failure;
- if (device &&
- nla_put_u32(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME, device->vnr))
- goto nla_put_failure;
- if (nla_put_string(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME, resource->name))
- goto nla_put_failure;
- if (connection) {
- if (connection->my_addr_len &&
- nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR,
- connection->my_addr_len,
- &connection->my_addr))
- goto nla_put_failure;
- if (connection->peer_addr_len &&
- nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR,
- connection->peer_addr_len,
- &connection->peer_addr))
- goto nla_put_failure;
- }
- nla_nest_end(skb, nla);
- return 0;
+ if (!adm_ctx->d->put_cbt_info) {
+ adm_ctx->result = ERR_INVALID_REQUEST;
+ return 0;
+ }
+ if (!get_ldev(device)) {
+ adm_ctx->result = ERR_NO_DISK;
+ return 0;
+ }
+ cbt = &device->ldev->md.cbt;
+ if (cbt->flags & CBT_BOOKMARK_IN_USE) {
+ info.cbt_active = 1;
+ info.cbt_valid = !(cbt->flags & CBT_BOOKMARK_INVALID);
+ info.cbt_bookmark_id = cbt->id;
+ info.cbt_since = cbt->since;
+ info.cbt_bitmap_uuid = cbt->bitmap_uuid;
+ info.cbt_bitmap_dagtag = cbt->bitmap_dagtag;
+ }
+ put_ldev(device);
-nla_put_failure:
- if (nla)
- nla_nest_cancel(skb, nla);
- return -EMSGSIZE;
+ adm_ctx->result = adm_ctx->d->put_cbt_info(adm_ctx, &info);
+ return 0;
}
-/*
- * net_conf_to_skb() serializes the shared secret verbatim. Any path that can
- * answer a request from an unprivileged process must pass exclude_sensitive,
- * so the secret is blanked in a private copy before it reaches the skb.
- */
-static int net_conf_to_skb_sanitized(struct sk_buff *skb, struct net_conf *nc,
- bool exclude_sensitive)
+int drbd_adm_outdate(struct drbd_adm_ctx *adm_ctx)
{
- struct net_conf nc_clean;
-
- if (!exclude_sensitive)
- return net_conf_to_skb(skb, nc);
-
- nc_clean = *nc;
- memset(nc_clean.shared_secret, 0, sizeof(nc_clean.shared_secret));
- nc_clean.shared_secret_len = 0;
+ enum drbd_ret_code retcode = NO_ERROR;
- return net_conf_to_skb(skb, &nc_clean);
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ } else {
+ retcode = stable_state_change(adm_ctx->device->resource,
+ change_disk_state(adm_ctx->device, D_OUTDATED,
+ CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE, "outdate", NULL));
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ }
+ adm_ctx->result = retcode;
+ return 0;
}
-/*
- * The generic netlink dump callbacks are called outside the genl_lock(), so
- * they cannot use the simple attribute parsing code which uses global
- * attribute tables.
- */
-static struct nlattr *find_cfg_context_attr(const struct nlmsghdr *nlh, int attr)
-{
- const unsigned int hdrlen = GENL_HDRLEN + sizeof(struct drbd_genlmsghdr);
- struct nlattr *nla;
+static void resource_to_info(struct drbd_resource_info *, struct drbd_resource *);
- nla = nla_find(nlmsg_attrdata(nlh, hdrlen), nlmsg_attrlen(nlh, hdrlen),
- DRBD_NLA_CFG_CONTEXT);
- if (!nla)
- return NULL;
- return nla_find_nested(nla, attr);
+void resource_to_statistics(struct drbd_resource_statistics *s, struct drbd_resource *resource)
+{
+ s->res_stat_write_ordering = resource->write_ordering;
}
-static void resource_to_info(struct resource_info *, struct drbd_resource *);
-
-int drbd_nl_get_resources_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+int drbd_dump_resources(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect)
{
- struct drbd_genlmsghdr *dh;
struct drbd_resource *resource;
- struct resource_info resource_info;
- struct resource_statistics resource_statistics;
+ struct drbd_resource_info resource_info;
+ struct drbd_resource_statistics resource_statistics;
int err;
rcu_read_lock();
@@ -3368,31 +6933,13 @@ int drbd_nl_get_resources_dumpit(struct sk_buff *skb, struct netlink_callback *c
goto out;
put_result:
- dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq, &drbd_nl_family,
- NLM_F_MULTI, DRBD_ADM_GET_RESOURCES);
- err = -ENOMEM;
- if (!dh)
- goto out;
- dh->minor = -1U;
- dh->ret_code = NO_ERROR;
- err = nla_put_drbd_cfg_context(skb, resource, NULL, NULL);
- if (err)
- goto out;
- err = res_opts_to_skb(skb, &resource->res_opts);
- if (err)
- goto out;
resource_to_info(&resource_info, resource);
- err = resource_info_to_skb(skb, &resource_info);
- if (err)
- goto out;
- resource_statistics.res_stat_write_ordering = resource->write_ordering;
- err = resource_statistics_to_skb(skb, &resource_statistics);
+ resource_to_statistics(&resource_statistics, resource);
+ err = dialect->emit_resource(skb, cb, resource, &resource_info,
+ &resource_statistics);
if (err)
goto out;
cb->args[0] = (long)resource;
- genlmsg_end(skb, dh);
- err = 0;
out:
rcu_read_unlock();
@@ -3401,8 +6948,8 @@ int drbd_nl_get_resources_dumpit(struct sk_buff *skb, struct netlink_callback *c
return skb->len;
}
-static void device_to_statistics(struct device_statistics *s,
- struct drbd_device *device)
+void device_to_statistics(struct drbd_device_statistics *s,
+ struct drbd_device *device)
{
memset(s, 0, sizeof(*s));
s->dev_upper_blocked = !may_inc_ap_bio(device);
@@ -3412,16 +6959,18 @@ static void device_to_statistics(struct device_statistics *s,
int n;
spin_lock_irq(&md->uuid_lock);
- s->dev_current_uuid = md->uuid[UI_CURRENT];
- BUILD_BUG_ON(sizeof(s->history_uuids) < UI_HISTORY_END - UI_HISTORY_START + 1);
- for (n = 0; n < UI_HISTORY_END - UI_HISTORY_START + 1; n++)
- history_uuids[n] = md->uuid[UI_HISTORY_START + n];
- for (; n < HISTORY_UUIDS; n++)
- history_uuids[n] = 0;
- s->history_uuids_len = HISTORY_UUIDS;
+ s->dev_current_uuid = md->current_uuid;
+ BUILD_BUG_ON(sizeof(s->history_uuids) != sizeof(md->history_uuids));
+ for (n = 0; n < ARRAY_SIZE(md->history_uuids); n++)
+ history_uuids[n] = md->history_uuids[n];
+ s->history_uuids_len = sizeof(s->history_uuids);
spin_unlock_irq(&md->uuid_lock);
s->dev_disk_flags = md->flags;
+ /* originally, this used the bdi congestion framework,
+ * but that was removed in linux 5.18.
+ * so just never report the lower device as congested. */
+ s->dev_lower_blocked = false;
put_ldev(device);
}
s->dev_size = get_capacity(device->vdisk);
@@ -3429,10 +6978,11 @@ static void device_to_statistics(struct device_statistics *s,
s->dev_write = device->writ_cnt;
s->dev_al_writes = device->al_writ_cnt;
s->dev_bm_writes = device->bm_writ_cnt;
- s->dev_upper_pending = atomic_read(&device->ap_bio_cnt);
+ s->dev_upper_pending = atomic_read(&device->ap_bio_cnt[READ]) +
+ atomic_read(&device->ap_bio_cnt[WRITE]);
s->dev_lower_pending = atomic_read(&device->local_cnt);
s->dev_al_suspended = test_bit(AL_SUSPENDED, &device->flags);
- s->dev_exposed_data_uuid = device->ed_uuid;
+ s->dev_exposed_data_uuid = device->exposed_data_uuid;
}
static int put_resource_in_arg0(struct netlink_callback *cb, int holder_nr)
@@ -3446,37 +6996,24 @@ static int put_resource_in_arg0(struct netlink_callback *cb, int holder_nr)
return 0;
}
-int drbd_adm_dump_devices_done(struct netlink_callback *cb) {
+int drbd_dump_devices_done(struct netlink_callback *cb)
+{
return put_resource_in_arg0(cb, 7);
}
-static void device_to_info(struct device_info *, struct drbd_device *);
-
-int drbd_nl_get_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+int drbd_dump_devices(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect)
{
- struct nlattr *resource_filter;
struct drbd_resource *resource;
struct drbd_device *device;
- int minor, err, retcode;
- struct drbd_genlmsghdr *dh;
- struct device_info device_info;
- struct device_statistics device_statistics;
+ struct drbd_disk_conf *disk_conf;
+ bool have_ldev;
+ int minor, err;
+ struct drbd_device_info device_info;
+ struct drbd_device_statistics device_statistics;
struct idr *idr_to_search;
resource = (struct drbd_resource *)cb->args[0];
- if (!cb->args[0] && !cb->args[1]) {
- resource_filter = find_cfg_context_attr(cb->nlh,
- DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
- if (resource_filter) {
- retcode = ERR_RES_NOT_KNOWN;
- resource = drbd_find_resource(nla_data(resource_filter));
- if (!resource) {
- rcu_read_lock();
- goto put_result;
- }
- cb->args[0] = (long)resource;
- }
- }
rcu_read_lock();
minor = cb->args[1];
@@ -3487,48 +7024,23 @@ int drbd_nl_get_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
goto out;
}
idr_for_each_entry_continue(idr_to_search, device, minor) {
- retcode = NO_ERROR;
goto put_result; /* only one iteration */
}
err = 0;
goto out; /* no more devices */
put_result:
- dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq, &drbd_nl_family,
- NLM_F_MULTI, DRBD_ADM_GET_DEVICES);
- err = -ENOMEM;
- if (!dh)
+ device_to_info(&device_info, device);
+ device_to_statistics(&device_statistics, device);
+ have_ldev = get_ldev_if_state(device, D_FAILED);
+ disk_conf = have_ldev ? rcu_dereference(device->ldev->disk_conf) : NULL;
+ err = dialect->emit_device(skb, cb, NO_ERROR, device, disk_conf,
+ &device_info, &device_statistics);
+ if (have_ldev)
+ put_ldev(device);
+ if (err)
goto out;
- dh->ret_code = retcode;
- dh->minor = -1U;
- if (retcode == NO_ERROR) {
- dh->minor = device->minor;
- err = nla_put_drbd_cfg_context(skb, device->resource, NULL, device);
- if (err)
- goto out;
- if (get_ldev(device)) {
- struct disk_conf *disk_conf =
- rcu_dereference(device->ldev->disk_conf);
-
- err = disk_conf_to_skb(skb, disk_conf);
- put_ldev(device);
- if (err)
- goto out;
- }
- device_to_info(&device_info, device);
- err = device_info_to_skb(skb, &device_info);
- if (err)
- goto out;
-
- device_to_statistics(&device_statistics, device);
- err = device_statistics_to_skb(skb, &device_statistics);
- if (err)
- goto out;
- cb->args[1] = minor + 1;
- }
- genlmsg_end(skb, dh);
- err = 0;
+ cb->args[1] = minor + 1;
out:
rcu_read_unlock();
@@ -3537,49 +7049,55 @@ int drbd_nl_get_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
return skb->len;
}
-int drbd_adm_dump_connections_done(struct netlink_callback *cb)
+int drbd_dump_connections_done(struct netlink_callback *cb)
{
return put_resource_in_arg0(cb, 6);
}
-enum { SINGLE_RESOURCE, ITERATE_RESOURCES };
+void connection_to_statistics(struct drbd_connection_statistics *s,
+ struct drbd_connection *connection)
+{
+ s->conn_congested = test_bit(NET_CONGESTED, &connection->transport.flags);
+ s->ap_in_flight = atomic_read(&connection->ap_in_flight);
+ s->rs_in_flight = atomic_read(&connection->rs_in_flight);
+}
-int drbd_nl_get_connections_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+int drbd_dump_connections(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect)
{
- struct nlattr *resource_filter;
struct drbd_resource *resource = NULL, *next_resource;
- struct drbd_connection *connection;
+ struct drbd_connection *connection = NULL;
+ struct drbd_net_conf *net_conf, nc_copy;
int err = 0, retcode;
- struct drbd_genlmsghdr *dh;
- struct connection_info connection_info;
- struct connection_statistics connection_statistics;
+ struct drbd_connection_info connection_info;
+ struct drbd_connection_statistics connection_statistics;
rcu_read_lock();
resource = (struct drbd_resource *)cb->args[0];
- if (!cb->args[0]) {
- resource_filter = find_cfg_context_attr(cb->nlh,
- DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
- if (resource_filter) {
- retcode = ERR_RES_NOT_KNOWN;
- resource = drbd_find_resource(nla_data(resource_filter));
- if (!resource)
- goto put_result;
- cb->args[0] = (long)resource;
- cb->args[1] = SINGLE_RESOURCE;
- }
- }
if (!resource) {
if (list_empty(&drbd_resources))
goto out;
resource = list_first_entry(&drbd_resources, struct drbd_resource, resources);
kref_get(&resource->kref);
cb->args[0] = (long)resource;
- cb->args[1] = ITERATE_RESOURCES;
+ cb->args[1] = DRBD_DUMP_ITERATE_RESOURCES;
}
- next_resource:
+next_resource:
rcu_read_unlock();
- mutex_lock(&resource->conf_update);
+ if (mutex_lock_interruptible(&resource->conf_update)) {
+ kref_put(&resource->kref, drbd_destroy_resource);
+ resource = NULL;
+ /* Drop the stale pointer so that neither a subsequent dump
+ * round nor the done() callback uses the reference we just
+ * dropped.
+ */
+ cb->args[0] = 0;
+ cb->args[2] = 0;
+ retcode = ERR_INTR;
+ rcu_read_lock();
+ goto put_result;
+ }
rcu_read_lock();
if (cb->args[2]) {
for_each_connection_rcu(connection, resource)
@@ -3592,14 +7110,12 @@ int drbd_nl_get_connections_dumpit(struct sk_buff *skb, struct netlink_callback
found_connection:
list_for_each_entry_continue_rcu(connection, &resource->connections, connections) {
- if (!has_net_conf(connection))
- continue;
retcode = NO_ERROR;
goto put_result; /* only one iteration */
}
no_more_connections:
- if (cb->args[1] == ITERATE_RESOURCES) {
+ if (cb->args[1] == DRBD_DUMP_ITERATE_RESOURCES) {
for_each_resource_rcu(next_resource, &drbd_resources) {
if (next_resource == resource)
goto found_resource;
@@ -3621,39 +7137,28 @@ int drbd_nl_get_connections_dumpit(struct sk_buff *skb, struct netlink_callback
goto out; /* no more resources */
put_result:
- dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq, &drbd_nl_family,
- NLM_F_MULTI, DRBD_ADM_GET_CONNECTIONS);
- err = -ENOMEM;
- if (!dh)
- goto out;
- dh->ret_code = retcode;
- dh->minor = -1U;
+ net_conf = NULL;
if (retcode == NO_ERROR) {
- struct net_conf *net_conf;
-
- err = nla_put_drbd_cfg_context(skb, resource, connection, NULL);
- if (err)
- goto out;
- net_conf = rcu_dereference(connection->net_conf);
- if (net_conf) {
- err = net_conf_to_skb_sanitized(skb, net_conf,
- !capable(CAP_SYS_ADMIN));
- if (err)
- goto out;
+ struct drbd_net_conf *nc = rcu_dereference(connection->transport.net_conf);
+
+ if (nc) {
+ nc_copy = *nc;
+ if (!capable(CAP_SYS_ADMIN)) {
+ memset(nc_copy.shared_secret, 0,
+ sizeof(nc_copy.shared_secret));
+ nc_copy.shared_secret_len = 0;
+ }
+ net_conf = &nc_copy;
}
connection_to_info(&connection_info, connection);
- err = connection_info_to_skb(skb, &connection_info);
- if (err)
- goto out;
- connection_statistics.conn_congested = test_bit(NET_CONGESTED, &connection->flags);
- err = connection_statistics_to_skb(skb, &connection_statistics);
- if (err)
- goto out;
- cb->args[2] = (long)connection;
+ connection_to_statistics(&connection_statistics, connection);
}
- genlmsg_end(skb, dh);
- err = 0;
+ err = dialect->emit_connection(skb, cb, retcode, resource, connection, net_conf,
+ &connection_info, &connection_statistics);
+ if (err)
+ goto out;
+ if (retcode == NO_ERROR)
+ cb->args[2] = (long)connection;
out:
rcu_read_unlock();
@@ -3664,74 +7169,113 @@ int drbd_nl_get_connections_dumpit(struct sk_buff *skb, struct netlink_callback
return skb->len;
}
-enum mdf_peer_flag {
- MDF_PEER_CONNECTED = 1 << 0,
- MDF_PEER_OUTDATED = 1 << 1,
- MDF_PEER_FENCING = 1 << 2,
- MDF_PEER_FULL_SYNC = 1 << 3,
-};
-
-static void peer_device_to_statistics(struct peer_device_statistics *s,
- struct drbd_peer_device *peer_device)
+void peer_device_to_statistics(struct drbd_peer_device_statistics *s,
+ struct drbd_peer_device *pd)
{
- struct drbd_device *device = peer_device->device;
+ struct drbd_device *device = pd->device;
+ struct drbd_md *md;
+ struct drbd_peer_md *peer_md;
+ struct drbd_bitmap *bm;
+ unsigned long now = jiffies;
+ unsigned long rs_left = 0;
+ int i;
+
+ /* userspace should get "future proof" units,
+ * convert to sectors or milli seconds as appropriate */
memset(s, 0, sizeof(*s));
- s->peer_dev_received = device->recv_cnt;
- s->peer_dev_sent = device->send_cnt;
- s->peer_dev_pending = atomic_read(&device->ap_pending_cnt) +
- atomic_read(&device->rs_pending_cnt);
- s->peer_dev_unacked = atomic_read(&device->unacked_cnt);
- s->peer_dev_out_of_sync = drbd_bm_total_weight(device) << (BM_BLOCK_SHIFT - 9);
- s->peer_dev_resync_failed = device->rs_failed << (BM_BLOCK_SHIFT - 9);
- if (get_ldev(device)) {
- struct drbd_md *md = &device->ldev->md;
+ s->peer_dev_received = pd->recv_cnt;
+ s->peer_dev_sent = pd->send_cnt;
+ s->peer_dev_pending = atomic_read(&pd->ap_pending_cnt) +
+ atomic_read(&pd->rs_pending_cnt);
+ s->peer_dev_unacked = atomic_read(&pd->unacked_cnt);
+ s->peer_dev_uuid_flags = pd->uuid_flags;
+
+ /* Below are resync / verify / bitmap / meta data stats.
+ * Without disk, we don't have those.
+ */
+ if (!get_ldev(device))
+ return;
- spin_lock_irq(&md->uuid_lock);
- s->peer_dev_bitmap_uuid = md->uuid[UI_BITMAP];
- spin_unlock_irq(&md->uuid_lock);
- s->peer_dev_flags =
- (drbd_md_test_flag(device->ldev, MDF_CONNECTED_IND) ?
- MDF_PEER_CONNECTED : 0) +
- (drbd_md_test_flag(device->ldev, MDF_CONSISTENT) &&
- !drbd_md_test_flag(device->ldev, MDF_WAS_UP_TO_DATE) ?
- MDF_PEER_OUTDATED : 0) +
- /* FIXME: MDF_PEER_FENCING? */
- (drbd_md_test_flag(device->ldev, MDF_FULL_SYNC) ?
- MDF_PEER_FULL_SYNC : 0);
+ md = &device->ldev->md;
+ peer_md = &md->peers[pd->node_id];
+
+ spin_lock_irq(&md->uuid_lock);
+ s->peer_dev_bitmap_uuid = peer_md->bitmap_uuid;
+ spin_unlock_irq(&md->uuid_lock);
+ s->peer_dev_flags = peer_md->flags;
+
+ /* A disk attached without a bitmap has no out-of-sync, resync or
+ * verify state to report, and no bm_block_shift to convert with.
+ */
+ bm = device->bitmap;
+ if (!bm) {
put_ldev(device);
+ return;
+ }
+
+ s->peer_dev_out_of_sync = bm_bit_to_sect(bm, drbd_bm_total_weight(pd));
+
+ if (is_verify_state(pd, NOW)) {
+ rs_left = bm_bit_to_sect(bm, atomic64_read(&pd->ov_left));
+ s->peer_dev_ov_start_sector = pd->ov_start_sector;
+ s->peer_dev_ov_stop_sector = pd->ov_stop_sector;
+ s->peer_dev_ov_position = pd->ov_position;
+ s->peer_dev_ov_left = bm_bit_to_sect(bm, atomic64_read(&pd->ov_left));
+ s->peer_dev_ov_skipped = bm_bit_to_sect(bm, pd->ov_skipped);
+ } else if (is_sync_state(pd, NOW)) {
+ rs_left = s->peer_dev_out_of_sync - bm_bit_to_sect(bm, pd->rs_failed);
+ s->peer_dev_resync_failed = bm_bit_to_sect(bm, pd->rs_failed);
+ s->peer_dev_rs_same_csum = bm_bit_to_sect(bm, pd->rs_same_csum);
+ }
+
+ if (rs_left) {
+ enum drbd_repl_state repl_state = pd->repl_state[NOW];
+
+ if (repl_state == L_SYNC_TARGET || repl_state == L_VERIFY_S)
+ s->peer_dev_rs_c_sync_rate = pd->c_sync_rate;
+
+ s->peer_dev_rs_total = bm_bit_to_sect(bm, pd->rs_total);
+
+ s->peer_dev_rs_dt_start_ms = jiffies_to_msecs(now - pd->rs_start);
+ s->peer_dev_rs_paused_ms = jiffies_to_msecs(pd->rs_paused);
+
+ i = (pd->rs_last_mark + 2) % DRBD_SYNC_MARKS;
+ s->peer_dev_rs_dt0_ms = jiffies_to_msecs(now - pd->rs_mark_time[i]);
+ s->peer_dev_rs_db0_sectors = bm_bit_to_sect(bm, pd->rs_mark_left[i]) - rs_left;
+
+ i = (pd->rs_last_mark + DRBD_SYNC_MARKS-1) % DRBD_SYNC_MARKS;
+ s->peer_dev_rs_dt1_ms = jiffies_to_msecs(now - pd->rs_mark_time[i]);
+ s->peer_dev_rs_db1_sectors = bm_bit_to_sect(bm, pd->rs_mark_left[i]) - rs_left;
+
+ /* long term average:
+ * dt = rs_dt_start_ms - rs_paused_ms;
+ * db = rs_total - rs_left, which is
+ * rs_total - (ov_left? ov_left : out_of_sync - rs_failed)
+ */
}
+
+ put_ldev(device);
}
-int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb)
+int drbd_dump_peer_devices_done(struct netlink_callback *cb)
{
return put_resource_in_arg0(cb, 9);
}
-int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+int drbd_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect)
{
- struct nlattr *resource_filter;
struct drbd_resource *resource;
struct drbd_device *device;
struct drbd_peer_device *peer_device = NULL;
- int minor, err, retcode;
- struct drbd_genlmsghdr *dh;
+ struct drbd_peer_device_info peer_device_info;
+ struct drbd_peer_device_statistics peer_device_statistics;
+ struct drbd_peer_device_conf *peer_device_conf;
+ int minor, err;
struct idr *idr_to_search;
resource = (struct drbd_resource *)cb->args[0];
- if (!cb->args[0] && !cb->args[1]) {
- resource_filter = find_cfg_context_attr(cb->nlh,
- DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
- if (resource_filter) {
- retcode = ERR_RES_NOT_KNOWN;
- resource = drbd_find_resource(nla_data(resource_filter));
- if (!resource) {
- rcu_read_lock();
- goto put_result;
- }
- }
- cb->args[0] = (long)resource;
- }
rcu_read_lock();
minor = cb->args[1];
@@ -3748,7 +7292,7 @@ int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, struct netlink_callback
}
}
if (cb->args[2]) {
- for_each_peer_device(peer_device, device)
+ for_each_peer_device_rcu(peer_device, device)
if (peer_device == (struct drbd_peer_device *)cb->args[2])
goto found_peer_device;
/* peer device was probably deleted */
@@ -3759,43 +7303,21 @@ int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, struct netlink_callback
found_peer_device:
list_for_each_entry_continue_rcu(peer_device, &device->peer_devices, peer_devices) {
- if (!has_net_conf(peer_device->connection))
- continue;
- retcode = NO_ERROR;
goto put_result; /* only one iteration */
}
goto next_device;
put_result:
- dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq, &drbd_nl_family,
- NLM_F_MULTI, DRBD_ADM_GET_PEER_DEVICES);
- err = -ENOMEM;
- if (!dh)
+ peer_device_to_info(&peer_device_info, peer_device);
+ peer_device_to_statistics(&peer_device_statistics, peer_device);
+ peer_device_conf = rcu_dereference(peer_device->conf);
+ err = dialect->emit_peer_device(skb, cb, NO_ERROR, peer_device, minor,
+ &peer_device_info, &peer_device_statistics,
+ peer_device_conf);
+ if (err)
goto out;
- dh->ret_code = retcode;
- dh->minor = -1U;
- if (retcode == NO_ERROR) {
- struct peer_device_info peer_device_info;
- struct peer_device_statistics peer_device_statistics;
-
- dh->minor = minor;
- err = nla_put_drbd_cfg_context(skb, device->resource, peer_device->connection, device);
- if (err)
- goto out;
- peer_device_to_info(&peer_device_info, peer_device);
- err = peer_device_info_to_skb(skb, &peer_device_info);
- if (err)
- goto out;
- peer_device_to_statistics(&peer_device_statistics, peer_device);
- err = peer_device_statistics_to_skb(skb, &peer_device_statistics);
- if (err)
- goto out;
- cb->args[1] = minor;
- cb->args[2] = (long)peer_device;
- }
- genlmsg_end(skb, dh);
- err = 0;
+ cb->args[1] = minor;
+ cb->args[2] = (long)peer_device;
out:
rcu_read_unlock();
@@ -3803,461 +7325,241 @@ int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, struct netlink_callback
return err;
return skb->len;
}
-/*
- * Return the connection of @resource if @resource has exactly one connection.
- */
-static struct drbd_connection *the_only_connection(struct drbd_resource *resource)
-{
- struct list_head *connections = &resource->connections;
-
- if (list_empty(connections) || connections->next->next != connections)
- return NULL;
- return list_first_entry(&resource->connections, struct drbd_connection, connections);
-}
-
-static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device,
- const struct sib_info *sib)
-{
- struct drbd_resource *resource = device->resource;
- struct state_info *si = NULL; /* for sizeof(si->member); */
- struct nlattr *nla;
- int got_ldev;
- int err = 0;
- int exclude_sensitive;
-
- /* If sib != NULL, this is drbd_bcast_event, which anyone can listen
- * to. So we better exclude_sensitive information.
- *
- * If sib == NULL, this is drbd_nl_get_status_doit, executed synchronously
- * in the context of the requesting user process. Exclude sensitive
- * information, unless current has superuser.
- *
- * NOTE: for drbd_nl_get_status_dumpit(), this is a netlink dump, and
- * relies on the current implementation of netlink_dump(), which
- * executes the dump callback successively from netlink_recvmsg(),
- * always in the context of the receiving process */
- exclude_sensitive = sib || !capable(CAP_SYS_ADMIN);
-
- got_ldev = get_ldev(device);
-
- /* We need to add connection name and volume number information still.
- * Minor number is in drbd_genlmsghdr. */
- if (nla_put_drbd_cfg_context(skb, resource, the_only_connection(resource), device))
- goto nla_put_failure;
-
- if (res_opts_to_skb(skb, &device->resource->res_opts))
- goto nla_put_failure;
-
- rcu_read_lock();
- if (got_ldev) {
- struct disk_conf *disk_conf;
-
- disk_conf = rcu_dereference(device->ldev->disk_conf);
- err = disk_conf_to_skb(skb, disk_conf);
- }
- if (!err) {
- struct net_conf *nc;
-
- nc = rcu_dereference(first_peer_device(device)->connection->net_conf);
- if (nc)
- err = net_conf_to_skb_sanitized(skb, nc, exclude_sensitive);
- }
- rcu_read_unlock();
- if (err)
- goto nla_put_failure;
-
- nla = nla_nest_start_noflag(skb, DRBD_NLA_STATE_INFO);
- if (!nla)
- goto nla_put_failure;
- if (nla_put_u32(skb, DRBD_A_STATE_INFO_SIB_REASON,
- sib ? sib->sib_reason : SIB_GET_STATUS_REPLY) ||
- nla_put_u32(skb, DRBD_A_STATE_INFO_CURRENT_STATE,
- device->state.i) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_ED_UUID,
- device->ed_uuid, 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_CAPACITY,
- get_capacity(device->vdisk), 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_SEND_CNT,
- device->send_cnt, 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_RECV_CNT,
- device->recv_cnt, 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_READ_CNT,
- device->read_cnt, 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_WRIT_CNT,
- device->writ_cnt, 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_AL_WRIT_CNT,
- device->al_writ_cnt, 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BM_WRIT_CNT,
- device->bm_writ_cnt, 0) ||
- nla_put_u32(skb, DRBD_A_STATE_INFO_AP_BIO_CNT,
- atomic_read(&device->ap_bio_cnt)) ||
- nla_put_u32(skb, DRBD_A_STATE_INFO_AP_PENDING_CNT,
- atomic_read(&device->ap_pending_cnt)) ||
- nla_put_u32(skb, DRBD_A_STATE_INFO_RS_PENDING_CNT,
- atomic_read(&device->rs_pending_cnt)))
- goto nla_put_failure;
-
- if (got_ldev) {
- int err;
-
- spin_lock_irq(&device->ldev->md.uuid_lock);
- err = nla_put(skb, DRBD_A_STATE_INFO_UUIDS,
- sizeof(si->uuids),
- device->ldev->md.uuid);
- spin_unlock_irq(&device->ldev->md.uuid_lock);
-
- if (err)
- goto nla_put_failure;
-
- if (nla_put_u32(skb, DRBD_A_STATE_INFO_DISK_FLAGS, device->ldev->md.flags) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_TOTAL, drbd_bm_bits(device), 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_OOS,
- drbd_bm_total_weight(device), 0))
- goto nla_put_failure;
- if (C_SYNC_SOURCE <= device->state.conn &&
- C_PAUSED_SYNC_T >= device->state.conn) {
- if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_TOTAL,
- device->rs_total, 0) ||
- nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_FAILED,
- device->rs_failed, 0))
- goto nla_put_failure;
- }
- }
-
- if (sib) {
- switch(sib->sib_reason) {
- case SIB_SYNC_PROGRESS:
- case SIB_GET_STATUS_REPLY:
- break;
- case SIB_STATE_CHANGE:
- if (nla_put_u32(skb, DRBD_A_STATE_INFO_PREV_STATE, sib->os.i) ||
- nla_put_u32(skb, DRBD_A_STATE_INFO_NEW_STATE, sib->ns.i))
- goto nla_put_failure;
- break;
- case SIB_HELPER_POST:
- if (nla_put_u32(skb, DRBD_A_STATE_INFO_HELPER_EXIT_CODE,
- sib->helper_exit_code))
- goto nla_put_failure;
- fallthrough;
- case SIB_HELPER_PRE:
- if (nla_put_string(skb, DRBD_A_STATE_INFO_HELPER, sib->helper_name))
- goto nla_put_failure;
- break;
- }
- }
- nla_nest_end(skb, nla);
-
- if (0)
-nla_put_failure:
- err = -EMSGSIZE;
- if (got_ldev)
- put_ldev(device);
- return err;
-}
-int drbd_nl_get_status_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_dump_paths_done(struct netlink_callback *cb)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
- int err;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
-
- err = nla_put_status_info(adm_ctx->reply_skb, adm_ctx->device, NULL);
- if (err) {
- nlmsg_free(adm_ctx->reply_skb);
- adm_ctx->reply_skb = NULL;
- return err;
- }
-out:
- adm_ctx->reply_dh->ret_code = retcode;
- return 0;
+ return put_resource_in_arg0(cb, 10);
}
-static int get_one_status(struct sk_buff *skb, struct netlink_callback *cb)
+int drbd_dump_paths(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect)
{
- struct drbd_device *device;
- struct drbd_genlmsghdr *dh;
- struct drbd_resource *pos = (struct drbd_resource *)cb->args[0];
- struct drbd_resource *resource = NULL;
- struct drbd_resource *tmp;
- unsigned volume = cb->args[1];
-
- /* Open coded, deferred, iteration:
- * for_each_resource_safe(resource, tmp, &drbd_resources) {
- * connection = "first connection of resource or undefined";
- * idr_for_each_entry(&resource->devices, device, i) {
- * ...
- * }
- * }
- * where resource is cb->args[0];
- * and i is cb->args[1];
- *
- * cb->args[2] indicates if we shall loop over all resources,
- * or just dump all volumes of a single resource.
- *
- * This may miss entries inserted after this dump started,
- * or entries deleted before they are reached.
- *
- * We need to make sure the device won't disappear while
- * we are looking at it, and revalidate our iterators
- * on each iteration.
- */
+ struct drbd_resource *resource = NULL, *next_resource;
+ struct drbd_connection *connection = NULL;
+ struct drbd_path *path = NULL;
+ struct drbd_nl_path_info path_info;
+ int err = 0;
- /* synchronize with conn_create()/drbd_destroy_connection() */
rcu_read_lock();
- /* revalidate iterator position */
- for_each_resource_rcu(tmp, &drbd_resources) {
- if (pos == NULL) {
- /* first iteration */
- pos = tmp;
- resource = pos;
- break;
- }
- if (tmp == pos) {
- resource = pos;
- break;
- }
+ resource = (struct drbd_resource *)cb->args[0];
+ if (!resource) {
+ if (list_empty(&drbd_resources))
+ goto out;
+ resource = list_first_entry(&drbd_resources, struct drbd_resource, resources);
+ kref_get(&resource->kref);
+ cb->args[0] = (long)resource;
+ cb->args[1] = DRBD_DUMP_ITERATE_RESOURCES;
}
- if (resource) {
+
next_resource:
- device = idr_get_next(&resource->devices, &volume);
- if (!device) {
- /* No more volumes to dump on this resource.
- * Advance resource iterator. */
- pos = list_entry_rcu(resource->resources.next,
- struct drbd_resource, resources);
- /* Did we dump any volume of this resource yet? */
- if (volume != 0) {
- /* If we reached the end of the list,
- * or only a single resource dump was requested,
- * we are done. */
- if (&pos->resources == &drbd_resources || cb->args[2])
- goto out;
- volume = 0;
- resource = pos;
- goto next_resource;
- }
+ rcu_read_unlock();
+ mutex_lock(&resource->conf_update);
+ rcu_read_lock();
+ if (cb->args[2]) {
+ for_each_connection_rcu(connection, resource) {
+ list_for_each_entry_rcu(path, &connection->transport.paths, list)
+ if (path == (struct drbd_path *)cb->args[2])
+ goto found_path;
}
+ /* path was probably deleted */
+ goto no_more_paths;
+ }
- dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq, &drbd_nl_family,
- NLM_F_MULTI, DRBD_ADM_GET_STATUS);
- if (!dh)
- goto out;
-
- if (!device) {
- /* This is a connection without a single volume.
- * Suprisingly enough, it may have a network
- * configuration. */
- struct drbd_connection *connection;
+ connection = first_connection(resource);
+ if (!connection)
+ goto no_more_paths;
- dh->minor = -1U;
- dh->ret_code = NO_ERROR;
- connection = the_only_connection(resource);
- if (nla_put_drbd_cfg_context(skb, resource, connection, NULL))
- goto cancel;
- if (connection) {
- struct net_conf *nc;
-
- nc = rcu_dereference(connection->net_conf);
- if (nc && net_conf_to_skb_sanitized(skb, nc, true) != 0)
- goto cancel;
- }
- goto done;
- }
+ path = list_entry(&connection->transport.paths, struct drbd_path, list);
- D_ASSERT(device, device->vnr == volume);
- D_ASSERT(device, device->resource == resource);
+found_path:
+ /* Advance to next path in connection. */
+ list_for_each_entry_continue_rcu(path, &connection->transport.paths, list) {
+ goto put_result; /* only one iteration */
+ }
- dh->minor = device_to_minor(device);
- dh->ret_code = NO_ERROR;
+ /* Advance to next connection. */
+ list_for_each_entry_continue_rcu(connection, &resource->connections, connections) {
+ path = first_path(connection);
+ if (!path)
+ continue;
+ goto put_result;
+ }
- if (nla_put_status_info(skb, device, NULL)) {
-cancel:
- genlmsg_cancel(skb, dh);
- goto out;
+no_more_paths:
+ if (cb->args[1] == DRBD_DUMP_ITERATE_RESOURCES) {
+ for_each_resource_rcu(next_resource, &drbd_resources) {
+ if (next_resource == resource)
+ goto found_resource;
}
-done:
- genlmsg_end(skb, dh);
+ /* resource was probably deleted */
+ }
+ goto out;
+
+found_resource:
+ list_for_each_entry_continue_rcu(next_resource, &drbd_resources, resources) {
+ mutex_unlock(&resource->conf_update);
+ kref_put(&resource->kref, drbd_destroy_resource);
+ resource = next_resource;
+ kref_get(&resource->kref);
+ cb->args[0] = (long)resource;
+ cb->args[2] = 0;
+ goto next_resource;
}
+ goto out; /* no more resources */
+
+put_result:
+ path_info.path_established = test_bit(TR_ESTABLISHED, &path->flags);
+ err = dialect->emit_path(skb, cb, NO_ERROR, resource, connection, path, &path_info);
+ if (err)
+ goto out;
+ cb->args[2] = (long)path;
out:
rcu_read_unlock();
- /* where to start the next iteration */
- cb->args[0] = (long)pos;
- cb->args[1] = (pos == resource) ? volume + 1 : 0;
-
- /* No more resources/volumes/minors found results in an empty skb.
- * Which will terminate the dump. */
- return skb->len;
+ if (resource)
+ mutex_unlock(&resource->conf_update);
+ if (err)
+ return err;
+ return skb->len;
}
-/*
- * Request status of all resources, or of all volumes within a single resource.
- *
- * This is a dump, as the answer may not fit in a single reply skb otherwise.
- * Which means we cannot use the family->attrbuf or other such members, because
- * dump is NOT protected by the genl_lock(). During dump, we only have access
- * to the incoming skb, and need to opencode "parsing" of the nlattr payload.
- *
- * Once things are setup properly, we call into get_one_status().
- */
-int drbd_nl_get_status_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+int drbd_adm_get_timeout_type(struct drbd_adm_ctx *adm_ctx)
{
- const unsigned int hdrlen = GENL_HDRLEN + sizeof(struct drbd_genlmsghdr);
- struct nlattr *nla;
- const char *resource_name;
- struct drbd_resource *resource;
-
- /* Is this a followup call? */
- if (cb->args[0]) {
- /* ... of a single resource dump,
- * and the resource iterator has been advanced already? */
- if (cb->args[2] && cb->args[2] != cb->args[0])
- return 0; /* DONE. */
- goto dump;
- }
-
- /* First call (from netlink_dump_start). We need to figure out
- * which resource(s) the user wants us to dump. */
- nla = nla_find(nlmsg_attrdata(cb->nlh, hdrlen),
- nlmsg_attrlen(cb->nlh, hdrlen),
- DRBD_NLA_CFG_CONTEXT);
-
- /* No explicit context given. Dump all. */
- if (!nla)
- goto dump;
- nla = nla_find_nested(nla, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
- /* context given, but no name present? */
- if (!nla)
- return -EINVAL;
- resource_name = nla_data(nla);
- if (!*resource_name)
- return -ENODEV;
- resource = drbd_find_resource(resource_name);
- if (!resource)
- return -ENODEV;
+ struct drbd_peer_device *peer_device;
+ enum drbd_timeout_flag timeout_type;
- kref_put(&resource->kref, drbd_destroy_resource); /* get_one_status() revalidates the resource */
+ peer_device = adm_ctx->peer_device;
- /* prime iterators, and set "filter" mode mark:
- * only dump this connection. */
- cb->args[0] = (long)resource;
- /* cb->args[1] = 0; passed in this way. */
- cb->args[2] = (long)resource;
+ timeout_type =
+ peer_device->disk_state[NOW] == D_OUTDATED ? UT_PEER_OUTDATED :
+ test_bit(USE_DEGR_WFC_T, peer_device->flags) ? UT_DEGRADED :
+ UT_DEFAULT;
-dump:
- return get_one_status(skb, cb);
+ adm_ctx->result = adm_ctx->d->put_timeout_type(adm_ctx, timeout_type);
+ return 0;
}
-int drbd_nl_get_timeout_type_doit(struct sk_buff *skb, struct genl_info *info)
+static enum drbd_ret_code check_verify_alg_matches_peer(struct drbd_peer_device *peer_device,
+ struct drbd_adm_ctx *ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
- struct timeout_parms tp;
- int err;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ struct drbd_connection *connection = peer_device->connection;
+ struct drbd_resource *resource = connection->resource;
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_net_conf *nc;
- tp.timeout_type =
- adm_ctx->device->state.pdsk == D_OUTDATED ? UT_PEER_OUTDATED :
- test_bit(USE_DEGR_WFC_T, &adm_ctx->device->flags) ? UT_DEGRADED :
- UT_DEFAULT;
-
- err = timeout_parms_to_skb(adm_ctx->reply_skb, &tp);
- if (err) {
- nlmsg_free(adm_ctx->reply_skb);
- adm_ctx->reply_skb = NULL;
- return err;
- }
-out:
- adm_ctx->reply_dh->ret_code = retcode;
- return 0;
+ mutex_lock(&resource->conf_update);
+ nc = connection->transport.net_conf;
+ if (connection->agreed_pro_version >= 88 &&
+ peer_device->repl_state[NOW] >= L_ESTABLISHED &&
+ nc && nc->verify_alg[0] &&
+ strcmp(nc->verify_alg, connection->peer_verify_alg)) {
+ drbd_adm_msg(ctx,
+ "Online verify refused: local verify-alg \"%s\" differs from peer verify-alg \"%s\"",
+ nc->verify_alg, connection->peer_verify_alg);
+ retcode = ERR_VERIFY_ALG;
+ }
+ mutex_unlock(&resource->conf_update);
+
+ return retcode;
}
-int drbd_nl_start_ov_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_start_ov(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_device *device;
- enum drbd_ret_code retcode;
- struct start_ov_parms parms;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ struct drbd_peer_device *peer_device;
+ enum drbd_ret_code retcode = NO_ERROR;
+ enum drbd_state_rv rv;
+ struct drbd_start_ov_parms parms;
- device = adm_ctx->device;
+ peer_device = adm_ctx->peer_device;
+ device = peer_device->device;
/* resume from last known position, if possible */
- parms.ov_start_sector = device->ov_start_sector;
+ parms.ov_start_sector = peer_device->ov_start_sector;
parms.ov_stop_sector = ULLONG_MAX;
- if (info->attrs[DRBD_NLA_START_OV_PARMS]) {
- int err = start_ov_parms_from_attrs(&parms, info);
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_START_OV_PARMS)) {
+ int err = drbd_adm_overlay_start_ov_parms(adm_ctx, &parms);
+
if (err) {
retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
+ drbd_adm_msg_overlay_error(adm_ctx, err);
goto out;
}
}
- mutex_lock(&adm_ctx->resource->adm_mutex);
+ if (!get_ldev(device)) {
+ retcode = ERR_NO_DISK;
+ goto out;
+ }
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_put_ldev;
+ }
+
+ retcode = check_verify_alg_matches_peer(peer_device, adm_ctx);
+ if (retcode != NO_ERROR) {
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ goto out_put_ldev;
+ }
/* w_make_ov_request expects position to be aligned */
- device->ov_start_sector = parms.ov_start_sector & ~(BM_SECT_PER_BIT-1);
- device->ov_stop_sector = parms.ov_stop_sector;
+ peer_device->ov_start_sector = parms.ov_start_sector & ~(bm_sect_per_bit(device->bitmap)-1);
+ peer_device->ov_stop_sector = parms.ov_stop_sector;
/* If there is still bitmap IO pending, e.g. previous resync or verify
* just being finished, wait for it before requesting a new resync. */
- drbd_suspend_io(device);
- wait_event(device->misc_wait, !test_bit(BITMAP_IO, &device->flags));
- retcode = drbd_request_state(device, NS(conn, C_VERIFY_S));
+ drbd_suspend_io(device, READ_AND_WRITE);
+ wait_event(device->misc_wait, !atomic_read(&device->pending_bitmap_work.n));
+ rv = stable_change_repl_state(peer_device,
+ L_VERIFY_S, CS_VERBOSE | CS_WAIT_COMPLETE | CS_SERIALIZE, "verify");
drbd_resume_io(device);
mutex_unlock(&adm_ctx->resource->adm_mutex);
+ put_ldev(device);
+ adm_ctx->result = rv;
+ return 0;
+
+out_put_ldev:
+ put_ldev(device);
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
+static bool should_skip_initial_sync(struct drbd_peer_device *peer_device)
+{
+ return peer_device->repl_state[NOW] == L_ESTABLISHED &&
+ peer_device->connection->agreed_pro_version >= 90 &&
+ drbd_current_uuid(peer_device->device) == UUID_JUST_CREATED;
+}
-int drbd_nl_new_c_uuid_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_new_c_uuid(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_device *device;
- enum drbd_ret_code retcode;
- int skip_initial_sync = 0;
+ struct drbd_peer_device *peer_device;
+ enum drbd_ret_code retcode = NO_ERROR;
int err;
- struct new_c_uuid_parms args;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out_nolock;
+ struct drbd_new_c_uuid_parms args;
+ u64 nodes = 0, diskful = 0;
device = adm_ctx->device;
memset(&args, 0, sizeof(args));
- if (info->attrs[DRBD_NLA_NEW_C_UUID_PARMS]) {
- err = new_c_uuid_parms_from_attrs(&args, info);
+ if (adm_ctx->d->has_set(adm_ctx, DRBD_NL_SET_NEW_C_UUID_PARMS)) {
+ err = drbd_adm_overlay_new_c_uuid_parms(adm_ctx, &args);
if (err) {
retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
- goto out_nolock;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto out_no_adm_mutex;
}
}
- mutex_lock(&adm_ctx->resource->adm_mutex);
- mutex_lock(device->state_mutex); /* Protects us against serialized state changes. */
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
+ if (down_interruptible(&device->resource->state_sem)) {
+ retcode = ERR_INTR;
+ goto out_no_state_sem;
+ }
if (!get_ldev(device)) {
retcode = ERR_NO_DISK;
@@ -4265,194 +7567,419 @@ int drbd_nl_new_c_uuid_doit(struct sk_buff *skb, struct genl_info *info)
}
/* this is "skip initial sync", assume to be clean */
- if (device->state.conn == C_CONNECTED &&
- first_peer_device(device)->connection->agreed_pro_version >= 90 &&
- device->ldev->md.uuid[UI_CURRENT] == UUID_JUST_CREATED && args.clear_bm) {
- drbd_info(device, "Preparing to skip initial sync\n");
- skip_initial_sync = 1;
- } else if (device->state.conn != C_STANDALONE) {
- retcode = ERR_CONNECTED;
- goto out_dec;
+ for_each_peer_device(peer_device, device) {
+ if ((args.clear_bm || args.force_resync) && should_skip_initial_sync(peer_device)) {
+ if (peer_device->disk_state[NOW] >= D_INCONSISTENT) {
+ drbd_info(peer_device, "Preparing to %s initial sync\n",
+ args.clear_bm ? "skip" : "force");
+ diskful |= NODE_MASK(peer_device->node_id);
+ }
+ nodes |= NODE_MASK(peer_device->node_id);
+ } else if (peer_device->repl_state[NOW] != L_OFF) {
+ retcode = ERR_CONNECTED;
+ goto out_dec;
+ }
}
- drbd_uuid_set(device, UI_BITMAP, 0); /* Rotate UI_BITMAP to History 1, etc... */
- drbd_uuid_new_current(device); /* New current, previous to UI_BITMAP */
+ /* gen-rotate reason: OTHER (admin new-current-uuid) */
+ drbd_uuid_new_current_by_user(device); /* New current, previous to UI_BITMAP */
+
+ if (args.force_resync) {
+ unsigned long irq_flags;
+
+ begin_state_change(device->resource, &irq_flags, CS_VERBOSE);
+ __change_disk_state(device, D_UP_TO_DATE);
+ end_state_change(device->resource, &irq_flags, "new-c-uuid");
+
+ for_each_peer_device(peer_device, device) {
+ if (NODE_MASK(peer_device->node_id) & nodes) {
+ if (NODE_MASK(peer_device->node_id) & diskful) {
+ drbd_info(peer_device, "Forcing resync");
+ set_bit(CONSIDER_RESYNC, peer_device->flags);
+ drbd_send_uuids(peer_device, UUID_FLAG_RESYNC, 0);
+ drbd_send_current_state(peer_device);
+ } else {
+ drbd_send_uuids(peer_device, 0, 0);
+ }
+
+ drbd_print_uuids(peer_device, "forced resync UUID");
+ }
+ }
+ }
if (args.clear_bm) {
- err = drbd_bitmap_io(device, &drbd_bmio_clear_n_write,
- "clear_n_write from new_c_uuid", BM_LOCKED_MASK, NULL);
+ unsigned long irq_flags;
+
+ err = drbd_bitmap_io(device, &drbd_bmio_clear_all_n_write,
+ "clear_n_write from new_c_uuid", BM_LOCK_ALL, NULL);
if (err) {
drbd_err(device, "Writing bitmap failed with %d\n", err);
retcode = ERR_IO_MD_DISK;
}
- if (skip_initial_sync) {
- drbd_send_uuids_skip_initial_sync(first_peer_device(device));
- _drbd_uuid_set(device, UI_BITMAP, 0);
- drbd_print_uuids(device, "cleared bitmap UUID");
- spin_lock_irq(&device->resource->req_lock);
- _drbd_set_state(_NS2(device, disk, D_UP_TO_DATE, pdsk, D_UP_TO_DATE),
- CS_VERBOSE, NULL);
- spin_unlock_irq(&device->resource->req_lock);
+ for_each_peer_device(peer_device, device) {
+ if (NODE_MASK(peer_device->node_id) & nodes) {
+ _drbd_uuid_set_bitmap(peer_device, 0);
+ drbd_send_uuids(peer_device, UUID_FLAG_SKIP_INITIAL_SYNC, 0);
+ /* The peer adopts our new UUID but does not send its
+ * UUID back, so update our cached record here to
+ * avoid a stale mismatch in sanitize_state().
+ */
+ peer_device->current_uuid = drbd_current_uuid(device);
+ drbd_print_uuids(peer_device, "cleared bitmap UUID");
+ }
+ }
+ begin_state_change(device->resource, &irq_flags, CS_VERBOSE);
+ __change_disk_state(device, D_UP_TO_DATE);
+ for_each_peer_device(peer_device, device) {
+ if (NODE_MASK(peer_device->node_id) & diskful)
+ __change_peer_disk_state(peer_device, D_UP_TO_DATE);
}
+ end_state_change(device->resource, &irq_flags, "new-c-uuid");
}
- drbd_md_sync(device);
+ drbd_md_sync_if_dirty(device);
out_dec:
put_ldev(device);
out:
- mutex_unlock(device->state_mutex);
+ up(&device->resource->state_sem);
+out_no_state_sem:
mutex_unlock(&adm_ctx->resource->adm_mutex);
-out_nolock:
- adm_ctx->reply_dh->ret_code = retcode;
+out_no_adm_mutex:
+ adm_ctx->result = retcode;
return 0;
}
-static enum drbd_ret_code
-drbd_check_resource_name(struct drbd_config_context *adm_ctx)
+/* name: a resource or connection name
+ * Comes from a NLA_NUL_STRING, and already passed validate_nla().
+ * It is known to be NUL-terminated within the bounds of our defined netlink
+ * attribute policy.
+ *
+ * It must not be empty.
+ * It must not be the literal "all".
+ *
+ * If strict:
+ * Only allow strict ascii alnum [0-9A-Za-z]
+ * and some hand selected punctuation characters
+ *
+ * If non strict:
+ * It must not contain '/', we use it as directory name in debugfs.
+ * It shall not contain "control characters" or space, as those may confuse
+ * utils when trying to parse the output of "drbdsetup events2" or similar.
+ * Otherwise, we don't care, it may be any tag that makes sense to userland,
+ * we do not enforce strict ascii or any other "encoding".
+ */
+static enum drbd_ret_code drbd_check_name_str(const char *name, const bool strict)
{
- const char *name = adm_ctx->resource_name;
- if (!name || !name[0]) {
- drbd_msg_put_info(adm_ctx->reply_skb, "resource name missing");
+ unsigned char c;
+
+ if (name == NULL || name[0] == 0)
return ERR_MANDATORY_TAG;
- }
- /* if we want to use these in sysfs/configfs/debugfs some day,
- * we must not allow slashes */
- if (strchr(name, '/')) {
- drbd_msg_put_info(adm_ctx->reply_skb, "invalid resource name");
+
+ /* Tools reserve the literal "all" to mean what you would expect. */
+ /* If we want to get really paranoid,
+ * we could add a number of "reserved" names,
+ * like the *_state_names defined in drbd_strings.c */
+ if (memcmp("all", name, 4) == 0)
return ERR_INVALID_REQUEST;
+
+ while ((c = *name++)) {
+ if (c == '/' || c <= ' ' || c == '\x7f')
+ return ERR_INVALID_REQUEST;
+ if (strict) {
+ switch (c) {
+ case '0' ... '9':
+ case 'A' ... 'Z':
+ case 'a' ... 'z':
+ /* if you change this, also change "strict_pattern" below */
+ case '+': case '-': case '.': case '_':
+ break;
+ default:
+ return ERR_INVALID_REQUEST;
+ }
+ }
}
return NO_ERROR;
}
-static void resource_to_info(struct resource_info *info,
+int param_set_drbd_strict_names(const char *val, const struct kernel_param *kp)
+{
+ int err = 0;
+ bool new_value;
+ bool orig_value = *(bool *)kp->arg;
+ struct kernel_param dummy_kp = *kp;
+
+ dummy_kp.arg = &new_value;
+
+ err = param_set_bool(val, &dummy_kp);
+ if (err || new_value == orig_value)
+ return err;
+
+ if (new_value) {
+ struct drbd_resource *resource;
+ struct drbd_connection *connection;
+ int non_strict_cnt = 0;
+
+ /* If we transition from "not enforced" to "enforcing strict names",
+ * we complain about all "non-strict names" that still exist,
+ * but intentionally still enable the enforcing.
+ *
+ * That way we can prevent new "non-strict" from being created,
+ * while allowing us to clean up the existing ones at some
+ * "convenient time" later.
+ */
+ rcu_read_lock();
+ for_each_resource_rcu(resource, &drbd_resources) {
+ for_each_connection_rcu(connection, resource) {
+ char *name = connection->transport.net_conf->name;
+
+ if (drbd_check_name_str(name, true) == NO_ERROR)
+ continue;
+ drbd_info(connection, "non-strict name still in use\n");
+ ++non_strict_cnt;
+ }
+ if (drbd_check_name_str(resource->name, true) == NO_ERROR)
+ continue;
+ drbd_info(resource, "non-strict name still in use\n");
+ ++non_strict_cnt;
+ }
+ rcu_read_unlock();
+ if (non_strict_cnt)
+ pr_notice("%u non-strict names still in use\n", non_strict_cnt);
+ }
+ if (!err) {
+ *(bool *)kp->arg = new_value;
+ pr_info("%s strict name checks\n", new_value ? "enabled" : "disabled");
+ }
+ return err;
+}
+
+static void drbd_msg_put_name_error(struct drbd_adm_ctx *ctx, enum drbd_ret_code ret_code)
+{
+ char *strict_pattern = " (strict_names=1 allows only [0-9A-Za-z+._-])";
+ char *non_strict_pat = " (disallowed: ascii control, space, slash)";
+
+ if (ret_code == NO_ERROR)
+ return;
+ if (ret_code == ERR_INVALID_REQUEST) {
+ drbd_adm_msg(ctx, "invalid name%s",
+ drbd_strict_names ? strict_pattern : non_strict_pat);
+ } else if (ret_code == ERR_MANDATORY_TAG) {
+ drbd_adm_msg(ctx, "%s", "name missing");
+ } else if (ret_code == ERR_ALREADY_EXISTS) {
+ drbd_adm_msg(ctx, "%s", "name already exists");
+ } else {
+ drbd_adm_msg(ctx, "%s", "unhandled error in drbd_check_name_str");
+ }
+}
+
+static enum drbd_ret_code drbd_check_resource_name(struct drbd_adm_ctx *const adm_ctx)
+{
+ enum drbd_ret_code ret_code = drbd_check_name_str(adm_ctx->resource_name, drbd_strict_names);
+
+ drbd_msg_put_name_error(adm_ctx, ret_code);
+ return ret_code;
+}
+
+static void resource_to_info(struct drbd_resource_info *info,
struct drbd_resource *resource)
{
- info->res_role = conn_highest_role(first_connection(resource));
- info->res_susp = resource->susp;
- info->res_susp_nod = resource->susp_nod;
- info->res_susp_fen = resource->susp_fen;
+ info->res_role = resource->role[NOW];
+ info->res_susp = resource->susp_user[NOW];
+ info->res_susp_nod = resource->susp_nod[NOW];
+ info->res_susp_fen = is_suspended_fen(resource, NOW);
+ info->res_susp_quorum = resource->susp_quorum[NOW];
+ info->res_fail_io = resource->fail_io[NOW];
+#ifdef CONFIG_DRBD_COMPAT_84
+ /*
+ * This snapshot has no "before" state of its own (unlike
+ * notify_resource_state_change()'s state_change-derived one, drbd_
+ * state.c) -- every caller of this function reports the live state
+ * as either a fresh object or an out-of-band status reply, not a
+ * transition. old == new here so a v1 dialect fused event, if one
+ * is ever built from this snapshot, reports no change and
+ * compat84_event_flush() skips it rather than printing a
+ * fabricated transition. Combines susp_user/susp_quorum/susp_uuid
+ * the same way drbd_get_resource_state()'s own .susp does, since
+ * res_susp above is susp_user alone.
+ */
+ info->old_res_role = info->res_role;
+ info->old_res_susp = resource->susp_user[NOW] || resource->susp_quorum[NOW] ||
+ resource->susp_uuid[NOW];
+ info->old_res_susp_nod = info->res_susp_nod;
+ info->old_res_susp_fen = info->res_susp_fen;
+#endif
}
-int drbd_nl_new_resource_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_new_resource(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_connection *connection;
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
- struct res_opts res_opts;
+ struct drbd_resource *resource;
+ enum drbd_ret_code retcode = NO_ERROR;
+ struct drbd_res_opts res_opts;
int err;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ mutex_lock(&resources_mutex);
- set_res_opts_defaults(&res_opts);
- err = res_opts_from_attrs(&res_opts, info);
- if (err && err != -ENOMSG) {
+ drbd_set_res_opts_defaults(&res_opts);
+ res_opts.node_id = -1;
+ err = drbd_adm_overlay_res_opts(adm_ctx, &res_opts);
+ /*
+ * -ENOMSG here means the whole RESOURCE_OPTS container was absent
+ * from the request. Only the v1 (8.4) dialect's plain "new-resource"
+ * legitimately does that: an 8.4 resource has no node id to send, so
+ * force_drbd8_compat (set only by the v1 adapter) is the signal that
+ * this is such a request. Legacy and drbd2 both require RESOURCE_OPTS
+ * with a node id and must keep rejecting its absence exactly as
+ * before, so the tolerance is conditional on force_drbd8_compat
+ * rather than blanket: res_opts.node_id is seeded to the sentinel -1
+ * just above, which happens to wrap to UINT_MAX (node_id is __u32)
+ * and so still trips the node_id >= DRBD_NODE_ID_MAX check below for
+ * legacy/drbd2 even without this guard -- but that is an incidental
+ * property of the field's type, not a designed guarantee, and must
+ * not be the only thing standing between an unset node id and a
+ * created resource.
+ */
+ if (err && !(err == -ENOMSG && adm_ctx->force_drbd8_compat)) {
retcode = ERR_MANDATORY_TAG;
- drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
+ drbd_adm_msg_overlay_error(adm_ctx, err);
goto out;
}
+ /* ERR_ALREADY_EXISTS? */
+ if (adm_ctx->resource)
+ goto out;
+
retcode = drbd_check_resource_name(adm_ctx);
if (retcode != NO_ERROR)
goto out;
- if (adm_ctx->resource) {
- if (info->nlhdr->nlmsg_flags & NLM_F_EXCL) {
- retcode = ERR_INVALID_REQUEST;
- drbd_msg_put_info(adm_ctx->reply_skb, "resource exists");
- }
- /* else: still NO_ERROR */
+ /*
+ * Record it as explicit, too: drbd_adm_resource_opts() re-derives
+ * drbd8_compat_mode from explicit_drbd8_compat once node_id is set.
+ */
+ if (adm_ctx->force_drbd8_compat)
+ res_opts.explicit_drbd8_compat = true;
+ if (res_opts.explicit_drbd8_compat)
+ res_opts.drbd8_compat_mode = true;
+
+ if (res_opts.drbd8_compat_mode) {
+#ifdef CONFIG_DRBD_COMPAT_84
+ pr_info("drbd: running in DRBD 8 compatibility mode.\n");
+ /*
+ * That means we ignore the value of node_id for now. That
+ * will be set to an actual value when the resource is
+ * connected later.
+ */
+ atomic_inc(&nr_drbd8_devices);
+ res_opts.auto_promote = false;
+#else
+ drbd_adm_msg(adm_ctx, "%s", "CONFIG_DRBD_COMPAT_84 not enabled");
+ goto out;
+#endif
+ } else if (res_opts.node_id >= DRBD_NODE_ID_MAX) {
+ pr_err("drbd: invalid node id (%d)\n", res_opts.node_id);
+ retcode = ERR_INVALID_REQUEST;
goto out;
}
- /* not yet safe for genl_family.parallel_ops */
- mutex_lock(&resources_mutex);
- connection = conn_create(adm_ctx->resource_name, &res_opts);
+ if (!try_module_get(THIS_MODULE)) {
+ pr_err("drbd: Could not get a module reference\n");
+ retcode = ERR_INVALID_REQUEST;
+ goto out;
+ }
+
+ resource = drbd_create_resource(adm_ctx->resource_name, &res_opts);
mutex_unlock(&resources_mutex);
- if (connection) {
- struct resource_info resource_info;
+ if (resource) {
+ struct drbd_resource_info resource_info;
mutex_lock(¬ification_mutex);
- resource_to_info(&resource_info, connection->resource);
- notify_resource_state(NULL, 0, connection->resource,
- &resource_info, NOTIFY_CREATE);
+ resource_to_info(&resource_info, resource);
+ notify_resource_state(NULL, 0, resource, &resource_info, NULL, NOTIFY_CREATE);
mutex_unlock(¬ification_mutex);
- } else
+ } else {
+ module_put(THIS_MODULE);
retcode = ERR_NOMEM;
-
+ }
+ goto out_no_unlock;
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ mutex_unlock(&resources_mutex);
+out_no_unlock:
+ adm_ctx->result = retcode;
return 0;
}
-static void device_to_info(struct device_info *info,
- struct drbd_device *device)
-{
- info->dev_disk_state = device->state.disk;
-}
-
-
-int drbd_nl_new_minor_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_new_minor(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct drbd_genlmsghdr *dh = genl_info_userhdr(info);
- enum drbd_ret_code retcode;
+ struct drbd_device_conf device_conf;
+ struct drbd_resource *resource;
+ struct drbd_device *device;
+ enum drbd_ret_code retcode = NO_ERROR;
+ int err;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
+ drbd_set_device_conf_defaults(&device_conf);
+ err = drbd_adm_overlay_device_conf(adm_ctx, &device_conf);
+ if (err && err != -ENOMSG) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
goto out;
+ }
- if (dh->minor > MINORMASK) {
- drbd_msg_put_info(adm_ctx->reply_skb, "requested minor out of range");
+ if (adm_ctx->minor > MINORMASK) {
+ drbd_adm_msg(adm_ctx, "%s", "requested minor out of range");
retcode = ERR_INVALID_REQUEST;
goto out;
}
if (adm_ctx->volume > DRBD_VOLUME_MAX) {
- drbd_msg_put_info(adm_ctx->reply_skb, "requested volume id out of range");
+ drbd_adm_msg(adm_ctx, "%s", "requested volume id out of range");
retcode = ERR_INVALID_REQUEST;
goto out;
}
-
- /* drbd_adm_prepare made sure already
- * that first_peer_device(device)->connection and device->vnr match the request. */
- if (adm_ctx->device) {
- if (info->nlhdr->nlmsg_flags & NLM_F_EXCL)
- retcode = ERR_MINOR_OR_VOLUME_EXISTS;
- /* else: still NO_ERROR */
+ if (device_conf.block_size != 512 && device_conf.block_size != 1024 &&
+ device_conf.block_size != 2048 && device_conf.block_size != 4096) {
+ drbd_adm_msg(adm_ctx, "%s", "block_size not 512, 1024, 2048, or 4096");
+ retcode = ERR_INVALID_REQUEST;
+ goto out;
+ }
+ if (device_conf.discard_granularity != DRBD_DISCARD_GRANULARITY_DEF &&
+ device_conf.discard_granularity != 0 &&
+ device_conf.discard_granularity % device_conf.block_size != 0) {
+ drbd_adm_msg(adm_ctx, "%s",
+ "discard_granularity must be 0 or a multiple of block_size");
+ retcode = ERR_INVALID_REQUEST;
goto out;
}
- mutex_lock(&adm_ctx->resource->adm_mutex);
- retcode = drbd_create_device(adm_ctx, dh->minor);
+ if (adm_ctx->device)
+ goto out;
+
+ resource = adm_ctx->resource;
+ mutex_lock(&resource->conf_update);
+ for (;;) {
+ retcode = drbd_create_device(adm_ctx, adm_ctx->minor, &device_conf, &device);
+ if (retcode != ERR_NOMEM ||
+ schedule_timeout_interruptible(HZ / 10))
+ break;
+ /* Keep retrying until the memory allocations eventually succeed. */
+ }
if (retcode == NO_ERROR) {
- struct drbd_device *device;
struct drbd_peer_device *peer_device;
- struct device_info info;
+ struct drbd_device_info info;
unsigned int peer_devices = 0;
enum drbd_notification_type flags;
- device = minor_to_device(dh->minor);
- for_each_peer_device(peer_device, device) {
- if (!has_net_conf(peer_device->connection))
- continue;
+ drbd_reconsider_queue_parameters(device, NULL);
+
+ for_each_peer_device(peer_device, device)
peer_devices++;
- }
device_to_info(&info, device);
mutex_lock(¬ification_mutex);
flags = (peer_devices--) ? NOTIFY_CONTINUES : 0;
notify_device_state(NULL, 0, device, &info, NOTIFY_CREATE | flags);
for_each_peer_device(peer_device, device) {
- struct peer_device_info peer_device_info;
+ struct drbd_peer_device_info peer_device_info;
- if (!has_net_conf(peer_device->connection))
- continue;
peer_device_to_info(&peer_device_info, peer_device);
flags = (peer_devices--) ? NOTIFY_CONTINUES : 0;
notify_peer_device_state(NULL, 0, peer_device, &peer_device_info,
@@ -4460,502 +7987,422 @@ int drbd_nl_new_minor_doit(struct sk_buff *skb, struct genl_info *info)
}
mutex_unlock(¬ification_mutex);
}
- mutex_unlock(&adm_ctx->resource->adm_mutex);
+ mutex_unlock(&resource->conf_update);
out:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
static enum drbd_ret_code adm_del_minor(struct drbd_device *device)
{
+ struct drbd_resource *resource = device->resource;
struct drbd_peer_device *peer_device;
+ enum drbd_ret_code ret;
+ u64 im;
- if (device->state.disk == D_DISKLESS &&
- /* no need to be device->state.conn == C_STANDALONE &&
- * we may want to delete a minor from a live replication group.
- */
- device->state.role == R_SECONDARY) {
- struct drbd_connection *connection =
- first_connection(device->resource);
+ read_lock_irq(&resource->state_rwlock);
+ if (device->disk_state[NOW] == D_DISKLESS)
+ ret = test_and_set_bit(UNREGISTERED, &device->flags) ? ERR_MINOR_INVALID : NO_ERROR;
+ else
+ ret = ERR_MINOR_CONFIGURED;
+ read_unlock_irq(&resource->state_rwlock);
- _drbd_request_state(device, NS(conn, C_WF_REPORT_PARAMS),
- CS_VERBOSE + CS_WAIT_COMPLETE);
+ if (ret != NO_ERROR)
+ return ret;
- /* If the state engine hasn't stopped the sender thread yet, we
- * need to flush the sender work queue before generating the
- * DESTROY events here. */
- if (get_t_state(&connection->worker) == RUNNING)
- drbd_flush_workqueue(&connection->sender_work);
+ for_each_peer_device_ref(peer_device, im, device) {
+ enum drbd_state_rv rv;
- mutex_lock(¬ification_mutex);
- for_each_peer_device(peer_device, device) {
- if (!has_net_conf(peer_device->connection))
- continue;
- notify_peer_device_state(NULL, 0, peer_device, NULL,
- NOTIFY_DESTROY | NOTIFY_CONTINUES);
- }
- notify_device_state(NULL, 0, device, NULL, NOTIFY_DESTROY);
- mutex_unlock(¬ification_mutex);
+ rv = stable_change_repl_state(peer_device, L_OFF,
+ CS_VERBOSE | CS_WAIT_COMPLETE, "del-minor");
+ if (rv < SS_SUCCESS)
+ drbd_err(peer_device,
+ "Deleting the volume with replication not stopped (%s)\n",
+ drbd_set_st_err_str(rv));
+ }
+
+ /* If drbd_ldev_destroy() is pending, wait for it to run before
+ * unregistering the device. */
+ wait_event(device->misc_wait, !test_bit(GOING_DISKLESS, &device->flags));
+ /*
+ * Flush the resource work queue to make sure that no more events like
+ * state change notifications for this device are queued: we want the
+ * "destroy" event to come last.
+ */
+ drbd_flush_workqueue(&resource->work);
+
+ drbd_unregister_device(device);
- set_bit(UNREGISTERED, &device->flags);
- drbd_delete_device(device);
- return NO_ERROR;
- } else
- return ERR_MINOR_CONFIGURED;
+ mutex_lock(¬ification_mutex);
+ for_each_peer_device_ref(peer_device, im, device)
+ notify_peer_device_state(NULL, 0, peer_device, NULL,
+ NOTIFY_DESTROY | NOTIFY_CONTINUES);
+ notify_device_state(NULL, 0, device, NULL, NOTIFY_DESTROY);
+ mutex_unlock(¬ification_mutex);
+
+ if (device->open_cnt == 0 && !test_and_set_bit(DESTROYING_DEV, &device->flags))
+ call_rcu(&device->rcu, drbd_reclaim_device);
+
+ return ret;
}
-int drbd_nl_del_minor_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_del_minor(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- enum drbd_ret_code retcode;
+ enum drbd_ret_code retcode = NO_ERROR;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto out;
+ if (mutex_lock_interruptible(&adm_ctx->resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ } else {
+ retcode = adm_del_minor(adm_ctx->device);
+ mutex_unlock(&adm_ctx->resource->adm_mutex);
+ }
- mutex_lock(&adm_ctx->resource->adm_mutex);
- retcode = adm_del_minor(adm_ctx->device);
- mutex_unlock(&adm_ctx->resource->adm_mutex);
-out:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
static int adm_del_resource(struct drbd_resource *resource)
{
- struct drbd_connection *connection;
-
- for_each_connection(connection, resource) {
- if (connection->cstate > C_STANDALONE)
- return ERR_NET_CONFIGURED;
- }
- if (!idr_is_empty(&resource->devices))
- return ERR_RES_IN_USE;
+ int err;
- /* The state engine has stopped the sender thread, so we don't
- * need to flush the sender work queue before generating the
- * DESTROY event here. */
- mutex_lock(¬ification_mutex);
- notify_resource_state(NULL, 0, resource, NULL, NOTIFY_DESTROY);
- mutex_unlock(¬ification_mutex);
+ /*
+ * Flush the resource work queue to make sure that no more events like
+ * state change notifications are queued: we want the "destroy" event
+ * to come last.
+ */
+ drbd_flush_workqueue(&resource->work);
mutex_lock(&resources_mutex);
+ err = ERR_RES_NOT_KNOWN;
+ if (test_bit(R_UNREGISTERED, &resource->flags))
+ goto out;
+ err = ERR_NET_CONFIGURED;
+ if (!list_empty(&resource->connections))
+ goto out;
+ err = ERR_RES_IN_USE;
+ if (!idr_is_empty(&resource->devices))
+ goto out;
+
set_bit(R_UNREGISTERED, &resource->flags);
list_del_rcu(&resource->resources);
+ drbd_debugfs_resource_cleanup(resource);
mutex_unlock(&resources_mutex);
- /* Make sure all threads have actually stopped: state handling only
- * does drbd_thread_stop_nowait(). */
- list_for_each_entry(connection, &resource->connections, connections)
- drbd_thread_stop(&connection->worker);
- synchronize_rcu();
- drbd_free_resource(resource);
+
+ drbd_free_send_oos_leftovers(resource);
+
+ if (cancel_work_sync(&resource->empty_twopc)) {
+ kref_put(&resource->kref, drbd_destroy_resource);
+ }
+ if (cancel_work_sync(&resource->resume_twopc)) {
+ kref_put(&resource->kref, drbd_destroy_resource);
+ }
+ timer_shutdown_sync(&resource->twopc_timer);
+ timer_shutdown_sync(&resource->peer_ack_timer);
+ call_rcu(&resource->rcu, drbd_reclaim_resource);
+
+ mutex_lock(¬ification_mutex);
+ notify_resource_state(NULL, 0, resource, NULL, NULL, NOTIFY_DESTROY);
+ mutex_unlock(¬ification_mutex);
+
+ /* When the last resource was removed do an explicit synchronize RCU.
+ Without this a immediately following rmmod would fail, since the
+ resource's worker thread still has a reference count to the module. */
+ if (list_empty(&drbd_resources))
+ synchronize_rcu();
return NO_ERROR;
+out:
+ mutex_unlock(&resources_mutex);
+ return err;
}
-int drbd_nl_down_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_down(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_resource *resource;
struct drbd_connection *connection;
struct drbd_device *device;
int retcode; /* enum drbd_ret_code rsp. enum drbd_state_rv */
- unsigned i;
-
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto finish;
+ enum drbd_ret_code ret;
+ int i;
+ u64 im;
resource = adm_ctx->resource;
- mutex_lock(&resource->adm_mutex);
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
+ set_bit(DOWN_IN_PROGRESS, &resource->flags);
/* demote */
- for_each_connection(connection, resource) {
- struct drbd_peer_device *peer_device;
-
- idr_for_each_entry(&connection->peer_devices, peer_device, i) {
- retcode = drbd_set_role(peer_device->device, R_SECONDARY, 0);
- if (retcode < SS_SUCCESS) {
- drbd_msg_put_info(adm_ctx->reply_skb, "failed to demote");
- goto out;
- }
- }
+ retcode = drbd_set_role(resource, R_SECONDARY, 0, "down", adm_ctx);
+ if (retcode < SS_SUCCESS) {
+ drbd_adm_msg(adm_ctx, "%s", "failed to demote");
+ goto out;
+ }
- retcode = conn_try_disconnect(connection, 0);
- if (retcode < SS_SUCCESS) {
- drbd_msg_put_info(adm_ctx->reply_skb, "failed to disconnect");
+ for_each_connection_ref(connection, im, resource) {
+ retcode = SS_SUCCESS;
+ if (connection->cstate[NOW] > C_STANDALONE)
+ retcode = conn_try_disconnect(connection, 0, "down", adm_ctx);
+ if (retcode >= SS_SUCCESS) {
+ del_connection(connection, "down");
+ } else {
+ kref_put(&connection->kref, drbd_destroy_connection);
goto out;
}
}
- /* detach */
+ /* detach and delete minor */
+ rcu_read_lock();
idr_for_each_entry(&resource->devices, device, i) {
- retcode = adm_detach(device, 0);
+ kref_get(&device->kref);
+ rcu_read_unlock();
+ retcode = adm_detach(device, 0, 0, "down", adm_ctx);
+ mutex_lock(&resource->conf_update);
+ ret = adm_del_minor(device);
+ mutex_unlock(&resource->conf_update);
+ kref_put(&device->kref, drbd_destroy_device);
if (retcode < SS_SUCCESS || retcode > NO_ERROR) {
- drbd_msg_put_info(adm_ctx->reply_skb, "failed to detach");
+ drbd_adm_msg(adm_ctx, "%s", "failed to detach");
goto out;
}
- }
-
- /* delete volumes */
- idr_for_each_entry(&resource->devices, device, i) {
- retcode = adm_del_minor(device);
- if (retcode != NO_ERROR) {
+ if (ret != NO_ERROR) {
/* "can not happen" */
- drbd_msg_put_info(adm_ctx->reply_skb, "failed to delete volume");
+ drbd_adm_msg(adm_ctx, "%s", "failed to delete volume");
goto out;
}
+ rcu_read_lock();
}
+ rcu_read_unlock();
+ mutex_lock(&resource->conf_update);
retcode = adm_del_resource(resource);
+ /* holding a reference to resource in adm_ctx until post_doit kfree */
+ mutex_unlock(&resource->conf_update);
out:
+ opener_info(adm_ctx->resource, adm_ctx, (enum drbd_state_rv)retcode);
+ clear_bit(DOWN_IN_PROGRESS, &resource->flags);
mutex_unlock(&resource->adm_mutex);
-finish:
- adm_ctx->reply_dh->ret_code = retcode;
+out_no_adm_mutex:
+ adm_ctx->result = retcode;
return 0;
}
-int drbd_nl_del_resource_doit(struct sk_buff *skb, struct genl_info *info)
+int drbd_adm_del_resource(struct drbd_adm_ctx *adm_ctx)
{
- struct drbd_config_context *adm_ctx = info->user_ptr[0];
- struct drbd_resource *resource;
- enum drbd_ret_code retcode;
+ enum drbd_ret_code retcode = NO_ERROR;
- if (!adm_ctx->reply_skb)
- return 0;
- retcode = adm_ctx->reply_dh->ret_code;
- if (retcode != NO_ERROR)
- goto finish;
- resource = adm_ctx->resource;
+ retcode = adm_del_resource(adm_ctx->resource);
- mutex_lock(&resource->adm_mutex);
- retcode = adm_del_resource(resource);
- mutex_unlock(&resource->adm_mutex);
-finish:
- adm_ctx->reply_dh->ret_code = retcode;
+ adm_ctx->result = retcode;
return 0;
}
-void drbd_bcast_event(struct drbd_device *device, const struct sib_info *sib)
+/*
+ * Announce an event to every registered dialect. All of them describe
+ * the same event, so it gets a single sequence number, drawn here.
+ *
+ * When a dialect is given (and then there is always an skb) this is the
+ * initial state replay of drbd_dump_initial_state() instead: build the
+ * message into the dump skb of that one dialect, under the sequence
+ * number of the dump.
+ */
+int drbd_notify_resource_state(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *resource_info,
+ struct drbd_rename_resource_info *rename_resource_info,
+ enum drbd_notification_type type)
{
- struct sk_buff *msg;
- struct drbd_genlmsghdr *d_out;
- unsigned seq;
- int err = -ENOMEM;
+ unsigned int i;
+ int err = 0;
+ if (dialect)
+ return dialect->notify_resource_state(skb, seq, resource, resource_info,
+ rename_resource_info, type);
+
+ WARN_ON_ONCE(skb);
seq = atomic_inc_return(&drbd_genl_seq);
- msg = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
- if (!msg)
- goto failed;
-
- err = -EMSGSIZE;
- d_out = genlmsg_put(msg, 0, seq, &drbd_nl_family, 0, DRBD_ADM_EVENT);
- if (!d_out) /* cannot happen, but anyways. */
- goto nla_put_failure;
- d_out->minor = device_to_minor(device);
- d_out->ret_code = NO_ERROR;
-
- if (nla_put_status_info(msg, device, sib))
- goto nla_put_failure;
- genlmsg_end(msg, d_out);
- err = drbd_genl_multicast_events(msg, GFP_NOWAIT);
- /* msg has been consumed or freed in netlink_broadcast() */
- if (err && err != -ESRCH)
- goto failed;
-
- return;
-
-nla_put_failure:
- nlmsg_free(msg);
-failed:
- drbd_err(device, "Error %d while broadcasting event. "
- "Event seq:%u sib_reason:%u\n",
- err, seq, sib->sib_reason);
-}
-
-static int nla_put_notification_header(struct sk_buff *msg,
- enum drbd_notification_type type)
-{
- struct drbd_notification_header nh = {
- .nh_type = type,
- };
+ for (i = 0; i < drbd_nl_n_dialects; i++) {
+ int e = drbd_nl_dialects[i]->notify_resource_state(NULL, seq, resource,
+ resource_info, rename_resource_info, type);
- return drbd_notification_header_to_skb(msg, &nh);
+ if (e && !err)
+ err = e;
+ }
+ return err;
}
int notify_resource_state(struct sk_buff *skb,
- unsigned int seq,
- struct drbd_resource *resource,
- struct resource_info *resource_info,
- enum drbd_notification_type type)
-{
- struct resource_statistics resource_statistics;
- struct drbd_genlmsghdr *dh;
- bool multicast = false;
- int err;
+ unsigned int seq,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *resource_info,
+ struct drbd_rename_resource_info *rename_resource_info,
+ enum drbd_notification_type type)
+{
+ return drbd_notify_resource_state(skb, seq, NULL, resource, resource_info,
+ rename_resource_info, type);
+}
- if (!skb) {
- seq = atomic_inc_return(¬ify_genl_seq);
- skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
- err = -ENOMEM;
- if (!skb)
- goto failed;
- multicast = true;
- }
-
- err = -EMSGSIZE;
- dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_RESOURCE_STATE);
- if (!dh)
- goto nla_put_failure;
- dh->minor = -1U;
- dh->ret_code = NO_ERROR;
- if (nla_put_drbd_cfg_context(skb, resource, NULL, NULL) ||
- nla_put_notification_header(skb, type) ||
- ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
- resource_info_to_skb(skb, resource_info)))
- goto nla_put_failure;
- resource_statistics.res_stat_write_ordering = resource->write_ordering;
- err = resource_statistics_to_skb(skb, &resource_statistics);
- if (err)
- goto nla_put_failure;
- genlmsg_end(skb, dh);
- if (multicast) {
- err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
- /* skb has been consumed or freed in netlink_broadcast() */
- if (err && err != -ESRCH)
- goto failed;
- }
- return 0;
+int drbd_notify_device_state(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_device *device,
+ struct drbd_device_info *device_info,
+ enum drbd_notification_type type)
+{
+ unsigned int i;
+ int err = 0;
+
+ if (dialect)
+ return dialect->notify_device_state(skb, seq, device, device_info, type);
+
+ WARN_ON_ONCE(skb);
+ seq = atomic_inc_return(&drbd_genl_seq);
+ for (i = 0; i < drbd_nl_n_dialects; i++) {
+ int e = drbd_nl_dialects[i]->notify_device_state(NULL, seq, device,
+ device_info, type);
-nla_put_failure:
- nlmsg_free(skb);
-failed:
- drbd_err(resource, "Error %d while broadcasting event. Event seq:%u\n",
- err, seq);
+ if (e && !err)
+ err = e;
+ }
return err;
}
int notify_device_state(struct sk_buff *skb,
- unsigned int seq,
- struct drbd_device *device,
- struct device_info *device_info,
- enum drbd_notification_type type)
-{
- struct device_statistics device_statistics;
- struct drbd_genlmsghdr *dh;
- bool multicast = false;
- int err;
+ unsigned int seq,
+ struct drbd_device *device,
+ struct drbd_device_info *device_info,
+ enum drbd_notification_type type)
+{
+ return drbd_notify_device_state(skb, seq, NULL, device, device_info, type);
+}
- if (!skb) {
- seq = atomic_inc_return(¬ify_genl_seq);
- skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
- err = -ENOMEM;
- if (!skb)
- goto failed;
- multicast = true;
- }
-
- err = -EMSGSIZE;
- dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_DEVICE_STATE);
- if (!dh)
- goto nla_put_failure;
- dh->minor = device->minor;
- dh->ret_code = NO_ERROR;
- if (nla_put_drbd_cfg_context(skb, device->resource, NULL, device) ||
- nla_put_notification_header(skb, type) ||
- ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
- device_info_to_skb(skb, device_info)))
- goto nla_put_failure;
- device_to_statistics(&device_statistics, device);
- device_statistics_to_skb(skb, &device_statistics);
- genlmsg_end(skb, dh);
- if (multicast) {
- err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
- /* skb has been consumed or freed in netlink_broadcast() */
- if (err && err != -ESRCH)
- goto failed;
- }
- return 0;
+int drbd_notify_connection_state(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_connection *connection,
+ struct drbd_connection_info *connection_info,
+ enum drbd_notification_type type)
+{
+ unsigned int i;
+ int err = 0;
+
+ if (dialect)
+ return dialect->notify_connection_state(skb, seq, connection,
+ connection_info, type);
+
+ WARN_ON_ONCE(skb);
+ seq = atomic_inc_return(&drbd_genl_seq);
+ for (i = 0; i < drbd_nl_n_dialects; i++) {
+ int e = drbd_nl_dialects[i]->notify_connection_state(NULL, seq, connection,
+ connection_info, type);
-nla_put_failure:
- nlmsg_free(skb);
-failed:
- drbd_err(device, "Error %d while broadcasting event. Event seq:%u\n",
- err, seq);
+ if (e && !err)
+ err = e;
+ }
return err;
}
int notify_connection_state(struct sk_buff *skb,
- unsigned int seq,
- struct drbd_connection *connection,
- struct connection_info *connection_info,
- enum drbd_notification_type type)
+ unsigned int seq,
+ struct drbd_connection *connection,
+ struct drbd_connection_info *connection_info,
+ enum drbd_notification_type type)
{
- struct connection_statistics connection_statistics;
- struct drbd_genlmsghdr *dh;
- bool multicast = false;
- int err;
+ return drbd_notify_connection_state(skb, seq, NULL, connection,
+ connection_info, type);
+}
- if (!skb) {
- seq = atomic_inc_return(¬ify_genl_seq);
- skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
- err = -ENOMEM;
- if (!skb)
- goto failed;
- multicast = true;
- }
-
- err = -EMSGSIZE;
- dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_CONNECTION_STATE);
- if (!dh)
- goto nla_put_failure;
- dh->minor = -1U;
- dh->ret_code = NO_ERROR;
- if (nla_put_drbd_cfg_context(skb, connection->resource, connection, NULL) ||
- nla_put_notification_header(skb, type) ||
- ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
- connection_info_to_skb(skb, connection_info)))
- goto nla_put_failure;
- connection_statistics.conn_congested = test_bit(NET_CONGESTED, &connection->flags);
- connection_statistics_to_skb(skb, &connection_statistics);
- genlmsg_end(skb, dh);
- if (multicast) {
- err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
- /* skb has been consumed or freed in netlink_broadcast() */
- if (err && err != -ESRCH)
- goto failed;
- }
- return 0;
+int drbd_notify_peer_device_state(struct sk_buff *skb,
+ unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_info *peer_device_info,
+ enum drbd_notification_type type)
+{
+ unsigned int i;
+ int err = 0;
+
+ if (dialect)
+ return dialect->notify_peer_device_state(skb, seq, peer_device,
+ peer_device_info, type);
+
+ WARN_ON_ONCE(skb);
+ seq = atomic_inc_return(&drbd_genl_seq);
+ for (i = 0; i < drbd_nl_n_dialects; i++) {
+ int e = drbd_nl_dialects[i]->notify_peer_device_state(NULL, seq, peer_device,
+ peer_device_info, type);
-nla_put_failure:
- nlmsg_free(skb);
-failed:
- drbd_err(connection, "Error %d while broadcasting event. Event seq:%u\n",
- err, seq);
+ if (e && !err)
+ err = e;
+ }
return err;
}
int notify_peer_device_state(struct sk_buff *skb,
- unsigned int seq,
- struct drbd_peer_device *peer_device,
- struct peer_device_info *peer_device_info,
- enum drbd_notification_type type)
-{
- struct peer_device_statistics peer_device_statistics;
- struct drbd_resource *resource = peer_device->device->resource;
- struct drbd_genlmsghdr *dh;
- bool multicast = false;
- int err;
+ unsigned int seq,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_info *peer_device_info,
+ enum drbd_notification_type type)
+{
+ return drbd_notify_peer_device_state(skb, seq, NULL, peer_device,
+ peer_device_info, type);
+}
- if (!skb) {
- seq = atomic_inc_return(¬ify_genl_seq);
- skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
- err = -ENOMEM;
- if (!skb)
- goto failed;
- multicast = true;
- }
-
- err = -EMSGSIZE;
- dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_PEER_DEVICE_STATE);
- if (!dh)
- goto nla_put_failure;
- dh->minor = -1U;
- dh->ret_code = NO_ERROR;
- if (nla_put_drbd_cfg_context(skb, resource, peer_device->connection, peer_device->device) ||
- nla_put_notification_header(skb, type) ||
- ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
- peer_device_info_to_skb(skb, peer_device_info)))
- goto nla_put_failure;
- peer_device_to_statistics(&peer_device_statistics, peer_device);
- peer_device_statistics_to_skb(skb, &peer_device_statistics);
- genlmsg_end(skb, dh);
- if (multicast) {
- err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
- /* skb has been consumed or freed in netlink_broadcast() */
- if (err && err != -ESRCH)
- goto failed;
- }
- return 0;
+void drbd_broadcast_peer_device_state(struct drbd_peer_device *peer_device)
+{
+ struct drbd_peer_device_info peer_device_info;
-nla_put_failure:
- nlmsg_free(skb);
-failed:
- drbd_err(peer_device, "Error %d while broadcasting event. Event seq:%u\n",
- err, seq);
- return err;
+ mutex_lock(¬ification_mutex);
+ peer_device_to_info(&peer_device_info, peer_device);
+ notify_peer_device_state(NULL, 0, peer_device, &peer_device_info, NOTIFY_CHANGE);
+ mutex_unlock(¬ification_mutex);
}
-void notify_helper(enum drbd_notification_type type,
- struct drbd_device *device, struct drbd_connection *connection,
- const char *name, int status)
+static int notify_path_state(struct drbd_connection *connection,
+ struct drbd_path *path,
+ struct drbd_nl_path_info *path_info,
+ enum drbd_notification_type type)
{
- struct drbd_resource *resource = device ? device->resource : connection->resource;
- struct drbd_helper_info helper_info;
- unsigned int seq = atomic_inc_return(¬ify_genl_seq);
- struct sk_buff *skb = NULL;
- struct drbd_genlmsghdr *dh;
- int err;
+ unsigned int i, seq;
+ int err = 0;
- strscpy(helper_info.helper_name, name, sizeof(helper_info.helper_name));
- helper_info.helper_name_len = min(strlen(name), sizeof(helper_info.helper_name));
- helper_info.helper_status = status;
+ seq = atomic_inc_return(&drbd_genl_seq);
+ for (i = 0; i < drbd_nl_n_dialects; i++) {
+ int e = drbd_nl_dialects[i]->notify_path_state(NULL, seq, connection, path,
+ path_info, type);
- skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
- err = -ENOMEM;
- if (!skb)
- goto fail;
+ if (e && !err)
+ err = e;
+ }
+ return err;
+}
- err = -EMSGSIZE;
- dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_HELPER);
- if (!dh)
- goto fail;
- dh->minor = device ? device->minor : -1;
- dh->ret_code = NO_ERROR;
+int notify_path(struct drbd_connection *connection, struct drbd_path *path, enum drbd_notification_type type)
+{
+ struct drbd_nl_path_info path_info;
+ int err;
+
+ path_info.path_established = test_bit(TR_ESTABLISHED, &path->flags);
mutex_lock(¬ification_mutex);
- if (nla_put_drbd_cfg_context(skb, resource, connection, device) ||
- nla_put_notification_header(skb, type) ||
- drbd_helper_info_to_skb(skb, &helper_info))
- goto unlock_fail;
- genlmsg_end(skb, dh);
- err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
- skb = NULL;
- /* skb has been consumed or freed in netlink_broadcast() */
- if (err && err != -ESRCH)
- goto unlock_fail;
+ err = notify_path_state(connection, path, &path_info, type);
mutex_unlock(¬ification_mutex);
- return;
+ return err;
-unlock_fail:
- mutex_unlock(¬ification_mutex);
-fail:
- nlmsg_free(skb);
- drbd_err(resource, "Error %d while broadcasting event. Event seq:%u\n",
- err, seq);
}
-static int notify_initial_state_done(struct sk_buff *skb, unsigned int seq)
+void notify_helper(enum drbd_notification_type type,
+ struct drbd_device *device, struct drbd_connection *connection,
+ const char *name, int status)
{
- struct drbd_genlmsghdr *dh;
- int err;
-
- err = -EMSGSIZE;
- dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_INITIAL_STATE_DONE);
- if (!dh)
- goto nla_put_failure;
- dh->minor = -1U;
- dh->ret_code = NO_ERROR;
- if (nla_put_notification_header(skb, NOTIFY_EXISTS))
- goto nla_put_failure;
- genlmsg_end(skb, dh);
- return 0;
+ unsigned int seq = atomic_inc_return(&drbd_genl_seq);
+ unsigned int i;
-nla_put_failure:
- nlmsg_free(skb);
- pr_err("Error %d sending event. Event seq:%u\n", err, seq);
- return err;
+ mutex_lock(¬ification_mutex);
+ for (i = 0; i < drbd_nl_n_dialects; i++)
+ drbd_nl_dialects[i]->notify_helper(NULL, seq, device, connection,
+ name, status, type);
+ mutex_unlock(¬ification_mutex);
}
static void free_state_changes(struct list_head *list)
@@ -4973,53 +8420,72 @@ static unsigned int notifications_for_state_change(struct drbd_state_change *sta
return 1 +
state_change->n_connections +
state_change->n_devices +
- state_change->n_devices * state_change->n_connections;
+ state_change->n_devices * state_change->n_connections +
+ state_change->n_paths;
}
-static int get_initial_state(struct sk_buff *skb, struct netlink_callback *cb)
+static int get_initial_state(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect, unsigned int seq)
{
- struct drbd_state_change *state_change = (struct drbd_state_change *)cb->args[0];
- unsigned int seq = cb->args[2];
+ struct drbd_state_change *state_change;
unsigned int n;
- enum drbd_notification_type flags = 0;
+ enum drbd_notification_type flags;
int err = 0;
/* There is no need for taking notification_mutex here: it doesn't
- matter if the initial state events mix with later state chage
+ matter if the initial state events mix with later state change
events; we can always tell the events apart by the NOTIFY_EXISTS
flag. */
+again:
+ state_change = (struct drbd_state_change *)cb->args[0];
+ flags = 0;
cb->args[5]--;
if (cb->args[5] == 1) {
- err = notify_initial_state_done(skb, seq);
+ err = dialect->notify_initial_state_done(skb, seq);
goto out;
}
n = cb->args[4]++;
if (cb->args[4] < cb->args[3])
flags |= NOTIFY_CONTINUES;
if (n < 1) {
- err = notify_resource_state_change(skb, seq, state_change->resource,
+ err = notify_resource_state_change(skb, seq, dialect, state_change,
NOTIFY_EXISTS | flags);
goto next;
}
n--;
if (n < state_change->n_connections) {
- err = notify_connection_state_change(skb, seq, &state_change->connections[n],
+ err = notify_connection_state_change(skb, seq, dialect,
+ &state_change->connections[n],
NOTIFY_EXISTS | flags);
goto next;
}
n -= state_change->n_connections;
+ if (n < state_change->n_paths) {
+ struct drbd_path_state *path_state = &state_change->paths[n];
+ struct drbd_nl_path_info path_info;
+
+ path_info.path_established = path_state->path_established;
+ err = dialect->notify_path_state(skb, seq,
+ path_state->connection,
+ path_state->path,
+ &path_info, NOTIFY_EXISTS | flags);
+ goto next;
+ }
+ n -= state_change->n_paths;
if (n < state_change->n_devices) {
- err = notify_device_state_change(skb, seq, &state_change->devices[n],
+ err = notify_device_state_change(skb, seq, dialect, &state_change->devices[n],
NOTIFY_EXISTS | flags);
goto next;
}
n -= state_change->n_devices;
if (n < state_change->n_devices * state_change->n_connections) {
- err = notify_peer_device_state_change(skb, seq, &state_change->peer_devices[n],
+ err = notify_peer_device_state_change(skb, seq, dialect,
+ &state_change->peer_devices[n],
NOTIFY_EXISTS | flags);
goto next;
}
+ n -= state_change->n_devices * state_change->n_connections;
next:
if (cb->args[4] == cb->args[3]) {
@@ -5030,29 +8496,44 @@ static int get_initial_state(struct sk_buff *skb, struct netlink_callback *cb)
cb->args[3] = notifications_for_state_change(next_state_change);
cb->args[4] = 0;
}
+ /*
+ * A dialect may have nothing to send for an object (v1 has no paths).
+ * An empty skb would end the dump before the INITIAL_STATE_DONE
+ * message, so go on to the next notification.
+ */
+ if (!err && !skb->len)
+ goto again;
out:
if (err)
return err;
- else
- return skb->len;
+ return skb->len;
+}
+
+int drbd_dump_initial_state_done(struct netlink_callback *cb)
+{
+ LIST_HEAD(head);
+
+ if (cb->args[0]) {
+ struct drbd_state_change *state_change =
+ (struct drbd_state_change *)cb->args[0];
+ cb->args[0] = 0;
+
+ /* connect list to head */
+ list_add(&head, &state_change->list);
+ free_state_changes(&head);
+ }
+ return 0;
}
-int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+int drbd_dump_initial_state(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect, unsigned int seq)
{
struct drbd_resource *resource;
LIST_HEAD(head);
if (cb->args[5] >= 1) {
if (cb->args[5] > 1)
- return get_initial_state(skb, cb);
- if (cb->args[0]) {
- struct drbd_state_change *state_change =
- (struct drbd_state_change *)cb->args[0];
-
- /* connect list to head */
- list_add(&head, &state_change->list);
- free_state_changes(&head);
- }
+ return get_initial_state(skb, cb, dialect, seq);
return 0;
}
@@ -5061,7 +8542,9 @@ int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb, struct netlink_callbac
for_each_resource(resource, &drbd_resources) {
struct drbd_state_change *state_change;
- state_change = remember_old_state(resource, GFP_KERNEL);
+ read_lock_irq(&resource->state_rwlock);
+ state_change = remember_state_change(resource, GFP_ATOMIC);
+ read_unlock_irq(&resource->state_rwlock);
if (!state_change) {
if (!list_empty(&head))
free_state_changes(&head);
@@ -5082,23 +8565,136 @@ int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb, struct netlink_callbac
list_del(&head); /* detach list from head */
}
- cb->args[2] = cb->nlh->nlmsg_seq;
- return get_initial_state(skb, cb);
+ return get_initial_state(skb, cb, dialect, seq);
}
-static const struct genl_multicast_group drbd_nl_mcgrps[] = {
- [DRBD_NLGRP_EVENTS] = { .name = "events", },
-};
+int drbd_adm_forget_peer(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_resource *resource;
+ struct drbd_device *device;
+ struct drbd_forget_peer_parms parms = { };
+ enum drbd_ret_code retcode = NO_ERROR;
+ int vnr, peer_node_id, err;
-struct genl_family drbd_nl_family __ro_after_init = {
- .name = "drbd",
- .version = DRBD_FAMILY_VERSION,
- .hdrsize = NLA_ALIGN(sizeof(struct drbd_genlmsghdr)),
- .split_ops = drbd_nl_ops,
- .n_split_ops = ARRAY_SIZE(drbd_nl_ops),
- .mcgrps = drbd_nl_mcgrps,
- .n_mcgrps = ARRAY_SIZE(drbd_nl_mcgrps),
- .resv_start_op = 42,
- .module = THIS_MODULE,
- .netnsok = true,
-};
+ resource = adm_ctx->resource;
+
+ err = drbd_adm_overlay_forget_peer_parms(adm_ctx, &parms);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto out_no_adm_mutex;
+ }
+
+ if (mutex_lock_interruptible(&resource->adm_mutex)) {
+ retcode = ERR_INTR;
+ goto out_no_adm_mutex;
+ }
+
+ peer_node_id = parms.forget_peer_node_id;
+ if (drbd_connection_by_node_id(resource, peer_node_id)) {
+ retcode = ERR_NET_CONFIGURED;
+ goto out;
+ }
+
+ if (peer_node_id < 0 || peer_node_id >= DRBD_NODE_ID_MAX) {
+ retcode = ERR_INVALID_PEER_NODE_ID;
+ goto out;
+ }
+
+ idr_for_each_entry(&resource->devices, device, vnr)
+ clear_peer_slot(device, peer_node_id, 0);
+out:
+ mutex_unlock(&resource->adm_mutex);
+out_no_adm_mutex:
+ idr_for_each_entry(&resource->devices, device, vnr)
+ drbd_md_sync_if_dirty(device);
+
+ adm_ctx->result = (enum drbd_ret_code)retcode;
+ return 0;
+
+}
+
+static enum drbd_ret_code validate_new_resource_name(const struct drbd_resource *resource, const char *new_name)
+{
+ enum drbd_ret_code retcode = drbd_check_name_str(new_name, drbd_strict_names);
+
+ if (retcode == NO_ERROR) {
+ struct drbd_resource *next_resource;
+
+ rcu_read_lock();
+ for_each_resource_rcu(next_resource, &drbd_resources) {
+ if (strcmp(next_resource->name, new_name) == 0) {
+ retcode = ERR_ALREADY_EXISTS;
+ break;
+ }
+ }
+ rcu_read_unlock();
+ }
+ return retcode;
+}
+
+int drbd_adm_rename_resource(struct drbd_adm_ctx *adm_ctx)
+{
+ struct drbd_resource *resource;
+ struct drbd_device *device;
+ struct drbd_rename_resource_info rename_resource_info;
+ struct drbd_rename_resource_parms parms = { };
+ char *old_res_name, *new_res_name;
+ enum drbd_ret_code retcode = NO_ERROR;
+ enum drbd_ret_code validate_err;
+ int err;
+ int vnr;
+
+ mutex_lock(&resources_mutex);
+
+ resource = adm_ctx->resource;
+
+ err = drbd_adm_overlay_rename_resource_parms(adm_ctx, &parms);
+ if (err) {
+ retcode = ERR_MANDATORY_TAG;
+ drbd_adm_msg_overlay_error(adm_ctx, err);
+ goto out;
+ }
+
+ validate_err = validate_new_resource_name(resource, parms.new_resource_name);
+ if (validate_err != NO_ERROR) {
+ if (validate_err == ERR_ALREADY_EXISTS) {
+ drbd_adm_msg(adm_ctx,
+ "Cannot rename to %s: a resource with that name already exists\n",
+ parms.new_resource_name);
+ } else {
+ drbd_msg_put_name_error(adm_ctx, validate_err);
+ }
+ retcode = validate_err;
+ goto out;
+ }
+
+ drbd_info(resource, "Renaming to %s\n", parms.new_resource_name);
+
+ strscpy(rename_resource_info.res_new_name, parms.new_resource_name, sizeof(rename_resource_info.res_new_name));
+ rename_resource_info.res_new_name_len = min(strlen(parms.new_resource_name), sizeof(rename_resource_info.res_new_name));
+
+ mutex_lock(¬ification_mutex);
+ notify_resource_state(NULL, 0, resource, NULL, &rename_resource_info, NOTIFY_RENAME);
+ mutex_unlock(¬ification_mutex);
+
+ new_res_name = kstrdup(parms.new_resource_name, GFP_KERNEL);
+ if (!new_res_name) {
+ retcode = ERR_NOMEM;
+ goto out;
+ }
+ old_res_name = resource->name;
+ resource->name = new_res_name;
+ kvfree_rcu_mightsleep(old_res_name);
+
+ drbd_debugfs_resource_rename(resource, new_res_name);
+
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ kobject_uevent(&disk_to_dev(device->vdisk)->kobj, KOBJ_CHANGE);
+ }
+
+out:
+ mutex_unlock(&resources_mutex);
+ adm_ctx->result = retcode;
+ return 0;
+}
diff --git a/drivers/block/drbd/drbd_nl.h b/drivers/block/drbd/drbd_nl.h
new file mode 100644
index 000000000000..f1c0a4eefd7b
--- /dev/null
+++ b/drivers/block/drbd/drbd_nl.h
@@ -0,0 +1,400 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Context and outcome of one DRBD netlink configuration command.
+ *
+ * The command implementations are independent of the wire format: they
+ * receive the identity of the object to act on and report back a result
+ * code plus a few informational messages. Translating that into the
+ * bytes of a particular netlink dialect is the job of the per-dialect
+ * pre_doit/post_doit handlers and of the struct drbd_nl_dialect ops
+ * below.
+ */
+#ifndef __DRBD_NL_H
+#define __DRBD_NL_H
+
+#include <linux/compiler.h>
+#include "drbd_nl_types.h"
+
+struct drbd_device;
+struct drbd_resource;
+struct drbd_connection;
+struct drbd_peer_device;
+struct drbd_path;
+struct net;
+struct netlink_callback;
+struct sk_buff;
+struct drbd_nl_dialect;
+
+/* per-message cap, same as drbd_msg_sprintf_info()'s reserve */
+#define DRBD_ADM_MSG_MAX 256
+/*
+ * Total info text per request; a strict superset of what the legacy
+ * NLMSG_GOODSIZE reply could carry (that is capped at
+ * SKB_WITH_OVERHEAD(8192UL) on every page size).
+ */
+#define DRBD_ADM_MSG_BUF 8192
+
+struct drbd_adm_ctx {
+ /* the dialect the request arrived in, and its private request state */
+ const struct drbd_nl_dialect *d;
+ void *req;
+
+ /* identity, filled from the request by the dialect's pre_doit */
+ unsigned int minor;
+ unsigned int volume;
+#define VOLUME_UNSPECIFIED (-1U)
+ unsigned int peer_node_id;
+#define PEER_NODE_ID_UNSPECIFIED (-1U)
+ const char *resource_name; /* points into the request; limited lifetime */
+ struct net *net;
+ bool set_defaults;
+ /*
+ * Set by a dialect that has no attribute to request DRBD 8.4
+ * compatibility mode explicitly (the version 1 dialect: every
+ * resource it creates is a DRBD 8.4 resource). Honoured by
+ * drbd_adm_new_resource() the same way as res_opts.explicit_drbd8_compat.
+ */
+ bool force_drbd8_compat;
+
+ /* resolved by drbd_adm_ctx_resolve() */
+ struct drbd_device *device;
+ struct drbd_resource *resource;
+ struct drbd_connection *connection;
+ struct drbd_peer_device *peer_device;
+
+ /* outcome, serialized by the dialect's post_doit */
+ int result; /* NO_ERROR, ERR_*, SS_* or -errno */
+ unsigned int msg_len; /* bytes used in msg[] */
+ /*
+ * NUL-separated info messages, in order. Each message is truncated
+ * at DRBD_ADM_MSG_MAX; a message that does not fit into what is
+ * left of the buffer is dropped, exactly as an over-full reply skb
+ * dropped it before.
+ */
+ char msg[DRBD_ADM_MSG_BUF];
+};
+
+/* Attribute sets the core overlays onto live data or reads as parameters. */
+enum drbd_nl_attr_set {
+ DRBD_NL_SET_DISK_CONF,
+ DRBD_NL_SET_NET_CONF,
+ DRBD_NL_SET_RES_OPTS,
+ DRBD_NL_SET_PEER_DEVICE_CONF,
+ DRBD_NL_SET_DEVICE_CONF,
+ DRBD_NL_SET_SET_ROLE_PARMS,
+ DRBD_NL_SET_RESIZE_PARMS,
+ DRBD_NL_SET_START_OV_PARMS,
+ DRBD_NL_SET_NEW_C_UUID_PARMS,
+ DRBD_NL_SET_DISCONNECT_PARMS,
+ DRBD_NL_SET_DETACH_PARMS,
+ DRBD_NL_SET_INVALIDATE_PARMS,
+ DRBD_NL_SET_INVALIDATE_PEER_PARMS,
+ DRBD_NL_SET_FORGET_PEER_PARMS,
+ DRBD_NL_SET_CONNECT_PARMS,
+ DRBD_NL_SET_PATH_PARMS,
+ DRBD_NL_SET_RENAME_RESOURCE_PARMS,
+ DRBD_NL_SET_SUSPEND_IO_PARMS,
+ DRBD_NL_SET_CBT_START_PARMS,
+ __DRBD_NL_SET_MAX,
+};
+
+/* Fields the core must refuse to change once set. */
+enum drbd_adm_field {
+ DRBD_ADM_F_DISK_BACKING_DEV,
+ DRBD_ADM_F_DISK_META_DEV,
+ DRBD_ADM_F_DISK_META_DEV_IDX,
+ DRBD_ADM_F_DISK_SIZE,
+ DRBD_ADM_F_NET_TRANSPORT_NAME,
+ DRBD_ADM_F_NET_LOAD_BALANCE_PATHS,
+ DRBD_ADM_F_RES_NODE_ID,
+};
+
+struct drbd_nl_dialect {
+ const char *name;
+ /* Is the attribute set present in the request at all? */
+ bool (*has_set)(struct drbd_adm_ctx *ctx, enum drbd_nl_attr_set set);
+ /*
+ * Apply the request's attributes of "set" onto "dst" (a struct of
+ * the matching type); attributes absent from the request leave
+ * "dst" untouched. Returns 0, -ENOMSG (required attribute
+ * missing), -EEXIST (invariant change attempted), or -EINVAL.
+ */
+ int (*overlay)(struct drbd_adm_ctx *ctx, enum drbd_nl_attr_set set, void *dst);
+ /* Did the request carry this invariant field? Logs if it did. */
+ bool (*attr_present)(struct drbd_adm_ctx *ctx, enum drbd_adm_field field);
+ /* Reply payloads */
+ int (*put_timeout_type)(struct drbd_adm_ctx *ctx, enum drbd_timeout_flag type);
+ int (*put_cbt_info)(struct drbd_adm_ctx *ctx, struct drbd_cbt_info *info);
+
+ /*
+ * Dump emitters: append one message describing the object to the
+ * dump skb; return 0 or -errno. A "retcode" other than NO_ERROR
+ * describes a failure instead of an object: only the result code
+ * goes on the wire, and the object pointers must not be
+ * dereferenced (they may be NULL or stale).
+ */
+ int (*emit_resource)(struct sk_buff *skb, struct netlink_callback *cb,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *info,
+ struct drbd_resource_statistics *statistics);
+ int (*emit_device)(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_device *device,
+ struct drbd_disk_conf *disk_conf /* NULL if diskless */,
+ struct drbd_device_info *info,
+ struct drbd_device_statistics *statistics);
+ int (*emit_connection)(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_resource *resource,
+ struct drbd_connection *connection,
+ struct drbd_net_conf *net_conf /* NULL if none */,
+ struct drbd_connection_info *info,
+ struct drbd_connection_statistics *statistics);
+ int (*emit_peer_device)(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_peer_device *peer_device, unsigned int minor,
+ struct drbd_peer_device_info *info,
+ struct drbd_peer_device_statistics *statistics,
+ struct drbd_peer_device_conf *conf /* NULL if none */);
+ int (*emit_path)(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_resource *resource, struct drbd_connection *connection,
+ struct drbd_path *path, struct drbd_nl_path_info *info);
+
+ /*
+ * Notification emitters. skb == NULL: allocate a message, build it
+ * and multicast it to the listeners of this dialect. skb != NULL:
+ * this is the initial state replay, append the message to that
+ * dump skb instead. "seq" is the sequence number of the event,
+ * assigned once per event by the core.
+ */
+ int (*notify_resource_state)(struct sk_buff *skb, unsigned int seq,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *info,
+ struct drbd_rename_resource_info *rename_info,
+ enum drbd_notification_type type);
+ int (*notify_device_state)(struct sk_buff *skb, unsigned int seq,
+ struct drbd_device *device,
+ struct drbd_device_info *info,
+ enum drbd_notification_type type);
+ int (*notify_connection_state)(struct sk_buff *skb, unsigned int seq,
+ struct drbd_connection *connection,
+ struct drbd_connection_info *info,
+ enum drbd_notification_type type);
+ int (*notify_peer_device_state)(struct sk_buff *skb, unsigned int seq,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_info *info,
+ enum drbd_notification_type type);
+ int (*notify_path_state)(struct sk_buff *skb, unsigned int seq,
+ struct drbd_connection *connection, struct drbd_path *path,
+ struct drbd_nl_path_info *info,
+ enum drbd_notification_type type);
+ int (*notify_helper)(struct sk_buff *skb, unsigned int seq,
+ struct drbd_device *device, struct drbd_connection *connection,
+ const char *name, int status,
+ enum drbd_notification_type type);
+ int (*notify_initial_state_done)(struct sk_buff *skb, unsigned int seq);
+};
+
+/* Dialects that receive notifications; registered at module init. */
+int drbd_nl_register_dialect(const struct drbd_nl_dialect *dialect);
+
+/*
+ * The "drbd" family, at version 1 for DRBD 8.4 userland (drbd_nl_84.c),
+ * exists only with CONFIG_DRBD_COMPAT_84. The out-of-tree module can serve
+ * version 2 of it instead (DRBD_NL_FAMILY_V2, see DRBD_PROC_VERSION).
+ */
+#if defined(CONFIG_DRBD_COMPAT_84) || defined(DRBD_NL_FAMILY_V2)
+int drbd_nl_legacy_init(void);
+void drbd_nl_legacy_exit(void);
+#else
+static inline int drbd_nl_legacy_init(void) { return 0; }
+static inline void drbd_nl_legacy_exit(void) { }
+#endif
+
+int drbd_nl_drbd2_init(void);
+void drbd_nl_drbd2_exit(void);
+
+static inline int drbd_adm_overlay_disk_conf(struct drbd_adm_ctx *ctx, struct drbd_disk_conf *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_DISK_CONF, c); }
+
+static inline int drbd_adm_overlay_net_conf(struct drbd_adm_ctx *ctx, struct drbd_net_conf *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_NET_CONF, c); }
+
+static inline int drbd_adm_overlay_res_opts(struct drbd_adm_ctx *ctx, struct drbd_res_opts *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_RES_OPTS, c); }
+
+static inline int drbd_adm_overlay_peer_device_conf(struct drbd_adm_ctx *ctx,
+ struct drbd_peer_device_conf *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_PEER_DEVICE_CONF, c); }
+
+static inline int drbd_adm_overlay_device_conf(struct drbd_adm_ctx *ctx, struct drbd_device_conf *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_DEVICE_CONF, c); }
+
+static inline int drbd_adm_overlay_set_role_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_set_role_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_SET_ROLE_PARMS, c); }
+
+static inline int drbd_adm_overlay_resize_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_resize_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_RESIZE_PARMS, c); }
+
+static inline int drbd_adm_overlay_start_ov_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_start_ov_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_START_OV_PARMS, c); }
+
+static inline int drbd_adm_overlay_new_c_uuid_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_new_c_uuid_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_NEW_C_UUID_PARMS, c); }
+
+static inline int drbd_adm_overlay_disconnect_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_disconnect_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_DISCONNECT_PARMS, c); }
+
+static inline int drbd_adm_overlay_detach_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_detach_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_DETACH_PARMS, c); }
+
+static inline int drbd_adm_overlay_invalidate_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_invalidate_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_INVALIDATE_PARMS, c); }
+
+static inline int drbd_adm_overlay_invalidate_peer_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_invalidate_peer_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_INVALIDATE_PEER_PARMS, c); }
+
+static inline int drbd_adm_overlay_forget_peer_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_forget_peer_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_FORGET_PEER_PARMS, c); }
+
+static inline int drbd_adm_overlay_connect_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_connect_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_CONNECT_PARMS, c); }
+
+static inline int drbd_adm_overlay_path_parms(struct drbd_adm_ctx *ctx, struct drbd_path_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_PATH_PARMS, c); }
+
+static inline int drbd_adm_overlay_rename_resource_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_rename_resource_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_RENAME_RESOURCE_PARMS, c); }
+
+static inline int drbd_adm_overlay_suspend_io_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_suspend_io_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_SUSPEND_IO_PARMS, c); }
+
+static inline int drbd_adm_overlay_cbt_start_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_cbt_start_parms *c)
+{ return ctx->d->overlay(ctx, DRBD_NL_SET_CBT_START_PARMS, c); }
+
+__printf(2, 3) void drbd_adm_msg(struct drbd_adm_ctx *ctx, const char *fmt, ...);
+
+/* Flags for drbd_adm_ctx_resolve() */
+#define DRBD_ADM_NEED_MINOR (1 << 0)
+#define DRBD_ADM_NEED_RESOURCE (1 << 1)
+#define DRBD_ADM_NEED_CONNECTION (1 << 2)
+#define DRBD_ADM_NEED_PEER_DEVICE (1 << 3)
+#define DRBD_ADM_NEED_PEER_NODE (1 << 4)
+#define DRBD_ADM_IGNORE_VERSION (1 << 5)
+
+int drbd_adm_ctx_resolve(struct drbd_adm_ctx *ctx, unsigned int flags);
+void drbd_adm_ctx_release(struct drbd_adm_ctx *ctx);
+
+/* The netlink commands; each reports its outcome through ctx->result. */
+int drbd_adm_primary(struct drbd_adm_ctx *ctx);
+int drbd_adm_secondary(struct drbd_adm_ctx *ctx);
+int drbd_adm_disk_opts(struct drbd_adm_ctx *ctx);
+int drbd_adm_attach(struct drbd_adm_ctx *ctx);
+int drbd_adm_detach(struct drbd_adm_ctx *ctx);
+int drbd_adm_net_opts(struct drbd_adm_ctx *ctx);
+int drbd_adm_peer_device_opts(struct drbd_adm_ctx *ctx);
+int drbd_adm_connect(struct drbd_adm_ctx *ctx);
+int drbd_adm_new_peer(struct drbd_adm_ctx *ctx);
+int drbd_adm_new_path(struct drbd_adm_ctx *ctx);
+int drbd_adm_del_path(struct drbd_adm_ctx *ctx);
+int drbd_adm_disconnect(struct drbd_adm_ctx *ctx);
+int drbd_adm_del_peer(struct drbd_adm_ctx *ctx);
+int drbd_adm_resize(struct drbd_adm_ctx *ctx);
+int drbd_adm_resource_opts(struct drbd_adm_ctx *ctx);
+int drbd_adm_invalidate(struct drbd_adm_ctx *ctx);
+int drbd_adm_invalidate_peer(struct drbd_adm_ctx *ctx);
+int drbd_adm_pause_sync(struct drbd_adm_ctx *ctx);
+int drbd_adm_resume_sync(struct drbd_adm_ctx *ctx);
+int drbd_adm_suspend_io(struct drbd_adm_ctx *ctx);
+int drbd_adm_resume_io(struct drbd_adm_ctx *ctx);
+int drbd_adm_outdate(struct drbd_adm_ctx *ctx);
+int drbd_adm_get_timeout_type(struct drbd_adm_ctx *ctx);
+int drbd_adm_start_ov(struct drbd_adm_ctx *ctx);
+int drbd_adm_new_c_uuid(struct drbd_adm_ctx *ctx);
+int drbd_adm_new_resource(struct drbd_adm_ctx *ctx);
+int drbd_adm_new_minor(struct drbd_adm_ctx *ctx);
+int drbd_adm_del_minor(struct drbd_adm_ctx *ctx);
+int drbd_adm_down(struct drbd_adm_ctx *ctx);
+int drbd_adm_del_resource(struct drbd_adm_ctx *ctx);
+int drbd_adm_forget_peer(struct drbd_adm_ctx *ctx);
+int drbd_adm_rename_resource(struct drbd_adm_ctx *ctx);
+int drbd_adm_cbt_flush(struct drbd_adm_ctx *ctx);
+int drbd_adm_cbt_start(struct drbd_adm_ctx *ctx);
+int drbd_adm_cbt_stop(struct drbd_adm_ctx *ctx);
+int drbd_adm_cbt_status(struct drbd_adm_ctx *ctx);
+
+/*
+ * The dumps. The core walks the objects and keeps its cursor in
+ * cb->args[]; the dialect turns each object into a message. The
+ * optional resource-name filter is resolved by the dialect before it
+ * calls in: cb->args[0] then holds that resource with a reference
+ * (dropped again by the matching _done callback).
+ */
+enum { DRBD_DUMP_SINGLE_RESOURCE, DRBD_DUMP_ITERATE_RESOURCES };
+
+int drbd_dump_resources(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect);
+int drbd_dump_devices(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect);
+int drbd_dump_devices_done(struct netlink_callback *cb);
+int drbd_dump_connections(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect);
+int drbd_dump_connections_done(struct netlink_callback *cb);
+int drbd_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect);
+int drbd_dump_peer_devices_done(struct netlink_callback *cb);
+int drbd_dump_paths(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect);
+int drbd_dump_paths_done(struct netlink_callback *cb);
+int drbd_dump_initial_state(struct sk_buff *skb, struct netlink_callback *cb,
+ const struct drbd_nl_dialect *dialect, unsigned int seq);
+int drbd_dump_initial_state_done(struct netlink_callback *cb);
+
+/* Statistics of a live object, for the dumps and for the notifications. */
+void resource_to_statistics(struct drbd_resource_statistics *s, struct drbd_resource *resource);
+void device_to_statistics(struct drbd_device_statistics *s, struct drbd_device *device);
+void connection_to_statistics(struct drbd_connection_statistics *s,
+ struct drbd_connection *connection);
+void peer_device_to_statistics(struct drbd_peer_device_statistics *s,
+ struct drbd_peer_device *peer_device);
+
+/*
+ * Notifications. With "dialect" NULL the event is announced to every
+ * registered dialect under a freshly drawn sequence number; that is what
+ * the notify_*_state() wrappers in drbd_int.h do. A non-NULL dialect
+ * (and then always a non-NULL skb) is the initial state replay into the
+ * dump skb of that one dialect.
+ */
+int drbd_notify_resource_state(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *resource_info,
+ struct drbd_rename_resource_info *rename_resource_info,
+ enum drbd_notification_type type);
+int drbd_notify_device_state(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_device *device,
+ struct drbd_device_info *device_info,
+ enum drbd_notification_type type);
+int drbd_notify_connection_state(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_connection *connection,
+ struct drbd_connection_info *connection_info,
+ enum drbd_notification_type type);
+int drbd_notify_peer_device_state(struct sk_buff *skb, unsigned int seq,
+ const struct drbd_nl_dialect *dialect,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_info *peer_device_info,
+ enum drbd_notification_type type);
+
+#endif /* __DRBD_NL_H */
diff --git a/drivers/block/drbd/drbd_nl_defaults.c b/drivers/block/drbd/drbd_nl_defaults.c
new file mode 100644
index 000000000000..52ddd1249354
--- /dev/null
+++ b/drivers/block/drbd/drbd_nl_defaults.c
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+/*
+ * Default setters for the structs in drbd_nl_types.h, generated from
+ * drbd_genl_ynl.yaml by the YNL generator carried with the out-of-tree
+ * DRBD sources (drbd-headers, linux/generate.sh). The kernel's
+ * tools/net/ynl cannot regenerate this file.
+ */
+
+#include "drbd_nl_types.h"
+#include <linux/string.h>
+
+#include <linux/drbd.h>
+#include <linux/drbd_limits.h>
+
+void drbd_set_nl_cfg_context_defaults(struct drbd_nl_cfg_context *x)
+{
+ memset(x->ctx_conn_name, 0, sizeof(x->ctx_conn_name));
+ x->ctx_conn_name_len = 0;
+}
+
+void drbd_set_disk_conf_defaults(struct drbd_disk_conf *x)
+{
+ x->on_io_error = DRBD_ON_IO_ERROR_DEF;
+ x->resync_after = DRBD_MINOR_NUMBER_DEF;
+ x->al_extents = DRBD_AL_EXTENTS_DEF;
+ x->disk_barrier = DRBD_DISK_BARRIER_DEF;
+ x->disk_flushes = DRBD_DISK_FLUSHES_DEF;
+ x->disk_drain = DRBD_DISK_DRAIN_DEF;
+ x->md_flushes = DRBD_MD_FLUSHES_DEF;
+ x->disk_timeout = DRBD_DISK_TIMEOUT_DEF;
+ x->read_balancing = DRBD_READ_BALANCING_DEF;
+ x->unplug_watermark = DRBD_UNPLUG_WATERMARK_DEF;
+ x->al_updates = DRBD_AL_UPDATES_DEF;
+ x->discard_zeroes_if_aligned = DRBD_DISCARD_ZEROES_IF_ALIGNED_DEF;
+ x->rs_discard_granularity = DRBD_RS_DISCARD_GRANULARITY_DEF;
+ x->disable_write_same = DRBD_DISABLE_WRITE_SAME_DEF;
+ x->d_bitmap = DRBD_BITMAP_DEF;
+}
+
+void drbd_set_res_opts_defaults(struct drbd_res_opts *x)
+{
+ memset(x->cpu_mask, 0, sizeof(x->cpu_mask));
+ x->cpu_mask_len = 0;
+ x->on_no_data = DRBD_ON_NO_DATA_DEF;
+ x->auto_promote = DRBD_AUTO_PROMOTE_DEF;
+ x->peer_ack_window = DRBD_PEER_ACK_WINDOW_DEF;
+ x->twopc_timeout = DRBD_TWOPC_TIMEOUT_DEF;
+ x->twopc_retry_timeout = DRBD_TWOPC_RETRY_TIMEOUT_DEF;
+ x->peer_ack_delay = DRBD_PEER_ACK_DELAY_DEF;
+ x->auto_promote_timeout = DRBD_AUTO_PROMOTE_TIMEOUT_DEF;
+ x->nr_requests = DRBD_NR_REQUESTS_DEF;
+ x->quorum = DRBD_QUORUM_DEF;
+ x->on_no_quorum = DRBD_ON_NO_QUORUM_DEF;
+ x->quorum_min_redundancy = DRBD_QUORUM_DEF;
+ x->on_susp_primary_outdated = DRBD_ON_SUSP_PRI_OUTD_DEF;
+ x->drbd8_compat_mode = DRBD_DRBD8_COMPAT_MODE_DEF;
+ x->explicit_drbd8_compat = DRBD_DRBD8_COMPAT_MODE_DEF;
+}
+
+void drbd_set_net_conf_defaults(struct drbd_net_conf *x)
+{
+ memset(x->shared_secret, 0, sizeof(x->shared_secret));
+ x->shared_secret_len = 0;
+ memset(x->cram_hmac_alg, 0, sizeof(x->cram_hmac_alg));
+ x->cram_hmac_alg_len = 0;
+ memset(x->integrity_alg, 0, sizeof(x->integrity_alg));
+ x->integrity_alg_len = 0;
+ memset(x->verify_alg, 0, sizeof(x->verify_alg));
+ x->verify_alg_len = 0;
+ memset(x->csums_alg, 0, sizeof(x->csums_alg));
+ x->csums_alg_len = 0;
+ x->wire_protocol = DRBD_PROTOCOL_DEF;
+ x->connect_int = DRBD_CONNECT_INT_DEF;
+ x->timeout = DRBD_TIMEOUT_DEF;
+ x->ping_int = DRBD_PING_INT_DEF;
+ x->ping_timeo = DRBD_PING_TIMEO_DEF;
+ x->sndbuf_size = DRBD_SNDBUF_SIZE_DEF;
+ x->rcvbuf_size = DRBD_RCVBUF_SIZE_DEF;
+ x->ko_count = DRBD_KO_COUNT_DEF;
+ x->max_epoch_size = DRBD_MAX_EPOCH_SIZE_DEF;
+ x->after_sb_0p = DRBD_AFTER_SB_0P_DEF;
+ x->after_sb_1p = DRBD_AFTER_SB_1P_DEF;
+ x->after_sb_2p = DRBD_AFTER_SB_2P_DEF;
+ x->rr_conflict = DRBD_RR_CONFLICT_DEF;
+ x->on_congestion = DRBD_ON_CONGESTION_DEF;
+ x->cong_fill = DRBD_CONG_FILL_DEF;
+ x->cong_extents = DRBD_CONG_EXTENTS_DEF;
+ x->two_primaries = DRBD_ALLOW_TWO_PRIMARIES_DEF;
+ x->tcp_cork = DRBD_TCP_CORK_DEF;
+ x->always_asbp = DRBD_ALWAYS_ASBP_DEF;
+ x->use_rle = DRBD_USE_RLE_DEF;
+ x->fencing_policy = DRBD_FENCING_DEF;
+ memset(x->name, 0, sizeof(x->name));
+ x->name_len = 0;
+ x->csums_after_crash_only = DRBD_CSUMS_AFTER_CRASH_ONLY_DEF;
+ x->sock_check_timeo = DRBD_SOCKET_CHECK_TIMEO_DEF;
+ memset(x->transport_name, 0, sizeof(x->transport_name));
+ x->transport_name_len = 0;
+ x->max_buffers = DRBD_MAX_BUFFERS_DEF;
+ x->allow_remote_read = DRBD_ALLOW_REMOTE_READ_DEF;
+ x->tls = DRBD_TLS_DEF;
+ x->tls_privkey = DRBD_TLS_PRIVKEY_DEF;
+ x->tls_certificate = DRBD_TLS_CERTIFICATE_DEF;
+ x->tls_keyring = DRBD_TLS_KEYRING_DEF;
+ x->load_balance_paths = DRBD_LOAD_BALANCE_PATHS_DEF;
+ x->rdma_ctrl_rcvbuf_size = DRBD_RDMA_CTRL_RCVBUF_SIZE_DEF;
+ x->rdma_ctrl_sndbuf_size = DRBD_RDMA_CTRL_SNDBUF_SIZE_DEF;
+}
+
+void drbd_set_resize_parms_defaults(struct drbd_resize_parms *x)
+{
+ x->al_stripes = DRBD_AL_STRIPES_DEF;
+ x->al_stripe_size = DRBD_AL_STRIPE_SIZE_DEF;
+}
+
+void drbd_set_detach_parms_defaults(struct drbd_detach_parms *x)
+{
+ x->intentional_diskless_detach = DRBD_DISK_DISKLESS_DEF;
+}
+
+void drbd_set_device_conf_defaults(struct drbd_device_conf *x)
+{
+ x->max_bio_size = DRBD_MAX_BIO_SIZE_DEF;
+ x->intentional_diskless = DRBD_DISK_DISKLESS_DEF;
+ x->block_size = DRBD_BLOCK_SIZE_DEF;
+ x->discard_granularity = DRBD_DISCARD_GRANULARITY_DEF;
+}
+
+void drbd_set_invalidate_parms_defaults(struct drbd_invalidate_parms *x)
+{
+ x->sync_from_peer_node_id = DRBD_SYNC_FROM_NID_DEF;
+ x->reset_bitmap = DRBD_INVALIDATE_RESET_BITMAP_DEF;
+}
+
+void drbd_set_forget_peer_parms_defaults(struct drbd_forget_peer_parms *x)
+{
+ x->forget_peer_node_id = DRBD_SYNC_FROM_NID_DEF;
+}
+
+void drbd_set_peer_device_conf_defaults(struct drbd_peer_device_conf *x)
+{
+ x->resync_rate = DRBD_RESYNC_RATE_DEF;
+ x->c_plan_ahead = DRBD_C_PLAN_AHEAD_DEF;
+ x->c_delay_target = DRBD_C_DELAY_TARGET_DEF;
+ x->c_fill_target = DRBD_C_FILL_TARGET_DEF;
+ x->c_max_rate = DRBD_C_MAX_RATE_DEF;
+ x->c_min_rate = DRBD_C_MIN_RATE_DEF;
+ x->bitmap = DRBD_BITMAP_DEF;
+ x->resync_without_replication = DRBD_RESYNC_WITHOUT_REPLICATION_DEF;
+ x->peer_tiebreaker = DRBD_PEER_TIEBREAKER_DEF;
+}
+
+void drbd_set_connect_parms_defaults(struct drbd_connect_parms *x)
+{
+ x->tentative = 0;
+ x->discard_my_data = 0;
+}
+
+void drbd_set_invalidate_peer_parms_defaults(struct drbd_invalidate_peer_parms *x)
+{
+ x->p_reset_bitmap = DRBD_INVALIDATE_RESET_BITMAP_DEF;
+}
+
+void drbd_set_suspend_io_parms_defaults(struct drbd_suspend_io_parms *x)
+{
+ x->bdev_freeze = DRBD_SUSPEND_IO_BDEV_FREEZE_DEF;
+ x->flush_to_peers = DRBD_SUSPEND_IO_FLUSH_TO_PEERS_DEF;
+}
diff --git a/drivers/block/drbd/drbd_nl_types.h b/drivers/block/drbd/drbd_nl_types.h
new file mode 100644
index 000000000000..10e9f99babfe
--- /dev/null
+++ b/drivers/block/drbd/drbd_nl_types.h
@@ -0,0 +1,402 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Internal representation of DRBD configuration options, parameters,
+ * state information and statistics as exchanged over generic netlink.
+ *
+ * These structs are shared by every netlink dialect the kernel module
+ * serves (the legacy "drbd" family and "drbd2"); the per-dialect
+ * marshalling code translates between them and the wire format. They
+ * used to be generated from linux/drbd_genl_ynl.yaml; keep field names
+ * and types in sync with the specs when adding attributes.
+ */
+#ifndef __DRBD_NL_TYPES_H
+#define __DRBD_NL_TYPES_H
+
+#include <linux/types.h>
+#include <linux/drbd.h>
+
+struct drbd_nl_cfg_reply {
+ char info_text[0];
+ __u32 info_text_len;
+};
+
+struct drbd_nl_cfg_context {
+ __u32 ctx_peer_node_id;
+ __u32 ctx_volume;
+ char ctx_resource_name[128];
+ __u32 ctx_resource_name_len;
+ char ctx_my_addr[128];
+ __u32 ctx_my_addr_len;
+ char ctx_peer_addr[128];
+ __u32 ctx_peer_addr_len;
+ char ctx_conn_name[SHARED_SECRET_MAX];
+ __u32 ctx_conn_name_len;
+};
+
+struct drbd_disk_conf {
+ char backing_dev[128];
+ __u32 backing_dev_len;
+ char meta_dev[128];
+ __u32 meta_dev_len;
+ __s32 meta_dev_idx;
+ __u64 disk_size;
+ __u32 on_io_error;
+ __s32 resync_after;
+ __u32 al_extents;
+ unsigned char disk_barrier;
+ unsigned char disk_flushes;
+ unsigned char disk_drain;
+ unsigned char md_flushes;
+ __u32 disk_timeout;
+ __u32 read_balancing;
+ __u32 unplug_watermark;
+ __u32 rs_discard_granularity;
+ unsigned char al_updates;
+ unsigned char discard_zeroes_if_aligned;
+ unsigned char disable_write_same;
+ unsigned char d_bitmap;
+};
+
+struct drbd_res_opts {
+ char cpu_mask[DRBD_CPU_MASK_SIZE];
+ __u32 cpu_mask_len;
+ __u32 on_no_data;
+ unsigned char auto_promote;
+ __u32 node_id;
+ __u32 peer_ack_window;
+ __u32 twopc_timeout;
+ __u32 twopc_retry_timeout;
+ __u32 peer_ack_delay;
+ __u32 auto_promote_timeout;
+ __u32 nr_requests;
+ __s32 quorum;
+ __u32 on_no_quorum;
+ __s32 quorum_min_redundancy;
+ __u32 on_susp_primary_outdated;
+ unsigned char drbd8_compat_mode;
+ unsigned char explicit_drbd8_compat;
+};
+
+struct drbd_net_conf {
+ char shared_secret[SHARED_SECRET_MAX];
+ __u32 shared_secret_len;
+ char cram_hmac_alg[SHARED_SECRET_MAX];
+ __u32 cram_hmac_alg_len;
+ char integrity_alg[SHARED_SECRET_MAX];
+ __u32 integrity_alg_len;
+ char verify_alg[SHARED_SECRET_MAX];
+ __u32 verify_alg_len;
+ char csums_alg[SHARED_SECRET_MAX];
+ __u32 csums_alg_len;
+ __u32 wire_protocol;
+ __u32 connect_int;
+ __u32 timeout;
+ __u32 ping_int;
+ __u32 ping_timeo;
+ __u32 sndbuf_size;
+ __u32 rcvbuf_size;
+ __u32 ko_count;
+ __u32 max_epoch_size;
+ __u32 after_sb_0p;
+ __u32 after_sb_1p;
+ __u32 after_sb_2p;
+ __u32 rr_conflict;
+ __u32 on_congestion;
+ __u32 cong_fill;
+ __u32 cong_extents;
+ unsigned char two_primaries;
+ unsigned char tcp_cork;
+ unsigned char always_asbp;
+ unsigned char use_rle;
+ __u32 fencing_policy;
+ char name[SHARED_SECRET_MAX];
+ __u32 name_len;
+ unsigned char csums_after_crash_only;
+ __u32 sock_check_timeo;
+ char transport_name[SHARED_SECRET_MAX];
+ __u32 transport_name_len;
+ __u32 max_buffers;
+ unsigned char allow_remote_read;
+ unsigned char tls;
+ __s32 tls_privkey;
+ __s32 tls_certificate;
+ __s32 tls_keyring;
+ unsigned char load_balance_paths;
+ __u32 rdma_ctrl_rcvbuf_size;
+ __u32 rdma_ctrl_sndbuf_size;
+};
+
+struct drbd_set_role_parms {
+ unsigned char force;
+};
+
+struct drbd_resize_parms {
+ __u64 resize_size;
+ unsigned char resize_force;
+ unsigned char no_resync;
+ __u32 al_stripes;
+ __u32 al_stripe_size;
+};
+
+struct drbd_start_ov_parms {
+ __u64 ov_start_sector;
+ __u64 ov_stop_sector;
+};
+
+struct drbd_new_c_uuid_parms {
+ unsigned char clear_bm;
+ unsigned char force_resync;
+};
+
+struct drbd_timeout_parms {
+ __u32 timeout_type;
+};
+
+struct drbd_disconnect_parms {
+ unsigned char force_disconnect;
+};
+
+struct drbd_detach_parms {
+ unsigned char force_detach;
+ unsigned char intentional_diskless_detach;
+};
+
+struct drbd_device_conf {
+ __u32 max_bio_size;
+ unsigned char intentional_diskless;
+ __u32 block_size;
+ __u32 discard_granularity;
+};
+
+struct drbd_resource_info {
+ __u32 res_role;
+ unsigned char res_susp;
+ unsigned char res_susp_nod;
+ unsigned char res_susp_fen;
+ unsigned char res_susp_quorum;
+ unsigned char res_fail_io;
+#ifdef CONFIG_DRBD_COMPAT_84
+ /*
+ * The pre-transition values of the four fields above, for the v1
+ * dialect's fused DRBD_EVENT (drbd_nl_84.c's compat84_emit_event()):
+ * 8.4's ST-prev/ST-new event lines need a real "before" state, not
+ * just the "after" state every other field here already carries.
+ * Filled only at notify_resource_state_change()'s NOTIFY_CHANGE call
+ * site (drbd_state.c); left unset (and unread) everywhere else,
+ * including every notify_resource_state() call the v1 dialect never
+ * sees a fused event for.
+ */
+ __u32 old_res_role;
+ unsigned char old_res_susp;
+ unsigned char old_res_susp_nod;
+ unsigned char old_res_susp_fen;
+#endif
+};
+
+struct drbd_device_info {
+ __u32 dev_disk_state;
+ unsigned char is_intentional_diskless;
+ unsigned char dev_has_quorum;
+ unsigned char dev_is_open;
+ char backing_dev_path[128];
+ __u32 backing_dev_path_len;
+#ifdef CONFIG_DRBD_COMPAT_84
+ /* dev_disk_state's pre-transition value; see resource_info above. */
+ __u32 old_dev_disk_state;
+#endif
+};
+
+struct drbd_connection_info {
+ __u32 conn_connection_state;
+ __u32 conn_role;
+#ifdef CONFIG_DRBD_COMPAT_84
+ /* Pre-transition values of the two fields above; see resource_info
+ * above.
+ */
+ __u32 old_conn_connection_state;
+ __u32 old_conn_role;
+#endif
+};
+
+struct drbd_peer_device_info {
+ __u32 peer_repl_state;
+ __u32 peer_disk_state;
+ __u32 peer_resync_susp_user;
+ __u32 peer_resync_susp_peer;
+ __u32 peer_resync_susp_dependency;
+ unsigned char peer_is_intentional_diskless;
+ __u32 peer_resync_susp_max_parallel;
+#ifdef CONFIG_DRBD_COMPAT_84
+ /* peer_repl_state's and peer_disk_state's pre-transition values; see
+ * resource_info above. old_peer_resync_susp_user/_peer are the exact
+ * pre-transition values of the two fields above them; old_peer_
+ * resync_susp_dependency approximates peer_resync_susp_dependency's
+ * combined derivation (resync_susp_dependency || resync_susp_other_c
+ * || (sync-source with an inconsistent local disk)) with only its
+ * first two terms, since the third needs the sibling device's
+ * pre-transition disk state, not reachable from a peer-device-only
+ * snapshot without widening the notification callback's signature.
+ */
+ __u32 old_peer_repl_state;
+ __u32 old_peer_disk_state;
+ __u32 old_peer_resync_susp_user;
+ __u32 old_peer_resync_susp_peer;
+ __u32 old_peer_resync_susp_dependency;
+#endif
+};
+
+struct drbd_resource_statistics {
+ __u32 res_stat_write_ordering;
+};
+
+struct drbd_device_statistics {
+ __u64 dev_size;
+ __u64 dev_read;
+ __u64 dev_write;
+ __u64 dev_al_writes;
+ __u64 dev_bm_writes;
+ __u32 dev_upper_pending;
+ __u32 dev_lower_pending;
+ unsigned char dev_upper_blocked;
+ unsigned char dev_lower_blocked;
+ unsigned char dev_al_suspended;
+ __u64 dev_exposed_data_uuid;
+ __u64 dev_current_uuid;
+ __u32 dev_disk_flags;
+ char history_uuids[HISTORY_UUIDS_SIZE];
+ __u32 history_uuids_len;
+};
+
+struct drbd_connection_statistics {
+ unsigned char conn_congested;
+ __u64 ap_in_flight;
+ __u64 rs_in_flight;
+};
+
+struct drbd_peer_device_statistics {
+ __u64 peer_dev_received;
+ __u64 peer_dev_sent;
+ __u32 peer_dev_pending;
+ __u32 peer_dev_unacked;
+ __u64 peer_dev_out_of_sync;
+ __u64 peer_dev_resync_failed;
+ __u64 peer_dev_bitmap_uuid;
+ __u32 peer_dev_flags;
+ __u64 peer_dev_rs_total;
+ __u64 peer_dev_ov_start_sector;
+ __u64 peer_dev_ov_stop_sector;
+ __u64 peer_dev_ov_position;
+ __u64 peer_dev_ov_left;
+ __u64 peer_dev_ov_skipped;
+ __u64 peer_dev_rs_same_csum;
+ __u64 peer_dev_rs_dt_start_ms;
+ __u64 peer_dev_rs_paused_ms;
+ __u64 peer_dev_rs_dt0_ms;
+ __u64 peer_dev_rs_db0_sectors;
+ __u64 peer_dev_rs_dt1_ms;
+ __u64 peer_dev_rs_db1_sectors;
+ __u32 peer_dev_rs_c_sync_rate;
+ __u64 peer_dev_uuid_flags;
+};
+
+struct drbd_nl_notification_header {
+ __u32 nh_type;
+};
+
+struct drbd_nl_helper_info {
+ char helper_name[32];
+ __u32 helper_name_len;
+ __u32 helper_status;
+};
+
+struct drbd_invalidate_parms {
+ __s32 sync_from_peer_node_id;
+ unsigned char reset_bitmap;
+};
+
+struct drbd_forget_peer_parms {
+ __s32 forget_peer_node_id;
+};
+
+struct drbd_peer_device_conf {
+ __u32 resync_rate;
+ __u32 c_plan_ahead;
+ __u32 c_delay_target;
+ __u32 c_fill_target;
+ __u32 c_max_rate;
+ __u32 c_min_rate;
+ unsigned char bitmap;
+ unsigned char resync_without_replication;
+ unsigned char peer_tiebreaker;
+};
+
+struct drbd_path_parms {
+ char my_addr[128];
+ __u32 my_addr_len;
+ char peer_addr[128];
+ __u32 peer_addr_len;
+};
+
+struct drbd_connect_parms {
+ unsigned char tentative;
+ unsigned char discard_my_data;
+};
+
+struct drbd_nl_path_info {
+ unsigned char path_established;
+};
+
+struct drbd_rename_resource_parms {
+ char new_resource_name[128];
+ __u32 new_resource_name_len;
+};
+
+struct drbd_rename_resource_info {
+ char res_new_name[128];
+ __u32 res_new_name_len;
+};
+
+struct drbd_invalidate_peer_parms {
+ unsigned char p_reset_bitmap;
+};
+
+struct drbd_suspend_io_parms {
+ unsigned char bdev_freeze;
+ unsigned char flush_to_peers;
+};
+
+struct drbd_cbt_start_parms {
+ __u64 cbt_id;
+};
+
+struct drbd_cbt_info {
+ unsigned char cbt_active;
+ unsigned char cbt_valid;
+ __u64 cbt_bookmark_id;
+ __u64 cbt_since;
+ __u64 cbt_bitmap_uuid;
+ __u64 cbt_bitmap_dagtag;
+};
+
+/*
+ * Neutral config defaults, shared by every dialect: default-value setters
+ * for the structs above that have optional fields. The core calls these
+ * before parsing a partial request, regardless of which dialect received
+ * it; the definitions live in linux/drbd_nl_defaults.c, which is always
+ * built into the module regardless of dialect configuration.
+ */
+void drbd_set_nl_cfg_context_defaults(struct drbd_nl_cfg_context *x);
+void drbd_set_disk_conf_defaults(struct drbd_disk_conf *x);
+void drbd_set_res_opts_defaults(struct drbd_res_opts *x);
+void drbd_set_net_conf_defaults(struct drbd_net_conf *x);
+void drbd_set_resize_parms_defaults(struct drbd_resize_parms *x);
+void drbd_set_detach_parms_defaults(struct drbd_detach_parms *x);
+void drbd_set_device_conf_defaults(struct drbd_device_conf *x);
+void drbd_set_invalidate_parms_defaults(struct drbd_invalidate_parms *x);
+void drbd_set_forget_peer_parms_defaults(struct drbd_forget_peer_parms *x);
+void drbd_set_peer_device_conf_defaults(struct drbd_peer_device_conf *x);
+void drbd_set_connect_parms_defaults(struct drbd_connect_parms *x);
+void drbd_set_invalidate_peer_parms_defaults(struct drbd_invalidate_peer_parms *x);
+void drbd_set_suspend_io_parms_defaults(struct drbd_suspend_io_parms *x);
+
+#endif /* __DRBD_NL_TYPES_H */
diff --git a/include/uapi/linux/drbd.h b/include/uapi/linux/drbd.h
index b7d6b1c52df0..0eaa02a4f3b2 100644
--- a/include/uapi/linux/drbd.h
+++ b/include/uapi/linux/drbd.h
@@ -1,10 +1,11 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later WITH Linux-syscall-note */
+/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
/*
* Copyright (C) 2014, LINBIT HA-Solutions GmbH.
*/
-#ifndef _UAPI_LINUX_DRBD_H
-#define _UAPI_LINUX_DRBD_H
+#ifndef DRBD_H
+#define DRBD_H
+
#include <linux/types.h>
#include <asm/byteorder.h>
@@ -14,8 +15,7 @@ enum drbd_io_error_p {
EP_DETACH
};
-enum drbd_fencing_p {
- FP_NOT_AVAIL = -1, /* Not a policy */
+enum drbd_fencing_policy {
FP_DONT_CARE = 0,
FP_RESOURCE,
FP_STONITH
@@ -38,7 +38,9 @@ enum drbd_after_sb_p {
ASB_CONSENSUS,
ASB_DISCARD_SECONDARY,
ASB_CALL_HELPER,
- ASB_VIOLENTLY
+ ASB_VIOLENTLY,
+ ASB_RETRY_CONNECT,
+ ASB_AUTO_DISCARD,
};
enum drbd_on_no_data {
@@ -46,6 +48,16 @@ enum drbd_on_no_data {
OND_SUSPEND_IO
};
+enum drbd_on_no_quorum {
+ ONQ_IO_ERROR = OND_IO_ERROR,
+ ONQ_SUSPEND_IO = OND_SUSPEND_IO
+};
+
+enum drbd_on_susp_primary_outdated {
+ SPO_DISCONNECT,
+ SPO_FORCE_SECONDARY,
+};
+
enum drbd_on_congestion {
OC_BLOCK,
OC_PULL_AHEAD,
@@ -66,6 +78,11 @@ enum drbd_read_balancing {
RB_1M_STRIPING,
};
+/* Windows km/dderror.h has that a 0L */
+#ifdef NO_ERROR
+#undef NO_ERROR
+#endif
+
/* KEEP the order, do not delete or insert. Only append. */
enum drbd_ret_code {
ERR_CODE_BASE = 100,
@@ -109,7 +126,7 @@ enum drbd_ret_code {
ERR_CSUMS_ALG_ND = 145, /* DRBD 8.2 only */
ERR_VERIFY_ALG = 146, /* DRBD 8.2 only */
ERR_VERIFY_ALG_ND = 147, /* DRBD 8.2 only */
- ERR_CSUMS_RESYNC_RUNNING= 148, /* DRBD 8.2 only */
+ ERR_CSUMS_RESYNC_RUNNING = 148, /* DRBD 8.2 only */
ERR_VERIFY_RUNNING = 149, /* DRBD 8.2 only */
ERR_DATA_NOT_CURRENT = 150,
ERR_CONNECTED = 151, /* DRBD 8.3 only */
@@ -132,6 +149,13 @@ enum drbd_ret_code {
ERR_MD_LAYOUT_TOO_SMALL = 168,
ERR_MD_LAYOUT_NO_FIT = 169,
ERR_IMPLICIT_SHRINK = 170,
+ ERR_INVALID_PEER_NODE_ID = 171,
+ ERR_CREATE_TRANSPORT = 172,
+ ERR_LOCAL_AND_PEER_ADDR = 173,
+ ERR_ALREADY_EXISTS = 174,
+ ERR_APV_TOO_LOW = 175,
+ ERR_PATH_COLLISION = 176,
+
/* insert new ones above this line */
AFTER_LAST_ERR_CODE
};
@@ -148,54 +172,65 @@ enum drbd_role {
};
/* The order of these constants is important.
- * The lower ones (<C_WF_REPORT_PARAMS) indicate
+ * The lower ones (< C_CONNECTED) indicate
* that there is no socket!
- * >=C_WF_REPORT_PARAMS ==> There is a socket
+ * >= C_CONNECTED ==> There is a socket
*/
-enum drbd_conns {
+enum drbd_conn_state {
C_STANDALONE,
- C_DISCONNECTING, /* Temporal state on the way to StandAlone. */
+ C_DISCONNECTING, /* Temporary state on the way to C_STANDALONE. */
C_UNCONNECTED, /* >= C_UNCONNECTED -> inc_net() succeeds */
- /* These temporal states are all used on the way
- * from >= C_CONNECTED to Unconnected.
+ /* These temporary states are used on the way
+ * from C_CONNECTED to C_UNCONNECTED.
* The 'disconnect reason' states
- * I do not allow to change between them. */
+ * I do not allow to change between them.
+ */
C_TIMEOUT,
C_BROKEN_PIPE,
C_NETWORK_FAILURE,
C_PROTOCOL_ERROR,
C_TEAR_DOWN,
- C_WF_CONNECTION,
- C_WF_REPORT_PARAMS, /* we have a socket */
- C_CONNECTED, /* we have introduced each other */
- C_STARTING_SYNC_S, /* starting full sync by admin request. */
- C_STARTING_SYNC_T, /* starting full sync by admin request. */
- C_WF_BITMAP_S,
- C_WF_BITMAP_T,
- C_WF_SYNC_UUID,
+ C_CONNECTING,
+
+ C_CONNECTED, /* we have a socket */
+
+ C_MASK = 31,
+};
+
+enum drbd_repl_state {
+ L_NEGOTIATING = C_CONNECTED, /* used for peer_device->negotiation_result only */
+ L_OFF = C_CONNECTED,
+
+ L_ESTABLISHED, /* we have introduced each other */
+ L_STARTING_SYNC_S, /* starting full sync by admin request. */
+ L_STARTING_SYNC_T, /* starting full sync by admin request. */
+ L_WF_BITMAP_S,
+ L_WF_BITMAP_T,
+ L_WF_SYNC_UUID,
/* All SyncStates are tested with this comparison
- * xx >= C_SYNC_SOURCE && xx <= C_PAUSED_SYNC_T */
- C_SYNC_SOURCE,
- C_SYNC_TARGET,
- C_VERIFY_S,
- C_VERIFY_T,
- C_PAUSED_SYNC_S,
- C_PAUSED_SYNC_T,
-
- C_AHEAD,
- C_BEHIND,
-
- C_MASK = 31
+ * xx >= L_SYNC_SOURCE && xx <= L_PAUSED_SYNC_T
+ */
+ L_SYNC_SOURCE,
+ L_SYNC_TARGET,
+ L_VERIFY_S,
+ L_VERIFY_T,
+ L_PAUSED_SYNC_S,
+ L_PAUSED_SYNC_T,
+
+ L_AHEAD,
+ L_BEHIND,
+ L_NEG_NO_RESULT = L_BEHIND, /* used for peer_device->negotiation_result only */
};
enum drbd_disk_state {
D_DISKLESS,
D_ATTACHING, /* In the process of reading the meta-data */
+ D_DETACHING, /* Added in protocol version 110 */
D_FAILED, /* Becomes D_DISKLESS as soon as we told it the peer */
- /* when >= D_FAILED it is legal to access mdev->ldev */
+ /* when >= D_FAILED it is legal to access device->ldev */
D_NEGOTIATING, /* Late attaching state, we need to talk to the peer */
D_INCONSISTENT,
D_OUTDATED,
@@ -216,31 +251,33 @@ union drbd_state {
*/
struct {
#if defined(__LITTLE_ENDIAN_BITFIELD)
- unsigned role:2 ; /* 3/4 primary/secondary/unknown */
- unsigned peer:2 ; /* 3/4 primary/secondary/unknown */
- unsigned conn:5 ; /* 17/32 cstates */
- unsigned disk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned pdsk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned susp:1 ; /* 2/2 IO suspended no/yes (by user) */
- unsigned aftr_isp:1 ; /* isp .. imposed sync pause */
- unsigned peer_isp:1 ;
- unsigned user_isp:1 ;
- unsigned susp_nod:1 ; /* IO suspended because no data */
- unsigned susp_fen:1 ; /* IO suspended because fence peer handler runs*/
- unsigned _pad:9; /* 0 unused */
+ unsigned role:2; /* 3/4 primary/secondary/unknown */
+ unsigned peer:2; /* 3/4 primary/secondary/unknown */
+ unsigned conn:5; /* 17/32 cstates */
+ unsigned disk:4; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
+ unsigned pdsk:4; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
+ unsigned susp:1; /* 2/2 IO suspended no/yes (by user) */
+ unsigned aftr_isp:1; /* isp .. imposed sync pause */
+ unsigned peer_isp:1;
+ unsigned user_isp:1;
+ unsigned susp_nod:1; /* IO suspended because no data */
+ unsigned susp_fen:1; /* IO suspended because fence peer handler runs*/
+ unsigned quorum:1;
+ unsigned _pad:8; /* 0 unused */
#elif defined(__BIG_ENDIAN_BITFIELD)
- unsigned _pad:9;
- unsigned susp_fen:1 ;
- unsigned susp_nod:1 ;
- unsigned user_isp:1 ;
- unsigned peer_isp:1 ;
- unsigned aftr_isp:1 ; /* isp .. imposed sync pause */
- unsigned susp:1 ; /* 2/2 IO suspended no/yes */
- unsigned pdsk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned disk:4 ; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
- unsigned conn:5 ; /* 17/32 cstates */
- unsigned peer:2 ; /* 3/4 primary/secondary/unknown */
- unsigned role:2 ; /* 3/4 primary/secondary/unknown */
+ unsigned _pad:8;
+ unsigned quorum:1;
+ unsigned susp_fen:1;
+ unsigned susp_nod:1;
+ unsigned user_isp:1;
+ unsigned peer_isp:1;
+ unsigned aftr_isp:1; /* isp .. imposed sync pause */
+ unsigned susp:1; /* 2/2 IO suspended no/yes */
+ unsigned pdsk:4; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
+ unsigned disk:4; /* 8/16 from D_DISKLESS to D_UP_TO_DATE */
+ unsigned conn:5; /* 17/32 cstates */
+ unsigned peer:2; /* 3/4 primary/secondary/unknown */
+ unsigned role:2; /* 3/4 primary/secondary/unknown */
#else
# error "this endianness is not supported"
#endif
@@ -267,29 +304,76 @@ enum drbd_state_rv {
SS_DEVICE_IN_USE = -12,
SS_NO_NET_CONFIG = -13,
SS_NO_VERIFY_ALG = -14, /* drbd-8.2 only */
- SS_NEED_CONNECTION = -15, /* drbd-8.2 only */
+ SS_NEED_CONNECTION = -15, /* Need connection this state change affects to be connected */
SS_LOWER_THAN_OUTDATED = -16,
- SS_NOT_SUPPORTED = -17, /* drbd-8.2 only */
+ SS_NOT_SUPPORTED = -17,
SS_IN_TRANSIENT_STATE = -18, /* Retry after the next state change */
SS_CONCURRENT_ST_CHG = -19, /* Concurrent cluster side state change! */
SS_O_VOL_PEER_PRI = -20,
- SS_OUTDATE_WO_CONN = -21,
- SS_AFTER_LAST_ERROR = -22, /* Keep this at bottom */
+ SS_INTERRUPTED = -21, /* interrupted in stable_state_change() */
+ SS_PRIMARY_READER = -22,
+ SS_TIMEOUT = -23,
+ SS_WEAKLY_CONNECTED = -24,
+ SS_NO_QUORUM = -25,
+ SS_ATTACH_NO_BITMAP = -26,
+ SS_HANDSHAKE_DISCONNECT = -27,
+ SS_HANDSHAKE_RETRY = -28,
+ SS_AFTER_LAST_ERROR = -29, /* Keep this at bottom */
};
#define SHARED_SECRET_MAX 64
-#define MDF_CONSISTENT (1 << 0)
-#define MDF_PRIMARY_IND (1 << 1)
-#define MDF_CONNECTED_IND (1 << 2)
-#define MDF_FULL_SYNC (1 << 3)
-#define MDF_WAS_UP_TO_DATE (1 << 4)
-#define MDF_PEER_OUT_DATED (1 << 5)
-#define MDF_CRASHED_PRIMARY (1 << 6)
-#define MDF_AL_CLEAN (1 << 7)
-#define MDF_AL_DISABLED (1 << 8)
+/* Meta data feature flags */
+#define DRBD_MDFF_DIVERGENCE_BITMAP (1ULL << 0)
+#define DRBD_MDFF_BITMAP_AUTHORITATIVE (1ULL << 1) /* MDF_PEER_BITMAP_AUTHORITATIVE is maintained */
+#define DRBD_MDFF_CBT (1ULL << 2) /* struct cbt_bookmark_on_disk_9 is maintained */
+
+/* Flags of the changed block tracking bookmark in the meta data */
+#define CBT_BOOKMARK_IN_USE (1U << 0)
+#define CBT_BOOKMARK_INVALID (1U << 1)
+#define CBT_BOOKMARK_FLUSHED (1U << 2) /* the bitmap on disk holds every write up to now */
+
+enum mdf_flag {
+ MDF_CONSISTENT = 1 << 0,
+ MDF_PRIMARY_IND = 1 << 1,
+ MDF_WAS_UP_TO_DATE = 1 << 4,
+ MDF_CRASHED_PRIMARY = 1 << 6,
+ MDF_AL_CLEAN = 1 << 7,
+ MDF_AL_DISABLED = 1 << 8,
+ MDF_PRIMARY_LOST_QUORUM = 1 << 9,
+ MDF_HAVE_QUORUM = 1 << 10,
+};
+
+/* Bit numbers, for the atomic bit operations on struct drbd_peer_md flags.
+ * Maximum value 31 because the flags are persisted as be32.
+ */
+enum mdf_peer_flag_bit {
+ __MDF_PEER_CONNECTED = 0,
+ __MDF_PEER_OUTDATED = 1,
+ __MDF_PEER_FENCING = 2,
+ __MDF_PEER_FULL_SYNC = 3,
+ __MDF_PEER_DEVICE_SEEN = 4,
+ __MDF_PEER_DIVERGENCE_BITMAP = 5, /* bitmap fully records divergence; safe to copy from */
+ __MDF_PEER_BITMAP_AUTHORITATIVE = 6, /* out-of-sync bits were set for blocks the peer lacks, not by a resync or an invalidate */
+ __MDF_NODE_EXISTS = 16,
+ __MDF_HAVE_BITMAP = 31, /* For in core use; no meaning when persisted */
+};
+
+/* Masks, for the on-disk and netlink representations. */
+enum mdf_peer_flag {
+ MDF_PEER_CONNECTED = 1U << __MDF_PEER_CONNECTED,
+ MDF_PEER_OUTDATED = 1U << __MDF_PEER_OUTDATED,
+ MDF_PEER_FENCING = 1U << __MDF_PEER_FENCING,
+ MDF_PEER_FULL_SYNC = 1U << __MDF_PEER_FULL_SYNC,
+ MDF_PEER_DEVICE_SEEN = 1U << __MDF_PEER_DEVICE_SEEN,
+ MDF_PEER_DIVERGENCE_BITMAP = 1U << __MDF_PEER_DIVERGENCE_BITMAP,
+ MDF_PEER_BITMAP_AUTHORITATIVE = 1U << __MDF_PEER_BITMAP_AUTHORITATIVE,
+ MDF_NODE_EXISTS = 1U << __MDF_NODE_EXISTS,
+ MDF_HAVE_BITMAP = 1U << __MDF_HAVE_BITMAP,
+};
-#define MAX_PEERS 32
+#define DRBD_PEERS_MAX 32
+#define DRBD_NODE_ID_MAX DRBD_PEERS_MAX
enum drbd_uuid_index {
UI_CURRENT,
@@ -301,10 +385,17 @@ enum drbd_uuid_index {
UI_EXTENDED_SIZE /* Everything. */
};
-#define HISTORY_UUIDS MAX_PEERS
+#define HISTORY_UUIDS_V08 (UI_HISTORY_END - UI_HISTORY_START + 1)
+#define HISTORY_UUIDS DRBD_PEERS_MAX
+#define HISTORY_UUIDS_SIZE (HISTORY_UUIDS * sizeof(__u64))
+/*
+ * Wire sizes of the uuid attributes of the "drbd" netlink family at
+ * version 1, as served by drbd_nl_84.c. Same values as the 8.4 driver's:
+ * UI_SIZE and HISTORY_UUIDS are identical in 8.4 and 9.
+ */
#define DRBD_NL_UUIDS_SIZE (UI_SIZE * sizeof(__u64))
-#define DRBD_NL_HISTORY_UUIDS_SIZE (HISTORY_UUIDS * sizeof(__u64))
+#define DRBD_NL_HISTORY_UUIDS_SIZE HISTORY_UUIDS_SIZE
enum drbd_timeout_flag {
UT_DEFAULT = 0,
@@ -312,6 +403,16 @@ enum drbd_timeout_flag {
UT_PEER_OUTDATED = 2,
};
+#define UUID_JUST_CREATED ((__u64)4)
+#define UUID_PRIMARY ((__u64)1)
+
+enum write_ordering_e {
+ WO_NONE,
+ WO_DRAIN_IO,
+ WO_BDEV_FLUSH,
+ WO_BIO_BARRIER
+};
+
enum drbd_notification_type {
NOTIFY_EXISTS,
NOTIFY_CREATE,
@@ -319,11 +420,13 @@ enum drbd_notification_type {
NOTIFY_DESTROY,
NOTIFY_CALL,
NOTIFY_RESPONSE,
+ NOTIFY_RENAME,
NOTIFY_CONTINUES = 0x8000,
NOTIFY_FLAGS = NOTIFY_CONTINUES,
};
+/* These values are part of the ABI! */
enum drbd_peer_state {
P_INCONSISTENT = 3,
P_OUTDATED = 4,
@@ -332,15 +435,6 @@ enum drbd_peer_state {
P_FENCING = 7,
};
-#define UUID_JUST_CREATED ((__u64)4)
-
-enum write_ordering_e {
- WO_NONE,
- WO_DRAIN_IO,
- WO_BDEV_FLUSH,
- WO_BIO_BARRIER
-};
-
/* magic numbers used in meta data and network packets */
#define DRBD_MAGIC 0x83740267
#define DRBD_MAGIC_BIG 0x835a
@@ -349,18 +443,24 @@ enum write_ordering_e {
#define DRBD_MD_MAGIC_07 (DRBD_MAGIC+3)
#define DRBD_MD_MAGIC_08 (DRBD_MAGIC+4)
#define DRBD_MD_MAGIC_84_UNCLEAN (DRBD_MAGIC+5)
-
-
-/* how I came up with this magic?
- * base64 decode "actlog==" ;) */
-#define DRBD_AL_MAGIC 0x69cb65a2
+#define DRBD_MD_MAGIC_09 (DRBD_MAGIC+6)
/* these are of type "int" */
#define DRBD_MD_INDEX_INTERNAL -1
#define DRBD_MD_INDEX_FLEX_EXT -2
#define DRBD_MD_INDEX_FLEX_INT -3
-#define DRBD_CPU_MASK_SIZE 32
+/*
+ * This is the maximum string length accepted by drbdadm.
+ * It allows a full mask for up to 908 CPUs.
+ */
+#define DRBD_CPU_MASK_SIZE 256
+
+#define DRBD_MAX_BIO_SIZE (1U << 20)
+
+#define QOU_OFF 0
+#define QOU_MAJORITY 1024
+#define QOU_ALL 1025
/**
* struct drbd_genlmsghdr - DRBD specific header used in NETLINK_GENERIC requests
@@ -373,19 +473,13 @@ enum write_ordering_e {
* is used instead.
* @flags: possible operation modifiers (relevant only for user->kernel):
* DRBD_GENL_F_SET_DEFAULTS
- * @volume:
- * When creating a new minor (adding it to a resource), the resource needs
- * to know which volume number within the resource this is supposed to be.
- * The volume number corresponds to the same volume number on the remote side,
- * whereas the minor number on the remote side may be different
- * (union with flags).
* @ret_code: kernel->userland unicast cfg reply return code (union with flags);
*/
struct drbd_genlmsghdr {
__u32 minor;
union {
- __u32 flags;
- __s32 ret_code;
+ __u32 flags;
+ __s32 ret_code;
};
};
@@ -394,12 +488,11 @@ enum {
DRBD_GENL_F_SET_DEFAULTS = 1,
};
-enum drbd_state_info_bcast_reason {
- SIB_GET_STATUS_REPLY = 1,
- SIB_STATE_CHANGE = 2,
- SIB_HELPER_PRE = 3,
- SIB_HELPER_POST = 4,
- SIB_SYNC_PROGRESS = 5,
-};
+/*
+ * DRBD_GENLA_F_MANDATORY: netlink ignores attributes it does not know
+ * about by default. This flag in nlattr->nla_type indicates that this
+ * attribute must not be ignored. Checked and stripped in pre_doit.
+ */
+#define DRBD_GENLA_F_MANDATORY (1 << 14)
-#endif /* _UAPI_LINUX_DRBD_H */
+#endif
diff --git a/include/uapi/linux/drbd_limits.h b/include/uapi/linux/drbd_limits.h
index bba0a0aa8792..f8d56ba2207c 100644
--- a/include/uapi/linux/drbd_limits.h
+++ b/include/uapi/linux/drbd_limits.h
@@ -10,10 +10,10 @@
* feedback about nonsense settings for certain configurable values.
*/
-#ifndef _UAPI_LINUX_DRBD_LIMITS_H
-#define _UAPI_LINUX_DRBD_LIMITS_H
+#ifndef DRBD_LIMITS_H
+#define DRBD_LIMITS_H 1
-#include <linux/drbd.h>
+#define DEBUG_RANGE_CHECK 0
#define DRBD_MINOR_COUNT_MIN 1U
#define DRBD_MINOR_COUNT_MAX 255U
@@ -51,7 +51,8 @@
/* net { */
/* timeout, unit centi seconds
- * more than one minute timeout is not useful */
+ * more than one minute timeout is not useful
+ */
#define DRBD_TIMEOUT_MIN 1U
#define DRBD_TIMEOUT_MAX 600U
#define DRBD_TIMEOUT_DEF 60U /* 6 seconds */
@@ -63,7 +64,7 @@
#define DRBD_DISK_TIMEOUT_DEF 0U /* disabled */
#define DRBD_DISK_TIMEOUT_SCALE '1'
- /* active connection retries when C_WF_CONNECTION */
+ /* active connection retries when C_CONNECTING */
#define DRBD_CONNECT_INT_MIN 1U
#define DRBD_CONNECT_INT_MAX 120U
#define DRBD_CONNECT_INT_DEF 10U /* seconds */
@@ -88,12 +89,12 @@
#define DRBD_MAX_EPOCH_SIZE_SCALE '1'
#define DRBD_SNDBUF_SIZE_MIN 0U
-#define DRBD_SNDBUF_SIZE_MAX (128U << 20)
+#define DRBD_SNDBUF_SIZE_MAX (128U<<20)
#define DRBD_SNDBUF_SIZE_DEF 0U
#define DRBD_SNDBUF_SIZE_SCALE '1'
#define DRBD_RCVBUF_SIZE_MIN 0U
-#define DRBD_RCVBUF_SIZE_MAX (128U << 20)
+#define DRBD_RCVBUF_SIZE_MAX (128U<<20)
#define DRBD_RCVBUF_SIZE_DEF 0U
#define DRBD_RCVBUF_SIZE_SCALE '1'
@@ -110,11 +111,14 @@
#define DRBD_UNPLUG_WATERMARK_SCALE '1'
/* 0 is disabled.
- * 200 should be more than enough even for very short timeouts */
+ * 200 should be more than enough even for very short timeouts
+ */
#define DRBD_KO_COUNT_MIN 0U
#define DRBD_KO_COUNT_MAX 200U
#define DRBD_KO_COUNT_DEF 7U
#define DRBD_KO_COUNT_SCALE '1'
+
+#define DRBD_ALLOW_REMOTE_READ_DEF 1U
/* } */
/* syncer { */
@@ -125,10 +129,12 @@
#define DRBD_RESYNC_RATE_DEF 250U
#define DRBD_RESYNC_RATE_SCALE 'k' /* kilobytes */
+ /* less than 67 would hit performance unnecessarily. */
#define DRBD_AL_EXTENTS_MIN 67U
/* we use u16 as "slot number", (u16)~0 is "FREE".
* If you use >= 292 kB on-disk ring buffer,
- * this is the maximum you can use: */
+ * this is the maximum you can use:
+ */
#define DRBD_AL_EXTENTS_MAX 0xfffeU
#define DRBD_AL_EXTENTS_DEF 1237U
#define DRBD_AL_EXTENTS_SCALE '1'
@@ -143,7 +149,8 @@
/* drbdsetup XY resize -d Z
* you are free to reduce the device size to nothing, if you want to.
* the upper limit with 64bit kernel, enough ram and flexible meta data
- * is 1 PiB, currently. */
+ * is 1 PiB, currently.
+ */
/* DRBD_MAX_SECTORS */
#define DRBD_DISK_SIZE_MIN 0LLU
#define DRBD_DISK_SIZE_MAX (1LLU * (2LLU << 40))
@@ -180,7 +187,7 @@
#define DRBD_C_FILL_TARGET_DEF 100U /* Try to place 50KiB in socket send buffer during resync */
#define DRBD_C_FILL_TARGET_SCALE 's' /* sectors */
-#define DRBD_C_MAX_RATE_MIN 250U
+#define DRBD_C_MAX_RATE_MIN 0U
#define DRBD_C_MAX_RATE_MAX (4U << 20)
#define DRBD_C_MAX_RATE_DEF 102400U
#define DRBD_C_MAX_RATE_SCALE 'k' /* kilobytes */
@@ -205,26 +212,75 @@
#define DRBD_DISK_BARRIER_DEF 0U
#define DRBD_DISK_FLUSHES_DEF 1U
#define DRBD_DISK_DRAIN_DEF 1U
+#define DRBD_DISK_DISKLESS_DEF 0U
#define DRBD_MD_FLUSHES_DEF 1U
#define DRBD_TCP_CORK_DEF 1U
#define DRBD_AL_UPDATES_DEF 1U
-
+#define DRBD_INVALIDATE_RESET_BITMAP_DEF 1U
/* We used to ignore the discard_zeroes_data setting.
* To not change established (and expected) behaviour,
* by default assume that, for discard_zeroes_data=0,
* we can make that an effective discard_zeroes_data=1,
- * if we only explicitly zero-out unaligned partial chunks. */
+ * if we only explicitly zero-out unaligned partial chunks.
+ */
#define DRBD_DISCARD_ZEROES_IF_ALIGNED_DEF 1U
/* Some backends pretend to support WRITE SAME,
* but fail such requests when they are actually submitted.
- * This is to tell DRBD to not even try. */
+ * This is to tell DRBD to not even try.
+ */
#define DRBD_DISABLE_WRITE_SAME_DEF 0U
#define DRBD_ALLOW_TWO_PRIMARIES_DEF 0U
#define DRBD_ALWAYS_ASBP_DEF 0U
#define DRBD_USE_RLE_DEF 1U
#define DRBD_CSUMS_AFTER_CRASH_ONLY_DEF 0U
+#define DRBD_AUTO_PROMOTE_DEF 1U
+#define DRBD_BITMAP_DEF 1U
+#define DRBD_RESYNC_WITHOUT_REPLICATION_DEF 1U
+
+#define DRBD_NR_REQUESTS_MIN 4U
+#define DRBD_NR_REQUESTS_DEF 8000U
+#define DRBD_NR_REQUESTS_MAX -1U
+#define DRBD_NR_REQUESTS_SCALE '1'
+
+#define DRBD_MAX_BIO_SIZE_DEF DRBD_MAX_BIO_SIZE
+#define DRBD_MAX_BIO_SIZE_MIN (1U << 9)
+#define DRBD_MAX_BIO_SIZE_MAX DRBD_MAX_BIO_SIZE
+#define DRBD_MAX_BIO_SIZE_SCALE '1'
+
+#define DRBD_NODE_ID_DEF 0U
+#define DRBD_NODE_ID_MIN 0U
+#ifndef DRBD_NODE_ID_MAX /* Is also defined in drbd.h */
+#define DRBD_NODE_ID_MAX DRBD_PEERS_MAX
+#endif
+#define DRBD_NODE_ID_SCALE '1'
+
+#define DRBD_PEER_ACK_WINDOW_DEF 4096U /* 2 MiByte */
+#define DRBD_PEER_ACK_WINDOW_MIN 2048U /* 1 MiByte */
+#define DRBD_PEER_ACK_WINDOW_MAX 204800U /* 100 MiByte */
+#define DRBD_PEER_ACK_WINDOW_SCALE 's' /* sectors*/
+
+#define DRBD_PEER_ACK_DELAY_DEF 100U /* 100ms */
+#define DRBD_PEER_ACK_DELAY_MIN 1U
+#define DRBD_PEER_ACK_DELAY_MAX 10000U /* 10 seconds */
+#define DRBD_PEER_ACK_DELAY_SCALE '1' /* milliseconds */
+
+/* Two-phase commit timeout (1/10 seconds). */
+#define DRBD_TWOPC_TIMEOUT_MIN 50U
+#define DRBD_TWOPC_TIMEOUT_MAX 600U
+#define DRBD_TWOPC_TIMEOUT_DEF 300U
+#define DRBD_TWOPC_TIMEOUT_SCALE '1'
+
+#define DRBD_TWOPC_RETRY_TIMEOUT_MIN 1U
+#define DRBD_TWOPC_RETRY_TIMEOUT_MAX 50U
+#define DRBD_TWOPC_RETRY_TIMEOUT_DEF 1U
+#define DRBD_TWOPC_RETRY_TIMEOUT_SCALE '1'
+
+#define DRBD_SYNC_FROM_NID_DEF -1
+#define DRBD_SYNC_FROM_NID_MIN -1
+#define DRBD_SYNC_FROM_NID_MAX DRBD_PEERS_MAX
+#define DRBD_SYNC_FROM_NID_SCALE '1'
#define DRBD_AL_STRIPES_MIN 1U
#define DRBD_AL_STRIPES_MAX 1024U
@@ -241,9 +297,61 @@
#define DRBD_SOCKET_CHECK_TIMEO_DEF 0U
#define DRBD_SOCKET_CHECK_TIMEO_SCALE '1'
+/* Auto promote timeout (1/10 seconds). */
+#define DRBD_AUTO_PROMOTE_TIMEOUT_MIN 0U
+#define DRBD_AUTO_PROMOTE_TIMEOUT_MAX 600U
+#define DRBD_AUTO_PROMOTE_TIMEOUT_DEF 20U
+#define DRBD_AUTO_PROMOTE_TIMEOUT_SCALE '1'
+
#define DRBD_RS_DISCARD_GRANULARITY_MIN 0U
#define DRBD_RS_DISCARD_GRANULARITY_MAX (1U<<20) /* 1MiByte */
#define DRBD_RS_DISCARD_GRANULARITY_DEF 0U /* disabled by default */
#define DRBD_RS_DISCARD_GRANULARITY_SCALE '1' /* bytes */
-#endif /* _UAPI_LINUX_DRBD_LIMITS_H */
+#define DRBD_QUORUM_MIN 0U
+#define DRBD_QUORUM_MAX QOU_ALL /* Note: user visible min/max different */
+#define DRBD_QUORUM_DEF QOU_OFF /* kernel min/max includes symbolic values */
+#define DRBD_QUORUM_SCALE '1' /* nodes */
+
+#define DRBD_BLOCK_SIZE_MIN 512
+#define DRBD_BLOCK_SIZE_MAX 4096
+#define DRBD_BLOCK_SIZE_DEF 512
+#define DRBD_BLOCK_SIZE_SCALE '1' /* Bytes */
+
+#define DRBD_DISCARD_GRANULARITY_SCALE '1' /* Bytes */
+#define DRBD_DISCARD_GRANULARITY_MIN 0U /* 0 = disable discards */
+#define DRBD_DISCARD_GRANULARITY_MAX (128U<<20) /* 128 MiB, current DRBD_MAX_BATCH_BIO_SIZE */
+#define DRBD_DISCARD_GRANULARITY_DEF 0xFFFFFFFFU /* sentinel: not configured; use legacy behavior */
+
+/* By default freeze IO, if set error all IOs as quick as possible */
+#define DRBD_ON_NO_QUORUM_DEF ONQ_SUSPEND_IO
+
+#define DRBD_ON_SUSP_PRI_OUTD_DEF SPO_DISCONNECT
+#define DRBD_DRBD8_COMPAT_MODE_DEF 0U
+
+#define DRBD_TLS_DEF 0U /* disabled by default */
+#define DRBD_TLS_PRIVKEY_DEF 0 /* disabled by default */
+#define DRBD_TLS_CERTIFICATE_DEF 0 /* disabled by default */
+#define DRBD_TLS_KEYRING_DEF 0 /* disabled by default */
+
+#define DRBD_LOAD_BALANCE_PATHS_DEF 0U
+
+#define DRBD_PEER_TIEBREAKER_DEF 1U
+
+#define DRBD_RDMA_CTRL_RCVBUF_SIZE_MIN 0U
+#define DRBD_RDMA_CTRL_RCVBUF_SIZE_MAX (10U<<20)
+#define DRBD_RDMA_CTRL_RCVBUF_SIZE_DEF 0
+#define DRBD_RDMA_CTRL_RCVBUF_SIZE_SCALE '1'
+
+#define DRBD_RDMA_CTRL_SNDBUF_SIZE_MIN 0U
+#define DRBD_RDMA_CTRL_SNDBUF_SIZE_MAX (10U<<20)
+#define DRBD_RDMA_CTRL_SNDBUF_SIZE_DEF 0
+#define DRBD_RDMA_CTRL_SNDBUF_SIZE_SCALE '1'
+
+/* Enable bdev_freeze/lockfs by default */
+#define DRBD_SUSPEND_IO_BDEV_FREEZE_DEF 1U
+
+/* Wait for the barrier ack of the last epoch by default */
+#define DRBD_SUSPEND_IO_FLUSH_TO_PEERS_DEF 1U
+
+#endif
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 17/20] drbd: serve the "drbd" genl family at version 1 for existing userspace
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (15 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 16/20] drbd: rework the netlink core for DRBD 9 Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 18/20] drbd: add the "drbd2" genl family Christoph Böhmwalder
` (2 subsequent siblings)
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder
The "drbd" generic netlink family that this driver has always served
at version 1 is what every shipped drbd-utils release talks, and its
wire format is incompatible with what DRBD 9's own utilities speak.
Keep serving it unchanged: drbd_nl_84.c implements struct
drbd_nl_dialect for the version-1 wire, on top of the existing
generated serialization in drbd_nl_gen.[ch] and include/uapi/linux/
drbd_genl.h, which this series does not modify.
The adapter translates in both directions between mainline's wire
structs (struct net_conf, struct disk_conf, ...) and DRBD 9's neutral
ones (struct drbd_net_conf, ...), remapping the values the two
generations disagree about: disk-state numbering, since DRBD 9
inserted D_DETACHING, the metadata flag bits, the state-change result
codes below SS_UNKNOWN_ERROR, and the defaults that only version 1
carries. Version-1 fields that DRBD 9 keeps in a different object
(the fencing policy, the resync tuning, the connect flags) are parsed
and applied by the command that owns that object. Requests without a
node id create a resource in DRBD 8.4 compatibility mode, so a
version-1 client sees the single-peer semantics it expects,
including the fused DRBD_EVENT broadcast that "drbdsetup events"
consumes and the GET_STATUS dump that drbdadm relies on.
The dialect needs the 8.4 metadata and /proc/drbd support, so it is
built with DRBD_COMPAT_84, and so is drbd_nl_gen.o, which only calls
into it. Without that option the driver does not register the "drbd"
family at all.
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Makefile | 3 +-
drivers/block/drbd/drbd_nl_84.c | 3622 +++++++++++++++++++++++++++++++
2 files changed, 3624 insertions(+), 1 deletion(-)
create mode 100644 drivers/block/drbd/drbd_nl_84.c
diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index f430536636c2..0b7bc8f5d53a 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -3,8 +3,9 @@ drbd-y := drbd_buildtag.o drbd_bitmap.o drbd_proc.o
drbd-y += drbd_sender.o drbd_receiver.o drbd_req.o drbd_actlog.o
drbd-y += drbd_main.o drbd_strings.o drbd_nl.o
drbd-y += drbd_interval.o drbd_state.o
-drbd-y += drbd_nl_gen.o
+drbd-$(CONFIG_DRBD_COMPAT_84) += drbd_nl_gen.o
drbd-y += drbd_nl_defaults.o
+drbd-$(CONFIG_DRBD_COMPAT_84) += drbd_nl_84.o
drbd-y += drbd_transport.o
drbd-$(CONFIG_DEV_DAX_PMEM) += drbd_dax_pmem.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
diff --git a/drivers/block/drbd/drbd_nl_84.c b/drivers/block/drbd/drbd_nl_84.c
new file mode 100644
index 000000000000..f9128bc0f3f7
--- /dev/null
+++ b/drivers/block/drbd/drbd_nl_84.c
@@ -0,0 +1,3622 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026, LINBIT HA-Solutions GmbH.
+ */
+
+/*
+ * The "drbd" generic netlink family at version 1: the dialect spoken by
+ * DRBD 8.4 userland (drbd-utils 8.9.x). Everything here knows about the
+ * bytes on the wire; the command implementations live in drbd_nl.c and
+ * only ever see a struct drbd_adm_ctx.
+ *
+ * Needs CONFIG_DRBD_COMPAT_84 for the 8.4 metadata and /proc/drbd support
+ * the dialect translates to.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/slab.h>
+#include <linux/drbd.h>
+#include <linux/drbd_limits.h>
+#include <linux/in.h>
+#include <linux/in6.h>
+#include <linux/crc32c.h>
+#include <net/genetlink.h>
+#include <net/sock.h>
+
+#include "drbd_int.h"
+#include "drbd_nl.h"
+#include "drbd_legacy_84.h"
+#include "drbd_nl_gen.h"
+
+static const struct drbd_nl_dialect drbd_nl_84_dialect;
+
+/*
+ * 8.4's reason codes for a DRBD_ADM_EVENT message and for the reply to
+ * DRBD_ADM_GET_STATUS. DRBD 9 has no use for these, so they live here
+ * rather than in drbd.h. Copied from mainline
+ * include/uapi/linux/drbd.h:427-432.
+ */
+enum drbd_state_info_bcast_reason {
+ SIB_GET_STATUS_REPLY = 1,
+ SIB_STATE_CHANGE = 2,
+ SIB_HELPER_PRE = 3,
+ SIB_HELPER_POST = 4,
+ SIB_SYNC_PROGRESS = 5,
+};
+
+/*
+ * Defined below compat84_fill_state_info(); forward declared for the
+ * notify hooks above it. @prev_state_84/@new_state_84 are read only for
+ * SIB_STATE_CHANGE, @helper_name/@helper_exit_code only for
+ * SIB_HELPER_PRE/SIB_HELPER_POST.
+ */
+static int compat84_emit_event(struct drbd_device *device,
+ enum drbd_state_info_bcast_reason reason,
+ u32 prev_state_84, u32 new_state_84,
+ const char *helper_name, int helper_exit_code);
+
+/*
+ * 8.4 reports one SIB_STATE_CHANGE event per device and state change, with
+ * the packed state before and after it; DRBD 9 notifies each object on its
+ * own. The notify hooks write their object's old and new fields into the
+ * device's pending pair, both taken from the state-change snapshot, and
+ * the hook that ends the state change (no NOTIFY_CONTINUES) sends one event
+ * per device whose 8.4 state moved. Fields no hook touches come from the
+ * live state and are equal on both sides. notification_mutex serializes
+ * all of this.
+ */
+static void compat84_event_get(struct drbd_device *device,
+ union drbd_state *prev, union drbd_state *new)
+{
+ if (!device->bcast_pending_84) {
+ device->bcast_prev_84 = drbd_pack_state_84(device);
+ device->bcast_new_84 = device->bcast_prev_84;
+ device->bcast_cstate_prev_84 = -1;
+ device->bcast_cstate_new_84 = -1;
+ device->bcast_pending_84 = true;
+ }
+ prev->i = device->bcast_prev_84;
+ new->i = device->bcast_new_84;
+}
+
+static void compat84_event_put(struct drbd_device *device,
+ union drbd_state prev, union drbd_state new)
+{
+ device->bcast_prev_84 = prev.i;
+ device->bcast_new_84 = new.i;
+}
+
+static void compat84_event_flush(struct drbd_resource *resource,
+ enum drbd_notification_type type)
+{
+ struct drbd_device *device;
+ int vnr;
+
+ if (type & NOTIFY_CONTINUES)
+ return;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ u32 prev = device->bcast_prev_84, new = device->bcast_new_84;
+
+ if (!device->bcast_pending_84)
+ continue;
+ device->bcast_pending_84 = false;
+ if (prev == new)
+ continue;
+
+ kref_get(&device->kref);
+ rcu_read_unlock();
+ compat84_emit_event(device, SIB_STATE_CHANGE, prev, new, NULL, 0);
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+}
+
+/* Per-request state of the v1 dialect; hangs off drbd_adm_ctx.req. */
+struct compat84_req {
+ struct genl_info *info;
+ struct sk_buff *reply_skb;
+ struct drbd_genlmsghdr *reply_dh;
+
+ /*
+ * Attributes that arrive inside one v1 container but belong to a
+ * different DRBD 9 struct than that container's overlay() dst.
+ * compat84_overlay() stashes them here; the handler that owns the
+ * real struct applies them. Each field carries its own "has_*"
+ * flag, set only when that attribute itself was present: the
+ * fields are independently optional on the wire, so a
+ * container-level flag would misreport unsent siblings as zero.
+ */
+
+ /* the v1 disk_conf's resync-tuning fields -> struct drbd_peer_device_conf.
+ * Applied by compat84_apply_disk_conf_stash() if a peer device
+ * exists, else deferred to device->pending_peer_device_conf_84
+ * for the connect handler.
+ */
+ struct {
+ bool has_resync_rate;
+ u32 resync_rate;
+ bool has_c_plan_ahead;
+ u32 c_plan_ahead;
+ bool has_c_delay_target;
+ u32 c_delay_target;
+ bool has_c_fill_target;
+ u32 c_fill_target;
+ bool has_c_max_rate;
+ u32 c_max_rate;
+ bool has_c_min_rate;
+ u32 c_min_rate;
+ } peer_device_conf_84;
+
+ /* v1 disk_conf.fencing -> drbd_net_conf.fencing_policy. Applied if a
+ * connection exists, else deferred to
+ * resource->pending_fencing_policy_84 for the connect handler.
+ */
+ bool has_fencing_policy_84;
+ u32 fencing_policy_84;
+
+ /* v1 net_conf.{discard_my_data, tentative} -> struct drbd_connect_parms,
+ * applied through the DRBD_NL_SET_CONNECT_PARMS overlay case.
+ */
+ struct {
+ bool has_discard_my_data;
+ unsigned char discard_my_data;
+ bool has_tentative;
+ unsigned char tentative;
+ } connect_parms_84;
+
+ /*
+ * ctx_my_addr/ctx_peer_addr from DRBD_NLA_CFG_CONTEXT, raw
+ * sockaddr bytes; v1 has no NLA_PATH_PARMS container. Parsed once
+ * in drbd_pre_doit(), read by the DRBD_NL_SET_PATH_PARMS overlay
+ * case and by compat84_resolve_peer_node_id(). *_len is 0 when the
+ * attribute was not sent.
+ */
+ u8 ctx_my_addr[128];
+ u32 ctx_my_addr_len;
+ u8 ctx_peer_addr[128];
+ u32 ctx_peer_addr_len;
+};
+
+/* One allocation for both, so that pre_doit zeroes the context only once. */
+struct compat84_ctx {
+ struct drbd_adm_ctx ctx;
+ struct compat84_req req;
+};
+
+static inline struct compat84_req *compat84_req(struct drbd_adm_ctx *ctx)
+{
+ return ctx->req;
+}
+
+static const struct genl_multicast_group drbd_nl_mcgrps[] = {
+ [DRBD_NLGRP_EVENTS] = { .name = "events", },
+};
+
+static struct genl_family drbd_nl_family __ro_after_init = {
+ .name = DRBD_FAMILY_NAME,
+ .version = DRBD_FAMILY_VERSION,
+ .hdrsize = NLA_ALIGN(sizeof(struct drbd_genlmsghdr)),
+ .split_ops = drbd_nl_ops,
+ .n_split_ops = ARRAY_SIZE(drbd_nl_ops),
+ /*
+ * Every v1 command predates the reserved-field/flags checks that
+ * genl applies from resv_start_op onward; matches mainline's
+ * drbd_nl_family.
+ */
+ .resv_start_op = DRBD_ADM_INITIAL_STATE_DONE + 1,
+ .mcgrps = drbd_nl_mcgrps,
+ .n_mcgrps = ARRAY_SIZE(drbd_nl_mcgrps),
+ .module = THIS_MODULE,
+ /* Register in every network namespace, like the v2 family. */
+ .netnsok = true,
+};
+
+/*
+ * Multicast sequence number of the events2 notifications, as mainline
+ * keeps it. The core's drbd_genl_seq also counts path and rename
+ * notifications, which v1 does not send, and drbdsetup-84 events2 reports
+ * every gap as "- skipped N".
+ */
+static atomic_t compat84_notify_seq = ATOMIC_INIT(2);
+/* ... and of DRBD_EVENT, mainline's drbd_bcast_event() drbd_genl_seq. */
+static atomic_t compat84_event_seq = ATOMIC_INIT(2);
+
+static int compat84_genl_multicast_events(struct sk_buff *skb)
+{
+ return genlmsg_multicast_allns(&drbd_nl_family, skb, 0,
+ DRBD_NLGRP_EVENTS);
+}
+
+static void drbd_adm_send_reply(struct sk_buff *skb, struct genl_info *info)
+{
+ genlmsg_end(skb, genlmsg_data(nlmsg_data(nlmsg_hdr(skb))));
+ if (genlmsg_reply(skb, info))
+ pr_err("error sending genl reply\n");
+}
+
+/* Used on a fresh reply_skb, this cannot fail: The only reason it could
+ * fail was no space in skb, and there are 4k available.
+ */
+static int drbd_msg_put_info(struct sk_buff *skb, const char *info)
+{
+ struct nlattr *nla;
+ int err = -EMSGSIZE;
+
+ if (!info || !info[0])
+ return 0;
+
+ nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_REPLY);
+ if (!nla)
+ return err;
+
+ err = nla_put_string(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, info);
+ if (err) {
+ nla_nest_cancel(skb, nla);
+ return err;
+ }
+ nla_nest_end(skb, nla);
+ return 0;
+}
+
+/*
+ * Serialize ctx->result and ctx->msg[] into the v1 reply. Only enum
+ * drbd_state_rv values need remapping for the v1 wire; drbd_state_rv_84()
+ * passes everything else through unchanged.
+ */
+static void compat84_put_outcome(struct drbd_adm_ctx *ctx)
+{
+ struct compat84_req *req = compat84_req(ctx);
+ const char *p, *end;
+
+ req->reply_dh->ret_code = drbd_state_rv_84(ctx->result);
+ for (p = ctx->msg, end = ctx->msg + ctx->msg_len; p < end; p += strlen(p) + 1)
+ drbd_msg_put_info(req->reply_skb, p);
+}
+
+static bool compat84_need_sys_admin(u8 cmd)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(drbd_nl_ops); i++)
+ if (drbd_nl_ops[i].cmd == cmd)
+ return 0 != (drbd_nl_ops[i].flags & GENL_ADMIN_PERM);
+ return true;
+}
+
+/* The v1 command set: a subset of v2's, with a few commands renamed
+ * (CHG_DISK_OPTS/CHG_NET_OPTS/INVAL_PEER) and none of the DRBD 9 only
+ * commands (peers, paths, rename, forget-peer).
+ */
+static const unsigned int drbd_genl_cmd_flags_84[] = {
+ [DRBD_ADM_NEW_MINOR] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD_ADM_DEL_MINOR] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_NEW_RESOURCE] = 0,
+ [DRBD_ADM_DEL_RESOURCE] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD_ADM_RESOURCE_OPTS] = DRBD_ADM_NEED_RESOURCE,
+ /*
+ * The command that creates a connection cannot require one to
+ * exist; the connect handler derives the peer node id from the
+ * address pair itself.
+ */
+ [DRBD_ADM_CONNECT] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD_ADM_DISCONNECT] = DRBD_ADM_NEED_CONNECTION,
+ [DRBD_ADM_ATTACH] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_RESIZE] = DRBD_ADM_NEED_MINOR,
+ /*
+ * v1 addresses primary/secondary by minor only (CTX_MINOR), unlike
+ * v2, which sends a resource name. drbd_adm_ctx_resolve() derives
+ * the resource from the device, which is all drbd_adm_set_role()
+ * needs.
+ */
+ [DRBD_ADM_PRIMARY] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_SECONDARY] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_NEW_C_UUID] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_START_OV] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD_ADM_DETACH] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_INVALIDATE] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_INVAL_PEER] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD_ADM_PAUSE_SYNC] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD_ADM_RESUME_SYNC] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD_ADM_SUSPEND_IO] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_RESUME_IO] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_OUTDATE] = DRBD_ADM_NEED_MINOR,
+ /*
+ * The doit path serves the per-minor queries (state, role, cstate,
+ * dstate, show-gi, get-gi); the dump path (status, show, events)
+ * needs no minor.
+ */
+ [DRBD_ADM_GET_STATUS] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_GET_TIMEOUT_TYPE] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD_ADM_DOWN] = DRBD_ADM_NEED_RESOURCE | DRBD_ADM_IGNORE_VERSION,
+ [DRBD_ADM_CHG_DISK_OPTS] = DRBD_ADM_NEED_MINOR,
+ [DRBD_ADM_CHG_NET_OPTS] = DRBD_ADM_NEED_CONNECTION,
+ [DRBD_ADM_GET_RESOURCES] = 0,
+ [DRBD_ADM_GET_DEVICES] = 0,
+ [DRBD_ADM_GET_CONNECTIONS] = 0,
+ [DRBD_ADM_GET_PEER_DEVICES] = 0,
+ [DRBD_ADM_GET_INITIAL_STATE] = 0,
+};
+
+/*
+ * Highest nested attribute type the kernel knows for each top-level
+ * attribute, for compat84_check_mandatory(). Kept apart from the netlink
+ * policies: kernels before v4.20 take .len of an NLA_NESTED entry as a
+ * minimum payload length, so the policies cannot carry it.
+ */
+static const u16 drbd_tla_nested_max_84[__DRBD_NLA_MAX] = {
+ [DRBD_NLA_CFG_REPLY] = DRBD_A_DRBD_CFG_REPLY_MAX,
+ [DRBD_NLA_CFG_CONTEXT] = DRBD_A_DRBD_CFG_CONTEXT_MAX,
+ [DRBD_NLA_DISK_CONF] = DRBD_A_DISK_CONF_MAX,
+ [DRBD_NLA_RESOURCE_OPTS] = DRBD_A_RES_OPTS_MAX,
+ [DRBD_NLA_NET_CONF] = DRBD_A_NET_CONF_MAX,
+ [DRBD_NLA_SET_ROLE_PARMS] = DRBD_A_SET_ROLE_PARMS_MAX,
+ [DRBD_NLA_RESIZE_PARMS] = DRBD_A_RESIZE_PARMS_MAX,
+ [DRBD_NLA_STATE_INFO] = DRBD_A_STATE_INFO_MAX,
+ [DRBD_NLA_START_OV_PARMS] = DRBD_A_START_OV_PARMS_MAX,
+ [DRBD_NLA_NEW_C_UUID_PARMS] = DRBD_A_NEW_C_UUID_PARMS_MAX,
+ [DRBD_NLA_TIMEOUT_PARMS] = DRBD_A_TIMEOUT_PARMS_MAX,
+ [DRBD_NLA_DISCONNECT_PARMS] = DRBD_A_DISCONNECT_PARMS_MAX,
+ [DRBD_NLA_DETACH_PARMS] = DRBD_A_DETACH_PARMS_MAX,
+ [DRBD_NLA_RESOURCE_INFO] = DRBD_A_RESOURCE_INFO_MAX,
+ [DRBD_NLA_DEVICE_INFO] = DRBD_A_DEVICE_INFO_MAX,
+ [DRBD_NLA_CONNECTION_INFO] = DRBD_A_CONNECTION_INFO_MAX,
+ [DRBD_NLA_PEER_DEVICE_INFO] = DRBD_A_PEER_DEVICE_INFO_MAX,
+ [DRBD_NLA_RESOURCE_STATISTICS] = DRBD_A_RESOURCE_STATISTICS_MAX,
+ [DRBD_NLA_DEVICE_STATISTICS] = DRBD_A_DEVICE_STATISTICS_MAX,
+ [DRBD_NLA_CONNECTION_STATISTICS] = DRBD_A_CONNECTION_STATISTICS_MAX,
+ [DRBD_NLA_PEER_DEVICE_STATISTICS] = DRBD_A_PEER_DEVICE_STATISTICS_MAX,
+ [DRBD_NLA_NOTIFICATION_HEADER] = DRBD_A_DRBD_NOTIFICATION_HEADER_MAX,
+ [DRBD_NLA_HELPER] = DRBD_A_DRBD_HELPER_INFO_MAX,
+};
+
+/* Strip DRBD_GENLA_F_MANDATORY from nested attrs before standard parsing.
+ * Reject unknown attrs that had the mandatory bit set.
+ */
+static int compat84_check_mandatory(const struct genl_split_ops *ops,
+ struct genl_info *info)
+{
+ int i;
+
+ for (i = 0; i <= ops->maxattr && i < ARRAY_SIZE(drbd_tla_nested_max_84); i++) {
+ struct nlattr *tla = info->attrs[i];
+ struct nlattr *nla;
+ int rem;
+
+ if (!tla || !drbd_tla_nested_max_84[i])
+ continue;
+
+ nla_for_each_nested(nla, tla, rem) {
+ if (nla->nla_type & DRBD_GENLA_F_MANDATORY) {
+ nla->nla_type &= ~DRBD_GENLA_F_MANDATORY;
+ if (nla_type(nla) > drbd_tla_nested_max_84[i])
+ return -EOPNOTSUPP;
+ }
+ }
+ }
+ return 0;
+}
+
+/*
+ * v1 has no ctx_peer_node_id attribute: an 8.4 resource has at most one
+ * peer. Resolve adm_ctx->peer_node_id (and adm_ctx->resource, when
+ * nothing else identifies it) here, before drbd_adm_ctx_resolve() runs,
+ * so that function needs no v1-specific fallback.
+ *
+ * Disconnect and net-options address the connection by its endpoint
+ * addresses (CTX_CONNECTION) and carry neither a resource name nor a
+ * minor, so the addresses are matched against every 8.4-mode resource's
+ * connections. Connect never gets here: it needs only the resource name
+ * and derives its own peer node id. The remaining peer-scoped commands
+ * (start-ov, invalidate-remote, pause-sync, resume-sync,
+ * get-timeout-type) address a minor only; an 8.4-mode resource has at
+ * most one connection, and the minor also pins the volume.
+ *
+ * Only 8.4-mode resources are considered, so a v1 request cannot act on
+ * a DRBD 9 native resource by naming its addresses. Both branches are
+ * skipped when a resource name was sent as well: drbd_adm_ctx_resolve()
+ * re-resolves adm_ctx->resource from the name unconditionally and
+ * would leak the reference taken here.
+ */
+static void compat84_resolve_peer_node_id(struct drbd_adm_ctx *adm_ctx)
+{
+ struct compat84_req *req = compat84_req(adm_ctx);
+ struct drbd_connection *connection = NULL;
+ struct drbd_resource *resource = NULL;
+ unsigned int volume = VOLUME_UNSPECIFIED;
+
+ rcu_read_lock();
+ if (req->ctx_my_addr_len && req->ctx_peer_addr_len && !adm_ctx->resource_name) {
+ struct drbd_resource *r;
+
+ for_each_resource_rcu(r, &drbd_resources) {
+ struct drbd_connection *c;
+
+ if (!r->res_opts.drbd8_compat_mode)
+ continue;
+
+ for_each_connection_rcu(c, r) {
+ struct drbd_path *path;
+
+ list_for_each_entry_rcu(path, &c->transport.paths, list) {
+ if (path->my_addr_len != req->ctx_my_addr_len ||
+ memcmp(&path->my_addr, req->ctx_my_addr,
+ path->my_addr_len))
+ continue;
+ if (path->peer_addr_len != req->ctx_peer_addr_len ||
+ memcmp(&path->peer_addr, req->ctx_peer_addr,
+ path->peer_addr_len))
+ continue;
+ connection = c;
+ break;
+ }
+ if (connection)
+ break;
+ }
+ if (connection) {
+ resource = r;
+ break;
+ }
+ }
+ } else if (adm_ctx->minor != -1U && !adm_ctx->resource_name) {
+ struct drbd_device *device = minor_to_device(adm_ctx->minor);
+
+ if (device && device->resource->res_opts.drbd8_compat_mode) {
+ resource = device->resource;
+ connection = list_first_or_null_rcu(&resource->connections,
+ struct drbd_connection,
+ connections);
+ volume = device->vnr;
+ }
+ }
+
+ if (connection) {
+ kref_get(&resource->kref);
+ adm_ctx->resource = resource;
+ adm_ctx->peer_node_id = connection->peer_node_id;
+ if (volume != VOLUME_UNSPECIFIED)
+ adm_ctx->volume = volume;
+ }
+ rcu_read_unlock();
+}
+
+/*
+ * Every resource this dialect creates is in DRBD 8.4 compatibility mode.
+ * A native DRBD 9 resource (from the drbd2 family) is not one 8.4
+ * userland can drive, whether it is named by resource or by minor.
+ */
+static bool compat84_reject_native(struct drbd_adm_ctx *adm_ctx)
+{
+ if (!adm_ctx->resource || adm_ctx->resource->res_opts.drbd8_compat_mode)
+ return false;
+
+ drbd_adm_msg(adm_ctx, "%s", "not a DRBD 8.4 compatibility mode resource");
+ adm_ctx->result = ERR_INVALID_REQUEST;
+ return true;
+}
+
+/*
+ * Allocates the command context together with the v1 request state,
+ * stores it in info->user_ptr[0], prepares the reply skb and resolves the
+ * objects the command refers to. Rejects unknown netlink versions with
+ * -EINVAL.
+ */
+int drbd_pre_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct drbd_genlmsghdr *d_in = genl_info_userhdr(info);
+ const u8 cmd = info->genlhdr->cmd;
+ struct drbd_adm_ctx *adm_ctx;
+ struct compat84_ctx *cctx;
+ struct compat84_req *req;
+ unsigned int flags;
+ int err;
+
+ err = compat84_check_mandatory(ops, info);
+ if (err)
+ return err;
+
+ /* Look up per-command flags */
+ flags = (cmd < ARRAY_SIZE(drbd_genl_cmd_flags_84)) ? drbd_genl_cmd_flags_84[cmd] : 0;
+
+ if (info->genlhdr->version != DRBD_FAMILY_VERSION && !(flags & DRBD_ADM_IGNORE_VERSION))
+ return -EINVAL;
+
+ /*
+ * genl_rcv_msg() only checks if commands with the GENL_ADMIN_PERM flag
+ * set have CAP_NET_ADMIN; we also require CAP_SYS_ADMIN for
+ * administrative commands.
+ */
+ if (compat84_need_sys_admin(cmd) && !capable(CAP_SYS_ADMIN))
+ return -EPERM;
+
+ cctx = kzalloc_obj(struct compat84_ctx);
+ if (!cctx)
+ return -ENOMEM;
+
+ adm_ctx = &cctx->ctx;
+ req = &cctx->req;
+ adm_ctx->d = &drbd_nl_84_dialect;
+ adm_ctx->req = req;
+ req->info = info;
+
+ adm_ctx->net = sock_net(skb->sk);
+
+ req->reply_skb = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL);
+ if (!req->reply_skb) {
+ err = -ENOMEM;
+ goto fail;
+ }
+
+ req->reply_dh = genlmsg_put_reply(req->reply_skb,
+ info, &drbd_nl_family, 0, cmd);
+ /* put of a few bytes into a fresh skb of >= 4k will always succeed.
+ * but anyways
+ */
+ if (!req->reply_dh) {
+ err = -ENOMEM;
+ goto fail;
+ }
+
+ req->reply_dh->minor = d_in->minor;
+ adm_ctx->minor = d_in->minor;
+ adm_ctx->result = NO_ERROR;
+ adm_ctx->set_defaults = !!(d_in->flags & DRBD_GENL_F_SET_DEFAULTS);
+
+ adm_ctx->volume = VOLUME_UNSPECIFIED;
+ adm_ctx->peer_node_id = PEER_NODE_ID_UNSPECIFIED;
+ if (info->attrs[DRBD_NLA_CFG_CONTEXT]) {
+ struct nlattr *nla;
+ struct nlattr **nested_attr_tb;
+ /* parse and validate only */
+ err = drbd_cfg_context_ntb_from_attrs(&nested_attr_tb, info);
+ if (err)
+ goto fail;
+
+ /* It was present, and valid,
+ * copy it over to the reply skb.
+ */
+ err = nla_put_nohdr(req->reply_skb,
+ info->attrs[DRBD_NLA_CFG_CONTEXT]->nla_len,
+ info->attrs[DRBD_NLA_CFG_CONTEXT]);
+ if (err) {
+ kfree(nested_attr_tb);
+ goto fail;
+ }
+
+ /* and assign stuff to the adm_ctx */
+ nla = nested_attr_tb[DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME];
+ if (nla)
+ adm_ctx->volume = nla_get_u32(nla);
+ nla = nested_attr_tb[DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME];
+ if (nla)
+ adm_ctx->resource_name = nla_data(nla);
+ /* No ctx_peer_node_id in v1; the addresses stand in for it. */
+ nla = nested_attr_tb[DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR];
+ if (nla) {
+ req->ctx_my_addr_len = min_t(u32, nla_len(nla), sizeof(req->ctx_my_addr));
+ memcpy(req->ctx_my_addr, nla_data(nla), req->ctx_my_addr_len);
+ }
+ nla = nested_attr_tb[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR];
+ if (nla) {
+ req->ctx_peer_addr_len =
+ min_t(u32, nla_len(nla), sizeof(req->ctx_peer_addr));
+ memcpy(req->ctx_peer_addr, nla_data(nla), req->ctx_peer_addr_len);
+ }
+ kfree(nested_attr_tb);
+ }
+
+ /*
+ * Only commands that need a connection or peer device inside
+ * drbd_adm_ctx_resolve(); connect is deliberately excluded.
+ */
+ if (flags & (DRBD_ADM_NEED_CONNECTION | DRBD_ADM_NEED_PEER_DEVICE))
+ compat84_resolve_peer_node_id(adm_ctx);
+
+ if (drbd_adm_ctx_resolve(adm_ctx, flags) != NO_ERROR ||
+ compat84_reject_native(adm_ctx)) {
+ /* Send error reply now; NULL reply_skb so the handler bails
+ * out. post_doit will drop the kref references.
+ */
+ compat84_put_outcome(adm_ctx);
+ drbd_adm_send_reply(req->reply_skb, info);
+ req->reply_skb = NULL;
+ }
+
+ info->user_ptr[0] = adm_ctx;
+ return 0;
+
+fail:
+ /* Fatal error while preparing the reply; nothing is resolved yet. */
+ nlmsg_free(req->reply_skb);
+ kfree(cctx);
+ info->user_ptr[0] = NULL;
+ return err;
+}
+
+/*
+ * Sends the reply, drops the references acquired while resolving, and
+ * frees the context.
+ */
+void drbd_post_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct drbd_adm_ctx *adm_ctx = info->user_ptr[0];
+ struct compat84_req *req;
+
+ if (!adm_ctx)
+ return;
+
+ req = compat84_req(adm_ctx);
+ if (req->reply_skb) {
+ compat84_put_outcome(adm_ctx);
+ drbd_adm_send_reply(req->reply_skb, info);
+ }
+
+ drbd_adm_ctx_release(adm_ctx);
+ kfree(container_of(adm_ctx, struct compat84_ctx, ctx));
+}
+
+static bool compat84_has_set(struct drbd_adm_ctx *ctx, enum drbd_nl_attr_set set)
+{
+ static const u16 tla[__DRBD_NL_SET_MAX] = {
+ [DRBD_NL_SET_DISK_CONF] = DRBD_NLA_DISK_CONF,
+ [DRBD_NL_SET_NET_CONF] = DRBD_NLA_NET_CONF,
+ [DRBD_NL_SET_RES_OPTS] = DRBD_NLA_RESOURCE_OPTS,
+ [DRBD_NL_SET_SET_ROLE_PARMS] = DRBD_NLA_SET_ROLE_PARMS,
+ [DRBD_NL_SET_RESIZE_PARMS] = DRBD_NLA_RESIZE_PARMS,
+ [DRBD_NL_SET_START_OV_PARMS] = DRBD_NLA_START_OV_PARMS,
+ [DRBD_NL_SET_NEW_C_UUID_PARMS] = DRBD_NLA_NEW_C_UUID_PARMS,
+ [DRBD_NL_SET_DISCONNECT_PARMS] = DRBD_NLA_DISCONNECT_PARMS,
+ [DRBD_NL_SET_DETACH_PARMS] = DRBD_NLA_DETACH_PARMS,
+ };
+
+ /*
+ * v1 has no NLA_CONNECT_PARMS container: discard_my_data and
+ * tentative arrive inside net_conf and are stashed while
+ * overlaying it, so report whether there is a stashed value.
+ */
+ if (set == DRBD_NL_SET_CONNECT_PARMS) {
+ struct compat84_req *req = compat84_req(ctx);
+
+ return req->connect_parms_84.has_discard_my_data ||
+ req->connect_parms_84.has_tentative;
+ }
+
+ if (!tla[set])
+ return false;
+ return compat84_req(ctx)->info->attrs[tla[set]] != NULL;
+}
+
+/*
+ * The generated *_from_attrs() parsers fill the vendored v1 wire
+ * structs (struct disk_conf, struct net_conf, ...), whose field sets
+ * differ from the core's neutral structs, so every set below parses into
+ * a local v1 struct and translates field by field into dst.
+ *
+ * For the persistent config sets (disk_conf, net_conf, res_opts), dst
+ * already holds the object's current values. The parser only writes a
+ * field whose attribute is present, so the local struct is pre-seeded
+ * from dst and copied back whole: an absent attribute round-trips its
+ * old value, preserving drbd_nl.h's "absent attributes leave dst
+ * untouched" contract. Most one-shot action parms start from a zeroed
+ * dst, so a zeroed local struct already matches; resize_parms and
+ * start_ov_parms do not, and are pre-seeded the same way.
+ *
+ * The v1 disk_conf treats backing_dev/meta_dev/meta_dev_idx as required, so
+ * a plain disk-options call parses with err == -ENOMSG while every other
+ * field is still filled in. drbd_nl.c tolerates -ENOMSG as non-fatal, so
+ * each case copies back (and stashes) before returning err, never on an
+ * early return.
+ */
+static int compat84_overlay(struct drbd_adm_ctx *ctx, enum drbd_nl_attr_set set, void *dst)
+{
+ struct genl_info *info = compat84_req(ctx)->info;
+ struct compat84_req *req = compat84_req(ctx);
+
+ switch (set) {
+ case DRBD_NL_SET_DISK_CONF: {
+ struct drbd_disk_conf *d = dst;
+ struct disk_conf c = {
+ .meta_dev_idx = d->meta_dev_idx,
+ .disk_size = d->disk_size,
+ .on_io_error = d->on_io_error,
+ .resync_after = d->resync_after,
+ .al_extents = d->al_extents,
+ .disk_barrier = d->disk_barrier,
+ .disk_flushes = d->disk_flushes,
+ .disk_drain = d->disk_drain,
+ .md_flushes = d->md_flushes,
+ .disk_timeout = d->disk_timeout,
+ .read_balancing = d->read_balancing,
+ .al_updates = d->al_updates,
+ .discard_zeroes_if_aligned = d->discard_zeroes_if_aligned,
+ .rs_discard_granularity = d->rs_discard_granularity,
+ .disable_write_same = d->disable_write_same,
+ };
+ struct nlattr **ntb;
+ int err, ntb_err;
+
+ memcpy(c.backing_dev, d->backing_dev, sizeof(c.backing_dev));
+ c.backing_dev_len = d->backing_dev_len;
+ memcpy(c.meta_dev, d->meta_dev, sizeof(c.meta_dev));
+ c.meta_dev_len = d->meta_dev_len;
+
+ err = disk_conf_from_attrs(&c, info);
+
+ /*
+ * Copy back and stash before returning err: -ENOMSG from a
+ * plain disk-options call (see above) must not discard the
+ * fields it did send. Any other error leaves c at its
+ * pre-seeded values, so copying back is a no-op.
+ */
+ memcpy(d->backing_dev, c.backing_dev, sizeof(d->backing_dev));
+ d->backing_dev_len = c.backing_dev_len;
+ memcpy(d->meta_dev, c.meta_dev, sizeof(d->meta_dev));
+ d->meta_dev_len = c.meta_dev_len;
+ d->meta_dev_idx = c.meta_dev_idx;
+ d->disk_size = c.disk_size;
+ d->on_io_error = c.on_io_error;
+ d->resync_after = c.resync_after;
+ d->al_extents = c.al_extents;
+ d->disk_barrier = c.disk_barrier;
+ d->disk_flushes = c.disk_flushes;
+ d->disk_drain = c.disk_drain;
+ d->md_flushes = c.md_flushes;
+ d->disk_timeout = c.disk_timeout;
+ d->read_balancing = c.read_balancing;
+ d->al_updates = c.al_updates;
+ d->discard_zeroes_if_aligned = c.discard_zeroes_if_aligned;
+ d->rs_discard_granularity = c.rs_discard_granularity;
+ d->disable_write_same = c.disable_write_same;
+ /* max_bio_bvecs: no DRBD 9 equivalent, dropped. */
+
+ /*
+ * The resync-tuning fields belong to struct drbd_peer_device_conf,
+ * fencing to net_conf.fencing_policy; neither is reachable
+ * through this dst. Re-parse the nested table for per-field
+ * presence and stash them for the attach/disk-options
+ * handlers.
+ */
+ ntb_err = disk_conf_ntb_from_attrs(&ntb, info);
+ if ((!ntb_err || ntb_err == -ENOMSG) && ntb) {
+ req->peer_device_conf_84.resync_rate = c.resync_rate;
+ req->peer_device_conf_84.has_resync_rate =
+ ntb[DRBD_A_DISK_CONF_RESYNC_RATE] != NULL;
+ req->peer_device_conf_84.c_plan_ahead = c.c_plan_ahead;
+ req->peer_device_conf_84.has_c_plan_ahead =
+ ntb[DRBD_A_DISK_CONF_C_PLAN_AHEAD] != NULL;
+ req->peer_device_conf_84.c_delay_target = c.c_delay_target;
+ req->peer_device_conf_84.has_c_delay_target =
+ ntb[DRBD_A_DISK_CONF_C_DELAY_TARGET] != NULL;
+ req->peer_device_conf_84.c_fill_target = c.c_fill_target;
+ req->peer_device_conf_84.has_c_fill_target =
+ ntb[DRBD_A_DISK_CONF_C_FILL_TARGET] != NULL;
+ req->peer_device_conf_84.c_max_rate = c.c_max_rate;
+ req->peer_device_conf_84.has_c_max_rate =
+ ntb[DRBD_A_DISK_CONF_C_MAX_RATE] != NULL;
+ req->peer_device_conf_84.c_min_rate = c.c_min_rate;
+ req->peer_device_conf_84.has_c_min_rate =
+ ntb[DRBD_A_DISK_CONF_C_MIN_RATE] != NULL;
+
+ req->fencing_policy_84 = c.fencing;
+ req->has_fencing_policy_84 =
+ ntb[DRBD_A_DISK_CONF_FENCING] != NULL;
+ }
+ kfree(ntb);
+
+ return err;
+ }
+ case DRBD_NL_SET_NET_CONF: {
+ struct drbd_net_conf *d = dst;
+ struct net_conf c = {
+ .wire_protocol = d->wire_protocol,
+ .connect_int = d->connect_int,
+ .timeout = d->timeout,
+ .ping_int = d->ping_int,
+ .ping_timeo = d->ping_timeo,
+ .sndbuf_size = d->sndbuf_size,
+ .rcvbuf_size = d->rcvbuf_size,
+ .ko_count = d->ko_count,
+ .max_buffers = d->max_buffers,
+ .max_epoch_size = d->max_epoch_size,
+ .after_sb_0p = d->after_sb_0p,
+ .after_sb_1p = d->after_sb_1p,
+ .after_sb_2p = d->after_sb_2p,
+ .rr_conflict = d->rr_conflict,
+ .on_congestion = d->on_congestion,
+ .cong_fill = d->cong_fill,
+ .cong_extents = d->cong_extents,
+ .two_primaries = d->two_primaries,
+ .tcp_cork = d->tcp_cork,
+ .always_asbp = d->always_asbp,
+ .use_rle = d->use_rle,
+ .csums_after_crash_only = d->csums_after_crash_only,
+ .sock_check_timeo = d->sock_check_timeo,
+ };
+ struct nlattr **ntb;
+ int err, ntb_err;
+
+ memcpy(c.shared_secret, d->shared_secret, sizeof(c.shared_secret));
+ c.shared_secret_len = d->shared_secret_len;
+ memcpy(c.cram_hmac_alg, d->cram_hmac_alg, sizeof(c.cram_hmac_alg));
+ c.cram_hmac_alg_len = d->cram_hmac_alg_len;
+ memcpy(c.integrity_alg, d->integrity_alg, sizeof(c.integrity_alg));
+ c.integrity_alg_len = d->integrity_alg_len;
+ memcpy(c.verify_alg, d->verify_alg, sizeof(c.verify_alg));
+ c.verify_alg_len = d->verify_alg_len;
+ memcpy(c.csums_alg, d->csums_alg, sizeof(c.csums_alg));
+ c.csums_alg_len = d->csums_alg_len;
+
+ err = net_conf_from_attrs(&c, info);
+
+ /* Copy back and stash before returning err, as for DISK_CONF. */
+ memcpy(d->shared_secret, c.shared_secret, sizeof(d->shared_secret));
+ d->shared_secret_len = c.shared_secret_len;
+ memcpy(d->cram_hmac_alg, c.cram_hmac_alg, sizeof(d->cram_hmac_alg));
+ d->cram_hmac_alg_len = c.cram_hmac_alg_len;
+ memcpy(d->integrity_alg, c.integrity_alg, sizeof(d->integrity_alg));
+ d->integrity_alg_len = c.integrity_alg_len;
+ memcpy(d->verify_alg, c.verify_alg, sizeof(d->verify_alg));
+ d->verify_alg_len = c.verify_alg_len;
+ memcpy(d->csums_alg, c.csums_alg, sizeof(d->csums_alg));
+ d->csums_alg_len = c.csums_alg_len;
+ d->wire_protocol = c.wire_protocol;
+ d->connect_int = c.connect_int;
+ d->timeout = c.timeout;
+ d->ping_int = c.ping_int;
+ d->ping_timeo = c.ping_timeo;
+ d->sndbuf_size = c.sndbuf_size;
+ d->rcvbuf_size = c.rcvbuf_size;
+ d->ko_count = c.ko_count;
+ d->max_buffers = c.max_buffers;
+ d->max_epoch_size = c.max_epoch_size;
+ d->after_sb_0p = c.after_sb_0p;
+ d->after_sb_1p = c.after_sb_1p;
+ d->after_sb_2p = c.after_sb_2p;
+ d->rr_conflict = c.rr_conflict;
+ d->on_congestion = c.on_congestion;
+ d->cong_fill = c.cong_fill;
+ d->cong_extents = c.cong_extents;
+ d->two_primaries = c.two_primaries;
+ d->tcp_cork = c.tcp_cork;
+ d->always_asbp = c.always_asbp;
+ d->use_rle = c.use_rle;
+ d->csums_after_crash_only = c.csums_after_crash_only;
+ d->sock_check_timeo = c.sock_check_timeo;
+
+ /*
+ * fencing is a disk option in 8.4, so "net-options
+ * --set-defaults" must not reset it: keep the live value.
+ * drbd_adm_net_opts() holds conf_update.
+ */
+ if (ctx->set_defaults && ctx->connection) {
+ struct drbd_net_conf *old;
+
+ rcu_read_lock();
+ old = rcu_dereference(ctx->connection->transport.net_conf);
+ if (old)
+ d->fencing_policy = old->fencing_policy;
+ rcu_read_unlock();
+ }
+
+ /*
+ * unplug_watermark is dropped: no DRBD 9 code reads it, and
+ * net_conf is connection-scoped while disk_conf, its DRBD 9
+ * home, is per device. discard_my_data/tentative belong to
+ * struct drbd_connect_parms; stash them for the connect handler.
+ */
+ ntb_err = net_conf_ntb_from_attrs(&ntb, info);
+ if ((!ntb_err || ntb_err == -ENOMSG) && ntb) {
+ req->connect_parms_84.discard_my_data = c.discard_my_data;
+ req->connect_parms_84.has_discard_my_data =
+ ntb[DRBD_A_NET_CONF_DISCARD_MY_DATA] != NULL;
+ req->connect_parms_84.tentative = c.tentative;
+ req->connect_parms_84.has_tentative =
+ ntb[DRBD_A_NET_CONF_TENTATIVE] != NULL;
+ }
+ kfree(ntb);
+
+ /*
+ * fencing lives in the v1 disk_conf on the wire but in
+ * drbd_net_conf.fencing_policy on DRBD 9. The attach/disk-options
+ * handler stashed it while parsing disk_conf and re-enters
+ * this overlay for NET_CONF once it has the connection. A
+ * real NET_CONF request never has this flag set.
+ */
+ if (req->has_fencing_policy_84)
+ d->fencing_policy = req->fencing_policy_84;
+
+ /*
+ * DRBD 9 requires a connection name; v1 has no such concept.
+ * Default it only for a freshly created connection (name_len
+ * == 0), matching the name drbd9 drbdsetup's own compat84 shim
+ * passes (--_name=remote).
+ */
+ if (!d->name_len) {
+ static const char v1_conn_name[] = "remote";
+
+ strscpy(d->name, v1_conn_name, sizeof(d->name));
+ d->name_len = strlen(v1_conn_name);
+ }
+
+ return err;
+ }
+ case DRBD_NL_SET_RES_OPTS: {
+ struct drbd_res_opts *d = dst;
+ struct res_opts c = {
+ .on_no_data = d->on_no_data,
+ };
+ int err;
+
+ memcpy(c.cpu_mask, d->cpu_mask, sizeof(c.cpu_mask));
+ c.cpu_mask_len = d->cpu_mask_len;
+
+ err = res_opts_from_attrs(&c, info);
+
+ /*
+ * The other fourteen DRBD 9 res_opts fields have no v1
+ * attribute, so v1's --set-defaults must not reset them
+ * either: that would drop drbd8_compat_mode and
+ * explicit_drbd8_compat and turn auto_promote back on.
+ * drbd_adm_resource_opts() holds adm_mutex, so the live
+ * res_opts are stable here.
+ */
+ if (ctx->set_defaults && ctx->resource)
+ *d = ctx->resource->res_opts;
+
+ /* Copy back before returning err, as for DISK_CONF. */
+ memcpy(d->cpu_mask, c.cpu_mask, sizeof(d->cpu_mask));
+ d->cpu_mask_len = c.cpu_mask_len;
+ d->on_no_data = c.on_no_data;
+ return err;
+ }
+ case DRBD_NL_SET_SET_ROLE_PARMS: {
+ struct drbd_set_role_parms *d = dst;
+ struct set_role_parms c = { };
+ int err = set_role_parms_from_attrs(&c, info);
+
+ if (err)
+ return err;
+ /* v1's assume_uptodate is drbd-utils 8.9.x's wire name for --force. */
+ d->force = c.assume_uptodate;
+ return 0;
+ }
+ case DRBD_NL_SET_RESIZE_PARMS: {
+ struct drbd_resize_parms *d = dst;
+ /* drbd_adm_resize() pre-fills the current AL layout. */
+ struct resize_parms c = {
+ .resize_size = d->resize_size,
+ .resize_force = d->resize_force,
+ .no_resync = d->no_resync,
+ .al_stripes = d->al_stripes,
+ .al_stripe_size = d->al_stripe_size,
+ };
+ int err = resize_parms_from_attrs(&c, info);
+
+ if (err)
+ return err;
+ d->resize_size = c.resize_size;
+ d->resize_force = c.resize_force;
+ d->no_resync = c.no_resync;
+ d->al_stripes = c.al_stripes;
+ d->al_stripe_size = c.al_stripe_size;
+ return 0;
+ }
+ case DRBD_NL_SET_START_OV_PARMS: {
+ struct drbd_start_ov_parms *d = dst;
+ /* drbd_adm_start_ov() pre-fills the resume position. */
+ struct start_ov_parms c = {
+ .ov_start_sector = d->ov_start_sector,
+ .ov_stop_sector = d->ov_stop_sector,
+ };
+ int err = start_ov_parms_from_attrs(&c, info);
+
+ if (err)
+ return err;
+ d->ov_start_sector = c.ov_start_sector;
+ d->ov_stop_sector = c.ov_stop_sector;
+ return 0;
+ }
+ case DRBD_NL_SET_NEW_C_UUID_PARMS: {
+ struct drbd_new_c_uuid_parms *d = dst;
+ struct new_c_uuid_parms c = { };
+ int err = new_c_uuid_parms_from_attrs(&c, info);
+
+ if (err)
+ return err;
+ d->clear_bm = c.clear_bm;
+ /* force_resync: DRBD 9 only, no v1 attribute; leave untouched. */
+ return 0;
+ }
+ case DRBD_NL_SET_DISCONNECT_PARMS: {
+ struct drbd_disconnect_parms *d = dst;
+ struct disconnect_parms c = { };
+ int err = disconnect_parms_from_attrs(&c, info);
+
+ if (err)
+ return err;
+ d->force_disconnect = c.force_disconnect;
+ return 0;
+ }
+ case DRBD_NL_SET_DETACH_PARMS: {
+ struct drbd_detach_parms *d = dst;
+ struct detach_parms c = { };
+ int err = detach_parms_from_attrs(&c, info);
+
+ if (err)
+ return err;
+ d->force_detach = c.force_detach;
+ /* intentional_diskless_detach: DRBD 9 only; leave untouched. */
+ return 0;
+ }
+ case DRBD_NL_SET_PEER_DEVICE_CONF: {
+ struct drbd_peer_device_conf *d = dst;
+
+ /*
+ * Not parsed from the wire: the DISK_CONF case stashed these
+ * while parsing the disk_conf that carried them. The
+ * attach/disk-options handler re-enters overlay() for
+ * PEER_DEVICE_CONF once it has the peer device.
+ */
+ if (req->peer_device_conf_84.has_resync_rate)
+ d->resync_rate = req->peer_device_conf_84.resync_rate;
+ if (req->peer_device_conf_84.has_c_plan_ahead)
+ d->c_plan_ahead = req->peer_device_conf_84.c_plan_ahead;
+ if (req->peer_device_conf_84.has_c_delay_target)
+ d->c_delay_target = req->peer_device_conf_84.c_delay_target;
+ if (req->peer_device_conf_84.has_c_fill_target)
+ d->c_fill_target = req->peer_device_conf_84.c_fill_target;
+ if (req->peer_device_conf_84.has_c_max_rate)
+ d->c_max_rate = req->peer_device_conf_84.c_max_rate;
+ if (req->peer_device_conf_84.has_c_min_rate)
+ d->c_min_rate = req->peer_device_conf_84.c_min_rate;
+
+ return 0;
+ }
+ case DRBD_NL_SET_PATH_PARMS: {
+ struct drbd_path_parms *d = dst;
+
+ /*
+ * v1 carries the endpoint addresses in DRBD_NLA_CFG_CONTEXT,
+ * parsed once per request into req; drbd_adm_new_path()
+ * calls here once per path.
+ */
+ if (!req->ctx_my_addr_len || !req->ctx_peer_addr_len)
+ return -ENOMSG;
+
+ memcpy(d->my_addr, req->ctx_my_addr, req->ctx_my_addr_len);
+ d->my_addr_len = req->ctx_my_addr_len;
+ memcpy(d->peer_addr, req->ctx_peer_addr, req->ctx_peer_addr_len);
+ d->peer_addr_len = req->ctx_peer_addr_len;
+ return 0;
+ }
+ case DRBD_NL_SET_CONNECT_PARMS: {
+ struct drbd_connect_parms *d = dst;
+
+ /*
+ * Stashed by the NET_CONF case; dst is a fresh zeroed struct,
+ * so an unsent field already defaults to false.
+ */
+ if (req->connect_parms_84.has_discard_my_data)
+ d->discard_my_data = req->connect_parms_84.discard_my_data;
+ if (req->connect_parms_84.has_tentative)
+ d->tentative = req->connect_parms_84.tentative;
+ return 0;
+ }
+ case DRBD_NL_SET_DEVICE_CONF:
+ case DRBD_NL_SET_INVALIDATE_PARMS:
+ case DRBD_NL_SET_INVALIDATE_PEER_PARMS:
+ case DRBD_NL_SET_FORGET_PEER_PARMS:
+ case DRBD_NL_SET_RENAME_RESOURCE_PARMS:
+ case DRBD_NL_SET_SUSPEND_IO_PARMS:
+ case DRBD_NL_SET_CBT_START_PARMS:
+ /* No v1 counterpart: report a missing required attribute
+ * rather than silently proceeding with defaults.
+ */
+ return -ENOMSG;
+ case __DRBD_NL_SET_MAX:
+ break;
+ }
+ return -EINVAL;
+}
+
+/*
+ * The generated parser returns the nested attribute table both on success
+ * and on -ENOMSG (some required attrs missing), which is the typical case
+ * for a change op since drbdsetup omits the invariant attrs. Inspect and
+ * free the table in both cases. It is NULL when the nested container
+ * attribute is absent entirely (command without any options).
+ */
+static bool compat84_attr_present(struct drbd_adm_ctx *ctx, enum drbd_adm_field field)
+{
+ static const struct {
+ int (*ntb_from_attrs)(struct nlattr ***ntb, struct genl_info *info);
+ int attr;
+ const char *name;
+ } invariant[] = {
+ [DRBD_ADM_F_DISK_BACKING_DEV] = { disk_conf_ntb_from_attrs,
+ DRBD_A_DISK_CONF_BACKING_DEV, "DRBD_A_DISK_CONF_BACKING_DEV" },
+ [DRBD_ADM_F_DISK_META_DEV] = { disk_conf_ntb_from_attrs,
+ DRBD_A_DISK_CONF_META_DEV, "DRBD_A_DISK_CONF_META_DEV" },
+ [DRBD_ADM_F_DISK_META_DEV_IDX] = { disk_conf_ntb_from_attrs,
+ DRBD_A_DISK_CONF_META_DEV_IDX, "DRBD_A_DISK_CONF_META_DEV_IDX" },
+ [DRBD_ADM_F_DISK_SIZE] = { disk_conf_ntb_from_attrs,
+ DRBD_A_DISK_CONF_DISK_SIZE, "DRBD_A_DISK_CONF_DISK_SIZE" },
+ };
+ struct nlattr **ntb;
+ bool found;
+ int err;
+
+ /* v1 has no transport name, no load-balance-paths and no explicit
+ * node id: none of these can be an invariant-change attempt.
+ */
+ switch (field) {
+ case DRBD_ADM_F_NET_TRANSPORT_NAME:
+ case DRBD_ADM_F_NET_LOAD_BALANCE_PATHS:
+ case DRBD_ADM_F_RES_NODE_ID:
+ return false;
+ default:
+ break;
+ }
+
+ err = invariant[field].ntb_from_attrs(&ntb, compat84_req(ctx)->info);
+ found = (!err || err == -ENOMSG) && ntb && ntb[invariant[field].attr];
+ kfree(ntb);
+ if (found)
+ pr_info("must not change invariant attr: %s\n", invariant[field].name);
+ return found;
+}
+
+static int compat84_put_timeout_type(struct drbd_adm_ctx *ctx, enum drbd_timeout_flag type)
+{
+ struct compat84_req *req = compat84_req(ctx);
+ struct timeout_parms tp = { .timeout_type = type };
+ int err;
+
+ err = timeout_parms_to_skb(req->reply_skb, &tp);
+ if (err) {
+ nlmsg_free(req->reply_skb);
+ req->reply_skb = NULL;
+ return err;
+ }
+ return NO_ERROR;
+}
+
+/*
+ * The emit_/notify_ hooks below are called unconditionally by the core,
+ * so none may be NULL. notify_path_state and emit_path are permanent
+ * no-ops: DRBD 8.4 has no path objects.
+ */
+
+/*
+ * DRBD_NLA_CFG_CONTEXT for a dump row. v1's container carries
+ * ctx_volume, ctx_resource_name, ctx_my_addr and ctx_peer_addr; no
+ * ctx_peer_node_id or ctx_conn_name (8.4 has neither concept).
+ *
+ * Takes a connection rather than a path: v1 has no path objects at the
+ * wire level, but an 8.4-mode connection carries exactly one transport
+ * path, and its address pair is what mainline's nla_put_drbd_cfg_context()
+ * emits from the connection directly. The rcu_read_lock() is local
+ * because not every caller holds one.
+ */
+static int compat84_put_cfg_context(struct sk_buff *skb, struct drbd_resource *resource,
+ struct drbd_connection *connection,
+ struct drbd_device *device)
+{
+ struct nlattr *nla;
+
+ nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_CONTEXT);
+ if (!nla)
+ goto nla_put_failure;
+ if (device)
+ nla_put_u32(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME, device->vnr);
+ if (resource)
+ nla_put_string(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME, resource->name);
+ if (connection) {
+ struct drbd_path *path;
+
+ rcu_read_lock();
+ path = list_first_or_null_rcu(&connection->transport.paths,
+ struct drbd_path, list);
+ if (path) {
+ if (path->my_addr_len)
+ nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR,
+ path->my_addr_len, &path->my_addr);
+ if (path->peer_addr_len)
+ nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR,
+ path->peer_addr_len, &path->peer_addr);
+ }
+ rcu_read_unlock();
+ }
+ nla_nest_end(skb, nla);
+ return 0;
+
+nla_put_failure:
+ if (nla)
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * The single peer device of an 8.4-mode device, or NULL if none exists
+ * yet. Callers hold rcu_read_lock(). Kept apart from drbd_get_state_84()
+ * (drbd_legacy_84.c), which couples the same lookup to a state fetch.
+ */
+static struct drbd_peer_device *compat84_single_peer_device(struct drbd_device *device)
+{
+ return list_first_or_null_rcu(&device->peer_devices, struct drbd_peer_device,
+ peer_devices);
+}
+
+/*
+ * The single connection of an 8.4-mode resource, or NULL if none exists
+ * yet. Callers hold rcu_read_lock().
+ */
+static struct drbd_connection *compat84_single_connection(struct drbd_resource *resource)
+{
+ return list_first_or_null_rcu(&resource->connections, struct drbd_connection,
+ connections);
+}
+
+/*
+ * DRBD_CPU_MASK_SIZE is 256 on DRBD 9 but 32 in mainline 8.4's drbd.h,
+ * and the vendored v1 policy compiles against the shared uapi header.
+ * drbd-utils 8.9.x rejects a longer cpu_mask with -ERANGE and fails the
+ * whole RESOURCE_OPTS container, so bound what goes on the wire to 8.4's
+ * real limit.
+ */
+#define COMPAT84_CPU_MASK_SIZE 32
+
+/* struct drbd_res_opts -> struct res_opts; shared by GET_RESOURCES and GET_STATUS. */
+static void compat84_pack_res_opts_84(struct res_opts *out, struct drbd_res_opts *res_opts)
+{
+ out->on_no_data = res_opts->on_no_data;
+ memcpy(out->cpu_mask, res_opts->cpu_mask, sizeof(out->cpu_mask));
+ out->cpu_mask_len = res_opts->cpu_mask_len;
+ if (out->cpu_mask_len > COMPAT84_CPU_MASK_SIZE - 1) {
+ out->cpu_mask_len = COMPAT84_CPU_MASK_SIZE - 1;
+ out->cpu_mask[COMPAT84_CPU_MASK_SIZE - 1] = 0;
+ }
+}
+
+static int compat84_emit_resource(struct sk_buff *skb, struct netlink_callback *cb,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *info,
+ struct drbd_resource_statistics *statistics)
+{
+ struct resource_info info_84 = {
+ .res_role = info->res_role,
+ .res_susp = info->res_susp,
+ .res_susp_nod = info->res_susp_nod,
+ .res_susp_fen = info->res_susp_fen,
+ /* res_susp_quorum, res_fail_io: DRBD 9 only, no v1 attribute. */
+ };
+ struct resource_statistics statistics_84 = {
+ .res_stat_write_ordering = statistics->res_stat_write_ordering,
+ };
+ struct res_opts res_opts;
+ struct drbd_genlmsghdr *dh;
+ int err;
+
+ compat84_pack_res_opts_84(&res_opts, &resource->res_opts);
+
+ dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, &drbd_nl_family,
+ NLM_F_MULTI, DRBD_ADM_GET_RESOURCES);
+ if (!dh)
+ return -ENOMEM;
+ dh->minor = -1U;
+ dh->ret_code = NO_ERROR;
+ err = compat84_put_cfg_context(skb, resource, NULL, NULL);
+ if (err)
+ return err;
+ err = res_opts_to_skb(skb, &res_opts);
+ if (err)
+ return err;
+ err = resource_info_to_skb(skb, &info_84);
+ if (err)
+ return err;
+ err = resource_statistics_to_skb(skb, &statistics_84);
+ if (err)
+ return err;
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+/*
+ * struct drbd_disk_conf -> struct disk_conf. fencing and the resync-tuning
+ * fields live in net_conf and struct drbd_peer_device_conf on DRBD 9, so they
+ * are sourced from @peer_device, 0 before one exists. Shared by
+ * GET_DEVICES and GET_STATUS.
+ */
+static void compat84_pack_disk_conf_84(struct disk_conf *dc, struct drbd_disk_conf *disk_conf,
+ struct drbd_peer_device *peer_device)
+{
+ struct drbd_peer_device_conf *pdc = peer_device ? rcu_dereference(peer_device->conf) : NULL;
+ struct drbd_net_conf *nc = peer_device ?
+ rcu_dereference(peer_device->connection->transport.net_conf) : NULL;
+
+ *dc = (struct disk_conf){
+ .meta_dev_idx = disk_conf->meta_dev_idx,
+ .disk_size = disk_conf->disk_size,
+ .on_io_error = disk_conf->on_io_error,
+ .fencing = nc ? nc->fencing_policy : 0,
+ .resync_rate = pdc ? pdc->resync_rate : 0,
+ .resync_after = disk_conf->resync_after,
+ .al_extents = disk_conf->al_extents,
+ .c_plan_ahead = pdc ? pdc->c_plan_ahead : 0,
+ .c_delay_target = pdc ? pdc->c_delay_target : 0,
+ .c_fill_target = pdc ? pdc->c_fill_target : 0,
+ .c_max_rate = pdc ? pdc->c_max_rate : 0,
+ .c_min_rate = pdc ? pdc->c_min_rate : 0,
+ .disk_barrier = disk_conf->disk_barrier,
+ .disk_flushes = disk_conf->disk_flushes,
+ .disk_drain = disk_conf->disk_drain,
+ .md_flushes = disk_conf->md_flushes,
+ .disk_timeout = disk_conf->disk_timeout,
+ .read_balancing = disk_conf->read_balancing,
+ .al_updates = disk_conf->al_updates,
+ .discard_zeroes_if_aligned = disk_conf->discard_zeroes_if_aligned,
+ .rs_discard_granularity = disk_conf->rs_discard_granularity,
+ .disable_write_same = disk_conf->disable_write_same,
+ /* max_bio_bvecs: no DRBD 9 equivalent, left 0. */
+ };
+
+ memcpy(dc->backing_dev, disk_conf->backing_dev, sizeof(dc->backing_dev));
+ dc->backing_dev_len = disk_conf->backing_dev_len;
+ memcpy(dc->meta_dev, disk_conf->meta_dev, sizeof(dc->meta_dev));
+ dc->meta_dev_len = disk_conf->meta_dev_len;
+}
+
+static int compat84_emit_device(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_device *device,
+ struct drbd_disk_conf *disk_conf,
+ struct drbd_device_info *info,
+ struct drbd_device_statistics *statistics)
+{
+ struct drbd_genlmsghdr *dh;
+ int err;
+
+ dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, &drbd_nl_family,
+ NLM_F_MULTI, DRBD_ADM_GET_DEVICES);
+ if (!dh)
+ return -ENOMEM;
+ dh->ret_code = retcode;
+ dh->minor = -1U;
+ if (retcode != NO_ERROR) {
+ genlmsg_end(skb, dh);
+ return 0;
+ }
+
+ dh->minor = device->minor;
+ err = compat84_put_cfg_context(skb, device->resource, NULL, device);
+ if (err)
+ return err;
+
+ /* v1 has no DRBD_NLA_DEVICE_CONF container. */
+
+ if (disk_conf) {
+ struct drbd_peer_device *peer_device = compat84_single_peer_device(device);
+ struct disk_conf dc;
+
+ compat84_pack_disk_conf_84(&dc, disk_conf, peer_device);
+
+ err = disk_conf_to_skb(skb, &dc);
+ if (err)
+ return err;
+ }
+
+ {
+ struct device_info info_84 = {
+ .dev_disk_state = drbd_disk_state_84(info->dev_disk_state),
+ /* is_intentional_diskless, dev_has_quorum, dev_is_open,
+ * backing_dev_path: DRBD 9 only, no v1 attribute.
+ */
+ };
+
+ err = device_info_to_skb(skb, &info_84);
+ if (err)
+ return err;
+ }
+ {
+ struct device_statistics stat_84 = {
+ .dev_size = statistics->dev_size,
+ .dev_read = statistics->dev_read,
+ .dev_write = statistics->dev_write,
+ .dev_al_writes = statistics->dev_al_writes,
+ .dev_bm_writes = statistics->dev_bm_writes,
+ .dev_upper_pending = statistics->dev_upper_pending,
+ .dev_lower_pending = statistics->dev_lower_pending,
+ .dev_upper_blocked = statistics->dev_upper_blocked,
+ .dev_lower_blocked = statistics->dev_lower_blocked,
+ .dev_al_suspended = statistics->dev_al_suspended,
+ .dev_exposed_data_uuid = statistics->dev_exposed_data_uuid,
+ .dev_current_uuid = statistics->dev_current_uuid,
+ .dev_disk_flags = statistics->dev_disk_flags & MDF_84_MASK,
+ .history_uuids_len = statistics->history_uuids_len,
+ };
+
+ BUILD_BUG_ON(sizeof(stat_84.history_uuids) != sizeof(statistics->history_uuids));
+ memcpy(stat_84.history_uuids, statistics->history_uuids,
+ sizeof(stat_84.history_uuids));
+
+ err = device_statistics_to_skb(skb, &stat_84);
+ if (err)
+ return err;
+ }
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+/*
+ * struct drbd_net_conf -> struct net_conf. Does not mask shared_secret;
+ * callers reachable by an unprivileged listener use
+ * compat84_put_net_conf_masked(). Shared by GET_CONNECTIONS and
+ * GET_STATUS.
+ *
+ * unplug_watermark has no persistent DRBD 9 home, so it is reported as
+ * 8.4's compiled-in default rather than 0: drbdsetup-84 show prints any
+ * non-default value, and drbdadm-84 adjust would then reissue
+ * net-options --set-defaults on every run. A non-default value
+ * configured on genuine 8.4 cannot be reflected back.
+ */
+static void compat84_pack_net_conf_84(struct net_conf *out, struct drbd_net_conf *net_conf)
+{
+ *out = (struct net_conf){
+ .wire_protocol = net_conf->wire_protocol,
+ .unplug_watermark = DRBD_UNPLUG_WATERMARK_DEF,
+ .connect_int = net_conf->connect_int,
+ .timeout = net_conf->timeout,
+ .ping_int = net_conf->ping_int,
+ .ping_timeo = net_conf->ping_timeo,
+ .sndbuf_size = net_conf->sndbuf_size,
+ .rcvbuf_size = net_conf->rcvbuf_size,
+ .ko_count = net_conf->ko_count,
+ .max_buffers = net_conf->max_buffers,
+ .max_epoch_size = net_conf->max_epoch_size,
+ .after_sb_0p = net_conf->after_sb_0p,
+ .after_sb_1p = net_conf->after_sb_1p,
+ .after_sb_2p = net_conf->after_sb_2p,
+ .rr_conflict = net_conf->rr_conflict,
+ .on_congestion = net_conf->on_congestion,
+ .cong_fill = net_conf->cong_fill,
+ .cong_extents = net_conf->cong_extents,
+ .two_primaries = net_conf->two_primaries,
+ .tcp_cork = net_conf->tcp_cork,
+ .always_asbp = net_conf->always_asbp,
+ .use_rle = net_conf->use_rle,
+ .csums_after_crash_only = net_conf->csums_after_crash_only,
+ .sock_check_timeo = net_conf->sock_check_timeo,
+ /* discard_my_data/tentative: one-shot connect flags, no persistent home. */
+ };
+
+ memcpy(out->shared_secret, net_conf->shared_secret, sizeof(out->shared_secret));
+ out->shared_secret_len = net_conf->shared_secret_len;
+ memcpy(out->cram_hmac_alg, net_conf->cram_hmac_alg, sizeof(out->cram_hmac_alg));
+ out->cram_hmac_alg_len = net_conf->cram_hmac_alg_len;
+ memcpy(out->integrity_alg, net_conf->integrity_alg, sizeof(out->integrity_alg));
+ out->integrity_alg_len = net_conf->integrity_alg_len;
+ memcpy(out->verify_alg, net_conf->verify_alg, sizeof(out->verify_alg));
+ out->verify_alg_len = net_conf->verify_alg_len;
+ memcpy(out->csums_alg, net_conf->csums_alg, sizeof(out->csums_alg));
+ out->csums_alg_len = net_conf->csums_alg_len;
+}
+
+/*
+ * Pack and serialize @net_conf, zeroing the shared secret when
+ * @exclude_sensitive: a reply reachable by an unprivileged caller must not
+ * carry it (GET_STATUS and GET_CONNECTIONS carry no GENL_ADMIN_PERM).
+ */
+static int compat84_put_net_conf_masked(struct sk_buff *skb, struct drbd_net_conf *net_conf,
+ bool exclude_sensitive)
+{
+ struct net_conf nc;
+
+ compat84_pack_net_conf_84(&nc, net_conf);
+ if (exclude_sensitive) {
+ memset(nc.shared_secret, 0, sizeof(nc.shared_secret));
+ nc.shared_secret_len = 0;
+ }
+ return net_conf_to_skb(skb, &nc);
+}
+
+static int compat84_emit_connection(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_resource *resource,
+ struct drbd_connection *connection,
+ struct drbd_net_conf *net_conf,
+ struct drbd_connection_info *info,
+ struct drbd_connection_statistics *statistics)
+{
+ struct drbd_genlmsghdr *dh;
+ int err;
+
+ dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, &drbd_nl_family,
+ NLM_F_MULTI, DRBD_ADM_GET_CONNECTIONS);
+ if (!dh)
+ return -ENOMEM;
+ dh->ret_code = retcode;
+ dh->minor = -1U;
+ if (retcode != NO_ERROR) {
+ genlmsg_end(skb, dh);
+ return 0;
+ }
+
+ err = compat84_put_cfg_context(skb, resource, connection, NULL);
+ if (err)
+ return err;
+
+ /* v1 has no DRBD_NLA_PATH_PARMS container. */
+
+ if (net_conf) {
+ /* This dump carries no GENL_ADMIN_PERM. */
+ err = compat84_put_net_conf_masked(skb, net_conf, !capable(CAP_SYS_ADMIN));
+ if (err)
+ return err;
+ }
+ {
+ struct connection_info info_84 = {
+ .conn_connection_state = info->conn_connection_state,
+ .conn_role = info->conn_role,
+ };
+
+ err = connection_info_to_skb(skb, &info_84);
+ if (err)
+ return err;
+ }
+ {
+ struct connection_statistics stat_84 = {
+ .conn_congested = statistics->conn_congested,
+ /* ap_in_flight, rs_in_flight: DRBD 9 only, no v1 attribute. */
+ };
+
+ err = connection_statistics_to_skb(skb, &stat_84);
+ if (err)
+ return err;
+ }
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+static int compat84_emit_peer_device(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_peer_device *peer_device, unsigned int minor,
+ struct drbd_peer_device_info *info,
+ struct drbd_peer_device_statistics *statistics,
+ struct drbd_peer_device_conf *conf)
+{
+ struct drbd_genlmsghdr *dh;
+ int err;
+
+ dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, &drbd_nl_family,
+ NLM_F_MULTI, DRBD_ADM_GET_PEER_DEVICES);
+ if (!dh)
+ return -ENOMEM;
+ dh->ret_code = retcode;
+ dh->minor = -1U;
+ if (retcode != NO_ERROR) {
+ genlmsg_end(skb, dh);
+ return 0;
+ }
+
+ dh->minor = minor;
+ err = compat84_put_cfg_context(skb, peer_device->device->resource,
+ peer_device->connection, peer_device->device);
+ if (err)
+ return err;
+
+ {
+ struct peer_device_info info_84 = {
+ .peer_repl_state = info->peer_repl_state,
+ .peer_disk_state = drbd_disk_state_84(info->peer_disk_state),
+ .peer_resync_susp_user = info->peer_resync_susp_user,
+ .peer_resync_susp_peer = info->peer_resync_susp_peer,
+ .peer_resync_susp_dependency = info->peer_resync_susp_dependency,
+ /* peer_is_intentional_diskless, peer_resync_susp_max_parallel:
+ * DRBD 9 only, no v1 attribute.
+ */
+ };
+
+ err = peer_device_info_to_skb(skb, &info_84);
+ if (err)
+ return err;
+ }
+ {
+ struct peer_device_statistics stat_84 = {
+ .peer_dev_received = statistics->peer_dev_received,
+ .peer_dev_sent = statistics->peer_dev_sent,
+ .peer_dev_pending = statistics->peer_dev_pending,
+ .peer_dev_unacked = statistics->peer_dev_unacked,
+ .peer_dev_out_of_sync = statistics->peer_dev_out_of_sync,
+ .peer_dev_resync_failed = statistics->peer_dev_resync_failed,
+ .peer_dev_bitmap_uuid = statistics->peer_dev_bitmap_uuid,
+ .peer_dev_flags = statistics->peer_dev_flags & PEER_DEV_FLAGS_84_MASK,
+ /* the fifteen resync/OV telemetry fields past this point:
+ * DRBD 9 only, no v1 attribute.
+ */
+ };
+
+ err = peer_device_statistics_to_skb(skb, &stat_84);
+ if (err)
+ return err;
+ }
+
+ /*
+ * v1 has no DRBD_NLA_PEER_DEVICE_OPTS container; the resync-tuning
+ * fields ride on GET_DEVICES's disk_conf instead.
+ */
+
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+/* Permanent no-op: DRBD 8.4 has no path objects. */
+static int compat84_emit_path(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_resource *resource, struct drbd_connection *connection,
+ struct drbd_path *path, struct drbd_nl_path_info *info)
+{
+ return 0;
+}
+
+/*
+ * Notifications run from process context (state-change work, admin paths,
+ * drbd_khelper()), never atomic, so GFP_NOIO as in the v2 adapter.
+ */
+
+static int compat84_put_notification_header(struct sk_buff *msg,
+ enum drbd_notification_type type)
+{
+ struct drbd_notification_header nh = {
+ .nh_type = type,
+ };
+
+ return drbd_notification_header_to_skb(msg, &nh);
+}
+
+static int compat84_notify_resource_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *info,
+ struct drbd_rename_resource_info *rename_info,
+ enum drbd_notification_type type)
+{
+ struct drbd_resource_statistics resource_statistics;
+ struct drbd_genlmsghdr *dh;
+ bool multicast = false;
+ int err;
+
+ /*
+ * v1 has no rename notification: NOTIFY_RENAME is DRBD 9 only and
+ * a v1 client cannot parse it. The only caller passes skb == NULL.
+ */
+ if ((type & ~NOTIFY_FLAGS) == NOTIFY_RENAME)
+ return 0;
+
+ if (!skb) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ multicast = true;
+ seq = atomic_inc_return(&compat84_notify_seq);
+ }
+
+ err = -EMSGSIZE;
+ dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_RESOURCE_STATE);
+ if (!dh)
+ goto nla_put_failure;
+ dh->minor = -1U;
+ dh->ret_code = NO_ERROR;
+ if (compat84_put_cfg_context(skb, resource, NULL, NULL) ||
+ compat84_put_notification_header(skb, type))
+ goto nla_put_failure;
+
+ if (info) {
+ struct resource_info info_84 = {
+ .res_role = info->res_role,
+ .res_susp = info->res_susp,
+ .res_susp_nod = info->res_susp_nod,
+ .res_susp_fen = info->res_susp_fen,
+ /* res_susp_quorum, res_fail_io: DRBD 9 only, no v1 attribute. */
+ };
+
+ err = resource_info_to_skb(skb, &info_84);
+ if (err)
+ goto nla_put_failure;
+ }
+
+ resource_to_statistics(&resource_statistics, resource);
+ {
+ struct resource_statistics statistics_84 = {
+ .res_stat_write_ordering = resource_statistics.res_stat_write_ordering,
+ };
+
+ err = resource_statistics_to_skb(skb, &statistics_84);
+ if (err)
+ goto nla_put_failure;
+ }
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = compat84_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+
+ /*
+ * Fused SIB_STATE_CHANGE DRBD_EVENT, see compat84_event_get(): a
+ * role/suspend change is resource-wide, but 8.4's wire word is per
+ * device. Multicast path only; the initial-state replay appends to a
+ * dump. !info means NOTIFY_DESTROY, which has no transition to report.
+ */
+ if (multicast && info && (type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY) {
+ struct drbd_device *device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ union drbd_state prev, new;
+
+ compat84_event_get(device, &prev, &new);
+ prev.role = info->old_res_role;
+ new.role = info->res_role;
+ prev.susp = info->old_res_susp;
+ new.susp = info->res_susp || info->res_susp_quorum;
+ prev.susp_nod = info->old_res_susp_nod;
+ new.susp_nod = info->res_susp_nod;
+ prev.susp_fen = info->old_res_susp_fen;
+ new.susp_fen = info->res_susp_fen;
+ compat84_event_put(device, prev, new);
+ }
+ rcu_read_unlock();
+ }
+ if (multicast)
+ compat84_event_flush(resource, type);
+
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(resource, "Error %d while broadcasting event. Event seq:%u\n",
+ err, seq);
+ return err;
+}
+
+static int compat84_notify_device_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_device *device,
+ struct drbd_device_info *info,
+ enum drbd_notification_type type)
+{
+ struct drbd_device_statistics device_statistics;
+ struct device_statistics statistics_84;
+ struct drbd_genlmsghdr *dh;
+ bool multicast = false;
+ int err;
+
+ if (!skb) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ multicast = true;
+ seq = atomic_inc_return(&compat84_notify_seq);
+ }
+
+ err = -EMSGSIZE;
+ dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_DEVICE_STATE);
+ if (!dh)
+ goto nla_put_failure;
+ dh->minor = device->minor;
+ dh->ret_code = NO_ERROR;
+ if (compat84_put_cfg_context(skb, device->resource, NULL, device) ||
+ compat84_put_notification_header(skb, type))
+ goto nla_put_failure;
+ if ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY) {
+ struct device_info info_84 = {
+ .dev_disk_state = drbd_disk_state_84(info->dev_disk_state),
+ /* is_intentional_diskless, dev_has_quorum, dev_is_open,
+ * backing_dev_path: DRBD 9 only, no v1 attribute.
+ */
+ };
+
+ if (device_info_to_skb(skb, &info_84))
+ goto nla_put_failure;
+ }
+ device_to_statistics(&device_statistics, device);
+ statistics_84 = (struct device_statistics) {
+ .dev_size = device_statistics.dev_size,
+ .dev_read = device_statistics.dev_read,
+ .dev_write = device_statistics.dev_write,
+ .dev_al_writes = device_statistics.dev_al_writes,
+ .dev_bm_writes = device_statistics.dev_bm_writes,
+ .dev_upper_pending = device_statistics.dev_upper_pending,
+ .dev_lower_pending = device_statistics.dev_lower_pending,
+ .dev_upper_blocked = device_statistics.dev_upper_blocked,
+ .dev_lower_blocked = device_statistics.dev_lower_blocked,
+ .dev_al_suspended = device_statistics.dev_al_suspended,
+ .dev_exposed_data_uuid = device_statistics.dev_exposed_data_uuid,
+ .dev_current_uuid = device_statistics.dev_current_uuid,
+ .dev_disk_flags = device_statistics.dev_disk_flags & MDF_84_MASK,
+ .history_uuids_len = device_statistics.history_uuids_len,
+ };
+ BUILD_BUG_ON(sizeof(statistics_84.history_uuids) !=
+ sizeof(device_statistics.history_uuids));
+ memcpy(statistics_84.history_uuids, device_statistics.history_uuids,
+ sizeof(statistics_84.history_uuids));
+ err = device_statistics_to_skb(skb, &statistics_84);
+ if (err)
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = compat84_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+
+ /* Fused SIB_STATE_CHANGE DRBD_EVENT, see compat84_event_get(). */
+ if (multicast && info && (type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY) {
+ union drbd_state prev, new;
+
+ compat84_event_get(device, &prev, &new);
+ prev.disk = drbd_disk_state_84(info->old_dev_disk_state);
+ new.disk = drbd_disk_state_84(info->dev_disk_state);
+ compat84_event_put(device, prev, new);
+ }
+ if (multicast)
+ compat84_event_flush(device->resource, type);
+
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(device, "Error %d while broadcasting event. Event seq:%u\n",
+ err, seq);
+ return err;
+}
+
+static int compat84_notify_connection_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_connection *connection,
+ struct drbd_connection_info *info,
+ enum drbd_notification_type type)
+{
+ struct drbd_connection_statistics connection_statistics;
+ struct drbd_genlmsghdr *dh;
+ bool multicast = false;
+ int err;
+
+ if (!skb) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ multicast = true;
+ seq = atomic_inc_return(&compat84_notify_seq);
+ }
+
+ err = -EMSGSIZE;
+ dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_CONNECTION_STATE);
+ if (!dh)
+ goto nla_put_failure;
+ dh->minor = -1U;
+ dh->ret_code = NO_ERROR;
+ if (compat84_put_cfg_context(skb, connection->resource, connection, NULL) ||
+ compat84_put_notification_header(skb, type))
+ goto nla_put_failure;
+ if ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY) {
+ struct connection_info info_84 = {
+ .conn_connection_state = info->conn_connection_state,
+ .conn_role = info->conn_role,
+ };
+
+ if (connection_info_to_skb(skb, &info_84))
+ goto nla_put_failure;
+ }
+ /* v1 has no DRBD_NLA_PATH_PARMS container. */
+ connection_to_statistics(&connection_statistics, connection);
+ {
+ struct connection_statistics statistics_84 = {
+ .conn_congested = connection_statistics.conn_congested,
+ /* ap_in_flight, rs_in_flight: DRBD 9 only, no v1 attribute. */
+ };
+
+ err = connection_statistics_to_skb(skb, &statistics_84);
+ if (err)
+ goto nla_put_failure;
+ }
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = compat84_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+
+ /*
+ * Fused SIB_STATE_CHANGE DRBD_EVENT, see compat84_event_get(), for
+ * every device behind this connection. state.conn takes the cstate
+ * only when the peer device is not itself replicating (repl_state <=
+ * L_OFF), matching combined_conn_state(). A peer-device notification
+ * in the same state change knows the replication state before and
+ * after it, so the cstate is recorded for it to decide with.
+ */
+ if (multicast && info && (type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY) {
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ union drbd_state prev, new;
+
+ compat84_event_get(peer_device->device, &prev, &new);
+ prev.peer = info->old_conn_role;
+ new.peer = info->conn_role;
+ if (peer_device->repl_state[NOW] <= L_OFF) {
+ prev.conn = info->old_conn_connection_state;
+ new.conn = info->conn_connection_state;
+ }
+ compat84_event_put(peer_device->device, prev, new);
+ peer_device->device->bcast_cstate_prev_84 = info->old_conn_connection_state;
+ peer_device->device->bcast_cstate_new_84 = info->conn_connection_state;
+ }
+ rcu_read_unlock();
+ }
+ if (multicast)
+ compat84_event_flush(connection->resource, type);
+
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(connection, "Error %d while broadcasting event. Event seq:%u\n",
+ err, seq);
+ return err;
+}
+
+static int compat84_notify_peer_device_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_info *info,
+ enum drbd_notification_type type)
+{
+ struct drbd_peer_device_statistics peer_device_statistics;
+ struct peer_device_statistics statistics_84;
+ struct drbd_resource *resource = peer_device->device->resource;
+ struct drbd_genlmsghdr *dh;
+ bool multicast = false;
+ int err;
+
+ if (!skb) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ multicast = true;
+ seq = atomic_inc_return(&compat84_notify_seq);
+ }
+
+ err = -EMSGSIZE;
+ dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_PEER_DEVICE_STATE);
+ if (!dh)
+ goto nla_put_failure;
+ dh->minor = -1U;
+ dh->ret_code = NO_ERROR;
+ if (compat84_put_cfg_context(skb, resource, peer_device->connection, peer_device->device) ||
+ compat84_put_notification_header(skb, type))
+ goto nla_put_failure;
+ if ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY) {
+ struct peer_device_info info_84 = {
+ .peer_repl_state = info->peer_repl_state,
+ .peer_disk_state = drbd_disk_state_84(info->peer_disk_state),
+ .peer_resync_susp_user = info->peer_resync_susp_user,
+ .peer_resync_susp_peer = info->peer_resync_susp_peer,
+ .peer_resync_susp_dependency = info->peer_resync_susp_dependency,
+ /* peer_is_intentional_diskless, peer_resync_susp_max_parallel:
+ * DRBD 9 only, no v1 attribute.
+ */
+ };
+
+ if (peer_device_info_to_skb(skb, &info_84))
+ goto nla_put_failure;
+ }
+ peer_device_to_statistics(&peer_device_statistics, peer_device);
+ statistics_84 = (struct peer_device_statistics) {
+ .peer_dev_received = peer_device_statistics.peer_dev_received,
+ .peer_dev_sent = peer_device_statistics.peer_dev_sent,
+ .peer_dev_pending = peer_device_statistics.peer_dev_pending,
+ .peer_dev_unacked = peer_device_statistics.peer_dev_unacked,
+ .peer_dev_out_of_sync = peer_device_statistics.peer_dev_out_of_sync,
+ .peer_dev_resync_failed = peer_device_statistics.peer_dev_resync_failed,
+ .peer_dev_bitmap_uuid = peer_device_statistics.peer_dev_bitmap_uuid,
+ .peer_dev_flags = peer_device_statistics.peer_dev_flags & PEER_DEV_FLAGS_84_MASK,
+ /* the fifteen resync/OV telemetry fields past this point:
+ * DRBD 9 only, no v1 attribute.
+ */
+ };
+ err = peer_device_statistics_to_skb(skb, &statistics_84);
+ if (err)
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = compat84_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+
+ /*
+ * Fused SIB_STATE_CHANGE DRBD_EVENT, see compat84_event_get().
+ * state.conn takes the repl_state while it is actively replicating
+ * (> L_OFF) and otherwise the cstate the connection hook recorded,
+ * matching combined_conn_state(). The frequent RS_PROGRESS notifications
+ * arrive here with old == new and send nothing.
+ *
+ * aftr_isp comes from peer_resync_susp_dependency on both sides; see
+ * peer_device_state_change_to_info() for why the old side is exact.
+ */
+ if (multicast && info && (type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY) {
+ struct drbd_device *device = peer_device->device;
+ union drbd_state prev, new;
+
+ compat84_event_get(device, &prev, &new);
+ prev.pdsk = drbd_disk_state_84(info->old_peer_disk_state);
+ new.pdsk = drbd_disk_state_84(info->peer_disk_state);
+ if (info->old_peer_repl_state > L_OFF)
+ prev.conn = info->old_peer_repl_state;
+ else if (device->bcast_cstate_prev_84 >= 0)
+ prev.conn = device->bcast_cstate_prev_84;
+ if (info->peer_repl_state > L_OFF)
+ new.conn = info->peer_repl_state;
+ else if (device->bcast_cstate_new_84 >= 0)
+ new.conn = device->bcast_cstate_new_84;
+ prev.user_isp = info->old_peer_resync_susp_user;
+ new.user_isp = info->peer_resync_susp_user;
+ prev.peer_isp = info->old_peer_resync_susp_peer;
+ new.peer_isp = info->peer_resync_susp_peer;
+ prev.aftr_isp = info->old_peer_resync_susp_dependency;
+ new.aftr_isp = info->peer_resync_susp_dependency;
+ compat84_event_put(device, prev, new);
+ }
+ if (multicast)
+ compat84_event_flush(peer_device->device->resource, type);
+
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(peer_device, "Error %d while broadcasting event. Event seq:%u\n",
+ err, seq);
+ return err;
+}
+
+/* Permanent no-op: DRBD 8.4 has no path objects. */
+static int compat84_notify_path_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_connection *connection, struct drbd_path *path,
+ struct drbd_nl_path_info *info,
+ enum drbd_notification_type type)
+{
+ return 0;
+}
+
+/* Helper events are never replayed, so "skb" is always NULL here. */
+static int compat84_notify_helper(struct sk_buff *skb, unsigned int seq,
+ struct drbd_device *device, struct drbd_connection *connection,
+ const char *name, int status,
+ enum drbd_notification_type type)
+{
+ struct drbd_resource *resource = device ? device->resource : connection->resource;
+ struct drbd_helper_info helper_info;
+ struct drbd_genlmsghdr *dh;
+ int err;
+
+ strscpy(helper_info.helper_name, name, sizeof(helper_info.helper_name));
+ helper_info.helper_name_len = min(strlen(name), sizeof(helper_info.helper_name));
+ helper_info.helper_status = status;
+
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto fail;
+
+ err = -EMSGSIZE;
+ seq = atomic_inc_return(&compat84_notify_seq);
+ dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_HELPER);
+ if (!dh)
+ goto fail;
+ dh->minor = device ? device->minor : -1;
+ dh->ret_code = NO_ERROR;
+ if (compat84_put_cfg_context(skb, resource, connection, device) ||
+ compat84_put_notification_header(skb, type) ||
+ drbd_helper_info_to_skb(skb, &helper_info))
+ goto fail;
+ genlmsg_end(skb, dh);
+ err = compat84_genl_multicast_events(skb);
+ skb = NULL;
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto fail;
+
+ /*
+ * Fused SIB_HELPER_PRE/SIB_HELPER_POST DRBD_EVENT: drbd_khelper()
+ * notifies with NOTIFY_CALL before running the helper and
+ * NOTIFY_RESPONSE after, matching mainline's two drbd_bcast_event()
+ * calls. A connection-scoped helper (fence-peer) has no device, so
+ * fan out over every device behind the connection. Never deduped:
+ * every helper invocation is reported.
+ */
+ {
+ enum drbd_state_info_bcast_reason reason =
+ (type & ~NOTIFY_FLAGS) == NOTIFY_CALL ? SIB_HELPER_PRE : SIB_HELPER_POST;
+
+ if (device) {
+ compat84_emit_event(device, reason, 0, 0, name, status);
+ } else {
+ struct drbd_peer_device *peer_device;
+ int vnr;
+
+ rcu_read_lock();
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ struct drbd_device *dev = peer_device->device;
+
+ kref_get(&dev->kref);
+ rcu_read_unlock();
+ compat84_emit_event(dev, reason, 0, 0, name, status);
+ kref_put(&dev->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+ }
+ }
+
+ return 0;
+
+fail:
+ nlmsg_free(skb);
+ drbd_err(resource, "Error %d while broadcasting event. Event seq:%u\n",
+ err, seq);
+ return err;
+}
+
+/*
+ * SIB_SYNC_PROGRESS DRBD_EVENT, called from update_on_disk_bitmap()
+ * (drbd_sender.c), mainline 8.4's own call site for it. Always sent,
+ * although the packed state word does not move during a resync; the
+ * call site itself is throttled by
+ * drbd_lazy_bitmap_update_due().
+ */
+void compat84_notify_sync_progress(struct drbd_peer_device *peer_device)
+{
+ compat84_emit_event(peer_device->device, SIB_SYNC_PROGRESS, 0, 0, NULL, 0);
+}
+
+static int compat84_notify_initial_state_done(struct sk_buff *skb, unsigned int seq)
+{
+ struct drbd_genlmsghdr *dh;
+ int err;
+
+ err = -EMSGSIZE;
+ dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_INITIAL_STATE_DONE);
+ if (!dh)
+ goto nla_put_failure;
+ dh->minor = -1U;
+ dh->ret_code = NO_ERROR;
+ if (compat84_put_notification_header(skb, NOTIFY_EXISTS))
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+ pr_err("Error %d sending event. Event seq:%u\n", err, seq);
+ return err;
+}
+
+static const struct drbd_nl_dialect drbd_nl_84_dialect = {
+ .name = "drbd-8.4",
+ .has_set = compat84_has_set,
+ .overlay = compat84_overlay,
+ .attr_present = compat84_attr_present,
+ .put_timeout_type = compat84_put_timeout_type,
+ .emit_resource = compat84_emit_resource,
+ .emit_device = compat84_emit_device,
+ .emit_connection = compat84_emit_connection,
+ .emit_peer_device = compat84_emit_peer_device,
+ .emit_path = compat84_emit_path,
+ .notify_resource_state = compat84_notify_resource_state,
+ .notify_device_state = compat84_notify_device_state,
+ .notify_connection_state = compat84_notify_connection_state,
+ .notify_peer_device_state = compat84_notify_peer_device_state,
+ .notify_path_state = compat84_notify_path_state,
+ .notify_helper = compat84_notify_helper,
+ .notify_initial_state_done = compat84_notify_initial_state_done,
+};
+
+int drbd_adm_dump_devices_done(struct netlink_callback *cb)
+{
+ return drbd_dump_devices_done(cb);
+}
+
+int drbd_adm_dump_connections_done(struct netlink_callback *cb)
+{
+ return drbd_dump_connections_done(cb);
+}
+
+int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb)
+{
+ return drbd_dump_peer_devices_done(cb);
+}
+
+/*
+ * Ported from drbd_nl_legacy.c: dump callbacks run outside genl_lock(),
+ * so they cannot use the attribute parsing that relies on global tables.
+ * The attribute type numbers are the same in both dialects.
+ */
+static struct nlattr *compat84_find_cfg_context_attr(const struct nlmsghdr *nlh, int attr)
+{
+ const unsigned int hdrlen = GENL_HDRLEN + sizeof(struct drbd_genlmsghdr);
+ struct nlattr *nla;
+
+ nla = nla_find(nlmsg_attrdata(nlh, hdrlen), nlmsg_attrlen(nlh, hdrlen),
+ DRBD_NLA_CFG_CONTEXT);
+ if (!nla)
+ return NULL;
+ return nla_find_nested(nla, attr);
+}
+
+/*
+ * Resolve the optional resource-name filter of a dump on its first call.
+ * The core expects the resource in cb->args[0], with a reference that the
+ * matching _done callback drops again. Returns true when the name is not
+ * known: no resource to walk, the caller reports that in its message.
+ */
+static bool compat84_dump_filter(struct netlink_callback *cb, int holder_nr)
+{
+ struct drbd_resource *resource;
+ struct nlattr *resource_filter;
+
+ resource_filter =
+ compat84_find_cfg_context_attr(cb->nlh, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
+ if (IS_ERR_OR_NULL(resource_filter))
+ return false;
+ resource = drbd_find_resource(nla_data(resource_filter));
+ if (!resource)
+ return true;
+ cb->args[0] = (long)resource;
+ return false;
+}
+
+/*
+ * device->ldev->md.flags masked to the 8.4 bits, OR'd with the persisted
+ * peer's flags, exactly as drbd_md_encode_84() builds the on-disk value.
+ * Caller holds get_ldev(). Needs no live peer_device: the persisted peer
+ * slot (md.peers[!md.node_id]) is valid from attach on, so pre-existing
+ * 8.4 metadata reports correctly before the first connect.
+ */
+static u32 compat84_pack_disk_flags_84(struct drbd_device *device)
+{
+ struct drbd_md *md = &device->ldev->md;
+ int peer_node_id = !md->node_id;
+ struct drbd_peer_md *peer_md = &md->peers[peer_node_id];
+ u32 flags = md->flags & MDF_84_MASK;
+
+ flags |= peer_md->flags & MDF_84_PEER_MASK;
+ if (device->bitmap == NULL)
+ flags |= MDF_PEER_FULL_SYNC;
+ flags |= test_bit(__MDF_PEER_OUTDATED, &peer_md->flags) ?
+ MDF_84_PEER_OUTDATED : 0;
+ flags |= test_bit(__MDF_PEER_CONNECTED, &peer_md->flags) ?
+ MDF_84_CONNECTED_IND : 0;
+ return flags;
+}
+
+/*
+ * Fill @si from @device and its single peer device, ported from mainline's
+ * nla_put_status_info() with DRBD 9 sources substituted for 8.4's flat
+ * device fields: the send/recv/pending counters and rs_total/rs_failed
+ * moved to the peer device, ap_bio_cnt was split into READ and WRITE, and
+ * bits_oos is per peer bitmap slot. The uuids, disk_flags and bitmap
+ * counters need only get_ldev(), not a live peer device.
+ */
+static int compat84_fill_state_info(struct drbd_device *device, struct state_info *si,
+ enum drbd_state_info_bcast_reason reason,
+ u32 prev_state_84, u32 new_state_84)
+{
+ struct drbd_peer_device *peer_device;
+ union drbd_state s;
+ int got_ldev;
+
+ memset(si, 0, sizeof(*si));
+
+ si->sib_reason = reason;
+
+ got_ldev = get_ldev(device);
+
+ rcu_read_lock();
+ peer_device = compat84_single_peer_device(device);
+
+ s.i = drbd_pack_state_84(device);
+ si->current_state = s.i;
+ /*
+ * Only SIB_STATE_CHANGE carries a real transition; every other
+ * reason mirrors current_state, as mainline's sib == NULL case does.
+ */
+ if (reason == SIB_STATE_CHANGE) {
+ si->prev_state = prev_state_84;
+ si->new_state = new_state_84;
+ } else {
+ si->prev_state = si->current_state;
+ si->new_state = si->current_state;
+ }
+
+ si->capacity = (u64)get_capacity(device->vdisk);
+ si->ed_uuid = device->exposed_data_uuid;
+
+ si->send_cnt = peer_device ? peer_device->send_cnt : 0;
+ si->recv_cnt = peer_device ? peer_device->recv_cnt : 0;
+ si->read_cnt = device->read_cnt;
+ si->writ_cnt = device->writ_cnt;
+ si->al_writ_cnt = device->al_writ_cnt;
+ si->bm_writ_cnt = device->bm_writ_cnt;
+ si->ap_bio_cnt = atomic_read(&device->ap_bio_cnt[READ]) +
+ atomic_read(&device->ap_bio_cnt[WRITE]);
+ si->ap_pending_cnt = peer_device ? atomic_read(&peer_device->ap_pending_cnt) : 0;
+ si->rs_pending_cnt = peer_device ? atomic_read(&peer_device->rs_pending_cnt) : 0;
+
+ if (got_ldev) {
+ struct drbd_md *md = &device->ldev->md;
+ int peer_node_id = !md->node_id;
+ struct drbd_peer_md *peer_md = &md->peers[peer_node_id];
+ u64 uuid[UI_SIZE] = { };
+
+ spin_lock_irq(&md->uuid_lock);
+ uuid[UI_CURRENT] = md->current_uuid;
+ uuid[UI_BITMAP] = peer_md->bitmap_uuid;
+ uuid[UI_HISTORY_START] = md->history_uuids[0];
+ uuid[UI_HISTORY_END] = md->history_uuids[1];
+ BUILD_BUG_ON(sizeof(uuid) != sizeof(si->uuids));
+ memcpy(si->uuids, uuid, sizeof(uuid));
+ spin_unlock_irq(&md->uuid_lock);
+ si->uuids_len = DRBD_NL_UUIDS_SIZE;
+
+ si->disk_flags = compat84_pack_disk_flags_84(device);
+
+ /*
+ * drbd_adm_attach() publishes device->ldev before it allocates
+ * device->bitmap, and this path does not hold adm_mutex.
+ */
+ if (device->bitmap) {
+ si->bits_total = drbd_bm_bits(device);
+ si->bits_oos = peer_md->bitmap_index != -1 ?
+ _drbd_bm_total_weight(device, peer_md->bitmap_index) : 0;
+ }
+
+ if (s.conn >= L_SYNC_SOURCE && s.conn <= L_PAUSED_SYNC_T && peer_device) {
+ si->bits_rs_total = peer_device->rs_total;
+ si->bits_rs_failed = peer_device->rs_failed;
+ }
+ }
+ rcu_read_unlock();
+
+ if (got_ldev)
+ put_ldev(device);
+
+ return 0;
+}
+
+/*
+ * The STATE_INFO nest as mainline's nla_put_status_info() builds it. The
+ * vendored state_info_to_skb() puts every field, but 8.4 userland reads
+ * some of them by presence: without an ldev there are no uuids, disk_flags
+ * or bit counts, bits_rs_total/bits_rs_failed exist only during a resync
+ * (sh-status computes a resync percentage whenever they are there), and
+ * prev_state/new_state and the helper fields only for their reasons.
+ * si->uuids_len is nonzero exactly when compat84_fill_state_info() got the
+ * ldev.
+ */
+static int compat84_state_info_to_skb(struct sk_buff *skb, struct state_info *si)
+{
+ struct nlattr *nla = nla_nest_start(skb, DRBD_NLA_STATE_INFO);
+ union drbd_state s = { .i = si->current_state };
+
+ if (!nla)
+ return -EMSGSIZE;
+
+ if (nla_put_u32(skb, DRBD_A_STATE_INFO_SIB_REASON, si->sib_reason) ||
+ nla_put_u32(skb, DRBD_A_STATE_INFO_CURRENT_STATE, si->current_state) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_ED_UUID, si->ed_uuid, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_CAPACITY, si->capacity, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_SEND_CNT, si->send_cnt, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_RECV_CNT, si->recv_cnt, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_READ_CNT, si->read_cnt, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_WRIT_CNT, si->writ_cnt, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_AL_WRIT_CNT, si->al_writ_cnt, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BM_WRIT_CNT, si->bm_writ_cnt, 0) ||
+ nla_put_u32(skb, DRBD_A_STATE_INFO_AP_BIO_CNT, si->ap_bio_cnt) ||
+ nla_put_u32(skb, DRBD_A_STATE_INFO_AP_PENDING_CNT, si->ap_pending_cnt) ||
+ nla_put_u32(skb, DRBD_A_STATE_INFO_RS_PENDING_CNT, si->rs_pending_cnt))
+ goto nla_put_failure;
+
+ if (si->uuids_len) {
+ if (nla_put(skb, DRBD_A_STATE_INFO_UUIDS, si->uuids_len, si->uuids) ||
+ nla_put_u32(skb, DRBD_A_STATE_INFO_DISK_FLAGS, si->disk_flags) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_TOTAL, si->bits_total, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_OOS, si->bits_oos, 0))
+ goto nla_put_failure;
+ if (s.conn >= L_SYNC_SOURCE && s.conn <= L_PAUSED_SYNC_T &&
+ (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_TOTAL,
+ si->bits_rs_total, 0) ||
+ nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_FAILED,
+ si->bits_rs_failed, 0)))
+ goto nla_put_failure;
+ }
+
+ switch (si->sib_reason) {
+ case SIB_STATE_CHANGE:
+ if (nla_put_u32(skb, DRBD_A_STATE_INFO_PREV_STATE, si->prev_state) ||
+ nla_put_u32(skb, DRBD_A_STATE_INFO_NEW_STATE, si->new_state))
+ goto nla_put_failure;
+ break;
+ case SIB_HELPER_POST:
+ if (nla_put_u32(skb, DRBD_A_STATE_INFO_HELPER_EXIT_CODE, si->helper_exit_code))
+ goto nla_put_failure;
+ fallthrough;
+ case SIB_HELPER_PRE:
+ if (nla_put_string(skb, DRBD_A_STATE_INFO_HELPER, si->helper))
+ goto nla_put_failure;
+ break;
+ default:
+ break;
+ }
+
+ nla_nest_end(skb, nla);
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * Port of mainline's nla_put_status_info(): the whole GET_STATUS reply for
+ * one device. drbdsetup-84's show and print_broadcast_events() read
+ * drbd_cfg_context/res_opts/disk_conf/net_conf off this reply and never
+ * touch state_info, so all four must be present. Mainline's
+ * drbd_bcast_event() reuses this same payload for the fused DRBD_EVENT,
+ * and so does compat84_emit_event().
+ *
+ * exclude_sensitive mirrors mainline's (sib || !capable(CAP_SYS_ADMIN)).
+ * SIB_GET_STATUS_REPLY is the only reason the two GET_STATUS entry points
+ * pass; every other reason comes through compat84_emit_event(), which runs
+ * from kernel threads where capable() is always true, so those must be
+ * masked unconditionally.
+ */
+static int compat84_put_status_info(struct sk_buff *skb, struct drbd_device *device,
+ enum drbd_state_info_bcast_reason reason,
+ u32 prev_state_84, u32 new_state_84,
+ const char *helper_name, int helper_exit_code)
+{
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
+ struct res_opts res_opts;
+ struct state_info si;
+ bool exclude_sensitive = reason != SIB_GET_STATUS_REPLY || !capable(CAP_SYS_ADMIN);
+ int got_ldev;
+ int err;
+
+ got_ldev = get_ldev(device);
+
+ /*
+ * Mainline passes the_only_connection(resource) here for the doit
+ * reply, every dumpit row and the broadcast alike; the local RCU
+ * section is needed for the doit caller.
+ */
+ rcu_read_lock();
+ err = compat84_put_cfg_context(skb, resource, compat84_single_connection(resource), device);
+ rcu_read_unlock();
+ if (err)
+ goto out;
+
+ compat84_pack_res_opts_84(&res_opts, &resource->res_opts);
+ err = res_opts_to_skb(skb, &res_opts);
+ if (err)
+ goto out;
+
+ rcu_read_lock();
+ peer_device = compat84_single_peer_device(device);
+ if (got_ldev) {
+ struct drbd_disk_conf *disk_conf = rcu_dereference(device->ldev->disk_conf);
+ struct disk_conf dc;
+
+ compat84_pack_disk_conf_84(&dc, disk_conf, peer_device);
+ err = disk_conf_to_skb(skb, &dc);
+ }
+ if (!err && peer_device) {
+ struct drbd_net_conf *nc =
+ rcu_dereference(peer_device->connection->transport.net_conf);
+
+ if (nc)
+ err = compat84_put_net_conf_masked(skb, nc, exclude_sensitive);
+ }
+ rcu_read_unlock();
+ if (err)
+ goto out;
+
+ err = compat84_fill_state_info(device, &si, reason, prev_state_84, new_state_84);
+ if (err)
+ goto out;
+
+ if (reason == SIB_HELPER_PRE || reason == SIB_HELPER_POST) {
+ strscpy(si.helper, helper_name, sizeof(si.helper));
+ si.helper_len = min(strlen(helper_name), sizeof(si.helper));
+ si.helper_exit_code = helper_exit_code;
+ }
+
+ err = compat84_state_info_to_skb(skb, &si);
+
+out:
+ if (got_ldev)
+ put_ldev(device);
+ return err;
+}
+
+/*
+ * The fused v1 DRBD_EVENT, mainline's drbd_bcast_event(), reusing
+ * compat84_put_status_info() for the payload.
+ *
+ * SIB_STATE_CHANGE events come from compat84_event_flush(), which skips a
+ * state change that leaves the 8.4 state word unchanged. SIB_SYNC_PROGRESS
+ * (whose packed word does not move for the whole resync) and the helper
+ * reasons always go out; their call sites are throttled or one-shot on
+ * their own.
+ *
+ * Called from process or workqueue context only, hence GFP_NOIO.
+ */
+static int compat84_emit_event(struct drbd_device *device,
+ enum drbd_state_info_bcast_reason reason,
+ u32 prev_state_84, u32 new_state_84,
+ const char *helper_name, int helper_exit_code)
+{
+ struct drbd_genlmsghdr *dh;
+ struct sk_buff *skb;
+ int err;
+
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ if (!skb)
+ return -ENOMEM;
+
+ err = -EMSGSIZE;
+ dh = genlmsg_put(skb, 0, atomic_inc_return(&compat84_event_seq), &drbd_nl_family, 0,
+ DRBD_ADM_EVENT);
+ if (!dh)
+ goto fail;
+ dh->minor = device->minor;
+ dh->ret_code = NO_ERROR;
+
+ err = compat84_put_status_info(skb, device, reason, prev_state_84, new_state_84,
+ helper_name, helper_exit_code);
+ if (err)
+ goto fail;
+
+ genlmsg_end(skb, dh);
+ err = compat84_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed_sent;
+ return 0;
+
+fail:
+ nlmsg_free(skb);
+failed_sent:
+ drbd_err(device, "Error %d while broadcasting fused event. Reason:%u\n", err, reason);
+ return err;
+}
+
+int drbd_nl_get_status_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+ struct compat84_req *req = compat84_req(ctx);
+ int err;
+
+ if (!req->reply_skb)
+ return 0;
+
+ err = compat84_put_status_info(req->reply_skb, ctx->device, SIB_GET_STATUS_REPLY,
+ 0, 0, NULL, 0);
+ if (err) {
+ nlmsg_free(req->reply_skb);
+ req->reply_skb = NULL;
+ return err;
+ }
+ return 0;
+}
+
+/*
+ * Port of mainline's get_one_status(): one message per volume, walked
+ * resource by resource, including the volume-less resource row.
+ * cb->args[0] is the resource cursor, revalidated against the live list on
+ * every call rather than dereferenced blindly; cb->args[1] the volume
+ * cursor; cb->args[2], when set, pins the dump to a single resource.
+ */
+static int compat84_get_one_status(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ struct drbd_device *device;
+ struct drbd_genlmsghdr *dh;
+ struct drbd_resource *pos = (struct drbd_resource *)cb->args[0];
+ struct drbd_resource *resource = NULL;
+ struct drbd_resource *tmp;
+ unsigned int volume = cb->args[1];
+
+ rcu_read_lock();
+ for_each_resource_rcu(tmp, &drbd_resources) {
+ if (pos == NULL) {
+ pos = tmp;
+ resource = pos;
+ break;
+ }
+ if (tmp == pos) {
+ resource = pos;
+ break;
+ }
+ }
+ if (resource) {
+next_resource:
+ device = idr_get_next(&resource->devices, &volume);
+ if (!device) {
+ pos = list_entry_rcu(resource->resources.next,
+ struct drbd_resource, resources);
+ if (volume != 0) {
+ if (&pos->resources == &drbd_resources || cb->args[2])
+ goto out;
+ volume = 0;
+ resource = pos;
+ goto next_resource;
+ }
+ }
+
+ dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, &drbd_nl_family,
+ NLM_F_MULTI, DRBD_ADM_GET_STATUS);
+ if (!dh)
+ goto out;
+
+ if (!device) {
+ /*
+ * A connection without a volume may still have
+ * network configuration worth reporting.
+ */
+ struct drbd_connection *connection;
+
+ dh->minor = -1U;
+ dh->ret_code = NO_ERROR;
+ connection = compat84_single_connection(resource);
+ if (compat84_put_cfg_context(skb, resource, connection, NULL))
+ goto cancel;
+ if (connection) {
+ struct drbd_net_conf *nc = rcu_dereference(
+ connection->transport.net_conf);
+
+ if (nc && compat84_put_net_conf_masked(skb, nc,
+ !capable(CAP_SYS_ADMIN)))
+ goto cancel;
+ }
+ goto done;
+ }
+
+ D_ASSERT(device, device->vnr == volume);
+ D_ASSERT(device, device->resource == resource);
+
+ dh->minor = device->minor;
+ dh->ret_code = NO_ERROR;
+
+ if (compat84_put_status_info(skb, device, SIB_GET_STATUS_REPLY,
+ 0, 0, NULL, 0)) {
+cancel:
+ genlmsg_cancel(skb, dh);
+ goto out;
+ }
+done:
+ genlmsg_end(skb, dh);
+ }
+
+out:
+ rcu_read_unlock();
+ cb->args[0] = (long)pos;
+ cb->args[1] = (pos == resource) ? volume + 1 : 0;
+
+ return skb->len;
+}
+
+int drbd_nl_get_status_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ struct nlattr *nla;
+ struct drbd_resource *resource;
+
+ /* Followup call: the resource iterator, if any, is already set. */
+ if (cb->args[0]) {
+ if (cb->args[2] && cb->args[2] != cb->args[0])
+ return 0; /* single-resource dump, already exhausted */
+ goto dump;
+ }
+
+ /* First call: is the request scoped to one resource by name? */
+ nla = compat84_find_cfg_context_attr(cb->nlh, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
+ if (!nla)
+ goto dump; /* no filter: dump every resource */
+
+ resource = drbd_find_resource(nla_data(nla));
+ if (!resource)
+ return -ENODEV;
+ /*
+ * cb->args[0]/[2] are opaque positions revalidated on every call, so
+ * no reference survives across dump callbacks, as in mainline.
+ */
+ kref_put(&resource->kref, drbd_destroy_resource);
+
+ cb->args[0] = (long)resource;
+ cb->args[2] = (long)resource;
+
+dump:
+ return compat84_get_one_status(skb, cb);
+}
+
+/*
+ * Core functions such as drbd_adm_net_opts() and adm_disconnect() index
+ * compat84_req(ctx)->info->attrs[] unconditionally, sized to whichever v1
+ * command allocated this request. A synthetic re-entry on behalf of a
+ * different command (the fencing re-entry below, connect's rollback) can
+ * index past the end of that array, a real general protection fault.
+ * Substitute a full-sized, all-absent attrs[] for the call: every
+ * has_set()/attr_present() then reports "not sent", which is right for a
+ * re-entry that feeds its values through req's stashed fields.
+ *
+ * Also clear adm_ctx->set_defaults for the call. drbd_adm_net_opts()
+ * honours it unconditionally, and in 8.4 fencing is a disk option, so
+ * "disk-options --set-defaults --fencing=..." (which drbdadm-84 adjust
+ * issues routinely) would otherwise reset the whole net_conf, shared
+ * secret included, while reporting success.
+ *
+ * Not used for the drbd_adm_peer_device_opts() re-entries: the
+ * resync-tuning fields genuinely live in disk_conf on 8.4, so
+ * "disk-options --set-defaults" is supposed to reset them too.
+ */
+static void compat84_call_with_empty_attrs(struct drbd_adm_ctx *ctx,
+ int (*fn)(struct drbd_adm_ctx *))
+{
+ struct compat84_req *req = compat84_req(ctx);
+ struct nlattr *no_attrs[__DRBD_NLA_MAX] = { };
+ struct genl_info empty_info = *req->info;
+ struct genl_info *orig_info = req->info;
+ bool orig_set_defaults = ctx->set_defaults;
+
+ empty_info.attrs = no_attrs;
+ req->info = &empty_info;
+ ctx->set_defaults = false;
+ fn(ctx);
+ ctx->set_defaults = orig_set_defaults;
+ req->info = orig_info;
+}
+
+/*
+ * Apply the resync-tuning and fencing values compat84_overlay() stashed
+ * out of disk_conf, after drbd_adm_attach() or drbd_adm_disk_opts()
+ * succeeded. 8.4's normal order is attach, then connect, so the peer
+ * device (for struct drbd_peer_device_conf) and the connection (for
+ * net_conf.fencing_policy) usually do not exist yet; then the values are
+ * deferred onto the device and the resource for the connect handler.
+ */
+/*
+ * In 8.4, the resync tuning fields and fencing are disk options, so
+ * "disk-options --set-defaults" resets the ones it does not send. They are
+ * applied only as sent fields here, so turn the unsent ones into sent
+ * defaults.
+ */
+static void compat84_disk_conf_stash_defaults(struct compat84_req *req)
+{
+ typeof(req->peer_device_conf_84) *pdc = &req->peer_device_conf_84;
+
+#define STASH_DEFAULT(field, def) \
+ do { \
+ if (!pdc->has_##field) { \
+ pdc->field = def; \
+ pdc->has_##field = true; \
+ } \
+ } while (0)
+ STASH_DEFAULT(resync_rate, DRBD_RESYNC_RATE_DEF);
+ STASH_DEFAULT(c_plan_ahead, DRBD_C_PLAN_AHEAD_DEF);
+ STASH_DEFAULT(c_delay_target, DRBD_C_DELAY_TARGET_DEF);
+ STASH_DEFAULT(c_fill_target, DRBD_C_FILL_TARGET_DEF);
+ STASH_DEFAULT(c_max_rate, DRBD_C_MAX_RATE_DEF);
+ STASH_DEFAULT(c_min_rate, DRBD_C_MIN_RATE_DEF);
+#undef STASH_DEFAULT
+
+ if (!req->has_fencing_policy_84) {
+ req->fencing_policy_84 = DRBD_FENCING_DEF;
+ req->has_fencing_policy_84 = true;
+ }
+}
+
+static void compat84_apply_disk_conf_stash(struct drbd_adm_ctx *ctx)
+{
+ struct compat84_req *req = compat84_req(ctx);
+ struct drbd_device *device = ctx->device;
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
+ bool have_resync;
+
+ if (ctx->set_defaults)
+ compat84_disk_conf_stash_defaults(req);
+
+ have_resync = req->peer_device_conf_84.has_resync_rate ||
+ req->peer_device_conf_84.has_c_plan_ahead ||
+ req->peer_device_conf_84.has_c_delay_target ||
+ req->peer_device_conf_84.has_c_fill_target ||
+ req->peer_device_conf_84.has_c_max_rate ||
+ req->peer_device_conf_84.has_c_min_rate;
+
+ if (!have_resync && !req->has_fencing_policy_84)
+ return;
+
+ /*
+ * A connection creates a peer device for every existing device, so
+ * a peer device exists if and only if a connection does.
+ */
+retry:
+ rcu_read_lock();
+ peer_device = list_first_or_null_rcu(&device->peer_devices,
+ struct drbd_peer_device,
+ peer_devices);
+ if (peer_device) {
+ kref_get(&peer_device->connection->kref);
+ }
+ rcu_read_unlock();
+
+ if (!peer_device) {
+ mutex_lock(&resource->adm_mutex);
+ /*
+ * A connect may have created the peer device meanwhile; it
+ * consumes the stash only afterwards, so stash only while
+ * there still is none, under adm_mutex.
+ */
+ if (!list_empty(&device->peer_devices)) {
+ mutex_unlock(&resource->adm_mutex);
+ goto retry;
+ }
+ if (have_resync) {
+ typeof(req->peer_device_conf_84) *from = &req->peer_device_conf_84;
+ typeof(device->pending_peer_device_conf_84) *to =
+ &device->pending_peer_device_conf_84;
+
+ if (from->has_resync_rate) {
+ to->resync_rate = from->resync_rate;
+ to->has_resync_rate = true;
+ }
+ if (from->has_c_plan_ahead) {
+ to->c_plan_ahead = from->c_plan_ahead;
+ to->has_c_plan_ahead = true;
+ }
+ if (from->has_c_delay_target) {
+ to->c_delay_target = from->c_delay_target;
+ to->has_c_delay_target = true;
+ }
+ if (from->has_c_fill_target) {
+ to->c_fill_target = from->c_fill_target;
+ to->has_c_fill_target = true;
+ }
+ if (from->has_c_max_rate) {
+ to->c_max_rate = from->c_max_rate;
+ to->has_c_max_rate = true;
+ }
+ if (from->has_c_min_rate) {
+ to->c_min_rate = from->c_min_rate;
+ to->has_c_min_rate = true;
+ }
+ }
+ if (req->has_fencing_policy_84) {
+ resource->pending_fencing_policy_84 =
+ (enum drbd_fencing_policy)req->fencing_policy_84;
+ resource->pending_fencing_policy_84_set = true;
+ }
+ mutex_unlock(&resource->adm_mutex);
+ return;
+ }
+
+ ctx->connection = peer_device->connection;
+
+ if (have_resync) {
+ ctx->peer_device = peer_device;
+ drbd_adm_peer_device_opts(ctx);
+ }
+
+ if (req->has_fencing_policy_84 && (!have_resync || ctx->result == NO_ERROR))
+ compat84_call_with_empty_attrs(ctx, drbd_adm_net_opts);
+}
+
+int drbd_nl_new_minor_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_new_minor(ctx);
+}
+
+int drbd_nl_del_minor_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_del_minor(ctx);
+}
+
+int drbd_nl_new_resource_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ /*
+ * Every resource created through the version 1 dialect is a DRBD 8.4
+ * resource: single peer, node ids derived from the peer's. The core
+ * enforces that from res_opts.drbd8_compat_mode.
+ */
+ ctx->force_drbd8_compat = true;
+ return drbd_adm_new_resource(ctx);
+}
+
+int drbd_nl_del_resource_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_del_resource(ctx);
+}
+
+int drbd_nl_resource_opts_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_resource_opts(ctx);
+}
+
+/*
+ * Render a raw sockaddr into the "af:addr:port" / "af:[addr]:port" text
+ * drbdadm-84 passes to drbdsetup-84. The kernel only sees the parsed
+ * sockaddr, so this reproduces the canonical form, not whatever a user
+ * typed by hand.
+ */
+static int compat84_addr_to_str(char *buf, size_t size, const u8 *addr, u32 addr_len)
+{
+ const struct sockaddr *sa = (const struct sockaddr *)addr;
+
+ if (addr_len >= sizeof(struct sockaddr_in6) && sa->sa_family == AF_INET6) {
+ const struct sockaddr_in6 *a6 = (const struct sockaddr_in6 *)addr;
+
+ return scnprintf(buf, size, "ipv6:[%pI6c]:%u",
+ &a6->sin6_addr, ntohs(a6->sin6_port));
+ }
+ if (addr_len >= sizeof(struct sockaddr_in) && sa->sa_family == AF_INET) {
+ const struct sockaddr_in *a4 = (const struct sockaddr_in *)addr;
+
+ return scnprintf(buf, size, "ipv4:%pI4:%u",
+ &a4->sin_addr, ntohs(a4->sin_port));
+ }
+ return -EINVAL;
+}
+
+/*
+ * v1's connect carries only an address pair; DRBD 9 needs a peer node id.
+ * drbd9's own compat84 shim (user/v9/drbdsetup_compat84.c, compare_addr())
+ * invents one when it drives one end of a mixed deployment, and this must
+ * match it: node ids are baked into the on-disk metadata slot layout, so
+ * disagreeing is a data-integrity hazard, not just a failed connect.
+ *
+ * The rule: crc32c(0x1a656f21, "af:addr:port") for both addresses; the
+ * larger hash gets node id 1. Both ends apply it to the same inputs, so
+ * the results are always complementary. drbd-utils keeps the hashes in an
+ * int, so "larger" is a signed comparison.
+ */
+static int compat84_connect_peer_node_id(struct compat84_req *req, u32 *peer_node_id)
+{
+ char my_str[64], peer_str[64];
+ int my_len, peer_len;
+ u32 my_hash, peer_hash;
+
+ if (!req->ctx_my_addr_len || !req->ctx_peer_addr_len)
+ return -EINVAL;
+
+ my_len = compat84_addr_to_str(my_str, sizeof(my_str),
+ req->ctx_my_addr, req->ctx_my_addr_len);
+ peer_len = compat84_addr_to_str(peer_str, sizeof(peer_str),
+ req->ctx_peer_addr, req->ctx_peer_addr_len);
+ if (my_len < 0 || peer_len < 0)
+ return -EINVAL;
+
+ my_hash = crc32c(0x1a656f21, my_str, my_len);
+ peer_hash = crc32c(0x1a656f21, peer_str, peer_len);
+ if (my_hash == peer_hash)
+ return -EINVAL;
+
+ *peer_node_id = ((s32)my_hash > (s32)peer_hash) ? 0 : 1;
+ return 0;
+}
+
+/* The per-device half of compat84_apply_pending_stash(); @device is referenced. */
+static void compat84_apply_pending_resync(struct drbd_adm_ctx *ctx,
+ struct drbd_device *device)
+{
+ struct compat84_req *req = compat84_req(ctx);
+ struct drbd_resource *resource = device->resource;
+ struct drbd_peer_device *peer_device;
+ bool have_resync;
+
+ if (mutex_lock_interruptible(&resource->adm_mutex))
+ return;
+ have_resync = device->pending_peer_device_conf_84.has_resync_rate ||
+ device->pending_peer_device_conf_84.has_c_plan_ahead ||
+ device->pending_peer_device_conf_84.has_c_delay_target ||
+ device->pending_peer_device_conf_84.has_c_fill_target ||
+ device->pending_peer_device_conf_84.has_c_max_rate ||
+ device->pending_peer_device_conf_84.has_c_min_rate;
+ if (have_resync) {
+ /* Two distinct anonymous struct types: copy field by field. */
+ req->peer_device_conf_84.has_resync_rate =
+ device->pending_peer_device_conf_84.has_resync_rate;
+ req->peer_device_conf_84.resync_rate =
+ device->pending_peer_device_conf_84.resync_rate;
+ req->peer_device_conf_84.has_c_plan_ahead =
+ device->pending_peer_device_conf_84.has_c_plan_ahead;
+ req->peer_device_conf_84.c_plan_ahead =
+ device->pending_peer_device_conf_84.c_plan_ahead;
+ req->peer_device_conf_84.has_c_delay_target =
+ device->pending_peer_device_conf_84.has_c_delay_target;
+ req->peer_device_conf_84.c_delay_target =
+ device->pending_peer_device_conf_84.c_delay_target;
+ req->peer_device_conf_84.has_c_fill_target =
+ device->pending_peer_device_conf_84.has_c_fill_target;
+ req->peer_device_conf_84.c_fill_target =
+ device->pending_peer_device_conf_84.c_fill_target;
+ req->peer_device_conf_84.has_c_max_rate =
+ device->pending_peer_device_conf_84.has_c_max_rate;
+ req->peer_device_conf_84.c_max_rate =
+ device->pending_peer_device_conf_84.c_max_rate;
+ req->peer_device_conf_84.has_c_min_rate =
+ device->pending_peer_device_conf_84.has_c_min_rate;
+ req->peer_device_conf_84.c_min_rate =
+ device->pending_peer_device_conf_84.c_min_rate;
+ }
+ mutex_unlock(&resource->adm_mutex);
+
+ if (!have_resync)
+ return;
+
+ rcu_read_lock();
+ peer_device = list_first_or_null_rcu(&device->peer_devices,
+ struct drbd_peer_device,
+ peer_devices);
+ rcu_read_unlock();
+ if (!peer_device)
+ return;
+
+ /* drbd_adm_peer_device_opts() takes resource->adm_mutex
+ * itself; must not be held here.
+ */
+ ctx->peer_device = peer_device;
+ drbd_adm_peer_device_opts(ctx);
+ ctx->peer_device = NULL;
+
+ if (ctx->result == NO_ERROR) {
+ if (!mutex_lock_interruptible(&resource->adm_mutex)) {
+ memset(&device->pending_peer_device_conf_84, 0,
+ sizeof(device->pending_peer_device_conf_84));
+ mutex_unlock(&resource->adm_mutex);
+ }
+ } else {
+ drbd_warn(device,
+ "could not apply deferred 8.4 resync tuning after connect\n");
+ }
+}
+
+/*
+ * The other end of compat84_apply_disk_conf_stash(): apply the deferred
+ * values now that connect has created the connection and a peer device
+ * for every volume. A failure here does not undo the connect; the values
+ * are retried on the next connect.
+ */
+static void compat84_apply_pending_stash(struct drbd_adm_ctx *ctx)
+{
+ struct compat84_req *req = compat84_req(ctx);
+ struct drbd_resource *resource = ctx->resource;
+ struct drbd_device *device;
+ int vnr;
+
+ /*
+ * The per-device step sleeps on adm_mutex; hold a device reference
+ * across it so a concurrent del-minor cannot free the device.
+ */
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, device, vnr) {
+ kref_get(&device->kref);
+ rcu_read_unlock();
+ compat84_apply_pending_resync(ctx, device);
+ kref_put(&device->kref, drbd_destroy_device);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+
+ if (!resource->pending_fencing_policy_84_set)
+ return;
+
+ if (mutex_lock_interruptible(&resource->adm_mutex))
+ return;
+ req->fencing_policy_84 = resource->pending_fencing_policy_84;
+ req->has_fencing_policy_84 = true;
+ mutex_unlock(&resource->adm_mutex);
+
+ /*
+ * Routed through compat84_call_with_empty_attrs() like the fencing
+ * re-entry in compat84_apply_disk_conf_stash(): a client may set
+ * --set-defaults on connect too.
+ */
+ compat84_call_with_empty_attrs(ctx, drbd_adm_net_opts);
+ req->has_fencing_policy_84 = false;
+
+ if (ctx->result == NO_ERROR) {
+ if (!mutex_lock_interruptible(&resource->adm_mutex)) {
+ resource->pending_fencing_policy_84_set = false;
+ mutex_unlock(&resource->adm_mutex);
+ }
+ } else {
+ drbd_warn(resource, "could not apply deferred 8.4 fencing policy after connect\n");
+ }
+}
+
+/*
+ * v1's single CONNECT carries what DRBD 9 splits into new-peer, new-path
+ * and connect; drbdsetup-84 has no commands to issue them separately.
+ *
+ * drbd_adm_new_peer()/_new_path()/_connect() report their outcome in
+ * ctx->result, and drbd_adm_connect()'s success path writes an enum
+ * drbd_state_rv there on top of the enum drbd_ret_code its failure paths
+ * use, so success is the same range check drbd_adm_down() relies on, not
+ * "== NO_ERROR".
+ */
+static bool compat84_result_ok(int result)
+{
+ return result >= SS_SUCCESS && result <= NO_ERROR;
+}
+
+/*
+ * A v1 disconnect deletes the connection's path (8.4 forgets the
+ * connection's addresses once it is StandAlone) but keeps the connection
+ * and its peer devices, so the next connect has to pick it up again.
+ * Returns that path-less connection's peer node id, or -1 if there is none.
+ */
+static int compat84_pathless_peer_node_id(struct drbd_resource *resource)
+{
+ struct drbd_connection *connection;
+ int peer_node_id = -1;
+
+ rcu_read_lock();
+ connection = list_first_or_null_rcu(&resource->connections,
+ struct drbd_connection, connections);
+ if (connection && list_empty(&connection->transport.paths))
+ peer_node_id = connection->peer_node_id;
+ rcu_read_unlock();
+
+ return peer_node_id;
+}
+
+int drbd_nl_connect_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+ struct compat84_req *req = compat84_req(ctx);
+ struct drbd_connection *connection;
+ bool reused, path_added = false, node_ids_unset;
+ int pathless_id;
+ u32 peer_node_id;
+
+ if (!req->reply_skb)
+ return 0;
+
+ /* new-path assigns the node ids of an 8.4 resource on its first connect. */
+ mutex_lock(&ctx->resource->adm_mutex);
+ node_ids_unset = ctx->resource->res_opts.node_id == -1;
+ mutex_unlock(&ctx->resource->adm_mutex);
+
+ pathless_id = compat84_pathless_peer_node_id(ctx->resource);
+ reused = pathless_id >= 0;
+ if (reused) {
+ peer_node_id = pathless_id;
+ } else if (compat84_connect_peer_node_id(req, &peer_node_id)) {
+ drbd_adm_msg(ctx, "%s",
+ "could not determine a peer node id from the given addresses");
+ ctx->result = ERR_INVALID_REQUEST;
+ return 0;
+ }
+ ctx->peer_node_id = peer_node_id;
+
+ if (!reused) {
+ drbd_adm_new_peer(ctx);
+ if (!compat84_result_ok(ctx->result))
+ return 0;
+ }
+
+ /*
+ * drbd_adm_new_peer() only creates the connection; look it up the
+ * way drbd_adm_ctx_resolve() would, so the calls below and
+ * drbd_adm_ctx_release() see a normally resolved request.
+ */
+ connection = drbd_get_connection_by_node_id(ctx->resource, peer_node_id);
+ if (!connection) {
+ /*
+ * Cannot happen: nothing else removes a connection this
+ * request just created under adm_mutex. ctx->connection is
+ * still NULL, so drbd_adm_del_peer() has nothing to act on.
+ */
+ drbd_adm_msg(ctx, "%s", "internal error: new peer vanished");
+ ctx->result = ERR_INVALID_REQUEST;
+ return 0;
+ }
+ ctx->connection = connection;
+
+ /* A reused connection still has the previous connect's net options. */
+ if (reused)
+ drbd_adm_net_opts(ctx);
+ if (compat84_result_ok(ctx->result)) {
+ drbd_adm_new_path(ctx);
+ path_added = compat84_result_ok(ctx->result);
+ }
+ if (path_added)
+ drbd_adm_connect(ctx);
+
+ if (!compat84_result_ok(ctx->result)) {
+ int result = ctx->result;
+
+ if (!reused) {
+ /*
+ * v1 has no del-peer, so a failed connect must not leave a
+ * peer behind: tear down whatever of {peer, path} was
+ * created. adm_disconnect() reads an attrs[] index
+ * CONNECT's policy does not carry, hence
+ * compat84_call_with_empty_attrs().
+ */
+ compat84_call_with_empty_attrs(ctx, drbd_adm_del_peer);
+ /*
+ * Unassign the node ids again, or a retry with other
+ * addresses keeps the ones derived from these.
+ */
+ if (node_ids_unset) {
+ mutex_lock(&ctx->resource->adm_mutex);
+ if (list_empty(&ctx->resource->connections))
+ ctx->resource->res_opts.node_id = -1;
+ mutex_unlock(&ctx->resource->adm_mutex);
+ }
+ } else if (path_added) {
+ /* Back to the path-less state the last disconnect left. */
+ drbd_adm_del_path(ctx);
+ }
+ ctx->result = result;
+ return 0;
+ }
+
+ /*
+ * The connect succeeded and that is what the reply reports;
+ * compat84_apply_pending_stash() writes its own outcomes to
+ * ctx->result.
+ */
+ compat84_apply_pending_stash(ctx);
+ ctx->result = NO_ERROR;
+ return 0;
+}
+
+int drbd_nl_disconnect_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ drbd_adm_disconnect(ctx);
+ if (!compat84_result_ok(ctx->result))
+ return 0;
+
+ /*
+ * 8.4 forgets a StandAlone connection's addresses, so that the next
+ * connect may name different ones; do the same by deleting the path.
+ * drbd_nl_connect_doit() then reuses the path-less connection.
+ */
+ drbd_adm_del_path(ctx);
+ return 0;
+}
+
+int drbd_nl_attach_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ drbd_adm_attach(ctx);
+ if (ctx->result == NO_ERROR)
+ compat84_apply_disk_conf_stash(ctx);
+ return 0;
+}
+
+int drbd_nl_resize_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_resize(ctx);
+}
+
+int drbd_nl_primary_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_primary(ctx);
+}
+
+int drbd_nl_secondary_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_secondary(ctx);
+}
+
+int drbd_nl_new_c_uuid_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_new_c_uuid(ctx);
+}
+
+int drbd_nl_start_ov_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_start_ov(ctx);
+}
+
+/* True if no device in @resource has a live disk. */
+static bool compat84_resource_has_no_disk(struct drbd_resource *resource)
+{
+ struct drbd_device *d;
+ int vnr;
+ bool none = true;
+
+ rcu_read_lock();
+ idr_for_each_entry(&resource->devices, d, vnr) {
+ if (get_ldev_if_state(d, D_FAILED)) {
+ put_ldev(d);
+ none = false;
+ break;
+ }
+ }
+ rcu_read_unlock();
+ return none;
+}
+
+int drbd_nl_detach_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+ struct drbd_device *device = ctx->device;
+ struct drbd_resource *resource = ctx->resource;
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ drbd_adm_detach(ctx);
+ if (ctx->result == NO_ERROR) {
+ /*
+ * Bound the stashes to the attach that produced them, or a
+ * detach without ever connecting followed by an attach that
+ * does not resend them would still apply the old values on
+ * the next connect. The resync-tuning stash is per device.
+ * The fencing stash is per resource, and a multi-volume 8.4
+ * resource must not lose it when one volume detaches, so
+ * clear it only once no device has a disk left.
+ */
+ mutex_lock(&resource->adm_mutex);
+ memset(&device->pending_peer_device_conf_84, 0,
+ sizeof(device->pending_peer_device_conf_84));
+ if (compat84_resource_has_no_disk(resource))
+ resource->pending_fencing_policy_84_set = false;
+ mutex_unlock(&resource->adm_mutex);
+ }
+ return 0;
+}
+
+int drbd_nl_invalidate_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_invalidate(ctx);
+}
+
+int drbd_nl_inval_peer_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_invalidate_peer(ctx);
+}
+
+int drbd_nl_pause_sync_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_pause_sync(ctx);
+}
+
+int drbd_nl_resume_sync_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_resume_sync(ctx);
+}
+
+int drbd_nl_suspend_io_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_suspend_io(ctx);
+}
+
+int drbd_nl_resume_io_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_resume_io(ctx);
+}
+
+int drbd_nl_outdate_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_outdate(ctx);
+}
+
+int drbd_nl_get_timeout_type_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_get_timeout_type(ctx);
+}
+
+int drbd_nl_down_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_down(ctx);
+}
+
+int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ drbd_adm_disk_opts(ctx);
+ if (ctx->result == NO_ERROR)
+ compat84_apply_disk_conf_stash(ctx);
+ return 0;
+}
+
+int drbd_nl_chg_net_opts_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+
+ if (!compat84_req(ctx)->reply_skb)
+ return 0;
+ return drbd_adm_net_opts(ctx);
+}
+
+int drbd_nl_get_resources_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ return drbd_dump_resources(skb, cb, &drbd_nl_84_dialect);
+}
+
+int drbd_nl_get_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ if (!cb->args[0] && !cb->args[1] && compat84_dump_filter(cb, 7)) {
+ int err = compat84_emit_device(skb, cb, ERR_RES_NOT_KNOWN,
+ NULL, NULL, NULL, NULL);
+
+ return err ? err : skb->len;
+ }
+ return drbd_dump_devices(skb, cb, &drbd_nl_84_dialect);
+}
+
+int drbd_nl_get_connections_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ if (!cb->args[0]) {
+ if (compat84_dump_filter(cb, 6)) {
+ int err = compat84_emit_connection(skb, cb, ERR_RES_NOT_KNOWN,
+ NULL, NULL, NULL, NULL, NULL);
+
+ return err ? err : skb->len;
+ }
+ if (cb->args[0])
+ cb->args[1] = DRBD_DUMP_SINGLE_RESOURCE;
+ }
+ return drbd_dump_connections(skb, cb, &drbd_nl_84_dialect);
+}
+
+int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ if (!cb->args[0] && !cb->args[1] && compat84_dump_filter(cb, 9)) {
+ int err = compat84_emit_peer_device(skb, cb, ERR_RES_NOT_KNOWN,
+ NULL, 0, NULL, NULL, NULL);
+
+ return err ? err : skb->len;
+ }
+ return drbd_dump_peer_devices(skb, cb, &drbd_nl_84_dialect);
+}
+
+int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ int ret;
+
+ /* the replayed notifications are sent under the sequence number of the dump */
+ ret = drbd_dump_initial_state(skb, cb, &drbd_nl_84_dialect, cb->nlh->nlmsg_seq);
+
+ /*
+ * The vendored ops table has no .done for GET_INITIAL_STATE, so free the
+ * state-change snapshot here once the dump ends, as mainline does.
+ */
+ if (ret <= 0)
+ drbd_dump_initial_state_done(cb);
+ return ret;
+}
+
+int drbd_nl_legacy_init(void)
+{
+ int err = genl_register_family(&drbd_nl_family);
+
+ if (err)
+ return err;
+ err = drbd_nl_register_dialect(&drbd_nl_84_dialect);
+ if (err)
+ genl_unregister_family(&drbd_nl_family);
+ return err;
+}
+
+void drbd_nl_legacy_exit(void)
+{
+ genl_unregister_family(&drbd_nl_family);
+}
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 18/20] drbd: add the "drbd2" genl family
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (16 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 17/20] drbd: serve the "drbd" genl family at version 1 for existing userspace Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 19/20] drbd: update monitoring interfaces for multi-peer topology Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 20/20] drbd: remove BROKEN for DRBD Christoph Böhmwalder
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Donald Hunter,
Jakub Kicinski
DRBD 9's utilities need the multi-peer administration model that the
version-1 wire cannot express: peers identified by node id, several
paths per connection, per-peer-device options and statistics, and
notifications for all of those. Rather than bump the "drbd" family
and break every existing drbd-utils, add a second family, "drbd2",
written as a YNL specification in Documentation/netlink/specs/ and
following current generic netlink conventions: strict policies with
ranges, nested objects, split do/dump ops, extended ACKs for errors,
and a proper uapi header.
drbd2_nl_gen.[ch] and include/uapi/linux/drbd2.h are generated from
the spec with tools/net/ynl and are not edited by hand.
drbd_nl_drbd2.c is the struct drbd_nl_dialect adapter for the family.
Both families are registered by the same driver and operate on the
same objects; a resource created through either is visible through
both, within what each wire format can express. drbd2 is always
registered, so with DRBD_COMPAT_84=n it is the only family.
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
Documentation/netlink/specs/drbd2.yaml | 2600 ++++++++++++++++++++++++
drivers/block/drbd/Kconfig | 6 +
drivers/block/drbd/Makefile | 1 +
drivers/block/drbd/drbd2_nl_gen.c | 810 ++++++++
drivers/block/drbd/drbd2_nl_gen.h | 113 +
drivers/block/drbd/drbd_nl_drbd2.c | 2589 +++++++++++++++++++++++
include/uapi/linux/drbd2.h | 1038 ++++++++++
7 files changed, 7157 insertions(+)
create mode 100644 Documentation/netlink/specs/drbd2.yaml
create mode 100644 drivers/block/drbd/drbd2_nl_gen.c
create mode 100644 drivers/block/drbd/drbd2_nl_gen.h
create mode 100644 drivers/block/drbd/drbd_nl_drbd2.c
create mode 100644 include/uapi/linux/drbd2.h
diff --git a/Documentation/netlink/specs/drbd2.yaml b/Documentation/netlink/specs/drbd2.yaml
new file mode 100644
index 000000000000..02d2d5e09c10
--- /dev/null
+++ b/Documentation/netlink/specs/drbd2.yaml
@@ -0,0 +1,2600 @@
+# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+---
+name: drbd2
+
+protocol: genetlink
+
+doc: 'DRBD 9 management interface: configure and monitor replicated block device resources,
+ their devices, connections to peers, peer devices and network paths.'
+
+uapi-header: linux/drbd2.h
+
+definitions:
+ -
+ name: resource-name-max
+ type: const
+ value: 128
+ doc: Maximum length of a resource name including the terminating NUL.
+
+ -
+ name: connection-name-max
+ type: const
+ value: 64
+ doc: Maximum length of a connection name including the terminating NUL.
+
+ -
+ name: shared-secret-max
+ type: const
+ value: 64
+ doc: Maximum length of the shared secret including the terminating NUL.
+
+ -
+ name: alg-name-max
+ type: const
+ value: 64
+ doc: Maximum length of a crypto algorithm name including the terminating NUL.
+
+ -
+ name: transport-name-max
+ type: const
+ value: 64
+ doc: Maximum length of a network transport name including the terminating NUL.
+
+ -
+ name: device-path-max
+ type: const
+ value: 128
+ doc: Maximum length of a backing/meta device path including the terminating NUL.
+
+ -
+ name: cpu-mask-size
+ type: const
+ value: 256
+ doc: Maximum length of the cpu-mask string.
+
+ -
+ name: helper-name-max
+ type: const
+ value: 32
+ doc: Maximum length of a helper name.
+
+ -
+ name: history-uuids-size
+ type: const
+ value: 256
+ doc: Maximum size in bytes of the history UUID array (32 peer slots of 8 bytes).
+
+ -
+ name: io-error-policy
+ type: enum
+ doc: What to do when the backing device reports an I/O error (on-io-error).
+ entries:
+ -
+ name: pass-on
+ doc: Mark the failed block out of sync, change the disk state to inconsistent and retry the request on a peer.
+ -
+ name: call-local-io-error
+ doc: Call the local-io-error handler.
+ -
+ name: detach
+ doc: Detach the backing device and continue in diskless mode.
+
+ -
+ name: fencing-policy
+ type: enum
+ doc: Fencing policy (fencing).
+ entries:
+ -
+ name: dont-care
+ doc: Take no fencing action at all.
+ -
+ name: resource
+ doc: A node that becomes a disconnected primary calls the fence-peer handler to outdate the peer.
+ -
+ name: stonith
+ doc: A node that becomes a disconnected primary freezes its I/O and calls the fence-peer handler, which may have to shoot the peer.
+
+ -
+ name: after-sb-policy
+ type: enum
+ doc: Split-brain recovery policy (after-sb-0pri, after-sb-1pri, after-sb-2pri, rr-conflict).
+ entries:
+ -
+ name: disconnect
+ doc: Do not resynchronize automatically, simply disconnect.
+ -
+ name: discard-younger-primary
+ doc: Resynchronize from the node that became primary first.
+ -
+ name: discard-older-primary
+ doc: Resynchronize from the node that became primary last.
+ -
+ name: discard-zero-changes
+ doc: Resynchronize from the node that wrote data since the split brain was detected, or disconnect if both nodes wrote.
+ -
+ name: discard-least-changes
+ doc: Resynchronize from the node with more modified blocks.
+ -
+ name: discard-local
+ doc: Discard the data on this node and resynchronize it from the peer.
+ -
+ name: discard-remote
+ doc: Discard the data on the peer and resynchronize it from this node.
+ -
+ name: consensus
+ doc: Discard the data on the secondary if the after-sb-0pri policy would discard it as well, otherwise disconnect.
+ -
+ name: discard-secondary
+ doc: Discard the data on the secondary node.
+ -
+ name: call-pri-lost-after-sb
+ doc: Take the after-sb-0pri decision and call the pri-lost-after-sb handler on the node that has to give up its data.
+ -
+ name: violently-as0p
+ doc: Always take the after-sb-0pri decision, even if that changes the primary's view of the data; this can crash the primary.
+ -
+ name: retry-connect
+ doc: Disconnect now and immediately retry to connect (rr-conflict only).
+ -
+ name: auto-discard
+ doc: Reverse the resync direction so that the current primary is resynchronized from the current secondary (rr-conflict with protocol A only).
+
+ -
+ name: on-no-data-policy
+ type: enum
+ doc: What to do when no data is accessible (on-no-data-accessible).
+ entries:
+ -
+ name: io-error
+ doc: Complete the requests with an I/O error, so that system calls fail with EIO.
+ -
+ name: suspend-io
+ doc: Suspend I/O until the data becomes accessible again or the administrator forces I/O to resume.
+
+ -
+ name: on-no-quorum-policy
+ type: enum
+ doc: What to do when quorum is lost (on-no-quorum).
+ entries:
+ -
+ name: io-error
+ doc: Complete the requests with an I/O error while the resource has no quorum.
+ -
+ name: suspend-io
+ doc: Freeze I/O while the resource has no quorum.
+
+ -
+ name: on-suspended-primary-outdated-policy
+ type: enum
+ doc: What to do with a suspended primary that became outdated.
+ entries:
+ -
+ name: disconnect
+ doc: Reject connection attempts and stay isolated.
+ -
+ name: force-secondary
+ doc: Demote to secondary immediately and fail all pending and new I/O requests.
+
+ -
+ name: on-congestion-policy
+ type: enum
+ doc: What to do when the replication link is congested (on-congestion).
+ entries:
+ -
+ name: block
+ doc: Block application writes until the transport's send queue drains.
+ -
+ name: pull-ahead
+ doc: Switch into ahead/behind mode and record further changes in the bitmap instead of replicating them.
+ -
+ name: disconnect
+ doc: Disconnect from the peer instead of blocking application writes.
+
+ -
+ name: read-balancing-policy
+ type: enum
+ doc: Read balancing policy (read-balancing).
+ entries:
+ -
+ name: prefer-local
+ doc: Always read from the local disk.
+ -
+ name: prefer-remote
+ doc: Always read from a peer's disk.
+ -
+ name: round-robin
+ doc: Alternate between the local disk and a peer for consecutive reads.
+ -
+ name: least-pending
+ doc: Read from the disk with the fewest pending requests.
+ -
+ name: when-congested-remote
+ doc: Read from a peer only while the local disk is congested; deprecated since DRBD 9.1.12.
+ -
+ name: striping-32k
+ doc: Read alternating 32 KiB stripes from the local disk and from a peer.
+ -
+ name: striping-64k
+ doc: Read alternating 64 KiB stripes from the local disk and from a peer.
+ -
+ name: striping-128k
+ doc: Read alternating 128 KiB stripes from the local disk and from a peer.
+ -
+ name: striping-256k
+ doc: Read alternating 256 KiB stripes from the local disk and from a peer.
+ -
+ name: striping-512k
+ doc: Read alternating 512 KiB stripes from the local disk and from a peer.
+ -
+ name: striping-1m
+ doc: Read alternating 1 MiB stripes from the local disk and from a peer.
+
+ -
+ name: wire-protocol
+ type: enum
+ doc: Replication protocol (protocol).
+ entries:
+ -
+ name: a
+ value: 1
+ doc: Writes complete as soon as they reached the local disk and the local send buffer.
+ -
+ name: b
+ doc: Writes complete as soon as they reached the local disk and all peers acknowledged their receipt.
+ -
+ name: c
+ doc: Writes complete as soon as they reached the local disk and all remote disks.
+
+ -
+ name: role
+ type: enum
+ doc: Role of a resource on a node. Values mirror enum drbd_role in linux/drbd.h so both netlink dialects share one numbering.
+ entries:
+ -
+ name: unknown
+ doc: The role is not known, for example because the peer is not connected.
+ -
+ name: primary
+ doc: The resource is primary; its devices may be opened for writing.
+ -
+ name: secondary
+ doc: The resource is secondary; its devices may not be opened for writing.
+
+ -
+ name: connection-state
+ type: enum
+ doc: State of a connection to a peer. Values mirror enum drbd_conn_state in linux/drbd.h so both netlink dialects share one numbering.
+ entries:
+ -
+ name: standalone
+ doc: No network configuration is in effect and no connection is attempted.
+ -
+ name: disconnecting
+ doc: Temporary state on the way to standalone while the connection is torn down.
+ -
+ name: unconnected
+ doc: Temporary state before the next connection attempt starts.
+ -
+ name: timeout
+ doc: The connection was dropped because the peer did not answer in time.
+ -
+ name: broken-pipe
+ doc: The connection was dropped because the transport reported a broken connection.
+ -
+ name: network-failure
+ doc: The connection was dropped because of a network failure.
+ -
+ name: protocol-error
+ doc: The connection was dropped because the peer violated the replication protocol.
+ -
+ name: tear-down
+ doc: Temporary state while the peer closes the connection.
+ -
+ name: connecting
+ doc: Listening for the peer and trying to establish a connection.
+ -
+ name: connected
+ doc: A connection to the peer is established.
+
+ -
+ name: repl-state
+ type: enum
+ doc: Replication state of a peer device. Values mirror enum drbd_repl_state in linux/drbd.h so both netlink dialects share one numbering.
+ entries:
+ -
+ name: 'off'
+ value: 9
+ doc: The peer device does not replicate because the connection is not established.
+ -
+ name: established
+ doc: The peer device is connected and replicating normally.
+ -
+ name: starting-sync-s
+ doc: A full resynchronization was requested by the administrator and starts with this node as the source.
+ -
+ name: starting-sync-t
+ doc: A full resynchronization was requested by the administrator and starts with this node as the target.
+ -
+ name: wf-bitmap-s
+ doc: Waiting for the bitmap exchange to finish, with this node as the prospective resync source.
+ -
+ name: wf-bitmap-t
+ doc: Waiting for the bitmap exchange to finish, with this node as the prospective resync target.
+ -
+ name: wf-sync-uuid
+ doc: Waiting for the resync source to create and send the new resync UUID.
+ -
+ name: sync-source
+ doc: Resynchronization is running with this node as the source.
+ -
+ name: sync-target
+ doc: Resynchronization is running with this node as the target.
+ -
+ name: verify-s
+ doc: Online verification is running with this node as the initiator.
+ -
+ name: verify-t
+ doc: Online verification is running with this node as the target.
+ -
+ name: paused-sync-s
+ doc: Resynchronization with this node as the source is paused.
+ -
+ name: paused-sync-t
+ doc: Resynchronization with this node as the target is paused.
+ -
+ name: ahead
+ doc: Replication is suspended because the link is congested; changes are only recorded in the bitmap.
+ -
+ name: behind
+ doc: The peer is ahead of this node, which will be resynchronized once the congestion is over.
+
+ -
+ name: disk-state
+ type: enum
+ doc: State of a backing disk. Values mirror enum drbd_disk_state in linux/drbd.h so both netlink dialects share one numbering.
+ entries:
+ -
+ name: diskless
+ doc: No backing device is attached.
+ -
+ name: attaching
+ doc: The backing device's metadata is being read.
+ -
+ name: detaching
+ doc: The backing device is being detached while pending I/O drains.
+ -
+ name: failed
+ doc: The backing device failed; the disk becomes diskless once the peers have been told.
+ -
+ name: negotiating
+ doc: Attaching while connected, with the data generation still being negotiated with the peers.
+ -
+ name: inconsistent
+ doc: The data is inconsistent, for example because a resynchronization is in progress.
+ -
+ name: outdated
+ doc: The data is consistent but known to be stale relative to a peer.
+ -
+ name: unknown
+ doc: The disk state is not known; used only for a peer's disk.
+ -
+ name: consistent
+ doc: The data is consistent but it is not yet known whether it is up to date.
+ -
+ name: up-to-date
+ doc: The data is consistent and up to date; only this state allows application I/O.
+
+ -
+ name: write-ordering
+ type: enum
+ doc: Write ordering method in use for the backing device.
+ entries:
+ -
+ name: none
+ doc: No ordering between dependent writes is enforced on the backing device.
+ -
+ name: drain-io
+ doc: Dependent writes are ordered by draining the backing device's request queue.
+ -
+ name: bdev-flush
+ doc: Dependent writes are ordered by issuing flushes to the backing device.
+ -
+ name: bio-barrier
+ doc: Dependent writes are ordered with block device barriers, which the Linux block layer no longer provides.
+
+ -
+ name: timeout-type
+ type: enum
+ doc: Which wait-for-connection timeout applies (timeout-type-get reply).
+ entries:
+ -
+ name: default
+ doc: The regular wait-for-connection timeout applies.
+ -
+ name: degraded
+ doc: The degraded-cluster timeout applies because the local data is not up to date.
+ -
+ name: peer-outdated
+ doc: The timeout for an outdated peer applies.
+
+ -
+ name: helper-phase
+ type: enum
+ doc: Whether a helper notification announces the call or reports its result.
+ entries:
+ -
+ name: call
+ doc: The helper script is about to be called.
+ -
+ name: response
+ doc: The helper script has finished and its exit status is reported.
+
+ -
+ name: state-change-action
+ type: enum
+ doc: What a state-change message reports about the object it carries. Values are assigned independently of the legacy drbd_notification_type.
+ entries:
+ -
+ name: exists
+ doc: The object already existed when the notification stream was opened.
+ -
+ name: create
+ doc: The object was just created.
+ -
+ name: change
+ doc: The state or statistics of the object changed.
+ -
+ name: destroy
+ doc: The object was destroyed.
+ -
+ name: rename
+ doc: The resource was renamed; the new name is carried in resource.new-name.
+
+ -
+ name: state-result
+ type: enum
+ doc: Outcome of a state change request. Error values equal the negated kernel-internal SS_* codes (value 3 is unused); success and the other non-error outcomes follow at 29 and above.
+ entries:
+ -
+ name: unknown-error
+ doc: The state change failed for an unspecified reason.
+ -
+ name: two-primaries
+ doc: Multiple primaries are not allowed by the configuration.
+ -
+ name: no-up-to-date-disk
+ doc: The request needs access to up-to-date data.
+ -
+ name: no-local-disk
+ value: 4
+ doc: Cannot resynchronize without a local disk.
+ -
+ name: no-remote-disk
+ doc: Cannot resynchronize without a remote disk.
+ -
+ name: connected-outdates
+ doc: Refusing to become outdated while connected.
+ -
+ name: primary-nop
+ doc: Refusing to become primary while the peer is not outdated.
+ -
+ name: resync-running
+ doc: Cannot start online verify or resynchronization because one is already active.
+ -
+ name: already-standalone
+ doc: Cannot disconnect a connection that is already standalone.
+ -
+ name: cw-failed-by-peer
+ doc: The cluster-wide state change was refused by a peer node.
+ -
+ name: is-diskless
+ doc: The device is diskless but the requested operation requires a disk.
+ -
+ name: device-in-use
+ doc: The device is held open by someone.
+ -
+ name: no-net-config
+ doc: There is no network configuration for this connection.
+ -
+ name: no-verify-alg
+ doc: Starting online verify requires a verify algorithm to be configured.
+ -
+ name: need-connection
+ doc: The state change requires an established connection.
+ -
+ name: lower-than-outdated
+ doc: The disk state is lower than outdated.
+ -
+ name: not-supported
+ doc: The peer does not support the requested protocol feature.
+ -
+ name: in-transient-state
+ doc: The object is in a transient state; retry after the next state change.
+ -
+ name: concurrent-state-change
+ doc: Concurrent state changes were detected and this one was aborted.
+ -
+ name: other-volume-peer-primary
+ doc: Another volume is primary on the peer, which the configuration does not allow.
+ -
+ name: interrupted
+ doc: The state change was interrupted.
+ -
+ name: primary-reader
+ doc: The peer may not become primary while the device is open for reading here.
+ -
+ name: timeout
+ doc: The state change timed out.
+ -
+ name: weakly-connected
+ doc: Primary nodes must be strongly connected among each other.
+ -
+ name: no-quorum
+ doc: The resource does not have quorum.
+ -
+ name: attach-no-bitmap
+ doc: An intentionally diskless peer may not attach a disk.
+ -
+ name: handshake-disconnect
+ doc: Disconnect was chosen during the connection handshake.
+ -
+ name: handshake-retry
+ doc: Retry was chosen during the connection handshake.
+ -
+ name: success
+ value: 29
+ doc: The state change succeeded.
+ -
+ name: nothing-to-do
+ doc: The requested state was already in effect, so nothing had to be done.
+ -
+ name: cw-success
+ doc: The cluster-wide state change succeeded.
+ -
+ name: cw-no-need
+ doc: No cluster-wide state change was necessary.
+
+attribute-sets:
+ -
+ name: address
+ doc: A socket address.
+ attributes:
+ -
+ name: family
+ type: u16
+ doc: Address family (AF_INET or AF_INET6).
+ -
+ name: port
+ type: u16
+ byte-order: big-endian
+ doc: Port number.
+ -
+ name: ipv4
+ type: u32
+ byte-order: big-endian
+ display-hint: ipv4
+ doc: IPv4 address, present when family is AF_INET.
+ -
+ name: ipv6
+ type: binary
+ checks:
+ exact-len: 16
+ display-hint: ipv6
+ doc: IPv6 address, present when family is AF_INET6.
+
+ -
+ name: context
+ doc: Identifies the object a request, reply or notification refers to.
+ attributes:
+ -
+ name: resource-name
+ type: string
+ checks:
+ max-len: resource-name-max
+ doc: Resource name.
+ -
+ name: volume
+ type: u32
+ doc: Volume number within the resource.
+ -
+ name: minor
+ type: u32
+ doc: Minor number of the DRBD device.
+ -
+ name: peer-node-id
+ type: u32
+ doc: Node ID of the peer.
+ -
+ name: connection-name
+ type: string
+ checks:
+ max-len: connection-name-max
+ doc: Name of the connection to the peer.
+ -
+ name: my-address
+ type: nest
+ nested-attributes: address
+ doc: Local address of a path.
+ -
+ name: peer-address
+ type: nest
+ nested-attributes: address
+ doc: Peer address of a path.
+
+ -
+ name: disk-conf
+ doc: Options of an attached backing disk (drbd.conf disk section).
+ attributes:
+ -
+ name: backing-dev
+ type: string
+ checks:
+ max-len: device-path-max
+ doc: Path of the lower-level block device used to store the actual data (drbd.conf option disk).
+ -
+ name: meta-dev
+ type: string
+ checks:
+ max-len: device-path-max
+ doc: Path of the device holding the metadata, or "internal" to store it on the backing device (drbd.conf option meta-disk).
+ -
+ name: meta-dev-idx
+ type: s32
+ doc: Index selecting which slot to use on a shared external metadata device (drbd.conf option meta-disk index).
+ -
+ name: size
+ type: u64
+ doc: Size of the backing device to use, in sectors, overriding automatic detection.
+ -
+ name: on-io-error
+ type: u32
+ enum: io-error-policy
+ doc: Action to take when the backing disk reports an I/O error.
+ -
+ name: resync-after
+ type: s32
+ doc: Volume number of another volume that must finish resynchronizing before this one starts (drbd.conf option resync-after).
+ -
+ name: al-extents
+ type: u32
+ doc: Number of 4 MiB activity log extents that can be active at the same time.
+ -
+ name: disk-barrier
+ type: u8
+ doc: Whether to use disk barriers to enforce the ordering of dependent write requests on the backing device.
+ -
+ name: disk-flushes
+ type: u8
+ doc: Whether to issue disk flushes to the backing device to enforce the ordering of dependent write requests.
+ -
+ name: disk-drain
+ type: u8
+ doc: Whether to wait for the local I/O request queue to drain before completing a write barrier.
+ -
+ name: md-flushes
+ type: u8
+ doc: Whether to issue disk flushes to the metadata device.
+ -
+ name: disk-timeout
+ type: u32
+ doc: Time to wait for the backing device to complete an I/O request before treating it as failed, in units of 0.1 seconds, or 0 to disable (drbd.conf option disk-timeout).
+ -
+ name: read-balancing
+ type: u32
+ enum: read-balancing-policy
+ doc: Policy for balancing reads across the local disk and peers (drbd.conf option read-balancing).
+ -
+ name: unplug-watermark
+ type: u32
+ doc: Legacy watermark, in pages, that used to trigger an early unplug of the backing device's request queue (drbd.conf option unplug-watermark).
+ -
+ name: rs-discard-granularity
+ type: u32
+ doc: Size, in bytes, of the resync requests DRBD tries to discard on the sync target when the source's data is all zeroes (drbd.conf option rs-discard-granularity).
+ -
+ name: al-updates
+ type: u8
+ doc: Whether to update the activity log on disk, at the cost of a full resync after a primary crash if disabled (drbd.conf option al-updates).
+ -
+ name: discard-zeroes-if-aligned
+ type: u8
+ doc: Whether to treat discarded, aligned blocks on the backing device as reading back as zero (drbd.conf option discard-zeroes-if-aligned).
+ -
+ name: disable-write-same
+ type: u8
+ doc: Whether to avoid using the WRITE SAME command on the backing device, for devices that advertise support but fail such requests (drbd.conf option disable-write-same).
+ -
+ name: bitmap
+ type: u8
+ doc: Whether to allocate the on-disk change-tracking bitmap when the device is attached (drbd.conf option bitmap).
+
+ -
+ name: resource-opts
+ doc: Options of a resource (drbd.conf options section).
+ attributes:
+ -
+ name: cpu-mask
+ type: string
+ checks:
+ max-len: cpu-mask-size
+ doc: CPU affinity mask for DRBD kernel threads, as a hexadecimal number (drbd.conf option cpu-mask).
+ -
+ name: on-no-data-accessible
+ type: u32
+ enum: on-no-data-policy
+ doc: Action to take for I/O requests when the requested data is not available locally or remotely (drbd.conf option on-no-data-accessible).
+ -
+ name: auto-promote
+ type: u8
+ doc: Whether to automatically promote the resource to primary when one of its devices is mounted or opened for writing (drbd.conf option auto-promote).
+ -
+ name: node-id
+ type: u32
+ doc: Unique node identifier of this node in the cluster (drbd.conf option node-id).
+ -
+ name: peer-ack-window
+ type: u32
+ doc: Amount of data, in sectors, DRBD replicates to a peer before issuing a peer-ack for the writes in between (drbd.conf option peer-ack-window).
+ -
+ name: twopc-timeout
+ type: u32
+ doc: Timeout for a two-phase-commit cluster-wide state transition, in units of 0.1 seconds (drbd.conf option twopc-timeout).
+ -
+ name: twopc-retry-timeout
+ type: u32
+ doc: Base multiplier for the randomized backoff before retrying a conflicting two-phase commit, in units of 0.1 seconds (drbd.conf option twopc-retry-timeout).
+ -
+ name: peer-ack-delay
+ type: u32
+ doc: How long to wait after the last write request before sending a peer-ack if no new write request arrives, in milliseconds (drbd.conf option peer-ack-delay).
+ -
+ name: auto-promote-timeout
+ type: u32
+ doc: How long to wait for a device to become promotable when auto-promote triggers a promotion, in units of 0.1 seconds (drbd.conf option auto-promote-timeout).
+ -
+ name: max-io-depth
+ type: u32
+ doc: Maximum number of outstanding application I/O requests on a DRBD device.
+ -
+ name: quorum
+ type: s32
+ doc: Number of nodes with an accessible disk required for a partition to have quorum, or a symbolic value (off, majority, all) (drbd.conf option quorum).
+ -
+ name: on-no-quorum
+ type: u32
+ enum: on-no-quorum-policy
+ doc: Action to take for I/O requests when the resource loses quorum (drbd.conf option on-no-quorum).
+ -
+ name: quorum-min-redundancy
+ type: s32
+ doc: Minimum number of nodes with an UpToDate disk required for a partition to gain quorum, or a symbolic value (off, majority, all) (drbd.conf option quorum-minimum-redundancy).
+ -
+ name: on-suspended-primary-outdated
+ type: u32
+ enum: on-suspended-primary-outdated-policy
+ doc: Action to take for a primary that has I/O suspended due to lost quorum and learns its data is outdated (drbd.conf option on-suspended-primary-outdated).
+ -
+ name: drbd8-compat-mode
+ type: u8
+ doc: Whether the resource operates in DRBD 8 compatible, single-peer mode.
+ -
+ name: explicit-drbd8-compat
+ type: u8
+ doc: Whether to explicitly (re)configure DRBD 8 compatible mode whenever a node-id is configured, instead of leaving it unchanged.
+
+ -
+ name: net-conf
+ doc: Options of a connection to a peer (drbd.conf net section).
+ attributes:
+ -
+ name: shared-secret
+ type: string
+ checks:
+ max-len: shared-secret-max
+ doc: Shared secret used for peer authentication (drbd.conf option shared-secret).
+ -
+ name: cram-hmac-alg
+ type: string
+ checks:
+ max-len: alg-name-max
+ doc: Hash-based message authentication code (HMAC) algorithm used for peer authentication (drbd.conf option cram-hmac-alg).
+ -
+ name: integrity-alg
+ type: string
+ checks:
+ max-len: alg-name-max
+ doc: Data integrity algorithm used, in addition to the TCP/IP checksum, to detect corruption of data received over the network (drbd.conf option data-integrity-alg).
+ -
+ name: verify-alg
+ type: string
+ checks:
+ max-len: alg-name-max
+ doc: Hash algorithm used to compute and compare block checksums during online verification (drbd.conf option verify-alg).
+ -
+ name: csums-alg
+ type: string
+ checks:
+ max-len: alg-name-max
+ doc: Hash algorithm used to compare blocks before resending identical data during a resync (drbd.conf option csums-alg).
+ -
+ name: protocol
+ type: u32
+ enum: wire-protocol
+ doc: Replication protocol used on this connection.
+ -
+ name: connect-int
+ type: u32
+ doc: How long to wait before retrying to establish a connection to the peer, in seconds (drbd.conf option connect-int).
+ -
+ name: timeout
+ type: u32
+ doc: Timeout for replies over the network before considering a peer dead, in units of 0.1 seconds (drbd.conf option timeout).
+ -
+ name: ping-int
+ type: u32
+ doc: Interval of connection idle time after which DRBD sends a keep-alive packet, in seconds (drbd.conf option ping-int).
+ -
+ name: ping-timeo
+ type: u32
+ doc: Timeout for replies to keep-alive packets, in units of 0.1 seconds (drbd.conf option ping-timeout).
+ -
+ name: sndbuf-size
+ type: u32
+ doc: Size of the TCP/IP send buffer, in bytes, or 0 to size it automatically (drbd.conf option sndbuf-size).
+ -
+ name: rcvbuf-size
+ type: u32
+ doc: Size of the TCP/IP receive buffer, in bytes, or 0 to size it automatically (drbd.conf option rcvbuf-size).
+ -
+ name: ko-count
+ type: u32
+ doc: Number of times the timeout may be exceeded before a non-responding peer is excluded from the cluster (drbd.conf option ko-count).
+ -
+ name: max-epoch-size
+ type: u32
+ doc: Maximum number of write requests DRBD may issue before issuing a write barrier (drbd.conf option max-epoch-size).
+ -
+ name: after-sb-0pri
+ type: u32
+ enum: after-sb-policy
+ doc: Split-brain recovery policy when neither node is primary.
+ -
+ name: after-sb-1pri
+ type: u32
+ enum: after-sb-policy
+ doc: Split-brain recovery policy when exactly one node is primary.
+ -
+ name: after-sb-2pri
+ type: u32
+ enum: after-sb-policy
+ doc: Split-brain recovery policy when both nodes are primary.
+ -
+ name: rr-conflict
+ type: u32
+ enum: after-sb-policy
+ doc: Policy for resolving a role/resync conflict left over from a split-brain recovery (drbd.conf option rr-conflict).
+ -
+ name: on-congestion
+ type: u32
+ enum: on-congestion-policy
+ doc: Action to take when the replication link's TCP send queue is congested (drbd.conf option on-congestion).
+ -
+ name: cong-fill
+ type: u32
+ doc: Amount of in-flight data, in sectors, allowed on this connection before switching into ahead/behind mode, or 0 to disable (drbd.conf option congestion-fill).
+ -
+ name: cong-extents
+ type: u32
+ doc: Number of active bitmap extents allowed before switching into ahead/behind mode (drbd.conf option congestion-extents).
+ -
+ name: two-primaries
+ type: u8
+ doc: Whether to allow both nodes of this connection to be primary at the same time (drbd.conf option allow-two-primaries).
+ -
+ name: tcp-cork
+ type: u8
+ doc: Whether to use the TCP_CORK socket option to coalesce partial messages into fewer, bigger packets (drbd.conf option tcp-cork).
+ -
+ name: always-asbp
+ type: u8
+ doc: Whether to apply the automatic split-brain recovery policies even when the UUIDs indicate a third node may be involved (drbd.conf option always-asbp).
+ -
+ name: use-rle
+ type: u8
+ doc: Whether to run-length encode the bitmap exchanged with the peer when a connection is established (drbd.conf option use-rle).
+ -
+ name: fencing
+ type: u32
+ enum: fencing-policy
+ doc: Fencing policy used to avoid a split-brain when both nodes are primary and disconnected.
+ -
+ name: connection-name
+ type: string
+ checks:
+ max-len: connection-name-max
+ doc: Name of the connection this net-conf applies to.
+ -
+ name: csums-after-crash-only
+ type: u8
+ doc: Whether to compare block checksums only for the resync after a primary crash, rather than for every resync (drbd.conf option csums-after-crash-only).
+ -
+ name: sock-check-timeo
+ type: u32
+ doc: Timeout used to poll the send/receive socket for a broken TCP connection when DRBD-proxy is in use, in units of 0.1 seconds (drbd.conf option socket-check-timeout).
+ -
+ name: transport-name
+ type: string
+ checks:
+ max-len: transport-name-max
+ doc: Name of the network transport module used for this connection, e.g. tcp or rdma (drbd.conf option transport).
+ -
+ name: max-buffers
+ type: u32
+ doc: Maximum number of pages used for receive buffers and internal resync/verify buffers per device (drbd.conf option max-buffers).
+ -
+ name: allow-remote-read
+ type: u8
+ doc: Whether to allow reads to be served from a peer's disk when the local disk is not up to date (drbd.conf option allow-remote-read).
+ -
+ name: tls
+ type: u8
+ doc: Whether to encrypt this connection with TLS (drbd.conf option tls).
+ -
+ name: tls-privkey
+ type: s32
+ doc: Serial number of the kernel keyring key holding the private key used for TLS encryption (drbd.conf option tls-privkey).
+ -
+ name: tls-certificate
+ type: s32
+ doc: Serial number of the kernel keyring key holding the certificate used for TLS encryption (drbd.conf option tls-certificate).
+ -
+ name: tls-keyring
+ type: s32
+ doc: Serial number of the kernel keyring used to look up the TLS key material (drbd.conf option tls-keyring).
+ -
+ name: load-balance-paths
+ type: u8
+ doc: Whether to load-balance traffic for this connection across all configured paths instead of using only one at a time (drbd.conf option load-balance-paths).
+ -
+ name: rdma-ctrl-rcvbuf-size
+ type: u32
+ doc: Size, in bytes, of the RDMA transport's control-stream receive buffer, or 0 to derive it from rcvbuf-size (drbd.conf option rdma-ctrl-rcvbuf-size).
+ -
+ name: rdma-ctrl-sndbuf-size
+ type: u32
+ doc: Size, in bytes, of the RDMA transport's control-stream send buffer, or 0 to derive it from sndbuf-size (drbd.conf option rdma-ctrl-sndbuf-size).
+
+ -
+ name: set-role-parms
+ doc: Parameters of the set-role request.
+ attributes:
+ -
+ name: force
+ type: flag
+ doc: Request that the promotion to primary succeed even if some devices are not guaranteed to have up-to-date data (drbdsetup primary option force).
+
+ -
+ name: resize-parms
+ doc: Parameters of the resize request.
+ attributes:
+ -
+ name: size
+ type: u64
+ doc: New size of the device, in sectors, or 0 to use the full size of the backing devices (drbdsetup resize option size).
+ -
+ name: assume-peer-has-space
+ type: flag
+ doc: Request that the resize proceed even though not all peer devices are currently connected, assuming they already have enough space.
+ -
+ name: assume-clean
+ type: flag
+ doc: Request that the additional space not be resynchronized, assuming it is already identical on all nodes.
+ -
+ name: al-stripes
+ type: u32
+ doc: Number of activity log stripes to use after the resize (drbdsetup resize option al-stripes).
+ -
+ name: al-stripe-size
+ type: u32
+ doc: Size of each activity log stripe, in KiB, after the resize (drbdsetup resize option al-stripe-size).
+
+ -
+ name: start-ov-parms
+ doc: Parameters of the start-ov (online verify) request.
+ attributes:
+ -
+ name: start-sector
+ type: u64
+ doc: Sector at which to start online verification, to resume a previously stopped run.
+ -
+ name: stop-sector
+ type: u64
+ doc: Sector at which to stop online verification.
+
+ -
+ name: new-current-uuid-parms
+ doc: Parameters of the new-current-uuid request.
+ attributes:
+ -
+ name: clear-bm
+ type: flag
+ doc: Request that the peer's bitmap also be cleared, marking the peer as fully in sync without a resync (drbdsetup new-current-uuid option clear-bitmap).
+ -
+ name: force-resync
+ type: flag
+ doc: Request that the peer be marked out of sync so that a resync starts, instead of assuming it already has the same data (drbdsetup new-current-uuid option force-resync).
+
+ -
+ name: disconnect-parms
+ doc: Parameters of the disconnect request.
+ attributes:
+ -
+ name: force
+ type: flag
+ doc: Request that the connection be torn down immediately, discarding any pending, not yet acknowledged data instead of waiting for it (drbdsetup disconnect option force).
+
+ -
+ name: detach-parms
+ doc: Parameters of the detach request.
+ attributes:
+ -
+ name: force
+ type: flag
+ doc: Request that the detach proceed immediately, marking the lower-level device failed until pending I/O completes (drbdsetup detach option force).
+ -
+ name: intentional-diskless-detach
+ type: flag
+ doc: Request that the device be marked permanently, intentionally diskless instead of temporarily diskless (drbdsetup detach option diskless).
+
+ -
+ name: device-conf
+ doc: Options of a device, set when the device is created (drbdsetup new-minor).
+ attributes:
+ -
+ name: max-bio-size
+ type: u32
+ doc: Maximum size of a single I/O request the device accepts, in bytes (drbdsetup new-minor option max-bio-size).
+ -
+ name: intentional-diskless
+ type: u8
+ doc: Whether the device is deliberately created without a local disk, e.g. as a diskless client (drbdsetup new-minor option diskless).
+ -
+ name: block-size
+ type: u32
+ doc: Logical block size the device reports, needed to combine differing backing disk block sizes across nodes (drbd.conf option block-size).
+ -
+ name: discard-granularity
+ type: u32
+ doc: Discard granularity, in bytes, advertised by the device; discard requests must be aligned to and a multiple of this value (drbd.conf option discard-granularity).
+
+ -
+ name: resource-info
+ doc: State of a resource.
+ attributes:
+ -
+ name: role
+ type: u32
+ enum: role
+ doc: Role of the resource on this node.
+ -
+ name: susp
+ type: u8
+ doc: Whether I/O on the resource is suspended by the user.
+ -
+ name: susp-nod
+ type: u8
+ doc: Whether I/O on the resource is suspended because no data is accessible.
+ -
+ name: susp-fen
+ type: u8
+ doc: Whether I/O on the resource is suspended because a fence-peer handler is running.
+ -
+ name: susp-quorum
+ type: u8
+ doc: Whether I/O on the resource is suspended because it has lost quorum.
+ -
+ name: fail-io
+ type: u8
+ doc: Whether all I/O on the resource fails because it was forced to demote from primary.
+
+ -
+ name: device-info
+ doc: State of a device.
+ attributes:
+ -
+ name: disk-state
+ type: u32
+ enum: disk-state
+ doc: Current state of the backing disk.
+ -
+ name: is-intentional-diskless
+ type: u8
+ doc: Whether the device is deliberately configured without a local disk.
+ -
+ name: has-quorum
+ type: u8
+ doc: Whether the device currently has quorum.
+ -
+ name: is-open
+ type: u8
+ doc: Whether the device is currently opened by an application.
+ -
+ name: backing-dev-path
+ type: string
+ checks:
+ max-len: device-path-max
+ doc: Path of the backing device, present while it is still accessible after a disk failure.
+
+ -
+ name: connection-info
+ doc: State of a connection to a peer.
+ attributes:
+ -
+ name: connection-state
+ type: u32
+ enum: connection-state
+ doc: Current state of the connection to the peer.
+ -
+ name: role
+ type: u32
+ enum: role
+ doc: Current role of the peer.
+
+ -
+ name: peer-device-info
+ doc: State of a peer device.
+ attributes:
+ -
+ name: repl-state
+ type: u32
+ enum: repl-state
+ doc: Current replication state with the peer.
+ -
+ name: disk-state
+ type: u32
+ enum: disk-state
+ doc: Current state of the peer's disk.
+ -
+ name: resync-susp-user
+ type: u8
+ doc: Whether resynchronization with this peer is suspended by the user.
+ -
+ name: resync-susp-peer
+ type: u8
+ doc: Whether resynchronization with this peer is suspended by the peer.
+ -
+ name: resync-susp-dependency
+ type: u8
+ doc: Whether resynchronization with this peer is suspended because it depends on another volume or connection resynchronizing first.
+ -
+ name: is-intentional-diskless
+ type: u8
+ doc: Whether the peer is deliberately configured without a local disk.
+ -
+ name: resync-susp-max-parallel
+ type: u8
+ doc: Whether resynchronization with this peer is suspended to enforce the maximum number of parallel resyncs.
+
+ -
+ name: resource-statistics
+ doc: Statistics of a resource.
+ attributes:
+ -
+ name: write-ordering
+ type: u32
+ enum: write-ordering
+ doc: Write ordering method currently in use for the resource's backing devices.
+
+ -
+ name: device-statistics
+ doc: Statistics of a device.
+ attributes:
+ -
+ name: size
+ type: u64
+ doc: Capacity of the device, in sectors.
+ -
+ name: read
+ type: u64
+ doc: Total data read from the device, in sectors, since it was attached.
+ -
+ name: write
+ type: u64
+ doc: Total data written to the device, in sectors, since it was attached.
+ -
+ name: al-writes
+ type: u64
+ doc: Number of activity log metadata writes since the device was attached.
+ -
+ name: bm-writes
+ type: u64
+ doc: Number of bitmap metadata writes since the device was attached.
+ -
+ name: upper-pending
+ type: u32
+ doc: Number of application I/O requests currently in flight on the device.
+ -
+ name: lower-pending
+ type: u32
+ doc: Number of I/O requests currently in flight on the backing device.
+ -
+ name: upper-blocked
+ type: u8
+ doc: Whether application I/O to the device is currently throttled.
+ -
+ name: lower-blocked
+ type: u8
+ doc: Whether I/O to the backing device is currently blocked.
+ -
+ name: al-suspended
+ type: u8
+ doc: Whether updates to the activity log are currently suspended.
+ -
+ name: exposed-data-uuid
+ type: u64
+ doc: Data generation UUID currently exposed to peers, identifying the generation of data this node has.
+ -
+ name: current-uuid
+ type: u64
+ doc: Current data generation UUID of the device's metadata.
+ -
+ name: disk-flags
+ type: u32
+ doc: Bitmask of the backing disk's metadata flags.
+ -
+ name: history-uuids
+ type: binary
+ sub-type: u64
+ checks:
+ max-len: history-uuids-size
+ doc: Historical bitmap UUIDs of prior data generations, one slot per peer.
+
+ -
+ name: connection-statistics
+ doc: Statistics of a connection to a peer.
+ attributes:
+ -
+ name: congested
+ type: u8
+ doc: Whether the network transport to the peer is currently congested.
+ -
+ name: ap-in-flight
+ type: u64
+ doc: Application data sent to the peer but not yet acknowledged, in sectors.
+ -
+ name: rs-in-flight
+ type: u64
+ doc: Resync data sent to the peer but not yet acknowledged, in sectors.
+
+ -
+ name: peer-device-statistics
+ doc: Statistics of a peer device.
+ attributes:
+ -
+ name: received
+ type: u64
+ doc: Total data received from the peer for this device, in sectors.
+ -
+ name: sent
+ type: u64
+ doc: Total data sent to the peer for this device, in sectors.
+ -
+ name: pending
+ type: u32
+ doc: Number of requests to this peer waiting for an acknowledgement.
+ -
+ name: unacked
+ type: u32
+ doc: Number of requests from this peer that have not yet been acknowledged.
+ -
+ name: out-of-sync
+ type: u64
+ doc: Amount of data known to differ from this peer, in sectors.
+ -
+ name: resync-failed
+ type: u64
+ doc: Amount of data that failed to resynchronize with this peer, in sectors.
+ -
+ name: bitmap-uuid
+ type: u64
+ doc: Bitmap UUID recorded for this peer.
+ -
+ name: flags
+ type: u32
+ doc: Bitmask of this peer device's metadata flags.
+ -
+ name: rs-total
+ type: u64
+ doc: Total amount of data to resynchronize with this peer for the current resync, in sectors.
+ -
+ name: ov-start-sector
+ type: u64
+ doc: Sector at which the current online verify run started.
+ -
+ name: ov-stop-sector
+ type: u64
+ doc: Sector at which the current online verify run will stop.
+ -
+ name: ov-position
+ type: u64
+ doc: Current position of an ongoing online verify run, in sectors.
+ -
+ name: ov-left
+ type: u64
+ doc: Amount of data left to verify in an ongoing online verify run, in sectors.
+ -
+ name: ov-skipped
+ type: u64
+ doc: Amount of data skipped during an ongoing online verify run, in sectors.
+ -
+ name: rs-same-csum
+ type: u64
+ doc: Amount of resync data found identical by checksum comparison and not retransmitted, in sectors.
+ -
+ name: rs-dt-start-ms
+ type: u64
+ doc: Time elapsed since the current resync with this peer started, in milliseconds.
+ -
+ name: rs-paused-ms
+ type: u64
+ doc: Total time the current resync with this peer has been paused, in milliseconds.
+ -
+ name: rs-dt0-ms
+ type: u64
+ doc: Time since the most recent resync progress mark, in milliseconds.
+ -
+ name: rs-db0-sectors
+ type: u64
+ doc: Data resynchronized since the most recent resync progress mark, in sectors.
+ -
+ name: rs-dt1-ms
+ type: u64
+ doc: Time since the resync progress mark before the most recent one, in milliseconds.
+ -
+ name: rs-db1-sectors
+ type: u64
+ doc: Data resynchronized since the resync progress mark before the most recent one, in sectors.
+ -
+ name: rs-c-sync-rate
+ type: u32
+ doc: Current dynamically controlled resync rate for this peer, in KiB/s.
+ -
+ name: uuid-flags
+ type: u64
+ doc: Bitmask of UUID-related flags negotiated with this peer.
+
+ -
+ name: helper-info
+ doc: Identity and outcome of a handler script invocation.
+ attributes:
+ -
+ name: name
+ type: string
+ checks:
+ max-len: helper-name-max
+ doc: Name of the handler script being invoked.
+ -
+ name: status
+ type: u32
+ doc: Exit status of the handler script, present when phase is response.
+ -
+ name: phase
+ type: u32
+ enum: helper-phase
+ doc: Whether this notification announces the call or reports its result.
+
+ -
+ name: invalidate-parms
+ doc: Parameters of the invalidate request.
+ attributes:
+ -
+ name: sync-from-peer-node-id
+ type: u32
+ doc: Node ID of the peer to resynchronize from; if absent, the kernel chooses a peer.
+ -
+ name: reset-bitmap
+ type: u8
+ doc: Whether to mark the whole bitmap out of sync before starting the resync, instead of resyncing only the blocks already marked out of sync (drbdsetup invalidate option reset-bitmap).
+
+ -
+ name: peer-device-conf
+ doc: Options of a peer device (drbd.conf peer-device-options section).
+ attributes:
+ -
+ name: resync-rate
+ type: u32
+ doc: Bandwidth DRBD may use for resynchronizing with this peer when the dynamic resync controller is disabled, in KiB/s (drbd.conf option resync-rate).
+ -
+ name: c-plan-ahead
+ type: u32
+ doc: How fast the dynamic resync-rate controller adapts to changes in resync speed, in units of 0.1 seconds, or 0 to use a fixed resync-rate instead (drbd.conf option c-plan-ahead).
+ -
+ name: c-delay-target
+ type: u32
+ doc: Delay the dynamic resync-rate controller aims for along the resync path, in units of 0.1 seconds (drbd.conf option c-delay-target).
+ -
+ name: c-fill-target
+ type: u32
+ doc: Amount of resync data the dynamic resync-rate controller aims to keep in flight, in sectors (drbd.conf option c-fill-target).
+ -
+ name: c-max-rate
+ type: u32
+ doc: Maximum bandwidth used by the dynamic resync-rate controller, in KiB/s, or 0 for no limit (drbd.conf option c-max-rate).
+ -
+ name: c-min-rate
+ type: u32
+ doc: Bandwidth reserved for resync I/O against this peer when this node is the sync source and primary, in KiB/s (drbd.conf option c-min-rate).
+ -
+ name: bitmap
+ type: u8
+ doc: Whether to allocate a bitmap slot for this peer, overriding the automatic default for a diskless peer (drbd.conf option bitmap).
+ -
+ name: resync-without-replication
+ type: u8
+ doc: Whether to suspend replication of application writes to this peer while it is being resynchronized (drbd.conf option resync-without-replication).
+ -
+ name: peer-tiebreaker
+ type: u8
+ doc: Whether this peer participates as a tiebreaker in quorum decisions even though it is intentionally diskless (drbd.conf option peer-tiebreaker).
+
+ -
+ name: connect-parms
+ doc: Parameters of the connect request.
+ attributes:
+ -
+ name: tentative
+ type: flag
+ doc: Request that DRBD only determine whether a connection could be established and which resync direction would be used, without actually connecting or resyncing (drbdsetup connect option tentative).
+ -
+ name: discard-my-data
+ type: flag
+ doc: Request that local data be discarded and resynchronized from the peer that has the most up-to-date data (drbdsetup connect option discard-my-data).
+
+ -
+ name: path-info
+ doc: State of a path.
+ attributes:
+ -
+ name: established
+ type: u8
+ doc: Whether the path's transport connection is currently established.
+
+ -
+ name: rename-parms
+ doc: Parameters of the rename request.
+ attributes:
+ -
+ name: new-name
+ type: string
+ checks:
+ max-len: resource-name-max
+ doc: New name for the resource.
+
+ -
+ name: invalidate-peer-parms
+ doc: Parameters of the invalidate-peer request.
+ attributes:
+ -
+ name: reset-bitmap
+ type: u8
+ doc: Whether to mark the whole bitmap out of sync before starting the resync, instead of resyncing only the blocks already marked out of sync (drbdsetup invalidate-remote option reset-bitmap).
+
+ -
+ name: suspend-io-parms
+ doc: Parameters of the suspend-io request.
+ attributes:
+ -
+ name: bdev-freeze
+ type: u8
+ doc: Whether to freeze the filesystem mounted on the device to bring it into a consistent state before suspending I/O (drbdsetup suspend-io option bdev-freeze).
+ -
+ name: flush-to-peers
+ type: u8
+ doc: Whether to wait until the peers have acknowledged the barrier that closes the last epoch, so that the writes before the suspension are on their stable storage also with protocols A and B (drbdsetup suspend-io option flush-to-peers).
+
+ -
+ name: cbt-start-parms
+ doc: Parameters of the device-cbt-start request.
+ attributes:
+ -
+ name: bookmark-id
+ type: u64
+ doc: Caller-chosen identifier of the backup snapshot the new tracking interval starts at; required.
+
+ -
+ name: cbt-info
+ doc: Changed block tracking state of a device. All but active are absent while tracking is inactive.
+ attributes:
+ -
+ name: active
+ type: u8
+ doc: Whether a bitmap slot tracks the blocks changed since a bookmark.
+ -
+ name: valid
+ type: u8
+ doc: Whether the tracking bitmap holds every change since the bookmark; cleared when changes may have been missed, which requires a full backup.
+ -
+ name: bookmark-id
+ type: u64
+ doc: Identifier given to the device-cbt-start that started the interval.
+ -
+ name: since
+ type: u64
+ doc: Time of that device-cbt-start, in seconds since the epoch.
+ -
+ name: bitmap-uuid
+ type: u64
+ doc: Current UUID of the device at that device-cbt-start.
+ -
+ name: bitmap-dagtag
+ type: u64
+ doc: Data generation tag of the Primary at that device-cbt-start; on a Secondary the last one received from the Primary.
+
+ -
+ name: resource
+ doc: A resource with its context, state, statistics and options.
+ attributes:
+ -
+ name: context
+ type: nest
+ nested-attributes: context
+ doc: Identity of the object.
+ -
+ name: info
+ type: nest
+ nested-attributes: resource-info
+ doc: State of the object.
+ -
+ name: statistics
+ type: nest
+ nested-attributes: resource-statistics
+ doc: Statistics of the object.
+ -
+ name: new-name
+ type: string
+ checks:
+ max-len: resource-name-max
+ doc: New name of the resource; present in state-change messages with action
+ rename.
+ -
+ name: resource-opts
+ type: nest
+ nested-attributes: resource-opts
+ doc: Resource-wide options in effect.
+
+ -
+ name: device
+ doc: A device with its context, state, statistics and options.
+ attributes:
+ -
+ name: context
+ type: nest
+ nested-attributes: context
+ doc: Identity of the object.
+ -
+ name: info
+ type: nest
+ nested-attributes: device-info
+ doc: State of the object.
+ -
+ name: statistics
+ type: nest
+ nested-attributes: device-statistics
+ doc: Statistics of the object.
+ -
+ name: disk-conf
+ type: nest
+ nested-attributes: disk-conf
+ doc: Backing disk options in effect; absent while the device is diskless.
+ -
+ name: device-conf
+ type: nest
+ nested-attributes: device-conf
+ doc: Device options in effect.
+
+ -
+ name: connection
+ doc: A connection with its context, state, statistics, options and paths.
+ attributes:
+ -
+ name: context
+ type: nest
+ nested-attributes: context
+ doc: Identity of the object.
+ -
+ name: info
+ type: nest
+ nested-attributes: connection-info
+ doc: State of the object.
+ -
+ name: statistics
+ type: nest
+ nested-attributes: connection-statistics
+ doc: Statistics of the object.
+ -
+ name: net-conf
+ type: nest
+ nested-attributes: net-conf
+ doc: Connection options in effect; absent when the connection has none.
+ -
+ name: path
+ type: nest
+ nested-attributes: path
+ multi-attr: true
+ doc: The paths of this connection, one attribute each. Also available on
+ their own through path-get.
+
+ -
+ name: peer-device
+ doc: A peer device with its context, state, statistics and options.
+ attributes:
+ -
+ name: context
+ type: nest
+ nested-attributes: context
+ doc: Identity of the object.
+ -
+ name: info
+ type: nest
+ nested-attributes: peer-device-info
+ doc: State of the object.
+ -
+ name: statistics
+ type: nest
+ nested-attributes: peer-device-statistics
+ doc: Statistics of the object.
+ -
+ name: peer-device-conf
+ type: nest
+ nested-attributes: peer-device-conf
+ doc: Peer device options in effect; absent when the peer device has none.
+
+ -
+ name: path
+ doc: A path with its context and state.
+ attributes:
+ -
+ name: context
+ type: nest
+ nested-attributes: context
+ doc: Identity of the object.
+ -
+ name: info
+ type: nest
+ nested-attributes: path-info
+ doc: State of the object.
+
+ -
+ name: drbd2
+ doc: Top-level attributes of drbd2 requests, replies and notifications.
+ attributes:
+ -
+ name: context
+ type: nest
+ nested-attributes: context
+ doc: Object the message refers to.
+ -
+ name: set-defaults
+ type: flag
+ doc: Reset all options not given in this request to their defaults before
+ applying it.
+ -
+ name: state-result
+ type: u32
+ enum: state-result
+ doc: Outcome of the state change performed by the request.
+ -
+ name: message
+ type: string
+ doc: Human-readable explanation accompanying state-result.
+ -
+ name: timeout-type
+ type: u32
+ enum: timeout-type
+ doc: Reply of timeout-type-get.
+ -
+ name: disk-conf
+ type: nest
+ nested-attributes: disk-conf
+ doc: Backing disk options for device-attach and disk-set.
+ -
+ name: resource-opts
+ type: nest
+ nested-attributes: resource-opts
+ doc: Resource-wide options for resource-new and resource-set.
+ -
+ name: net-conf
+ type: nest
+ nested-attributes: net-conf
+ doc: Connection options for connection-new and connection-set.
+ -
+ name: set-role-parms
+ type: nest
+ nested-attributes: set-role-parms
+ doc: Parameters of resource-primary and resource-secondary.
+ -
+ name: resize-parms
+ type: nest
+ nested-attributes: resize-parms
+ doc: Parameters of device-resize.
+ -
+ name: start-ov-parms
+ type: nest
+ nested-attributes: start-ov-parms
+ doc: Parameters of peer-device-start-ov.
+ -
+ name: new-current-uuid-parms
+ type: nest
+ nested-attributes: new-current-uuid-parms
+ doc: Parameters of device-new-current-uuid.
+ -
+ name: disconnect-parms
+ type: nest
+ nested-attributes: disconnect-parms
+ doc: Parameters of connection-disconnect and connection-del.
+ -
+ name: detach-parms
+ type: nest
+ nested-attributes: detach-parms
+ doc: Parameters of device-detach.
+ -
+ name: device-conf
+ type: nest
+ nested-attributes: device-conf
+ doc: Device options set when the device is created by device-new.
+ -
+ name: helper
+ type: nest
+ nested-attributes: helper-info
+ doc: Identity and outcome of the handler script reported by helper-ntf.
+ -
+ name: invalidate-parms
+ type: nest
+ nested-attributes: invalidate-parms
+ doc: Parameters of device-invalidate.
+ -
+ name: peer-device-conf
+ type: nest
+ nested-attributes: peer-device-conf
+ doc: Peer device options for peer-device-set.
+ -
+ name: connect-parms
+ type: nest
+ nested-attributes: connect-parms
+ doc: Parameters of connection-connect.
+ -
+ name: rename-parms
+ type: nest
+ nested-attributes: rename-parms
+ doc: Parameters of resource-rename.
+ -
+ name: invalidate-peer-parms
+ type: nest
+ nested-attributes: invalidate-peer-parms
+ doc: Parameters of peer-device-invalidate.
+ -
+ name: suspend-io-parms
+ type: nest
+ nested-attributes: suspend-io-parms
+ doc: Parameters of resource-suspend-io.
+ -
+ name: resource
+ type: nest
+ nested-attributes: resource
+ doc: A resource (dump reply).
+ -
+ name: device
+ type: nest
+ nested-attributes: device
+ doc: A device (dump reply).
+ -
+ name: connection
+ type: nest
+ nested-attributes: connection
+ doc: A connection (dump reply).
+ -
+ name: peer-device
+ type: nest
+ nested-attributes: peer-device
+ doc: A peer device (dump reply).
+ -
+ name: path
+ type: nest
+ nested-attributes: path
+ doc: A path (dump reply).
+ -
+ name: cbt-start-parms
+ type: nest
+ nested-attributes: cbt-start-parms
+ doc: Parameters of device-cbt-start.
+ -
+ name: cbt-info
+ type: nest
+ nested-attributes: cbt-info
+ doc: Reply of device-cbt-status.
+
+ -
+ name: state-change
+ doc: One object of a state change or of the initial state snapshot. Exactly one
+ object nest is present.
+ attributes:
+ -
+ name: action
+ type: u32
+ enum: state-change-action
+ doc: Whether the object exists, was created, changed or destroyed.
+ -
+ name: more
+ type: flag
+ doc: Further messages belong to the same state change.
+ -
+ name: resource
+ type: nest
+ nested-attributes: resource
+ doc: Present when the message describes a resource.
+ -
+ name: device
+ type: nest
+ nested-attributes: device
+ doc: Present when the message describes a device.
+ -
+ name: connection
+ type: nest
+ nested-attributes: connection
+ doc: Present when the message describes a connection.
+ -
+ name: peer-device
+ type: nest
+ nested-attributes: peer-device
+ doc: Present when the message describes a peer device.
+ -
+ name: path
+ type: nest
+ nested-attributes: path
+ doc: Present when the message describes a path.
+
+operations:
+ list:
+ -
+ name: resource-new
+ doc: Create a new DRBD resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - resource-opts
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-del
+ doc: Delete a DRBD resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-set
+ doc: Change resource options.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - resource-opts
+ - set-defaults
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-rename
+ doc: Rename a DRBD resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - rename-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-down
+ doc: Shut down a DRBD resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-primary
+ doc: Promote the resource to primary role.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - set-role-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-secondary
+ doc: Demote the resource to secondary role.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - set-role-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-suspend-io
+ doc: Suspend I/O on a resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - suspend-io-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: resource-resume-io
+ doc: Resume I/O on a resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-new
+ doc: Add a DRBD device as a volume to a resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - device-conf
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-del
+ doc: Delete a DRBD device.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-attach
+ doc: Attach a local backing device to a DRBD device.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - disk-conf
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-detach
+ doc: Detach the local backing device from a DRBD device.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - detach-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: disk-set
+ doc: Change disk options. Applies to the attached backing disk; see
+ device-attach.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - disk-conf
+ - set-defaults
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-resize
+ doc: Resize the DRBD device.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - resize-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-outdate
+ doc: Mark the local data as outdated.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-invalidate
+ doc: Invalidate the local data of a DRBD device.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - invalidate-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-new-current-uuid
+ doc: Generate a new current UUID.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - new-current-uuid-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: connection-new
+ doc: Add a new peer to a resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - net-conf
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: connection-del
+ doc: Delete a peer from a resource.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - disconnect-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: connection-connect
+ doc: Connect to a peer.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - connect-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: connection-disconnect
+ doc: Disconnect from a peer.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - disconnect-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: connection-set
+ doc: Change network options.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - net-conf
+ - set-defaults
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: connection-forget
+ doc: Forget a previously connected peer. The peer is identified by
+ context.peer-node-id.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: path-new
+ doc: Add a new network path to a connection.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: path-del
+ doc: Delete a network path from a connection.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: peer-device-set
+ doc: Change peer device options.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - peer-device-conf
+ - set-defaults
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: peer-device-invalidate
+ doc: Invalidate the peer data of a DRBD peer device.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - invalidate-peer-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: peer-device-pause-sync
+ doc: Pause resynchronization.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: peer-device-resume-sync
+ doc: Resume resynchronization.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: peer-device-start-ov
+ doc: Start online verification.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - start-ov-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: timeout-type-get
+ doc: Get the timeout type for a peer device.
+ attribute-set: drbd2
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - timeout-type
+
+ -
+ name: resource-get
+ doc: Dump all DRBD resources. Optionally filtered by context.
+ attribute-set: drbd2
+ dump:
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - resource
+
+ -
+ name: device-get
+ doc: Dump all DRBD devices. Optionally filtered by context.
+ attribute-set: drbd2
+ dump:
+ post: drbd2-nl-device-get-done
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - device
+
+ -
+ name: connection-get
+ doc: Dump all DRBD connections. Optionally filtered by context.
+ attribute-set: drbd2
+ dump:
+ post: drbd2-nl-connection-get-done
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - connection
+
+ -
+ name: peer-device-get
+ doc: Dump all DRBD peer devices. Optionally filtered by context.
+ attribute-set: drbd2
+ dump:
+ post: drbd2-nl-peer-device-get-done
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - peer-device
+
+ -
+ name: path-get
+ doc: Dump all DRBD paths. Optionally filtered by context.
+ attribute-set: drbd2
+ dump:
+ post: drbd2-nl-path-get-done
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - path
+
+ -
+ name: state-get
+ doc: Dump a consistent snapshot of all resources, devices, connections, peer
+ devices and paths as state-change messages with action exists.
+ attribute-set: state-change
+ dump:
+ post: drbd2-nl-state-get-done
+ reply:
+ attributes:
+ - action
+ - more
+ - resource
+ - device
+ - connection
+ - peer-device
+ - path
+
+ -
+ name: state-change-ntf
+ doc: Notification about a state change; same layout as state-get replies. Messages
+ of one state change have more set on all but the last.
+ notify: state-get
+ mcgrp: events
+
+ -
+ name: helper-ntf
+ doc: A user-space helper is about to be called (phase call) or has finished
+ (phase response).
+ attribute-set: drbd2
+ event:
+ attributes:
+ - context
+ - helper
+ mcgrp: events
+
+ -
+ name: device-cbt-flush
+ doc: Write the bitmaps and the superblock to the meta data device, so that a
+ snapshot of the meta data taken while I/O stays suspended carries every
+ changed block. Requires I/O suspended and drained, no peer write still
+ executing, no resync running, and a disk that is at least Outdated.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-cbt-start
+ doc: Clear the changed block tracking bitmap and start a new tracking interval
+ at a bookmark. Allocates the tracking bitmap slot if needed. Requires a
+ device-cbt-flush within the same suspension of I/O and no write to the
+ backing device since.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ - cbt-start-parms
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-cbt-stop
+ doc: Stop changed block tracking and free its bitmap slot.
+ attribute-set: drbd2
+ flags:
+ - admin-perm
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - state-result
+ - message
+
+ -
+ name: device-cbt-status
+ doc: Get the changed block tracking state of a device.
+ attribute-set: drbd2
+ do:
+ pre: drbd2-pre-doit
+ post: drbd2-post-doit
+ request:
+ attributes:
+ - context
+ reply:
+ attributes:
+ - cbt-info
+
+mcast-groups:
+ list:
+ -
+ name: events
diff --git a/drivers/block/drbd/Kconfig b/drivers/block/drbd/Kconfig
index 1b8b3f6efc1b..261965d7b054 100644
--- a/drivers/block/drbd/Kconfig
+++ b/drivers/block/drbd/Kconfig
@@ -108,4 +108,10 @@ config DRBD_COMPAT_84
this compile-time option, creating a resource without a node
ID is not possible.
+ This option also provides the "drbd" generic netlink
+ family at version 1, which drbd-utils have always used to
+ configure DRBD. Without it, DRBD can only be configured
+ through the drbd2 family, which older drbd-utils releases
+ do not know.
+
If unsure, say Y.
diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index 0b7bc8f5d53a..74694d95fe39 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -6,6 +6,7 @@ drbd-y += drbd_interval.o drbd_state.o
drbd-$(CONFIG_DRBD_COMPAT_84) += drbd_nl_gen.o
drbd-y += drbd_nl_defaults.o
drbd-$(CONFIG_DRBD_COMPAT_84) += drbd_nl_84.o
+drbd-y += drbd_nl_drbd2.o drbd2_nl_gen.o
drbd-y += drbd_transport.o
drbd-$(CONFIG_DEV_DAX_PMEM) += drbd_dax_pmem.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
diff --git a/drivers/block/drbd/drbd2_nl_gen.c b/drivers/block/drbd/drbd2_nl_gen.c
new file mode 100644
index 000000000000..c0b19c0993de
--- /dev/null
+++ b/drivers/block/drbd/drbd2_nl_gen.c
@@ -0,0 +1,810 @@
+// SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+/* Do not edit directly, auto-generated from: */
+/* Documentation/netlink/specs/drbd2.yaml */
+/* YNL-GEN kernel source */
+/* To regenerate run: tools/net/ynl/ynl-regen.sh */
+
+#include <net/netlink.h>
+#include <net/genetlink.h>
+
+#include "drbd2_nl_gen.h"
+
+#include <uapi/linux/drbd2.h>
+
+/* Common nested types */
+const struct nla_policy drbd2_address_nl_policy[DRBD2_A_ADDRESS_IPV6 + 1] = {
+ [DRBD2_A_ADDRESS_FAMILY] = { .type = NLA_U16, },
+ [DRBD2_A_ADDRESS_PORT] = { .type = NLA_BE16, },
+ [DRBD2_A_ADDRESS_IPV4] = { .type = NLA_BE32, },
+ [DRBD2_A_ADDRESS_IPV6] = NLA_POLICY_EXACT_LEN(16),
+};
+
+const struct nla_policy drbd2_cbt_start_parms_nl_policy[DRBD2_A_CBT_START_PARMS_BOOKMARK_ID + 1] = {
+ [DRBD2_A_CBT_START_PARMS_BOOKMARK_ID] = { .type = NLA_U64, },
+};
+
+const struct nla_policy drbd2_connect_parms_nl_policy[DRBD2_A_CONNECT_PARMS_DISCARD_MY_DATA + 1] = {
+ [DRBD2_A_CONNECT_PARMS_TENTATIVE] = { .type = NLA_FLAG, },
+ [DRBD2_A_CONNECT_PARMS_DISCARD_MY_DATA] = { .type = NLA_FLAG, },
+};
+
+const struct nla_policy drbd2_context_nl_policy[DRBD2_A_CONTEXT_PEER_ADDRESS + 1] = {
+ [DRBD2_A_CONTEXT_RESOURCE_NAME] = { .type = NLA_NUL_STRING, .len = DRBD2_RESOURCE_NAME_MAX, },
+ [DRBD2_A_CONTEXT_VOLUME] = { .type = NLA_U32, },
+ [DRBD2_A_CONTEXT_MINOR] = { .type = NLA_U32, },
+ [DRBD2_A_CONTEXT_PEER_NODE_ID] = { .type = NLA_U32, },
+ [DRBD2_A_CONTEXT_CONNECTION_NAME] = { .type = NLA_NUL_STRING, .len = DRBD2_CONNECTION_NAME_MAX, },
+ [DRBD2_A_CONTEXT_MY_ADDRESS] = NLA_POLICY_NESTED(drbd2_address_nl_policy),
+ [DRBD2_A_CONTEXT_PEER_ADDRESS] = NLA_POLICY_NESTED(drbd2_address_nl_policy),
+};
+
+const struct nla_policy drbd2_detach_parms_nl_policy[DRBD2_A_DETACH_PARMS_INTENTIONAL_DISKLESS_DETACH + 1] = {
+ [DRBD2_A_DETACH_PARMS_FORCE] = { .type = NLA_FLAG, },
+ [DRBD2_A_DETACH_PARMS_INTENTIONAL_DISKLESS_DETACH] = { .type = NLA_FLAG, },
+};
+
+const struct nla_policy drbd2_device_conf_nl_policy[DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY + 1] = {
+ [DRBD2_A_DEVICE_CONF_MAX_BIO_SIZE] = { .type = NLA_U32, },
+ [DRBD2_A_DEVICE_CONF_INTENTIONAL_DISKLESS] = { .type = NLA_U8, },
+ [DRBD2_A_DEVICE_CONF_BLOCK_SIZE] = { .type = NLA_U32, },
+ [DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY] = { .type = NLA_U32, },
+};
+
+const struct nla_policy drbd2_disconnect_parms_nl_policy[DRBD2_A_DISCONNECT_PARMS_FORCE + 1] = {
+ [DRBD2_A_DISCONNECT_PARMS_FORCE] = { .type = NLA_FLAG, },
+};
+
+const struct nla_policy drbd2_disk_conf_nl_policy[DRBD2_A_DISK_CONF_BITMAP + 1] = {
+ [DRBD2_A_DISK_CONF_BACKING_DEV] = { .type = NLA_NUL_STRING, .len = DRBD2_DEVICE_PATH_MAX, },
+ [DRBD2_A_DISK_CONF_META_DEV] = { .type = NLA_NUL_STRING, .len = DRBD2_DEVICE_PATH_MAX, },
+ [DRBD2_A_DISK_CONF_META_DEV_IDX] = { .type = NLA_S32, },
+ [DRBD2_A_DISK_CONF_SIZE] = { .type = NLA_U64, },
+ [DRBD2_A_DISK_CONF_ON_IO_ERROR] = NLA_POLICY_MAX(NLA_U32, 2),
+ [DRBD2_A_DISK_CONF_RESYNC_AFTER] = { .type = NLA_S32, },
+ [DRBD2_A_DISK_CONF_AL_EXTENTS] = { .type = NLA_U32, },
+ [DRBD2_A_DISK_CONF_DISK_BARRIER] = { .type = NLA_U8, },
+ [DRBD2_A_DISK_CONF_DISK_FLUSHES] = { .type = NLA_U8, },
+ [DRBD2_A_DISK_CONF_DISK_DRAIN] = { .type = NLA_U8, },
+ [DRBD2_A_DISK_CONF_MD_FLUSHES] = { .type = NLA_U8, },
+ [DRBD2_A_DISK_CONF_DISK_TIMEOUT] = { .type = NLA_U32, },
+ [DRBD2_A_DISK_CONF_READ_BALANCING] = NLA_POLICY_MAX(NLA_U32, 10),
+ [DRBD2_A_DISK_CONF_UNPLUG_WATERMARK] = { .type = NLA_U32, },
+ [DRBD2_A_DISK_CONF_RS_DISCARD_GRANULARITY] = { .type = NLA_U32, },
+ [DRBD2_A_DISK_CONF_AL_UPDATES] = { .type = NLA_U8, },
+ [DRBD2_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED] = { .type = NLA_U8, },
+ [DRBD2_A_DISK_CONF_DISABLE_WRITE_SAME] = { .type = NLA_U8, },
+ [DRBD2_A_DISK_CONF_BITMAP] = { .type = NLA_U8, },
+};
+
+const struct nla_policy drbd2_invalidate_parms_nl_policy[DRBD2_A_INVALIDATE_PARMS_RESET_BITMAP + 1] = {
+ [DRBD2_A_INVALIDATE_PARMS_SYNC_FROM_PEER_NODE_ID] = { .type = NLA_U32, },
+ [DRBD2_A_INVALIDATE_PARMS_RESET_BITMAP] = { .type = NLA_U8, },
+};
+
+const struct nla_policy drbd2_invalidate_peer_parms_nl_policy[DRBD2_A_INVALIDATE_PEER_PARMS_RESET_BITMAP + 1] = {
+ [DRBD2_A_INVALIDATE_PEER_PARMS_RESET_BITMAP] = { .type = NLA_U8, },
+};
+
+const struct nla_policy drbd2_net_conf_nl_policy[DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE + 1] = {
+ [DRBD2_A_NET_CONF_SHARED_SECRET] = { .type = NLA_NUL_STRING, .len = DRBD2_SHARED_SECRET_MAX, },
+ [DRBD2_A_NET_CONF_CRAM_HMAC_ALG] = { .type = NLA_NUL_STRING, .len = DRBD2_ALG_NAME_MAX, },
+ [DRBD2_A_NET_CONF_INTEGRITY_ALG] = { .type = NLA_NUL_STRING, .len = DRBD2_ALG_NAME_MAX, },
+ [DRBD2_A_NET_CONF_VERIFY_ALG] = { .type = NLA_NUL_STRING, .len = DRBD2_ALG_NAME_MAX, },
+ [DRBD2_A_NET_CONF_CSUMS_ALG] = { .type = NLA_NUL_STRING, .len = DRBD2_ALG_NAME_MAX, },
+ [DRBD2_A_NET_CONF_PROTOCOL] = NLA_POLICY_RANGE(NLA_U32, 1, 3),
+ [DRBD2_A_NET_CONF_CONNECT_INT] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_TIMEOUT] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_PING_INT] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_PING_TIMEO] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_SNDBUF_SIZE] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_RCVBUF_SIZE] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_KO_COUNT] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_MAX_EPOCH_SIZE] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_AFTER_SB_0PRI] = NLA_POLICY_MAX(NLA_U32, 12),
+ [DRBD2_A_NET_CONF_AFTER_SB_1PRI] = NLA_POLICY_MAX(NLA_U32, 12),
+ [DRBD2_A_NET_CONF_AFTER_SB_2PRI] = NLA_POLICY_MAX(NLA_U32, 12),
+ [DRBD2_A_NET_CONF_RR_CONFLICT] = NLA_POLICY_MAX(NLA_U32, 12),
+ [DRBD2_A_NET_CONF_ON_CONGESTION] = NLA_POLICY_MAX(NLA_U32, 2),
+ [DRBD2_A_NET_CONF_CONG_FILL] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_CONG_EXTENTS] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_TWO_PRIMARIES] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_TCP_CORK] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_ALWAYS_ASBP] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_USE_RLE] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_FENCING] = NLA_POLICY_MAX(NLA_U32, 2),
+ [DRBD2_A_NET_CONF_CONNECTION_NAME] = { .type = NLA_NUL_STRING, .len = DRBD2_CONNECTION_NAME_MAX, },
+ [DRBD2_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_SOCK_CHECK_TIMEO] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_TRANSPORT_NAME] = { .type = NLA_NUL_STRING, .len = DRBD2_TRANSPORT_NAME_MAX, },
+ [DRBD2_A_NET_CONF_MAX_BUFFERS] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_ALLOW_REMOTE_READ] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_TLS] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_TLS_PRIVKEY] = { .type = NLA_S32, },
+ [DRBD2_A_NET_CONF_TLS_CERTIFICATE] = { .type = NLA_S32, },
+ [DRBD2_A_NET_CONF_TLS_KEYRING] = { .type = NLA_S32, },
+ [DRBD2_A_NET_CONF_LOAD_BALANCE_PATHS] = { .type = NLA_U8, },
+ [DRBD2_A_NET_CONF_RDMA_CTRL_RCVBUF_SIZE] = { .type = NLA_U32, },
+ [DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE] = { .type = NLA_U32, },
+};
+
+const struct nla_policy drbd2_new_current_uuid_parms_nl_policy[DRBD2_A_NEW_CURRENT_UUID_PARMS_FORCE_RESYNC + 1] = {
+ [DRBD2_A_NEW_CURRENT_UUID_PARMS_CLEAR_BM] = { .type = NLA_FLAG, },
+ [DRBD2_A_NEW_CURRENT_UUID_PARMS_FORCE_RESYNC] = { .type = NLA_FLAG, },
+};
+
+const struct nla_policy drbd2_peer_device_conf_nl_policy[DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER + 1] = {
+ [DRBD2_A_PEER_DEVICE_CONF_RESYNC_RATE] = { .type = NLA_U32, },
+ [DRBD2_A_PEER_DEVICE_CONF_C_PLAN_AHEAD] = { .type = NLA_U32, },
+ [DRBD2_A_PEER_DEVICE_CONF_C_DELAY_TARGET] = { .type = NLA_U32, },
+ [DRBD2_A_PEER_DEVICE_CONF_C_FILL_TARGET] = { .type = NLA_U32, },
+ [DRBD2_A_PEER_DEVICE_CONF_C_MAX_RATE] = { .type = NLA_U32, },
+ [DRBD2_A_PEER_DEVICE_CONF_C_MIN_RATE] = { .type = NLA_U32, },
+ [DRBD2_A_PEER_DEVICE_CONF_BITMAP] = { .type = NLA_U8, },
+ [DRBD2_A_PEER_DEVICE_CONF_RESYNC_WITHOUT_REPLICATION] = { .type = NLA_U8, },
+ [DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER] = { .type = NLA_U8, },
+};
+
+const struct nla_policy drbd2_rename_parms_nl_policy[DRBD2_A_RENAME_PARMS_NEW_NAME + 1] = {
+ [DRBD2_A_RENAME_PARMS_NEW_NAME] = { .type = NLA_NUL_STRING, .len = DRBD2_RESOURCE_NAME_MAX, },
+};
+
+const struct nla_policy drbd2_resize_parms_nl_policy[DRBD2_A_RESIZE_PARMS_AL_STRIPE_SIZE + 1] = {
+ [DRBD2_A_RESIZE_PARMS_SIZE] = { .type = NLA_U64, },
+ [DRBD2_A_RESIZE_PARMS_ASSUME_PEER_HAS_SPACE] = { .type = NLA_FLAG, },
+ [DRBD2_A_RESIZE_PARMS_ASSUME_CLEAN] = { .type = NLA_FLAG, },
+ [DRBD2_A_RESIZE_PARMS_AL_STRIPES] = { .type = NLA_U32, },
+ [DRBD2_A_RESIZE_PARMS_AL_STRIPE_SIZE] = { .type = NLA_U32, },
+};
+
+const struct nla_policy drbd2_resource_opts_nl_policy[DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT + 1] = {
+ [DRBD2_A_RESOURCE_OPTS_CPU_MASK] = { .type = NLA_NUL_STRING, .len = DRBD2_CPU_MASK_SIZE, },
+ [DRBD2_A_RESOURCE_OPTS_ON_NO_DATA_ACCESSIBLE] = NLA_POLICY_MAX(NLA_U32, 1),
+ [DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE] = { .type = NLA_U8, },
+ [DRBD2_A_RESOURCE_OPTS_NODE_ID] = { .type = NLA_U32, },
+ [DRBD2_A_RESOURCE_OPTS_PEER_ACK_WINDOW] = { .type = NLA_U32, },
+ [DRBD2_A_RESOURCE_OPTS_TWOPC_TIMEOUT] = { .type = NLA_U32, },
+ [DRBD2_A_RESOURCE_OPTS_TWOPC_RETRY_TIMEOUT] = { .type = NLA_U32, },
+ [DRBD2_A_RESOURCE_OPTS_PEER_ACK_DELAY] = { .type = NLA_U32, },
+ [DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE_TIMEOUT] = { .type = NLA_U32, },
+ [DRBD2_A_RESOURCE_OPTS_MAX_IO_DEPTH] = { .type = NLA_U32, },
+ [DRBD2_A_RESOURCE_OPTS_QUORUM] = { .type = NLA_S32, },
+ [DRBD2_A_RESOURCE_OPTS_ON_NO_QUORUM] = NLA_POLICY_MAX(NLA_U32, 1),
+ [DRBD2_A_RESOURCE_OPTS_QUORUM_MIN_REDUNDANCY] = { .type = NLA_S32, },
+ [DRBD2_A_RESOURCE_OPTS_ON_SUSPENDED_PRIMARY_OUTDATED] = NLA_POLICY_MAX(NLA_U32, 1),
+ [DRBD2_A_RESOURCE_OPTS_DRBD8_COMPAT_MODE] = { .type = NLA_U8, },
+ [DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT] = { .type = NLA_U8, },
+};
+
+const struct nla_policy drbd2_set_role_parms_nl_policy[DRBD2_A_SET_ROLE_PARMS_FORCE + 1] = {
+ [DRBD2_A_SET_ROLE_PARMS_FORCE] = { .type = NLA_FLAG, },
+};
+
+const struct nla_policy drbd2_start_ov_parms_nl_policy[DRBD2_A_START_OV_PARMS_STOP_SECTOR + 1] = {
+ [DRBD2_A_START_OV_PARMS_START_SECTOR] = { .type = NLA_U64, },
+ [DRBD2_A_START_OV_PARMS_STOP_SECTOR] = { .type = NLA_U64, },
+};
+
+const struct nla_policy drbd2_suspend_io_parms_nl_policy[DRBD2_A_SUSPEND_IO_PARMS_FLUSH_TO_PEERS + 1] = {
+ [DRBD2_A_SUSPEND_IO_PARMS_BDEV_FREEZE] = { .type = NLA_U8, },
+ [DRBD2_A_SUSPEND_IO_PARMS_FLUSH_TO_PEERS] = { .type = NLA_U8, },
+};
+
+/* DRBD2_CMD_RESOURCE_NEW - do */
+static const struct nla_policy drbd2_resource_new_nl_policy[DRBD2_A_RESOURCE_OPTS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_RESOURCE_OPTS] = NLA_POLICY_NESTED(drbd2_resource_opts_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_DEL - do */
+static const struct nla_policy drbd2_resource_del_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_SET - do */
+static const struct nla_policy drbd2_resource_set_nl_policy[DRBD2_A_RESOURCE_OPTS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_RESOURCE_OPTS] = NLA_POLICY_NESTED(drbd2_resource_opts_nl_policy),
+ [DRBD2_A_SET_DEFAULTS] = { .type = NLA_FLAG, },
+};
+
+/* DRBD2_CMD_RESOURCE_RENAME - do */
+static const struct nla_policy drbd2_resource_rename_nl_policy[DRBD2_A_RENAME_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_RENAME_PARMS] = NLA_POLICY_NESTED(drbd2_rename_parms_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_DOWN - do */
+static const struct nla_policy drbd2_resource_down_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_PRIMARY - do */
+static const struct nla_policy drbd2_resource_primary_nl_policy[DRBD2_A_SET_ROLE_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_SET_ROLE_PARMS] = NLA_POLICY_NESTED(drbd2_set_role_parms_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_SECONDARY - do */
+static const struct nla_policy drbd2_resource_secondary_nl_policy[DRBD2_A_SET_ROLE_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_SET_ROLE_PARMS] = NLA_POLICY_NESTED(drbd2_set_role_parms_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_SUSPEND_IO - do */
+static const struct nla_policy drbd2_resource_suspend_io_nl_policy[DRBD2_A_SUSPEND_IO_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_SUSPEND_IO_PARMS] = NLA_POLICY_NESTED(drbd2_suspend_io_parms_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_RESUME_IO - do */
+static const struct nla_policy drbd2_resource_resume_io_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_NEW - do */
+static const struct nla_policy drbd2_device_new_nl_policy[DRBD2_A_DEVICE_CONF + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_DEVICE_CONF] = NLA_POLICY_NESTED(drbd2_device_conf_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_DEL - do */
+static const struct nla_policy drbd2_device_del_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_ATTACH - do */
+static const struct nla_policy drbd2_device_attach_nl_policy[DRBD2_A_DISK_CONF + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_DISK_CONF] = NLA_POLICY_NESTED(drbd2_disk_conf_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_DETACH - do */
+static const struct nla_policy drbd2_device_detach_nl_policy[DRBD2_A_DETACH_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_DETACH_PARMS] = NLA_POLICY_NESTED(drbd2_detach_parms_nl_policy),
+};
+
+/* DRBD2_CMD_DISK_SET - do */
+static const struct nla_policy drbd2_disk_set_nl_policy[DRBD2_A_DISK_CONF + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_DISK_CONF] = NLA_POLICY_NESTED(drbd2_disk_conf_nl_policy),
+ [DRBD2_A_SET_DEFAULTS] = { .type = NLA_FLAG, },
+};
+
+/* DRBD2_CMD_DEVICE_RESIZE - do */
+static const struct nla_policy drbd2_device_resize_nl_policy[DRBD2_A_RESIZE_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_RESIZE_PARMS] = NLA_POLICY_NESTED(drbd2_resize_parms_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_OUTDATE - do */
+static const struct nla_policy drbd2_device_outdate_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_INVALIDATE - do */
+static const struct nla_policy drbd2_device_invalidate_nl_policy[DRBD2_A_INVALIDATE_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_INVALIDATE_PARMS] = NLA_POLICY_NESTED(drbd2_invalidate_parms_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_NEW_CURRENT_UUID - do */
+static const struct nla_policy drbd2_device_new_current_uuid_nl_policy[DRBD2_A_NEW_CURRENT_UUID_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_NEW_CURRENT_UUID_PARMS] = NLA_POLICY_NESTED(drbd2_new_current_uuid_parms_nl_policy),
+};
+
+/* DRBD2_CMD_CONNECTION_NEW - do */
+static const struct nla_policy drbd2_connection_new_nl_policy[DRBD2_A_NET_CONF + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_NET_CONF] = NLA_POLICY_NESTED(drbd2_net_conf_nl_policy),
+};
+
+/* DRBD2_CMD_CONNECTION_DEL - do */
+static const struct nla_policy drbd2_connection_del_nl_policy[DRBD2_A_DISCONNECT_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_DISCONNECT_PARMS] = NLA_POLICY_NESTED(drbd2_disconnect_parms_nl_policy),
+};
+
+/* DRBD2_CMD_CONNECTION_CONNECT - do */
+static const struct nla_policy drbd2_connection_connect_nl_policy[DRBD2_A_CONNECT_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_CONNECT_PARMS] = NLA_POLICY_NESTED(drbd2_connect_parms_nl_policy),
+};
+
+/* DRBD2_CMD_CONNECTION_DISCONNECT - do */
+static const struct nla_policy drbd2_connection_disconnect_nl_policy[DRBD2_A_DISCONNECT_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_DISCONNECT_PARMS] = NLA_POLICY_NESTED(drbd2_disconnect_parms_nl_policy),
+};
+
+/* DRBD2_CMD_CONNECTION_SET - do */
+static const struct nla_policy drbd2_connection_set_nl_policy[DRBD2_A_NET_CONF + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_NET_CONF] = NLA_POLICY_NESTED(drbd2_net_conf_nl_policy),
+ [DRBD2_A_SET_DEFAULTS] = { .type = NLA_FLAG, },
+};
+
+/* DRBD2_CMD_CONNECTION_FORGET - do */
+static const struct nla_policy drbd2_connection_forget_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_PATH_NEW - do */
+static const struct nla_policy drbd2_path_new_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_PATH_DEL - do */
+static const struct nla_policy drbd2_path_del_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_PEER_DEVICE_SET - do */
+static const struct nla_policy drbd2_peer_device_set_nl_policy[DRBD2_A_PEER_DEVICE_CONF + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_PEER_DEVICE_CONF] = NLA_POLICY_NESTED(drbd2_peer_device_conf_nl_policy),
+ [DRBD2_A_SET_DEFAULTS] = { .type = NLA_FLAG, },
+};
+
+/* DRBD2_CMD_PEER_DEVICE_INVALIDATE - do */
+static const struct nla_policy drbd2_peer_device_invalidate_nl_policy[DRBD2_A_INVALIDATE_PEER_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_INVALIDATE_PEER_PARMS] = NLA_POLICY_NESTED(drbd2_invalidate_peer_parms_nl_policy),
+};
+
+/* DRBD2_CMD_PEER_DEVICE_PAUSE_SYNC - do */
+static const struct nla_policy drbd2_peer_device_pause_sync_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_PEER_DEVICE_RESUME_SYNC - do */
+static const struct nla_policy drbd2_peer_device_resume_sync_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_PEER_DEVICE_START_OV - do */
+static const struct nla_policy drbd2_peer_device_start_ov_nl_policy[DRBD2_A_START_OV_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_START_OV_PARMS] = NLA_POLICY_NESTED(drbd2_start_ov_parms_nl_policy),
+};
+
+/* DRBD2_CMD_TIMEOUT_TYPE_GET - do */
+static const struct nla_policy drbd2_timeout_type_get_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_RESOURCE_GET - dump */
+static const struct nla_policy drbd2_resource_get_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_GET - dump */
+static const struct nla_policy drbd2_device_get_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_CONNECTION_GET - dump */
+static const struct nla_policy drbd2_connection_get_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_PEER_DEVICE_GET - dump */
+static const struct nla_policy drbd2_peer_device_get_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_PATH_GET - dump */
+static const struct nla_policy drbd2_path_get_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_CBT_FLUSH - do */
+static const struct nla_policy drbd2_device_cbt_flush_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_CBT_START - do */
+static const struct nla_policy drbd2_device_cbt_start_nl_policy[DRBD2_A_CBT_START_PARMS + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+ [DRBD2_A_CBT_START_PARMS] = NLA_POLICY_NESTED(drbd2_cbt_start_parms_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_CBT_STOP - do */
+static const struct nla_policy drbd2_device_cbt_stop_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* DRBD2_CMD_DEVICE_CBT_STATUS - do */
+static const struct nla_policy drbd2_device_cbt_status_nl_policy[DRBD2_A_CONTEXT + 1] = {
+ [DRBD2_A_CONTEXT] = NLA_POLICY_NESTED(drbd2_context_nl_policy),
+};
+
+/* Ops table for drbd2 */
+static const struct genl_split_ops drbd2_nl_ops[] = {
+ {
+ .cmd = DRBD2_CMD_RESOURCE_NEW,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_new_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_new_nl_policy,
+ .maxattr = DRBD2_A_RESOURCE_OPTS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_DEL,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_del_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_del_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_SET,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_set_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_set_nl_policy,
+ .maxattr = DRBD2_A_RESOURCE_OPTS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_RENAME,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_rename_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_rename_nl_policy,
+ .maxattr = DRBD2_A_RENAME_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_DOWN,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_down_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_down_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_PRIMARY,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_primary_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_primary_nl_policy,
+ .maxattr = DRBD2_A_SET_ROLE_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_SECONDARY,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_secondary_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_secondary_nl_policy,
+ .maxattr = DRBD2_A_SET_ROLE_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_SUSPEND_IO,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_suspend_io_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_suspend_io_nl_policy,
+ .maxattr = DRBD2_A_SUSPEND_IO_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_RESUME_IO,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_resource_resume_io_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_resource_resume_io_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_NEW,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_new_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_new_nl_policy,
+ .maxattr = DRBD2_A_DEVICE_CONF,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_DEL,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_del_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_del_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_ATTACH,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_attach_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_attach_nl_policy,
+ .maxattr = DRBD2_A_DISK_CONF,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_DETACH,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_detach_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_detach_nl_policy,
+ .maxattr = DRBD2_A_DETACH_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DISK_SET,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_disk_set_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_disk_set_nl_policy,
+ .maxattr = DRBD2_A_DISK_CONF,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_RESIZE,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_resize_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_resize_nl_policy,
+ .maxattr = DRBD2_A_RESIZE_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_OUTDATE,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_outdate_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_outdate_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_INVALIDATE,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_invalidate_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_invalidate_nl_policy,
+ .maxattr = DRBD2_A_INVALIDATE_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_NEW_CURRENT_UUID,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_new_current_uuid_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_new_current_uuid_nl_policy,
+ .maxattr = DRBD2_A_NEW_CURRENT_UUID_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_CONNECTION_NEW,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_connection_new_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_connection_new_nl_policy,
+ .maxattr = DRBD2_A_NET_CONF,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_CONNECTION_DEL,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_connection_del_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_connection_del_nl_policy,
+ .maxattr = DRBD2_A_DISCONNECT_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_CONNECTION_CONNECT,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_connection_connect_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_connection_connect_nl_policy,
+ .maxattr = DRBD2_A_CONNECT_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_CONNECTION_DISCONNECT,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_connection_disconnect_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_connection_disconnect_nl_policy,
+ .maxattr = DRBD2_A_DISCONNECT_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_CONNECTION_SET,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_connection_set_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_connection_set_nl_policy,
+ .maxattr = DRBD2_A_NET_CONF,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_CONNECTION_FORGET,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_connection_forget_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_connection_forget_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_PATH_NEW,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_path_new_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_path_new_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_PATH_DEL,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_path_del_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_path_del_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_PEER_DEVICE_SET,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_peer_device_set_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_peer_device_set_nl_policy,
+ .maxattr = DRBD2_A_PEER_DEVICE_CONF,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_PEER_DEVICE_INVALIDATE,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_peer_device_invalidate_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_peer_device_invalidate_nl_policy,
+ .maxattr = DRBD2_A_INVALIDATE_PEER_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_PEER_DEVICE_PAUSE_SYNC,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_peer_device_pause_sync_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_peer_device_pause_sync_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_PEER_DEVICE_RESUME_SYNC,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_peer_device_resume_sync_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_peer_device_resume_sync_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_PEER_DEVICE_START_OV,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_peer_device_start_ov_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_peer_device_start_ov_nl_policy,
+ .maxattr = DRBD2_A_START_OV_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_TIMEOUT_TYPE_GET,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_timeout_type_get_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_timeout_type_get_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_RESOURCE_GET,
+ .dumpit = drbd2_nl_resource_get_dumpit,
+ .policy = drbd2_resource_get_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_CMD_CAP_DUMP,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_GET,
+ .dumpit = drbd2_nl_device_get_dumpit,
+ .done = drbd2_nl_device_get_done,
+ .policy = drbd2_device_get_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_CMD_CAP_DUMP,
+ },
+ {
+ .cmd = DRBD2_CMD_CONNECTION_GET,
+ .dumpit = drbd2_nl_connection_get_dumpit,
+ .done = drbd2_nl_connection_get_done,
+ .policy = drbd2_connection_get_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_CMD_CAP_DUMP,
+ },
+ {
+ .cmd = DRBD2_CMD_PEER_DEVICE_GET,
+ .dumpit = drbd2_nl_peer_device_get_dumpit,
+ .done = drbd2_nl_peer_device_get_done,
+ .policy = drbd2_peer_device_get_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_CMD_CAP_DUMP,
+ },
+ {
+ .cmd = DRBD2_CMD_PATH_GET,
+ .dumpit = drbd2_nl_path_get_dumpit,
+ .done = drbd2_nl_path_get_done,
+ .policy = drbd2_path_get_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_CMD_CAP_DUMP,
+ },
+ {
+ .cmd = DRBD2_CMD_STATE_GET,
+ .dumpit = drbd2_nl_state_get_dumpit,
+ .done = drbd2_nl_state_get_done,
+ .flags = GENL_CMD_CAP_DUMP,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_CBT_FLUSH,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_cbt_flush_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_cbt_flush_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_CBT_START,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_cbt_start_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_cbt_start_nl_policy,
+ .maxattr = DRBD2_A_CBT_START_PARMS,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_CBT_STOP,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_cbt_stop_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_cbt_stop_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = DRBD2_CMD_DEVICE_CBT_STATUS,
+ .pre_doit = drbd2_pre_doit,
+ .doit = drbd2_nl_device_cbt_status_doit,
+ .post_doit = drbd2_post_doit,
+ .policy = drbd2_device_cbt_status_nl_policy,
+ .maxattr = DRBD2_A_CONTEXT,
+ .flags = GENL_CMD_CAP_DO,
+ },
+};
+
+static const struct genl_multicast_group drbd2_nl_mcgrps[] = {
+ [DRBD2_NLGRP_EVENTS] = { "events", },
+};
+
+struct genl_family drbd2_nl_family __ro_after_init = {
+ .name = DRBD2_FAMILY_NAME,
+ .version = DRBD2_FAMILY_VERSION,
+ .netnsok = true,
+ .parallel_ops = true,
+ .module = THIS_MODULE,
+ .split_ops = drbd2_nl_ops,
+ .n_split_ops = ARRAY_SIZE(drbd2_nl_ops),
+ .mcgrps = drbd2_nl_mcgrps,
+ .n_mcgrps = ARRAY_SIZE(drbd2_nl_mcgrps),
+};
diff --git a/drivers/block/drbd/drbd2_nl_gen.h b/drivers/block/drbd/drbd2_nl_gen.h
new file mode 100644
index 000000000000..705dee69cdce
--- /dev/null
+++ b/drivers/block/drbd/drbd2_nl_gen.h
@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/* Do not edit directly, auto-generated from: */
+/* Documentation/netlink/specs/drbd2.yaml */
+/* YNL-GEN kernel header */
+/* To regenerate run: tools/net/ynl/ynl-regen.sh */
+
+#ifndef _LINUX_DRBD2_GEN_H
+#define _LINUX_DRBD2_GEN_H
+
+#include <net/netlink.h>
+#include <net/genetlink.h>
+
+#include <uapi/linux/drbd2.h>
+
+/* Common nested types */
+extern const struct nla_policy drbd2_address_nl_policy[DRBD2_A_ADDRESS_IPV6 + 1];
+extern const struct nla_policy drbd2_cbt_start_parms_nl_policy[DRBD2_A_CBT_START_PARMS_BOOKMARK_ID + 1];
+extern const struct nla_policy drbd2_connect_parms_nl_policy[DRBD2_A_CONNECT_PARMS_DISCARD_MY_DATA + 1];
+extern const struct nla_policy drbd2_context_nl_policy[DRBD2_A_CONTEXT_PEER_ADDRESS + 1];
+extern const struct nla_policy drbd2_detach_parms_nl_policy[DRBD2_A_DETACH_PARMS_INTENTIONAL_DISKLESS_DETACH + 1];
+extern const struct nla_policy drbd2_device_conf_nl_policy[DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY + 1];
+extern const struct nla_policy drbd2_disconnect_parms_nl_policy[DRBD2_A_DISCONNECT_PARMS_FORCE + 1];
+extern const struct nla_policy drbd2_disk_conf_nl_policy[DRBD2_A_DISK_CONF_BITMAP + 1];
+extern const struct nla_policy drbd2_invalidate_parms_nl_policy[DRBD2_A_INVALIDATE_PARMS_RESET_BITMAP + 1];
+extern const struct nla_policy drbd2_invalidate_peer_parms_nl_policy[DRBD2_A_INVALIDATE_PEER_PARMS_RESET_BITMAP + 1];
+extern const struct nla_policy drbd2_net_conf_nl_policy[DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE + 1];
+extern const struct nla_policy drbd2_new_current_uuid_parms_nl_policy[DRBD2_A_NEW_CURRENT_UUID_PARMS_FORCE_RESYNC + 1];
+extern const struct nla_policy drbd2_peer_device_conf_nl_policy[DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER + 1];
+extern const struct nla_policy drbd2_rename_parms_nl_policy[DRBD2_A_RENAME_PARMS_NEW_NAME + 1];
+extern const struct nla_policy drbd2_resize_parms_nl_policy[DRBD2_A_RESIZE_PARMS_AL_STRIPE_SIZE + 1];
+extern const struct nla_policy drbd2_resource_opts_nl_policy[DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT + 1];
+extern const struct nla_policy drbd2_set_role_parms_nl_policy[DRBD2_A_SET_ROLE_PARMS_FORCE + 1];
+extern const struct nla_policy drbd2_start_ov_parms_nl_policy[DRBD2_A_START_OV_PARMS_STOP_SECTOR + 1];
+extern const struct nla_policy drbd2_suspend_io_parms_nl_policy[DRBD2_A_SUSPEND_IO_PARMS_FLUSH_TO_PEERS + 1];
+
+int drbd2_pre_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info);
+void
+drbd2_post_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_device_get_done(struct netlink_callback *cb);
+int drbd2_nl_connection_get_done(struct netlink_callback *cb);
+int drbd2_nl_peer_device_get_done(struct netlink_callback *cb);
+int drbd2_nl_path_get_done(struct netlink_callback *cb);
+int drbd2_nl_state_get_done(struct netlink_callback *cb);
+
+int drbd2_nl_resource_new_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_resource_del_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_resource_set_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_resource_rename_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_resource_down_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_resource_primary_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_resource_secondary_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_resource_suspend_io_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_resource_resume_io_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_device_new_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_del_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_attach_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_detach_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_disk_set_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_resize_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_outdate_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_invalidate_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_device_new_current_uuid_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_connection_new_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_connection_del_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_connection_connect_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_connection_disconnect_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_connection_set_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_connection_forget_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_path_new_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_path_del_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_peer_device_set_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_peer_device_invalidate_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_peer_device_pause_sync_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_peer_device_resume_sync_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_peer_device_start_ov_doit(struct sk_buff *skb,
+ struct genl_info *info);
+int drbd2_nl_timeout_type_get_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_resource_get_dumpit(struct sk_buff *skb,
+ struct netlink_callback *cb);
+int drbd2_nl_device_get_dumpit(struct sk_buff *skb,
+ struct netlink_callback *cb);
+int drbd2_nl_connection_get_dumpit(struct sk_buff *skb,
+ struct netlink_callback *cb);
+int drbd2_nl_peer_device_get_dumpit(struct sk_buff *skb,
+ struct netlink_callback *cb);
+int drbd2_nl_path_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
+int drbd2_nl_state_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
+int drbd2_nl_device_cbt_flush_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_cbt_start_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_cbt_stop_doit(struct sk_buff *skb, struct genl_info *info);
+int drbd2_nl_device_cbt_status_doit(struct sk_buff *skb,
+ struct genl_info *info);
+
+enum {
+ DRBD2_NLGRP_EVENTS,
+};
+
+extern struct genl_family drbd2_nl_family;
+
+#endif /* _LINUX_DRBD2_GEN_H */
diff --git a/drivers/block/drbd/drbd_nl_drbd2.c b/drivers/block/drbd/drbd_nl_drbd2.c
new file mode 100644
index 000000000000..3d9ff2516fcd
--- /dev/null
+++ b/drivers/block/drbd/drbd_nl_drbd2.c
@@ -0,0 +1,2589 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026, LINBIT HA-Solutions GmbH.
+ */
+
+/*
+ * The "drbd2" generic netlink family: everything that knows about the
+ * bytes on the wire of that dialect. The command implementations
+ * themselves live in drbd_nl.c and only ever see a struct drbd_adm_ctx.
+ *
+ * Unlike the legacy "drbd" family, drbd2 has no fixed message header and
+ * no place to carry a drbd_ret_code, so the outcome of a request is split
+ * in two, as the specification prescribes:
+ *
+ * - A state change reports its drbd_state_rv in the state-result
+ * attribute of the reply, together with the accumulated info messages
+ * in message. A refused state change is data, not a transport error.
+ * - Everything that failed for a configuration reason (the ERR_* codes)
+ * becomes an errno plus a description in the extended ACK, and no
+ * reply is sent at all.
+ *
+ * The description in the extended ACK is the command's own account of the
+ * failure where it left one -- the first of the info messages the core
+ * collected, formatted into the ACK's buffer so that it outlives the
+ * command context -- and the generic text of the ERR_* code otherwise.
+ * Failures that are about one attribute of the request also name that
+ * attribute, the way the netlink policy checks do. Further info messages
+ * are lost on the errno path; the reply of a completed command carries
+ * them all.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/slab.h>
+#include <linux/drbd.h>
+#include "drbd2_nl_gen.h"
+#include <net/genetlink.h>
+#include <net/sock.h>
+
+#include "drbd_int.h"
+#include "drbd_nl.h"
+
+static const struct drbd_nl_dialect drbd2_dialect;
+
+/* Per-request state of the drbd2 dialect; hangs off drbd_adm_ctx.req. */
+struct drbd2_req {
+ struct genl_info *info;
+ struct sk_buff *reply_skb;
+ void *reply_hdr;
+ /* the command filled the reply itself; do not add state-result */
+ bool payload_reply;
+};
+
+/* One allocation for both, so that pre_doit zeroes the context only once. */
+struct drbd2_ctx {
+ struct drbd_adm_ctx ctx;
+ struct drbd2_req req;
+};
+
+static inline struct drbd2_req *drbd2_req(struct drbd_adm_ctx *ctx)
+{
+ return ctx->req;
+}
+
+/*
+ * drbd2 numbers its attributes from 1, so attribute 0 is free to serve as
+ * the padding attribute of the 64 bit values, the same way the legacy
+ * marshalling code uses it.
+ */
+#define DRBD2_A_PAD 0
+
+static int drbd2_put_u64(struct sk_buff *skb, int attrtype, u64 value)
+{
+ return nla_put_u64_64bit(skb, attrtype, value, DRBD2_A_PAD);
+}
+
+/*
+ * The internal string buffers carry an explicit length and are
+ * NUL-terminated unless they are completely full; emit them including
+ * that terminating NUL, as the drbd2 policies expect.
+ */
+static int drbd2_put_str(struct sk_buff *skb, int attrtype,
+ const char *val, u32 len, u32 size)
+{
+ return nla_put(skb, attrtype, min_t(u32, size, len + (len < size)), val);
+}
+
+/* A socket address as the address nest of "attrtype". */
+static int drbd2_put_address(struct sk_buff *skb, int attrtype,
+ const struct sockaddr_storage *addr)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u16(skb, DRBD2_A_ADDRESS_FAMILY, addr->ss_family))
+ goto fail;
+ switch (addr->ss_family) {
+ case AF_INET: {
+ const struct sockaddr_in *sin = (const struct sockaddr_in *)addr;
+
+ if (nla_put_be16(skb, DRBD2_A_ADDRESS_PORT, sin->sin_port) ||
+ nla_put_be32(skb, DRBD2_A_ADDRESS_IPV4, sin->sin_addr.s_addr))
+ goto fail;
+ break;
+ }
+ case AF_INET6: {
+ const struct sockaddr_in6 *sin6 = (const struct sockaddr_in6 *)addr;
+
+ if (nla_put_be16(skb, DRBD2_A_ADDRESS_PORT, sin6->sin6_port) ||
+ nla_put(skb, DRBD2_A_ADDRESS_IPV6,
+ sizeof(sin6->sin6_addr), &sin6->sin6_addr))
+ goto fail;
+ break;
+ }
+ default:
+ /* the family alone is all this dialect can say about it */
+ break;
+ }
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * The reverse: one address nest into the sockaddr_storage the transports
+ * work with. Returns -ENOMSG if the nest does not describe an address of
+ * a family this dialect knows.
+ */
+static int drbd2_get_address(const struct nlattr *nla, char *dst, __u32 *dst_len)
+{
+ struct nlattr *tb[DRBD2_A_ADDRESS_IPV6 + 1];
+ int err;
+
+ err = nla_parse_nested(tb, DRBD2_A_ADDRESS_IPV6, nla,
+ drbd2_address_nl_policy, NULL);
+ if (err)
+ return err;
+ if (!tb[DRBD2_A_ADDRESS_FAMILY] || !tb[DRBD2_A_ADDRESS_PORT])
+ return -ENOMSG;
+
+ switch (nla_get_u16(tb[DRBD2_A_ADDRESS_FAMILY])) {
+ case AF_INET: {
+ struct sockaddr_in sin = { .sin_family = AF_INET };
+
+ if (!tb[DRBD2_A_ADDRESS_IPV4])
+ return -ENOMSG;
+ sin.sin_port = nla_get_be16(tb[DRBD2_A_ADDRESS_PORT]);
+ sin.sin_addr.s_addr = nla_get_be32(tb[DRBD2_A_ADDRESS_IPV4]);
+ memcpy(dst, &sin, sizeof(sin));
+ *dst_len = sizeof(sin);
+ return 0;
+ }
+ case AF_INET6: {
+ struct sockaddr_in6 sin6 = { .sin6_family = AF_INET6 };
+
+ if (!tb[DRBD2_A_ADDRESS_IPV6])
+ return -ENOMSG;
+ sin6.sin6_port = nla_get_be16(tb[DRBD2_A_ADDRESS_PORT]);
+ memcpy(&sin6.sin6_addr, nla_data(tb[DRBD2_A_ADDRESS_IPV6]),
+ sizeof(sin6.sin6_addr));
+ memcpy(dst, &sin6, sizeof(sin6));
+ *dst_len = sizeof(sin6);
+ return 0;
+ }
+ }
+ return -ENOMSG;
+}
+
+static int drbd2_genl_multicast_events(struct sk_buff *skb)
+{
+ return genlmsg_multicast_allns(&drbd2_nl_family, skb, 0,
+ DRBD2_NLGRP_EVENTS);
+}
+
+/*
+ * The ERR_* codes of the command core translated into what a modern
+ * netlink family reports: an errno and a description for the extended
+ * ACK. The texts are the ones drbdsetup prints for the legacy family.
+ */
+static const struct {
+ int err;
+ const char *msg;
+} drbd2_error_map[AFTER_LAST_ERR_CODE] = {
+ [ERR_LOCAL_ADDR] = { -EADDRINUSE, "local address(port) already in use" },
+ [ERR_PEER_ADDR] = { -EADDRINUSE, "remote address(port) already in use" },
+ [ERR_OPEN_DISK] = { -ENOENT, "can not open backing device" },
+ [ERR_OPEN_MD_DISK] = { -ENOENT, "can not open meta device" },
+ [ERR_DISK_NOT_BDEV] = { -ENOTBLK, "lower device is not a block device" },
+ [ERR_MD_NOT_BDEV] = { -ENOTBLK, "meta device is not a block device" },
+ [ERR_DISK_TOO_SMALL] = { -ENOSPC, "lower device smaller than the requested size" },
+ [ERR_MD_DISK_TOO_SMALL] = { -ENOSPC, "meta device too small" },
+ [ERR_BDCLAIM_DISK] = { -EBUSY, "lower device is already claimed" },
+ [ERR_BDCLAIM_MD_DISK] = { -EBUSY, "meta device is already claimed" },
+ [ERR_MD_IDX_INVALID] = { -EINVAL,
+ "lower device / meta device / index combination invalid" },
+ [ERR_IO_MD_DISK] = { -EIO, "I/O error during initial access to meta-data" },
+ [ERR_MD_INVALID] = { -EINVAL, "no valid meta-data signature found" },
+ [ERR_AUTH_ALG] = { -EINVAL, "cram-hmac-alg not known to the kernel" },
+ [ERR_AUTH_ALG_ND] = { -EINVAL, "cram-hmac-alg is not a digest" },
+ [ERR_NOMEM] = { -ENOMEM, "out of memory" },
+ [ERR_DISCARD_IMPOSSIBLE] = { -EINVAL, "discard-my-data not allowed when primary" },
+ [ERR_DISK_CONFIGURED] = { -EEXIST, "device is attached to a disk" },
+ [ERR_NET_CONFIGURED] = { -EEXIST, "connection has a net-config" },
+ [ERR_MANDATORY_TAG] = { -EINVAL, "invalid or missing attribute" },
+ [ERR_MINOR_INVALID] = { -ENOENT, "device minor not allocated" },
+ [ERR_INTR] = { -EINTR, "interrupted by signal" },
+ [ERR_RESIZE_RESYNC] = { -EBUSY, "resize not allowed during resync" },
+ [ERR_NO_PRIMARY] = { -EINVAL, "need one primary node to resize" },
+ [ERR_RESYNC_AFTER] = { -EINVAL, "the resync-after minor number is invalid" },
+ [ERR_RESYNC_AFTER_CYCLE] = { -EINVAL, "this would cause a resync-after dependency cycle" },
+ [ERR_PAUSE_IS_SET] = { -EALREADY, "sync-pause flag is already set" },
+ [ERR_PAUSE_IS_CLEAR] = { -EALREADY, "sync-pause flag is already cleared" },
+ [ERR_PACKET_NR] = { -EPROTO, "kernel does not know how to handle this request" },
+ [ERR_NO_DISK] = { -ENODEV, "device does not have a disk-config" },
+ [ERR_NOT_PROTO_C] = { -EINVAL, "protocol C required" },
+ [ERR_NOMEM_BITMAP] = { -ENOMEM, "out of memory for the bitmap" },
+ [ERR_INTEGRITY_ALG] = { -EINVAL, "data-integrity-alg not known to the kernel" },
+ [ERR_INTEGRITY_ALG_ND] = { -EINVAL, "data-integrity-alg is not a digest" },
+ [ERR_CPU_MASK_PARSE] = { -EINVAL, "invalid cpu-mask" },
+ [ERR_CSUMS_ALG] = { -EINVAL, "csums-alg not known to the kernel" },
+ [ERR_CSUMS_ALG_ND] = { -EINVAL, "csums-alg is not a digest" },
+ [ERR_VERIFY_ALG] = { -EINVAL, "verify-alg not known to the kernel" },
+ [ERR_VERIFY_ALG_ND] = { -EINVAL, "verify-alg is not a digest" },
+ [ERR_CSUMS_RESYNC_RUNNING] = { -EBUSY,
+ "can not change csums-alg while resync is in progress" },
+ [ERR_VERIFY_RUNNING] = { -EBUSY, "can not change verify-alg while online verify runs" },
+ [ERR_DATA_NOT_CURRENT] = { -EINVAL, "can only attach to the data we lost last" },
+ [ERR_CONNECTED] = { -EBUSY, "need to be standalone" },
+ [ERR_PERM] = { -EPERM, "permission denied" },
+ [ERR_NEED_APV_93] = { -EOPNOTSUPP, "protocol version 93 required for assume-clean" },
+ [ERR_STONITH_AND_PROT_A] = { -EINVAL,
+ "fencing resource-and-stonith needs protocol B or C" },
+ [ERR_CONG_NOT_PROTO_A] = { -EINVAL, "on-congestion pull-ahead needs protocol A" },
+ [ERR_PIC_AFTER_DEP] = { -EALREADY, "sync-pause is cleared, but a dependency is paused" },
+ [ERR_PIC_PEER_DEP] = { -EALREADY, "sync-pause is cleared, but the peer paused" },
+ [ERR_RES_NOT_KNOWN] = { -ENOENT, "unknown resource" },
+ [ERR_RES_IN_USE] = { -EBUSY, "resource still in use" },
+ [ERR_MINOR_CONFIGURED] = { -EBUSY, "minor still configured" },
+ [ERR_MINOR_OR_VOLUME_EXISTS] = { -EEXIST, "minor or volume exists already" },
+ [ERR_INVALID_REQUEST] = { -EINVAL, "invalid configuration request" },
+ [ERR_NEED_APV_100] = { -EOPNOTSUPP, "protocol version 100 required" },
+ [ERR_NEED_ALLOW_TWO_PRI] = { -EINVAL, "can not clear allow-two-primaries now" },
+ [ERR_MD_UNCLEAN] = { -EINVAL, "unclean meta-data found" },
+ [ERR_MD_LAYOUT_CONNECTED] = { -EBUSY, "online meta-data layout change needs a connection" },
+ [ERR_MD_LAYOUT_TOO_BIG] = { -ENOSPC, "resulting activity log area too big" },
+ [ERR_MD_LAYOUT_TOO_SMALL] = { -ENOSPC, "resulting activity log area too small" },
+ [ERR_MD_LAYOUT_NO_FIT] = { -ENOSPC, "resulting activity log does not fit the meta-data" },
+ [ERR_IMPLICIT_SHRINK] = { -EINVAL, "implicit device shrinking not allowed" },
+ [ERR_INVALID_PEER_NODE_ID] = { -EINVAL, "invalid peer-node-id" },
+ [ERR_CREATE_TRANSPORT] = { -ENOENT, "failed to create transport" },
+ [ERR_LOCAL_AND_PEER_ADDR] = { -EADDRINUSE,
+ "combination of local and remote address already in use" },
+ [ERR_ALREADY_EXISTS] = { -EEXIST, "already exists" },
+ [ERR_APV_TOO_LOW] = { -EOPNOTSUPP, "a higher DRBD protocol level is required" },
+ [ERR_PATH_COLLISION] = { -EEXIST, "path collides with an existing path" },
+};
+
+static bool drbd2_is_ret_code(int result)
+{
+ return result > NO_ERROR && result < AFTER_LAST_ERR_CODE;
+}
+
+/* The errno an ERR_* code becomes on the wire. */
+static int drbd2_errno(int result)
+{
+ if (result == NO_ERROR)
+ return 0;
+ if (!drbd2_is_ret_code(result))
+ return result < 0 ? result : -EINVAL;
+ return drbd2_error_map[result].err ? : -EINVAL;
+}
+
+/*
+ * The request attribute a failure is about, for the codes that are
+ * specific to one: the top level nest and the attribute within it.
+ */
+static const struct {
+ u16 nest;
+ u16 attr;
+} drbd2_error_attr[AFTER_LAST_ERR_CODE] = {
+ [ERR_OPEN_DISK] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_BACKING_DEV },
+ [ERR_DISK_NOT_BDEV] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_BACKING_DEV },
+ [ERR_BDCLAIM_DISK] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_BACKING_DEV },
+ [ERR_DISK_TOO_SMALL] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_BACKING_DEV },
+ [ERR_OPEN_MD_DISK] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_META_DEV },
+ [ERR_MD_NOT_BDEV] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_META_DEV },
+ [ERR_BDCLAIM_MD_DISK] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_META_DEV },
+ [ERR_MD_DISK_TOO_SMALL] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_META_DEV },
+ [ERR_MD_IDX_INVALID] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_META_DEV_IDX },
+ [ERR_RESYNC_AFTER] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_RESYNC_AFTER },
+ [ERR_RESYNC_AFTER_CYCLE] = { DRBD2_A_DISK_CONF, DRBD2_A_DISK_CONF_RESYNC_AFTER },
+ [ERR_CPU_MASK_PARSE] = { DRBD2_A_RESOURCE_OPTS, DRBD2_A_RESOURCE_OPTS_CPU_MASK },
+ [ERR_AUTH_ALG] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_CRAM_HMAC_ALG },
+ [ERR_AUTH_ALG_ND] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_CRAM_HMAC_ALG },
+ [ERR_INTEGRITY_ALG] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_INTEGRITY_ALG },
+ [ERR_INTEGRITY_ALG_ND] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_INTEGRITY_ALG },
+ [ERR_VERIFY_ALG] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_VERIFY_ALG },
+ [ERR_VERIFY_ALG_ND] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_VERIFY_ALG },
+ [ERR_VERIFY_RUNNING] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_VERIFY_ALG },
+ [ERR_CSUMS_ALG] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_CSUMS_ALG },
+ [ERR_CSUMS_ALG_ND] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_CSUMS_ALG },
+ [ERR_CSUMS_RESYNC_RUNNING] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_CSUMS_ALG },
+ [ERR_STONITH_AND_PROT_A] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_FENCING },
+ [ERR_CONG_NOT_PROTO_A] = { DRBD2_A_NET_CONF, DRBD2_A_NET_CONF_ON_CONGESTION },
+ [ERR_INVALID_PEER_NODE_ID] = { DRBD2_A_CONTEXT, DRBD2_A_CONTEXT_PEER_NODE_ID },
+};
+
+/*
+ * Describe a failed command in the extended ACK: its own first info
+ * message if it left one, the generic text of its ERR_* code otherwise,
+ * and the request attribute the failure is about if there is one.
+ */
+static void drbd2_set_extack(struct genl_info *info, struct drbd_adm_ctx *ctx)
+{
+ struct netlink_ext_ack *extack = info->extack;
+ int result = ctx->result;
+ const char *msg = NULL;
+ int len = 0;
+
+ if (!extack)
+ return;
+
+ /* what the command itself said, its first message only */
+ if (ctx->msg_len) {
+ len = strnlen(ctx->msg, ctx->msg_len);
+ while (len && (ctx->msg[len - 1] == '\n' || ctx->msg[len - 1] == ' '))
+ len--;
+ if (len)
+ msg = ctx->msg;
+ }
+ if (!msg) {
+ msg = drbd2_is_ret_code(result) && drbd2_error_map[result].msg ?
+ drbd2_error_map[result].msg : "DRBD command failed";
+ len = strlen(msg);
+ }
+ NL_SET_ERR_MSG_FMT(extack, "%.*s",
+ min(len, NETLINK_MAX_FMTMSG_LEN - 1), msg);
+
+ if (drbd2_is_ret_code(result) && drbd2_error_attr[result].nest) {
+ struct nlattr *nest = info->attrs[drbd2_error_attr[result].nest];
+ struct nlattr *attr = nest ?
+ nla_find_nested(nest, drbd2_error_attr[result].attr) : NULL;
+
+ if (attr)
+ NL_SET_BAD_ATTR(extack, attr);
+ }
+}
+
+/*
+ * The drbd_state_rv of a completed state change as the state-result
+ * attribute: the error values are the negated SS_* codes, the successful
+ * outcomes follow above them in the order of the enum.
+ */
+static u32 drbd2_state_result(int result)
+{
+ if (result == NO_ERROR)
+ return DRBD2_STATE_RESULT_SUCCESS;
+ if (result >= SS_SUCCESS && result <= SS_CW_NO_NEED)
+ return DRBD2_STATE_RESULT_SUCCESS + (result - SS_SUCCESS);
+ if (result <= SS_UNKNOWN_ERROR && result > SS_AFTER_LAST_ERROR)
+ return -result;
+ return DRBD2_STATE_RESULT_UNKNOWN_ERROR;
+}
+
+/*
+ * The info messages of the command, joined into the one string the
+ * message attribute carries. They are NUL-separated in ctx->msg and this
+ * is their last user, so the separators can be turned into newlines in
+ * place; the final NUL stays and terminates the string.
+ */
+static int drbd2_put_message(struct sk_buff *skb, struct drbd_adm_ctx *ctx)
+{
+ unsigned int i;
+
+ if (!ctx->msg_len)
+ return 0;
+ for (i = 0; i + 1 < ctx->msg_len; i++)
+ if (ctx->msg[i] == '\0')
+ ctx->msg[i] = '\n';
+ return nla_put(skb, DRBD2_A_MESSAGE, ctx->msg_len, ctx->msg);
+}
+
+/*
+ * Serialize the outcome of a completed command into the reply. The state
+ * result always fits into a fresh message; the info text may not, and is
+ * then dropped, exactly as the over-full legacy reply dropped it.
+ */
+static void drbd2_put_outcome(struct drbd_adm_ctx *ctx)
+{
+ struct drbd2_req *req = drbd2_req(ctx);
+
+ if (req->payload_reply)
+ return;
+ if (nla_put_u32(req->reply_skb, DRBD2_A_STATE_RESULT,
+ drbd2_state_result(ctx->result)))
+ return;
+ drbd2_put_message(req->reply_skb, ctx);
+}
+
+/*
+ * The identity of an object, as the context nest of "attrtype". Every
+ * pointer is optional; only the parts that are given are described.
+ */
+static int drbd2_put_context(struct sk_buff *skb, int attrtype,
+ struct drbd_resource *resource,
+ struct drbd_connection *connection,
+ struct drbd_device *device,
+ struct drbd_path *path)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (resource && nla_put_string(skb, DRBD2_A_CONTEXT_RESOURCE_NAME, resource->name))
+ goto fail;
+ if (device &&
+ (nla_put_u32(skb, DRBD2_A_CONTEXT_VOLUME, device->vnr) ||
+ nla_put_u32(skb, DRBD2_A_CONTEXT_MINOR, device->minor)))
+ goto fail;
+ if (connection) {
+ struct drbd_net_conf *nc;
+ int err = 0;
+
+ if (nla_put_u32(skb, DRBD2_A_CONTEXT_PEER_NODE_ID, connection->peer_node_id))
+ goto fail;
+ rcu_read_lock();
+ nc = rcu_dereference(connection->transport.net_conf);
+ if (nc)
+ err = nla_put_string(skb, DRBD2_A_CONTEXT_CONNECTION_NAME, nc->name);
+ rcu_read_unlock();
+ if (err)
+ goto fail;
+ }
+ if (path &&
+ (drbd2_put_address(skb, DRBD2_A_CONTEXT_MY_ADDRESS, &path->my_addr) ||
+ drbd2_put_address(skb, DRBD2_A_CONTEXT_PEER_ADDRESS, &path->peer_addr)))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * State and statistics of the five object types. The values of the drbd2
+ * enums are those of the kernel-internal ones, so they go on the wire
+ * unchanged.
+ */
+
+static int drbd2_put_resource_info(struct sk_buff *skb, int attrtype,
+ struct drbd_resource_info *info)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u32(skb, DRBD2_A_RESOURCE_INFO_ROLE, info->res_role) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_INFO_SUSP, info->res_susp) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_INFO_SUSP_NOD, info->res_susp_nod) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_INFO_SUSP_FEN, info->res_susp_fen) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_INFO_SUSP_QUORUM, info->res_susp_quorum) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_INFO_FAIL_IO, info->res_fail_io))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_resource_statistics(struct sk_buff *skb, int attrtype,
+ struct drbd_resource_statistics *s)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u32(skb, DRBD2_A_RESOURCE_STATISTICS_WRITE_ORDERING,
+ s->res_stat_write_ordering))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_device_info(struct sk_buff *skb, int attrtype,
+ struct drbd_device_info *info)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u32(skb, DRBD2_A_DEVICE_INFO_DISK_STATE, info->dev_disk_state) ||
+ nla_put_u8(skb, DRBD2_A_DEVICE_INFO_IS_INTENTIONAL_DISKLESS,
+ info->is_intentional_diskless) ||
+ nla_put_u8(skb, DRBD2_A_DEVICE_INFO_HAS_QUORUM, info->dev_has_quorum) ||
+ nla_put_u8(skb, DRBD2_A_DEVICE_INFO_IS_OPEN, info->dev_is_open) ||
+ drbd2_put_str(skb, DRBD2_A_DEVICE_INFO_BACKING_DEV_PATH,
+ info->backing_dev_path, info->backing_dev_path_len,
+ sizeof(info->backing_dev_path)))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_device_statistics(struct sk_buff *skb, int attrtype,
+ struct drbd_device_statistics *s)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_u64(skb, DRBD2_A_DEVICE_STATISTICS_SIZE, s->dev_size) ||
+ drbd2_put_u64(skb, DRBD2_A_DEVICE_STATISTICS_READ, s->dev_read) ||
+ drbd2_put_u64(skb, DRBD2_A_DEVICE_STATISTICS_WRITE, s->dev_write) ||
+ drbd2_put_u64(skb, DRBD2_A_DEVICE_STATISTICS_AL_WRITES, s->dev_al_writes) ||
+ drbd2_put_u64(skb, DRBD2_A_DEVICE_STATISTICS_BM_WRITES, s->dev_bm_writes) ||
+ nla_put_u32(skb, DRBD2_A_DEVICE_STATISTICS_UPPER_PENDING, s->dev_upper_pending) ||
+ nla_put_u32(skb, DRBD2_A_DEVICE_STATISTICS_LOWER_PENDING, s->dev_lower_pending) ||
+ nla_put_u8(skb, DRBD2_A_DEVICE_STATISTICS_UPPER_BLOCKED, s->dev_upper_blocked) ||
+ nla_put_u8(skb, DRBD2_A_DEVICE_STATISTICS_LOWER_BLOCKED, s->dev_lower_blocked) ||
+ nla_put_u8(skb, DRBD2_A_DEVICE_STATISTICS_AL_SUSPENDED, s->dev_al_suspended) ||
+ drbd2_put_u64(skb, DRBD2_A_DEVICE_STATISTICS_EXPOSED_DATA_UUID,
+ s->dev_exposed_data_uuid) ||
+ drbd2_put_u64(skb, DRBD2_A_DEVICE_STATISTICS_CURRENT_UUID, s->dev_current_uuid) ||
+ nla_put_u32(skb, DRBD2_A_DEVICE_STATISTICS_DISK_FLAGS, s->dev_disk_flags) ||
+ nla_put(skb, DRBD2_A_DEVICE_STATISTICS_HISTORY_UUIDS,
+ min_t(u32, HISTORY_UUIDS_SIZE, s->history_uuids_len), s->history_uuids))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_connection_info(struct sk_buff *skb, int attrtype,
+ struct drbd_connection_info *info)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u32(skb, DRBD2_A_CONNECTION_INFO_CONNECTION_STATE,
+ info->conn_connection_state) ||
+ nla_put_u32(skb, DRBD2_A_CONNECTION_INFO_ROLE, info->conn_role))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_connection_statistics(struct sk_buff *skb, int attrtype,
+ struct drbd_connection_statistics *s)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u8(skb, DRBD2_A_CONNECTION_STATISTICS_CONGESTED, s->conn_congested) ||
+ drbd2_put_u64(skb, DRBD2_A_CONNECTION_STATISTICS_AP_IN_FLIGHT, s->ap_in_flight) ||
+ drbd2_put_u64(skb, DRBD2_A_CONNECTION_STATISTICS_RS_IN_FLIGHT, s->rs_in_flight))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_peer_device_info(struct sk_buff *skb, int attrtype,
+ struct drbd_peer_device_info *info)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u32(skb, DRBD2_A_PEER_DEVICE_INFO_REPL_STATE, info->peer_repl_state) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_INFO_DISK_STATE, info->peer_disk_state) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_USER,
+ !!info->peer_resync_susp_user) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_PEER,
+ !!info->peer_resync_susp_peer) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_DEPENDENCY,
+ !!info->peer_resync_susp_dependency) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_INFO_IS_INTENTIONAL_DISKLESS,
+ info->peer_is_intentional_diskless) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_MAX_PARALLEL,
+ !!info->peer_resync_susp_max_parallel))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_peer_device_statistics(struct sk_buff *skb, int attrtype,
+ struct drbd_peer_device_statistics *s)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RECEIVED, s->peer_dev_received) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_SENT, s->peer_dev_sent) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_STATISTICS_PENDING, s->peer_dev_pending) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_STATISTICS_UNACKED, s->peer_dev_unacked) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_OUT_OF_SYNC,
+ s->peer_dev_out_of_sync) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RESYNC_FAILED,
+ s->peer_dev_resync_failed) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_BITMAP_UUID,
+ s->peer_dev_bitmap_uuid) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_STATISTICS_FLAGS, s->peer_dev_flags) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_TOTAL, s->peer_dev_rs_total) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_OV_START_SECTOR,
+ s->peer_dev_ov_start_sector) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_OV_STOP_SECTOR,
+ s->peer_dev_ov_stop_sector) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_OV_POSITION,
+ s->peer_dev_ov_position) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_OV_LEFT, s->peer_dev_ov_left) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_OV_SKIPPED,
+ s->peer_dev_ov_skipped) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_SAME_CSUM,
+ s->peer_dev_rs_same_csum) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_DT_START_MS,
+ s->peer_dev_rs_dt_start_ms) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_PAUSED_MS,
+ s->peer_dev_rs_paused_ms) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_DT0_MS,
+ s->peer_dev_rs_dt0_ms) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_DB0_SECTORS,
+ s->peer_dev_rs_db0_sectors) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_DT1_MS,
+ s->peer_dev_rs_dt1_ms) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_DB1_SECTORS,
+ s->peer_dev_rs_db1_sectors) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_STATISTICS_RS_C_SYNC_RATE,
+ s->peer_dev_rs_c_sync_rate) ||
+ drbd2_put_u64(skb, DRBD2_A_PEER_DEVICE_STATISTICS_UUID_FLAGS,
+ s->peer_dev_uuid_flags))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_path_info(struct sk_buff *skb, int attrtype,
+ struct drbd_nl_path_info *info)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u8(skb, DRBD2_A_PATH_INFO_ESTABLISHED, info->path_established))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * The configuration of a live object, for the dumps. The notifications
+ * describe state changes and carry no configuration, so they pass NULL.
+ */
+
+static int drbd2_put_res_opts(struct sk_buff *skb, int attrtype, struct drbd_res_opts *c)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_str(skb, DRBD2_A_RESOURCE_OPTS_CPU_MASK, c->cpu_mask,
+ c->cpu_mask_len, sizeof(c->cpu_mask)) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_ON_NO_DATA_ACCESSIBLE, c->on_no_data) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE, c->auto_promote) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_NODE_ID, c->node_id) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_PEER_ACK_WINDOW, c->peer_ack_window) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_TWOPC_TIMEOUT, c->twopc_timeout) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_TWOPC_RETRY_TIMEOUT,
+ c->twopc_retry_timeout) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_PEER_ACK_DELAY, c->peer_ack_delay) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE_TIMEOUT,
+ c->auto_promote_timeout) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_MAX_IO_DEPTH, c->nr_requests) ||
+ nla_put_s32(skb, DRBD2_A_RESOURCE_OPTS_QUORUM, c->quorum) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_ON_NO_QUORUM, c->on_no_quorum) ||
+ nla_put_s32(skb, DRBD2_A_RESOURCE_OPTS_QUORUM_MIN_REDUNDANCY,
+ c->quorum_min_redundancy) ||
+ nla_put_u32(skb, DRBD2_A_RESOURCE_OPTS_ON_SUSPENDED_PRIMARY_OUTDATED,
+ c->on_susp_primary_outdated) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_OPTS_DRBD8_COMPAT_MODE, c->drbd8_compat_mode) ||
+ nla_put_u8(skb, DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT,
+ c->explicit_drbd8_compat))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_disk_conf(struct sk_buff *skb, int attrtype, struct drbd_disk_conf *c)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_str(skb, DRBD2_A_DISK_CONF_BACKING_DEV, c->backing_dev,
+ c->backing_dev_len, sizeof(c->backing_dev)) ||
+ drbd2_put_str(skb, DRBD2_A_DISK_CONF_META_DEV, c->meta_dev,
+ c->meta_dev_len, sizeof(c->meta_dev)) ||
+ nla_put_s32(skb, DRBD2_A_DISK_CONF_META_DEV_IDX, c->meta_dev_idx) ||
+ drbd2_put_u64(skb, DRBD2_A_DISK_CONF_SIZE, c->disk_size) ||
+ nla_put_u32(skb, DRBD2_A_DISK_CONF_ON_IO_ERROR, c->on_io_error) ||
+ nla_put_s32(skb, DRBD2_A_DISK_CONF_RESYNC_AFTER, c->resync_after) ||
+ nla_put_u32(skb, DRBD2_A_DISK_CONF_AL_EXTENTS, c->al_extents) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_DISK_BARRIER, c->disk_barrier) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_DISK_FLUSHES, c->disk_flushes) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_DISK_DRAIN, c->disk_drain) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_MD_FLUSHES, c->md_flushes) ||
+ nla_put_u32(skb, DRBD2_A_DISK_CONF_DISK_TIMEOUT, c->disk_timeout) ||
+ nla_put_u32(skb, DRBD2_A_DISK_CONF_READ_BALANCING, c->read_balancing) ||
+ nla_put_u32(skb, DRBD2_A_DISK_CONF_UNPLUG_WATERMARK, c->unplug_watermark) ||
+ nla_put_u32(skb, DRBD2_A_DISK_CONF_RS_DISCARD_GRANULARITY,
+ c->rs_discard_granularity) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_AL_UPDATES, c->al_updates) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED,
+ c->discard_zeroes_if_aligned) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_DISABLE_WRITE_SAME, c->disable_write_same) ||
+ nla_put_u8(skb, DRBD2_A_DISK_CONF_BITMAP, c->d_bitmap))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * The caller decides what the reader may see: the dump hands us a copy
+ * with the shared secret cleared unless the reader has CAP_SYS_ADMIN.
+ */
+static int drbd2_put_net_conf(struct sk_buff *skb, int attrtype, struct drbd_net_conf *c)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_str(skb, DRBD2_A_NET_CONF_SHARED_SECRET, c->shared_secret,
+ c->shared_secret_len, sizeof(c->shared_secret)) ||
+ drbd2_put_str(skb, DRBD2_A_NET_CONF_CRAM_HMAC_ALG, c->cram_hmac_alg,
+ c->cram_hmac_alg_len, sizeof(c->cram_hmac_alg)) ||
+ drbd2_put_str(skb, DRBD2_A_NET_CONF_INTEGRITY_ALG, c->integrity_alg,
+ c->integrity_alg_len, sizeof(c->integrity_alg)) ||
+ drbd2_put_str(skb, DRBD2_A_NET_CONF_VERIFY_ALG, c->verify_alg,
+ c->verify_alg_len, sizeof(c->verify_alg)) ||
+ drbd2_put_str(skb, DRBD2_A_NET_CONF_CSUMS_ALG, c->csums_alg,
+ c->csums_alg_len, sizeof(c->csums_alg)) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_PROTOCOL, c->wire_protocol) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_CONNECT_INT, c->connect_int) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_TIMEOUT, c->timeout) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_PING_INT, c->ping_int) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_PING_TIMEO, c->ping_timeo) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_SNDBUF_SIZE, c->sndbuf_size) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_RCVBUF_SIZE, c->rcvbuf_size) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_KO_COUNT, c->ko_count) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_MAX_EPOCH_SIZE, c->max_epoch_size) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_AFTER_SB_0PRI, c->after_sb_0p) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_AFTER_SB_1PRI, c->after_sb_1p) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_AFTER_SB_2PRI, c->after_sb_2p) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_RR_CONFLICT, c->rr_conflict) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_ON_CONGESTION, c->on_congestion) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_CONG_FILL, c->cong_fill) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_CONG_EXTENTS, c->cong_extents) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_TWO_PRIMARIES, c->two_primaries) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_TCP_CORK, c->tcp_cork) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_ALWAYS_ASBP, c->always_asbp) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_USE_RLE, c->use_rle) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_FENCING, c->fencing_policy) ||
+ drbd2_put_str(skb, DRBD2_A_NET_CONF_CONNECTION_NAME, c->name,
+ c->name_len, sizeof(c->name)) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY,
+ c->csums_after_crash_only) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_SOCK_CHECK_TIMEO, c->sock_check_timeo) ||
+ drbd2_put_str(skb, DRBD2_A_NET_CONF_TRANSPORT_NAME, c->transport_name,
+ c->transport_name_len, sizeof(c->transport_name)) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_MAX_BUFFERS, c->max_buffers) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_ALLOW_REMOTE_READ, c->allow_remote_read) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_TLS, c->tls) ||
+ nla_put_s32(skb, DRBD2_A_NET_CONF_TLS_PRIVKEY, c->tls_privkey) ||
+ nla_put_s32(skb, DRBD2_A_NET_CONF_TLS_CERTIFICATE, c->tls_certificate) ||
+ nla_put_s32(skb, DRBD2_A_NET_CONF_TLS_KEYRING, c->tls_keyring) ||
+ nla_put_u8(skb, DRBD2_A_NET_CONF_LOAD_BALANCE_PATHS, c->load_balance_paths) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_RDMA_CTRL_RCVBUF_SIZE,
+ c->rdma_ctrl_rcvbuf_size) ||
+ nla_put_u32(skb, DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE,
+ c->rdma_ctrl_sndbuf_size))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_device_conf(struct sk_buff *skb, int attrtype, struct drbd_device_conf *c)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u32(skb, DRBD2_A_DEVICE_CONF_MAX_BIO_SIZE, c->max_bio_size) ||
+ nla_put_u8(skb, DRBD2_A_DEVICE_CONF_INTENTIONAL_DISKLESS,
+ c->intentional_diskless) ||
+ nla_put_u32(skb, DRBD2_A_DEVICE_CONF_BLOCK_SIZE, c->block_size) ||
+ nla_put_u32(skb, DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY,
+ c->discard_granularity))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_peer_device_conf(struct sk_buff *skb, int attrtype,
+ struct drbd_peer_device_conf *c)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (nla_put_u32(skb, DRBD2_A_PEER_DEVICE_CONF_RESYNC_RATE, c->resync_rate) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_CONF_C_PLAN_AHEAD, c->c_plan_ahead) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_CONF_C_DELAY_TARGET, c->c_delay_target) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_CONF_C_FILL_TARGET, c->c_fill_target) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_CONF_C_MAX_RATE, c->c_max_rate) ||
+ nla_put_u32(skb, DRBD2_A_PEER_DEVICE_CONF_C_MIN_RATE, c->c_min_rate) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_CONF_BITMAP, c->bitmap) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_CONF_RESYNC_WITHOUT_REPLICATION,
+ c->resync_without_replication) ||
+ nla_put_u8(skb, DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER, c->peer_tiebreaker))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * The paths of a connection, one attribute each, carried inside the
+ * connection nest as the legacy family carried them inside its
+ * connection message. Only the addresses identify a path here; the
+ * enclosing nest already names the resource and the peer.
+ */
+static int drbd2_put_connection_paths(struct sk_buff *skb,
+ struct drbd_connection *connection)
+{
+ struct drbd_path *path;
+ int err = 0;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(path, &connection->transport.paths, list) {
+ struct drbd_nl_path_info info = {
+ .path_established = test_bit(TR_ESTABLISHED, &path->flags),
+ };
+ struct nlattr *nla = nla_nest_start(skb, DRBD2_A_CONNECTION_PATH);
+
+ if (!nla) {
+ err = -EMSGSIZE;
+ break;
+ }
+ if (drbd2_put_context(skb, DRBD2_A_PATH_CONTEXT, NULL, NULL, NULL, path) ||
+ drbd2_put_path_info(skb, DRBD2_A_PATH_INFO, &info)) {
+ nla_nest_cancel(skb, nla);
+ err = -EMSGSIZE;
+ break;
+ }
+ nla_nest_end(skb, nla);
+ }
+ rcu_read_unlock();
+ return err;
+}
+
+/*
+ * The object nests shared by the dumps and the state change messages:
+ * one nest per object type, holding its context, state and statistics.
+ */
+
+static int drbd2_put_resource(struct sk_buff *skb, int attrtype,
+ struct drbd_resource *resource,
+ struct drbd_res_opts *res_opts,
+ struct drbd_resource_info *info,
+ struct drbd_resource_statistics *statistics,
+ struct drbd_rename_resource_info *rename_info)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_context(skb, DRBD2_A_RESOURCE_CONTEXT, resource, NULL, NULL, NULL))
+ goto fail;
+ if (info && drbd2_put_resource_info(skb, DRBD2_A_RESOURCE_INFO, info))
+ goto fail;
+ if (statistics &&
+ drbd2_put_resource_statistics(skb, DRBD2_A_RESOURCE_STATISTICS, statistics))
+ goto fail;
+ if (rename_info &&
+ drbd2_put_str(skb, DRBD2_A_RESOURCE_NEW_NAME, rename_info->res_new_name,
+ rename_info->res_new_name_len, sizeof(rename_info->res_new_name)))
+ goto fail;
+ if (res_opts && drbd2_put_res_opts(skb, DRBD2_A_RESOURCE_RESOURCE_OPTS, res_opts))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_device(struct sk_buff *skb, int attrtype,
+ struct drbd_device *device,
+ struct drbd_disk_conf *disk_conf,
+ struct drbd_device_conf *device_conf,
+ struct drbd_device_info *info,
+ struct drbd_device_statistics *statistics)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_context(skb, DRBD2_A_DEVICE_CONTEXT, device->resource, NULL, device, NULL))
+ goto fail;
+ if (info && drbd2_put_device_info(skb, DRBD2_A_DEVICE_INFO, info))
+ goto fail;
+ if (statistics &&
+ drbd2_put_device_statistics(skb, DRBD2_A_DEVICE_STATISTICS, statistics))
+ goto fail;
+ if (disk_conf && drbd2_put_disk_conf(skb, DRBD2_A_DEVICE_DISK_CONF, disk_conf))
+ goto fail;
+ if (device_conf &&
+ drbd2_put_device_conf(skb, DRBD2_A_DEVICE_DEVICE_CONF, device_conf))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_connection(struct sk_buff *skb, int attrtype,
+ struct drbd_resource *resource,
+ struct drbd_connection *connection,
+ struct drbd_net_conf *net_conf,
+ struct drbd_connection_info *info,
+ struct drbd_connection_statistics *statistics)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_context(skb, DRBD2_A_CONNECTION_CONTEXT, resource, connection, NULL, NULL))
+ goto fail;
+ if (info && drbd2_put_connection_info(skb, DRBD2_A_CONNECTION_INFO, info))
+ goto fail;
+ if (statistics &&
+ drbd2_put_connection_statistics(skb, DRBD2_A_CONNECTION_STATISTICS, statistics))
+ goto fail;
+ if (net_conf && drbd2_put_net_conf(skb, DRBD2_A_CONNECTION_NET_CONF, net_conf))
+ goto fail;
+ if (drbd2_put_connection_paths(skb, connection))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_peer_device(struct sk_buff *skb, int attrtype,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_conf *conf,
+ struct drbd_peer_device_info *info,
+ struct drbd_peer_device_statistics *statistics)
+{
+ struct drbd_device *device = peer_device->device;
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_context(skb, DRBD2_A_PEER_DEVICE_CONTEXT, device->resource,
+ peer_device->connection, device, NULL))
+ goto fail;
+ if (info && drbd2_put_peer_device_info(skb, DRBD2_A_PEER_DEVICE_INFO, info))
+ goto fail;
+ if (statistics &&
+ drbd2_put_peer_device_statistics(skb, DRBD2_A_PEER_DEVICE_STATISTICS, statistics))
+ goto fail;
+ if (conf &&
+ drbd2_put_peer_device_conf(skb, DRBD2_A_PEER_DEVICE_PEER_DEVICE_CONF, conf))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+static int drbd2_put_path(struct sk_buff *skb, int attrtype,
+ struct drbd_resource *resource,
+ struct drbd_connection *connection,
+ struct drbd_path *path,
+ struct drbd_nl_path_info *info)
+{
+ struct nlattr *nla = nla_nest_start(skb, attrtype);
+
+ if (!nla)
+ return -EMSGSIZE;
+ if (drbd2_put_context(skb, DRBD2_A_PATH_CONTEXT, resource, connection, NULL, path))
+ goto fail;
+ if (info && drbd2_put_path_info(skb, DRBD2_A_PATH_INFO, info))
+ goto fail;
+ nla_nest_end(skb, nla);
+ return 0;
+
+fail:
+ nla_nest_cancel(skb, nla);
+ return -EMSGSIZE;
+}
+
+/*
+ * pre_doit / post_doit.
+ */
+
+/* Which objects a command needs resolved, indexed by DRBD2_CMD_*. */
+static const unsigned int drbd2_cmd_flags[] = {
+ [DRBD2_CMD_RESOURCE_NEW] = 0,
+ [DRBD2_CMD_RESOURCE_DEL] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD2_CMD_RESOURCE_SET] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD2_CMD_RESOURCE_RENAME] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD2_CMD_RESOURCE_DOWN] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD2_CMD_RESOURCE_PRIMARY] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD2_CMD_RESOURCE_SECONDARY] = DRBD_ADM_NEED_RESOURCE,
+ /* both act on all devices of the resource, but reach it via a minor */
+ [DRBD2_CMD_RESOURCE_SUSPEND_IO] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_RESOURCE_RESUME_IO] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_NEW] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD2_CMD_DEVICE_DEL] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_ATTACH] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_DETACH] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DISK_SET] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_RESIZE] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_OUTDATE] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_INVALIDATE] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_NEW_CURRENT_UUID] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_CONNECTION_NEW] = DRBD_ADM_NEED_PEER_NODE,
+ [DRBD2_CMD_CONNECTION_DEL] = DRBD_ADM_NEED_CONNECTION,
+ [DRBD2_CMD_CONNECTION_CONNECT] = DRBD_ADM_NEED_CONNECTION,
+ [DRBD2_CMD_CONNECTION_DISCONNECT] = DRBD_ADM_NEED_CONNECTION,
+ [DRBD2_CMD_CONNECTION_SET] = DRBD_ADM_NEED_CONNECTION,
+ /* the peer to forget must not have a connection any more */
+ [DRBD2_CMD_CONNECTION_FORGET] = DRBD_ADM_NEED_RESOURCE,
+ [DRBD2_CMD_PATH_NEW] = DRBD_ADM_NEED_CONNECTION,
+ [DRBD2_CMD_PATH_DEL] = DRBD_ADM_NEED_CONNECTION,
+ [DRBD2_CMD_PEER_DEVICE_SET] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD2_CMD_PEER_DEVICE_INVALIDATE] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD2_CMD_PEER_DEVICE_PAUSE_SYNC] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD2_CMD_PEER_DEVICE_RESUME_SYNC] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD2_CMD_PEER_DEVICE_START_OV] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD2_CMD_TIMEOUT_TYPE_GET] = DRBD_ADM_NEED_PEER_DEVICE,
+ [DRBD2_CMD_DEVICE_CBT_FLUSH] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_CBT_START] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_CBT_STOP] = DRBD_ADM_NEED_MINOR,
+ [DRBD2_CMD_DEVICE_CBT_STATUS] = DRBD_ADM_NEED_MINOR,
+};
+
+/* Fill in the identity of the object the request refers to. */
+static int drbd2_parse_context(struct drbd_adm_ctx *ctx, struct genl_info *info)
+{
+ struct nlattr *tb[DRBD2_A_CONTEXT_PEER_ADDRESS + 1];
+ int err;
+
+ ctx->minor = MINORMASK + 1; /* no such minor */
+ ctx->volume = VOLUME_UNSPECIFIED;
+ ctx->peer_node_id = PEER_NODE_ID_UNSPECIFIED;
+
+ if (!info->attrs[DRBD2_A_CONTEXT])
+ return 0;
+
+ err = nla_parse_nested(tb, DRBD2_A_CONTEXT_PEER_ADDRESS,
+ info->attrs[DRBD2_A_CONTEXT],
+ drbd2_context_nl_policy, info->extack);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_CONTEXT_MINOR])
+ ctx->minor = nla_get_u32(tb[DRBD2_A_CONTEXT_MINOR]);
+ if (tb[DRBD2_A_CONTEXT_VOLUME])
+ ctx->volume = nla_get_u32(tb[DRBD2_A_CONTEXT_VOLUME]);
+ if (tb[DRBD2_A_CONTEXT_PEER_NODE_ID])
+ ctx->peer_node_id = nla_get_u32(tb[DRBD2_A_CONTEXT_PEER_NODE_ID]);
+ if (tb[DRBD2_A_CONTEXT_RESOURCE_NAME])
+ ctx->resource_name = nla_data(tb[DRBD2_A_CONTEXT_RESOURCE_NAME]);
+ return 0;
+}
+
+/*
+ * Allocates the command context together with the drbd2 request state,
+ * stores it in info->user_ptr[0], prepares the reply message and resolves
+ * the objects the command refers to. A failure here means post_doit is
+ * not called, so everything is cleaned up before returning.
+ */
+int drbd2_pre_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info)
+{
+ const u8 cmd = info->genlhdr->cmd;
+ struct drbd_adm_ctx *adm_ctx;
+ struct drbd2_ctx *dctx;
+ struct drbd2_req *req;
+ unsigned int flags;
+ int err;
+
+ /*
+ * genl_rcv_msg() checked CAP_NET_ADMIN for the commands with the
+ * admin-perm flag; as in the legacy family, configuring DRBD also
+ * requires CAP_SYS_ADMIN.
+ */
+ if ((ops->flags & GENL_ADMIN_PERM) && !capable(CAP_SYS_ADMIN)) {
+ GENL_SET_ERR_MSG(info, "CAP_SYS_ADMIN required");
+ return -EPERM;
+ }
+
+ dctx = kzalloc_obj(struct drbd2_ctx);
+ if (!dctx)
+ return -ENOMEM;
+
+ adm_ctx = &dctx->ctx;
+ req = &dctx->req;
+ adm_ctx->d = &drbd2_dialect;
+ adm_ctx->req = req;
+ adm_ctx->result = NO_ERROR;
+ adm_ctx->net = sock_net(skb->sk);
+ /*
+ * Only the *-set commands accept set-defaults; genl sizes info->attrs
+ * to the maxattr of the command, so do not index past it elsewhere.
+ */
+ adm_ctx->set_defaults = ops->maxattr >= DRBD2_A_SET_DEFAULTS &&
+ info->attrs[DRBD2_A_SET_DEFAULTS];
+ req->info = info;
+
+ err = drbd2_parse_context(adm_ctx, info);
+ if (err)
+ goto fail;
+
+ req->reply_skb = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL);
+ if (!req->reply_skb) {
+ err = -ENOMEM;
+ goto fail;
+ }
+ req->reply_hdr = genlmsg_put_reply(req->reply_skb, info,
+ &drbd2_nl_family, 0, cmd);
+ if (!req->reply_hdr) {
+ err = -EMSGSIZE;
+ goto fail;
+ }
+
+ flags = cmd < ARRAY_SIZE(drbd2_cmd_flags) ? drbd2_cmd_flags[cmd] : 0;
+ if (drbd_adm_ctx_resolve(adm_ctx, flags) != NO_ERROR) {
+ err = drbd2_errno(adm_ctx->result);
+ drbd2_set_extack(info, adm_ctx);
+ drbd_adm_ctx_release(adm_ctx);
+ goto fail;
+ }
+
+ info->user_ptr[0] = adm_ctx;
+ return 0;
+
+fail:
+ nlmsg_free(req->reply_skb);
+ kfree(dctx);
+ info->user_ptr[0] = NULL;
+ return err;
+}
+
+/*
+ * Sends the reply, drops the references acquired while resolving, and
+ * frees the context. The reply is gone when the command failed with an
+ * ERR_* code; the errno and the extended ACK describe that instead.
+ */
+void drbd2_post_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct drbd_adm_ctx *adm_ctx = info->user_ptr[0];
+ struct drbd2_req *req;
+
+ if (!adm_ctx)
+ return;
+
+ req = drbd2_req(adm_ctx);
+ if (req->reply_skb) {
+ drbd2_put_outcome(adm_ctx);
+ genlmsg_end(req->reply_skb, req->reply_hdr);
+ if (genlmsg_reply(req->reply_skb, info))
+ pr_err("error sending genl reply\n");
+ }
+
+ drbd_adm_ctx_release(adm_ctx);
+ kfree(container_of(adm_ctx, struct drbd2_ctx, ctx));
+}
+
+/*
+ * The attribute sets of a request.
+ */
+
+static bool drbd2_has_set(struct drbd_adm_ctx *ctx, enum drbd_nl_attr_set set)
+{
+ static const u16 tla[__DRBD_NL_SET_MAX] = {
+ [DRBD_NL_SET_DISK_CONF] = DRBD2_A_DISK_CONF,
+ [DRBD_NL_SET_NET_CONF] = DRBD2_A_NET_CONF,
+ [DRBD_NL_SET_RES_OPTS] = DRBD2_A_RESOURCE_OPTS,
+ [DRBD_NL_SET_PEER_DEVICE_CONF] = DRBD2_A_PEER_DEVICE_CONF,
+ [DRBD_NL_SET_DEVICE_CONF] = DRBD2_A_DEVICE_CONF,
+ [DRBD_NL_SET_SET_ROLE_PARMS] = DRBD2_A_SET_ROLE_PARMS,
+ [DRBD_NL_SET_RESIZE_PARMS] = DRBD2_A_RESIZE_PARMS,
+ [DRBD_NL_SET_START_OV_PARMS] = DRBD2_A_START_OV_PARMS,
+ [DRBD_NL_SET_NEW_C_UUID_PARMS] = DRBD2_A_NEW_CURRENT_UUID_PARMS,
+ [DRBD_NL_SET_DISCONNECT_PARMS] = DRBD2_A_DISCONNECT_PARMS,
+ [DRBD_NL_SET_DETACH_PARMS] = DRBD2_A_DETACH_PARMS,
+ [DRBD_NL_SET_INVALIDATE_PARMS] = DRBD2_A_INVALIDATE_PARMS,
+ [DRBD_NL_SET_INVALIDATE_PEER_PARMS] = DRBD2_A_INVALIDATE_PEER_PARMS,
+ [DRBD_NL_SET_CONNECT_PARMS] = DRBD2_A_CONNECT_PARMS,
+ [DRBD_NL_SET_RENAME_RESOURCE_PARMS] = DRBD2_A_RENAME_PARMS,
+ [DRBD_NL_SET_SUSPEND_IO_PARMS] = DRBD2_A_SUSPEND_IO_PARMS,
+ [DRBD_NL_SET_CBT_START_PARMS] = DRBD2_A_CBT_START_PARMS,
+ /* both are carried by the context nest of the request */
+ [DRBD_NL_SET_FORGET_PEER_PARMS] = DRBD2_A_CONTEXT,
+ [DRBD_NL_SET_PATH_PARMS] = DRBD2_A_CONTEXT,
+ };
+
+ return drbd2_req(ctx)->info->attrs[tla[set]] != NULL;
+}
+
+/*
+ * Parse the nest of "attrtype" into "tb". Returns -ENOMSG when the nest
+ * is not part of the request at all, which is what the core expects for
+ * a missing parameter set.
+ */
+static int drbd2_parse_set(struct drbd_adm_ctx *ctx, int attrtype, int maxtype,
+ const struct nla_policy *policy, struct nlattr **tb)
+{
+ struct genl_info *info = drbd2_req(ctx)->info;
+
+ if (!info->attrs[attrtype])
+ return -ENOMSG;
+ return nla_parse_nested(tb, maxtype, info->attrs[attrtype], policy,
+ info->extack);
+}
+
+#define DRBD2_PARSE_SET(ctx, name, NAME, tb) \
+ drbd2_parse_set((ctx), DRBD2_A_##NAME, ARRAY_SIZE(tb) - 1, \
+ drbd2_##name##_nl_policy, (tb))
+
+static int drbd2_overlay_disk_conf(struct drbd_adm_ctx *ctx, struct drbd_disk_conf *s)
+{
+ struct nlattr *tb[DRBD2_A_DISK_CONF_BITMAP + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, disk_conf, DISK_CONF, tb);
+ if (err)
+ return err;
+
+ /* the three attributes that describe the disk itself are required */
+ if (!tb[DRBD2_A_DISK_CONF_BACKING_DEV] || !tb[DRBD2_A_DISK_CONF_META_DEV] ||
+ !tb[DRBD2_A_DISK_CONF_META_DEV_IDX])
+ err = -ENOMSG;
+
+ if (tb[DRBD2_A_DISK_CONF_BACKING_DEV])
+ s->backing_dev_len = nla_strscpy(s->backing_dev,
+ tb[DRBD2_A_DISK_CONF_BACKING_DEV],
+ sizeof(s->backing_dev));
+ if (tb[DRBD2_A_DISK_CONF_META_DEV])
+ s->meta_dev_len = nla_strscpy(s->meta_dev, tb[DRBD2_A_DISK_CONF_META_DEV],
+ sizeof(s->meta_dev));
+ if (tb[DRBD2_A_DISK_CONF_META_DEV_IDX])
+ s->meta_dev_idx = nla_get_s32(tb[DRBD2_A_DISK_CONF_META_DEV_IDX]);
+ if (tb[DRBD2_A_DISK_CONF_SIZE])
+ s->disk_size = nla_get_u64(tb[DRBD2_A_DISK_CONF_SIZE]);
+ if (tb[DRBD2_A_DISK_CONF_ON_IO_ERROR])
+ s->on_io_error = nla_get_u32(tb[DRBD2_A_DISK_CONF_ON_IO_ERROR]);
+ if (tb[DRBD2_A_DISK_CONF_RESYNC_AFTER])
+ s->resync_after = nla_get_s32(tb[DRBD2_A_DISK_CONF_RESYNC_AFTER]);
+ if (tb[DRBD2_A_DISK_CONF_AL_EXTENTS])
+ s->al_extents = nla_get_u32(tb[DRBD2_A_DISK_CONF_AL_EXTENTS]);
+ if (tb[DRBD2_A_DISK_CONF_DISK_BARRIER])
+ s->disk_barrier = nla_get_u8(tb[DRBD2_A_DISK_CONF_DISK_BARRIER]);
+ if (tb[DRBD2_A_DISK_CONF_DISK_FLUSHES])
+ s->disk_flushes = nla_get_u8(tb[DRBD2_A_DISK_CONF_DISK_FLUSHES]);
+ if (tb[DRBD2_A_DISK_CONF_DISK_DRAIN])
+ s->disk_drain = nla_get_u8(tb[DRBD2_A_DISK_CONF_DISK_DRAIN]);
+ if (tb[DRBD2_A_DISK_CONF_MD_FLUSHES])
+ s->md_flushes = nla_get_u8(tb[DRBD2_A_DISK_CONF_MD_FLUSHES]);
+ if (tb[DRBD2_A_DISK_CONF_DISK_TIMEOUT])
+ s->disk_timeout = nla_get_u32(tb[DRBD2_A_DISK_CONF_DISK_TIMEOUT]);
+ if (tb[DRBD2_A_DISK_CONF_READ_BALANCING])
+ s->read_balancing = nla_get_u32(tb[DRBD2_A_DISK_CONF_READ_BALANCING]);
+ if (tb[DRBD2_A_DISK_CONF_UNPLUG_WATERMARK])
+ s->unplug_watermark = nla_get_u32(tb[DRBD2_A_DISK_CONF_UNPLUG_WATERMARK]);
+ if (tb[DRBD2_A_DISK_CONF_RS_DISCARD_GRANULARITY])
+ s->rs_discard_granularity =
+ nla_get_u32(tb[DRBD2_A_DISK_CONF_RS_DISCARD_GRANULARITY]);
+ if (tb[DRBD2_A_DISK_CONF_AL_UPDATES])
+ s->al_updates = nla_get_u8(tb[DRBD2_A_DISK_CONF_AL_UPDATES]);
+ if (tb[DRBD2_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED])
+ s->discard_zeroes_if_aligned =
+ nla_get_u8(tb[DRBD2_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED]);
+ if (tb[DRBD2_A_DISK_CONF_DISABLE_WRITE_SAME])
+ s->disable_write_same = nla_get_u8(tb[DRBD2_A_DISK_CONF_DISABLE_WRITE_SAME]);
+ if (tb[DRBD2_A_DISK_CONF_BITMAP])
+ s->d_bitmap = nla_get_u8(tb[DRBD2_A_DISK_CONF_BITMAP]);
+ return err;
+}
+
+static int drbd2_overlay_res_opts(struct drbd_adm_ctx *ctx, struct drbd_res_opts *s)
+{
+ struct nlattr *tb[DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, resource_opts, RESOURCE_OPTS, tb);
+ if (err)
+ return err;
+
+ /* the node id identifies this node in the resource; it is required */
+ if (!tb[DRBD2_A_RESOURCE_OPTS_NODE_ID])
+ err = -ENOMSG;
+
+ if (tb[DRBD2_A_RESOURCE_OPTS_CPU_MASK])
+ s->cpu_mask_len = nla_strscpy(s->cpu_mask, tb[DRBD2_A_RESOURCE_OPTS_CPU_MASK],
+ sizeof(s->cpu_mask));
+ if (tb[DRBD2_A_RESOURCE_OPTS_ON_NO_DATA_ACCESSIBLE])
+ s->on_no_data = nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_ON_NO_DATA_ACCESSIBLE]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE])
+ s->auto_promote = nla_get_u8(tb[DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_NODE_ID])
+ s->node_id = nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_NODE_ID]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_PEER_ACK_WINDOW])
+ s->peer_ack_window = nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_PEER_ACK_WINDOW]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_TWOPC_TIMEOUT])
+ s->twopc_timeout = nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_TWOPC_TIMEOUT]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_TWOPC_RETRY_TIMEOUT])
+ s->twopc_retry_timeout =
+ nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_TWOPC_RETRY_TIMEOUT]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_PEER_ACK_DELAY])
+ s->peer_ack_delay = nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_PEER_ACK_DELAY]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE_TIMEOUT])
+ s->auto_promote_timeout =
+ nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE_TIMEOUT]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_MAX_IO_DEPTH])
+ s->nr_requests = nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_MAX_IO_DEPTH]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_QUORUM])
+ s->quorum = nla_get_s32(tb[DRBD2_A_RESOURCE_OPTS_QUORUM]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_ON_NO_QUORUM])
+ s->on_no_quorum = nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_ON_NO_QUORUM]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_QUORUM_MIN_REDUNDANCY])
+ s->quorum_min_redundancy =
+ nla_get_s32(tb[DRBD2_A_RESOURCE_OPTS_QUORUM_MIN_REDUNDANCY]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_ON_SUSPENDED_PRIMARY_OUTDATED])
+ s->on_susp_primary_outdated =
+ nla_get_u32(tb[DRBD2_A_RESOURCE_OPTS_ON_SUSPENDED_PRIMARY_OUTDATED]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_DRBD8_COMPAT_MODE])
+ s->drbd8_compat_mode = nla_get_u8(tb[DRBD2_A_RESOURCE_OPTS_DRBD8_COMPAT_MODE]);
+ if (tb[DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT])
+ s->explicit_drbd8_compat =
+ nla_get_u8(tb[DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT]);
+ return err;
+}
+
+static int drbd2_overlay_net_conf(struct drbd_adm_ctx *ctx, struct drbd_net_conf *s)
+{
+ struct nlattr *tb[DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, net_conf, NET_CONF, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_NET_CONF_SHARED_SECRET])
+ s->shared_secret_len = nla_strscpy(s->shared_secret,
+ tb[DRBD2_A_NET_CONF_SHARED_SECRET],
+ sizeof(s->shared_secret));
+ if (tb[DRBD2_A_NET_CONF_CRAM_HMAC_ALG])
+ s->cram_hmac_alg_len = nla_strscpy(s->cram_hmac_alg,
+ tb[DRBD2_A_NET_CONF_CRAM_HMAC_ALG],
+ sizeof(s->cram_hmac_alg));
+ if (tb[DRBD2_A_NET_CONF_INTEGRITY_ALG])
+ s->integrity_alg_len = nla_strscpy(s->integrity_alg,
+ tb[DRBD2_A_NET_CONF_INTEGRITY_ALG],
+ sizeof(s->integrity_alg));
+ if (tb[DRBD2_A_NET_CONF_VERIFY_ALG])
+ s->verify_alg_len = nla_strscpy(s->verify_alg,
+ tb[DRBD2_A_NET_CONF_VERIFY_ALG],
+ sizeof(s->verify_alg));
+ if (tb[DRBD2_A_NET_CONF_CSUMS_ALG])
+ s->csums_alg_len = nla_strscpy(s->csums_alg, tb[DRBD2_A_NET_CONF_CSUMS_ALG],
+ sizeof(s->csums_alg));
+ if (tb[DRBD2_A_NET_CONF_PROTOCOL])
+ s->wire_protocol = nla_get_u32(tb[DRBD2_A_NET_CONF_PROTOCOL]);
+ if (tb[DRBD2_A_NET_CONF_CONNECT_INT])
+ s->connect_int = nla_get_u32(tb[DRBD2_A_NET_CONF_CONNECT_INT]);
+ if (tb[DRBD2_A_NET_CONF_TIMEOUT])
+ s->timeout = nla_get_u32(tb[DRBD2_A_NET_CONF_TIMEOUT]);
+ if (tb[DRBD2_A_NET_CONF_PING_INT])
+ s->ping_int = nla_get_u32(tb[DRBD2_A_NET_CONF_PING_INT]);
+ if (tb[DRBD2_A_NET_CONF_PING_TIMEO])
+ s->ping_timeo = nla_get_u32(tb[DRBD2_A_NET_CONF_PING_TIMEO]);
+ if (tb[DRBD2_A_NET_CONF_SNDBUF_SIZE])
+ s->sndbuf_size = nla_get_u32(tb[DRBD2_A_NET_CONF_SNDBUF_SIZE]);
+ if (tb[DRBD2_A_NET_CONF_RCVBUF_SIZE])
+ s->rcvbuf_size = nla_get_u32(tb[DRBD2_A_NET_CONF_RCVBUF_SIZE]);
+ if (tb[DRBD2_A_NET_CONF_KO_COUNT])
+ s->ko_count = nla_get_u32(tb[DRBD2_A_NET_CONF_KO_COUNT]);
+ if (tb[DRBD2_A_NET_CONF_MAX_EPOCH_SIZE])
+ s->max_epoch_size = nla_get_u32(tb[DRBD2_A_NET_CONF_MAX_EPOCH_SIZE]);
+ if (tb[DRBD2_A_NET_CONF_AFTER_SB_0PRI])
+ s->after_sb_0p = nla_get_u32(tb[DRBD2_A_NET_CONF_AFTER_SB_0PRI]);
+ if (tb[DRBD2_A_NET_CONF_AFTER_SB_1PRI])
+ s->after_sb_1p = nla_get_u32(tb[DRBD2_A_NET_CONF_AFTER_SB_1PRI]);
+ if (tb[DRBD2_A_NET_CONF_AFTER_SB_2PRI])
+ s->after_sb_2p = nla_get_u32(tb[DRBD2_A_NET_CONF_AFTER_SB_2PRI]);
+ if (tb[DRBD2_A_NET_CONF_RR_CONFLICT])
+ s->rr_conflict = nla_get_u32(tb[DRBD2_A_NET_CONF_RR_CONFLICT]);
+ if (tb[DRBD2_A_NET_CONF_ON_CONGESTION])
+ s->on_congestion = nla_get_u32(tb[DRBD2_A_NET_CONF_ON_CONGESTION]);
+ if (tb[DRBD2_A_NET_CONF_CONG_FILL])
+ s->cong_fill = nla_get_u32(tb[DRBD2_A_NET_CONF_CONG_FILL]);
+ if (tb[DRBD2_A_NET_CONF_CONG_EXTENTS])
+ s->cong_extents = nla_get_u32(tb[DRBD2_A_NET_CONF_CONG_EXTENTS]);
+ if (tb[DRBD2_A_NET_CONF_TWO_PRIMARIES])
+ s->two_primaries = nla_get_u8(tb[DRBD2_A_NET_CONF_TWO_PRIMARIES]);
+ if (tb[DRBD2_A_NET_CONF_TCP_CORK])
+ s->tcp_cork = nla_get_u8(tb[DRBD2_A_NET_CONF_TCP_CORK]);
+ if (tb[DRBD2_A_NET_CONF_ALWAYS_ASBP])
+ s->always_asbp = nla_get_u8(tb[DRBD2_A_NET_CONF_ALWAYS_ASBP]);
+ if (tb[DRBD2_A_NET_CONF_USE_RLE])
+ s->use_rle = nla_get_u8(tb[DRBD2_A_NET_CONF_USE_RLE]);
+ if (tb[DRBD2_A_NET_CONF_FENCING])
+ s->fencing_policy = nla_get_u32(tb[DRBD2_A_NET_CONF_FENCING]);
+ if (tb[DRBD2_A_NET_CONF_CONNECTION_NAME])
+ s->name_len = nla_strscpy(s->name, tb[DRBD2_A_NET_CONF_CONNECTION_NAME],
+ sizeof(s->name));
+ if (tb[DRBD2_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY])
+ s->csums_after_crash_only =
+ nla_get_u8(tb[DRBD2_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY]);
+ if (tb[DRBD2_A_NET_CONF_SOCK_CHECK_TIMEO])
+ s->sock_check_timeo = nla_get_u32(tb[DRBD2_A_NET_CONF_SOCK_CHECK_TIMEO]);
+ if (tb[DRBD2_A_NET_CONF_TRANSPORT_NAME])
+ s->transport_name_len = nla_strscpy(s->transport_name,
+ tb[DRBD2_A_NET_CONF_TRANSPORT_NAME],
+ sizeof(s->transport_name));
+ if (tb[DRBD2_A_NET_CONF_MAX_BUFFERS])
+ s->max_buffers = nla_get_u32(tb[DRBD2_A_NET_CONF_MAX_BUFFERS]);
+ if (tb[DRBD2_A_NET_CONF_ALLOW_REMOTE_READ])
+ s->allow_remote_read = nla_get_u8(tb[DRBD2_A_NET_CONF_ALLOW_REMOTE_READ]);
+ if (tb[DRBD2_A_NET_CONF_TLS])
+ s->tls = nla_get_u8(tb[DRBD2_A_NET_CONF_TLS]);
+ if (tb[DRBD2_A_NET_CONF_TLS_PRIVKEY])
+ s->tls_privkey = nla_get_s32(tb[DRBD2_A_NET_CONF_TLS_PRIVKEY]);
+ if (tb[DRBD2_A_NET_CONF_TLS_CERTIFICATE])
+ s->tls_certificate = nla_get_s32(tb[DRBD2_A_NET_CONF_TLS_CERTIFICATE]);
+ if (tb[DRBD2_A_NET_CONF_TLS_KEYRING])
+ s->tls_keyring = nla_get_s32(tb[DRBD2_A_NET_CONF_TLS_KEYRING]);
+ if (tb[DRBD2_A_NET_CONF_LOAD_BALANCE_PATHS])
+ s->load_balance_paths = nla_get_u8(tb[DRBD2_A_NET_CONF_LOAD_BALANCE_PATHS]);
+ if (tb[DRBD2_A_NET_CONF_RDMA_CTRL_RCVBUF_SIZE])
+ s->rdma_ctrl_rcvbuf_size =
+ nla_get_u32(tb[DRBD2_A_NET_CONF_RDMA_CTRL_RCVBUF_SIZE]);
+ if (tb[DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE])
+ s->rdma_ctrl_sndbuf_size =
+ nla_get_u32(tb[DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE]);
+ return 0;
+}
+
+static int drbd2_overlay_peer_device_conf(struct drbd_adm_ctx *ctx,
+ struct drbd_peer_device_conf *s)
+{
+ struct nlattr *tb[DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, peer_device_conf, PEER_DEVICE_CONF, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_RESYNC_RATE])
+ s->resync_rate = nla_get_u32(tb[DRBD2_A_PEER_DEVICE_CONF_RESYNC_RATE]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_C_PLAN_AHEAD])
+ s->c_plan_ahead = nla_get_u32(tb[DRBD2_A_PEER_DEVICE_CONF_C_PLAN_AHEAD]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_C_DELAY_TARGET])
+ s->c_delay_target = nla_get_u32(tb[DRBD2_A_PEER_DEVICE_CONF_C_DELAY_TARGET]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_C_FILL_TARGET])
+ s->c_fill_target = nla_get_u32(tb[DRBD2_A_PEER_DEVICE_CONF_C_FILL_TARGET]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_C_MAX_RATE])
+ s->c_max_rate = nla_get_u32(tb[DRBD2_A_PEER_DEVICE_CONF_C_MAX_RATE]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_C_MIN_RATE])
+ s->c_min_rate = nla_get_u32(tb[DRBD2_A_PEER_DEVICE_CONF_C_MIN_RATE]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_BITMAP])
+ s->bitmap = nla_get_u8(tb[DRBD2_A_PEER_DEVICE_CONF_BITMAP]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_RESYNC_WITHOUT_REPLICATION])
+ s->resync_without_replication =
+ nla_get_u8(tb[DRBD2_A_PEER_DEVICE_CONF_RESYNC_WITHOUT_REPLICATION]);
+ if (tb[DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER])
+ s->peer_tiebreaker = nla_get_u8(tb[DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER]);
+ return 0;
+}
+
+static int drbd2_overlay_device_conf(struct drbd_adm_ctx *ctx, struct drbd_device_conf *s)
+{
+ struct nlattr *tb[DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, device_conf, DEVICE_CONF, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_DEVICE_CONF_MAX_BIO_SIZE])
+ s->max_bio_size = nla_get_u32(tb[DRBD2_A_DEVICE_CONF_MAX_BIO_SIZE]);
+ if (tb[DRBD2_A_DEVICE_CONF_INTENTIONAL_DISKLESS])
+ s->intentional_diskless =
+ nla_get_u8(tb[DRBD2_A_DEVICE_CONF_INTENTIONAL_DISKLESS]);
+ if (tb[DRBD2_A_DEVICE_CONF_BLOCK_SIZE])
+ s->block_size = nla_get_u32(tb[DRBD2_A_DEVICE_CONF_BLOCK_SIZE]);
+ if (tb[DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY])
+ s->discard_granularity =
+ nla_get_u32(tb[DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY]);
+ return 0;
+}
+
+static int drbd2_overlay_set_role_parms(struct drbd_adm_ctx *ctx, struct drbd_set_role_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_SET_ROLE_PARMS_FORCE + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, set_role_parms, SET_ROLE_PARMS, tb);
+ if (err)
+ return err;
+
+ s->force = !!tb[DRBD2_A_SET_ROLE_PARMS_FORCE];
+ return 0;
+}
+
+static int drbd2_overlay_resize_parms(struct drbd_adm_ctx *ctx, struct drbd_resize_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_RESIZE_PARMS_AL_STRIPE_SIZE + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, resize_parms, RESIZE_PARMS, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_RESIZE_PARMS_SIZE])
+ s->resize_size = nla_get_u64(tb[DRBD2_A_RESIZE_PARMS_SIZE]);
+ s->resize_force = !!tb[DRBD2_A_RESIZE_PARMS_ASSUME_PEER_HAS_SPACE];
+ s->no_resync = !!tb[DRBD2_A_RESIZE_PARMS_ASSUME_CLEAN];
+ if (tb[DRBD2_A_RESIZE_PARMS_AL_STRIPES])
+ s->al_stripes = nla_get_u32(tb[DRBD2_A_RESIZE_PARMS_AL_STRIPES]);
+ if (tb[DRBD2_A_RESIZE_PARMS_AL_STRIPE_SIZE])
+ s->al_stripe_size = nla_get_u32(tb[DRBD2_A_RESIZE_PARMS_AL_STRIPE_SIZE]);
+ return 0;
+}
+
+static int drbd2_overlay_start_ov_parms(struct drbd_adm_ctx *ctx, struct drbd_start_ov_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_START_OV_PARMS_STOP_SECTOR + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, start_ov_parms, START_OV_PARMS, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_START_OV_PARMS_START_SECTOR])
+ s->ov_start_sector = nla_get_u64(tb[DRBD2_A_START_OV_PARMS_START_SECTOR]);
+ if (tb[DRBD2_A_START_OV_PARMS_STOP_SECTOR])
+ s->ov_stop_sector = nla_get_u64(tb[DRBD2_A_START_OV_PARMS_STOP_SECTOR]);
+ return 0;
+}
+
+static int drbd2_overlay_new_c_uuid_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_new_c_uuid_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_NEW_CURRENT_UUID_PARMS_FORCE_RESYNC + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, new_current_uuid_parms, NEW_CURRENT_UUID_PARMS, tb);
+ if (err)
+ return err;
+
+ s->clear_bm = !!tb[DRBD2_A_NEW_CURRENT_UUID_PARMS_CLEAR_BM];
+ s->force_resync = !!tb[DRBD2_A_NEW_CURRENT_UUID_PARMS_FORCE_RESYNC];
+ return 0;
+}
+
+static int drbd2_overlay_disconnect_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_disconnect_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_DISCONNECT_PARMS_FORCE + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, disconnect_parms, DISCONNECT_PARMS, tb);
+ if (err)
+ return err;
+
+ s->force_disconnect = !!tb[DRBD2_A_DISCONNECT_PARMS_FORCE];
+ return 0;
+}
+
+static int drbd2_overlay_detach_parms(struct drbd_adm_ctx *ctx, struct drbd_detach_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_DETACH_PARMS_INTENTIONAL_DISKLESS_DETACH + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, detach_parms, DETACH_PARMS, tb);
+ if (err)
+ return err;
+
+ s->force_detach = !!tb[DRBD2_A_DETACH_PARMS_FORCE];
+ s->intentional_diskless_detach =
+ !!tb[DRBD2_A_DETACH_PARMS_INTENTIONAL_DISKLESS_DETACH];
+ return 0;
+}
+
+static int drbd2_overlay_invalidate_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_invalidate_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_INVALIDATE_PARMS_RESET_BITMAP + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, invalidate_parms, INVALIDATE_PARMS, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_INVALIDATE_PARMS_SYNC_FROM_PEER_NODE_ID])
+ s->sync_from_peer_node_id =
+ nla_get_u32(tb[DRBD2_A_INVALIDATE_PARMS_SYNC_FROM_PEER_NODE_ID]);
+ if (tb[DRBD2_A_INVALIDATE_PARMS_RESET_BITMAP])
+ s->reset_bitmap = nla_get_u8(tb[DRBD2_A_INVALIDATE_PARMS_RESET_BITMAP]);
+ return 0;
+}
+
+static int drbd2_overlay_invalidate_peer_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_invalidate_peer_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_INVALIDATE_PEER_PARMS_RESET_BITMAP + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, invalidate_peer_parms, INVALIDATE_PEER_PARMS, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_INVALIDATE_PEER_PARMS_RESET_BITMAP])
+ s->p_reset_bitmap =
+ nla_get_u8(tb[DRBD2_A_INVALIDATE_PEER_PARMS_RESET_BITMAP]);
+ return 0;
+}
+
+/*
+ * connection-forget names the peer in the context of the request, not in
+ * a parameter set of its own.
+ */
+static int drbd2_overlay_forget_peer_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_forget_peer_parms *s)
+{
+ if (ctx->peer_node_id == PEER_NODE_ID_UNSPECIFIED)
+ return -ENOMSG;
+ s->forget_peer_node_id = ctx->peer_node_id;
+ return 0;
+}
+
+static int drbd2_overlay_connect_parms(struct drbd_adm_ctx *ctx, struct drbd_connect_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_CONNECT_PARMS_DISCARD_MY_DATA + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, connect_parms, CONNECT_PARMS, tb);
+ if (err)
+ return err;
+
+ s->tentative = !!tb[DRBD2_A_CONNECT_PARMS_TENTATIVE];
+ s->discard_my_data = !!tb[DRBD2_A_CONNECT_PARMS_DISCARD_MY_DATA];
+ return 0;
+}
+
+/*
+ * A path is identified by the two addresses in the context of the
+ * request; both are required.
+ */
+static int drbd2_overlay_path_parms(struct drbd_adm_ctx *ctx, struct drbd_path_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_CONTEXT_PEER_ADDRESS + 1];
+ struct genl_info *info = drbd2_req(ctx)->info;
+ int err;
+
+ if (!info->attrs[DRBD2_A_CONTEXT])
+ return -ENOMSG;
+ err = nla_parse_nested(tb, DRBD2_A_CONTEXT_PEER_ADDRESS,
+ info->attrs[DRBD2_A_CONTEXT],
+ drbd2_context_nl_policy, info->extack);
+ if (err)
+ return err;
+ if (!tb[DRBD2_A_CONTEXT_MY_ADDRESS] || !tb[DRBD2_A_CONTEXT_PEER_ADDRESS])
+ return -ENOMSG;
+
+ err = drbd2_get_address(tb[DRBD2_A_CONTEXT_MY_ADDRESS], s->my_addr, &s->my_addr_len);
+ if (err)
+ return err;
+ return drbd2_get_address(tb[DRBD2_A_CONTEXT_PEER_ADDRESS],
+ s->peer_addr, &s->peer_addr_len);
+}
+
+static int drbd2_overlay_rename_resource_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_rename_resource_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_RENAME_PARMS_NEW_NAME + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, rename_parms, RENAME_PARMS, tb);
+ if (err)
+ return err;
+ if (!tb[DRBD2_A_RENAME_PARMS_NEW_NAME])
+ return -ENOMSG;
+
+ s->new_resource_name_len = nla_strscpy(s->new_resource_name,
+ tb[DRBD2_A_RENAME_PARMS_NEW_NAME],
+ sizeof(s->new_resource_name));
+ return 0;
+}
+
+static int drbd2_overlay_suspend_io_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_suspend_io_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_SUSPEND_IO_PARMS_FLUSH_TO_PEERS + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, suspend_io_parms, SUSPEND_IO_PARMS, tb);
+ if (err)
+ return err;
+
+ if (tb[DRBD2_A_SUSPEND_IO_PARMS_BDEV_FREEZE])
+ s->bdev_freeze = nla_get_u8(tb[DRBD2_A_SUSPEND_IO_PARMS_BDEV_FREEZE]);
+ if (tb[DRBD2_A_SUSPEND_IO_PARMS_FLUSH_TO_PEERS])
+ s->flush_to_peers = nla_get_u8(tb[DRBD2_A_SUSPEND_IO_PARMS_FLUSH_TO_PEERS]);
+ return 0;
+}
+
+static int drbd2_overlay_cbt_start_parms(struct drbd_adm_ctx *ctx,
+ struct drbd_cbt_start_parms *s)
+{
+ struct nlattr *tb[DRBD2_A_CBT_START_PARMS_BOOKMARK_ID + 1];
+ int err;
+
+ err = DRBD2_PARSE_SET(ctx, cbt_start_parms, CBT_START_PARMS, tb);
+ if (err)
+ return err;
+ if (!tb[DRBD2_A_CBT_START_PARMS_BOOKMARK_ID])
+ return -ENOMSG;
+
+ s->cbt_id = nla_get_u64(tb[DRBD2_A_CBT_START_PARMS_BOOKMARK_ID]);
+ return 0;
+}
+
+static int drbd2_overlay(struct drbd_adm_ctx *ctx, enum drbd_nl_attr_set set, void *dst)
+{
+ switch (set) {
+ case DRBD_NL_SET_DISK_CONF: return drbd2_overlay_disk_conf(ctx, dst);
+ case DRBD_NL_SET_NET_CONF: return drbd2_overlay_net_conf(ctx, dst);
+ case DRBD_NL_SET_RES_OPTS: return drbd2_overlay_res_opts(ctx, dst);
+ case DRBD_NL_SET_PEER_DEVICE_CONF: return drbd2_overlay_peer_device_conf(ctx, dst);
+ case DRBD_NL_SET_DEVICE_CONF: return drbd2_overlay_device_conf(ctx, dst);
+ case DRBD_NL_SET_SET_ROLE_PARMS: return drbd2_overlay_set_role_parms(ctx, dst);
+ case DRBD_NL_SET_RESIZE_PARMS: return drbd2_overlay_resize_parms(ctx, dst);
+ case DRBD_NL_SET_START_OV_PARMS: return drbd2_overlay_start_ov_parms(ctx, dst);
+ case DRBD_NL_SET_NEW_C_UUID_PARMS: return drbd2_overlay_new_c_uuid_parms(ctx, dst);
+ case DRBD_NL_SET_DISCONNECT_PARMS: return drbd2_overlay_disconnect_parms(ctx, dst);
+ case DRBD_NL_SET_DETACH_PARMS: return drbd2_overlay_detach_parms(ctx, dst);
+ case DRBD_NL_SET_INVALIDATE_PARMS: return drbd2_overlay_invalidate_parms(ctx, dst);
+ case DRBD_NL_SET_INVALIDATE_PEER_PARMS:
+ return drbd2_overlay_invalidate_peer_parms(ctx, dst);
+ case DRBD_NL_SET_FORGET_PEER_PARMS: return drbd2_overlay_forget_peer_parms(ctx, dst);
+ case DRBD_NL_SET_CONNECT_PARMS: return drbd2_overlay_connect_parms(ctx, dst);
+ case DRBD_NL_SET_PATH_PARMS: return drbd2_overlay_path_parms(ctx, dst);
+ case DRBD_NL_SET_RENAME_RESOURCE_PARMS:
+ return drbd2_overlay_rename_resource_parms(ctx, dst);
+ case DRBD_NL_SET_SUSPEND_IO_PARMS: return drbd2_overlay_suspend_io_parms(ctx, dst);
+ case DRBD_NL_SET_CBT_START_PARMS: return drbd2_overlay_cbt_start_parms(ctx, dst);
+ case __DRBD_NL_SET_MAX:
+ break;
+ }
+ return -EINVAL;
+}
+
+/*
+ * The core asks about one invariant field at a time and stops at the
+ * first one that is present, so exactly one message is logged, for that
+ * field. Parsing the nest per query keeps that short-circuit intact.
+ */
+static bool drbd2_attr_present(struct drbd_adm_ctx *ctx, enum drbd_adm_field field)
+{
+ static const struct {
+ u16 tla;
+ u16 maxtype;
+ const struct nla_policy *policy;
+ u16 attr;
+ const char *name;
+ } invariant[] = {
+ [DRBD_ADM_F_DISK_BACKING_DEV] = { DRBD2_A_DISK_CONF,
+ DRBD2_A_DISK_CONF_BITMAP, drbd2_disk_conf_nl_policy,
+ DRBD2_A_DISK_CONF_BACKING_DEV, "disk-conf.backing-dev" },
+ [DRBD_ADM_F_DISK_META_DEV] = { DRBD2_A_DISK_CONF,
+ DRBD2_A_DISK_CONF_BITMAP, drbd2_disk_conf_nl_policy,
+ DRBD2_A_DISK_CONF_META_DEV, "disk-conf.meta-dev" },
+ [DRBD_ADM_F_DISK_META_DEV_IDX] = { DRBD2_A_DISK_CONF,
+ DRBD2_A_DISK_CONF_BITMAP, drbd2_disk_conf_nl_policy,
+ DRBD2_A_DISK_CONF_META_DEV_IDX, "disk-conf.meta-dev-idx" },
+ [DRBD_ADM_F_DISK_SIZE] = { DRBD2_A_DISK_CONF,
+ DRBD2_A_DISK_CONF_BITMAP, drbd2_disk_conf_nl_policy,
+ DRBD2_A_DISK_CONF_SIZE, "disk-conf.size" },
+ [DRBD_ADM_F_NET_TRANSPORT_NAME] = { DRBD2_A_NET_CONF,
+ DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE, drbd2_net_conf_nl_policy,
+ DRBD2_A_NET_CONF_TRANSPORT_NAME, "net-conf.transport-name" },
+ [DRBD_ADM_F_NET_LOAD_BALANCE_PATHS] = { DRBD2_A_NET_CONF,
+ DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE, drbd2_net_conf_nl_policy,
+ DRBD2_A_NET_CONF_LOAD_BALANCE_PATHS, "net-conf.load-balance-paths" },
+ [DRBD_ADM_F_RES_NODE_ID] = { DRBD2_A_RESOURCE_OPTS,
+ DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT,
+ drbd2_resource_opts_nl_policy,
+ DRBD2_A_RESOURCE_OPTS_NODE_ID, "resource-opts.node-id" },
+ };
+ struct genl_info *info = drbd2_req(ctx)->info;
+ struct nlattr **tb;
+ bool found;
+
+ if (!info->attrs[invariant[field].tla])
+ return false;
+
+ tb = kcalloc(invariant[field].maxtype + 1, sizeof(*tb), GFP_KERNEL);
+ if (!tb)
+ return false;
+ found = !nla_parse_nested(tb, invariant[field].maxtype,
+ info->attrs[invariant[field].tla],
+ invariant[field].policy, NULL) &&
+ tb[invariant[field].attr];
+ kfree(tb);
+ if (found)
+ pr_info("must not change invariant attr: %s\n", invariant[field].name);
+ return found;
+}
+
+static int drbd2_put_timeout_type(struct drbd_adm_ctx *ctx, enum drbd_timeout_flag type)
+{
+ struct drbd2_req *req = drbd2_req(ctx);
+
+ if (nla_put_u32(req->reply_skb, DRBD2_A_TIMEOUT_TYPE, type)) {
+ nlmsg_free(req->reply_skb);
+ req->reply_skb = NULL;
+ return -EMSGSIZE;
+ }
+ req->payload_reply = true;
+ return NO_ERROR;
+}
+
+static int drbd2_put_cbt_info(struct drbd_adm_ctx *ctx, struct drbd_cbt_info *ci)
+{
+ struct drbd2_req *req = drbd2_req(ctx);
+ struct sk_buff *skb = req->reply_skb;
+ struct nlattr *nla = nla_nest_start(skb, DRBD2_A_CBT_INFO);
+
+ if (!nla)
+ goto fail;
+ if (nla_put_u8(skb, DRBD2_A_CBT_INFO_ACTIVE, ci->cbt_active))
+ goto fail;
+ if (ci->cbt_active &&
+ (nla_put_u8(skb, DRBD2_A_CBT_INFO_VALID, ci->cbt_valid) ||
+ drbd2_put_u64(skb, DRBD2_A_CBT_INFO_BOOKMARK_ID, ci->cbt_bookmark_id) ||
+ drbd2_put_u64(skb, DRBD2_A_CBT_INFO_SINCE, ci->cbt_since) ||
+ drbd2_put_u64(skb, DRBD2_A_CBT_INFO_BITMAP_UUID, ci->cbt_bitmap_uuid) ||
+ drbd2_put_u64(skb, DRBD2_A_CBT_INFO_BITMAP_DAGTAG, ci->cbt_bitmap_dagtag)))
+ goto fail;
+ nla_nest_end(skb, nla);
+ req->payload_reply = true;
+ return NO_ERROR;
+
+fail:
+ nlmsg_free(req->reply_skb);
+ req->reply_skb = NULL;
+ return -EMSGSIZE;
+}
+
+/*
+ * The generated entry points. Each runs the wire-format independent
+ * command implementation in drbd_nl.c and turns an ERR_* outcome into the
+ * errno and extended ACK this dialect reports errors with.
+ */
+static int drbd2_do(struct genl_info *info, int (*fn)(struct drbd_adm_ctx *ctx))
+{
+ struct drbd_adm_ctx *ctx = info->user_ptr[0];
+ struct drbd2_req *req = drbd2_req(ctx);
+ int err;
+
+ err = fn(ctx);
+ if (!err) {
+ if (drbd2_is_ret_code(ctx->result))
+ err = drbd2_errno(ctx->result);
+ else if (ctx->result < SS_AFTER_LAST_ERROR)
+ err = ctx->result; /* a plain -errno */
+ }
+ if (err) {
+ drbd2_set_extack(info, ctx);
+ nlmsg_free(req->reply_skb);
+ req->reply_skb = NULL;
+ }
+ return err;
+}
+
+#define DRBD2_DOIT(cmd, fn) \
+int drbd2_nl_##cmd##_doit(struct sk_buff *skb, struct genl_info *info) \
+{ \
+ return drbd2_do(info, fn); \
+}
+
+DRBD2_DOIT(resource_new, drbd_adm_new_resource)
+DRBD2_DOIT(resource_del, drbd_adm_del_resource)
+DRBD2_DOIT(resource_set, drbd_adm_resource_opts)
+DRBD2_DOIT(resource_rename, drbd_adm_rename_resource)
+DRBD2_DOIT(resource_down, drbd_adm_down)
+DRBD2_DOIT(resource_primary, drbd_adm_primary)
+DRBD2_DOIT(resource_secondary, drbd_adm_secondary)
+DRBD2_DOIT(resource_suspend_io, drbd_adm_suspend_io)
+DRBD2_DOIT(resource_resume_io, drbd_adm_resume_io)
+DRBD2_DOIT(device_new, drbd_adm_new_minor)
+DRBD2_DOIT(device_del, drbd_adm_del_minor)
+DRBD2_DOIT(device_attach, drbd_adm_attach)
+DRBD2_DOIT(device_detach, drbd_adm_detach)
+DRBD2_DOIT(disk_set, drbd_adm_disk_opts)
+DRBD2_DOIT(device_resize, drbd_adm_resize)
+DRBD2_DOIT(device_outdate, drbd_adm_outdate)
+DRBD2_DOIT(device_invalidate, drbd_adm_invalidate)
+DRBD2_DOIT(device_new_current_uuid, drbd_adm_new_c_uuid)
+DRBD2_DOIT(connection_new, drbd_adm_new_peer)
+DRBD2_DOIT(connection_del, drbd_adm_del_peer)
+DRBD2_DOIT(connection_connect, drbd_adm_connect)
+DRBD2_DOIT(connection_disconnect, drbd_adm_disconnect)
+DRBD2_DOIT(connection_set, drbd_adm_net_opts)
+DRBD2_DOIT(connection_forget, drbd_adm_forget_peer)
+DRBD2_DOIT(path_new, drbd_adm_new_path)
+DRBD2_DOIT(path_del, drbd_adm_del_path)
+DRBD2_DOIT(peer_device_set, drbd_adm_peer_device_opts)
+DRBD2_DOIT(peer_device_invalidate, drbd_adm_invalidate_peer)
+DRBD2_DOIT(peer_device_pause_sync, drbd_adm_pause_sync)
+DRBD2_DOIT(peer_device_resume_sync, drbd_adm_resume_sync)
+DRBD2_DOIT(peer_device_start_ov, drbd_adm_start_ov)
+DRBD2_DOIT(timeout_type_get, drbd_adm_get_timeout_type)
+DRBD2_DOIT(device_cbt_flush, drbd_adm_cbt_flush)
+DRBD2_DOIT(device_cbt_start, drbd_adm_cbt_start)
+DRBD2_DOIT(device_cbt_stop, drbd_adm_cbt_stop)
+DRBD2_DOIT(device_cbt_status, drbd_adm_cbt_status)
+
+/*
+ * Dumps.
+ *
+ * The core walks the objects; everything below turns one object into one
+ * message of this dialect. This dialect has no room for a result code in
+ * a dump message, so a "retcode" other than NO_ERROR aborts the dump with
+ * the matching errno instead.
+ */
+
+static void *drbd2_dump_put(struct sk_buff *skb, struct netlink_callback *cb, u8 cmd)
+{
+ return genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
+ &drbd2_nl_family, NLM_F_MULTI, cmd);
+}
+
+static int drbd2_emit_resource(struct sk_buff *skb, struct netlink_callback *cb,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *info,
+ struct drbd_resource_statistics *statistics)
+{
+ void *dh = drbd2_dump_put(skb, cb, DRBD2_CMD_RESOURCE_GET);
+
+ if (!dh)
+ return -EMSGSIZE;
+ if (drbd2_put_resource(skb, DRBD2_A_RESOURCE, resource, &resource->res_opts,
+ info, statistics, NULL)) {
+ genlmsg_cancel(skb, dh);
+ return -EMSGSIZE;
+ }
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+static int drbd2_emit_device(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_device *device, struct drbd_disk_conf *disk_conf,
+ struct drbd_device_info *info,
+ struct drbd_device_statistics *statistics)
+{
+ void *dh;
+
+ if (retcode != NO_ERROR)
+ return drbd2_errno(retcode);
+ dh = drbd2_dump_put(skb, cb, DRBD2_CMD_DEVICE_GET);
+ if (!dh)
+ return -EMSGSIZE;
+ if (drbd2_put_device(skb, DRBD2_A_DEVICE, device, disk_conf,
+ &device->device_conf, info, statistics)) {
+ genlmsg_cancel(skb, dh);
+ return -EMSGSIZE;
+ }
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+static int drbd2_emit_connection(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_resource *resource,
+ struct drbd_connection *connection,
+ struct drbd_net_conf *net_conf,
+ struct drbd_connection_info *info,
+ struct drbd_connection_statistics *statistics)
+{
+ void *dh;
+
+ if (retcode != NO_ERROR)
+ return drbd2_errno(retcode);
+ dh = drbd2_dump_put(skb, cb, DRBD2_CMD_CONNECTION_GET);
+ if (!dh)
+ return -EMSGSIZE;
+ if (drbd2_put_connection(skb, DRBD2_A_CONNECTION, resource, connection,
+ net_conf, info, statistics)) {
+ genlmsg_cancel(skb, dh);
+ return -EMSGSIZE;
+ }
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+static int drbd2_emit_peer_device(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_peer_device *peer_device, unsigned int minor,
+ struct drbd_peer_device_info *info,
+ struct drbd_peer_device_statistics *statistics,
+ struct drbd_peer_device_conf *conf)
+{
+ void *dh;
+
+ if (retcode != NO_ERROR)
+ return drbd2_errno(retcode);
+ dh = drbd2_dump_put(skb, cb, DRBD2_CMD_PEER_DEVICE_GET);
+ if (!dh)
+ return -EMSGSIZE;
+ if (drbd2_put_peer_device(skb, DRBD2_A_PEER_DEVICE, peer_device, conf,
+ info, statistics)) {
+ genlmsg_cancel(skb, dh);
+ return -EMSGSIZE;
+ }
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+static int drbd2_emit_path(struct sk_buff *skb, struct netlink_callback *cb, int retcode,
+ struct drbd_resource *resource, struct drbd_connection *connection,
+ struct drbd_path *path, struct drbd_nl_path_info *info)
+{
+ void *dh;
+
+ if (retcode != NO_ERROR)
+ return drbd2_errno(retcode);
+ dh = drbd2_dump_put(skb, cb, DRBD2_CMD_PATH_GET);
+ if (!dh)
+ return -EMSGSIZE;
+ if (drbd2_put_path(skb, DRBD2_A_PATH, resource, connection, path, info)) {
+ genlmsg_cancel(skb, dh);
+ return -EMSGSIZE;
+ }
+ genlmsg_end(skb, dh);
+ return 0;
+}
+
+/*
+ * The dump callbacks run outside the genl lock and get the request as a
+ * raw message, so the optional context nest is looked up by hand.
+ */
+static struct nlattr *drbd2_find_context_attr(const struct nlmsghdr *nlh, int attr)
+{
+ struct nlattr *nla;
+
+ nla = nla_find(nlmsg_attrdata(nlh, GENL_HDRLEN),
+ nlmsg_attrlen(nlh, GENL_HDRLEN), DRBD2_A_CONTEXT);
+ if (!nla)
+ return NULL;
+ return nla_find_nested(nla, attr);
+}
+
+/*
+ * Resolve the optional resource-name filter of a dump on its first call.
+ * The core expects the resource in cb->args[0], with a reference that the
+ * matching _done callback drops again. Returns -ENOENT when the name does
+ * not name a resource, and 0 when there is nothing to filter by.
+ */
+static int drbd2_dump_filter(struct netlink_callback *cb, int holder_nr)
+{
+ struct drbd_resource *resource;
+ struct nlattr *filter;
+
+ filter = drbd2_find_context_attr(cb->nlh, DRBD2_A_CONTEXT_RESOURCE_NAME);
+ if (IS_ERR_OR_NULL(filter))
+ return 0;
+ resource = drbd_find_resource(nla_data(filter));
+ if (!resource)
+ return -ENOENT;
+ cb->args[0] = (long)resource;
+ return 0;
+}
+
+int drbd2_nl_resource_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ return drbd_dump_resources(skb, cb, &drbd2_dialect);
+}
+
+int drbd2_nl_device_get_done(struct netlink_callback *cb)
+{
+ return drbd_dump_devices_done(cb);
+}
+
+int drbd2_nl_device_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ if (!cb->args[0] && !cb->args[1]) {
+ int err = drbd2_dump_filter(cb, 7);
+
+ if (err)
+ return err;
+ }
+ return drbd_dump_devices(skb, cb, &drbd2_dialect);
+}
+
+int drbd2_nl_connection_get_done(struct netlink_callback *cb)
+{
+ return drbd_dump_connections_done(cb);
+}
+
+int drbd2_nl_connection_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ if (!cb->args[0]) {
+ int err = drbd2_dump_filter(cb, 6);
+
+ if (err)
+ return err;
+ if (cb->args[0])
+ cb->args[1] = DRBD_DUMP_SINGLE_RESOURCE;
+ }
+ return drbd_dump_connections(skb, cb, &drbd2_dialect);
+}
+
+int drbd2_nl_peer_device_get_done(struct netlink_callback *cb)
+{
+ return drbd_dump_peer_devices_done(cb);
+}
+
+int drbd2_nl_peer_device_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ if (!cb->args[0] && !cb->args[1]) {
+ int err = drbd2_dump_filter(cb, 9);
+
+ if (err)
+ return err;
+ }
+ return drbd_dump_peer_devices(skb, cb, &drbd2_dialect);
+}
+
+int drbd2_nl_path_get_done(struct netlink_callback *cb)
+{
+ return drbd_dump_paths_done(cb);
+}
+
+int drbd2_nl_path_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ if (!cb->args[0]) {
+ int err = drbd2_dump_filter(cb, 10);
+
+ if (err)
+ return err;
+ if (cb->args[0])
+ cb->args[1] = DRBD_DUMP_SINGLE_RESOURCE;
+ }
+ return drbd_dump_paths(skb, cb, &drbd2_dialect);
+}
+
+int drbd2_nl_state_get_done(struct netlink_callback *cb)
+{
+ return drbd_dump_initial_state_done(cb);
+}
+
+int drbd2_nl_state_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ /* the replayed state changes are sent under the sequence number of the dump */
+ return drbd_dump_initial_state(skb, cb, &drbd2_dialect, cb->nlh->nlmsg_seq);
+}
+
+/*
+ * Notifications.
+ *
+ * With an skb this is the initial state snapshot of the state-get dump:
+ * append the message to that skb. Without one, allocate a message and
+ * multicast it to the "events" group. Both carry the same attributes; the
+ * command tells them apart. The sequence number comes from the core; it
+ * is the same for every dialect describing the event.
+ */
+
+static u32 drbd2_state_change_action(enum drbd_notification_type type)
+{
+ switch (type & ~NOTIFY_FLAGS) {
+ case NOTIFY_CREATE: return DRBD2_STATE_CHANGE_ACTION_CREATE;
+ case NOTIFY_CHANGE: return DRBD2_STATE_CHANGE_ACTION_CHANGE;
+ case NOTIFY_DESTROY: return DRBD2_STATE_CHANGE_ACTION_DESTROY;
+ case NOTIFY_RENAME: return DRBD2_STATE_CHANGE_ACTION_RENAME;
+ default: return DRBD2_STATE_CHANGE_ACTION_EXISTS;
+ }
+}
+
+static void *drbd2_state_change_put(struct sk_buff *skb, unsigned int seq, bool replay,
+ enum drbd_notification_type type)
+{
+ void *dh = genlmsg_put(skb, 0, seq, &drbd2_nl_family,
+ replay ? NLM_F_MULTI : 0,
+ replay ? DRBD2_CMD_STATE_GET : DRBD2_CMD_STATE_CHANGE_NTF);
+
+ if (!dh)
+ return NULL;
+ if (nla_put_u32(skb, DRBD2_A_STATE_CHANGE_ACTION,
+ drbd2_state_change_action(type)))
+ goto fail;
+ if ((type & NOTIFY_CONTINUES) &&
+ nla_put_flag(skb, DRBD2_A_STATE_CHANGE_MORE))
+ goto fail;
+ return dh;
+
+fail:
+ genlmsg_cancel(skb, dh);
+ return NULL;
+}
+
+static int drbd2_notify_resource_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_resource *resource,
+ struct drbd_resource_info *info,
+ struct drbd_rename_resource_info *rename_info,
+ enum drbd_notification_type type)
+{
+ struct drbd_resource_statistics statistics;
+ bool multicast = !skb;
+ void *dh;
+ int err;
+
+ if (multicast) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ }
+
+ err = -EMSGSIZE;
+ dh = drbd2_state_change_put(skb, seq, !multicast, type);
+ if (!dh)
+ goto nla_put_failure;
+ resource_to_statistics(&statistics, resource);
+ if (drbd2_put_resource(skb, DRBD2_A_STATE_CHANGE_RESOURCE, resource, NULL,
+ info, &statistics, rename_info))
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = drbd2_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(resource, "Error %d while broadcasting event. Event seq:%u\n", err, seq);
+ return err;
+}
+
+static int drbd2_notify_device_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_device *device,
+ struct drbd_device_info *info,
+ enum drbd_notification_type type)
+{
+ struct drbd_device_statistics statistics;
+ bool multicast = !skb;
+ void *dh;
+ int err;
+
+ if (multicast) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ }
+
+ err = -EMSGSIZE;
+ dh = drbd2_state_change_put(skb, seq, !multicast, type);
+ if (!dh)
+ goto nla_put_failure;
+ device_to_statistics(&statistics, device);
+ if (drbd2_put_device(skb, DRBD2_A_STATE_CHANGE_DEVICE, device, NULL, NULL,
+ (type & ~NOTIFY_FLAGS) == NOTIFY_DESTROY ? NULL : info,
+ &statistics))
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = drbd2_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(device, "Error %d while broadcasting event. Event seq:%u\n", err, seq);
+ return err;
+}
+
+static int drbd2_notify_connection_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_connection *connection,
+ struct drbd_connection_info *info,
+ enum drbd_notification_type type)
+{
+ struct drbd_connection_statistics statistics;
+ bool multicast = !skb;
+ void *dh;
+ int err;
+
+ if (multicast) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ }
+
+ err = -EMSGSIZE;
+ dh = drbd2_state_change_put(skb, seq, !multicast, type);
+ if (!dh)
+ goto nla_put_failure;
+ connection_to_statistics(&statistics, connection);
+ if (drbd2_put_connection(skb, DRBD2_A_STATE_CHANGE_CONNECTION,
+ connection->resource, connection, NULL,
+ (type & ~NOTIFY_FLAGS) == NOTIFY_DESTROY ? NULL : info,
+ &statistics))
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = drbd2_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(connection, "Error %d while broadcasting event. Event seq:%u\n", err, seq);
+ return err;
+}
+
+static int drbd2_notify_peer_device_state(struct sk_buff *skb, unsigned int seq,
+ struct drbd_peer_device *peer_device,
+ struct drbd_peer_device_info *info,
+ enum drbd_notification_type type)
+{
+ struct drbd_peer_device_statistics statistics;
+ bool multicast = !skb;
+ void *dh;
+ int err;
+
+ if (multicast) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ }
+
+ err = -EMSGSIZE;
+ dh = drbd2_state_change_put(skb, seq, !multicast, type);
+ if (!dh)
+ goto nla_put_failure;
+ peer_device_to_statistics(&statistics, peer_device);
+ if (drbd2_put_peer_device(skb, DRBD2_A_STATE_CHANGE_PEER_DEVICE, peer_device, NULL,
+ (type & ~NOTIFY_FLAGS) == NOTIFY_DESTROY ? NULL : info,
+ &statistics))
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = drbd2_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ drbd_err(peer_device, "Error %d while broadcasting event. Event seq:%u\n", err, seq);
+ return err;
+}
+
+static int drbd2_notify_path_state(struct sk_buff *skb, unsigned int seq,
+ /* until we have a backpointer in drbd_path,
+ * we need an explicit connection:
+ */
+ struct drbd_connection *connection,
+ struct drbd_path *path,
+ struct drbd_nl_path_info *info,
+ enum drbd_notification_type type)
+{
+ bool multicast = !skb;
+ void *dh;
+ int err;
+
+ if (multicast) {
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto failed;
+ }
+
+ err = -EMSGSIZE;
+ dh = drbd2_state_change_put(skb, seq, !multicast, type);
+ if (!dh)
+ goto nla_put_failure;
+ if (drbd2_put_path(skb, DRBD2_A_STATE_CHANGE_PATH, connection->resource,
+ connection, path, info))
+ goto nla_put_failure;
+ genlmsg_end(skb, dh);
+ if (multicast) {
+ err = drbd2_genl_multicast_events(skb);
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto failed;
+ }
+ return 0;
+
+nla_put_failure:
+ nlmsg_free(skb);
+failed:
+ /* FIXME add path specifics to our drbd_polymorph_printk.h */
+ drbd_err(connection, "path: Error %d while broadcasting event. Event seq:%u\n",
+ err, seq);
+ return err;
+}
+
+/* Helper events are never replayed, so "skb" is always NULL here. */
+static int drbd2_notify_helper(struct sk_buff *skb, unsigned int seq,
+ struct drbd_device *device, struct drbd_connection *connection,
+ const char *name, int status,
+ enum drbd_notification_type type)
+{
+ struct drbd_resource *resource = device ? device->resource : connection->resource;
+ struct nlattr *nla;
+ void *dh;
+ int err;
+
+ skb = genlmsg_new(NLMSG_GOODSIZE, GFP_NOIO);
+ err = -ENOMEM;
+ if (!skb)
+ goto fail;
+
+ err = -EMSGSIZE;
+ dh = genlmsg_put(skb, 0, seq, &drbd2_nl_family, 0, DRBD2_CMD_HELPER_NTF);
+ if (!dh)
+ goto fail;
+ if (drbd2_put_context(skb, DRBD2_A_CONTEXT, resource, connection, device, NULL))
+ goto fail;
+ nla = nla_nest_start(skb, DRBD2_A_HELPER);
+ if (!nla)
+ goto fail;
+ if (nla_put_string(skb, DRBD2_A_HELPER_INFO_NAME, name) ||
+ nla_put_u32(skb, DRBD2_A_HELPER_INFO_STATUS, status) ||
+ nla_put_u32(skb, DRBD2_A_HELPER_INFO_PHASE,
+ (type & ~NOTIFY_FLAGS) == NOTIFY_RESPONSE ?
+ DRBD2_HELPER_PHASE_RESPONSE : DRBD2_HELPER_PHASE_CALL))
+ goto fail;
+ nla_nest_end(skb, nla);
+ genlmsg_end(skb, dh);
+ err = drbd2_genl_multicast_events(skb);
+ skb = NULL;
+ /* skb has been consumed or freed in netlink_broadcast() */
+ if (err && err != -ESRCH)
+ goto fail;
+ return 0;
+
+fail:
+ nlmsg_free(skb);
+ drbd_err(resource, "Error %d while broadcasting event. Event seq:%u\n", err, seq);
+ return err;
+}
+
+/*
+ * The end of the state-get dump needs no message of its own: appending
+ * nothing ends the dump, and netlink terminates it with NLMSG_DONE.
+ */
+static int drbd2_notify_initial_state_done(struct sk_buff *skb, unsigned int seq)
+{
+ return 0;
+}
+
+static const struct drbd_nl_dialect drbd2_dialect = {
+ .name = "drbd2",
+ .has_set = drbd2_has_set,
+ .overlay = drbd2_overlay,
+ .attr_present = drbd2_attr_present,
+ .put_timeout_type = drbd2_put_timeout_type,
+ .put_cbt_info = drbd2_put_cbt_info,
+ .emit_resource = drbd2_emit_resource,
+ .emit_device = drbd2_emit_device,
+ .emit_connection = drbd2_emit_connection,
+ .emit_peer_device = drbd2_emit_peer_device,
+ .emit_path = drbd2_emit_path,
+ .notify_resource_state = drbd2_notify_resource_state,
+ .notify_device_state = drbd2_notify_device_state,
+ .notify_connection_state = drbd2_notify_connection_state,
+ .notify_peer_device_state = drbd2_notify_peer_device_state,
+ .notify_path_state = drbd2_notify_path_state,
+ .notify_helper = drbd2_notify_helper,
+ .notify_initial_state_done = drbd2_notify_initial_state_done,
+};
+
+int drbd_nl_drbd2_init(void)
+{
+ int err = genl_register_family(&drbd2_nl_family);
+
+ if (err)
+ return err;
+ err = drbd_nl_register_dialect(&drbd2_dialect);
+ if (err)
+ genl_unregister_family(&drbd2_nl_family);
+ return err;
+}
+
+void drbd_nl_drbd2_exit(void)
+{
+ genl_unregister_family(&drbd2_nl_family);
+}
diff --git a/include/uapi/linux/drbd2.h b/include/uapi/linux/drbd2.h
new file mode 100644
index 000000000000..d1ccf783caee
--- /dev/null
+++ b/include/uapi/linux/drbd2.h
@@ -0,0 +1,1038 @@
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/* Do not edit directly, auto-generated from: */
+/* Documentation/netlink/specs/drbd2.yaml */
+/* YNL-GEN uapi header */
+/* To regenerate run: tools/net/ynl/ynl-regen.sh */
+
+#ifndef _UAPI_LINUX_DRBD2_H
+#define _UAPI_LINUX_DRBD2_H
+
+#define DRBD2_FAMILY_NAME "drbd2"
+#define DRBD2_FAMILY_VERSION 1
+
+#define DRBD2_RESOURCE_NAME_MAX 128
+#define DRBD2_CONNECTION_NAME_MAX 64
+#define DRBD2_SHARED_SECRET_MAX 64
+#define DRBD2_ALG_NAME_MAX 64
+#define DRBD2_TRANSPORT_NAME_MAX 64
+#define DRBD2_DEVICE_PATH_MAX 128
+#define DRBD2_CPU_MASK_SIZE 256
+#define DRBD2_HELPER_NAME_MAX 32
+#define DRBD2_HISTORY_UUIDS_SIZE 256
+
+/**
+ * enum drbd2_io_error_policy - What to do when the backing device reports an
+ * I/O error (on-io-error).
+ * @DRBD2_IO_ERROR_POLICY_PASS_ON: Mark the failed block out of sync, change
+ * the disk state to inconsistent and retry the request on a peer.
+ * @DRBD2_IO_ERROR_POLICY_CALL_LOCAL_IO_ERROR: Call the local-io-error handler.
+ * @DRBD2_IO_ERROR_POLICY_DETACH: Detach the backing device and continue in
+ * diskless mode.
+ */
+enum drbd2_io_error_policy {
+ DRBD2_IO_ERROR_POLICY_PASS_ON,
+ DRBD2_IO_ERROR_POLICY_CALL_LOCAL_IO_ERROR,
+ DRBD2_IO_ERROR_POLICY_DETACH,
+};
+
+/**
+ * enum drbd2_fencing_policy - Fencing policy (fencing).
+ * @DRBD2_FENCING_POLICY_DONT_CARE: Take no fencing action at all.
+ * @DRBD2_FENCING_POLICY_RESOURCE: A node that becomes a disconnected primary
+ * calls the fence-peer handler to outdate the peer.
+ * @DRBD2_FENCING_POLICY_STONITH: A node that becomes a disconnected primary
+ * freezes its I/O and calls the fence-peer handler, which may have to shoot
+ * the peer.
+ */
+enum drbd2_fencing_policy {
+ DRBD2_FENCING_POLICY_DONT_CARE,
+ DRBD2_FENCING_POLICY_RESOURCE,
+ DRBD2_FENCING_POLICY_STONITH,
+};
+
+/**
+ * enum drbd2_after_sb_policy - Split-brain recovery policy (after-sb-0pri,
+ * after-sb-1pri, after-sb-2pri, rr-conflict).
+ * @DRBD2_AFTER_SB_POLICY_DISCONNECT: Do not resynchronize automatically,
+ * simply disconnect.
+ * @DRBD2_AFTER_SB_POLICY_DISCARD_YOUNGER_PRIMARY: Resynchronize from the node
+ * that became primary first.
+ * @DRBD2_AFTER_SB_POLICY_DISCARD_OLDER_PRIMARY: Resynchronize from the node
+ * that became primary last.
+ * @DRBD2_AFTER_SB_POLICY_DISCARD_ZERO_CHANGES: Resynchronize from the node
+ * that wrote data since the split brain was detected, or disconnect if both
+ * nodes wrote.
+ * @DRBD2_AFTER_SB_POLICY_DISCARD_LEAST_CHANGES: Resynchronize from the node
+ * with more modified blocks.
+ * @DRBD2_AFTER_SB_POLICY_DISCARD_LOCAL: Discard the data on this node and
+ * resynchronize it from the peer.
+ * @DRBD2_AFTER_SB_POLICY_DISCARD_REMOTE: Discard the data on the peer and
+ * resynchronize it from this node.
+ * @DRBD2_AFTER_SB_POLICY_CONSENSUS: Discard the data on the secondary if the
+ * after-sb-0pri policy would discard it as well, otherwise disconnect.
+ * @DRBD2_AFTER_SB_POLICY_DISCARD_SECONDARY: Discard the data on the secondary
+ * node.
+ * @DRBD2_AFTER_SB_POLICY_CALL_PRI_LOST_AFTER_SB: Take the after-sb-0pri
+ * decision and call the pri-lost-after-sb handler on the node that has to
+ * give up its data.
+ * @DRBD2_AFTER_SB_POLICY_VIOLENTLY_AS0P: Always take the after-sb-0pri
+ * decision, even if that changes the primary's view of the data; this can
+ * crash the primary.
+ * @DRBD2_AFTER_SB_POLICY_RETRY_CONNECT: Disconnect now and immediately retry
+ * to connect (rr-conflict only).
+ * @DRBD2_AFTER_SB_POLICY_AUTO_DISCARD: Reverse the resync direction so that
+ * the current primary is resynchronized from the current secondary
+ * (rr-conflict with protocol A only).
+ */
+enum drbd2_after_sb_policy {
+ DRBD2_AFTER_SB_POLICY_DISCONNECT,
+ DRBD2_AFTER_SB_POLICY_DISCARD_YOUNGER_PRIMARY,
+ DRBD2_AFTER_SB_POLICY_DISCARD_OLDER_PRIMARY,
+ DRBD2_AFTER_SB_POLICY_DISCARD_ZERO_CHANGES,
+ DRBD2_AFTER_SB_POLICY_DISCARD_LEAST_CHANGES,
+ DRBD2_AFTER_SB_POLICY_DISCARD_LOCAL,
+ DRBD2_AFTER_SB_POLICY_DISCARD_REMOTE,
+ DRBD2_AFTER_SB_POLICY_CONSENSUS,
+ DRBD2_AFTER_SB_POLICY_DISCARD_SECONDARY,
+ DRBD2_AFTER_SB_POLICY_CALL_PRI_LOST_AFTER_SB,
+ DRBD2_AFTER_SB_POLICY_VIOLENTLY_AS0P,
+ DRBD2_AFTER_SB_POLICY_RETRY_CONNECT,
+ DRBD2_AFTER_SB_POLICY_AUTO_DISCARD,
+};
+
+/**
+ * enum drbd2_on_no_data_policy - What to do when no data is accessible
+ * (on-no-data-accessible).
+ * @DRBD2_ON_NO_DATA_POLICY_IO_ERROR: Complete the requests with an I/O error,
+ * so that system calls fail with EIO.
+ * @DRBD2_ON_NO_DATA_POLICY_SUSPEND_IO: Suspend I/O until the data becomes
+ * accessible again or the administrator forces I/O to resume.
+ */
+enum drbd2_on_no_data_policy {
+ DRBD2_ON_NO_DATA_POLICY_IO_ERROR,
+ DRBD2_ON_NO_DATA_POLICY_SUSPEND_IO,
+};
+
+/**
+ * enum drbd2_on_no_quorum_policy - What to do when quorum is lost
+ * (on-no-quorum).
+ * @DRBD2_ON_NO_QUORUM_POLICY_IO_ERROR: Complete the requests with an I/O error
+ * while the resource has no quorum.
+ * @DRBD2_ON_NO_QUORUM_POLICY_SUSPEND_IO: Freeze I/O while the resource has no
+ * quorum.
+ */
+enum drbd2_on_no_quorum_policy {
+ DRBD2_ON_NO_QUORUM_POLICY_IO_ERROR,
+ DRBD2_ON_NO_QUORUM_POLICY_SUSPEND_IO,
+};
+
+/**
+ * enum drbd2_on_suspended_primary_outdated_policy - What to do with a
+ * suspended primary that became outdated.
+ * @DRBD2_ON_SUSPENDED_PRIMARY_OUTDATED_POLICY_DISCONNECT: Reject connection
+ * attempts and stay isolated.
+ * @DRBD2_ON_SUSPENDED_PRIMARY_OUTDATED_POLICY_FORCE_SECONDARY: Demote to
+ * secondary immediately and fail all pending and new I/O requests.
+ */
+enum drbd2_on_suspended_primary_outdated_policy {
+ DRBD2_ON_SUSPENDED_PRIMARY_OUTDATED_POLICY_DISCONNECT,
+ DRBD2_ON_SUSPENDED_PRIMARY_OUTDATED_POLICY_FORCE_SECONDARY,
+};
+
+/**
+ * enum drbd2_on_congestion_policy - What to do when the replication link is
+ * congested (on-congestion).
+ * @DRBD2_ON_CONGESTION_POLICY_BLOCK: Block application writes until the
+ * transport's send queue drains.
+ * @DRBD2_ON_CONGESTION_POLICY_PULL_AHEAD: Switch into ahead/behind mode and
+ * record further changes in the bitmap instead of replicating them.
+ * @DRBD2_ON_CONGESTION_POLICY_DISCONNECT: Disconnect from the peer instead of
+ * blocking application writes.
+ */
+enum drbd2_on_congestion_policy {
+ DRBD2_ON_CONGESTION_POLICY_BLOCK,
+ DRBD2_ON_CONGESTION_POLICY_PULL_AHEAD,
+ DRBD2_ON_CONGESTION_POLICY_DISCONNECT,
+};
+
+/**
+ * enum drbd2_read_balancing_policy - Read balancing policy (read-balancing).
+ * @DRBD2_READ_BALANCING_POLICY_PREFER_LOCAL: Always read from the local disk.
+ * @DRBD2_READ_BALANCING_POLICY_PREFER_REMOTE: Always read from a peer's disk.
+ * @DRBD2_READ_BALANCING_POLICY_ROUND_ROBIN: Alternate between the local disk
+ * and a peer for consecutive reads.
+ * @DRBD2_READ_BALANCING_POLICY_LEAST_PENDING: Read from the disk with the
+ * fewest pending requests.
+ * @DRBD2_READ_BALANCING_POLICY_WHEN_CONGESTED_REMOTE: Read from a peer only
+ * while the local disk is congested; deprecated since DRBD 9.1.12.
+ * @DRBD2_READ_BALANCING_POLICY_STRIPING_32K: Read alternating 32 KiB stripes
+ * from the local disk and from a peer.
+ * @DRBD2_READ_BALANCING_POLICY_STRIPING_64K: Read alternating 64 KiB stripes
+ * from the local disk and from a peer.
+ * @DRBD2_READ_BALANCING_POLICY_STRIPING_128K: Read alternating 128 KiB stripes
+ * from the local disk and from a peer.
+ * @DRBD2_READ_BALANCING_POLICY_STRIPING_256K: Read alternating 256 KiB stripes
+ * from the local disk and from a peer.
+ * @DRBD2_READ_BALANCING_POLICY_STRIPING_512K: Read alternating 512 KiB stripes
+ * from the local disk and from a peer.
+ * @DRBD2_READ_BALANCING_POLICY_STRIPING_1M: Read alternating 1 MiB stripes
+ * from the local disk and from a peer.
+ */
+enum drbd2_read_balancing_policy {
+ DRBD2_READ_BALANCING_POLICY_PREFER_LOCAL,
+ DRBD2_READ_BALANCING_POLICY_PREFER_REMOTE,
+ DRBD2_READ_BALANCING_POLICY_ROUND_ROBIN,
+ DRBD2_READ_BALANCING_POLICY_LEAST_PENDING,
+ DRBD2_READ_BALANCING_POLICY_WHEN_CONGESTED_REMOTE,
+ DRBD2_READ_BALANCING_POLICY_STRIPING_32K,
+ DRBD2_READ_BALANCING_POLICY_STRIPING_64K,
+ DRBD2_READ_BALANCING_POLICY_STRIPING_128K,
+ DRBD2_READ_BALANCING_POLICY_STRIPING_256K,
+ DRBD2_READ_BALANCING_POLICY_STRIPING_512K,
+ DRBD2_READ_BALANCING_POLICY_STRIPING_1M,
+};
+
+/**
+ * enum drbd2_wire_protocol - Replication protocol (protocol).
+ * @DRBD2_WIRE_PROTOCOL_A: Writes complete as soon as they reached the local
+ * disk and the local send buffer.
+ * @DRBD2_WIRE_PROTOCOL_B: Writes complete as soon as they reached the local
+ * disk and all peers acknowledged their receipt.
+ * @DRBD2_WIRE_PROTOCOL_C: Writes complete as soon as they reached the local
+ * disk and all remote disks.
+ */
+enum drbd2_wire_protocol {
+ DRBD2_WIRE_PROTOCOL_A = 1,
+ DRBD2_WIRE_PROTOCOL_B,
+ DRBD2_WIRE_PROTOCOL_C,
+};
+
+/**
+ * enum drbd2_role - Role of a resource on a node. Values mirror enum drbd_role
+ * in linux/drbd.h so both netlink dialects share one numbering.
+ * @DRBD2_ROLE_UNKNOWN: The role is not known, for example because the peer is
+ * not connected.
+ * @DRBD2_ROLE_PRIMARY: The resource is primary; its devices may be opened for
+ * writing.
+ * @DRBD2_ROLE_SECONDARY: The resource is secondary; its devices may not be
+ * opened for writing.
+ */
+enum drbd2_role {
+ DRBD2_ROLE_UNKNOWN,
+ DRBD2_ROLE_PRIMARY,
+ DRBD2_ROLE_SECONDARY,
+};
+
+/**
+ * enum drbd2_connection_state - State of a connection to a peer. Values mirror
+ * enum drbd_conn_state in linux/drbd.h so both netlink dialects share one
+ * numbering.
+ * @DRBD2_CONNECTION_STATE_STANDALONE: No network configuration is in effect
+ * and no connection is attempted.
+ * @DRBD2_CONNECTION_STATE_DISCONNECTING: Temporary state on the way to
+ * standalone while the connection is torn down.
+ * @DRBD2_CONNECTION_STATE_UNCONNECTED: Temporary state before the next
+ * connection attempt starts.
+ * @DRBD2_CONNECTION_STATE_TIMEOUT: The connection was dropped because the peer
+ * did not answer in time.
+ * @DRBD2_CONNECTION_STATE_BROKEN_PIPE: The connection was dropped because the
+ * transport reported a broken connection.
+ * @DRBD2_CONNECTION_STATE_NETWORK_FAILURE: The connection was dropped because
+ * of a network failure.
+ * @DRBD2_CONNECTION_STATE_PROTOCOL_ERROR: The connection was dropped because
+ * the peer violated the replication protocol.
+ * @DRBD2_CONNECTION_STATE_TEAR_DOWN: Temporary state while the peer closes the
+ * connection.
+ * @DRBD2_CONNECTION_STATE_CONNECTING: Listening for the peer and trying to
+ * establish a connection.
+ * @DRBD2_CONNECTION_STATE_CONNECTED: A connection to the peer is established.
+ */
+enum drbd2_connection_state {
+ DRBD2_CONNECTION_STATE_STANDALONE,
+ DRBD2_CONNECTION_STATE_DISCONNECTING,
+ DRBD2_CONNECTION_STATE_UNCONNECTED,
+ DRBD2_CONNECTION_STATE_TIMEOUT,
+ DRBD2_CONNECTION_STATE_BROKEN_PIPE,
+ DRBD2_CONNECTION_STATE_NETWORK_FAILURE,
+ DRBD2_CONNECTION_STATE_PROTOCOL_ERROR,
+ DRBD2_CONNECTION_STATE_TEAR_DOWN,
+ DRBD2_CONNECTION_STATE_CONNECTING,
+ DRBD2_CONNECTION_STATE_CONNECTED,
+};
+
+/**
+ * enum drbd2_repl_state - Replication state of a peer device. Values mirror
+ * enum drbd_repl_state in linux/drbd.h so both netlink dialects share one
+ * numbering.
+ * @DRBD2_REPL_STATE_OFF: The peer device does not replicate because the
+ * connection is not established.
+ * @DRBD2_REPL_STATE_ESTABLISHED: The peer device is connected and replicating
+ * normally.
+ * @DRBD2_REPL_STATE_STARTING_SYNC_S: A full resynchronization was requested by
+ * the administrator and starts with this node as the source.
+ * @DRBD2_REPL_STATE_STARTING_SYNC_T: A full resynchronization was requested by
+ * the administrator and starts with this node as the target.
+ * @DRBD2_REPL_STATE_WF_BITMAP_S: Waiting for the bitmap exchange to finish,
+ * with this node as the prospective resync source.
+ * @DRBD2_REPL_STATE_WF_BITMAP_T: Waiting for the bitmap exchange to finish,
+ * with this node as the prospective resync target.
+ * @DRBD2_REPL_STATE_WF_SYNC_UUID: Waiting for the resync source to create and
+ * send the new resync UUID.
+ * @DRBD2_REPL_STATE_SYNC_SOURCE: Resynchronization is running with this node
+ * as the source.
+ * @DRBD2_REPL_STATE_SYNC_TARGET: Resynchronization is running with this node
+ * as the target.
+ * @DRBD2_REPL_STATE_VERIFY_S: Online verification is running with this node as
+ * the initiator.
+ * @DRBD2_REPL_STATE_VERIFY_T: Online verification is running with this node as
+ * the target.
+ * @DRBD2_REPL_STATE_PAUSED_SYNC_S: Resynchronization with this node as the
+ * source is paused.
+ * @DRBD2_REPL_STATE_PAUSED_SYNC_T: Resynchronization with this node as the
+ * target is paused.
+ * @DRBD2_REPL_STATE_AHEAD: Replication is suspended because the link is
+ * congested; changes are only recorded in the bitmap.
+ * @DRBD2_REPL_STATE_BEHIND: The peer is ahead of this node, which will be
+ * resynchronized once the congestion is over.
+ */
+enum drbd2_repl_state {
+ DRBD2_REPL_STATE_OFF = 9,
+ DRBD2_REPL_STATE_ESTABLISHED,
+ DRBD2_REPL_STATE_STARTING_SYNC_S,
+ DRBD2_REPL_STATE_STARTING_SYNC_T,
+ DRBD2_REPL_STATE_WF_BITMAP_S,
+ DRBD2_REPL_STATE_WF_BITMAP_T,
+ DRBD2_REPL_STATE_WF_SYNC_UUID,
+ DRBD2_REPL_STATE_SYNC_SOURCE,
+ DRBD2_REPL_STATE_SYNC_TARGET,
+ DRBD2_REPL_STATE_VERIFY_S,
+ DRBD2_REPL_STATE_VERIFY_T,
+ DRBD2_REPL_STATE_PAUSED_SYNC_S,
+ DRBD2_REPL_STATE_PAUSED_SYNC_T,
+ DRBD2_REPL_STATE_AHEAD,
+ DRBD2_REPL_STATE_BEHIND,
+};
+
+/**
+ * enum drbd2_disk_state - State of a backing disk. Values mirror enum
+ * drbd_disk_state in linux/drbd.h so both netlink dialects share one
+ * numbering.
+ * @DRBD2_DISK_STATE_DISKLESS: No backing device is attached.
+ * @DRBD2_DISK_STATE_ATTACHING: The backing device's metadata is being read.
+ * @DRBD2_DISK_STATE_DETACHING: The backing device is being detached while
+ * pending I/O drains.
+ * @DRBD2_DISK_STATE_FAILED: The backing device failed; the disk becomes
+ * diskless once the peers have been told.
+ * @DRBD2_DISK_STATE_NEGOTIATING: Attaching while connected, with the data
+ * generation still being negotiated with the peers.
+ * @DRBD2_DISK_STATE_INCONSISTENT: The data is inconsistent, for example
+ * because a resynchronization is in progress.
+ * @DRBD2_DISK_STATE_OUTDATED: The data is consistent but known to be stale
+ * relative to a peer.
+ * @DRBD2_DISK_STATE_UNKNOWN: The disk state is not known; used only for a
+ * peer's disk.
+ * @DRBD2_DISK_STATE_CONSISTENT: The data is consistent but it is not yet known
+ * whether it is up to date.
+ * @DRBD2_DISK_STATE_UP_TO_DATE: The data is consistent and up to date; only
+ * this state allows application I/O.
+ */
+enum drbd2_disk_state {
+ DRBD2_DISK_STATE_DISKLESS,
+ DRBD2_DISK_STATE_ATTACHING,
+ DRBD2_DISK_STATE_DETACHING,
+ DRBD2_DISK_STATE_FAILED,
+ DRBD2_DISK_STATE_NEGOTIATING,
+ DRBD2_DISK_STATE_INCONSISTENT,
+ DRBD2_DISK_STATE_OUTDATED,
+ DRBD2_DISK_STATE_UNKNOWN,
+ DRBD2_DISK_STATE_CONSISTENT,
+ DRBD2_DISK_STATE_UP_TO_DATE,
+};
+
+/**
+ * enum drbd2_write_ordering - Write ordering method in use for the backing
+ * device.
+ * @DRBD2_WRITE_ORDERING_NONE: No ordering between dependent writes is enforced
+ * on the backing device.
+ * @DRBD2_WRITE_ORDERING_DRAIN_IO: Dependent writes are ordered by draining the
+ * backing device's request queue.
+ * @DRBD2_WRITE_ORDERING_BDEV_FLUSH: Dependent writes are ordered by issuing
+ * flushes to the backing device.
+ * @DRBD2_WRITE_ORDERING_BIO_BARRIER: Dependent writes are ordered with block
+ * device barriers, which the Linux block layer no longer provides.
+ */
+enum drbd2_write_ordering {
+ DRBD2_WRITE_ORDERING_NONE,
+ DRBD2_WRITE_ORDERING_DRAIN_IO,
+ DRBD2_WRITE_ORDERING_BDEV_FLUSH,
+ DRBD2_WRITE_ORDERING_BIO_BARRIER,
+};
+
+/**
+ * enum drbd2_timeout_type - Which wait-for-connection timeout applies
+ * (timeout-type-get reply).
+ * @DRBD2_TIMEOUT_TYPE_DEFAULT: The regular wait-for-connection timeout
+ * applies.
+ * @DRBD2_TIMEOUT_TYPE_DEGRADED: The degraded-cluster timeout applies because
+ * the local data is not up to date.
+ * @DRBD2_TIMEOUT_TYPE_PEER_OUTDATED: The timeout for an outdated peer applies.
+ */
+enum drbd2_timeout_type {
+ DRBD2_TIMEOUT_TYPE_DEFAULT,
+ DRBD2_TIMEOUT_TYPE_DEGRADED,
+ DRBD2_TIMEOUT_TYPE_PEER_OUTDATED,
+};
+
+/**
+ * enum drbd2_helper_phase - Whether a helper notification announces the call
+ * or reports its result.
+ * @DRBD2_HELPER_PHASE_CALL: The helper script is about to be called.
+ * @DRBD2_HELPER_PHASE_RESPONSE: The helper script has finished and its exit
+ * status is reported.
+ */
+enum drbd2_helper_phase {
+ DRBD2_HELPER_PHASE_CALL,
+ DRBD2_HELPER_PHASE_RESPONSE,
+};
+
+/**
+ * enum drbd2_state_change_action - What a state-change message reports about
+ * the object it carries. Values are assigned independently of the legacy
+ * drbd_notification_type.
+ * @DRBD2_STATE_CHANGE_ACTION_EXISTS: The object already existed when the
+ * notification stream was opened.
+ * @DRBD2_STATE_CHANGE_ACTION_CREATE: The object was just created.
+ * @DRBD2_STATE_CHANGE_ACTION_CHANGE: The state or statistics of the object
+ * changed.
+ * @DRBD2_STATE_CHANGE_ACTION_DESTROY: The object was destroyed.
+ * @DRBD2_STATE_CHANGE_ACTION_RENAME: The resource was renamed; the new name is
+ * carried in resource.new-name.
+ */
+enum drbd2_state_change_action {
+ DRBD2_STATE_CHANGE_ACTION_EXISTS,
+ DRBD2_STATE_CHANGE_ACTION_CREATE,
+ DRBD2_STATE_CHANGE_ACTION_CHANGE,
+ DRBD2_STATE_CHANGE_ACTION_DESTROY,
+ DRBD2_STATE_CHANGE_ACTION_RENAME,
+};
+
+/**
+ * enum drbd2_state_result - Outcome of a state change request. Error values
+ * equal the negated kernel-internal SS_* codes (value 3 is unused); success
+ * and the other non-error outcomes follow at 29 and above.
+ * @DRBD2_STATE_RESULT_UNKNOWN_ERROR: The state change failed for an
+ * unspecified reason.
+ * @DRBD2_STATE_RESULT_TWO_PRIMARIES: Multiple primaries are not allowed by the
+ * configuration.
+ * @DRBD2_STATE_RESULT_NO_UP_TO_DATE_DISK: The request needs access to
+ * up-to-date data.
+ * @DRBD2_STATE_RESULT_NO_LOCAL_DISK: Cannot resynchronize without a local
+ * disk.
+ * @DRBD2_STATE_RESULT_NO_REMOTE_DISK: Cannot resynchronize without a remote
+ * disk.
+ * @DRBD2_STATE_RESULT_CONNECTED_OUTDATES: Refusing to become outdated while
+ * connected.
+ * @DRBD2_STATE_RESULT_PRIMARY_NOP: Refusing to become primary while the peer
+ * is not outdated.
+ * @DRBD2_STATE_RESULT_RESYNC_RUNNING: Cannot start online verify or
+ * resynchronization because one is already active.
+ * @DRBD2_STATE_RESULT_ALREADY_STANDALONE: Cannot disconnect a connection that
+ * is already standalone.
+ * @DRBD2_STATE_RESULT_CW_FAILED_BY_PEER: The cluster-wide state change was
+ * refused by a peer node.
+ * @DRBD2_STATE_RESULT_IS_DISKLESS: The device is diskless but the requested
+ * operation requires a disk.
+ * @DRBD2_STATE_RESULT_DEVICE_IN_USE: The device is held open by someone.
+ * @DRBD2_STATE_RESULT_NO_NET_CONFIG: There is no network configuration for
+ * this connection.
+ * @DRBD2_STATE_RESULT_NO_VERIFY_ALG: Starting online verify requires a verify
+ * algorithm to be configured.
+ * @DRBD2_STATE_RESULT_NEED_CONNECTION: The state change requires an
+ * established connection.
+ * @DRBD2_STATE_RESULT_LOWER_THAN_OUTDATED: The disk state is lower than
+ * outdated.
+ * @DRBD2_STATE_RESULT_NOT_SUPPORTED: The peer does not support the requested
+ * protocol feature.
+ * @DRBD2_STATE_RESULT_IN_TRANSIENT_STATE: The object is in a transient state;
+ * retry after the next state change.
+ * @DRBD2_STATE_RESULT_CONCURRENT_STATE_CHANGE: Concurrent state changes were
+ * detected and this one was aborted.
+ * @DRBD2_STATE_RESULT_OTHER_VOLUME_PEER_PRIMARY: Another volume is primary on
+ * the peer, which the configuration does not allow.
+ * @DRBD2_STATE_RESULT_INTERRUPTED: The state change was interrupted.
+ * @DRBD2_STATE_RESULT_PRIMARY_READER: The peer may not become primary while
+ * the device is open for reading here.
+ * @DRBD2_STATE_RESULT_TIMEOUT: The state change timed out.
+ * @DRBD2_STATE_RESULT_WEAKLY_CONNECTED: Primary nodes must be strongly
+ * connected among each other.
+ * @DRBD2_STATE_RESULT_NO_QUORUM: The resource does not have quorum.
+ * @DRBD2_STATE_RESULT_ATTACH_NO_BITMAP: An intentionally diskless peer may not
+ * attach a disk.
+ * @DRBD2_STATE_RESULT_HANDSHAKE_DISCONNECT: Disconnect was chosen during the
+ * connection handshake.
+ * @DRBD2_STATE_RESULT_HANDSHAKE_RETRY: Retry was chosen during the connection
+ * handshake.
+ * @DRBD2_STATE_RESULT_SUCCESS: The state change succeeded.
+ * @DRBD2_STATE_RESULT_NOTHING_TO_DO: The requested state was already in
+ * effect, so nothing had to be done.
+ * @DRBD2_STATE_RESULT_CW_SUCCESS: The cluster-wide state change succeeded.
+ * @DRBD2_STATE_RESULT_CW_NO_NEED: No cluster-wide state change was necessary.
+ */
+enum drbd2_state_result {
+ DRBD2_STATE_RESULT_UNKNOWN_ERROR,
+ DRBD2_STATE_RESULT_TWO_PRIMARIES,
+ DRBD2_STATE_RESULT_NO_UP_TO_DATE_DISK,
+ DRBD2_STATE_RESULT_NO_LOCAL_DISK = 4,
+ DRBD2_STATE_RESULT_NO_REMOTE_DISK,
+ DRBD2_STATE_RESULT_CONNECTED_OUTDATES,
+ DRBD2_STATE_RESULT_PRIMARY_NOP,
+ DRBD2_STATE_RESULT_RESYNC_RUNNING,
+ DRBD2_STATE_RESULT_ALREADY_STANDALONE,
+ DRBD2_STATE_RESULT_CW_FAILED_BY_PEER,
+ DRBD2_STATE_RESULT_IS_DISKLESS,
+ DRBD2_STATE_RESULT_DEVICE_IN_USE,
+ DRBD2_STATE_RESULT_NO_NET_CONFIG,
+ DRBD2_STATE_RESULT_NO_VERIFY_ALG,
+ DRBD2_STATE_RESULT_NEED_CONNECTION,
+ DRBD2_STATE_RESULT_LOWER_THAN_OUTDATED,
+ DRBD2_STATE_RESULT_NOT_SUPPORTED,
+ DRBD2_STATE_RESULT_IN_TRANSIENT_STATE,
+ DRBD2_STATE_RESULT_CONCURRENT_STATE_CHANGE,
+ DRBD2_STATE_RESULT_OTHER_VOLUME_PEER_PRIMARY,
+ DRBD2_STATE_RESULT_INTERRUPTED,
+ DRBD2_STATE_RESULT_PRIMARY_READER,
+ DRBD2_STATE_RESULT_TIMEOUT,
+ DRBD2_STATE_RESULT_WEAKLY_CONNECTED,
+ DRBD2_STATE_RESULT_NO_QUORUM,
+ DRBD2_STATE_RESULT_ATTACH_NO_BITMAP,
+ DRBD2_STATE_RESULT_HANDSHAKE_DISCONNECT,
+ DRBD2_STATE_RESULT_HANDSHAKE_RETRY,
+ DRBD2_STATE_RESULT_SUCCESS,
+ DRBD2_STATE_RESULT_NOTHING_TO_DO,
+ DRBD2_STATE_RESULT_CW_SUCCESS,
+ DRBD2_STATE_RESULT_CW_NO_NEED,
+};
+
+enum {
+ DRBD2_A_ADDRESS_FAMILY = 1,
+ DRBD2_A_ADDRESS_PORT,
+ DRBD2_A_ADDRESS_IPV4,
+ DRBD2_A_ADDRESS_IPV6,
+
+ __DRBD2_A_ADDRESS_MAX,
+ DRBD2_A_ADDRESS_MAX = (__DRBD2_A_ADDRESS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CONTEXT_RESOURCE_NAME = 1,
+ DRBD2_A_CONTEXT_VOLUME,
+ DRBD2_A_CONTEXT_MINOR,
+ DRBD2_A_CONTEXT_PEER_NODE_ID,
+ DRBD2_A_CONTEXT_CONNECTION_NAME,
+ DRBD2_A_CONTEXT_MY_ADDRESS,
+ DRBD2_A_CONTEXT_PEER_ADDRESS,
+
+ __DRBD2_A_CONTEXT_MAX,
+ DRBD2_A_CONTEXT_MAX = (__DRBD2_A_CONTEXT_MAX - 1)
+};
+
+enum {
+ DRBD2_A_DISK_CONF_BACKING_DEV = 1,
+ DRBD2_A_DISK_CONF_META_DEV,
+ DRBD2_A_DISK_CONF_META_DEV_IDX,
+ DRBD2_A_DISK_CONF_SIZE,
+ DRBD2_A_DISK_CONF_ON_IO_ERROR,
+ DRBD2_A_DISK_CONF_RESYNC_AFTER,
+ DRBD2_A_DISK_CONF_AL_EXTENTS,
+ DRBD2_A_DISK_CONF_DISK_BARRIER,
+ DRBD2_A_DISK_CONF_DISK_FLUSHES,
+ DRBD2_A_DISK_CONF_DISK_DRAIN,
+ DRBD2_A_DISK_CONF_MD_FLUSHES,
+ DRBD2_A_DISK_CONF_DISK_TIMEOUT,
+ DRBD2_A_DISK_CONF_READ_BALANCING,
+ DRBD2_A_DISK_CONF_UNPLUG_WATERMARK,
+ DRBD2_A_DISK_CONF_RS_DISCARD_GRANULARITY,
+ DRBD2_A_DISK_CONF_AL_UPDATES,
+ DRBD2_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED,
+ DRBD2_A_DISK_CONF_DISABLE_WRITE_SAME,
+ DRBD2_A_DISK_CONF_BITMAP,
+
+ __DRBD2_A_DISK_CONF_MAX,
+ DRBD2_A_DISK_CONF_MAX = (__DRBD2_A_DISK_CONF_MAX - 1)
+};
+
+enum {
+ DRBD2_A_RESOURCE_OPTS_CPU_MASK = 1,
+ DRBD2_A_RESOURCE_OPTS_ON_NO_DATA_ACCESSIBLE,
+ DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE,
+ DRBD2_A_RESOURCE_OPTS_NODE_ID,
+ DRBD2_A_RESOURCE_OPTS_PEER_ACK_WINDOW,
+ DRBD2_A_RESOURCE_OPTS_TWOPC_TIMEOUT,
+ DRBD2_A_RESOURCE_OPTS_TWOPC_RETRY_TIMEOUT,
+ DRBD2_A_RESOURCE_OPTS_PEER_ACK_DELAY,
+ DRBD2_A_RESOURCE_OPTS_AUTO_PROMOTE_TIMEOUT,
+ DRBD2_A_RESOURCE_OPTS_MAX_IO_DEPTH,
+ DRBD2_A_RESOURCE_OPTS_QUORUM,
+ DRBD2_A_RESOURCE_OPTS_ON_NO_QUORUM,
+ DRBD2_A_RESOURCE_OPTS_QUORUM_MIN_REDUNDANCY,
+ DRBD2_A_RESOURCE_OPTS_ON_SUSPENDED_PRIMARY_OUTDATED,
+ DRBD2_A_RESOURCE_OPTS_DRBD8_COMPAT_MODE,
+ DRBD2_A_RESOURCE_OPTS_EXPLICIT_DRBD8_COMPAT,
+
+ __DRBD2_A_RESOURCE_OPTS_MAX,
+ DRBD2_A_RESOURCE_OPTS_MAX = (__DRBD2_A_RESOURCE_OPTS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_NET_CONF_SHARED_SECRET = 1,
+ DRBD2_A_NET_CONF_CRAM_HMAC_ALG,
+ DRBD2_A_NET_CONF_INTEGRITY_ALG,
+ DRBD2_A_NET_CONF_VERIFY_ALG,
+ DRBD2_A_NET_CONF_CSUMS_ALG,
+ DRBD2_A_NET_CONF_PROTOCOL,
+ DRBD2_A_NET_CONF_CONNECT_INT,
+ DRBD2_A_NET_CONF_TIMEOUT,
+ DRBD2_A_NET_CONF_PING_INT,
+ DRBD2_A_NET_CONF_PING_TIMEO,
+ DRBD2_A_NET_CONF_SNDBUF_SIZE,
+ DRBD2_A_NET_CONF_RCVBUF_SIZE,
+ DRBD2_A_NET_CONF_KO_COUNT,
+ DRBD2_A_NET_CONF_MAX_EPOCH_SIZE,
+ DRBD2_A_NET_CONF_AFTER_SB_0PRI,
+ DRBD2_A_NET_CONF_AFTER_SB_1PRI,
+ DRBD2_A_NET_CONF_AFTER_SB_2PRI,
+ DRBD2_A_NET_CONF_RR_CONFLICT,
+ DRBD2_A_NET_CONF_ON_CONGESTION,
+ DRBD2_A_NET_CONF_CONG_FILL,
+ DRBD2_A_NET_CONF_CONG_EXTENTS,
+ DRBD2_A_NET_CONF_TWO_PRIMARIES,
+ DRBD2_A_NET_CONF_TCP_CORK,
+ DRBD2_A_NET_CONF_ALWAYS_ASBP,
+ DRBD2_A_NET_CONF_USE_RLE,
+ DRBD2_A_NET_CONF_FENCING,
+ DRBD2_A_NET_CONF_CONNECTION_NAME,
+ DRBD2_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY,
+ DRBD2_A_NET_CONF_SOCK_CHECK_TIMEO,
+ DRBD2_A_NET_CONF_TRANSPORT_NAME,
+ DRBD2_A_NET_CONF_MAX_BUFFERS,
+ DRBD2_A_NET_CONF_ALLOW_REMOTE_READ,
+ DRBD2_A_NET_CONF_TLS,
+ DRBD2_A_NET_CONF_TLS_PRIVKEY,
+ DRBD2_A_NET_CONF_TLS_CERTIFICATE,
+ DRBD2_A_NET_CONF_TLS_KEYRING,
+ DRBD2_A_NET_CONF_LOAD_BALANCE_PATHS,
+ DRBD2_A_NET_CONF_RDMA_CTRL_RCVBUF_SIZE,
+ DRBD2_A_NET_CONF_RDMA_CTRL_SNDBUF_SIZE,
+
+ __DRBD2_A_NET_CONF_MAX,
+ DRBD2_A_NET_CONF_MAX = (__DRBD2_A_NET_CONF_MAX - 1)
+};
+
+enum {
+ DRBD2_A_SET_ROLE_PARMS_FORCE = 1,
+
+ __DRBD2_A_SET_ROLE_PARMS_MAX,
+ DRBD2_A_SET_ROLE_PARMS_MAX = (__DRBD2_A_SET_ROLE_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_RESIZE_PARMS_SIZE = 1,
+ DRBD2_A_RESIZE_PARMS_ASSUME_PEER_HAS_SPACE,
+ DRBD2_A_RESIZE_PARMS_ASSUME_CLEAN,
+ DRBD2_A_RESIZE_PARMS_AL_STRIPES,
+ DRBD2_A_RESIZE_PARMS_AL_STRIPE_SIZE,
+
+ __DRBD2_A_RESIZE_PARMS_MAX,
+ DRBD2_A_RESIZE_PARMS_MAX = (__DRBD2_A_RESIZE_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_START_OV_PARMS_START_SECTOR = 1,
+ DRBD2_A_START_OV_PARMS_STOP_SECTOR,
+
+ __DRBD2_A_START_OV_PARMS_MAX,
+ DRBD2_A_START_OV_PARMS_MAX = (__DRBD2_A_START_OV_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_NEW_CURRENT_UUID_PARMS_CLEAR_BM = 1,
+ DRBD2_A_NEW_CURRENT_UUID_PARMS_FORCE_RESYNC,
+
+ __DRBD2_A_NEW_CURRENT_UUID_PARMS_MAX,
+ DRBD2_A_NEW_CURRENT_UUID_PARMS_MAX = (__DRBD2_A_NEW_CURRENT_UUID_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_DISCONNECT_PARMS_FORCE = 1,
+
+ __DRBD2_A_DISCONNECT_PARMS_MAX,
+ DRBD2_A_DISCONNECT_PARMS_MAX = (__DRBD2_A_DISCONNECT_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_DETACH_PARMS_FORCE = 1,
+ DRBD2_A_DETACH_PARMS_INTENTIONAL_DISKLESS_DETACH,
+
+ __DRBD2_A_DETACH_PARMS_MAX,
+ DRBD2_A_DETACH_PARMS_MAX = (__DRBD2_A_DETACH_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_DEVICE_CONF_MAX_BIO_SIZE = 1,
+ DRBD2_A_DEVICE_CONF_INTENTIONAL_DISKLESS,
+ DRBD2_A_DEVICE_CONF_BLOCK_SIZE,
+ DRBD2_A_DEVICE_CONF_DISCARD_GRANULARITY,
+
+ __DRBD2_A_DEVICE_CONF_MAX,
+ DRBD2_A_DEVICE_CONF_MAX = (__DRBD2_A_DEVICE_CONF_MAX - 1)
+};
+
+enum {
+ DRBD2_A_RESOURCE_INFO_ROLE = 1,
+ DRBD2_A_RESOURCE_INFO_SUSP,
+ DRBD2_A_RESOURCE_INFO_SUSP_NOD,
+ DRBD2_A_RESOURCE_INFO_SUSP_FEN,
+ DRBD2_A_RESOURCE_INFO_SUSP_QUORUM,
+ DRBD2_A_RESOURCE_INFO_FAIL_IO,
+
+ __DRBD2_A_RESOURCE_INFO_MAX,
+ DRBD2_A_RESOURCE_INFO_MAX = (__DRBD2_A_RESOURCE_INFO_MAX - 1)
+};
+
+enum {
+ DRBD2_A_DEVICE_INFO_DISK_STATE = 1,
+ DRBD2_A_DEVICE_INFO_IS_INTENTIONAL_DISKLESS,
+ DRBD2_A_DEVICE_INFO_HAS_QUORUM,
+ DRBD2_A_DEVICE_INFO_IS_OPEN,
+ DRBD2_A_DEVICE_INFO_BACKING_DEV_PATH,
+
+ __DRBD2_A_DEVICE_INFO_MAX,
+ DRBD2_A_DEVICE_INFO_MAX = (__DRBD2_A_DEVICE_INFO_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CONNECTION_INFO_CONNECTION_STATE = 1,
+ DRBD2_A_CONNECTION_INFO_ROLE,
+
+ __DRBD2_A_CONNECTION_INFO_MAX,
+ DRBD2_A_CONNECTION_INFO_MAX = (__DRBD2_A_CONNECTION_INFO_MAX - 1)
+};
+
+enum {
+ DRBD2_A_PEER_DEVICE_INFO_REPL_STATE = 1,
+ DRBD2_A_PEER_DEVICE_INFO_DISK_STATE,
+ DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_USER,
+ DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_PEER,
+ DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_DEPENDENCY,
+ DRBD2_A_PEER_DEVICE_INFO_IS_INTENTIONAL_DISKLESS,
+ DRBD2_A_PEER_DEVICE_INFO_RESYNC_SUSP_MAX_PARALLEL,
+
+ __DRBD2_A_PEER_DEVICE_INFO_MAX,
+ DRBD2_A_PEER_DEVICE_INFO_MAX = (__DRBD2_A_PEER_DEVICE_INFO_MAX - 1)
+};
+
+enum {
+ DRBD2_A_RESOURCE_STATISTICS_WRITE_ORDERING = 1,
+
+ __DRBD2_A_RESOURCE_STATISTICS_MAX,
+ DRBD2_A_RESOURCE_STATISTICS_MAX = (__DRBD2_A_RESOURCE_STATISTICS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_DEVICE_STATISTICS_SIZE = 1,
+ DRBD2_A_DEVICE_STATISTICS_READ,
+ DRBD2_A_DEVICE_STATISTICS_WRITE,
+ DRBD2_A_DEVICE_STATISTICS_AL_WRITES,
+ DRBD2_A_DEVICE_STATISTICS_BM_WRITES,
+ DRBD2_A_DEVICE_STATISTICS_UPPER_PENDING,
+ DRBD2_A_DEVICE_STATISTICS_LOWER_PENDING,
+ DRBD2_A_DEVICE_STATISTICS_UPPER_BLOCKED,
+ DRBD2_A_DEVICE_STATISTICS_LOWER_BLOCKED,
+ DRBD2_A_DEVICE_STATISTICS_AL_SUSPENDED,
+ DRBD2_A_DEVICE_STATISTICS_EXPOSED_DATA_UUID,
+ DRBD2_A_DEVICE_STATISTICS_CURRENT_UUID,
+ DRBD2_A_DEVICE_STATISTICS_DISK_FLAGS,
+ DRBD2_A_DEVICE_STATISTICS_HISTORY_UUIDS,
+
+ __DRBD2_A_DEVICE_STATISTICS_MAX,
+ DRBD2_A_DEVICE_STATISTICS_MAX = (__DRBD2_A_DEVICE_STATISTICS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CONNECTION_STATISTICS_CONGESTED = 1,
+ DRBD2_A_CONNECTION_STATISTICS_AP_IN_FLIGHT,
+ DRBD2_A_CONNECTION_STATISTICS_RS_IN_FLIGHT,
+
+ __DRBD2_A_CONNECTION_STATISTICS_MAX,
+ DRBD2_A_CONNECTION_STATISTICS_MAX = (__DRBD2_A_CONNECTION_STATISTICS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_PEER_DEVICE_STATISTICS_RECEIVED = 1,
+ DRBD2_A_PEER_DEVICE_STATISTICS_SENT,
+ DRBD2_A_PEER_DEVICE_STATISTICS_PENDING,
+ DRBD2_A_PEER_DEVICE_STATISTICS_UNACKED,
+ DRBD2_A_PEER_DEVICE_STATISTICS_OUT_OF_SYNC,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RESYNC_FAILED,
+ DRBD2_A_PEER_DEVICE_STATISTICS_BITMAP_UUID,
+ DRBD2_A_PEER_DEVICE_STATISTICS_FLAGS,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_TOTAL,
+ DRBD2_A_PEER_DEVICE_STATISTICS_OV_START_SECTOR,
+ DRBD2_A_PEER_DEVICE_STATISTICS_OV_STOP_SECTOR,
+ DRBD2_A_PEER_DEVICE_STATISTICS_OV_POSITION,
+ DRBD2_A_PEER_DEVICE_STATISTICS_OV_LEFT,
+ DRBD2_A_PEER_DEVICE_STATISTICS_OV_SKIPPED,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_SAME_CSUM,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_DT_START_MS,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_PAUSED_MS,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_DT0_MS,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_DB0_SECTORS,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_DT1_MS,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_DB1_SECTORS,
+ DRBD2_A_PEER_DEVICE_STATISTICS_RS_C_SYNC_RATE,
+ DRBD2_A_PEER_DEVICE_STATISTICS_UUID_FLAGS,
+
+ __DRBD2_A_PEER_DEVICE_STATISTICS_MAX,
+ DRBD2_A_PEER_DEVICE_STATISTICS_MAX = (__DRBD2_A_PEER_DEVICE_STATISTICS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_HELPER_INFO_NAME = 1,
+ DRBD2_A_HELPER_INFO_STATUS,
+ DRBD2_A_HELPER_INFO_PHASE,
+
+ __DRBD2_A_HELPER_INFO_MAX,
+ DRBD2_A_HELPER_INFO_MAX = (__DRBD2_A_HELPER_INFO_MAX - 1)
+};
+
+enum {
+ DRBD2_A_INVALIDATE_PARMS_SYNC_FROM_PEER_NODE_ID = 1,
+ DRBD2_A_INVALIDATE_PARMS_RESET_BITMAP,
+
+ __DRBD2_A_INVALIDATE_PARMS_MAX,
+ DRBD2_A_INVALIDATE_PARMS_MAX = (__DRBD2_A_INVALIDATE_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_PEER_DEVICE_CONF_RESYNC_RATE = 1,
+ DRBD2_A_PEER_DEVICE_CONF_C_PLAN_AHEAD,
+ DRBD2_A_PEER_DEVICE_CONF_C_DELAY_TARGET,
+ DRBD2_A_PEER_DEVICE_CONF_C_FILL_TARGET,
+ DRBD2_A_PEER_DEVICE_CONF_C_MAX_RATE,
+ DRBD2_A_PEER_DEVICE_CONF_C_MIN_RATE,
+ DRBD2_A_PEER_DEVICE_CONF_BITMAP,
+ DRBD2_A_PEER_DEVICE_CONF_RESYNC_WITHOUT_REPLICATION,
+ DRBD2_A_PEER_DEVICE_CONF_PEER_TIEBREAKER,
+
+ __DRBD2_A_PEER_DEVICE_CONF_MAX,
+ DRBD2_A_PEER_DEVICE_CONF_MAX = (__DRBD2_A_PEER_DEVICE_CONF_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CONNECT_PARMS_TENTATIVE = 1,
+ DRBD2_A_CONNECT_PARMS_DISCARD_MY_DATA,
+
+ __DRBD2_A_CONNECT_PARMS_MAX,
+ DRBD2_A_CONNECT_PARMS_MAX = (__DRBD2_A_CONNECT_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_PATH_INFO_ESTABLISHED = 1,
+
+ __DRBD2_A_PATH_INFO_MAX,
+ DRBD2_A_PATH_INFO_MAX = (__DRBD2_A_PATH_INFO_MAX - 1)
+};
+
+enum {
+ DRBD2_A_RENAME_PARMS_NEW_NAME = 1,
+
+ __DRBD2_A_RENAME_PARMS_MAX,
+ DRBD2_A_RENAME_PARMS_MAX = (__DRBD2_A_RENAME_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_INVALIDATE_PEER_PARMS_RESET_BITMAP = 1,
+
+ __DRBD2_A_INVALIDATE_PEER_PARMS_MAX,
+ DRBD2_A_INVALIDATE_PEER_PARMS_MAX = (__DRBD2_A_INVALIDATE_PEER_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_SUSPEND_IO_PARMS_BDEV_FREEZE = 1,
+ DRBD2_A_SUSPEND_IO_PARMS_FLUSH_TO_PEERS,
+
+ __DRBD2_A_SUSPEND_IO_PARMS_MAX,
+ DRBD2_A_SUSPEND_IO_PARMS_MAX = (__DRBD2_A_SUSPEND_IO_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CBT_START_PARMS_BOOKMARK_ID = 1,
+
+ __DRBD2_A_CBT_START_PARMS_MAX,
+ DRBD2_A_CBT_START_PARMS_MAX = (__DRBD2_A_CBT_START_PARMS_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CBT_INFO_ACTIVE = 1,
+ DRBD2_A_CBT_INFO_VALID,
+ DRBD2_A_CBT_INFO_BOOKMARK_ID,
+ DRBD2_A_CBT_INFO_SINCE,
+ DRBD2_A_CBT_INFO_BITMAP_UUID,
+ DRBD2_A_CBT_INFO_BITMAP_DAGTAG,
+
+ __DRBD2_A_CBT_INFO_MAX,
+ DRBD2_A_CBT_INFO_MAX = (__DRBD2_A_CBT_INFO_MAX - 1)
+};
+
+enum {
+ DRBD2_A_RESOURCE_CONTEXT = 1,
+ DRBD2_A_RESOURCE_INFO,
+ DRBD2_A_RESOURCE_STATISTICS,
+ DRBD2_A_RESOURCE_NEW_NAME,
+ DRBD2_A_RESOURCE_RESOURCE_OPTS,
+
+ __DRBD2_A_RESOURCE_MAX,
+ DRBD2_A_RESOURCE_MAX = (__DRBD2_A_RESOURCE_MAX - 1)
+};
+
+enum {
+ DRBD2_A_DEVICE_CONTEXT = 1,
+ DRBD2_A_DEVICE_INFO,
+ DRBD2_A_DEVICE_STATISTICS,
+ DRBD2_A_DEVICE_DISK_CONF,
+ DRBD2_A_DEVICE_DEVICE_CONF,
+
+ __DRBD2_A_DEVICE_MAX,
+ DRBD2_A_DEVICE_MAX = (__DRBD2_A_DEVICE_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CONNECTION_CONTEXT = 1,
+ DRBD2_A_CONNECTION_INFO,
+ DRBD2_A_CONNECTION_STATISTICS,
+ DRBD2_A_CONNECTION_NET_CONF,
+ DRBD2_A_CONNECTION_PATH,
+
+ __DRBD2_A_CONNECTION_MAX,
+ DRBD2_A_CONNECTION_MAX = (__DRBD2_A_CONNECTION_MAX - 1)
+};
+
+enum {
+ DRBD2_A_PEER_DEVICE_CONTEXT = 1,
+ DRBD2_A_PEER_DEVICE_INFO,
+ DRBD2_A_PEER_DEVICE_STATISTICS,
+ DRBD2_A_PEER_DEVICE_PEER_DEVICE_CONF,
+
+ __DRBD2_A_PEER_DEVICE_MAX,
+ DRBD2_A_PEER_DEVICE_MAX = (__DRBD2_A_PEER_DEVICE_MAX - 1)
+};
+
+enum {
+ DRBD2_A_PATH_CONTEXT = 1,
+ DRBD2_A_PATH_INFO,
+
+ __DRBD2_A_PATH_MAX,
+ DRBD2_A_PATH_MAX = (__DRBD2_A_PATH_MAX - 1)
+};
+
+enum {
+ DRBD2_A_CONTEXT = 1,
+ DRBD2_A_SET_DEFAULTS,
+ DRBD2_A_STATE_RESULT,
+ DRBD2_A_MESSAGE,
+ DRBD2_A_TIMEOUT_TYPE,
+ DRBD2_A_DISK_CONF,
+ DRBD2_A_RESOURCE_OPTS,
+ DRBD2_A_NET_CONF,
+ DRBD2_A_SET_ROLE_PARMS,
+ DRBD2_A_RESIZE_PARMS,
+ DRBD2_A_START_OV_PARMS,
+ DRBD2_A_NEW_CURRENT_UUID_PARMS,
+ DRBD2_A_DISCONNECT_PARMS,
+ DRBD2_A_DETACH_PARMS,
+ DRBD2_A_DEVICE_CONF,
+ DRBD2_A_HELPER,
+ DRBD2_A_INVALIDATE_PARMS,
+ DRBD2_A_PEER_DEVICE_CONF,
+ DRBD2_A_CONNECT_PARMS,
+ DRBD2_A_RENAME_PARMS,
+ DRBD2_A_INVALIDATE_PEER_PARMS,
+ DRBD2_A_SUSPEND_IO_PARMS,
+ DRBD2_A_RESOURCE,
+ DRBD2_A_DEVICE,
+ DRBD2_A_CONNECTION,
+ DRBD2_A_PEER_DEVICE,
+ DRBD2_A_PATH,
+ DRBD2_A_CBT_START_PARMS,
+ DRBD2_A_CBT_INFO,
+
+ __DRBD2_A_MAX,
+ DRBD2_A_MAX = (__DRBD2_A_MAX - 1)
+};
+
+enum {
+ DRBD2_A_STATE_CHANGE_ACTION = 1,
+ DRBD2_A_STATE_CHANGE_MORE,
+ DRBD2_A_STATE_CHANGE_RESOURCE,
+ DRBD2_A_STATE_CHANGE_DEVICE,
+ DRBD2_A_STATE_CHANGE_CONNECTION,
+ DRBD2_A_STATE_CHANGE_PEER_DEVICE,
+ DRBD2_A_STATE_CHANGE_PATH,
+
+ __DRBD2_A_STATE_CHANGE_MAX,
+ DRBD2_A_STATE_CHANGE_MAX = (__DRBD2_A_STATE_CHANGE_MAX - 1)
+};
+
+enum {
+ DRBD2_CMD_RESOURCE_NEW = 1,
+ DRBD2_CMD_RESOURCE_DEL,
+ DRBD2_CMD_RESOURCE_SET,
+ DRBD2_CMD_RESOURCE_RENAME,
+ DRBD2_CMD_RESOURCE_DOWN,
+ DRBD2_CMD_RESOURCE_PRIMARY,
+ DRBD2_CMD_RESOURCE_SECONDARY,
+ DRBD2_CMD_RESOURCE_SUSPEND_IO,
+ DRBD2_CMD_RESOURCE_RESUME_IO,
+ DRBD2_CMD_DEVICE_NEW,
+ DRBD2_CMD_DEVICE_DEL,
+ DRBD2_CMD_DEVICE_ATTACH,
+ DRBD2_CMD_DEVICE_DETACH,
+ DRBD2_CMD_DISK_SET,
+ DRBD2_CMD_DEVICE_RESIZE,
+ DRBD2_CMD_DEVICE_OUTDATE,
+ DRBD2_CMD_DEVICE_INVALIDATE,
+ DRBD2_CMD_DEVICE_NEW_CURRENT_UUID,
+ DRBD2_CMD_CONNECTION_NEW,
+ DRBD2_CMD_CONNECTION_DEL,
+ DRBD2_CMD_CONNECTION_CONNECT,
+ DRBD2_CMD_CONNECTION_DISCONNECT,
+ DRBD2_CMD_CONNECTION_SET,
+ DRBD2_CMD_CONNECTION_FORGET,
+ DRBD2_CMD_PATH_NEW,
+ DRBD2_CMD_PATH_DEL,
+ DRBD2_CMD_PEER_DEVICE_SET,
+ DRBD2_CMD_PEER_DEVICE_INVALIDATE,
+ DRBD2_CMD_PEER_DEVICE_PAUSE_SYNC,
+ DRBD2_CMD_PEER_DEVICE_RESUME_SYNC,
+ DRBD2_CMD_PEER_DEVICE_START_OV,
+ DRBD2_CMD_TIMEOUT_TYPE_GET,
+ DRBD2_CMD_RESOURCE_GET,
+ DRBD2_CMD_DEVICE_GET,
+ DRBD2_CMD_CONNECTION_GET,
+ DRBD2_CMD_PEER_DEVICE_GET,
+ DRBD2_CMD_PATH_GET,
+ DRBD2_CMD_STATE_GET,
+ DRBD2_CMD_STATE_CHANGE_NTF,
+ DRBD2_CMD_HELPER_NTF,
+ DRBD2_CMD_DEVICE_CBT_FLUSH,
+ DRBD2_CMD_DEVICE_CBT_START,
+ DRBD2_CMD_DEVICE_CBT_STOP,
+ DRBD2_CMD_DEVICE_CBT_STATUS,
+
+ __DRBD2_CMD_MAX,
+ DRBD2_CMD_MAX = (__DRBD2_CMD_MAX - 1)
+};
+
+#define DRBD2_MCGRP_EVENTS "events"
+
+#endif /* _UAPI_LINUX_DRBD2_H */
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 19/20] drbd: update monitoring interfaces for multi-peer topology
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (17 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 18/20] drbd: add the "drbd2" genl family Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-10 16:26 ` [PATCH v3 20/20] drbd: remove BROKEN for DRBD Christoph Böhmwalder
19 siblings, 0 replies; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder, Andreas Gruenbacher,
Joel Colledge, Roland Kammerer
Remove the /proc/drbd inline status display; detailed per-peer
monitoring moves to debugfs and netlink.
In DRBD 9, only version and build information is exposed in /proc/drbd.
The "legacy 8.4" compat mechanism restores compatible output to
/proc/drbd if necessary, ensuring userspace compat.
Restructure the debugfs tree from a fixed single-peer layout to a
per-connection hierarchy, reflecting that DRBD 9 resources can have
multiple simultaneous peers.
Request state display now iterates over all peer connections rather
than a single network state field.
Peer request tracking moves from per-device to per-connection lists,
and the transfer log is walked under RCU.
Timing statistics switch from jiffies to ktime for sub-millisecond
precision.
Transport buffer statistics are abstracted through a transport ops
callback instead of reaching into TCP internals.
New debugfs files expose two-phase commit state, transport details,
interval tree contents, activity log histograms, and per-peer resync
progress.
The connection and replication state string tables are split to match
the DRBD 9 split-state model, where transport-level connection state
and replication-level sync state are tracked separately per peer.
Co-developed-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Co-developed-by: Andreas Gruenbacher <agruen@linbit.com>
Signed-off-by: Andreas Gruenbacher <agruen@linbit.com>
Co-developed-by: Joel Colledge <joel.colledge@linbit.com>
Signed-off-by: Joel Colledge <joel.colledge@linbit.com>
Co-developed-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/drbd_debugfs.c | 1758 +++++++++++++++++++++++-----
drivers/block/drbd/drbd_interval.c | 37 +-
drivers/block/drbd/drbd_proc.c | 327 +-----
drivers/block/drbd/drbd_strings.c | 103 +-
4 files changed, 1629 insertions(+), 596 deletions(-)
diff --git a/drivers/block/drbd/drbd_debugfs.c b/drivers/block/drbd/drbd_debugfs.c
index b8e698dbb475..69eb0cc1a36c 100644
--- a/drivers/block/drbd/drbd_debugfs.c
+++ b/drivers/block/drbd/drbd_debugfs.c
@@ -3,19 +3,21 @@
* Copyright (C) 2014, LINBIT HA-Solutions GmbH.
*/
-#define pr_fmt(fmt) "drbd debugfs: " fmt
+#define pr_fmt(fmt) KBUILD_MODNAME " debugfs: " fmt
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/debugfs.h>
#include <linux/seq_file.h>
-#include <linux/stat.h>
#include <linux/jiffies.h>
#include <linux/list.h>
+#include <generated/utsrelease.h>
+#include <linux/drbd_genl.h>
#include "drbd_int.h"
#include "drbd_req.h"
#include "drbd_debugfs.h"
-
+#include "drbd_transport.h"
+#include "drbd_dax_pmem.h"
/**********************************************************************
* Whenever you change the file format, remember to bump the version. *
@@ -23,26 +25,48 @@
static struct dentry *drbd_debugfs_root;
static struct dentry *drbd_debugfs_version;
+static struct dentry *drbd_debugfs_refcounts;
static struct dentry *drbd_debugfs_resources;
static struct dentry *drbd_debugfs_minors;
+static struct dentry *drbd_debugfs_compat;
+
+static void seq_print_node_mask(struct seq_file *m, struct drbd_resource *resource, u64 nodes)
+{
+ struct drbd_connection *connection;
+
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ if (NODE_MASK(connection->peer_node_id) & nodes) {
+ char *name = rcu_dereference((connection)->transport.net_conf)->name;
+
+ seq_printf(m, "%s, ", name);
+ }
+ }
+ rcu_read_unlock();
+ seq_puts(m, "\n");
+}
-static void seq_print_age_or_dash(struct seq_file *m, bool valid, unsigned long dt)
+#ifdef CONFIG_DRBD_TIMING_STATS
+static void seq_print_age_or_dash(struct seq_file *m, bool valid, ktime_t dt)
{
if (valid)
- seq_printf(m, "\t%d", jiffies_to_msecs(dt));
+ seq_printf(m, "\t%d", (int)ktime_to_ms(dt));
else
- seq_printf(m, "\t-");
+ seq_puts(m, "\t-");
}
+#endif
static void __seq_print_rq_state_bit(struct seq_file *m,
bool is_set, char *sep, const char *set_name, const char *unset_name)
{
if (is_set && set_name) {
- seq_putc(m, *sep);
+ if (*sep)
+ seq_putc(m, *sep);
seq_puts(m, set_name);
*sep = '|';
} else if (!is_set && unset_name) {
- seq_putc(m, *sep);
+ if (*sep)
+ seq_putc(m, *sep);
seq_puts(m, unset_name);
*sep = '|';
}
@@ -57,17 +81,21 @@ static void seq_print_rq_state_bit(struct seq_file *m,
/* pretty print enum drbd_req_state_bits req->rq_state */
static void seq_print_request_state(struct seq_file *m, struct drbd_request *req)
{
- unsigned int s = req->rq_state;
+ struct drbd_device *device = req->device;
+ struct drbd_peer_device *peer_device;
+ unsigned int s = req->local_rq_state;
char sep = ' ';
+
seq_printf(m, "\t0x%08x", s);
- seq_printf(m, "\tmaster: %s", req->master_bio ? "pending" : "completed");
+ seq_puts(m, "\tmaster:");
+ __seq_print_rq_state_bit(m, req->master_bio, &sep, "pending", "completed");
+ seq_print_rq_state_bit(m, s & RQ_POSTPONED, &sep, "postponed");
+ seq_print_rq_state_bit(m, s & RQ_COMPLETION_SUSP, &sep, "suspended");
/* RQ_WRITE ignored, already reported */
seq_puts(m, "\tlocal:");
- seq_print_rq_state_bit(m, s & RQ_IN_ACT_LOG, &sep, "in-AL");
- seq_print_rq_state_bit(m, s & RQ_POSTPONED, &sep, "postponed");
- seq_print_rq_state_bit(m, s & RQ_COMPLETION_SUSP, &sep, "suspended");
sep = ' ';
+ seq_print_rq_state_bit(m, s & RQ_IN_ACT_LOG, &sep, "in-AL");
seq_print_rq_state_bit(m, s & RQ_LOCAL_PENDING, &sep, "pending");
seq_print_rq_state_bit(m, s & RQ_LOCAL_COMPLETED, &sep, "completed");
seq_print_rq_state_bit(m, s & RQ_LOCAL_ABORTED, &sep, "aborted");
@@ -75,64 +103,129 @@ static void seq_print_request_state(struct seq_file *m, struct drbd_request *req
if (sep == ' ')
seq_puts(m, " -");
- /* for_each_connection ... */
- seq_printf(m, "\tnet:");
- sep = ' ';
- seq_print_rq_state_bit(m, s & RQ_NET_PENDING, &sep, "pending");
- seq_print_rq_state_bit(m, s & RQ_NET_QUEUED, &sep, "queued");
- seq_print_rq_state_bit(m, s & RQ_NET_SENT, &sep, "sent");
- seq_print_rq_state_bit(m, s & RQ_NET_DONE, &sep, "done");
- seq_print_rq_state_bit(m, s & RQ_NET_SIS, &sep, "sis");
- seq_print_rq_state_bit(m, s & RQ_NET_OK, &sep, "ok");
- if (sep == ' ')
- seq_puts(m, " -");
+ for_each_peer_device(peer_device, device) {
+ s = req->net_rq_state[peer_device->node_id];
+ seq_printf(m, "\tnet[%d]:", peer_device->node_id);
+ sep = ' ';
+ seq_print_rq_state_bit(m, s & RQ_NET_PENDING, &sep, "pending");
+ seq_print_rq_state_bit(m, s & RQ_NET_PENDING_OOS, &sep, "pending-oos");
+ seq_print_rq_state_bit(m, s & RQ_NET_QUEUED, &sep, "queued");
+ seq_print_rq_state_bit(m, s & RQ_NET_READY, &sep, "ready");
+ seq_print_rq_state_bit(m, s & RQ_NET_SENT, &sep, "sent");
+ seq_print_rq_state_bit(m, s & RQ_NET_DONE, &sep, "done");
+ seq_print_rq_state_bit(m, s & RQ_NET_SIS, &sep, "sis");
+ seq_print_rq_state_bit(m, s & RQ_NET_OK, &sep, "ok");
+ if (sep == ' ')
+ seq_puts(m, " -");
+
+ seq_puts(m, " :");
+ sep = ' ';
+ seq_print_rq_state_bit(m, s & RQ_EXP_RECEIVE_ACK, &sep, "B");
+ seq_print_rq_state_bit(m, s & RQ_EXP_WRITE_ACK, &sep, "C");
+ seq_print_rq_state_bit(m, s & RQ_EXP_BARR_ACK, &sep, "barr");
+ if (sep == ' ')
+ seq_puts(m, " -");
+ }
+ seq_putc(m, '\n');
+}
- seq_printf(m, " :");
- sep = ' ';
- seq_print_rq_state_bit(m, s & RQ_EXP_RECEIVE_ACK, &sep, "B");
- seq_print_rq_state_bit(m, s & RQ_EXP_WRITE_ACK, &sep, "C");
- seq_print_rq_state_bit(m, s & RQ_EXP_BARR_ACK, &sep, "barr");
- if (sep == ' ')
- seq_puts(m, " -");
- seq_printf(m, "\n");
+#define memberat(PTR, TYPE, OFFSET) (*(TYPE *)((char *)PTR + OFFSET))
+
+#ifdef CONFIG_DRBD_TIMING_STATS
+static void print_one_age_or_dash(struct seq_file *m, struct drbd_request *req,
+ unsigned int set_mask, unsigned int clear_mask,
+ ktime_t now, size_t offset)
+{
+ struct drbd_device *device = req->device;
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device(peer_device, device) {
+ unsigned int s = req->net_rq_state[peer_device->node_id];
+
+ if (s & set_mask && !(s & clear_mask)) {
+ ktime_t ktime = ktime_sub(now, memberat(req, ktime_t,
+ offset + peer_device->node_id * sizeof(ktime_t)));
+
+ seq_printf(m, "\t[%d]%d", peer_device->node_id, (int)ktime_to_ms(ktime));
+ return;
+ }
+ }
+ seq_puts(m, "\t-");
+}
+#else
+static void seq_print_jif_age_or_dash(struct seq_file *m, bool valid, unsigned long dt)
+{
+ if (valid)
+ seq_printf(m, "\t%d", (int)jiffies_to_msecs(dt));
+ else
+ seq_puts(m, "\t-");
+}
+
+static void print_sent_age_or_dash(struct seq_file *m, struct drbd_request *req, unsigned long jif)
+{
+ struct drbd_device *device = req->device;
+ struct drbd_peer_device *peer_device;
+
+ for_each_peer_device(peer_device, device) {
+ int node_id = peer_device->node_id;
+
+ if (req->net_rq_state[node_id] & RQ_NET_SENT) {
+ seq_printf(m, "\t[%d]%d", node_id,
+ (int)jiffies_to_msecs(jif - req->pre_send_jif[node_id]));
+ return;
+ }
+ }
+ seq_puts(m, "\t-");
}
+#endif
-static void seq_print_one_request(struct seq_file *m, struct drbd_request *req, unsigned long now)
+static void seq_print_one_request(struct seq_file *m, struct drbd_request *req, ktime_t now, unsigned long jif)
{
/* change anything here, fixup header below! */
- unsigned int s = req->rq_state;
+ unsigned int s = req->local_rq_state;
+ unsigned long flags;
+ spin_lock_irqsave(&req->rq_lock, flags);
#define RQ_HDR_1 "epoch\tsector\tsize\trw"
seq_printf(m, "0x%x\t%llu\t%u\t%s",
req->epoch,
(unsigned long long)req->i.sector, req->i.size >> 9,
(s & RQ_WRITE) ? "W" : "R");
+#ifdef CONFIG_DRBD_TIMING_STATS
#define RQ_HDR_2 "\tstart\tin AL\tsubmit"
- seq_printf(m, "\t%d", jiffies_to_msecs(now - req->start_jif));
- seq_print_age_or_dash(m, s & RQ_IN_ACT_LOG, now - req->in_actlog_jif);
- seq_print_age_or_dash(m, s & RQ_LOCAL_PENDING, now - req->pre_submit_jif);
+ seq_printf(m, "\t%d", (int)ktime_to_ms(ktime_sub(now, req->start_kt)));
+ seq_print_age_or_dash(m, s & RQ_IN_ACT_LOG, ktime_sub(now, req->in_actlog_kt));
+ seq_print_age_or_dash(m, s & RQ_LOCAL_PENDING, ktime_sub(now, req->pre_submit_kt));
#define RQ_HDR_3 "\tsent\tacked\tdone"
- seq_print_age_or_dash(m, s & RQ_NET_SENT, now - req->pre_send_jif);
- seq_print_age_or_dash(m, (s & RQ_NET_SENT) && !(s & RQ_NET_PENDING), now - req->acked_jif);
- seq_print_age_or_dash(m, s & RQ_NET_DONE, now - req->net_done_jif);
-
+ print_one_age_or_dash(m, req, RQ_NET_SENT, 0, now, offsetof(typeof(*req), pre_send_kt));
+ print_one_age_or_dash(m, req, RQ_NET_SENT, RQ_NET_PENDING, now, offsetof(typeof(*req), acked_kt));
+ print_one_age_or_dash(m, req, RQ_NET_DONE, 0, now, offsetof(typeof(*req), net_done_kt));
+#else
+#define RQ_HDR_2 "\tstart\tsubmit"
+ seq_printf(m, "\t%d", (int)jiffies_to_msecs(jif - req->start_jif));
+ seq_print_jif_age_or_dash(m, s & RQ_LOCAL_PENDING, jif - req->pre_submit_jif);
+
+#define RQ_HDR_3 "\tsent"
+ print_sent_age_or_dash(m, req, jif);
+#endif
#define RQ_HDR_4 "\tstate\n"
seq_print_request_state(m, req);
+ spin_unlock_irqrestore(&req->rq_lock, flags);
}
#define RQ_HDR RQ_HDR_1 RQ_HDR_2 RQ_HDR_3 RQ_HDR_4
-static void seq_print_minor_vnr_req(struct seq_file *m, struct drbd_request *req, unsigned long now)
+static void seq_print_minor_vnr_req(struct seq_file *m, struct drbd_request *req, ktime_t now, unsigned long jif)
{
seq_printf(m, "%u\t%u\t", req->device->minor, req->device->vnr);
- seq_print_one_request(m, req, now);
+ seq_print_one_request(m, req, now, jif);
}
-static void seq_print_resource_pending_meta_io(struct seq_file *m, struct drbd_resource *resource, unsigned long now)
+static void seq_print_resource_pending_meta_io(struct seq_file *m, struct drbd_resource *resource, unsigned long jif)
{
struct drbd_device *device;
- unsigned int i;
+ int i;
seq_puts(m, "minor\tvnr\tstart\tsubmit\tintent\n");
rcu_read_lock();
@@ -146,45 +239,47 @@ static void seq_print_resource_pending_meta_io(struct seq_file *m, struct drbd_r
if (atomic_read(&tmp.in_use)) {
seq_printf(m, "%u\t%u\t%d\t",
device->minor, device->vnr,
- jiffies_to_msecs(now - tmp.start_jif));
+ jiffies_to_msecs(jif - tmp.start_jif));
if (time_before(tmp.submit_jif, tmp.start_jif))
seq_puts(m, "-\t");
else
- seq_printf(m, "%d\t", jiffies_to_msecs(now - tmp.submit_jif));
+ seq_printf(m, "%d\t", jiffies_to_msecs(jif - tmp.submit_jif));
seq_printf(m, "%s\n", tmp.current_use);
}
}
rcu_read_unlock();
}
-static void seq_print_waiting_for_AL(struct seq_file *m, struct drbd_resource *resource, unsigned long now)
+static void seq_print_waiting_for_AL(struct seq_file *m, struct drbd_resource *resource, ktime_t now, unsigned long jif)
{
struct drbd_device *device;
- unsigned int i;
+ int i;
seq_puts(m, "minor\tvnr\tage\t#waiting\n");
rcu_read_lock();
idr_for_each_entry(&resource->devices, device, i) {
- unsigned long jif;
struct drbd_request *req;
int n = atomic_read(&device->ap_actlog_cnt);
+
if (n) {
- spin_lock_irq(&device->resource->req_lock);
+ spin_lock_irq(&device->pending_completion_lock);
req = list_first_entry_or_null(&device->pending_master_completion[1],
struct drbd_request, req_pending_master_completion);
/* if the oldest request does not wait for the activity log
* it is not interesting for us here */
- if (req && !(req->rq_state & RQ_IN_ACT_LOG))
- jif = req->start_jif;
- else
+ if (req && (req->local_rq_state & RQ_IN_ACT_LOG))
req = NULL;
- spin_unlock_irq(&device->resource->req_lock);
+ spin_unlock_irq(&device->pending_completion_lock);
}
if (n) {
seq_printf(m, "%u\t%u\t", device->minor, device->vnr);
- if (req)
- seq_printf(m, "%u\t", jiffies_to_msecs(now - jif));
- else
+ if (req) {
+#ifdef CONFIG_DRBD_TIMING_STATS
+ seq_printf(m, "%d\t", (int)ktime_to_ms(ktime_sub(now, req->start_kt)));
+#else
+ seq_printf(m, "%d\t", (int)jiffies_to_msecs(jif - req->start_jif));
+#endif
+ } else
seq_puts(m, "-\t");
seq_printf(m, "%u\n", n);
}
@@ -192,13 +287,14 @@ static void seq_print_waiting_for_AL(struct seq_file *m, struct drbd_resource *r
rcu_read_unlock();
}
-static void seq_print_device_bitmap_io(struct seq_file *m, struct drbd_device *device, unsigned long now)
+static void seq_print_device_bitmap_io(struct seq_file *m, struct drbd_device *device, unsigned long jif)
{
struct drbd_bm_aio_ctx *ctx;
unsigned long start_jif;
unsigned int in_flight;
unsigned int flags;
- spin_lock_irq(&device->resource->req_lock);
+
+ spin_lock_irq(&device->pending_bmio_lock);
ctx = list_first_entry_or_null(&device->pending_bitmap_io, struct drbd_bm_aio_ctx, list);
if (ctx && ctx->done)
ctx = NULL;
@@ -207,25 +303,25 @@ static void seq_print_device_bitmap_io(struct seq_file *m, struct drbd_device *d
in_flight = atomic_read(&ctx->in_flight);
flags = ctx->flags;
}
- spin_unlock_irq(&device->resource->req_lock);
+ spin_unlock_irq(&device->pending_bmio_lock);
if (ctx) {
seq_printf(m, "%u\t%u\t%c\t%u\t%u\n",
device->minor, device->vnr,
(flags & BM_AIO_READ) ? 'R' : 'W',
- jiffies_to_msecs(now - start_jif),
+ jiffies_to_msecs(jif - start_jif),
in_flight);
}
}
-static void seq_print_resource_pending_bitmap_io(struct seq_file *m, struct drbd_resource *resource, unsigned long now)
+static void seq_print_resource_pending_bitmap_io(struct seq_file *m, struct drbd_resource *resource, unsigned long jif)
{
struct drbd_device *device;
- unsigned int i;
+ int i;
seq_puts(m, "minor\tvnr\trw\tage\t#in-flight\n");
rcu_read_lock();
idr_for_each_entry(&resource->devices, device, i) {
- seq_print_device_bitmap_io(m, device, now);
+ seq_print_device_bitmap_io(m, device, jif);
}
rcu_read_unlock();
}
@@ -234,104 +330,201 @@ static void seq_print_resource_pending_bitmap_io(struct seq_file *m, struct drbd
static void seq_print_peer_request_flags(struct seq_file *m, struct drbd_peer_request *peer_req)
{
unsigned long f = peer_req->flags;
- char sep = ' ';
-
- __seq_print_rq_state_bit(m, f & EE_SUBMITTED, &sep, "submitted", "preparing");
- __seq_print_rq_state_bit(m, f & EE_APPLICATION, &sep, "application", "internal");
- seq_print_rq_state_bit(m, f & EE_CALL_AL_COMPLETE_IO, &sep, "in-AL");
+ char sep = 0;
+
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &peer_req->i.flags),
+ &sep, "submit-conflict-queued");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_SUBMITTED, &peer_req->i.flags),
+ &sep, "submitted");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_CONFLICT, &peer_req->i.flags),
+ &sep, "conflict");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_SENT, &peer_req->i.flags),
+ &sep, "sent");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags),
+ &sep, "ready-to-send");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_RECEIVED, &peer_req->i.flags),
+ &sep, "received");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_BACKING_COMPLETED, &peer_req->i.flags),
+ &sep, "backing-completed");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_COMPLETED, &peer_req->i.flags),
+ &sep, "completed");
+ seq_print_rq_state_bit(m, f & EE_IS_BARRIER, &sep, "barr");
seq_print_rq_state_bit(m, f & EE_SEND_WRITE_ACK, &sep, "C");
seq_print_rq_state_bit(m, f & EE_MAY_SET_IN_SYNC, &sep, "set-in-sync");
+ seq_print_rq_state_bit(m, f & EE_SET_OUT_OF_SYNC, &sep, "set-out-of-sync");
+ seq_print_rq_state_bit(m, peer_req->i.type == INTERVAL_PEER_WRITE && !(f & EE_IN_ACTLOG), &sep, "blocked-on-al");
seq_print_rq_state_bit(m, f & EE_TRIM, &sep, "trim");
seq_print_rq_state_bit(m, f & EE_ZEROOUT, &sep, "zero-out");
seq_print_rq_state_bit(m, f & EE_WRITE_SAME, &sep, "write-same");
seq_putc(m, '\n');
}
-static void seq_print_peer_request(struct seq_file *m,
- struct drbd_device *device, struct list_head *lh,
- unsigned long now)
+enum drbd_peer_request_state {
+ PRS_NEW,
+ PRS_READY_TO_SEND,
+ PRS_SUBMITTED,
+ PRS_LAST,
+};
+
+static enum drbd_peer_request_state drbd_get_peer_request_state(struct drbd_peer_request *peer_req)
{
- bool reported_preparing = false;
+ if (test_bit(INTERVAL_SUBMITTED, &peer_req->i.flags))
+ return PRS_SUBMITTED;
+
+ if (test_bit(INTERVAL_READY_TO_SEND, &peer_req->i.flags))
+ return PRS_READY_TO_SEND;
+
+ return PRS_NEW;
+}
+
+static void seq_print_peer_request_one(struct seq_file *m,
+ struct drbd_peer_request *peer_req,
+ const char *list_name, unsigned long jif)
+{
+ struct drbd_peer_device *peer_device = peer_req->peer_device;
+ struct drbd_device *device = peer_device ? peer_device->device : NULL;
+
+ seq_printf(m, "%s\t", list_name);
+
+ if (device)
+ seq_printf(m, "%u\t%u\t", device->minor, device->vnr);
+
+ seq_printf(m, "%llu\t%u\t%s\t%u\t",
+ (unsigned long long)peer_req->i.sector, peer_req->i.size >> 9,
+ drbd_interval_type_str(&peer_req->i),
+ jiffies_to_msecs(jif - peer_req->submit_jif));
+ seq_print_peer_request_flags(m, peer_req);
+}
+
+static void seq_print_peer_request_w(struct seq_file *m,
+ struct drbd_connection *connection, struct list_head *lh,
+ const char *list_name, unsigned long jif)
+{
+ int count[PRS_LAST] = {0};
struct drbd_peer_request *peer_req;
+
list_for_each_entry(peer_req, lh, w.list) {
- if (reported_preparing && !(peer_req->flags & EE_SUBMITTED))
- continue;
+ enum drbd_peer_request_state state = drbd_get_peer_request_state(peer_req);
- if (device)
- seq_printf(m, "%u\t%u\t", device->minor, device->vnr);
+ count[state]++;
+ if (count[state] <= 16)
+ seq_print_peer_request_one(m, peer_req, list_name, jif);
+ }
+}
- seq_printf(m, "%llu\t%u\t%c\t%u\t",
- (unsigned long long)peer_req->i.sector, peer_req->i.size >> 9,
- (peer_req->flags & EE_WRITE) ? 'W' : 'R',
- jiffies_to_msecs(now - peer_req->submit_jif));
- seq_print_peer_request_flags(m, peer_req);
- if (peer_req->flags & EE_SUBMITTED)
- break;
- else
- reported_preparing = true;
+static void seq_print_peer_request(struct seq_file *m,
+ struct drbd_connection *connection, struct list_head *lh,
+ const char *list_name, unsigned long jif)
+{
+ int count[2] = {0};
+ struct drbd_peer_request *peer_req;
+
+ /* Cap completed and not yet completed entries separately, so that
+ * completed ones waiting for a peer ack never hide a stuck one.
+ */
+ list_for_each_entry(peer_req, lh, recv_order) {
+ bool completed = test_bit(INTERVAL_COMPLETED, &peer_req->i.flags);
+
+ count[completed]++;
+ if (count[completed] <= 16)
+ seq_print_peer_request_one(m, peer_req, list_name, jif);
}
}
-static void seq_print_device_peer_requests(struct seq_file *m,
- struct drbd_device *device, unsigned long now)
+static void seq_print_connection_peer_requests(struct seq_file *m,
+ struct drbd_connection *connection, unsigned long jif)
+{
+ struct drbd_peer_device *peer_device;
+ int i;
+
+ seq_puts(m, "list\t\tminor\tvnr\tsector\tsize\ttype\t\tage\tflags\n");
+ spin_lock_irq(&connection->peer_reqs_lock);
+ seq_print_peer_request_w(m, connection, &connection->done_ee, "done\t", jif);
+ seq_print_peer_request_w(m, connection, &connection->dagtag_wait_ee, "dagtag_wait", jif);
+ seq_print_peer_request(m, connection, &connection->peer_requests, "peer_requests", jif);
+ seq_print_peer_request(m, connection, &connection->peer_reads, "peer_reads", jif);
+ idr_for_each_entry(&connection->peer_devices, peer_device, i)
+ seq_print_peer_request(m, connection, &peer_device->resync_requests,
+ "resync_requests", jif);
+ spin_unlock_irq(&connection->peer_reqs_lock);
+}
+
+static void seq_print_device_peer_flushes(struct seq_file *m,
+ struct drbd_device *device, unsigned long jif)
{
- seq_puts(m, "minor\tvnr\tsector\tsize\trw\tage\tflags\n");
- spin_lock_irq(&device->resource->req_lock);
- seq_print_peer_request(m, device, &device->active_ee, now);
- seq_print_peer_request(m, device, &device->read_ee, now);
- seq_print_peer_request(m, device, &device->sync_ee, now);
- spin_unlock_irq(&device->resource->req_lock);
if (test_bit(FLUSH_PENDING, &device->flags)) {
seq_printf(m, "%u\t%u\t-\t-\tF\t%u\tflush\n",
device->minor, device->vnr,
- jiffies_to_msecs(now - device->flush_jif));
+ jiffies_to_msecs(jif - device->flush_jif));
}
}
static void seq_print_resource_pending_peer_requests(struct seq_file *m,
- struct drbd_resource *resource, unsigned long now)
+ struct drbd_resource *resource, unsigned long jif)
{
+ struct drbd_connection *connection;
struct drbd_device *device;
- unsigned int i;
+ int i;
rcu_read_lock();
+
+ for_each_connection_rcu(connection, resource) {
+ seq_printf(m, "oldest peer requests (peer: %s)\n",
+ rcu_dereference(connection->transport.net_conf)->name);
+ seq_print_connection_peer_requests(m, connection, jif);
+ seq_putc(m, '\n');
+ }
+
+ seq_puts(m, "flushes\n");
idr_for_each_entry(&resource->devices, device, i) {
- seq_print_device_peer_requests(m, device, now);
+ seq_print_device_peer_flushes(m, device, jif);
}
+ seq_putc(m, '\n');
+
rcu_read_unlock();
}
static void seq_print_resource_transfer_log_summary(struct seq_file *m,
struct drbd_resource *resource,
- struct drbd_connection *connection,
- unsigned long now)
+ ktime_t now, unsigned long jif)
{
struct drbd_request *req;
unsigned int count = 0;
unsigned int show_state = 0;
seq_puts(m, "n\tdevice\tvnr\t" RQ_HDR);
- spin_lock_irq(&resource->req_lock);
- list_for_each_entry(req, &connection->transfer_log, tl_requests) {
+ rcu_read_lock();
+ list_for_each_entry_rcu(req, &resource->transfer_log, tl_requests) {
+ struct drbd_device *device = req->device;
+ struct drbd_peer_device *peer_device;
unsigned int tmp = 0;
unsigned int s;
- ++count;
- /* don't disable irq "forever" */
- if (!(count & 0x1ff)) {
- struct drbd_request *req_next;
- kref_get(&req->kref);
- spin_unlock_irq(&resource->req_lock);
+ /* don't disable preemption "forever" */
+ if ((count & 0x1ff) == 0x1ff) {
+ struct list_head *next_hdr;
+ /* Only get if the request hasn't already been removed from transfer_log. */
+ if (!refcount_inc_not_zero(&req->oos_send_ref))
+ continue;
+ rcu_read_unlock();
cond_resched();
- spin_lock_irq(&resource->req_lock);
- req_next = list_next_entry(req, tl_requests);
- if (kref_put(&req->kref, drbd_req_destroy))
- req = req_next;
- if (&req->tl_requests == &connection->transfer_log)
- break;
+ rcu_read_lock();
+ next_hdr = rcu_dereference(list_next_rcu(&req->tl_requests));
+ read_lock_irq(&resource->state_rwlock);
+ drbd_put_ref_tl_walk(req, 0, 1);
+ read_unlock_irq(&resource->state_rwlock);
+ if (!refcount_read(&req->done_ref)) {
+ if (next_hdr == &resource->transfer_log)
+ break;
+ req = list_entry_rcu(next_hdr,
+ struct drbd_request,
+ tl_requests);
+ }
}
+ ++count;
- s = req->rq_state;
+ spin_lock_irq(&req->rq_lock);
+ s = req->local_rq_state;
/* This is meant to summarize timing issues, to be able to tell
* local disk problems from network problems.
@@ -341,40 +534,43 @@ static void seq_print_resource_transfer_log_summary(struct seq_file *m,
tmp |= 1;
if ((s & RQ_LOCAL_MASK) && (s & RQ_LOCAL_PENDING))
tmp |= 2;
- if (s & RQ_NET_MASK) {
- if (!(s & RQ_NET_SENT))
- tmp |= 4;
- if (s & RQ_NET_PENDING)
- tmp |= 8;
- if (!(s & RQ_NET_DONE))
- tmp |= 16;
+
+ for_each_peer_device_rcu(peer_device, device) {
+ s = READ_ONCE(req->net_rq_state[peer_device->node_id]);
+ if (s & RQ_NET_MASK) {
+ if (!(s & RQ_NET_SENT))
+ tmp |= 4;
+ if (s & RQ_NET_PENDING)
+ tmp |= 8;
+ if (!(s & RQ_NET_DONE))
+ tmp |= 16;
+ }
}
+ spin_unlock_irq(&req->rq_lock);
+
if ((tmp & show_state) == tmp)
continue;
show_state |= tmp;
seq_printf(m, "%u\t", count);
- seq_print_minor_vnr_req(m, req, now);
+ seq_print_minor_vnr_req(m, req, now, jif);
if (show_state == 0x1f)
break;
}
- spin_unlock_irq(&resource->req_lock);
+ rcu_read_unlock();
+ seq_printf(m, "%u total\n", count);
}
-/* TODO: transfer_log and friends should be moved to resource */
-static int in_flight_summary_show(struct seq_file *m, void *pos)
+static int resource_in_flight_summary_show(struct seq_file *m, void *pos)
{
struct drbd_resource *resource = m->private;
struct drbd_connection *connection;
+ struct drbd_transport *transport;
+ struct drbd_transport_stats transport_stats;
+ ktime_t now = ktime_get();
unsigned long jif = jiffies;
- connection = first_connection(resource);
- /* This does not happen, actually.
- * But be robust and prepare for future code changes. */
- if (!connection || !kref_get_unless_zero(&connection->kref))
- return -ESTALE;
-
/* BUMP me if you change the file format/content/presentation */
- seq_printf(m, "v: %u\n\n", 0);
+ seq_printf(m, "v: %u\n\n", 3);
seq_puts(m, "oldest bitmap IO\n");
seq_print_resource_pending_bitmap_io(m, resource, jif);
@@ -384,43 +580,132 @@ static int in_flight_summary_show(struct seq_file *m, void *pos)
seq_print_resource_pending_meta_io(m, resource, jif);
seq_putc(m, '\n');
- seq_puts(m, "socket buffer stats\n");
- /* for each connection ... once we have more than one */
+ seq_puts(m, "transport buffer stats\n");
+ seq_puts(m, "peer\ttransport class\tunread receive buffer\tunacked send buffer\n");
rcu_read_lock();
- if (connection->data.socket) {
- /* open coded SIOCINQ, the "relevant" part */
- struct tcp_sock *tp = tcp_sk(connection->data.socket->sk);
- int answ = tp->rcv_nxt - tp->copied_seq;
- seq_printf(m, "unread receive buffer: %u Byte\n", answ);
- /* open coded SIOCOUTQ, the "relevant" part */
- answ = tp->write_seq - tp->snd_una;
- seq_printf(m, "unacked send buffer: %u Byte\n", answ);
+ for_each_connection_rcu(connection, resource) {
+ char *name;
+
+ transport = &connection->transport;
+ name = rcu_dereference(transport->net_conf)->name;
+ seq_printf(m, "%s\t%s\t", name, transport->class->name);
+
+ if (transport->class->ops.stream_ok(transport, DATA_STREAM)) {
+ transport->class->ops.stats(transport, &transport_stats);
+ seq_printf(m, "%u\t%u\n",
+ transport_stats.unread_received,
+ transport_stats.unacked_send);
+ } else {
+ seq_puts(m, "-\t-\n");
+ }
}
rcu_read_unlock();
seq_putc(m, '\n');
- seq_puts(m, "oldest peer requests\n");
seq_print_resource_pending_peer_requests(m, resource, jif);
- seq_putc(m, '\n');
seq_puts(m, "application requests waiting for activity log\n");
- seq_print_waiting_for_AL(m, resource, jif);
+ seq_print_waiting_for_AL(m, resource, now, jif);
seq_putc(m, '\n');
seq_puts(m, "oldest application requests\n");
- seq_print_resource_transfer_log_summary(m, resource, connection, jif);
+ seq_print_resource_transfer_log_summary(m, resource, now, jif);
seq_putc(m, '\n');
jif = jiffies - jif;
if (jif)
seq_printf(m, "generated in %d ms\n", jiffies_to_msecs(jif));
- kref_put(&connection->kref, drbd_destroy_connection);
+ return 0;
+}
+
+static int resource_state_twopc_show(struct seq_file *m, void *pos)
+{
+ struct drbd_resource *resource = m->private;
+ struct twopc_reply twopc;
+ bool active = false;
+ unsigned long jif;
+
+ read_lock_irq(&resource->state_rwlock);
+ if (resource->remote_state_change) {
+ twopc = resource->twopc_reply;
+ active = true;
+ }
+ read_unlock_irq(&resource->state_rwlock);
+
+ seq_printf(m, "v: %u\n\n", 1);
+ if (active) {
+ struct drbd_connection *connection;
+
+ seq_printf(m,
+ "Executing tid: %u\n"
+ " initiator_node_id: %d\n"
+ " target_node_id: %d\n",
+ twopc.tid, twopc.initiator_node_id,
+ twopc.target_node_id);
+
+ if (twopc.initiator_node_id != resource->res_opts.node_id) {
+ seq_puts(m, " parent node mask: ");
+ seq_print_node_mask(m, resource, resource->twopc_parent_nodes);
+
+ if (resource->twopc_prepare_reply_cmd)
+ seq_printf(m,
+ " Reply sent: %s\n",
+ resource->twopc_prepare_reply_cmd == P_TWOPC_YES ? "yes" :
+ resource->twopc_prepare_reply_cmd == P_TWOPC_NO ? "no" :
+ resource->twopc_prepare_reply_cmd == P_TWOPC_RETRY ? "retry" :
+ "else!?!");
+ }
+
+ seq_puts(m, " received replies: ");
+ rcu_read_lock();
+ for_each_connection_rcu(connection, resource) {
+ char *name = rcu_dereference((connection)->transport.net_conf)->name;
+
+ if (!test_bit(TWOPC_PREPARED, &connection->flags))
+ /* seq_printf(m, "%s n.p., ", name) * print nothing! */;
+ else if (test_bit(TWOPC_NO, &connection->flags))
+ seq_printf(m, "%s no, ", name);
+ else if (test_bit(TWOPC_RETRY, &connection->flags))
+ seq_printf(m, "%s ret, ", name);
+ else if (test_bit(TWOPC_YES, &connection->flags))
+ seq_printf(m, "%s yes, ", name);
+ else
+ seq_printf(m, "%s ___, ", name);
+ }
+ rcu_read_unlock();
+ seq_puts(m, "\n");
+ if (twopc.initiator_node_id != resource->res_opts.node_id) {
+ /* The timer is only relevant for twopcs initiated by other nodes */
+ jif = resource->twopc_timer.expires - jiffies;
+ seq_printf(m, " timer expires in: %d ms\n", jiffies_to_msecs(jif));
+ }
+ } else {
+ seq_puts(m, "No ongoing two phase state transaction\n");
+ }
+
+ return 0;
+}
+
+static int resource_worker_pid_show(struct seq_file *m, void *pos)
+{
+ struct drbd_resource *resource = m->private;
+
+ if (resource->worker.task)
+ seq_printf(m, "%d\n", resource->worker.task->pid);
+ return 0;
+}
+
+static int resource_members_show(struct seq_file *m, void *pos)
+{
+ struct drbd_resource *resource = m->private;
+
+ seq_printf(m, "0x%016llX\n", resource->members);
return 0;
}
/* make sure at *open* time that the respective object won't go away. */
static int drbd_single_open(struct file *file, int (*show)(struct seq_file *, void *),
- void *data, struct kref *kref,
+ void *data, struct kref *kref,
void (*release)(struct kref *))
{
struct dentry *parent;
@@ -429,6 +714,9 @@ static int drbd_single_open(struct file *file, int (*show)(struct seq_file *, vo
/* Are we still linked,
* or has debugfs_remove() already been called? */
parent = file->f_path.dentry->d_parent;
+ /* not sure if this can happen: */
+ if (!parent || !parent->d_inode)
+ goto out;
/* serialize with d_delete() */
inode_lock(d_inode(parent));
/* Make sure the object is still alive */
@@ -441,31 +729,56 @@ static int drbd_single_open(struct file *file, int (*show)(struct seq_file *, vo
if (ret)
kref_put(kref, release);
}
+out:
return ret;
}
-static int in_flight_summary_open(struct inode *inode, struct file *file)
+static int resource_attr_release(struct inode *inode, struct file *file)
{
struct drbd_resource *resource = inode->i_private;
- return drbd_single_open(file, in_flight_summary_show, resource,
- &resource->kref, drbd_destroy_resource);
-}
-static int in_flight_summary_release(struct inode *inode, struct file *file)
-{
- struct drbd_resource *resource = inode->i_private;
kref_put(&resource->kref, drbd_destroy_resource);
return single_release(inode, file);
}
-static const struct file_operations in_flight_summary_fops = {
- .owner = THIS_MODULE,
- .open = in_flight_summary_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = in_flight_summary_release,
+#define drbd_debugfs_resource_attr(name) \
+static int resource_ ## name ## _open(struct inode *inode, struct file *file) \
+{ \
+ struct drbd_resource *resource = inode->i_private; \
+ return drbd_single_open(file, resource_ ## name ## _show, resource, \
+ &resource->kref, drbd_destroy_resource); \
+} \
+static const struct file_operations resource_ ## name ## _fops = { \
+ .owner = THIS_MODULE, \
+ .open = resource_ ## name ## _open, \
+ .read = seq_read, \
+ .llseek = seq_lseek, \
+ .release = resource_attr_release, \
};
+drbd_debugfs_resource_attr(in_flight_summary)
+drbd_debugfs_resource_attr(state_twopc)
+drbd_debugfs_resource_attr(worker_pid)
+drbd_debugfs_resource_attr(members)
+
+#define drbd_dcf(top, obj, attr, perm) do { \
+ dentry = debugfs_create_file(#attr, perm, \
+ top, obj, &obj ## _ ## attr ## _fops); \
+ top ## _ ## attr = dentry; \
+ } while (0)
+
+#define res_dcf(attr) \
+ drbd_dcf(resource->debugfs_res, resource, attr, 0400)
+
+#define conn_dcf(attr) \
+ drbd_dcf(connection->debugfs_conn, connection, attr, 0400)
+
+#define vol_dcf(attr) \
+ drbd_dcf(device->debugfs_vol, device, attr, 0400)
+
+#define peer_dev_dcf(attr) \
+ drbd_dcf(peer_device->debugfs_peer_dev, peer_device, attr, 0400)
+
void drbd_debugfs_resource_add(struct drbd_resource *resource)
{
struct dentry *dentry;
@@ -479,10 +792,11 @@ void drbd_debugfs_resource_add(struct drbd_resource *resource)
dentry = debugfs_create_dir("connections", resource->debugfs_res);
resource->debugfs_res_connections = dentry;
- dentry = debugfs_create_file("in_flight_summary", 0440,
- resource->debugfs_res, resource,
- &in_flight_summary_fops);
- resource->debugfs_res_in_flight_summary = dentry;
+ /* debugfs create file */
+ res_dcf(in_flight_summary);
+ res_dcf(state_twopc);
+ res_dcf(worker_pid);
+ res_dcf(members);
}
static void drbd_debugfs_remove(struct dentry **dp)
@@ -493,16 +807,35 @@ static void drbd_debugfs_remove(struct dentry **dp)
void drbd_debugfs_resource_cleanup(struct drbd_resource *resource)
{
+ /* Older kernels have a broken implementation of
+ * debugfs_remove_recursive (prior to upstream commit 776164c1f)
+ * That unfortunately includes a number of "enterprise" kernels.
+ * Even older kernels do not even have the _recursive() helper at all.
+ * For now, remember all debugfs nodes we created,
+ * and call debugfs_remove on all of them separately.
+ */
/* it is ok to call debugfs_remove(NULL) */
+ drbd_debugfs_remove(&resource->debugfs_res_members);
+ drbd_debugfs_remove(&resource->debugfs_res_worker_pid);
+ drbd_debugfs_remove(&resource->debugfs_res_state_twopc);
drbd_debugfs_remove(&resource->debugfs_res_in_flight_summary);
drbd_debugfs_remove(&resource->debugfs_res_connections);
drbd_debugfs_remove(&resource->debugfs_res_volumes);
drbd_debugfs_remove(&resource->debugfs_res);
}
+void drbd_debugfs_resource_rename(struct drbd_resource *resource, const char *new_name)
+{
+ int err;
+
+ err = debugfs_change_name(resource->debugfs_res, "%s", new_name);
+ if (err)
+ drbd_err(resource, "failed to rename debugfs entry for resource\n");
+}
+
static void seq_print_one_timing_detail(struct seq_file *m,
const struct drbd_thread_timing_details *tdp,
- unsigned long now)
+ unsigned long jif)
{
struct drbd_thread_timing_details td;
/* No locking...
@@ -514,14 +847,14 @@ static void seq_print_one_timing_detail(struct seq_file *m,
return;
seq_printf(m, "%u\t%d\t%s:%u\t%ps\n",
td.cb_nr,
- jiffies_to_msecs(now - td.start_jif),
+ jiffies_to_msecs(jif - td.start_jif),
td.caller_fn, td.line,
td.cb_addr);
}
static void seq_print_timing_details(struct seq_file *m,
const char *title,
- unsigned int cb_nr, struct drbd_thread_timing_details *tdp, unsigned long now)
+ unsigned int cb_nr, struct drbd_thread_timing_details *tdp, unsigned long jif)
{
unsigned int start_idx;
unsigned int i;
@@ -533,135 +866,315 @@ static void seq_print_timing_details(struct seq_file *m,
*/
start_idx = cb_nr % DRBD_THREAD_DETAILS_HIST;
for (i = start_idx; i < DRBD_THREAD_DETAILS_HIST; i++)
- seq_print_one_timing_detail(m, tdp+i, now);
+ seq_print_one_timing_detail(m, tdp+i, jif);
for (i = 0; i < start_idx; i++)
- seq_print_one_timing_detail(m, tdp+i, now);
+ seq_print_one_timing_detail(m, tdp+i, jif);
}
-static int callback_history_show(struct seq_file *m, void *ignored)
+static int connection_callback_history_show(struct seq_file *m, void *ignored)
{
struct drbd_connection *connection = m->private;
+ struct drbd_resource *resource = connection->resource;
unsigned long jif = jiffies;
/* BUMP me if you change the file format/content/presentation */
seq_printf(m, "v: %u\n\n", 0);
seq_puts(m, "n\tage\tcallsite\tfn\n");
- seq_print_timing_details(m, "worker", connection->w_cb_nr, connection->w_timing_details, jif);
+ seq_print_timing_details(m, "sender", connection->s_cb_nr, connection->s_timing_details, jif);
seq_print_timing_details(m, "receiver", connection->r_cb_nr, connection->r_timing_details, jif);
+ seq_print_timing_details(m, "worker", resource->w_cb_nr, resource->w_timing_details, jif);
return 0;
}
-static int callback_history_open(struct inode *inode, struct file *file)
-{
- struct drbd_connection *connection = inode->i_private;
- return drbd_single_open(file, callback_history_show, connection,
- &connection->kref, drbd_destroy_connection);
-}
-
-static int callback_history_release(struct inode *inode, struct file *file)
-{
- struct drbd_connection *connection = inode->i_private;
- kref_put(&connection->kref, drbd_destroy_connection);
- return single_release(inode, file);
-}
-
-static const struct file_operations connection_callback_history_fops = {
- .owner = THIS_MODULE,
- .open = callback_history_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = callback_history_release,
-};
-
static int connection_oldest_requests_show(struct seq_file *m, void *ignored)
{
struct drbd_connection *connection = m->private;
- unsigned long now = jiffies;
+ ktime_t now = ktime_get();
+ unsigned long jif = jiffies;
struct drbd_request *r1, *r2;
/* BUMP me if you change the file format/content/presentation */
- seq_printf(m, "v: %u\n\n", 0);
+ seq_printf(m, "v: %u\n\n", 2);
- spin_lock_irq(&connection->resource->req_lock);
- r1 = connection->req_next;
+ rcu_read_lock();
+ r1 = READ_ONCE(connection->todo.req_next);
if (r1)
- seq_print_minor_vnr_req(m, r1, now);
- r2 = connection->req_ack_pending;
+ seq_print_minor_vnr_req(m, r1, now, jif);
+ r2 = READ_ONCE(connection->req_next_ready);
if (r2 && r2 != r1) {
r1 = r2;
- seq_print_minor_vnr_req(m, r1, now);
+ seq_print_minor_vnr_req(m, r1, now, jif);
}
- r2 = connection->req_not_net_done;
+ r2 = READ_ONCE(connection->req_ack_pending);
+ if (r2 && r2 != r1) {
+ r1 = r2;
+ seq_print_minor_vnr_req(m, r1, now, jif);
+ }
+ r2 = READ_ONCE(connection->req_not_net_done);
if (r2 && r2 != r1)
- seq_print_minor_vnr_req(m, r2, now);
- spin_unlock_irq(&connection->resource->req_lock);
+ seq_print_minor_vnr_req(m, r2, now, jif);
+ rcu_read_unlock();
return 0;
}
-static int connection_oldest_requests_open(struct inode *inode, struct file *file)
+static int connection_transport_show(struct seq_file *m, void *ignored)
{
- struct drbd_connection *connection = inode->i_private;
- return drbd_single_open(file, connection_oldest_requests_show, connection,
- &connection->kref, drbd_destroy_connection);
+ struct drbd_connection *connection = m->private;
+ struct drbd_transport *transport = &connection->transport;
+ struct drbd_transport_ops *tr_ops = &transport->class->ops;
+ enum drbd_stream i;
+
+ /* BUMP me if you change the file format/content/presentation */
+ seq_printf(m, "v: %u\n\n", 1);
+
+ for (i = DATA_STREAM; i <= CONTROL_STREAM; i++) {
+ struct drbd_send_buffer *sbuf = &connection->send_buffer[i];
+
+ seq_printf(m, "%s stream\n", i == DATA_STREAM ? "data" : "control");
+ seq_printf(m, " corked: %d\n", test_bit(CORKED + i, &connection->flags));
+ seq_printf(m, " unsent: %ld bytes\n", (long)(sbuf->pos - sbuf->unsent));
+ seq_printf(m, " allocated: %d bytes\n", sbuf->allocated_size);
+ }
+
+ seq_printf(m, "\nrx_misaligned_copies: %u\n", connection->rx_misaligned_copies);
+
+ seq_printf(m, "transport_type: %s\n", transport->class->name);
+
+ tr_ops->debugfs_show(transport, m);
+
+ return 0;
+}
+
+static int connection_debug_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_connection *connection = m->private;
+ struct drbd_resource *resource = connection->resource;
+ unsigned long flags = connection->flags;
+ unsigned int u1, u2;
+ unsigned long long ull1, ull2;
+ int in_flight;
+ char sep = ' ';
+
+ seq_puts(m, "content and format of this will change without notice\n");
+
+ seq_printf(m, "flags: 0x%04lx :", flags);
+#define pretty_print_bit(n) \
+ seq_print_rq_state_bit(m, test_bit(n, &flags), &sep, #n)
+ pretty_print_bit(PING_PENDING);
+ pretty_print_bit(TWOPC_PREPARED);
+ pretty_print_bit(TWOPC_YES);
+ pretty_print_bit(TWOPC_NO);
+ pretty_print_bit(TWOPC_RETRY);
+ pretty_print_bit(CONN_DRY_RUN);
+ pretty_print_bit(DISCONNECT_EXPECTED);
+ pretty_print_bit(DATA_CORKED);
+ pretty_print_bit(CONTROL_CORKED);
+ pretty_print_bit(C_UNREGISTERED);
+ pretty_print_bit(RECONNECT);
+ pretty_print_bit(CONN_DISCARD_MY_DATA);
+ pretty_print_bit(SEND_STATE_AFTER_AHEAD_C);
+ pretty_print_bit(NOTIFY_PEERS_LOST_PRIMARY);
+ pretty_print_bit(CHECKING_PEER);
+ pretty_print_bit(CONN_CONGESTED);
+ pretty_print_bit(CONN_HANDSHAKE_DISCONNECT);
+ pretty_print_bit(CONN_HANDSHAKE_RETRY);
+ pretty_print_bit(CONN_HANDSHAKE_READY);
+#undef pretty_print_bit
+ seq_putc(m, '\n');
+
+ u1 = atomic_read(&resource->current_tle_nr);
+ u2 = connection->send.current_epoch_nr;
+ seq_printf(m, "resource->current_tle_nr: %u\n", u1);
+ seq_printf(m, " send.current_epoch_nr: %u (%d)\n", u2, (int)(u2 - u1));
+ seq_printf(m, " last barrier sent, acked: %u, %u\n",
+ READ_ONCE(connection->send.last_sent_epoch_nr),
+ READ_ONCE(connection->last_acked_epoch_nr));
+
+ ull1 = resource->dagtag_sector;
+ ull2 = resource->last_peer_acked_dagtag;
+ seq_printf(m, " resource->dagtag_sector: %llu\n", ull1);
+ seq_printf(m, " last_peer_acked_dagtag: %llu (%lld)\n", ull2, (long long)(ull2 - ull1));
+ ull2 = connection->send.current_dagtag_sector;
+ seq_printf(m, " send.current_dagtag_sec: %llu (%lld)\n", ull2, (long long)(ull2 - ull1));
+ ull2 = atomic64_read(&connection->last_dagtag_sector);
+ seq_printf(m, " last_dagtag_sector: %llu\n", ull2);
+ seq_printf(m, "last_peer_ack_dagtag_seen: %llu\n",
+ (unsigned long long) connection->last_peer_ack_dagtag_seen);
+
+ spin_lock_irq(&resource->initiator_flush_lock);
+ seq_printf(m, "resource->current_flush_sequence: %llu\n",
+ (unsigned long long) resource->current_flush_sequence);
+ seq_puts(m, " pending_flush_mask: ");
+ seq_print_node_mask(m, resource, connection->pending_flush_mask);
+ spin_unlock_irq(&resource->initiator_flush_lock);
+
+ spin_lock_irq(&connection->primary_flush_lock);
+ seq_printf(m, " flush_requests_dagtag: %llu\n",
+ (unsigned long long) connection->flush_requests_dagtag);
+ seq_printf(m, " flush_sequence: %llu\n",
+ (unsigned long long) connection->flush_sequence);
+ seq_puts(m, " flush_forward_sent_mask: ");
+ seq_print_node_mask(m, resource, connection->flush_forward_sent_mask);
+ spin_unlock_irq(&connection->primary_flush_lock);
+
+ spin_lock_irq(&connection->flush_ack_lock);
+ for (u1 = 0; u1 < DRBD_PEERS_MAX; u1++) {
+ if (connection->flush_ack_sequence[u1])
+ seq_printf(m, " flush_ack_sequence[%u]: %llu\n", u1,
+ (unsigned long long) connection->flush_ack_sequence[u1]);
+ }
+ spin_unlock_irq(&connection->flush_ack_lock);
+
+ in_flight = atomic_read(&connection->ap_in_flight);
+ seq_printf(m, " ap_in_flight: %d KiB (%d sectors)\n", in_flight / 2, in_flight);
+
+ in_flight = atomic_read(&connection->rs_in_flight);
+ seq_printf(m, " rs_in_flight: %d KiB (%d sectors)\n", in_flight / 2, in_flight);
+
+ seq_printf(m, " done_ee_cnt: %d\n"
+ " backing_ee_cnt: %d\n"
+ " active_ee_cnt: %d\n",
+ atomic_read(&connection->done_ee_cnt),
+ atomic_read(&connection->backing_ee_cnt),
+ atomic_read(&connection->active_ee_cnt));
+ seq_printf(m, " agreed_pro_version: %d\n", connection->agreed_pro_version);
+ seq_printf(m, " send control: %u bytes/pckt (%u bytes, %u pckts)\n",
+ connection->ctl_bytes / (connection->ctl_packets ?: 1),
+ connection->ctl_bytes, connection->ctl_packets);
+ return 0;
+}
+
+static void pid_show(struct seq_file *m, struct drbd_thread *thi)
+{
+ struct task_struct *task = NULL;
+ pid_t pid;
+
+ spin_lock_irq(&thi->t_lock);
+ task = thi->task;
+ if (task)
+ pid = task->pid;
+ spin_unlock_irq(&thi->t_lock);
+ if (task)
+ seq_printf(m, "%d\n", pid);
+}
+
+static int connection_receiver_pid_show(struct seq_file *m, void *pos)
+{
+ struct drbd_connection *connection = m->private;
+
+ pid_show(m, &connection->receiver);
+ return 0;
}
-static int connection_oldest_requests_release(struct inode *inode, struct file *file)
+static int connection_sender_pid_show(struct seq_file *m, void *pos)
+{
+ struct drbd_connection *connection = m->private;
+
+ pid_show(m, &connection->sender);
+ return 0;
+}
+
+static int connection_attr_release(struct inode *inode, struct file *file)
{
struct drbd_connection *connection = inode->i_private;
+
kref_put(&connection->kref, drbd_destroy_connection);
return single_release(inode, file);
}
-static const struct file_operations connection_oldest_requests_fops = {
- .owner = THIS_MODULE,
- .open = connection_oldest_requests_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = connection_oldest_requests_release,
+#define drbd_debugfs_connection_attr(name) \
+static int connection_ ## name ## _open(struct inode *inode, struct file *file) \
+{ \
+ struct drbd_connection *connection = inode->i_private; \
+ return drbd_single_open(file, connection_ ## name ## _show, \
+ connection, &connection->kref, \
+ drbd_destroy_connection); \
+} \
+static const struct file_operations connection_ ## name ## _fops = { \
+ .owner = THIS_MODULE, \
+ .open = connection_ ## name ##_open, \
+ .read = seq_read, \
+ .llseek = seq_lseek, \
+ .release = connection_attr_release, \
};
+drbd_debugfs_connection_attr(oldest_requests)
+drbd_debugfs_connection_attr(callback_history)
+drbd_debugfs_connection_attr(transport)
+drbd_debugfs_connection_attr(debug)
+drbd_debugfs_connection_attr(receiver_pid)
+drbd_debugfs_connection_attr(sender_pid)
+
void drbd_debugfs_connection_add(struct drbd_connection *connection)
{
struct dentry *conns_dir = connection->resource->debugfs_res_connections;
+ struct drbd_peer_device *peer_device;
+ char conn_name[SHARED_SECRET_MAX];
struct dentry *dentry;
+ int vnr;
- /* Once we enable mutliple peers,
- * these connections will have descriptive names.
- * For now, it is just the one connection to the (only) "peer". */
- dentry = debugfs_create_dir("peer", conns_dir);
- connection->debugfs_conn = dentry;
+ rcu_read_lock();
+ strscpy(conn_name, rcu_dereference(connection->transport.net_conf)->name);
+ rcu_read_unlock();
- dentry = debugfs_create_file("callback_history", 0440,
- connection->debugfs_conn, connection,
- &connection_callback_history_fops);
- connection->debugfs_conn_callback_history = dentry;
+ dentry = debugfs_create_dir(conn_name, conns_dir);
+ connection->debugfs_conn = dentry;
- dentry = debugfs_create_file("oldest_requests", 0440,
- connection->debugfs_conn, connection,
- &connection_oldest_requests_fops);
- connection->debugfs_conn_oldest_requests = dentry;
+ /* debugfs create file */
+ conn_dcf(callback_history);
+ conn_dcf(oldest_requests);
+ conn_dcf(transport);
+ conn_dcf(debug);
+ conn_dcf(receiver_pid);
+ conn_dcf(sender_pid);
+
+ idr_for_each_entry(&connection->peer_devices, peer_device, vnr) {
+ if (!peer_device->debugfs_peer_dev)
+ drbd_debugfs_peer_device_add(peer_device);
+ }
}
void drbd_debugfs_connection_cleanup(struct drbd_connection *connection)
{
+ drbd_debugfs_remove(&connection->debugfs_conn_sender_pid);
+ drbd_debugfs_remove(&connection->debugfs_conn_receiver_pid);
+ drbd_debugfs_remove(&connection->debugfs_conn_debug);
+ drbd_debugfs_remove(&connection->debugfs_conn_transport);
drbd_debugfs_remove(&connection->debugfs_conn_callback_history);
drbd_debugfs_remove(&connection->debugfs_conn_oldest_requests);
drbd_debugfs_remove(&connection->debugfs_conn);
}
-static void resync_dump_detail(struct seq_file *m, struct lc_element *e)
+static void seq_printf_nice_histogram(struct seq_file *m, unsigned int *hist, unsigned int const n)
{
- struct bm_extent *bme = lc_entry(e, struct bm_extent, lce);
+ unsigned int i;
+ unsigned int max = 0;
+ unsigned int n_transactions = 0;
+ unsigned long n_updates = 0;
+
+ for (i = 1; i <= n; i++) {
+ if (hist[i] > max)
+ max = hist[i];
+ n_updates += i * hist[i];
+ n_transactions += hist[i];
+ }
+
+ seq_puts(m, "updates per activity log transaction\n");
+ seq_printf(m, "avg: %lu\n", n_transactions == 0 ? 0 : n_updates / n_transactions);
+
+ if (!max)
+ return;
- seq_printf(m, "%5d %s %s %s", bme->rs_left,
- test_bit(BME_NO_WRITES, &bme->flags) ? "NO_WRITES" : "---------",
- test_bit(BME_LOCKED, &bme->flags) ? "LOCKED" : "------",
- test_bit(BME_PRIORITY, &bme->flags) ? "PRIORITY" : "--------"
- );
+ for (i = 0; i <= n; i++) {
+ unsigned int v = (hist[i] * 60UL + max-1) / max;
+ seq_printf(m, "%2u : %10u : %-60.*s\n", i, hist[i], v,
+ "############################################################");
+ }
}
-static int device_resync_extents_show(struct seq_file *m, void *ignored)
+
+static int device_act_log_histogram_show(struct seq_file *m, void *ignored)
{
struct drbd_device *device = m->private;
@@ -669,8 +1182,7 @@ static int device_resync_extents_show(struct seq_file *m, void *ignored)
seq_printf(m, "v: %u\n\n", 0);
if (get_ldev_if_state(device, D_FAILED)) {
- lc_seq_printf_stats(m, device->resync);
- lc_seq_dump_details(m, device->resync, "rs_left flags", resync_dump_detail);
+ seq_printf_nice_histogram(m, device->al_histogram, AL_UPDATES_PER_TRANSACTION);
put_ldev(device);
}
return 0;
@@ -694,16 +1206,16 @@ static int device_act_log_extents_show(struct seq_file *m, void *ignored)
static int device_oldest_requests_show(struct seq_file *m, void *ignored)
{
struct drbd_device *device = m->private;
- struct drbd_resource *resource = device->resource;
- unsigned long now = jiffies;
+ ktime_t now = ktime_get();
+ unsigned long jif = jiffies;
struct drbd_request *r1, *r2;
int i;
/* BUMP me if you change the file format/content/presentation */
- seq_printf(m, "v: %u\n\n", 0);
+ seq_printf(m, "v: %u\n\n", 1);
seq_puts(m, RQ_HDR);
- spin_lock_irq(&resource->req_lock);
+ spin_lock_irq(&device->pending_completion_lock);
/* WRITE, then READ */
for (i = 1; i >= 0; --i) {
r1 = list_first_entry_or_null(&device->pending_master_completion[i],
@@ -711,11 +1223,89 @@ static int device_oldest_requests_show(struct seq_file *m, void *ignored)
r2 = list_first_entry_or_null(&device->pending_completion[i],
struct drbd_request, req_pending_local);
if (r1)
- seq_print_one_request(m, r1, now);
+ seq_print_one_request(m, r1, now, jif);
if (r2 && r2 != r1)
- seq_print_one_request(m, r2, now);
+ seq_print_one_request(m, r2, now, jif);
}
- spin_unlock_irq(&resource->req_lock);
+ spin_unlock_irq(&device->pending_completion_lock);
+ return 0;
+}
+
+static int device_openers_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_device *device = m->private;
+ struct drbd_resource *resource = device->resource;
+ ktime_t now = ktime_get_real();
+ struct opener *tmp;
+
+ spin_lock(&device->openers_lock);
+ list_for_each_entry(tmp, &device->openers, list)
+ seq_printf(m, "%s\t%d\t%lld\n", tmp->comm, tmp->pid,
+ ktime_to_ms(ktime_sub(now, tmp->opened)));
+ spin_unlock(&device->openers_lock);
+ if (mutex_trylock(&resource->open_release)) {
+ if (resource->auto_promoted_by.pid != 0
+ && device->minor == resource->auto_promoted_by.minor) {
+ seq_printf(m, "+%s\t%d\t%lld\n",
+ resource->auto_promoted_by.comm,
+ resource->auto_promoted_by.pid,
+ ktime_to_ms(ktime_sub(now, resource->auto_promoted_by.opened)));
+ }
+ mutex_unlock(&resource->open_release);
+ }
+
+ return 0;
+}
+
+static int device_md_io_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_device *device = m->private;
+
+ if (get_ldev_if_state(device, D_FAILED)) {
+ seq_puts(m, drbd_md_dax_active(device->ldev) ? "dax-pmem\n" : "blk-bio\n");
+ put_ldev(device);
+ }
+
+ return 0;
+}
+
+static void seq_printf_interval_tree(struct seq_file *m, struct rb_root *root)
+{
+ struct rb_node *node;
+
+ node = rb_first(root);
+ while (node) {
+ struct drbd_interval *i = rb_entry(node, struct drbd_interval, rb);
+ char sep = ' ';
+
+ seq_printf(m, "%llus+%u %s", (unsigned long long) i->sector, i->size, drbd_interval_type_str(i));
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_READY_TO_SEND, &i->flags), &sep,
+ "ready-to-send");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_SENT, &i->flags), &sep, "sent");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_RECEIVED, &i->flags), &sep, "received");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_SUBMIT_CONFLICT_QUEUED, &i->flags), &sep, "submit-conflict-queued");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_SUBMITTED, &i->flags), &sep, "submitted");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_BACKING_COMPLETED, &i->flags), &sep, "backing-completed");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_COMPLETED, &i->flags), &sep, "completed");
+ seq_print_rq_state_bit(m, test_bit(INTERVAL_CANCELED, &i->flags), &sep, "canceled");
+ seq_putc(m, '\n');
+
+ node = rb_next(node);
+ }
+}
+
+static int device_interval_tree_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_device *device = m->private;
+
+ spin_lock_irq(&device->interval_lock);
+ seq_puts(m, "Write requests:\n");
+ seq_printf_interval_tree(m, &device->requests);
+ seq_putc(m, '\n');
+ seq_puts(m, "Read requests:\n");
+ seq_printf_interval_tree(m, &device->read_requests);
+ spin_unlock_irq(&device->interval_lock);
+
return 0;
}
@@ -723,63 +1313,251 @@ static int device_data_gen_id_show(struct seq_file *m, void *ignored)
{
struct drbd_device *device = m->private;
struct drbd_md *md;
- enum drbd_uuid_index idx;
+ int node_id, i = 0;
if (!get_ldev_if_state(device, D_FAILED))
return -ENODEV;
md = &device->ldev->md;
+
spin_lock_irq(&md->uuid_lock);
- for (idx = UI_CURRENT; idx <= UI_HISTORY_END; idx++) {
- seq_printf(m, "0x%016llX\n", md->uuid[idx]);
+ seq_printf(m, "0x%016llX\n", drbd_current_uuid(device));
+
+ for (node_id = 0; node_id < DRBD_NODE_ID_MAX; node_id++) {
+ struct drbd_peer_md *peer_md = &md->peers[node_id];
+
+ if (!test_bit(__MDF_HAVE_BITMAP, &peer_md->flags))
+ continue;
+ /* A divergence bitmap fully records the divergence and is safe to
+ * copy; without the flag it is a convergence bitmap being cleared
+ * by a resync.
+ */
+ seq_printf(m, "%s[%d]0x%016llX%s%s", i++ ? " " : "", node_id,
+ peer_md->bitmap_uuid,
+ test_bit(__MDF_PEER_DIVERGENCE_BITMAP, &peer_md->flags) ?
+ "(DIVERGENCE_BITMAP)" : "",
+ test_bit(__MDF_PEER_BITMAP_AUTHORITATIVE, &peer_md->flags) ?
+ "(BITMAP_AUTHORITATIVE)" : "");
}
+ seq_putc(m, '\n');
+
+ for (i = 0; i < HISTORY_UUIDS; i++)
+ seq_printf(m, "0x%016llX\n", drbd_history_uuid(device, i));
spin_unlock_irq(&md->uuid_lock);
put_ldev(device);
return 0;
}
+static int device_io_frozen_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_device *device = m->private;
+ unsigned long flags = device->flags;
+ char obligation[GEN_OBL_STR_MAX];
+ char sep = ' ';
+
+ if (!get_ldev_if_state(device, D_FAILED))
+ return -ENODEV;
+
+ /* BUMP me if you change the file format/content/presentation */
+ seq_printf(m, "v: %u\n\n", 2);
+
+ seq_printf(m, "drbd_suspended(): %d\n", drbd_suspended(device));
+ seq_printf(m, "suspend_cnt: %d\n", atomic_read(&device->suspend_cnt));
+ seq_printf(m, "!drbd_state_is_stable(): %d\n", device->cached_state_unstable);
+ seq_printf(m, "ap_bio_cnt[READ]: %d\n", atomic_read(&device->ap_bio_cnt[READ]));
+ seq_printf(m, "ap_bio_cnt[WRITE]: %d\n", atomic_read(&device->ap_bio_cnt[WRITE]));
+ seq_printf(m, "device->pending_bitmap_work.n: %d\n", atomic_read(&device->pending_bitmap_work.n));
+ seq_printf(m, "may_inc_ap_bio(): %d\n", may_inc_ap_bio(device));
+ drbd_gen_obligation_str(READ_ONCE(device->gen_obligation), obligation, sizeof(obligation));
+ seq_printf(m, "gen-obligation: %s\n", obligation);
+ seq_printf(m, "flags: 0x%04lx :", flags);
+#define pretty_print_bit(n) \
+ seq_print_rq_state_bit(m, test_bit(n, &flags), &sep, #n)
+ pretty_print_bit(MAKE_NEW_CUR_UUID);
+#undef pretty_print_bit
+ seq_putc(m, '\n');
+ put_ldev(device);
+
+ return 0;
+}
+
+static int device_al_updates_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_device *device = m->private;
+ bool al_updates, cfg_al_updates;
+
+ if (!get_ldev_if_state(device, D_FAILED))
+ return -ENODEV;
+
+ al_updates = !(device->ldev->md.flags & MDF_AL_DISABLED) ||
+ test_bit(AL_ENABLE_PENDING, &device->flags);
+ rcu_read_lock();
+ cfg_al_updates = rcu_dereference(device->ldev->disk_conf)->al_updates;
+ rcu_read_unlock();
+ put_ldev(device);
+
+ seq_printf(m, "%s\n",
+ al_updates && cfg_al_updates ? "yes" :
+ !al_updates && cfg_al_updates ? "no (optimized)" :
+ !al_updates && !cfg_al_updates ? "no" :
+ "?");
+ return 0;
+}
+
static int device_ed_gen_id_show(struct seq_file *m, void *ignored)
{
struct drbd_device *device = m->private;
- seq_printf(m, "0x%016llX\n", (unsigned long long)device->ed_uuid);
+
+ seq_printf(m, "0x%016llX\n", (unsigned long long)device->exposed_data_uuid);
+ return 0;
+}
+
+static int device_multi_bio_cnt_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_device *device = m->private;
+
+ seq_printf(m, "%u\n", device->multi_bio_cnt);
+ return 0;
+}
+
+#define show_per_peer(M) do { \
+ seq_printf(m, "%-16s", #M ":"); \
+ for_each_peer_device(peer_device, device) \
+ seq_printf(m, " %12lld", ktime_to_ns(peer_device->M)); \
+ seq_printf(m, "\n"); \
+ } while (0)
+
+#define PRId64 "lld"
+
+#ifdef CONFIG_DRBD_TIMING_STATS
+static int device_req_timing_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_device *device = m->private;
+ struct drbd_peer_device *peer_device;
+
+ seq_printf(m,
+ "timing values are nanoseconds; write an 'r' to reset all to 0\n\n"
+ "requests: %12lu\n"
+ "before_queue: %12" PRId64 "\n"
+ "before_al_begin %12" PRId64 "\n"
+ "in_actlog: %12" PRId64 "\n"
+ "pre_submit: %12" PRId64 "\n\n"
+ "al_updates: %12u\n"
+ "before_bm_write %12" PRId64 "\n"
+ "mid %12" PRId64 "\n"
+ "after_sync_page %12" PRId64 "\n",
+ device->reqs,
+ ktime_to_ns(device->before_queue_kt),
+ ktime_to_ns(device->before_al_begin_io_kt),
+ ktime_to_ns(device->in_actlog_kt),
+ ktime_to_ns(device->pre_submit_kt),
+ device->al_writ_cnt,
+ ktime_to_ns(device->al_before_bm_write_hinted_kt),
+ ktime_to_ns(device->al_mid_kt),
+ ktime_to_ns(device->al_after_sync_page_kt));
+
+ seq_puts(m, "\npeer: ");
+ for_each_peer_device(peer_device, device) {
+ struct drbd_connection *connection = peer_device->connection;
+
+ seq_printf(m, " %12.12s", rcu_dereference(connection->transport.net_conf)->name);
+ }
+ seq_puts(m, "\n");
+ show_per_peer(pre_send_kt);
+ show_per_peer(acked_kt);
+ show_per_peer(net_done_kt);
+
return 0;
}
-#define drbd_debugfs_device_attr(name) \
+static ssize_t device_req_timing_write(struct file *file, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ struct drbd_device *device = file_inode(file)->i_private;
+ char buffer;
+
+ if (copy_from_user(&buffer, ubuf, 1))
+ return -EFAULT;
+
+ if (buffer == 'r' || buffer == 'R') {
+ struct drbd_peer_device *peer_device;
+ unsigned long flags;
+
+ spin_lock_irqsave(&device->timing_lock, flags);
+ device->reqs = 0;
+ device->in_actlog_kt = ns_to_ktime(0);
+ device->pre_submit_kt = ns_to_ktime(0);
+
+ device->before_queue_kt = ns_to_ktime(0);
+ device->before_al_begin_io_kt = ns_to_ktime(0);
+ device->al_writ_cnt = 0;
+ device->al_before_bm_write_hinted_kt = ns_to_ktime(0);
+ device->al_mid_kt = ns_to_ktime(0);
+ device->al_after_sync_page_kt = ns_to_ktime(0);
+
+ for_each_peer_device(peer_device, device) {
+ peer_device->pre_send_kt = ns_to_ktime(0);
+ peer_device->acked_kt = ns_to_ktime(0);
+ peer_device->net_done_kt = ns_to_ktime(0);
+ }
+ spin_unlock_irqrestore(&device->timing_lock, flags);
+ }
+
+ *ppos += cnt;
+ return cnt;
+}
+#endif
+
+static int device_attr_release(struct inode *inode, struct file *file)
+{
+ struct drbd_device *device = inode->i_private;
+
+ kref_put(&device->kref, drbd_destroy_device);
+ return single_release(inode, file);
+}
+
+#define __drbd_debugfs_device_attr(name, write_fn) \
static int device_ ## name ## _open(struct inode *inode, struct file *file) \
{ \
struct drbd_device *device = inode->i_private; \
return drbd_single_open(file, device_ ## name ## _show, device, \
&device->kref, drbd_destroy_device); \
} \
-static int device_ ## name ## _release(struct inode *inode, struct file *file) \
-{ \
- struct drbd_device *device = inode->i_private; \
- kref_put(&device->kref, drbd_destroy_device); \
- return single_release(inode, file); \
-} \
static const struct file_operations device_ ## name ## _fops = { \
.owner = THIS_MODULE, \
.open = device_ ## name ## _open, \
+ .write = write_fn, \
.read = seq_read, \
.llseek = seq_lseek, \
- .release = device_ ## name ## _release, \
+ .release = device_attr_release, \
};
+#define drbd_debugfs_device_attr(name) __drbd_debugfs_device_attr(name, NULL)
drbd_debugfs_device_attr(oldest_requests)
drbd_debugfs_device_attr(act_log_extents)
-drbd_debugfs_device_attr(resync_extents)
+drbd_debugfs_device_attr(act_log_histogram)
drbd_debugfs_device_attr(data_gen_id)
+drbd_debugfs_device_attr(io_frozen)
drbd_debugfs_device_attr(ed_gen_id)
+drbd_debugfs_device_attr(openers)
+drbd_debugfs_device_attr(md_io)
+drbd_debugfs_device_attr(interval_tree)
+drbd_debugfs_device_attr(al_updates)
+drbd_debugfs_device_attr(multi_bio_cnt)
+#ifdef CONFIG_DRBD_TIMING_STATS
+__drbd_debugfs_device_attr(req_timing, device_req_timing_write)
+#endif
void drbd_debugfs_device_add(struct drbd_device *device)
{
struct dentry *vols_dir = device->resource->debugfs_res_volumes;
+ struct drbd_peer_device *peer_device;
char minor_buf[8]; /* MINORMASK, MINORBITS == 20; */
char vnr_buf[8]; /* volume number vnr is even 16 bit only; */
char *slink_name = NULL;
struct dentry *dentry;
+
if (!vols_dir || !drbd_debugfs_minors)
return;
@@ -797,19 +1575,28 @@ void drbd_debugfs_device_add(struct drbd_device *device)
kfree(slink_name);
slink_name = NULL;
-#define DCF(name) do { \
- dentry = debugfs_create_file(#name, 0440, \
- device->debugfs_vol, device, \
- &device_ ## name ## _fops); \
- device->debugfs_vol_ ## name = dentry; \
- } while (0)
+ /* debugfs create file */
+ vol_dcf(oldest_requests);
+ vol_dcf(act_log_extents);
+ vol_dcf(act_log_histogram);
+ vol_dcf(data_gen_id);
+ vol_dcf(io_frozen);
+ vol_dcf(ed_gen_id);
+ vol_dcf(openers);
+ vol_dcf(md_io);
+ vol_dcf(interval_tree);
+ vol_dcf(al_updates);
+ vol_dcf(multi_bio_cnt);
+#ifdef CONFIG_DRBD_TIMING_STATS
+ drbd_dcf(device->debugfs_vol, device, req_timing, 0600);
+#endif
+
+ /* Caller holds conf_update */
+ for_each_peer_device(peer_device, device) {
+ if (!peer_device->debugfs_peer_dev)
+ drbd_debugfs_peer_device_add(peer_device);
+ }
- DCF(oldest_requests);
- DCF(act_log_extents);
- DCF(resync_extents);
- DCF(data_gen_id);
- DCF(ed_gen_id);
-#undef DCF
return;
fail:
@@ -822,12 +1609,383 @@ void drbd_debugfs_device_cleanup(struct drbd_device *device)
drbd_debugfs_remove(&device->debugfs_minor);
drbd_debugfs_remove(&device->debugfs_vol_oldest_requests);
drbd_debugfs_remove(&device->debugfs_vol_act_log_extents);
- drbd_debugfs_remove(&device->debugfs_vol_resync_extents);
+ drbd_debugfs_remove(&device->debugfs_vol_act_log_histogram);
drbd_debugfs_remove(&device->debugfs_vol_data_gen_id);
+ drbd_debugfs_remove(&device->debugfs_vol_io_frozen);
drbd_debugfs_remove(&device->debugfs_vol_ed_gen_id);
+ drbd_debugfs_remove(&device->debugfs_vol_openers);
+ drbd_debugfs_remove(&device->debugfs_vol_md_io);
+ drbd_debugfs_remove(&device->debugfs_vol_interval_tree);
+ drbd_debugfs_remove(&device->debugfs_vol_al_updates);
+ drbd_debugfs_remove(&device->debugfs_vol_multi_bio_cnt);
+#ifdef CONFIG_DRBD_TIMING_STATS
+ drbd_debugfs_remove(&device->debugfs_vol_req_timing);
+#endif
drbd_debugfs_remove(&device->debugfs_vol);
}
+static int drbd_single_open_peer_device(struct file *file,
+ int (*show)(struct seq_file *, void *),
+ struct drbd_peer_device *peer_device)
+{
+ struct drbd_device *device = peer_device->device;
+ struct drbd_connection *connection = peer_device->connection;
+ bool got_connection, got_device;
+ struct dentry *parent;
+
+ parent = file->f_path.dentry->d_parent;
+ if (!parent || !parent->d_inode)
+ goto out;
+ inode_lock(d_inode(parent));
+ if (!simple_positive(file->f_path.dentry))
+ goto out_unlock;
+
+ got_connection = kref_get_unless_zero(&connection->kref);
+ got_device = kref_get_unless_zero(&device->kref);
+
+ if (got_connection && got_device) {
+ int ret;
+
+ inode_unlock(d_inode(parent));
+ ret = single_open(file, show, peer_device);
+ if (ret) {
+ kref_put(&connection->kref, drbd_destroy_connection);
+ kref_put(&device->kref, drbd_destroy_device);
+ }
+ return ret;
+ }
+
+ if (got_connection)
+ kref_put(&connection->kref, drbd_destroy_connection);
+ if (got_device)
+ kref_put(&device->kref, drbd_destroy_device);
+out_unlock:
+ inode_unlock(d_inode(parent));
+out:
+ return -ESTALE;
+}
+
+static void seq_printf_with_thousands_grouping(struct seq_file *seq, long v)
+{
+ /* v is in kB/sec. We don't expect TiByte/sec yet.
+ * v can be negative: while the resync loses ground to incoming
+ * application writes the out-of-sync count grows, so the rate at
+ * which it shrinks is negative. Show that honestly.
+ */
+ if (v < 0) {
+ seq_putc(seq, '-');
+ v = -v;
+ }
+ if (unlikely(v >= 1000000)) {
+ /* cool: > GiByte/s */
+ seq_printf(seq, "%ld,", v / 1000000);
+ v %= 1000000;
+ seq_printf(seq, "%03ld,%03ld", v/1000, v % 1000);
+ } else if (likely(v >= 1000))
+ seq_printf(seq, "%ld,%03ld", v/1000, v % 1000);
+ else
+ seq_printf(seq, "%ld", v);
+}
+
+/* like bit_to_kb(), but accepts a signed bit count so we can express a
+ * negative kB/sec rate when the out-of-sync set is growing.
+ */
+static long signed_bit_to_kb(long bits, unsigned int bm_block_shift)
+{
+ if (bits < 0)
+ return -(long)bit_to_kb(-bits, bm_block_shift);
+ return bit_to_kb(bits, bm_block_shift);
+}
+
+static void drbd_get_syncer_progress(struct drbd_peer_device *pd,
+ enum drbd_repl_state repl_state, unsigned long *rs_total,
+ unsigned long *bits_left, unsigned int *per_mil_done)
+{
+ /* this is to break it at compile time when we change that, in case we
+ * want to support more than (1<<32) bits on a 32bit arch. */
+ typecheck(unsigned long, pd->rs_total);
+ *rs_total = pd->rs_total;
+
+ /* note: both rs_total and rs_left are in bits, i.e. in
+ * units of BM_BLOCK_SIZE.
+ * for the percentage, we don't care. */
+
+ if (repl_state == L_VERIFY_S || repl_state == L_VERIFY_T)
+ *bits_left = atomic64_read(&pd->ov_left);
+ else
+ *bits_left = drbd_bm_total_weight(pd) - pd->rs_failed;
+ /* >> 10 to prevent overflow,
+ * +1 to prevent division by zero */
+ if (*bits_left > *rs_total) {
+ /* D'oh. Maybe a logic bug somewhere. More likely just a race
+ * between state change and reset of rs_total.
+ */
+ *bits_left = *rs_total;
+ *per_mil_done = *rs_total ? 0 : 1000;
+ } else {
+ /* Make sure the division happens in long context.
+ * We allow up to one petabyte storage right now,
+ * at a granularity of 4k per bit that is 2**38 bits.
+ * After shift right and multiplication by 1000,
+ * this should still fit easily into a 32bit long,
+ * so we don't need a 64bit division on 32bit arch.
+ * Note: currently we don't support such large bitmaps on 32bit
+ * arch anyways, but no harm done to be prepared for it here.
+ */
+ unsigned int shift = *rs_total > UINT_MAX ? 16 : 10;
+ unsigned long left = *bits_left >> shift;
+ unsigned long total = 1UL + (*rs_total >> shift);
+ unsigned long tmp = 1000UL - left * 1000UL/total;
+ *per_mil_done = tmp;
+ }
+}
+
+static void drbd_syncer_progress(struct drbd_peer_device *pd, struct seq_file *seq,
+ enum drbd_repl_state repl_state)
+{
+ unsigned long dt, rt, rs_total, rs_left;
+ long db, dbdt;
+ unsigned int res;
+ int i, x, y;
+ int stalled = 0;
+ unsigned int bm_block_shift = pd->device->last_bm_block_shift;
+
+ drbd_get_syncer_progress(pd, repl_state, &rs_total, &rs_left, &res);
+
+ x = res/50;
+ y = 20-x;
+ seq_puts(seq, "\t[");
+ for (i = 1; i < x; i++)
+ seq_putc(seq, '=');
+ seq_putc(seq, '>');
+ for (i = 0; i < y; i++)
+ seq_putc(seq, '.');
+ seq_puts(seq, "] ");
+
+ if (repl_state == L_VERIFY_S || repl_state == L_VERIFY_T)
+ seq_puts(seq, "verified:");
+ else
+ seq_puts(seq, "sync'ed:");
+ seq_printf(seq, "%3u.%u%% ", res / 10, res % 10);
+
+ /* if more than a few GB, display in MB */
+ if (rs_total > (4UL << (30 - bm_block_shift)))
+ seq_printf(seq, "(%llu/%llu)M",
+ bit_to_kb(rs_left >> 10, bm_block_shift),
+ bit_to_kb(rs_total >> 10, bm_block_shift));
+ else
+ seq_printf(seq, "(%llu/%llu)K",
+ bit_to_kb(rs_left, bm_block_shift),
+ bit_to_kb(rs_total, bm_block_shift));
+
+ seq_puts(seq, "\n\t");
+
+ /* see drivers/md/md.c
+ * We do not want to overflow, so the order of operands and
+ * the * 100 / 100 trick are important. We do a +1 to be
+ * safe against division by zero. We only estimate anyway.
+ *
+ * dt: time from mark until now
+ * db: blocks written from mark until now
+ * rt: remaining time
+ */
+ /* Rolling marks. last_mark+1 may just now be modified. last_mark+2 is
+ * at least (DRBD_SYNC_MARKS-2)*DRBD_SYNC_MARK_STEP old, and has at
+ * least DRBD_SYNC_MARK_STEP time before it will be modified. */
+ /* ------------------------ ~18s average ------------------------ */
+ i = (pd->rs_last_mark + 2) % DRBD_SYNC_MARKS;
+ dt = (jiffies - pd->rs_mark_time[i]) / HZ;
+ if (dt > 180)
+ stalled = 1;
+
+ if (!dt)
+ dt++;
+ db = (long)pd->rs_mark_left[i] - (long)rs_left;
+ if (db <= 0) {
+ /* out-of-sync is not shrinking: under this write load the
+ * resync is not converging, so there is no finish time.
+ */
+ seq_puts(seq, "finish: ∞");
+ } else {
+ rt = (dt * (rs_left / ((unsigned long)db/100+1)))/100; /* seconds */
+ seq_printf(seq, "finish: %lu:%02lu:%02lu",
+ rt / 3600, (rt % 3600) / 60, rt % 60);
+ }
+
+ dbdt = signed_bit_to_kb(db / (long)dt, bm_block_shift);
+ seq_puts(seq, " speed: ");
+ seq_printf_with_thousands_grouping(seq, dbdt);
+ seq_puts(seq, " (");
+ /* ------------------------- ~3s average ------------------------ */
+ if (1) {
+ /* this is what drbd_rs_should_slow_down() uses */
+ i = (pd->rs_last_mark + DRBD_SYNC_MARKS-1) % DRBD_SYNC_MARKS;
+ dt = (jiffies - pd->rs_mark_time[i]) / HZ;
+ if (!dt)
+ dt++;
+ db = (long)pd->rs_mark_left[i] - (long)rs_left;
+ dbdt = signed_bit_to_kb(db / (long)dt, bm_block_shift);
+ seq_printf_with_thousands_grouping(seq, dbdt);
+ seq_puts(seq, " -- ");
+ }
+
+ /* --------------------- long term average ---------------------- */
+ /* mean speed since syncer started
+ * we do account for PausedSync periods */
+ dt = (jiffies - pd->rs_start - pd->rs_paused) / HZ;
+ if (dt == 0)
+ dt = 1;
+ db = (long)rs_total - (long)rs_left;
+ dbdt = signed_bit_to_kb(db / (long)dt, bm_block_shift);
+ seq_printf_with_thousands_grouping(seq, dbdt);
+ seq_putc(seq, ')');
+
+ if (repl_state == L_SYNC_TARGET ||
+ repl_state == L_VERIFY_S) {
+ seq_puts(seq, " want: ");
+ seq_printf_with_thousands_grouping(seq, pd->c_sync_rate);
+ }
+ seq_printf(seq, " K/sec%s\n", stalled ? " (stalled)" : "");
+
+ {
+ /* 64 bit:
+ * we convert to sectors in the display below. */
+ unsigned long bm_bits = drbd_bm_bits(pd->device);
+ unsigned long bit_pos;
+ unsigned long long stop_sector = 0;
+
+ if (repl_state == L_VERIFY_S ||
+ repl_state == L_VERIFY_T) {
+ bit_pos = bm_bits - (unsigned long)atomic64_read(&pd->ov_left);
+ if (verify_can_do_stop_sector(pd))
+ stop_sector = pd->ov_stop_sector;
+ } else
+ bit_pos = pd->resync_next_bit;
+ /* Total sectors may be slightly off for oddly
+ * sized devices. So what. */
+ seq_printf(seq,
+ "\t%3d%% sector pos: %llu/%llu",
+ (int)(bit_pos / (bm_bits/100+1)),
+ (unsigned long long)bit_pos * sect_per_bit(bm_block_shift),
+ (unsigned long long)bm_bits * sect_per_bit(bm_block_shift));
+ if (stop_sector != 0 && stop_sector != ULLONG_MAX)
+ seq_printf(seq, " stop sector: %llu", stop_sector);
+ seq_putc(seq, '\n');
+ }
+}
+
+static int peer_device_proc_drbd_show(struct seq_file *m, void *ignored)
+{
+ struct drbd_peer_device *peer_device = m->private;
+ struct drbd_device *device = peer_device->device;
+ union drbd_state state;
+ const char *sn;
+ struct drbd_net_conf *nc;
+ bool have_ldev;
+ char wp;
+
+ state.disk = device->disk_state[NOW];
+ state.pdsk = peer_device->disk_state[NOW];
+ state.conn = peer_device->repl_state[NOW];
+ state.role = device->resource->role[NOW];
+ state.peer = peer_device->connection->peer_role[NOW];
+
+ state.user_isp = peer_device->resync_susp_user[NOW];
+ state.peer_isp = peer_device->resync_susp_peer[NOW];
+ state.aftr_isp = peer_device->resync_susp_dependency[NOW];
+
+ sn = drbd_repl_str(state.conn);
+
+ rcu_read_lock();
+ have_ldev = get_ldev_if_state(device, D_FAILED);
+
+ /* reset device->congestion_reason */
+
+ nc = rcu_dereference(peer_device->connection->transport.net_conf);
+ wp = nc ? nc->wire_protocol - DRBD_PROT_A + 'A' : ' ';
+ seq_printf(m,
+ "%2d: cs:%s ro:%s/%s ds:%s/%s %c %c%c%c%c%c%c\n"
+ " ns:%u nr:%u dw:%u dr:%u al:%u bm:%u "
+ "lo:%d pe:[%d;%d] ua:%d ap:[%d;%d] ep:%d wo:%d",
+ device->minor, sn,
+ drbd_role_str(state.role),
+ drbd_role_str(state.peer),
+ drbd_disk_str(state.disk),
+ drbd_disk_str(state.pdsk),
+ wp,
+ drbd_suspended(device) ? 's' : 'r',
+ state.aftr_isp ? 'a' : '-',
+ state.peer_isp ? 'p' : '-',
+ state.user_isp ? 'u' : '-',
+ '-' /* congestion reason... FIXME */,
+ test_bit(AL_SUSPENDED, &device->flags) ? 's' : '-',
+ peer_device->send_cnt/2,
+ peer_device->recv_cnt/2,
+ device->writ_cnt/2,
+ device->read_cnt/2,
+ device->al_writ_cnt,
+ device->bm_writ_cnt,
+ atomic_read(&device->local_cnt),
+ atomic_read(&peer_device->ap_pending_cnt),
+ atomic_read(&peer_device->rs_pending_cnt),
+ atomic_read(&peer_device->unacked_cnt),
+ atomic_read(&device->ap_bio_cnt[WRITE]),
+ atomic_read(&device->ap_bio_cnt[READ]),
+ peer_device->connection->epochs,
+ device->resource->write_ordering
+ );
+
+ seq_printf(m, " oos:%llu\n",
+ have_ldev ? device_bit_to_kb(device, drbd_bm_total_weight(peer_device)) : 0);
+
+ if (have_ldev) {
+ if (state.conn == L_SYNC_SOURCE ||
+ state.conn == L_SYNC_TARGET ||
+ state.conn == L_VERIFY_S ||
+ state.conn == L_VERIFY_T)
+ drbd_syncer_progress(peer_device, m, state.conn);
+
+ lc_seq_printf_stats(m, device->act_log);
+
+ put_ldev(device);
+ }
+
+ seq_printf(m, "\tblocked on activity log: %d/%d/%d\n",
+ atomic_read(&device->ap_actlog_cnt), /* requests */
+ atomic_read(&device->wait_for_actlog), /* peer_requests */
+ /* nr extents needed to satisfy the above in the worst case */
+ atomic_read(&device->wait_for_actlog_ecnt));
+
+ rcu_read_unlock();
+
+ return 0;
+}
+
+#define drbd_debugfs_peer_device_attr(name) \
+static int peer_device_ ## name ## _open(struct inode *inode, struct file *file)\
+{ \
+ struct drbd_peer_device *peer_device = inode->i_private; \
+ return drbd_single_open_peer_device(file, \
+ peer_device_ ## name ## _show, \
+ peer_device); \
+} \
+static int peer_device_ ## name ## _release(struct inode *inode, struct file *file)\
+{ \
+ struct drbd_peer_device *peer_device = inode->i_private; \
+ kref_put(&peer_device->connection->kref, drbd_destroy_connection); \
+ kref_put(&peer_device->device->kref, drbd_destroy_device); \
+ return single_release(inode, file); \
+} \
+static const struct file_operations peer_device_ ## name ## _fops = { \
+ .owner = THIS_MODULE, \
+ .open = peer_device_ ## name ## _open, \
+ .read = seq_read, \
+ .llseek = seq_lseek, \
+ .release = peer_device_ ## name ## _release, \
+};
+
+drbd_debugfs_peer_device_attr(proc_drbd)
+
void drbd_debugfs_peer_device_add(struct drbd_peer_device *peer_device)
{
struct dentry *conn_dir = peer_device->connection->debugfs_conn;
@@ -837,10 +1995,14 @@ void drbd_debugfs_peer_device_add(struct drbd_peer_device *peer_device)
snprintf(vnr_buf, sizeof(vnr_buf), "%u", peer_device->device->vnr);
dentry = debugfs_create_dir(vnr_buf, conn_dir);
peer_device->debugfs_peer_dev = dentry;
+
+ /* debugfs create file */
+ peer_dev_dcf(proc_drbd);
}
void drbd_debugfs_peer_device_cleanup(struct drbd_peer_device *peer_device)
{
+ drbd_debugfs_remove(&peer_device->debugfs_peer_dev_proc_drbd);
drbd_debugfs_remove(&peer_device->debugfs_peer_dev);
}
@@ -851,6 +2013,11 @@ static int drbd_version_show(struct seq_file *m, void *ignored)
seq_printf(m, "API_VERSION=%u\n", DRBD_FAMILY_VERSION);
seq_printf(m, "PRO_VERSION_MIN=%u\n", PRO_VERSION_MIN);
seq_printf(m, "PRO_VERSION_MAX=%u\n", PRO_VERSION_MAX);
+#ifdef UTS_RELEASE
+ /* the UTS_RELEASE string of the prepared kernel source tree this
+ * module was built against */
+ seq_printf(m, "UTS_RELEASE=%s\n", UTS_RELEASE);
+#endif
return 0;
}
@@ -867,13 +2034,52 @@ static const struct file_operations drbd_version_fops = {
.release = single_release,
};
+static int drbd_refcounts_show(struct seq_file *m, void *ignored)
+{
+ seq_printf(m, "v: %u\n\n", 0);
+ return 0;
+}
+
+static int drbd_refcounts_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, drbd_refcounts_show, NULL);
+}
+
+static const struct file_operations drbd_refcounts_fops = {
+ .owner = THIS_MODULE,
+ .open = drbd_refcounts_open,
+ .llseek = seq_lseek,
+ .read = seq_read,
+ .release = single_release,
+};
+
+static int drbd_compat_show(struct seq_file *m, void *ignored)
+{
+ return 0;
+}
+
+static int drbd_compat_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, drbd_compat_show, NULL);
+}
+
+static const struct file_operations drbd_compat_fops = {
+ .owner = THIS_MODULE,
+ .open = drbd_compat_open,
+ .llseek = seq_lseek,
+ .read = seq_read,
+ .release = single_release,
+};
+
/* not __exit, may be indirectly called
* from the module-load-failure path as well. */
void drbd_debugfs_cleanup(void)
{
+ drbd_debugfs_remove(&drbd_debugfs_compat);
drbd_debugfs_remove(&drbd_debugfs_resources);
drbd_debugfs_remove(&drbd_debugfs_minors);
drbd_debugfs_remove(&drbd_debugfs_version);
+ drbd_debugfs_remove(&drbd_debugfs_refcounts);
drbd_debugfs_remove(&drbd_debugfs_root);
}
@@ -887,9 +2093,15 @@ void __init drbd_debugfs_init(void)
dentry = debugfs_create_file("version", 0444, drbd_debugfs_root, NULL, &drbd_version_fops);
drbd_debugfs_version = dentry;
+ dentry = debugfs_create_file("reference_counts", 0444, drbd_debugfs_root, NULL, &drbd_refcounts_fops);
+ drbd_debugfs_refcounts = dentry;
+
dentry = debugfs_create_dir("resources", drbd_debugfs_root);
drbd_debugfs_resources = dentry;
dentry = debugfs_create_dir("minors", drbd_debugfs_root);
drbd_debugfs_minors = dentry;
+
+ dentry = debugfs_create_file("compat", 0444, drbd_debugfs_root, NULL, &drbd_compat_fops);
+ drbd_debugfs_compat = dentry;
}
diff --git a/drivers/block/drbd/drbd_interval.c b/drivers/block/drbd/drbd_interval.c
index 2c12297ffdaf..8f4606e5bc62 100644
--- a/drivers/block/drbd/drbd_interval.c
+++ b/drivers/block/drbd/drbd_interval.c
@@ -4,6 +4,7 @@
*/
#include <asm/bug.h>
+#include <linux/kernel.h>
#include <linux/rbtree_augmented.h>
#include "drbd_interval.h"
@@ -14,13 +15,33 @@ static inline
sector_t interval_end(struct rb_node *node)
{
struct drbd_interval *this = rb_entry(node, struct drbd_interval, rb);
+
return this->end;
}
#define NODE_END(node) ((node)->sector + ((node)->size >> 9))
+RB_DECLARE_CALLBACKS_MAX(static, augment_callbacks, struct drbd_interval, rb,
+ sector_t, end, NODE_END);
+
+static const char * const drbd_interval_type_names[] = {
+ [INTERVAL_LOCAL_WRITE] = "LocalWrite",
+ [INTERVAL_PEER_WRITE] = "PeerWrite",
+ [INTERVAL_RESYNC_WRITE] = "ResyncWrite",
+ [INTERVAL_RESYNC_READ] = "ResyncRead",
+ [INTERVAL_OV_READ_SOURCE] = "VerifySource",
+ [INTERVAL_OV_READ_TARGET] = "VerifyTarget",
+ [INTERVAL_PEERS_IN_SYNC_LOCK] = "PeersInSync",
+};
+
+const char *drbd_interval_type_str(struct drbd_interval *i)
+{
+ enum drbd_interval_type type = i->type;
+ unsigned int size = ARRAY_SIZE(drbd_interval_type_names);
-RB_DECLARE_CALLBACKS_MAX(static, augment_callbacks,
- struct drbd_interval, rb, sector_t, end, NODE_END);
+ return (type < 0 || type >= size ||
+ !drbd_interval_type_names[type]) ?
+ "?" : drbd_interval_type_names[type];
+}
/*
* drbd_insert_interval - insert a new interval into a tree
@@ -106,6 +127,18 @@ drbd_remove_interval(struct rb_root *root, struct drbd_interval *this)
rb_erase_augmented(&this->rb, root, &augment_callbacks);
}
+void drbd_update_interval_size(struct drbd_interval *this, unsigned int new_size)
+{
+ this->size = new_size;
+
+ /* The size is one of the inputs to calculate the tree node's
+ * augmented value. When we change it we need to update the augmented
+ * value in this node and maybe in some parent nodes. That might be
+ * all the way up to the root. As this function is used for joining
+ * intervals, usually it will propagate only to the parent node. */
+ augment_callbacks_propagate(&this->rb, NULL);
+}
+
/**
* drbd_find_overlap - search for an interval overlapping with [sector, sector + size)
* @root: red black tree root
diff --git a/drivers/block/drbd/drbd_proc.c b/drivers/block/drbd/drbd_proc.c
index fb2b68c5c5f7..0f5a83288a44 100644
--- a/drivers/block/drbd/drbd_proc.c
+++ b/drivers/block/drbd/drbd_proc.c
@@ -3,313 +3,44 @@
* Copyright (C) 2001, LINBIT HA-Solutions GmbH.
*/
-#include <linux/module.h>
-
-#include <linux/uaccess.h>
-#include <linux/fs.h>
-#include <linux/file.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
-#include <linux/drbd.h>
+#include <linux/drbd_genl.h>
#include "drbd_int.h"
+#include "drbd_transport.h"
+#include "drbd_legacy_84.h"
struct proc_dir_entry *drbd_proc;
-static void seq_printf_with_thousands_grouping(struct seq_file *seq, long v)
-{
- /* v is in kB/sec. We don't expect TiByte/sec yet. */
- if (unlikely(v >= 1000000)) {
- /* cool: > GiByte/s */
- seq_printf(seq, "%ld,", v / 1000000);
- v %= 1000000;
- seq_printf(seq, "%03ld,%03ld", v/1000, v % 1000);
- } else if (likely(v >= 1000))
- seq_printf(seq, "%ld,%03ld", v/1000, v % 1000);
- else
- seq_printf(seq, "%ld", v);
-}
-
-static void drbd_get_syncer_progress(struct drbd_device *device,
- union drbd_dev_state state, unsigned long *rs_total,
- unsigned long *bits_left, unsigned int *per_mil_done)
-{
- /* this is to break it at compile time when we change that, in case we
- * want to support more than (1<<32) bits on a 32bit arch. */
- typecheck(unsigned long, device->rs_total);
- *rs_total = device->rs_total;
-
- /* note: both rs_total and rs_left are in bits, i.e. in
- * units of BM_BLOCK_SIZE.
- * for the percentage, we don't care. */
-
- if (state.conn == C_VERIFY_S || state.conn == C_VERIFY_T)
- *bits_left = device->ov_left;
- else
- *bits_left = drbd_bm_total_weight(device) - device->rs_failed;
- /* >> 10 to prevent overflow,
- * +1 to prevent division by zero */
- if (*bits_left > *rs_total) {
- /* D'oh. Maybe a logic bug somewhere. More likely just a race
- * between state change and reset of rs_total.
- */
- *bits_left = *rs_total;
- *per_mil_done = *rs_total ? 0 : 1000;
- } else {
- /* Make sure the division happens in long context.
- * We allow up to one petabyte storage right now,
- * at a granularity of 4k per bit that is 2**38 bits.
- * After shift right and multiplication by 1000,
- * this should still fit easily into a 32bit long,
- * so we don't need a 64bit division on 32bit arch.
- * Note: currently we don't support such large bitmaps on 32bit
- * arch anyways, but no harm done to be prepared for it here.
- */
- unsigned int shift = *rs_total > UINT_MAX ? 16 : 10;
- unsigned long left = *bits_left >> shift;
- unsigned long total = 1UL + (*rs_total >> shift);
- unsigned long tmp = 1000UL - left * 1000UL/total;
- *per_mil_done = tmp;
- }
-}
-
-
-/*lge
- * progress bars shamelessly adapted from driver/md/md.c
- * output looks like
- * [=====>..............] 33.5% (23456/123456)
- * finish: 2:20:20 speed: 6,345 (6,456) K/sec
- */
-static void drbd_syncer_progress(struct drbd_device *device, struct seq_file *seq,
- union drbd_dev_state state)
-{
- unsigned long db, dt, dbdt, rt, rs_total, rs_left;
- unsigned int res;
- int i, x, y;
- int stalled = 0;
-
- drbd_get_syncer_progress(device, state, &rs_total, &rs_left, &res);
-
- x = res/50;
- y = 20-x;
- seq_puts(seq, "\t[");
- for (i = 1; i < x; i++)
- seq_putc(seq, '=');
- seq_putc(seq, '>');
- for (i = 0; i < y; i++)
- seq_putc(seq, '.');
- seq_puts(seq, "] ");
-
- if (state.conn == C_VERIFY_S || state.conn == C_VERIFY_T)
- seq_puts(seq, "verified:");
- else
- seq_puts(seq, "sync'ed:");
- seq_printf(seq, "%3u.%u%% ", res / 10, res % 10);
-
- /* if more than a few GB, display in MB */
- if (rs_total > (4UL << (30 - BM_BLOCK_SHIFT)))
- seq_printf(seq, "(%lu/%lu)M",
- (unsigned long) Bit2KB(rs_left >> 10),
- (unsigned long) Bit2KB(rs_total >> 10));
- else
- seq_printf(seq, "(%lu/%lu)K",
- (unsigned long) Bit2KB(rs_left),
- (unsigned long) Bit2KB(rs_total));
-
- seq_puts(seq, "\n\t");
-
- /* see drivers/md/md.c
- * We do not want to overflow, so the order of operands and
- * the * 100 / 100 trick are important. We do a +1 to be
- * safe against division by zero. We only estimate anyway.
- *
- * dt: time from mark until now
- * db: blocks written from mark until now
- * rt: remaining time
- */
- /* Rolling marks. last_mark+1 may just now be modified. last_mark+2 is
- * at least (DRBD_SYNC_MARKS-2)*DRBD_SYNC_MARK_STEP old, and has at
- * least DRBD_SYNC_MARK_STEP time before it will be modified. */
- /* ------------------------ ~18s average ------------------------ */
- i = (device->rs_last_mark + 2) % DRBD_SYNC_MARKS;
- dt = (jiffies - device->rs_mark_time[i]) / HZ;
- if (dt > 180)
- stalled = 1;
-
- if (!dt)
- dt++;
- db = device->rs_mark_left[i] - rs_left;
- rt = (dt * (rs_left / (db/100+1)))/100; /* seconds */
-
- seq_printf(seq, "finish: %lu:%02lu:%02lu",
- rt / 3600, (rt % 3600) / 60, rt % 60);
-
- dbdt = Bit2KB(db/dt);
- seq_puts(seq, " speed: ");
- seq_printf_with_thousands_grouping(seq, dbdt);
- seq_puts(seq, " (");
- /* ------------------------- ~3s average ------------------------ */
- if (drbd_proc_details >= 1) {
- /* this is what drbd_rs_should_slow_down() uses */
- i = (device->rs_last_mark + DRBD_SYNC_MARKS-1) % DRBD_SYNC_MARKS;
- dt = (jiffies - device->rs_mark_time[i]) / HZ;
- if (!dt)
- dt++;
- db = device->rs_mark_left[i] - rs_left;
- dbdt = Bit2KB(db/dt);
- seq_printf_with_thousands_grouping(seq, dbdt);
- seq_puts(seq, " -- ");
- }
-
- /* --------------------- long term average ---------------------- */
- /* mean speed since syncer started
- * we do account for PausedSync periods */
- dt = (jiffies - device->rs_start - device->rs_paused) / HZ;
- if (dt == 0)
- dt = 1;
- db = rs_total - rs_left;
- dbdt = Bit2KB(db/dt);
- seq_printf_with_thousands_grouping(seq, dbdt);
- seq_putc(seq, ')');
-
- if (state.conn == C_SYNC_TARGET ||
- state.conn == C_VERIFY_S) {
- seq_puts(seq, " want: ");
- seq_printf_with_thousands_grouping(seq, device->c_sync_rate);
- }
- seq_printf(seq, " K/sec%s\n", stalled ? " (stalled)" : "");
-
- if (drbd_proc_details >= 1) {
- /* 64 bit:
- * we convert to sectors in the display below. */
- unsigned long bm_bits = drbd_bm_bits(device);
- unsigned long bit_pos;
- unsigned long long stop_sector = 0;
- if (state.conn == C_VERIFY_S ||
- state.conn == C_VERIFY_T) {
- bit_pos = bm_bits - device->ov_left;
- if (verify_can_do_stop_sector(device))
- stop_sector = device->ov_stop_sector;
- } else
- bit_pos = device->bm_resync_fo;
- /* Total sectors may be slightly off for oddly
- * sized devices. So what. */
- seq_printf(seq,
- "\t%3d%% sector pos: %llu/%llu",
- (int)(bit_pos / (bm_bits/100+1)),
- (unsigned long long)bit_pos * BM_SECT_PER_BIT,
- (unsigned long long)bm_bits * BM_SECT_PER_BIT);
- if (stop_sector != 0 && stop_sector != ULLONG_MAX)
- seq_printf(seq, " stop sector: %llu", stop_sector);
- seq_putc(seq, '\n');
- }
-}
-
int drbd_seq_show(struct seq_file *seq, void *v)
{
- int i, prev_i = -1;
- const char *sn;
- struct drbd_device *device;
- struct net_conf *nc;
- union drbd_dev_state state;
- char wp;
-
- static char write_ordering_chars[] = {
- [WO_NONE] = 'n',
- [WO_DRAIN_IO] = 'd',
- [WO_BDEV_FLUSH] = 'f',
- };
-
- seq_printf(seq, "version: " REL_VERSION " (api:%d/proto:%d-%d)\n%s\n",
- DRBD_FAMILY_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX, drbd_buildtag());
-
+ bool any_legacy;
/*
- cs .. connection state
- ro .. node role (local/remote)
- ds .. disk state (local/remote)
- protocol
- various flags
- ns .. network send
- nr .. network receive
- dw .. disk write
- dr .. disk read
- al .. activity log write count
- bm .. bitmap update write count
- pe .. pending (waiting for ack or data reply)
- ua .. unack'd (still need to send ack or data reply)
- ap .. application requests accepted, but not yet completed
- ep .. number of epochs currently "on the fly", P_BARRIER_ACK pending
- wo .. write ordering mode currently in use
- oos .. known out-of-sync kB
- */
-
- rcu_read_lock();
- idr_for_each_entry(&drbd_devices, device, i) {
- if (prev_i != i - 1)
- seq_putc(seq, '\n');
- prev_i = i;
-
- state = device->state;
- sn = drbd_conn_str(state.conn);
-
- if (state.conn == C_STANDALONE &&
- state.disk == D_DISKLESS &&
- state.role == R_SECONDARY) {
- seq_printf(seq, "%2d: cs:Unconfigured\n", i);
- } else {
- /* reset device->congestion_reason */
-
- nc = rcu_dereference(first_peer_device(device)->connection->net_conf);
- wp = nc ? nc->wire_protocol - DRBD_PROT_A + 'A' : ' ';
- seq_printf(seq,
- "%2d: cs:%s ro:%s/%s ds:%s/%s %c %c%c%c%c%c%c\n"
- " ns:%u nr:%u dw:%u dr:%u al:%u bm:%u "
- "lo:%d pe:%d ua:%d ap:%d ep:%d wo:%c",
- i, sn,
- drbd_role_str(state.role),
- drbd_role_str(state.peer),
- drbd_disk_str(state.disk),
- drbd_disk_str(state.pdsk),
- wp,
- drbd_suspended(device) ? 's' : 'r',
- state.aftr_isp ? 'a' : '-',
- state.peer_isp ? 'p' : '-',
- state.user_isp ? 'u' : '-',
- device->congestion_reason ?: '-',
- test_bit(AL_SUSPENDED, &device->flags) ? 's' : '-',
- device->send_cnt/2,
- device->recv_cnt/2,
- device->writ_cnt/2,
- device->read_cnt/2,
- device->al_writ_cnt,
- device->bm_writ_cnt,
- atomic_read(&device->local_cnt),
- atomic_read(&device->ap_pending_cnt) +
- atomic_read(&device->rs_pending_cnt),
- atomic_read(&device->unacked_cnt),
- atomic_read(&device->ap_bio_cnt),
- first_peer_device(device)->connection->epochs,
- write_ordering_chars[device->resource->write_ordering]
- );
- seq_printf(seq, " oos:%llu\n",
- Bit2KB((unsigned long long)
- drbd_bm_total_weight(device)));
- }
- if (state.conn == C_SYNC_SOURCE ||
- state.conn == C_SYNC_TARGET ||
- state.conn == C_VERIFY_S ||
- state.conn == C_VERIFY_T)
- drbd_syncer_progress(device, seq, state);
-
- if (drbd_proc_details >= 1 && get_ldev_if_state(device, D_FAILED)) {
- lc_seq_printf_stats(seq, device->resync);
- lc_seq_printf_stats(seq, device->act_log);
- put_ldev(device);
- }
-
- if (drbd_proc_details >= 2)
- seq_printf(seq, "\tblocked on activity log: %d\n", atomic_read(&device->ap_actlog_cnt));
+ * drbd-utils pick their tools by the token after "version:". A v1
+ * build also names its real release as "core:". "(compat 8.4)" tells
+ * drbdadm that "new-resource" without a node id gives an 8.4
+ * compatibility resource; drbdadm-84 only acts on it after a 9.x token.
+ */
+ static const char legacy_info[] =
+#if defined(CONFIG_DRBD_COMPAT_84) && !defined(DRBD_NL_FAMILY_V2)
+ " (compat 8.4) core: " REL_VERSION;
+#elif defined(CONFIG_DRBD_COMPAT_84)
+ " (compat 8.4)";
+#else
+ "";
+#endif
+
+ seq_printf(seq, "version: " DRBD_PROC_VERSION " (api:%d/proto:%d-%d)%s\n%s\n",
+ DRBD_FAMILY_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX, legacy_info,
+ drbd_buildtag());
+
+ any_legacy = drbd_show_legacy_device(seq, v);
+ if (!any_legacy) {
+ /*
+ * DRBD 8 did not output the transport information, so do not
+ * display it if any resources are in DRBD 8 compatibility mode.
+ */
+ drbd_print_transports_loaded(seq);
}
- rcu_read_unlock();
-
return 0;
}
diff --git a/drivers/block/drbd/drbd_strings.c b/drivers/block/drbd/drbd_strings.c
index f9fde9defedb..2db43bd60365 100644
--- a/drivers/block/drbd/drbd_strings.c
+++ b/drivers/block/drbd/drbd_strings.c
@@ -16,23 +16,9 @@ static const char * const __conn_state_names[] = {
[C_BROKEN_PIPE] = "BrokenPipe",
[C_NETWORK_FAILURE] = "NetworkFailure",
[C_PROTOCOL_ERROR] = "ProtocolError",
- [C_WF_CONNECTION] = "WFConnection",
- [C_WF_REPORT_PARAMS] = "WFReportParams",
[C_TEAR_DOWN] = "TearDown",
- [C_CONNECTED] = "Connected",
- [C_STARTING_SYNC_S] = "StartingSyncS",
- [C_STARTING_SYNC_T] = "StartingSyncT",
- [C_WF_BITMAP_S] = "WFBitMapS",
- [C_WF_BITMAP_T] = "WFBitMapT",
- [C_WF_SYNC_UUID] = "WFSyncUUID",
- [C_SYNC_SOURCE] = "SyncSource",
- [C_SYNC_TARGET] = "SyncTarget",
- [C_PAUSED_SYNC_S] = "PausedSyncS",
- [C_PAUSED_SYNC_T] = "PausedSyncT",
- [C_VERIFY_S] = "VerifyS",
- [C_VERIFY_T] = "VerifyT",
- [C_AHEAD] = "Ahead",
- [C_BEHIND] = "Behind",
+ [C_CONNECTING] = "Connecting",
+ [C_CONNECTED] = "Connected",
};
struct state_names drbd_conn_state_names = {
@@ -40,6 +26,29 @@ struct state_names drbd_conn_state_names = {
.size = ARRAY_SIZE(__conn_state_names),
};
+static const char * const __repl_state_names[] = {
+ [L_OFF] = "Off",
+ [L_ESTABLISHED] = "Established",
+ [L_STARTING_SYNC_S] = "StartingSyncS",
+ [L_STARTING_SYNC_T] = "StartingSyncT",
+ [L_WF_BITMAP_S] = "WFBitMapS",
+ [L_WF_BITMAP_T] = "WFBitMapT",
+ [L_WF_SYNC_UUID] = "WFSyncUUID",
+ [L_SYNC_SOURCE] = "SyncSource",
+ [L_SYNC_TARGET] = "SyncTarget",
+ [L_VERIFY_S] = "VerifyS",
+ [L_VERIFY_T] = "VerifyT",
+ [L_PAUSED_SYNC_S] = "PausedSyncS",
+ [L_PAUSED_SYNC_T] = "PausedSyncT",
+ [L_AHEAD] = "Ahead",
+ [L_BEHIND] = "Behind",
+};
+
+struct state_names drbd_repl_state_names = {
+ .names = __repl_state_names,
+ .size = ARRAY_SIZE(__repl_state_names),
+};
+
static const char * const __role_state_names[] = {
[R_UNKNOWN] = "Unknown",
[R_PRIMARY] = "Primary",
@@ -54,6 +63,7 @@ struct state_names drbd_role_state_names = {
static const char * const __disk_state_names[] = {
[D_DISKLESS] = "Diskless",
[D_ATTACHING] = "Attaching",
+ [D_DETACHING] = "Detaching",
[D_FAILED] = "Failed",
[D_NEGOTIATING] = "Negotiating",
[D_INCONSISTENT] = "Inconsistent",
@@ -82,13 +92,20 @@ static const char * const __error_messages[] = {
[-SS_DEVICE_IN_USE] = "Device is held open by someone",
[-SS_NO_NET_CONFIG] = "Have no net/connection configuration",
[-SS_NO_VERIFY_ALG] = "Need a verify algorithm to start online verify",
- [-SS_NEED_CONNECTION] = "Need a connection to start verify or resync",
+ [-SS_NEED_CONNECTION] = "State change requires a connection",
[-SS_NOT_SUPPORTED] = "Peer does not support protocol",
[-SS_LOWER_THAN_OUTDATED] = "Disk state is lower than outdated",
[-SS_IN_TRANSIENT_STATE] = "In transient state, retry after next state change",
[-SS_CONCURRENT_ST_CHG] = "Concurrent state changes detected and aborted",
- [-SS_OUTDATE_WO_CONN] = "Need a connection for a graceful disconnect/outdate peer",
[-SS_O_VOL_PEER_PRI] = "Other vol primary on peer not allowed by config",
+ [-SS_PRIMARY_READER] = "Peer may not become primary while device is opened read-only",
+ [-SS_INTERRUPTED] = "Interrupted state change",
+ [-SS_TIMEOUT] = "Timeout in operation",
+ [-SS_WEAKLY_CONNECTED] = "Primary nodes must be strongly connected among each other",
+ [-SS_NO_QUORUM] = "No quorum",
+ [-SS_ATTACH_NO_BITMAP] = "Intentional diskless peer may not attach a disk",
+ [-SS_HANDSHAKE_DISCONNECT] = "Disconnect chosen in handshake",
+ [-SS_HANDSHAKE_RETRY] = "Retry chosen in handshake",
};
struct state_names drbd_error_messages = {
@@ -140,12 +157,45 @@ static const char * const __packet_names[] = {
[P_OUT_OF_SYNC] = "P_OUT_OF_SYNC",
[P_RETRY_WRITE] = "P_RETRY_WRITE",
[P_RS_CANCEL] = "P_RS_CANCEL",
+ [P_RS_CANCEL_AHEAD] = "P_RS_CANCEL_AHEAD",
[P_CONN_ST_CHG_REQ] = "P_CONN_ST_CHG_REQ",
[P_CONN_ST_CHG_REPLY] = "P_CONN_ST_CHG_REPLY",
[P_PROTOCOL_UPDATE] = "P_PROTOCOL_UPDATE",
+ [P_TWOPC_PREPARE] = "P_TWOPC_PREPARE",
+ [P_TWOPC_ABORT] = "P_TWOPC_ABORT",
+ [P_DAGTAG] = "P_DAGTAG",
[P_RS_THIN_REQ] = "P_RS_THIN_REQ",
[P_RS_DEALLOCATED] = "P_RS_DEALLOCATED",
+ [P_TWOPC_PREP_RSZ] = "P_TWOPC_PREP_RSZ",
[P_ZEROES] = "P_ZEROES",
+ [P_PEER_ACK] = "P_PEER_ACK",
+ [P_PEERS_IN_SYNC] = "P_PEERS_IN_SYNC",
+ [P_UUIDS110] = "P_UUIDS110",
+ [P_PEER_DAGTAG] = "P_PEER_DAGTAG",
+ [P_CURRENT_UUID] = "P_CURRENT_UUID",
+ [P_TWOPC_COMMIT] = "P_TWOPC_COMMIT",
+ [P_TWOPC_YES] = "P_TWOPC_YES",
+ [P_TWOPC_NO] = "P_TWOPC_NO",
+ [P_TWOPC_RETRY] = "P_TWOPC_RETRY",
+ [P_CONFIRM_STABLE] = "P_CONFIRM_STABLE",
+ [P_DISCONNECT] = "P_DISCONNECT",
+ [P_RS_DAGTAG_REQ] = "P_RS_DAGTAG_REQ",
+ [P_RS_CSUM_DAGTAG_REQ] = "P_RS_CSUM_DAGTAG_REQ",
+ [P_RS_THIN_DAGTAG_REQ] = "P_RS_THIN_DAGTAG_REQ",
+ [P_OV_DAGTAG_REQ] = "P_OV_DAGTAG_REQ",
+ [P_OV_DAGTAG_REPLY] = "P_OV_DAGTAG_REPLY",
+ [P_WRITE_ACK_IN_SYNC] = "P_WRITE_ACK_IN_SYNC",
+ [P_RS_NEG_ACK] = "P_RS_NEG_ACK",
+ [P_OV_RESULT_ID] = "P_OV_RESULT_ID",
+ [P_RS_DEALLOCATED_ID] = "P_RS_DEALLOCATED_ID",
+ [P_FLUSH_REQUESTS] = "P_FLUSH_REQUESTS",
+ [P_FLUSH_FORWARD] = "P_FLUSH_FORWARD",
+ [P_FLUSH_REQUESTS_ACK] = "P_FLUSH_REQUESTS_ACK",
+ [P_ENABLE_REPLICATION_NEXT] = "P_ENABLE_REPLICATION_NEXT",
+ [P_ENABLE_REPLICATION] = "P_ENABLE_REPLICATION",
+ [P_RS_DAGTAG_WAIT_REQ] = "P_RS_DAGTAG_WAIT_REQ",
+ [P_RS_DAGTAG_REACHED] = "P_RS_DAGTAG_REACHED",
+ [P_RS_DAGTAG_UNREACHABLE] = "P_RS_DAGTAG_UNREACHABLE",
/* enum drbd_packet, but not commands - obsoleted flags:
* P_MAY_IGNORE
* P_MAX_OPT_CMD
@@ -157,32 +207,39 @@ struct state_names drbd_packet_names = {
.size = ARRAY_SIZE(__packet_names),
};
-const char *drbd_conn_str(enum drbd_conns s)
+const char *drbd_repl_str(enum drbd_repl_state s)
+{
+ return (s < 0 || s >= drbd_repl_state_names.size ||
+ !drbd_repl_state_names.names[s]) ?
+ "?" : drbd_repl_state_names.names[s];
+}
+
+const char *drbd_conn_str(enum drbd_conn_state s)
{
return (s < 0 || s >= drbd_conn_state_names.size ||
!drbd_conn_state_names.names[s]) ?
- "?" : drbd_conn_state_names.names[s];
+ "?" : drbd_conn_state_names.names[s];
}
const char *drbd_role_str(enum drbd_role s)
{
return (s < 0 || s >= drbd_role_state_names.size ||
!drbd_role_state_names.names[s]) ?
- "?" : drbd_role_state_names.names[s];
+ "?" : drbd_role_state_names.names[s];
}
const char *drbd_disk_str(enum drbd_disk_state s)
{
return (s < 0 || s >= drbd_disk_state_names.size ||
!drbd_disk_state_names.names[s]) ?
- "?" : drbd_disk_state_names.names[s];
+ "?" : drbd_disk_state_names.names[s];
}
const char *drbd_set_st_err_str(enum drbd_state_rv err)
{
return (-err < 0 || -err >= drbd_error_messages.size ||
!drbd_error_messages.names[-err]) ?
- "?" : drbd_error_messages.names[-err];
+ "?" : drbd_error_messages.names[-err];
}
const char *drbd_packet_name(enum drbd_packet cmd)
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v3 20/20] drbd: remove BROKEN for DRBD
2026-11-10 16:26 [PATCH v3 00/20] drbd: DRBD 9 rework Christoph Böhmwalder
` (18 preceding siblings ...)
2026-11-10 16:26 ` [PATCH v3 19/20] drbd: update monitoring interfaces for multi-peer topology Christoph Böhmwalder
@ 2026-11-10 16:26 ` Christoph Böhmwalder
2026-11-11 0:34 ` kernel test robot
19 siblings, 1 reply; 22+ messages in thread
From: Christoph Böhmwalder @ 2026-11-10 16:26 UTC (permalink / raw)
To: Jens Axboe
Cc: drbd-dev, linux-kernel, Lars Ellenberg, Philipp Reisner,
linux-block, Christoph Böhmwalder
Remove the BROKEN dependency now that the DRBD 9 rework is complete
and the driver compiles cleanly again.
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
drivers/block/drbd/Kconfig | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/block/drbd/Kconfig b/drivers/block/drbd/Kconfig
index 261965d7b054..06be6ce164f5 100644
--- a/drivers/block/drbd/Kconfig
+++ b/drivers/block/drbd/Kconfig
@@ -9,7 +9,6 @@ comment "DRBD disabled because PROC_FS or INET not selected"
config BLK_DEV_DRBD
tristate "DRBD Distributed Replicated Block Device support"
depends on PROC_FS && INET
- depends on BROKEN
select LRU_CACHE
select CRC32
help
--
2.56.0
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v3 20/20] drbd: remove BROKEN for DRBD
2026-11-10 16:26 ` [PATCH v3 20/20] drbd: remove BROKEN for DRBD Christoph Böhmwalder
@ 2026-11-11 0:34 ` kernel test robot
0 siblings, 0 replies; 22+ messages in thread
From: kernel test robot @ 2026-11-11 0:34 UTC (permalink / raw)
To: Christoph Böhmwalder, Jens Axboe
Cc: oe-kbuild-all, drbd-dev, linux-kernel, Lars Ellenberg,
Philipp Reisner, linux-block, Christoph Böhmwalder
Hi Christoph,
kernel test robot noticed the following build warnings:
[auto build test WARNING on aa7014f8d7565cfea117eac36a5de0a05eba65a4]
url: https://github.com/intel-lab-lkp/linux/commits/Christoph-B-hmwalder/drbd-mark-as-BROKEN-during-DRBD-9-rework/20261110-172626
base: aa7014f8d7565cfea117eac36a5de0a05eba65a4
patch link: https://lore.kernel.org/r/20261110162646.3028472-21-christoph.boehmwalder%40linbit.com
patch subject: [PATCH v3 20/20] drbd: remove BROKEN for DRBD
config: m68k-defconfig (https://download.01.org/0day-ci/archive/20261111/202611110819.YiyLkWx7-lkp@intel.com/config)
compiler: m68k-linux-gcc (GCC) 16.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261111/202611110819.YiyLkWx7-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202611110819.YiyLkWx7-lkp@intel.com/
All warnings (new ones prefixed by >>):
>> Warning: drivers/block/drbd/drbd_transport_tcp.c:670 function parameter 'transport' not described in 'dtt_socket_ok_or_free'
>> Warning: drivers/block/drbd/drbd_transport_tcp.c:670 function parameter 'what' not described in 'dtt_socket_ok_or_free'
>> Warning: drivers/block/drbd/drbd_transport_tcp.c:670 function parameter 'transport' not described in 'dtt_socket_ok_or_free'
>> Warning: drivers/block/drbd/drbd_transport_tcp.c:670 function parameter 'what' not described in 'dtt_socket_ok_or_free'
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 22+ messages in thread