From: James Simmons <jsimmons@infradead.org>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
devel@driverdev.osuosl.org,
Andreas Dilger <andreas.dilger@intel.com>,
Oleg Drokin <oleg.drokin@intel.com>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Lustre Development List <lustre-devel@lists.lustre.org>,
Doug Oucharek <doug.s.oucharek@intel.com>,
James Simmons <jsimmons@infradead.org>
Subject: [PATCH 80/80] staging: lustre: lnet: Stop Infinite CON RACE Condition
Date: Tue, 16 Aug 2016 16:19:33 -0400 [thread overview]
Message-ID: <1471378773-24590-81-git-send-email-jsimmons@infradead.org> (raw)
In-Reply-To: <1471378773-24590-1-git-send-email-jsimmons@infradead.org>
From: Doug Oucharek <doug.s.oucharek@intel.com>
In current code, when a CON RACE occurs, the passive side will
let the node with the higher NID value win the race.
We have a field case where a node can have a "stuck"
connection which never goes away and is the trigger of a
never-ending loop of re-connections.
This patch introduces a counter to how many times a
connection in a connecting state has been the cause of a CON RACE
rejection. After 20 times (constant MAX_CONN_RACES_BEFORE_ABORT),
we assume the connection is stuck and let the other side (with
lower NID) win.
Signed-off-by: Doug Oucharek <doug.s.oucharek@intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7646
Reviewed-on: http://review.whamcloud.com/19430
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 2 +
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 24 ++++++++++++++++---
2 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
index 078a0c3..fbc4f68 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
@@ -582,6 +582,8 @@ struct kib_peer {
unsigned short ibp_connecting;
/* reconnect this peer later */
unsigned short ibp_reconnecting:1;
+ /* counter of how many times we triggered a conn race */
+ unsigned char ibp_races;
/* # consecutive reconnection attempts to this peer */
unsigned int ibp_reconnected;
/* errno on closing this peer */
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 6d1b14a..430ff85 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -36,6 +36,8 @@
#include "o2iblnd.h"
+#define MAX_CONN_RACES_BEFORE_ABORT 20
+
static void kiblnd_peer_alive(struct kib_peer *peer);
static void kiblnd_peer_connect_failed(struct kib_peer *peer, int active, int error);
static void kiblnd_init_tx_msg(lnet_ni_t *ni, struct kib_tx *tx,
@@ -2405,23 +2407,37 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob)
goto failed;
}
- /* tie-break connection race in favour of the higher NID */
+ /*
+ * Tie-break connection race in favour of the higher NID.
+ * If we keep running into a race condition multiple times,
+ * we have to assume that the connection attempt with the
+ * higher NID is stuck in a connecting state and will never
+ * recover. As such, we pass through this if-block and let
+ * the lower NID connection win so we can move forward.
+ */
if (peer2->ibp_connecting &&
- nid < ni->ni_nid) {
+ nid < ni->ni_nid && peer2->ibp_races <
+ MAX_CONN_RACES_BEFORE_ABORT) {
+ peer2->ibp_races++;
write_unlock_irqrestore(g_lock, flags);
- CWARN("Conn race %s\n", libcfs_nid2str(peer2->ibp_nid));
+ CDEBUG(D_NET, "Conn race %s\n",
+ libcfs_nid2str(peer2->ibp_nid));
kiblnd_peer_decref(peer);
rej.ibr_why = IBLND_REJECT_CONN_RACE;
goto failed;
}
-
+ if (peer2->ibp_races >= MAX_CONN_RACES_BEFORE_ABORT)
+ CNETERR("Conn race %s: unresolved after %d attempts, letting lower NID win\n",
+ libcfs_nid2str(peer2->ibp_nid),
+ MAX_CONN_RACES_BEFORE_ABORT);
/**
* passive connection is allowed even this peer is waiting for
* reconnection.
*/
peer2->ibp_reconnecting = 0;
+ peer2->ibp_races = 0;
peer2->ibp_accepting++;
kiblnd_peer_addref(peer2);
--
1.7.1
prev parent reply other threads:[~2016-08-16 20:21 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-16 20:18 [PATCH 00/80] staging: lustre: majority of missing fixes for 2.6 release James Simmons
2016-08-16 20:18 ` [PATCH 01/80] staging: lustre: llite: add md_op_data parameter to ll_get_dir_page James Simmons
2016-08-16 20:18 ` [PATCH 02/80] staging: lustre: llite: remove comment from ll_dir_read James Simmons
2016-08-16 20:18 ` [PATCH 03/80] staging: lustre: llite: style cleanup for llite_internal.h James Simmons
2016-08-16 20:18 ` [PATCH 04/80] staging: lustre: llite: pass inode to ll_release_page James Simmons
2016-08-16 20:18 ` [PATCH 05/80] staging: lustre: llite: change remove parameter to bool James Simmons
2016-08-16 20:18 ` [PATCH 06/80] staging: lustre: mdc: don't take rpc lock for readdir case James Simmons
2016-08-16 20:18 ` [PATCH 07/80] staging: lustre: lmv: remove unused lmv_get_mea function James Simmons
2016-08-16 20:18 ` [PATCH 08/80] staging: lustre: lmv: remove duplicate MAX_HASH_* James Simmons
2016-08-16 20:18 ` [PATCH 09/80] staging: lustre: lmv: change handling of lmv striping information James Simmons
2016-08-16 20:18 ` [PATCH 10/80] staging: lustre: lmv: remove lmv_get_easize James Simmons
2016-08-16 20:18 ` [PATCH 11/80] staging: lustre: lmv: replace obd_free_memmd with lmv_free_memmd James Simmons
2016-08-16 20:18 ` [PATCH 12/80] staging: lustre: create striped directory James Simmons
2016-08-16 20:18 ` [PATCH 13/80] staging: lustre: llite: fix "getdirstripe" to show stripe info James Simmons
2016-08-16 20:18 ` [PATCH 14/80] staging: lustre: delete striped directory James Simmons
2016-08-16 20:18 ` [PATCH 15/80] staging: lustre: obdclass: fix lmd_parse() to handle comma-separated NIDs James Simmons
2016-08-16 20:18 ` [PATCH 16/80] staging: lustre: obdclass: bug fixes for lu_device_type handling James Simmons
2016-08-16 20:18 ` [PATCH 17/80] staging: lustre: add ability to migrate inodes James Simmons
2016-08-16 20:18 ` [PATCH 18/80] staging: lustre: lmv: fix issue found by Klocwork Insight tool James Simmons
2016-08-16 20:18 ` [PATCH 19/80] staging: lustre: libcfs: Only dump log once per sec. to avoid EEXIST James Simmons
2016-08-16 20:18 ` [PATCH 20/80] staging: lustre: llite: enable clients to inject error for lfsck James Simmons
2016-08-16 20:18 ` [PATCH 21/80] staging: lustre: osc: allow to call brw_commit() multiple times James Simmons
2016-08-16 20:18 ` [PATCH 22/80] staging: lustre: llite: a few fixes for migration James Simmons
2016-08-16 20:18 ` [PATCH 23/80] staging: lustre: mdc: fixup MDS_SWAP_LAYOUTS ELC handling James Simmons
2016-08-16 20:18 ` [PATCH 24/80] staging: lustre: don't need to const __u64 parameters for lustre_idl.h James Simmons
2016-08-16 20:18 ` [PATCH 25/80] staging: lustre: const correct FID/OSTID/... helpers James Simmons
2016-08-16 20:18 ` [PATCH 26/80] staging: lustre: use bool for several function in lustre_idl.h/lustre_fid.h James Simmons
2016-08-16 20:18 ` [PATCH 27/80] staging: lustre: simplify inline functions in lustre_fid.h James Simmons
2016-08-16 20:18 ` [PATCH 28/80] staging: lustre: lmv: access lum_stripe_offset as little endian James Simmons
2016-08-16 20:18 ` [PATCH 29/80] staging: lustre: lmv: lookup remote migrating object in LMV James Simmons
2016-08-16 20:18 ` [PATCH 30/80] staging: lustre: lmv: Ensure lmv_intent_lookup cleans up reqp James Simmons
2016-08-16 20:18 ` [PATCH 31/80] staging: lustre: llite: avoid a deadlock in page write James Simmons
2016-08-16 20:18 ` [PATCH 32/80] staging: lustre: lov: handle the case of stripe size is not power 2 James Simmons
2016-08-16 20:18 ` [PATCH 33/80] staging: lustre: lmv: cleanup req in lmv_getattr_name() James Simmons
2016-08-16 20:18 ` [PATCH 34/80] staging: lustre: lmv: rename request to preq " James Simmons
2016-08-16 20:18 ` [PATCH 35/80] staging: lustre: obdclass: unified flow control interfaces James Simmons
2016-08-16 20:18 ` [PATCH 36/80] staging: lustre: reorder LOV_MAGIC_* definition James Simmons
2016-08-16 20:18 ` [PATCH 37/80] staging: lustre: ldlm: flock completion fixes James Simmons
2016-08-16 20:18 ` [PATCH 38/80] staging: lustre: move ioctls to lustre_ioctl.h James Simmons
2016-08-16 20:18 ` [PATCH 39/80] staging: lustre: llite: add error handler in inode prepare phase James Simmons
2016-08-16 20:18 ` [PATCH 40/80] staging: lustre: ptlrpc: Early replies need to honor at_max James Simmons
2016-08-16 20:18 ` [PATCH 41/80] staging: lustre: lmv: separate master object with master stripe James Simmons
2018-02-09 1:39 ` NeilBrown
2018-02-09 2:01 ` Oleg Drokin
2018-02-09 3:10 ` NeilBrown
2018-02-09 3:50 ` [lustre-devel] " Oleg Drokin
2018-02-10 20:57 ` James Simmons
2018-02-11 23:50 ` NeilBrown
2018-02-12 0:06 ` Oleg Drokin
2018-02-11 23:44 ` NeilBrown
2018-02-12 0:52 ` Oleg Drokin
2018-02-10 22:14 ` James Simmons
2018-02-12 7:41 ` Dan Carpenter
2016-08-16 20:18 ` [PATCH 42/80] staging: lustre: llite: validate names James Simmons
2016-08-16 20:18 ` [PATCH 43/80] staging: lustre: llite: fix inconsistencies of root squash feature James Simmons
2016-08-16 20:18 ` [PATCH 44/80] staging: lustre: Remove static declaration in anonymous union James Simmons
2016-08-16 20:18 ` [PATCH 45/80] staging: lustre: llite: Fix the deadlock in balance_dirty_pages() James Simmons
2016-08-16 20:18 ` [PATCH 46/80] staging: lustre: llite: Change readdir BRW metrics James Simmons
2016-08-16 20:19 ` [PATCH 47/80] staging: lustre: uapi: reduce scope of lustre_idl.h James Simmons
2016-08-16 20:19 ` [PATCH 48/80] staging: lustre: llite: a few fixes about readdir of striped dir James Simmons
2016-08-16 20:19 ` [PATCH 49/80] staging: lustre: lmv: validate lock with correct stripe FID James Simmons
2016-08-16 20:19 ` [PATCH 50/80] staging: lustre: lov: new pattern flag for partially repaired file James Simmons
2016-08-16 20:19 ` [PATCH 51/80] staging: lustre: lmv: Match MDT where the FID locates first James Simmons
2016-08-16 20:19 ` [PATCH 52/80] staging: lustre: llite: use the correct mode for striped directory James Simmons
2016-08-16 20:19 ` [PATCH 53/80] staging: lustre: obd: rename lsr_padding to lsr_valid James Simmons
2016-08-16 20:19 ` [PATCH 54/80] staging: lustre: llite: set dir LOV xattr length variable James Simmons
2016-08-16 20:19 ` [PATCH 55/80] staging: lustre: mdt: add mbo_ prefix to members of struct mdt_body James Simmons
2016-08-16 20:19 ` [PATCH 56/80] staging: lustre: clio: Reduce memory overhead of per-page allocation James Simmons
2016-08-16 20:19 ` [PATCH 57/80] staging: lustre: osc: revise unstable pages accounting James Simmons
2016-10-16 15:14 ` Greg Kroah-Hartman
2016-10-16 17:16 ` Oleg Drokin
2016-08-16 20:19 ` [PATCH 58/80] staging: lustre: mdc: always use D_INFO for debug info when mdc_put_rpc_lock fails James Simmons
2016-08-16 20:19 ` [PATCH 59/80] staging: lustre: fld: add fld description documentation James Simmons
2016-08-16 20:19 ` [PATCH 60/80] staging: lustre: ldlm: improve ldlm_lock_create() return value James Simmons
2016-08-16 20:19 ` [PATCH 61/80] staging: lustre: obdclass: compile issues with variable not being initialized James Simmons
2016-08-16 20:19 ` [PATCH 62/80] staging: lustre: obd: limit lu_object cache James Simmons
2016-08-16 20:19 ` [PATCH 63/80] staging: lustre: fid: do open-by-fid by default James Simmons
2016-08-16 20:19 ` [PATCH 64/80] staging: lustre: ptlrpc: add OBD_CONNECT_UNLINK_CLOSE flag James Simmons
2016-08-16 20:19 ` [PATCH 65/80] staging: lustre: llog: keep llog ctxt indices constant James Simmons
2016-08-16 20:19 ` [PATCH 66/80] staging: lustre: lmv: try all stripes for unknown hash functions James Simmons
2016-08-16 20:19 ` [PATCH 67/80] staging: lustre: ptlrpc: request gets stuck in UNREGISTERING phase James Simmons
2016-08-16 20:19 ` [PATCH 68/80] staging: lustre: lmv: build master LMV EA dynamically build via readdir James Simmons
2016-08-16 20:19 ` [PATCH 69/80] staging: lustre: osc: Automatically increase the max_dirty_mb James Simmons
2016-08-16 20:19 ` [PATCH 70/80] staging: lustre: include: fix one off errors in lustre_id.h James Simmons
2016-08-16 20:19 ` [PATCH 71/80] staging: lustre: llite: remove assert for acl refcount James Simmons
2016-08-16 20:19 ` [PATCH 72/80] staging: lustre: obd: validate open handle cookies James Simmons
2016-08-16 20:19 ` [PATCH 73/80] staging: lustre: lmv: build error with gcc 4.7.0 20110509 James Simmons
2016-08-16 20:19 ` [PATCH 74/80] staging: lustre: obd: implement md_read_page James Simmons
2016-08-16 20:19 ` [PATCH 75/80] staging: lustre: llite: set op_max_pages James Simmons
2016-08-16 20:19 ` [PATCH 76/80] staging: lustre: lnet: Do not drop message when shutting down LNet James Simmons
2016-08-16 20:19 ` [PATCH 77/80] staging: lustre: lnet: Correct position of lnet_ni_decref() James Simmons
2016-08-16 20:19 ` [PATCH 78/80] staging: lustre: lnet: make connection more stable with packet loss James Simmons
2016-08-16 20:19 ` [PATCH 79/80] staging: lustre: lnet: lock improvement for ko2iblnd James Simmons
2016-08-16 20:19 ` James Simmons [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1471378773-24590-81-git-send-email-jsimmons@infradead.org \
--to=jsimmons@infradead.org \
--cc=andreas.dilger@intel.com \
--cc=devel@driverdev.osuosl.org \
--cc=doug.s.oucharek@intel.com \
--cc=gregkh@linuxfoundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lustre-devel@lists.lustre.org \
--cc=oleg.drokin@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®