From: James Simmons <jsimmons@infradead.org>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
devel@driverdev.osuosl.org,
Andreas Dilger <andreas.dilger@intel.com>,
Oleg Drokin <oleg.drokin@intel.com>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Lustre Development List <lustre-devel@lists.lustre.org>,
Amir Shehata <amir.shehata@intel.com>
Subject: [PATCH 22/27] staging: lustre: make ko2iblnd connect parameters persistent
Date: Wed, 2 Mar 2016 17:02:05 -0500 [thread overview]
Message-ID: <1456956130-6110-23-git-send-email-jsimmons@infradead.org> (raw)
In-Reply-To: <1456956130-6110-1-git-send-email-jsimmons@infradead.org>
From: Amir Shehata <amir.shehata@intel.com>
Store map-on-demand and peertx credits in the peer, since the peer
is persistent. Also made sure that when assigning the parameters
received on the connection to the peer structure through create,
that if another peer is added before grabbing the lock we assign
these parameters to it as well.
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3322
Reviewed-on: http://review.whamcloud.com/17074
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
---
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 14 +++-----
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 6 +++-
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 38 ++++++++++++++------
3 files changed, 37 insertions(+), 21 deletions(-)
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
index 0b1ffbe..56c221b 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
@@ -335,6 +335,8 @@ int kiblnd_create_peer(lnet_ni_t *ni, kib_peer_t **peerp, lnet_nid_t nid)
peer->ibp_nid = nid;
peer->ibp_error = 0;
peer->ibp_last_alive = 0;
+ peer->ibp_max_frags = IBLND_CFG_RDMA_FRAGS;
+ peer->ibp_queue_depth = *kiblnd_tunables.kib_peertxcredits;
atomic_set(&peer->ibp_refcount, 1); /* 1 ref for caller */
INIT_LIST_HEAD(&peer->ibp_list); /* not in the peer table yet */
@@ -631,7 +633,7 @@ static int kiblnd_get_completion_vector(kib_conn_t *conn, int cpt)
}
kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, struct rdma_cm_id *cmid,
- int state, int version, kib_connparams_t *cp)
+ int state, int version)
{
/*
* CAVEAT EMPTOR:
@@ -685,14 +687,8 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, struct rdma_cm_id *cmid,
conn->ibc_peer = peer; /* I take the caller's ref */
cmid->context = conn; /* for future CM callbacks */
conn->ibc_cmid = cmid;
-
- if (!cp) {
- conn->ibc_max_frags = IBLND_CFG_RDMA_FRAGS;
- conn->ibc_queue_depth = *kiblnd_tunables.kib_peertxcredits;
- } else {
- conn->ibc_max_frags = cp->ibcp_max_frags;
- conn->ibc_queue_depth = cp->ibcp_queue_depth;
- }
+ conn->ibc_max_frags = peer->ibp_max_frags;
+ conn->ibc_queue_depth = peer->ibp_queue_depth;
INIT_LIST_HEAD(&conn->ibc_early_rxs);
INIT_LIST_HEAD(&conn->ibc_tx_noops);
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
index 59a26c4..3db1413 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
@@ -586,6 +586,10 @@ typedef struct kib_peer {
int ibp_error; /* errno on closing this peer */
unsigned long ibp_last_alive; /* when (in jiffies) I was last alive
*/
+ /* max map_on_demand */
+ __u16 ibp_max_frags;
+ /* max_peer_credits */
+ __u16 ibp_queue_depth;
} kib_peer_t;
extern kib_data_t kiblnd_data;
@@ -946,7 +950,7 @@ int kiblnd_close_stale_conns_locked(kib_peer_t *peer,
int kiblnd_close_peer_conns_locked(kib_peer_t *peer, int why);
kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, struct rdma_cm_id *cmid,
- int state, int version, kib_connparams_t *cp);
+ int state, int version);
void kiblnd_destroy_conn(kib_conn_t *conn);
void kiblnd_close_conn(kib_conn_t *conn, int error);
void kiblnd_close_conn_locked(kib_conn_t *conn, int error);
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 22420c0..fb3873a 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -2323,6 +2323,10 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob)
goto failed;
}
+ /* We have validated the peer's parameters so use those */
+ peer->ibp_max_frags = reqmsg->ibm_u.connparams.ibcp_max_frags;
+ peer->ibp_queue_depth = reqmsg->ibm_u.connparams.ibcp_queue_depth;
+
write_lock_irqsave(g_lock, flags);
peer2 = kiblnd_find_peer_locked(nid);
@@ -2361,6 +2365,14 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob)
peer2->ibp_accepting++;
kiblnd_peer_addref(peer2);
+ /**
+ * Race with kiblnd_launch_tx (active connect) to create peer
+ * so copy validated parameters since we now know what the
+ * peer's limits are
+ */
+ peer2->ibp_max_frags = peer->ibp_max_frags;
+ peer2->ibp_queue_depth = peer->ibp_queue_depth;
+
write_unlock_irqrestore(g_lock, flags);
kiblnd_peer_decref(peer);
peer = peer2;
@@ -2383,8 +2395,8 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob)
write_unlock_irqrestore(g_lock, flags);
}
- conn = kiblnd_create_conn(peer, cmid, IBLND_CONN_PASSIVE_WAIT, version,
- &reqmsg->ibm_u.connparams);
+ conn = kiblnd_create_conn(peer, cmid, IBLND_CONN_PASSIVE_WAIT,
+ version);
if (!conn) {
kiblnd_peer_connect_failed(peer, 0, -ENOMEM);
kiblnd_peer_decref(peer);
@@ -2397,8 +2409,8 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob)
* CM callback doesn't destroy cmid.
*/
conn->ibc_incarnation = reqmsg->ibm_srcstamp;
- conn->ibc_credits = reqmsg->ibm_u.connparams.ibcp_queue_depth;
- conn->ibc_reserved_credits = reqmsg->ibm_u.connparams.ibcp_queue_depth;
+ conn->ibc_credits = conn->ibc_queue_depth;
+ conn->ibc_reserved_credits = conn->ibc_queue_depth;
LASSERT(conn->ibc_credits + conn->ibc_reserved_credits +
IBLND_OOB_MSGS(version) <= IBLND_RX_MSGS(conn));
@@ -2407,10 +2419,8 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob)
kiblnd_init_msg(ackmsg, IBLND_MSG_CONNACK,
sizeof(ackmsg->ibm_u.connparams));
- ackmsg->ibm_u.connparams.ibcp_queue_depth =
- reqmsg->ibm_u.connparams.ibcp_queue_depth;
- ackmsg->ibm_u.connparams.ibcp_max_frags =
- reqmsg->ibm_u.connparams.ibcp_max_frags;
+ ackmsg->ibm_u.connparams.ibcp_queue_depth = conn->ibc_queue_depth;
+ ackmsg->ibm_u.connparams.ibcp_max_frags = conn->ibc_max_frags;
ackmsg->ibm_u.connparams.ibcp_max_msg_size = IBLND_MSG_SIZE;
kiblnd_pack_msg(ni, ackmsg, version, 0, nid, reqmsg->ibm_srcstamp);
@@ -2495,6 +2505,9 @@ kiblnd_reconnect(kib_conn_t *conn, int version,
break;
case IBLND_REJECT_RDMA_FRAGS:
+ if (!cp)
+ goto failed;
+
if (conn->ibc_max_frags <= cp->ibcp_max_frags) {
CNETERR("Unsupported max frags, peer supports %d\n",
cp->ibcp_max_frags);
@@ -2504,18 +2517,21 @@ kiblnd_reconnect(kib_conn_t *conn, int version,
goto failed;
}
- conn->ibc_max_frags = cp->ibcp_max_frags;
+ peer->ibp_max_frags = cp->ibcp_max_frags;
reason = "rdma fragments";
break;
case IBLND_REJECT_MSG_QUEUE_SIZE:
+ if (!cp)
+ goto failed;
+
if (conn->ibc_queue_depth <= cp->ibcp_queue_depth) {
CNETERR("Unsupported queue depth, peer supports %d\n",
cp->ibcp_queue_depth);
goto failed;
}
- conn->ibc_queue_depth = cp->ibcp_queue_depth;
+ peer->ibp_queue_depth = cp->ibcp_queue_depth;
reason = "queue depth";
break;
@@ -2796,7 +2812,7 @@ kiblnd_active_connect(struct rdma_cm_id *cmid)
read_unlock_irqrestore(&kiblnd_data.kib_global_lock, flags);
conn = kiblnd_create_conn(peer, cmid, IBLND_CONN_ACTIVE_CONNECT,
- version, NULL);
+ version);
if (!conn) {
kiblnd_peer_connect_failed(peer, 1, -ENOMEM);
kiblnd_peer_decref(peer); /* lose cmid's ref */
--
1.7.1
next prev parent reply other threads:[~2016-03-02 22:04 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-02 22:01 [PATCH 00/27] Third batch of LNet fixes James Simmons
2016-03-02 22:01 ` [PATCH 01/27] staging: lustre: set downis to 1 if there's no NI for remote net James Simmons
2016-03-02 22:01 ` [PATCH 02/27] staging: lustre: recv could access freed message James Simmons
2016-03-02 22:01 ` [PATCH 03/27] staging: lustre: Ignore hops if not explicitly set James Simmons
2016-03-02 22:01 ` [PATCH 04/27] staging: lustre: return proper error code for LNet core James Simmons
2016-03-02 22:01 ` [PATCH 05/27] staging: lustre: remove annoying message in parse_nidrange James Simmons
2016-03-02 22:01 ` [PATCH 06/27] staging: lustre: Use after free in lnet_ptl_match_delay() James Simmons
2016-03-02 22:01 ` [PATCH 07/27] staging: lustre: issue in the offset in lnet match hash table James Simmons
2016-03-02 22:01 ` [PATCH 08/27] staging: lustre: fix 'copy into fixed size buffer' errors James Simmons
2016-03-02 22:01 ` [PATCH 09/27] staging: lustre: set task state before scheduling in lnet_sock_accept James Simmons
2016-03-02 22:01 ` [PATCH 10/27] staging: lustre: replace direct LNet HZ access with kernel APIs James Simmons
2016-03-02 22:01 ` [PATCH 11/27] staging: lustre: bind socklnd peers to a specific CPT James Simmons
2016-03-02 22:01 ` [PATCH 12/27] staging: lustre: fix socklnd issues found by Klocwork Insight tool James Simmons
2016-03-02 22:01 ` [PATCH 13/27] staging: lustre: fix api-ni.c " James Simmons
2016-03-02 22:01 ` [PATCH 14/27] staging: lustre: fix conctl.c " James Simmons
2016-03-02 22:01 ` [PATCH 15/27] staging: lustre: fix framework.c " James Simmons
2016-03-02 22:01 ` [PATCH 16/27] staging: lustre: reverse LNet and infinband header order James Simmons
2016-03-02 22:02 ` [PATCH 17/27] staging: lustre: make o2iblnd local functions static James Simmons
2016-03-02 22:02 ` [PATCH 18/27] staging: lustre: make o2iblnd_cb.c " James Simmons
2016-03-02 22:02 ` [PATCH 19/27] staging: lustre: corrected some typos and grammar errors James Simmons
2016-03-02 22:02 ` [PATCH 20/27] staging: lustre: change ibh_mrs from array to pointer James Simmons
2016-03-02 22:02 ` [PATCH] staging: lustre: Support different ko2iblnd configs between systems James Simmons
2016-03-02 23:22 ` Greg Kroah-Hartman
2016-03-02 23:35 ` [lustre-devel] " Simmons, James A.
2016-03-02 22:02 ` James Simmons [this message]
2016-03-02 22:02 ` [PATCH 23/27] staging: lustre: take extra refcount in kiblnd_connreq_done James Simmons
2016-03-02 22:02 ` [PATCH 24/27] staging: lustre: Change connect peer failed cleanup order James Simmons
2016-03-02 22:02 ` [PATCH 25/27] staging: lustre: check wr_id returned by ib_poll_cq James Simmons
2016-03-02 22:02 ` [PATCH 26/27] staging: lustre: avoid intensive reconnecting for ko2iblnd James Simmons
2016-03-02 22:02 ` [PATCH 27/27] staging: lustre: do less intense allocating retry " James Simmons
2016-03-02 23:24 ` [PATCH 00/27] Third batch of LNet fixes Greg Kroah-Hartman
2016-03-02 23:52 ` [lustre-devel] " Simmons, James A.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1456956130-6110-23-git-send-email-jsimmons@infradead.org \
--to=jsimmons@infradead.org \
--cc=amir.shehata@intel.com \
--cc=andreas.dilger@intel.com \
--cc=devel@driverdev.osuosl.org \
--cc=gregkh@linuxfoundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lustre-devel@lists.lustre.org \
--cc=oleg.drokin@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®