From: James Simmons <jsimmons@infradead.org>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
devel@driverdev.osuosl.org,
Andreas Dilger <andreas.dilger@intel.com>,
Oleg Drokin <oleg.drokin@intel.com>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Lustre Development List <lustre-devel@lists.lustre.org>,
Liang Zhen <liang.zhen@intel.com>
Subject: [PATCH 25/27] staging: lustre: check wr_id returned by ib_poll_cq
Date: Wed, 2 Mar 2016 17:02:08 -0500 [thread overview]
Message-ID: <1456956130-6110-26-git-send-email-jsimmons@infradead.org> (raw)
In-Reply-To: <1456956130-6110-1-git-send-email-jsimmons@infradead.org>
From: Liang Zhen <liang.zhen@intel.com>
If ib_poll_cq returned +ve without initialising ib_wc::wr_id (bug
in driver), then o2iblnd will run into unpredictable situation
because ib_wc::wr_id may refer to stale tx/rx pointer in stack.
It indicates bug in HCA driver if this happened, ko2iblnd should
output console error then close current connection.
This patch could also be helpful for LU-5271
Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-519
Reviewed-on: http://review.whamcloud.com/12747
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
---
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 9 ++++---
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 21 ++++++++++++++++++-
2 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
index 3db1413..6a4c4ac 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
@@ -762,10 +762,11 @@ kiblnd_queue2str(kib_conn_t *conn, struct list_head *q)
/* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the */
/* lowest bits of the work request id to stash the work item type. */
-#define IBLND_WID_TX 0
-#define IBLND_WID_RDMA 1
-#define IBLND_WID_RX 2
-#define IBLND_WID_MASK 3UL
+#define IBLND_WID_INVAL 0
+#define IBLND_WID_TX 1
+#define IBLND_WID_RX 2
+#define IBLND_WID_RDMA 3
+#define IBLND_WID_MASK 3UL
static inline __u64
kiblnd_ptr2wreqid(void *ptr, int type)
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 9428166..1c62875 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -768,7 +768,6 @@ kiblnd_post_tx_locked(kib_conn_t *conn, kib_tx_t *tx, int credit)
int ver = conn->ibc_version;
int rc;
int done;
- struct ib_send_wr *bad_wrq;
LASSERT(tx->tx_queued);
/* We rely on this for QP sizing */
@@ -852,7 +851,14 @@ kiblnd_post_tx_locked(kib_conn_t *conn, kib_tx_t *tx, int credit)
/* close_conn will launch failover */
rc = -ENETDOWN;
} else {
- rc = ib_post_send(conn->ibc_cmid->qp, &tx->tx_wrq->wr, &bad_wrq);
+ struct ib_send_wr *wrq = &tx->tx_wrq[tx->tx_nwrq - 1].wr;
+
+ LASSERTF(wrq->wr_id == kiblnd_ptr2wreqid(tx, IBLND_WID_TX),
+ "bad wr_id %llx, opc %d, flags %d, peer: %s\n",
+ wrq->wr_id, wrq->opcode, wrq->send_flags,
+ libcfs_nid2str(conn->ibc_peer->ibp_nid));
+ wrq = NULL;
+ rc = ib_post_send(conn->ibc_cmid->qp, &tx->tx_wrq->wr, &wrq);
}
conn->ibc_last_send = jiffies;
@@ -3421,6 +3427,8 @@ kiblnd_scheduler(void *arg)
spin_unlock_irqrestore(&sched->ibs_lock, flags);
+ wc.wr_id = IBLND_WID_INVAL;
+
rc = ib_poll_cq(conn->ibc_cq, 1, &wc);
if (!rc) {
rc = ib_req_notify_cq(conn->ibc_cq,
@@ -3438,6 +3446,15 @@ kiblnd_scheduler(void *arg)
rc = ib_poll_cq(conn->ibc_cq, 1, &wc);
}
+ if (unlikely(rc > 0 && wc.wr_id == IBLND_WID_INVAL)) {
+ LCONSOLE_ERROR("ib_poll_cq (rc: %d) returned invalid wr_id, opcode %d, status: %d, vendor_err: %d, conn: %s status: %d\nplease upgrade firmware and OFED or contact vendor.\n",
+ rc, wc.opcode, wc.status,
+ wc.vendor_err,
+ libcfs_nid2str(conn->ibc_peer->ibp_nid),
+ conn->ibc_state);
+ rc = -EINVAL;
+ }
+
if (rc < 0) {
CWARN("%s: ib_poll_cq failed: %d, closing connection\n",
libcfs_nid2str(conn->ibc_peer->ibp_nid),
--
1.7.1
next prev parent reply other threads:[~2016-03-02 22:03 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-02 22:01 [PATCH 00/27] Third batch of LNet fixes James Simmons
2016-03-02 22:01 ` [PATCH 01/27] staging: lustre: set downis to 1 if there's no NI for remote net James Simmons
2016-03-02 22:01 ` [PATCH 02/27] staging: lustre: recv could access freed message James Simmons
2016-03-02 22:01 ` [PATCH 03/27] staging: lustre: Ignore hops if not explicitly set James Simmons
2016-03-02 22:01 ` [PATCH 04/27] staging: lustre: return proper error code for LNet core James Simmons
2016-03-02 22:01 ` [PATCH 05/27] staging: lustre: remove annoying message in parse_nidrange James Simmons
2016-03-02 22:01 ` [PATCH 06/27] staging: lustre: Use after free in lnet_ptl_match_delay() James Simmons
2016-03-02 22:01 ` [PATCH 07/27] staging: lustre: issue in the offset in lnet match hash table James Simmons
2016-03-02 22:01 ` [PATCH 08/27] staging: lustre: fix 'copy into fixed size buffer' errors James Simmons
2016-03-02 22:01 ` [PATCH 09/27] staging: lustre: set task state before scheduling in lnet_sock_accept James Simmons
2016-03-02 22:01 ` [PATCH 10/27] staging: lustre: replace direct LNet HZ access with kernel APIs James Simmons
2016-03-02 22:01 ` [PATCH 11/27] staging: lustre: bind socklnd peers to a specific CPT James Simmons
2016-03-02 22:01 ` [PATCH 12/27] staging: lustre: fix socklnd issues found by Klocwork Insight tool James Simmons
2016-03-02 22:01 ` [PATCH 13/27] staging: lustre: fix api-ni.c " James Simmons
2016-03-02 22:01 ` [PATCH 14/27] staging: lustre: fix conctl.c " James Simmons
2016-03-02 22:01 ` [PATCH 15/27] staging: lustre: fix framework.c " James Simmons
2016-03-02 22:01 ` [PATCH 16/27] staging: lustre: reverse LNet and infinband header order James Simmons
2016-03-02 22:02 ` [PATCH 17/27] staging: lustre: make o2iblnd local functions static James Simmons
2016-03-02 22:02 ` [PATCH 18/27] staging: lustre: make o2iblnd_cb.c " James Simmons
2016-03-02 22:02 ` [PATCH 19/27] staging: lustre: corrected some typos and grammar errors James Simmons
2016-03-02 22:02 ` [PATCH 20/27] staging: lustre: change ibh_mrs from array to pointer James Simmons
2016-03-02 22:02 ` [PATCH] staging: lustre: Support different ko2iblnd configs between systems James Simmons
2016-03-02 23:22 ` Greg Kroah-Hartman
2016-03-02 23:35 ` [lustre-devel] " Simmons, James A.
2016-03-02 22:02 ` [PATCH 22/27] staging: lustre: make ko2iblnd connect parameters persistent James Simmons
2016-03-02 22:02 ` [PATCH 23/27] staging: lustre: take extra refcount in kiblnd_connreq_done James Simmons
2016-03-02 22:02 ` [PATCH 24/27] staging: lustre: Change connect peer failed cleanup order James Simmons
2016-03-02 22:02 ` James Simmons [this message]
2016-03-02 22:02 ` [PATCH 26/27] staging: lustre: avoid intensive reconnecting for ko2iblnd James Simmons
2016-03-02 22:02 ` [PATCH 27/27] staging: lustre: do less intense allocating retry " James Simmons
2016-03-02 23:24 ` [PATCH 00/27] Third batch of LNet fixes Greg Kroah-Hartman
2016-03-02 23:52 ` [lustre-devel] " Simmons, James A.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1456956130-6110-26-git-send-email-jsimmons@infradead.org \
--to=jsimmons@infradead.org \
--cc=andreas.dilger@intel.com \
--cc=devel@driverdev.osuosl.org \
--cc=gregkh@linuxfoundation.org \
--cc=liang.zhen@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lustre-devel@lists.lustre.org \
--cc=oleg.drokin@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®