[PATCH 3/3] blk-mq: Fix the queue freezing mechanism

[Bart Van Assche <bart.vanassche@sandisk.com>]

Ensure that blk_mq_queue_enter() waits if mq_freeze_depth is not
zero. Ensure that the update of mq_freeze_depth by blk_mq_freeze_queue()
is visible by all CPU cores before that function waits on
mq_usage_counter.

It is unfortunate that this patch introduces an smp_mb() in the
hot path (blk_mq_queue_enter()) but I have not yet found a way to
avoid this.

I came across this code while analyzing a lockup triggered by
deleting a SCSI host created by the SRP initiator immediately
followed by a relogin.

Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Tejun Heo <tj@kernel.org>
Cc: <stable@vger.kernel.org>
---
 block/blk-mq.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 2077f0d..e3ad411 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -83,8 +83,13 @@ static int blk_mq_queue_enter(struct request_queue *q, gfp_t gfp)
 	while (true) {
 		int ret;
 
-		if (percpu_ref_tryget_live(&q->mq_usage_counter))
-			return 0;
+		if (percpu_ref_tryget_live(&q->mq_usage_counter)) {
+			/* Order mq_use_counter and mq_freeze_depth accesses */
+			smp_mb();
+			if (!atomic_read(&q->mq_freeze_depth))
+				return 0;
+			percpu_ref_put(&q->mq_usage_counter);
+		}
 
 		if (!(gfp & __GFP_WAIT))
 			return -EBUSY;
@@ -136,6 +141,11 @@ static void blk_mq_freeze_queue_wait(struct request_queue *q)
 void blk_mq_freeze_queue(struct request_queue *q)
 {
 	blk_mq_freeze_queue_start(q);
+	/*
+	 * Ensure that the mq_freeze_depth update is visiable before
+	 * mq_use_counter is read.
+	 */
+	smp_mb();
 	blk_mq_freeze_queue_wait(q);
 }
 EXPORT_SYMBOL_GPL(blk_mq_freeze_queue);
-- 
2.1.4