From: Bart Van Assche <bart.vanassche@sandisk.com>
To: Tejun Heo <tj@kernel.org>
Cc: Ming Lei <tom.leiming@gmail.com>, Jens Axboe <axboe@fb.com>,
"Christoph Hellwig" <hch@lst.de>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Akinobu Mita <akinobu.mita@gmail.com>
Subject: Re: [PATCH 3/3] blk-mq: Fix the queue freezing mechanism
Date: Thu, 24 Sep 2015 11:09:33 -0700 [thread overview]
Message-ID: <56043C5D.3090307@sandisk.com> (raw)
In-Reply-To: <20150924174938.GC25415@mtj.duckdns.org>
On 09/24/2015 10:49 AM, Tejun Heo wrote:
> Again, that doesn't happen.
Hello Tejun,
In case anyone would be interested, the backtraces for the lockup I had
observed are as follows:
sysrq: SysRq : Show Blocked State
task PC stack pid father
kworker/4:0 D ffff88045c5d5a00 0 29 2 0x00000000
Workqueue: srp_remove srp_remove_work [ib_srp]
ffff88045c767c08 0000000000000086 ffffffff810ba11d ffff88047fd15ad8
ffff88045c5d5a00 ffff88045c768000 ffff88045c768000 ffff88041c737ab8
ffff880036f7bcf8 ffff8804158b55e8 0000000000000100 ffff88045c767c20
Call Trace:
[<ffffffff810ba11d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff81271a36>] blk_mq_freeze_queue_wait+0x56/0xb0
[<ffffffff810b4620>] ? prepare_to_wait_event+0xf0/0xf0
[<ffffffff81273b64>] blk_mq_update_tag_set_depth+0x44/0xb0
[<ffffffff81275620>] blk_mq_free_queue+0x50/0x110
[<ffffffff81266468>] blk_cleanup_queue+0x148/0x240
[<ffffffffa001aaf5>] __scsi_remove_device+0x65/0xd0 [scsi_mod]
[<ffffffffa0019249>] scsi_forget_host+0x69/0x70 [scsi_mod]
[<ffffffffa000d292>] scsi_remove_host+0x82/0x130 [scsi_mod]
[<ffffffffa02a5c80>] srp_remove_work+0x90/0x1f0 [ib_srp]
[<ffffffff8108d9b8>] process_one_work+0x1d8/0x610
[<ffffffff8108d92b>] ? process_one_work+0x14b/0x610
[<ffffffff8108df04>] worker_thread+0x114/0x460
[<ffffffff8108ddf0>] ? process_one_work+0x610/0x610
[<ffffffff810941f8>] kthread+0xf8/0x110
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
[<ffffffff814f5eaf>] ret_from_fork+0x3f/0x70
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
kworker/1:2 D ffff88045c648000 0 264 2 0x00000000
Workqueue: srp_remove srp_remove_work [ib_srp]
ffff880444d47c08 0000000000000086 ffffffff810ba11d ffff88047fc55ad8
ffff88045c648000 ffff880450a9da00 ffff880444d48000 ffff88042826c8d8
ffff880428969790 ffff880413b27b50 0000000000000040 ffff880444d47c20
Call Trace:
[<ffffffff810ba11d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff81271a36>] blk_mq_freeze_queue_wait+0x56/0xb0
[<ffffffff810b4620>] ? prepare_to_wait_event+0xf0/0xf0
[<ffffffff81273b64>] blk_mq_update_tag_set_depth+0x44/0xb0
[<ffffffff81275620>] blk_mq_free_queue+0x50/0x110
[<ffffffff81266468>] blk_cleanup_queue+0x148/0x240
[<ffffffffa001aaf5>] __scsi_remove_device+0x65/0xd0 [scsi_mod]
[<ffffffffa0019249>] scsi_forget_host+0x69/0x70 [scsi_mod]
[<ffffffffa000d292>] scsi_remove_host+0x82/0x130 [scsi_mod]
[<ffffffffa02a5c80>] srp_remove_work+0x90/0x1f0 [ib_srp]
[<ffffffff8108d9b8>] process_one_work+0x1d8/0x610
[<ffffffff8108d92b>] ? process_one_work+0x14b/0x610
[<ffffffff8108df04>] worker_thread+0x114/0x460
[<ffffffff8108ddf0>] ? process_one_work+0x610/0x610
[<ffffffff810941f8>] kthread+0xf8/0x110
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
[<ffffffff814f5eaf>] ret_from_fork+0x3f/0x70
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
kworker/7:0 D ffff88045c675a00 0 27179 2 0x00000000
Workqueue: srp_remove srp_remove_work [ib_srp]
ffff8803f7333c08 0000000000000086 ffffffff810ba11d ffff88047fdd5ad8
ffff88045c675a00 ffff8803f7139680 ffff8803f7334000 ffff880403d76e40
ffff88040c412408 ffff8803fe493cf8 00000000000001c0 ffff8803f7333c20
Call Trace:
[<ffffffff810ba11d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff81271a36>] blk_mq_freeze_queue_wait+0x56/0xb0
[<ffffffff810b4620>] ? prepare_to_wait_event+0xf0/0xf0
[<ffffffff81273b64>] blk_mq_update_tag_set_depth+0x44/0xb0
[<ffffffff81275620>] blk_mq_free_queue+0x50/0x110
[<ffffffff81266468>] blk_cleanup_queue+0x148/0x240
[<ffffffffa001aaf5>] __scsi_remove_device+0x65/0xd0 [scsi_mod]
[<ffffffffa0019249>] scsi_forget_host+0x69/0x70 [scsi_mod]
[<ffffffffa000d292>] scsi_remove_host+0x82/0x130 [scsi_mod]
[<ffffffffa02a5c80>] srp_remove_work+0x90/0x1f0 [ib_srp]
[<ffffffff8108d9b8>] process_one_work+0x1d8/0x610
[<ffffffff8108d92b>] ? process_one_work+0x14b/0x610
[<ffffffff8108df04>] worker_thread+0x114/0x460
[<ffffffff8108ddf0>] ? process_one_work+0x610/0x610
[<ffffffff810941f8>] kthread+0xf8/0x110
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
[<ffffffff814f5eaf>] ret_from_fork+0x3f/0x70
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
kworker/3:0 D ffff88045c649680 0 21529 2 0x00000000
Workqueue: srp_remove srp_remove_work [ib_srp]
ffff880392f2fc08 0000000000000086 ffffffff810ba11d ffff88047fcd5ad8
ffff88045c649680 ffff88039a72c380 ffff880392f30000 ffff880420f2fab8
ffff880425896ed8 ffff880416043cf8 00000000000000c0 ffff880392f2fc20
Call Trace:
[<ffffffff810ba11d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff81271a36>] blk_mq_freeze_queue_wait+0x56/0xb0
[<ffffffff810b4620>] ? prepare_to_wait_event+0xf0/0xf0
[<ffffffff81273b64>] blk_mq_update_tag_set_depth+0x44/0xb0
[<ffffffff81275620>] blk_mq_free_queue+0x50/0x110
[<ffffffff81266468>] blk_cleanup_queue+0x148/0x240
[<ffffffffa001aaf5>] __scsi_remove_device+0x65/0xd0 [scsi_mod]
[<ffffffffa0019249>] scsi_forget_host+0x69/0x70 [scsi_mod]
[<ffffffffa000d292>] scsi_remove_host+0x82/0x130 [scsi_mod]
[<ffffffffa02a5c80>] srp_remove_work+0x90/0x1f0 [ib_srp]
[<ffffffff8108d9b8>] process_one_work+0x1d8/0x610
[<ffffffff8108d92b>] ? process_one_work+0x14b/0x610
[<ffffffff8108df04>] worker_thread+0x114/0x460
[<ffffffff8108ddf0>] ? process_one_work+0x610/0x610
[<ffffffff810941f8>] kthread+0xf8/0x110
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
[<ffffffff814f5eaf>] ret_from_fork+0x3f/0x70
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
kworker/5:0 D ffff88045c64ad00 0 10950 2 0x00000000
Workqueue: srp_remove srp_remove_work [ib_srp]
ffff88037111bc08 0000000000000086 ffffffff810ba11d ffff88047fd55ad8
ffff88045c64ad00 ffff880055991680 ffff88037111c000 ffff88041ff348d8
ffff880424463080 ffff8804145155e8 0000000000000140 ffff88037111bc20
Call Trace:
[<ffffffff810ba11d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff81271a36>] blk_mq_freeze_queue_wait+0x56/0xb0
[<ffffffff810b4620>] ? prepare_to_wait_event+0xf0/0xf0
[<ffffffff81273b64>] blk_mq_update_tag_set_depth+0x44/0xb0
[<ffffffff81275620>] blk_mq_free_queue+0x50/0x110
[<ffffffff81266468>] blk_cleanup_queue+0x148/0x240
[<ffffffffa001aaf5>] __scsi_remove_device+0x65/0xd0 [scsi_mod]
[<ffffffffa0019249>] scsi_forget_host+0x69/0x70 [scsi_mod]
[<ffffffffa000d292>] scsi_remove_host+0x82/0x130 [scsi_mod]
[<ffffffffa02a5c80>] srp_remove_work+0x90/0x1f0 [ib_srp]
[<ffffffff8108d9b8>] process_one_work+0x1d8/0x610
[<ffffffff8108d92b>] ? process_one_work+0x14b/0x610
[<ffffffff8108df04>] worker_thread+0x114/0x460
[<ffffffff8108ddf0>] ? process_one_work+0x610/0x610
[<ffffffff810941f8>] kthread+0xf8/0x110
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
[<ffffffff814f5eaf>] ret_from_fork+0x3f/0x70
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
kworker/6:1 D ffff88045c670000 0 17446 2 0x00000000
Workqueue: srp_remove srp_remove_work [ib_srp]
ffff880052effc08 0000000000000086 ffffffff810ba11d ffff88047fd95ad8
ffff88045c670000 ffff88035e401680 ffff880052f00000 ffff8803ecf8ee40
ffff8803eed58b18 ffff880067a88b18 0000000000000180 ffff880052effc20
Call Trace:
[<ffffffff810ba11d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff81271a36>] blk_mq_freeze_queue_wait+0x56/0xb0
[<ffffffff810b4620>] ? prepare_to_wait_event+0xf0/0xf0
[<ffffffff81273b64>] blk_mq_update_tag_set_depth+0x44/0xb0
[<ffffffff81275620>] blk_mq_free_queue+0x50/0x110
[<ffffffff81266468>] blk_cleanup_queue+0x148/0x240
[<ffffffffa001aaf5>] __scsi_remove_device+0x65/0xd0 [scsi_mod]
[<ffffffffa0019249>] scsi_forget_host+0x69/0x70 [scsi_mod]
[<ffffffffa000d292>] scsi_remove_host+0x82/0x130 [scsi_mod]
[<ffffffffa02a5c80>] srp_remove_work+0x90/0x1f0 [ib_srp]
[<ffffffff8108d9b8>] process_one_work+0x1d8/0x610
[<ffffffff8108d92b>] ? process_one_work+0x14b/0x610
[<ffffffff8108df04>] worker_thread+0x114/0x460
[<ffffffff8108ddf0>] ? process_one_work+0x610/0x610
[<ffffffff810941f8>] kthread+0xf8/0x110
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
[<ffffffff814f5eaf>] ret_from_fork+0x3f/0x70
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
kworker/0:0 D ffff88035d205a00 0 23356 2 0x00000000
Workqueue: srp_remove srp_remove_work [ib_srp]
ffff88035ec83c08 0000000000000086 ffffffff810ba11d ffff88047fc15ad8
ffff88035d205a00 ffff880360a3ad00 ffff88035ec84000 ffff8803e75a61c8
ffff8803eed59790 ffff8803e4c10b18 0000000000000000 ffff88035ec83c20
Call Trace:
[<ffffffff810ba11d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff81271a36>] blk_mq_freeze_queue_wait+0x56/0xb0
[<ffffffff810b4620>] ? prepare_to_wait_event+0xf0/0xf0
[<ffffffff81273b64>] blk_mq_update_tag_set_depth+0x44/0xb0
[<ffffffff81275620>] blk_mq_free_queue+0x50/0x110
[<ffffffff81266468>] blk_cleanup_queue+0x148/0x240
[<ffffffffa001aaf5>] __scsi_remove_device+0x65/0xd0 [scsi_mod]
[<ffffffffa0019249>] scsi_forget_host+0x69/0x70 [scsi_mod]
[<ffffffffa000d292>] scsi_remove_host+0x82/0x130 [scsi_mod]
[<ffffffffa02a5c80>] srp_remove_work+0x90/0x1f0 [ib_srp]
[<ffffffff8108d9b8>] process_one_work+0x1d8/0x610
[<ffffffff8108d92b>] ? process_one_work+0x14b/0x610
[<ffffffff8108df04>] worker_thread+0x114/0x460
[<ffffffff8108ddf0>] ? process_one_work+0x610/0x610
[<ffffffff810941f8>] kthread+0xf8/0x110
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
[<ffffffff814f5eaf>] ret_from_fork+0x3f/0x70
[<ffffffff81094100>] ? kthread_create_on_node+0x200/0x200
modprobe D ffff88045c64ad00 0 24137 17294 0x00000004
ffff8804146e7c48 0000000000000082 ffff880400000001 ffff88047fd55ad8
ffff88045c64ad00 ffff88045c5d0000 ffff8804146e8000 ffff8804146e7df0
ffff8804146e7de8 ffff88045c5d0000 ffff8804146e7dd0 ffff8804146e7c60
Call Trace:
[<ffffffff814effea>] schedule+0x3a/0x90
[<ffffffff814f4463>] schedule_timeout+0x1f3/0x290
[<ffffffff810b9f16>] ? mark_held_locks+0x66/0x90
[<ffffffff814f520c>] ? _raw_spin_unlock_irq+0x2c/0x40
[<ffffffff810ba042>] ? trace_hardirqs_on_caller+0x102/0x1d0
[<ffffffff814f1246>] wait_for_completion+0xd6/0x110
[<ffffffff8109ebd0>] ? wake_up_q+0x70/0x70
[<ffffffff8108a7cf>] flush_workqueue+0x1cf/0x6b0
[<ffffffff8108a605>] ? flush_workqueue+0x5/0x6b0
[<ffffffffa02a51a1>] srp_remove_one+0xc1/0x130 [ib_srp]
[<ffffffffa02b77ae>] ib_unregister_client+0x11e/0x1a0 [ib_core]
[<ffffffffa02aa471>] srp_cleanup_module+0x10/0xb9f [ib_srp]
[<ffffffff810f106f>] SyS_delete_module+0x16f/0x1f0
[<ffffffff81003017>] ? trace_hardirqs_on_thunk+0x17/0x19
[<ffffffff814f5af6>] entry_SYSCALL_64_fastpath+0x16/0x7a
next prev parent reply other threads:[~2015-09-24 18:09 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-09-23 22:11 [PATCH 0/3] Block layer patches for kernel v4.4 Bart Van Assche
2015-09-23 22:12 ` [PATCH 1/3] blk-cgroup: Declare local symbols static Bart Van Assche
2015-09-25 15:35 ` Tejun Heo
2015-09-23 22:12 ` [PATCH 2/3] bsg: Add sparse annotations to bsg_request_fn() Bart Van Assche
2015-09-23 22:14 ` [PATCH 3/3] blk-mq: Fix the queue freezing mechanism Bart Van Assche
2015-09-24 3:22 ` Ming Lei
2015-09-24 16:43 ` Bart Van Assche
2015-09-24 16:53 ` Tejun Heo
2015-09-24 17:35 ` Bart Van Assche
2015-09-24 17:49 ` Tejun Heo
2015-09-24 18:09 ` Bart Van Assche [this message]
2015-09-24 18:14 ` Tejun Heo
2015-09-24 22:54 ` Bart Van Assche
2015-09-24 22:56 ` Tejun Heo
2015-09-24 19:04 ` Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56043C5D.3090307@sandisk.com \
--to=bart.vanassche@sandisk.com \
--cc=akinobu.mita@gmail.com \
--cc=axboe@fb.com \
--cc=hch@lst.de \
--cc=linux-kernel@vger.kernel.org \
--cc=tj@kernel.org \
--cc=tom.leiming@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome