From: Vivek Goyal <vgoyal@redhat.com>
To: Tejun Heo <tj@kernel.org>
Cc: axboe@kernel.dk, ctalbott@google.com, rni@google.com,
linux-kernel@vger.kernel.org, dm-devel@redhat.com
Subject: Re: [PATCHSET] blkcg: accumulated blkcg updates
Date: Tue, 6 Mar 2012 13:39:42 -0500 [thread overview]
Message-ID: <20120306183942.GC32148@redhat.com> (raw)
In-Reply-To: <20120306162455.GB32148@redhat.com>
On Tue, Mar 06, 2012 at 11:24:55AM -0500, Vivek Goyal wrote:
> On Tue, Mar 06, 2012 at 10:07:09AM -0500, Vivek Goyal wrote:
>
> [..]
> >
> > My system is hanging during reboot. Last message I see is "Detaching DM
> > devices" and nothing happens after that. I shall have to do some more
> > testing to figure out when did that start happening.
>
> Ok, git bisect shows that very first patch to drain the queue is culprit.
>
> 9e5b9f8 block: blk-throttle should be drained regardless of q->elevator
>
> Will do some more debugging.
Hmm..., haven't reached to the bottom of the issue yet, but there is more
data.
- We are spinning in blk_drain_queue() as we think there is request on the
request queue to be drained.
- The request queue we are spinning on, is created by dm.
- There is something queued on q->queue_head but we are not kicking queue
as q->request_fn is empty. I think you put this code to avoid issues
with loop etc. Though I am not sure why it is not a bug condition. If
a request queue does not have a request function, its a bio based
driver. Are these driver using q->queue_head to queue bios or something
internal? If yes, then it is still a BUG() condition as driver should
have cleaned up the queue before calling blk_cleanup_queue(). So I
don't know why you are not treating it as a bug() condition.
Captured two backtraces of queue creation and when we are spinning in
queue drain/cleanup.
CCing dm-devel. They might know what's happening.
Also there is no q->backing_dev_info.dev. So looks like either we never
registered a device or we cleaned up a device before calling
blk_cleanup_queue().
Thanks
Vivek
Queue creation backtrace.
------------------------
[ 23.382675] ------------[ cut here ]------------
[ 23.387628] WARNING: at block/blk-cgroup.c:1660 blkcg_init_queue+0x4b/0xb0()
[ 23.394739] Hardware name: HP xw6600 Workstation
[ 23.399426] Modules linked in: floppy [last unloaded: scsi_wait_scan]
[ 23.406077] Pid: 2739, comm: lvm Tainted: G W 3.3.0-rc3+ #3
[ 23.412583] Call Trace:
[ 23.415105] [<ffffffff81037f2f>] warn_slowpath_common+0x7f/0xc0
[ 23.421180] [<ffffffff81037f8a>] warn_slowpath_null+0x1a/0x20
[ 23.427080] [<ffffffff8130b76b>] blkcg_init_queue+0x4b/0xb0
[ 23.432811] [<ffffffff812f2c2a>] blk_alloc_queue_node+0x22a/0x270
[ 23.439057] [<ffffffff812f2c83>] blk_alloc_queue+0x13/0x20
[ 23.444700] [<ffffffff815722ee>] dm_create+0x21e/0x520
[ 23.449995] [<ffffffff8157890e>] dev_create+0x5e/0x360
[ 23.455289] [<ffffffff81578e9a>] ctl_ioctl+0x15a/0x2c0
[ 23.460586] [<ffffffff8112110c>] ? might_fault+0x5c/0xb0
[ 23.466053] [<ffffffff815788b0>] ? dev_suspend+0x240/0x240
[ 23.471693] [<ffffffff81579013>] dm_ctl_ioctl+0x13/0x20
[ 23.477075] [<ffffffff81163658>] do_vfs_ioctl+0x98/0x560
[ 23.482543] [<ffffffff81150b9f>] ? fget_light+0x1df/0x490
[ 23.488097] [<ffffffff81154ada>] ? sys_newstat+0x2a/0x40
[ 23.493564] [<ffffffff81163bb1>] sys_ioctl+0x91/0xa0
[ 23.498686] [<ffffffff81843ad2>] system_call_fastpath+0x16/0x1b
[ 23.504759] ---[ end trace 1de7f357c03667a3 ]---
Queue cleanup backtrace
------------------------
[ 147.977010] ------------[ cut here ]------------
[ 147.981696] WARNING: at block/blk-core.c:411
blk_drain_queue+0x124/0x180()
[ 147.988636] Hardware name: HP xw6600 Workstation
[ 147.993323] Modules linked in: floppy [last unloaded: scsi_wait_scan]
[ 147.999976] Pid: 1, comm: systemd-shutdow Tainted: G W
3.3.0-rc3+ #3
[ 148.007307] Call Trace:
[ 148.009831] [<ffffffff81037f2f>] warn_slowpath_common+0x7f/0xc0
[ 148.015911] [<ffffffff81037f8a>] warn_slowpath_null+0x1a/0x20
[ 148.021816] [<ffffffff812f78c4>] blk_drain_queue+0x124/0x180
[ 148.027630] [<ffffffff812f7a24>] blk_cleanup_queue+0x104/0x1f0
[ 148.033619] [<ffffffff81571ebe>] __dm_destroy+0x1ee/0x260
[ 148.039180] [<ffffffff81572c43>] dm_destroy+0x13/0x20
[ 148.044393] [<ffffffff815783cd>] dev_remove+0x8d/0xf0
[ 148.049601] [<ffffffff81578e9a>] ctl_ioctl+0x15a/0x2c0
[ 148.054895] [<ffffffff81578340>] ? __hash_remove+0xd0/0xd0
[ 148.060538] [<ffffffff81579013>] dm_ctl_ioctl+0x13/0x20
[ 148.065919] [<ffffffff81163658>] do_vfs_ioctl+0x98/0x560
[ 148.071389] [<ffffffff8115bb13>] ? putname+0x33/0x50
[ 148.076512] [<ffffffff81145445>] ? kmem_cache_free+0x235/0x240
[ 148.082499] [<ffffffff81150b9f>] ? fget_light+0x1df/0x490
[ 148.088055] [<ffffffff81163bb1>] sys_ioctl+0x91/0xa0
[ 148.093177] [<ffffffff81843ad2>] system_call_fastpath+0x16/0x1b
[ 148.099251] ---[ end trace 1de7f357c03667bc ]---
[ 148.103939] Sleeping waiting in drain_queue. q=ffff880136f847a0 drain=1
queue_empty=0 q->request_fn= (null)
next prev parent reply other threads:[~2012-03-06 19:17 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-22 1:46 Tejun Heo
2012-02-22 1:46 ` [PATCH 01/36] block: blk-throttle should be drained regardless of q->elevator Tejun Heo
2012-02-22 1:46 ` [PATCH 02/36] blkcg: make CONFIG_BLK_CGROUP bool Tejun Heo
2012-02-22 1:46 ` [PATCH 03/36] cfq: don't register propio policy if !CONFIG_CFQ_GROUP_IOSCHED Tejun Heo
2012-02-22 1:46 ` [PATCH 04/36] elevator: clear auxiliary data earlier during elevator switch Tejun Heo
2012-02-22 1:46 ` [PATCH 05/36] elevator: make elevator_init_fn() return 0/-errno Tejun Heo
2012-02-22 1:46 ` [PATCH 06/36] block: implement blk_queue_bypass_start/end() Tejun Heo
2012-02-22 1:46 ` [PATCH 07/36] block: extend queue bypassing to cover blkcg policies Tejun Heo
2012-02-22 1:46 ` [PATCH 08/36] blkcg: shoot down blkio_groups on elevator switch Tejun Heo
2012-02-22 1:46 ` [PATCH 09/36] blkcg: move rcu_read_lock() outside of blkio_group get functions Tejun Heo
2012-02-22 1:46 ` [PATCH 10/36] blkcg: update blkg get functions take blkio_cgroup as parameter Tejun Heo
2012-02-22 1:46 ` [PATCH 11/36] blkcg: use q and plid instead of opaque void * for blkio_group association Tejun Heo
2012-02-22 1:46 ` [PATCH 12/36] blkcg: add blkio_policy[] array and allow one policy per policy ID Tejun Heo
2012-02-22 1:46 ` [PATCH 13/36] blkcg: use the usual get blkg path for root blkio_group Tejun Heo
2012-02-22 1:46 ` [PATCH 14/36] blkcg: factor out blkio_group creation Tejun Heo
2012-02-22 1:46 ` [PATCH 15/36] blkcg: don't allow or retain configuration of missing devices Tejun Heo
2012-02-22 1:46 ` [PATCH 16/36] blkcg: kill blkio_policy_node Tejun Heo
2012-02-22 1:46 ` [PATCH 17/36] blkcg: kill the mind-bending blkg->dev Tejun Heo
2012-02-22 1:46 ` [PATCH 18/36] blkcg: let blkio_group point to blkio_cgroup directly Tejun Heo
2012-02-22 1:46 ` [PATCH 19/36] blkcg: add blkcg_{init|drain|exit}_queue() Tejun Heo
2012-02-22 1:46 ` [PATCH 20/36] blkcg: clear all request_queues on blkcg policy [un]registrations Tejun Heo
2012-02-22 1:46 ` [PATCH 21/36] blkcg: let blkcg core handle policy private data allocation Tejun Heo
2012-02-22 1:46 ` [PATCH 22/36] blkcg: move refcnt to blkcg core Tejun Heo
2012-02-22 1:46 ` [PATCH 23/36] blkcg: make blkg->pd an array and move configuration and stats into it Tejun Heo
2012-02-22 1:46 ` [PATCH 24/36] blkcg: don't use blkg->plid in stat related functions Tejun Heo
2012-02-22 1:46 ` [PATCH 25/36] blkcg: move per-queue blkg list heads and counters to queue and blkg Tejun Heo
2012-02-22 1:46 ` [PATCH 26/36] blkcg: let blkcg core manage per-queue blkg list and counter Tejun Heo
2012-02-22 1:46 ` [PATCH 27/36] blkcg: unify blkg's for blkcg policies Tejun Heo
2012-03-05 21:01 ` [PATCH UPDATED " Tejun Heo
2012-02-22 1:46 ` [PATCH 28/36] blkcg: use double locking instead of RCU for blkg synchronization Tejun Heo
2012-02-22 1:46 ` [PATCH 29/36] blkcg: drop unnecessary RCU locking Tejun Heo
2012-02-23 18:51 ` [PATCH UPDATED " Tejun Heo
2012-02-22 1:46 ` [PATCH 30/36] block: restructure get_request() Tejun Heo
2012-02-22 1:46 ` [PATCH 31/36] block: interface update for ioc/icq creation functions Tejun Heo
2012-02-22 1:46 ` [PATCH 32/36] block: ioc_task_link() can't fail Tejun Heo
2012-02-22 1:47 ` [PATCH 33/36] block: add io_context->active_ref Tejun Heo
2012-02-22 18:47 ` Vivek Goyal
2012-02-22 19:13 ` Tejun Heo
2012-02-23 18:20 ` Vivek Goyal
2012-02-22 1:47 ` [PATCH 34/36] block: implement bio_associate_current() Tejun Heo
2012-02-22 13:45 ` Jeff Moyer
2012-02-22 19:07 ` Tejun Heo
2012-02-22 19:33 ` Jeff Moyer
2012-02-22 19:37 ` Vivek Goyal
2012-02-22 19:41 ` Jeff Moyer
2012-02-22 1:47 ` [PATCH 35/36] block: make block cgroup policies follow bio task association Tejun Heo
2012-02-22 1:47 ` [PATCH 36/36] block: make blk-throttle preserve the issuing task on delayed bios Tejun Heo
2012-02-22 19:34 ` [PATCHSET] blkcg: accumulated blkcg updates Vivek Goyal
2012-02-22 22:04 ` Tejun Heo
2012-03-05 20:59 ` [PATCH 17.5] blkcg: skip blkg printing if q isn't associated with disk Tejun Heo
2012-03-05 21:07 ` [PATCHSET] blkcg: accumulated blkcg updates Tejun Heo
2012-03-05 21:08 ` Tejun Heo
2012-03-06 15:07 ` Vivek Goyal
2012-03-06 16:24 ` Vivek Goyal
2012-03-06 18:39 ` Vivek Goyal [this message]
2012-03-06 19:02 ` Vivek Goyal
2012-03-08 0:06 ` Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120306183942.GC32148@redhat.com \
--to=vgoyal@redhat.com \
--cc=axboe@kernel.dk \
--cc=ctalbott@google.com \
--cc=dm-devel@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=rni@google.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome