From: Mark Hills <mark@pogo.org.uk>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Balbir Singh <balbir@linux.vnet.ibm.com>,
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>,
Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>,
linux-kernel@vger.kernel.org
Subject: Re: cgroup: rmdir() does not complete
Date: Fri, 10 Sep 2010 00:04:31 +0100 (BST) [thread overview]
Message-ID: <alpine.NEB.2.01.1009091416100.26432@jrf.vwaro.pbz> (raw)
In-Reply-To: <1284033017.402.8.camel@laptop>
On Thu, 9 Sep 2010, Peter Zijlstra wrote:
> On Thu, 2010-09-09 at 12:36 +0100, Mark Hills wrote:
>
> > I am still finding the problem incredibly hard to reproduce, so I'd like
> > to observe as much data as possible from the current case before
> > rebooting. If I could capture some kind of stack trace in the kernel for
> > the running process that would be great, any suggestions appreciated.
>
> echo l > /proc/sysrq-trigger
Despite running this many times, I never 'catch' the process on a CPU,
despite it using 70% in top. But...
> another thing you can do is run something like: perf record -gp $pid
> which will give you a profile of that task.
This is very useful, thanks.
The report on the spinning process (23586) is dominated by calls from
mem_cgroup_force_empty.
It seems to show lru_add_drain_all and drain_all_stock_sync are causing
the load (I assume drain_all_stock_sync has been optimised out). But I
don't think this is as important as what causes the spin.
There are no tasks in the cgroup, but memory usage is non-zero and
constant. It seems mem_cgroup_force_empty is unable to empty the cgroup in
this case.
# cat /cgroup/soaked-23586/tasks
# cat /cgroup/soaked-23586/memory.usage_in_bytes
24576
# cat /cgroup/soaked-23586/memsw.usage_in_bytes
<hangs>
Here are the first few entries from the perf output, I can provide the
rest if needed, but all result from mem_cgroup_force_empty.
8.13% :23586 [kernel] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--45.14%-- probe_workqueue_insertion
| insert_work
| |
| |--99.09%-- __queue_work
| | queue_work_on
| | schedule_work_on
| | schedule_on_each_cpu
| | |
| | |--50.59%-- lru_add_drain_all
| | | mem_cgroup_force_empty
| | | mem_cgroup_pre_destroy
| | | cgroup_rmdir
| | | vfs_rmdir
| | | do_rmdir
| | | sys_rmdir
| | | system_call_fastpath
| | | 0x3f504d27d7
| | | 0x405687
| | | 0x406ef0
| | | 0x402f31
| | | 0x3f5041eb1d
| | |
| | --49.41%-- mem_cgroup_force_empty
| | mem_cgroup_pre_destroy
| | cgroup_rmdir
| | vfs_rmdir
| | do_rmdir
| | sys_rmdir
| | system_call_fastpath
| | 0x3f504d27d7
| | 0x405687
| | 0x406ef0
| | 0x402f31
| | 0x3f5041eb1d
| --0.91%-- [...]
|
|--22.92%-- mem_cgroup_force_empty
| mem_cgroup_pre_destroy
| cgroup_rmdir
| vfs_rmdir
| do_rmdir
| sys_rmdir
| system_call_fastpath
| 0x3f504d27d7
| 0x405687
| 0x406ef0
| 0x402f31
| 0x3f5041eb1d
|
|--8.17%-- __queue_work
| queue_work_on
| schedule_work_on
| schedule_on_each_cpu
| |
| |--52.09%-- lru_add_drain_all
| | mem_cgroup_force_empty
| | mem_cgroup_pre_destroy
| | cgroup_rmdir
| | vfs_rmdir
| | do_rmdir
| | sys_rmdir
| | system_call_fastpath
| | 0x3f504d27d7
| | 0x405687
| | 0x406ef0
| | 0x402f31
| | 0x3f5041eb1d
| |
| --47.91%-- mem_cgroup_force_empty
| mem_cgroup_pre_destroy
| cgroup_rmdir
| vfs_rmdir
| do_rmdir
| sys_rmdir
| system_call_fastpath
| 0x3f504d27d7
| 0x405687
| 0x406ef0
| 0x402f31
| 0x3f5041eb1d
|
|--7.94%-- __wake_up
| |
| |--99.71%-- insert_work
| | |
| | |--97.70%-- __queue_work
| | | queue_work_on
| | | schedule_work_on
| | | schedule_on_each_cpu
| | | |
| | | |--50.59%-- mem_cgroup_force_empty
| | | | mem_cgroup_pre_destroy
| | | | cgroup_rmdir
| | | | vfs_rmdir
| | | | do_rmdir
| | | | sys_rmdir
| | | | system_call_fastpath
| | | | 0x3f504d27d7
| | | | 0x405687
| | | | 0x406ef0
| | | | 0x402f31
| | | | 0x3f5041eb1d
| | | |
| | | --49.41%-- lru_add_drain_all
| | | mem_cgroup_force_empty
| | | mem_cgroup_pre_destroy
| | | cgroup_rmdir
| | | vfs_rmdir
| | | do_rmdir
| | | sys_rmdir
| | | system_call_fastpath
| | | 0x3f504d27d7
| | | 0x405687
| | | 0x406ef0
| | | 0x402f31
| | | 0x3f5041eb1d
| | --2.30%-- [...]
| --0.29%-- [...]
|
|--4.35%-- mem_cgroup_pre_destroy
| cgroup_rmdir
| vfs_rmdir
| do_rmdir
| sys_rmdir
| system_call_fastpath
| 0x3f504d27d7
| 0x405687
| 0x406ef0
| 0x402f31
| 0x3f5041eb1d
--11.47%-- [...]
7.25% :23586 [kernel] [k] sched_clock_cpu
|
--- sched_clock_cpu
|
|--97.11%-- update_rq_clock
| |
| |--98.89%-- try_to_wake_up
| | default_wake_function
| | autoremove_wake_function
| | __wake_up_common
| | __wake_up
| | insert_work
| | __queue_work
| | queue_work_on
| | schedule_work_on
| | schedule_on_each_cpu
| | |
| | |--50.69%-- lru_add_drain_all
| | | mem_cgroup_force_empty
| | | mem_cgroup_pre_destroy
| | | cgroup_rmdir
| | | vfs_rmdir
| | | do_rmdir
| | | sys_rmdir
| | | system_call_fastpath
| | | 0x3f504d27d7
| | | 0x405687
| | | 0x406ef0
| | | 0x402f31
| | | 0x3f5041eb1d
| | |
| | --49.31%-- mem_cgroup_force_empty
| | mem_cgroup_pre_destroy
| | cgroup_rmdir
| | vfs_rmdir
| | do_rmdir
| | sys_rmdir
| | system_call_fastpath
| | 0x3f504d27d7
| | 0x405687
| | 0x406ef0
| | 0x402f31
| | 0x3f5041eb1d
| --1.11%-- [...]
--2.89%-- [...]
5.54% :23586 [kernel] [k] try_to_wake_up
|
--- try_to_wake_up
|
|--99.13%-- default_wake_function
| autoremove_wake_function
| __wake_up_common
| __wake_up
| insert_work
| __queue_work
| queue_work_on
| schedule_work_on
| schedule_on_each_cpu
| |
| |--52.03%-- lru_add_drain_all
| | mem_cgroup_force_empty
| | mem_cgroup_pre_destroy
| | cgroup_rmdir
| | vfs_rmdir
| | do_rmdir
| | sys_rmdir
| | system_call_fastpath
| | 0x3f504d27d7
| | 0x405687
| | 0x406ef0
| | 0x402f31
| | 0x3f5041eb1d
| |
| --47.97%-- mem_cgroup_force_empty
| mem_cgroup_pre_destroy
| cgroup_rmdir
| vfs_rmdir
| do_rmdir
| sys_rmdir
| system_call_fastpath
| 0x3f504d27d7
| 0x405687
| 0x406ef0
| 0x402f31
| 0x3f5041eb1d
--0.87%-- [...]
--
Mark
next prev parent reply other threads:[~2010-09-09 23:05 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-26 15:51 Mark Hills
2010-08-27 0:56 ` Daisuke Nishimura
2010-08-27 1:20 ` Balbir Singh
2010-08-27 2:35 ` KAMEZAWA Hiroyuki
2010-08-27 3:39 ` Daisuke Nishimura
2010-08-27 5:42 ` KAMEZAWA Hiroyuki
2010-08-27 6:29 ` KAMEZAWA Hiroyuki
2010-08-30 7:32 ` Balbir Singh
2010-08-30 9:13 ` Mark Hills
2010-09-01 11:10 ` Mark Hills
2010-09-01 23:42 ` KAMEZAWA Hiroyuki
2010-09-02 9:45 ` Mark Hills
2010-09-09 10:01 ` Mark Hills
2010-09-09 10:09 ` Balbir Singh
2010-09-09 11:36 ` Mark Hills
2010-09-09 11:50 ` Peter Zijlstra
2010-09-09 23:04 ` Mark Hills [this message]
2010-09-09 23:43 ` KAMEZAWA Hiroyuki
2010-09-10 2:16 ` KAMEZAWA Hiroyuki
2010-09-10 4:05 ` Daisuke Nishimura
2010-09-10 4:11 ` KAMEZAWA Hiroyuki
2010-09-10 7:28 ` Mark Hills
2010-09-10 7:33 ` KAMEZAWA Hiroyuki
2010-09-10 7:51 ` Mark Hills
2010-08-27 1:25 ` KAMEZAWA Hiroyuki
2010-08-30 9:25 ` Mark Hills
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.NEB.2.01.1009091416100.26432@jrf.vwaro.pbz \
--to=mark@pogo.org.uk \
--cc=balbir@linux.vnet.ibm.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=nishimura@mxp.nes.nec.co.jp \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®