Re: 3.10.16 cgroup_mutex deadlock

[Tejun Heo <tj@kernel.org>]

Hello,

On Thu, Nov 14, 2013 at 04:56:49PM -0600, Shawn Bohrer wrote:
> After running both concurrently on 40 machines for about 12 hours I've
> managed to reproduce the issue at least once, possibly more.  One
> machine looked identical to this reported issue.  It has a bunch of
> stuck cgroup_free_fn() kworker threads and one thread in cpuset_attach
> waiting on lru_add_drain_all().  A sysrq+l shows all CPUs are idle
> except for the one triggering the sysrq+l.  The sysrq+w unfortunately
> wrapped dmesg so we didn't get the stacks of all blocked tasks.  We
> did however also cat /proc/<pid>/stack of all kworker threads on the
> system.  There were 265 kworker threads that all have the following
> stack:

Umm... so, WQ_DFL_ACTIVE is 256.  It's just an arbitrarily largish
number which is supposed to serve as protection against runaway
kworker creation.  The assumption there is that there won't be a
dependency chain which can be longer than that and if there are it
should be separated out into a separate workqueue.  It looks like we
*can* have such long chain of dependency with high enough rate of
cgroup destruction.  kworkers trying to destroy cgroups get blocked by
an earlier one which is holding cgroup_mutex.  If the blocked ones
completely consume max_active and then the earlier one tries to
perform an operation which makes use of the system_wq, the forward
progress guarantee gets broken.

So, yeah, it makes sense now.  We're just gonna have to separate out
cgroup destruction to a separate workqueue.  Hugh's temp fix achieved
about the same effect by putting the affected part of destruction to a
different workqueue.  I probably should have realized that we were
hitting max_active when I was told that moving some part to a
different workqueue makes the problem go away.

Will send out a patch soon.

Thanks.

-- 
tejun