From: David Rientjes <rientjes@google.com>
To: Eric Paris <eparis@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>,
linux-kernel@vger.kernel.org,
Linus Torvalds <torvalds@linux-foundation.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>, Greg KH <greg@kroah.com>,
linux-hotplug@vger.kernel.org
Subject: Re: kernel panic - not syncing: out of memory and no killable processes
Date: Fri, 18 Sep 2009 12:58:01 -0700 (PDT) [thread overview]
Message-ID: <alpine.DEB.1.00.0909181246410.27556@chino.kir.corp.google.com> (raw)
In-Reply-To: <1253285923.30531.17.camel@dhcp231-106.rdu.redhat.com>
On Fri, 18 Sep 2009, Eric Paris wrote:
> > Isolating udevd down to an interactivity scheduling change isn't _that_
> > bizarre. I think the setting of UDEVD_PRIORITY is already mostly
> > arbitrary anyway and it'll allow 192 children on your 512M machine by
> > default unless you changed UDEVD_MAX_CHILDS for uid 0.
> >
> > The default timeout for idle workers is 3 seconds, which may just happen
> > to be long enough to panic your machine because of low memory. If that's
> > the case, I don't believe that it's a scheduler issue but rather a root
> > abuse of setting all udevd threads to be OOM_DISABLE.
> >
> > What is your udevd --version? The latest is udev-146 released last month.
>
> 145
>
> Let me try and clone the vm some I don't break my reproducer. I'll see
> if adding more memory fixes it. Doesn't look like Fedora has built a
> -146 yet, I'll see if I can get one of those as well.
>
> udev bug, configuration issue, whatever, or not, it's a regression that
> I used to be able to boot and updating my kernel leaves me unable to
> boot. I think we all agree when 512M of memory isn't enough to boot to
> runlevel 3 we've got a problem :)
>
I totally agree, and my hypothesis is that the idle child workers are not
being killed in time that they quickly accumulate approaching
UDEVD_MAX_CHILDS and when the oom killer is called because of a write to
shared memory, it can't kill any of these threads either since udevd sets
them all to OOM_DISABLE and everything else is an unkillable kthread.
Bisecting that to a scheduler change would suggest that each udevd thread
isn't returning from its poll() timeout fast enough; there's essentially a
street race between udevd killing its own threads off because the poll
timeout was exceeded and all your memory being used up and the machine
panicking. The scheduling change seems to have affected the speed of the
former.
UDEVD_MAX_CHILDS defaults to 192 on your 512M machine unless overridden by
an environment variable of the same name, so you may find it helpful to
reduce this to a saner value. I'd suggest a value lower than the number
of udevd threads that were shown in your latest oom killer dump.
If that turns out to fix the issue for you, perhaps max_childs needs to be
calculated in a slightly more conservative way in the userspace package
since all threads come with the prerequisite of being OOM_DISABLE.
next prev parent reply other threads:[~2009-09-18 19:58 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-09-17 3:36 Eric Paris
2009-09-17 12:43 ` Eric Paris
2009-09-17 20:29 ` David Rientjes
2009-09-17 20:35 ` Eric Paris
2009-09-17 21:01 ` Eric Paris
2009-09-17 21:15 ` Eric Paris
2009-09-17 21:24 ` David Rientjes
2009-09-17 21:31 ` Eric Paris
2009-09-17 21:59 ` David Rientjes
2009-09-18 14:58 ` Eric Paris
2009-09-18 19:58 ` David Rientjes [this message]
2009-09-17 21:27 ` Eric Paris
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.1.00.0909181246410.27556@chino.kir.corp.google.com \
--to=rientjes@google.com \
--cc=a.p.zijlstra@chello.nl \
--cc=eparis@redhat.com \
--cc=greg@kroah.com \
--cc=linux-hotplug@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®