mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: David Rientjes <rientjes@google.com>
To: Eric Paris <eparis@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>,
	linux-kernel@vger.kernel.org,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Peter Zijlstra <a.p.zijlstra@chello.nl>, Greg KH <greg@kroah.com>,
	linux-hotplug@vger.kernel.org
Subject: Re: kernel panic - not syncing: out of memory and no killable processes
Date: Fri, 18 Sep 2009 12:58:01 -0700 (PDT)	[thread overview]
Message-ID: <alpine.DEB.1.00.0909181246410.27556@chino.kir.corp.google.com> (raw)
In-Reply-To: <1253285923.30531.17.camel@dhcp231-106.rdu.redhat.com>

On Fri, 18 Sep 2009, Eric Paris wrote:

> > Isolating udevd down to an interactivity scheduling change isn't _that_ 
> > bizarre.  I think the setting of UDEVD_PRIORITY is already mostly 
> > arbitrary anyway and it'll allow 192 children on your 512M machine by 
> > default unless you changed UDEVD_MAX_CHILDS for uid 0.
> > 
> > The default timeout for idle workers is 3 seconds, which may just happen 
> > to be long enough to panic your machine because of low memory.  If that's 
> > the case, I don't believe that it's a scheduler issue but rather a root 
> > abuse of setting all udevd threads to be OOM_DISABLE.
> > 
> > What is your udevd --version?  The latest is udev-146 released last month.
> 
> 145
> 
> Let me try and clone the vm some I don't break my reproducer.  I'll see
> if adding more memory fixes it.  Doesn't look like Fedora has built a
> -146 yet, I'll see if I can get one of those as well.
> 
> udev bug, configuration issue, whatever, or not, it's a regression that
> I used to be able to boot and updating my kernel leaves me unable to
> boot.  I think we all agree when 512M of memory isn't enough to boot to
> runlevel 3 we've got a problem   :)
> 

I totally agree, and my hypothesis is that the idle child workers are not 
being killed in time that they quickly accumulate approaching 
UDEVD_MAX_CHILDS and when the oom killer is called because of a write to 
shared memory, it can't kill any of these threads either since udevd sets 
them all to OOM_DISABLE and everything else is an unkillable kthread.

Bisecting that to a scheduler change would suggest that each udevd thread 
isn't returning from its poll() timeout fast enough; there's essentially a 
street race between udevd killing its own threads off because the poll 
timeout was exceeded and all your memory being used up and the machine 
panicking.  The scheduling change seems to have affected the speed of the 
former.

UDEVD_MAX_CHILDS defaults to 192 on your 512M machine unless overridden by 
an environment variable of the same name, so you may find it helpful to 
reduce this to a saner value.  I'd suggest a value lower than the number 
of udevd threads that were shown in your latest oom killer dump.

If that turns out to fix the issue for you, perhaps max_childs needs to be 
calculated in a slightly more conservative way in the userspace package 
since all threads come with the prerequisite of being OOM_DISABLE.

  reply	other threads:[~2009-09-18 19:58 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2009-09-17  3:36 Eric Paris
2009-09-17 12:43 ` Eric Paris
2009-09-17 20:29   ` David Rientjes
2009-09-17 20:35     ` Eric Paris
2009-09-17 21:01       ` Eric Paris
2009-09-17 21:15         ` Eric Paris
2009-09-17 21:24           ` David Rientjes
2009-09-17 21:31             ` Eric Paris
2009-09-17 21:59               ` David Rientjes
2009-09-18 14:58                 ` Eric Paris
2009-09-18 19:58                   ` David Rientjes [this message]
2009-09-17 21:27         ` Eric Paris

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.1.00.0909181246410.27556@chino.kir.corp.google.com \
    --to=rientjes@google.com \
    --cc=a.p.zijlstra@chello.nl \
    --cc=eparis@redhat.com \
    --cc=greg@kroah.com \
    --cc=linux-hotplug@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®