kernel: request_module[net-pf-10]: fork failed, errno 11

kernel: request_module[net-pf-10]: fork failed, errno 11

[Athanasius]

Hi,
  I'm seeing a lot of:

	kernel: request_module[net-pf-10]: fork failed, errno 11

in syslog, despite the fact I a) I have no IPv6 compiled in the kernel,
and b) have "alias net-pf-10 off             # IPv6" in
/etc/modules.conf.
  This is using a stock 2.4.18 kernel.  I was under the impression that
the /etc/modules.conf line would lead to such things as above not
happening.  Is the network code doing something slightly askew with
modules?

thanks,

-Ath
-- 
- Athanasius = Athanasius(at)miggy.org.uk / http://www.clan-lovely.org/~athan/
                  Finger athan(at)fysh.org for PGP key
	   "And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME

Re: kernel: request_module[net-pf-10]: fork failed, errno 11

[Keith Owens]

On Sat, 8 Jun 2002 10:43:07 +0100, 
Athanasius <link@gurus.tf> wrote:
>  I'm seeing a lot of:
>
>	kernel: request_module[net-pf-10]: fork failed, errno 11
>
>in syslog, despite the fact I a) I have no IPv6 compiled in the kernel,
>and b) have "alias net-pf-10 off             # IPv6" in
>/etc/modules.conf.
>  This is using a stock 2.4.18 kernel.  I was under the impression that
>the /etc/modules.conf line would lead to such things as above not
>happening.  Is the network code doing something slightly askew with
>modules?

That error is occurring before modprobe has run, long before it gets to
modules.conf.  You need to find out why fork() for modprobe on your
system is failing with EAGAIN.  Have you reached the limit on the
number of tasks?

Re: kernel: request_module[net-pf-10]: fork failed, errno 11

[Athanasius]

On Sat, Jun 08, 2002 at 08:22:05PM +1000, Keith Owens wrote:
> On Sat, 8 Jun 2002 10:43:07 +0100, 
> Athanasius <link@gurus.tf> wrote:
> >  I'm seeing a lot of:
> >
> >	kernel: request_module[net-pf-10]: fork failed, errno 11
[snip]
> 
> That error is occurring before modprobe has run, long before it gets to
> modules.conf.  You need to find out why fork() for modprobe on your
> system is failing with EAGAIN.  Have you reached the limit on the
> number of tasks?

   Not as far as I can tell.  It would be easier to tell if 'sa' (BSD
process accounting) carried useful information like timestamps per
process or even just their PIDs (to match to a *.* syslog'd file).

  What I can say is that in that *.* syslog file the only thing with
'fork' let alone 'fork failed' is the lines as above.  I can't find
where to query the max number of tasks (/proc/sys/kernel/threads-max is
the only likely looking candidate I can find and is 4095) in /proc, and
couldn't seem to find any indication of such with a quick glance at
kernel headers and source.

  The machine in question is somewhat busy, running a lot of shell
accounts, news, http and https, email including mailman lists and other
miscellaneous processes.  However I'd expect to see some other
indication of task starvation if it was occurring.  Right now there
around 220-225 processes running.

-Ath

-- 
- Athanasius = Athanasius(at)miggy.org.uk / http://www.clan-lovely.org/~athan/
                  Finger athan(at)fysh.org for PGP key
	   "And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME

Re: kernel: request_module[net-pf-10]: fork failed, errno 11

[Athanasius]

On Sun, Jun 09, 2002 at 06:54:14PM +0100, Athanasius wrote:
> On Sat, Jun 08, 2002 at 08:22:05PM +1000, Keith Owens wrote:
> > On Sat, 8 Jun 2002 10:43:07 +0100, 
> > Athanasius <link@gurus.tf> wrote:
> > >  I'm seeing a lot of:
> > >
> > >	kernel: request_module[net-pf-10]: fork failed, errno 11
> [snip]
> > 
> > That error is occurring before modprobe has run, long before it gets to
> > modules.conf.  You need to find out why fork() for modprobe on your
> > system is failing with EAGAIN.  Have you reached the limit on the
> > number of tasks?
> 
>    Not as far as I can tell.  It would be easier to tell if 'sa' (BSD
> process accounting) carried useful information like timestamps per
> process or even just their PIDs (to match to a *.* syslog'd file).

  Ok, a little more investigation:

root@bowl:/sbin# cat modprobe-logging 
#!/bin/sh

echo "`date` `ps axh | wc -l`: $@" >> /var/log/modprobe.log
exec /sbin/modprobe $@
root@bowl:/sbin# cat /proc/sys/kernel/modprobe 
/sbin/modprobe-logging

In /var/log/kern.log:
Jun 11 14:36:58 bowl kernel: request_module[net-pf-10]: fork failed, errno 11

And in /var/log/modprobe.log:

Tue Jun 11 14:36:41 BST 2002     229: -s -k -- net-pf-10
Tue Jun 11 14:36:58 BST 2002     228: -s -k -- net-pf-10
Tue Jun 11 14:36:58 BST 2002     227: -s -k -- net-pf-10
Tue Jun 11 14:37:02 BST 2002     227: -s -k -- net-pf-10

/proc/sys/kernel/threads-max is 4095.

  So, nowhere near threads-max (if ps axh is a good test of that),
certainly not processes-wise.  Yet at times the request_module is
failing anyway.  It would seem that an immediate followup attempt works.
Yup, just had another instance:

Jun 11 14:40:11 bowl kernel: request_module[net-pf-10]: fork failed, errno 11
Tue Jun 11 14:40:07 BST 2002     235: -s -k -- net-pf-10
Tue Jun 11 14:40:11 BST 2002     229: -s -k -- net-pf-10
Tue Jun 11 14:40:11 BST 2002     230: -s -k -- net-pf-10
Tue Jun 11 14:40:42 BST 2002     227: -s -k -- net-pf-10

  Note that ulimit -u is 256, but that's per login instance normally and
I'd not have thought a kernel thread goes through PAM anyway...

-Ath
-- 
- Athanasius = Athanasius(at)miggy.org.uk / http://www.clan-lovely.org/~athan/
                  Finger athan(at)fysh.org for PGP key
	   "And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME

Re: kernel: request_module[net-pf-10]: fork failed, errno 11

[Keith Owens]

On Tue, 11 Jun 2002 14:42:25 +0100, 
Athanasius <Athanasius@miggy.org.uk> wrote:
>  Ok, a little more investigation:
>
>root@bowl:/sbin# cat modprobe-logging 
>#!/bin/sh
>
>echo "`date` `ps axh | wc -l`: $@" >> /var/log/modprobe.log
>exec /sbin/modprobe $@
>root@bowl:/sbin# cat /proc/sys/kernel/modprobe 
>/sbin/modprobe-logging

You do not need a special modprobe.log script.  Just mkdir
/var/log/ksymoops and you will get module logging automatically.  man
insmod, look for ksymoops.

>In /var/log/kern.log:
>Jun 11 14:36:58 bowl kernel: request_module[net-pf-10]: fork failed, errno 11
>
>And in /var/log/modprobe.log:
>
>Tue Jun 11 14:36:41 BST 2002     229: -s -k -- net-pf-10
>Tue Jun 11 14:36:58 BST 2002     228: -s -k -- net-pf-10
>Tue Jun 11 14:36:58 BST 2002     227: -s -k -- net-pf-10
>Tue Jun 11 14:37:02 BST 2002     227: -s -k -- net-pf-10

>/proc/sys/kernel/threads-max is 4095.
>  Note that ulimit -u is 256, but that's per login instance normally and
>I'd not have thought a kernel thread goes through PAM anyway...

This is weird.  'fork failed' is issued when kernel_thread() fails.
This means that the fork() syscall for kernel threads is failing,
before the application has even been started.  The fact that you see
other modprobes working only shows that multiple requests for net-pf-10
were issued and some of them got as far as running modprobe but others
failed.