2.6.20, Java, unresponsive userspace

[Alistair John Strachan <s0348365@sms.ed.ac.uk>]

Hi,

I've had some problems with 2.6.20 (but also with earlier kernels, like 
Debian's 2.6.18) where one of the users on our system, running a Java6 
application, is somehow able to make the userspace on the machine totally 
unresponsive, denying all remote _and_ local access to the machine.

- The AMD64 port of Linux and Java are in use
- The process runs as an unprivileged user
- The OOM killer doesn't kick in
- No swap is enabled on the machine
- I'm told that the program doesn't fork bomb
  (but I've not got a reproducible test case yet)
- The program is able to deny service even to a getty login.

If I SAK the machine, init respawns getty and everything is fine again.

alistair@clarent:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) 1000
max memory size         (kbytes, -m) 1000
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) 0
stack size              (kbytes, -s) 2048
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Any ideas with how to proceed for debugging this issue?

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.