Large memory application exhuasts buffers during write.

[Robin Holt <holt@sgi.com>]

We have a large memory application which is being killed by the OOM
killer.

This is a 2.4 based kernel with many of the redhat patches applied.
Before the application is started, there is approx 350GB of memory
free according to top.  When the app starts, it mallocs a 300GB
buffer, initializes it, does computations into it, and then starts
to write it to a disk file.

What we see happen is the first approx 30GBs gets written and then
swap starts getting utilized.  Once swap has been heavily utilized,
the OOM killer kicks in and kills the job.

The application is a vendor provided app and probably cannot be
modified.  Does anybody have any suggestion on possible changes
to make to the kernel to eliminate or significantly reduce the
likelihood that the job gets terminated.

Thanks,
Robin Holt