* Large memory application exhuasts buffers during write.
@ 2004-03-26 1:20 Robin Holt
2004-03-26 10:47 ` Denis Vlasenko
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Robin Holt @ 2004-03-26 1:20 UTC (permalink / raw)
To: linux-kernel
We have a large memory application which is being killed by the OOM
killer.
This is a 2.4 based kernel with many of the redhat patches applied.
Before the application is started, there is approx 350GB of memory
free according to top. When the app starts, it mallocs a 300GB
buffer, initializes it, does computations into it, and then starts
to write it to a disk file.
What we see happen is the first approx 30GBs gets written and then
swap starts getting utilized. Once swap has been heavily utilized,
the OOM killer kicks in and kills the job.
The application is a vendor provided app and probably cannot be
modified. Does anybody have any suggestion on possible changes
to make to the kernel to eliminate or significantly reduce the
likelihood that the job gets terminated.
Thanks,
Robin Holt
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Large memory application exhuasts buffers during write.
2004-03-26 1:20 Robin Holt
@ 2004-03-26 10:47 ` Denis Vlasenko
2004-03-26 10:49 ` Robin Holt
2004-03-26 10:57 ` Christoph Hellwig
2004-03-26 14:09 ` John Stoffel
2 siblings, 1 reply; 6+ messages in thread
From: Denis Vlasenko @ 2004-03-26 10:47 UTC (permalink / raw)
To: Robin Holt, linux-kernel
On Friday 26 March 2004 03:20, Robin Holt wrote:
> We have a large memory application which is being killed by the OOM
> killer.
>
> This is a 2.4 based kernel with many of the redhat patches applied.
> Before the application is started, there is approx 350GB of memory
> free according to top. When the app starts, it mallocs a 300GB
> buffer, initializes it, does computations into it, and then starts
> to write it to a disk file.
>
> What we see happen is the first approx 30GBs gets written and then
> swap starts getting utilized. Once swap has been heavily utilized,
> the OOM killer kicks in and kills the job.
How many swap do you have? What do you see in top?
> The application is a vendor provided app and probably cannot be
> modified. Does anybody have any suggestion on possible changes
> to make to the kernel to eliminate or significantly reduce the
> likelihood that the job gets terminated.
--
vda
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Large memory application exhuasts buffers during write.
2004-03-26 10:47 ` Denis Vlasenko
@ 2004-03-26 10:49 ` Robin Holt
0 siblings, 0 replies; 6+ messages in thread
From: Robin Holt @ 2004-03-26 10:49 UTC (permalink / raw)
To: Denis Vlasenko; +Cc: Robin Holt, linux-kernel
On Fri, Mar 26, 2004 at 12:47:10PM +0200, Denis Vlasenko wrote:
> On Friday 26 March 2004 03:20, Robin Holt wrote:
> > We have a large memory application which is being killed by the OOM
> > killer.
> >
> > This is a 2.4 based kernel with many of the redhat patches applied.
> > Before the application is started, there is approx 350GB of memory
> > free according to top. When the app starts, it mallocs a 300GB
> > buffer, initializes it, does computations into it, and then starts
> > to write it to a disk file.
> >
> > What we see happen is the first approx 30GBs gets written and then
> > swap starts getting utilized. Once swap has been heavily utilized,
> > the OOM killer kicks in and kills the job.
>
> How many swap do you have? What do you see in top?
I am not sure how much swap is configured or available, but I
highly doubt that there is 350GB of swap.
Robin
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Large memory application exhuasts buffers during write.
2004-03-26 1:20 Robin Holt
2004-03-26 10:47 ` Denis Vlasenko
@ 2004-03-26 10:57 ` Christoph Hellwig
2004-03-26 14:09 ` John Stoffel
2 siblings, 0 replies; 6+ messages in thread
From: Christoph Hellwig @ 2004-03-26 10:57 UTC (permalink / raw)
To: Robin Holt; +Cc: linux-kernel
On Thu, Mar 25, 2004 at 07:20:56PM -0600, Robin Holt wrote:
> This is a 2.4 based kernel with many of the redhat patches applied.
> Before the application is started, there is approx 350GB of memory
> free according to top. When the app starts, it mallocs a 300GB
> buffer, initializes it, does computations into it, and then starts
> to write it to a disk file.
>
> What we see happen is the first approx 30GBs gets written and then
> swap starts getting utilized. Once swap has been heavily utilized,
> the OOM killer kicks in and kills the job.
Buffered writes or O_DIRECT? I guess you're doing the former and
actually want the latter. Try preloading a tiny library stub that
adds O_DIRECT to open for the interesting fds.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Large memory application exhuasts buffers during write.
2004-03-26 1:20 Robin Holt
2004-03-26 10:47 ` Denis Vlasenko
2004-03-26 10:57 ` Christoph Hellwig
@ 2004-03-26 14:09 ` John Stoffel
2 siblings, 0 replies; 6+ messages in thread
From: John Stoffel @ 2004-03-26 14:09 UTC (permalink / raw)
To: Robin Holt; +Cc: linux-kernel
Robin> What we see happen is the first approx 30GBs gets written and then
Robin> swap starts getting utilized. Once swap has been heavily utilized,
Robin> the OOM killer kicks in and kills the job.
Can you post the output of 'vmstat 1' starting just before your app is
fired up, and until OOM kicks in an kills it off? And more details
on exactly which version of Linux kernel you're running and how the
system is configured as well.
John
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2004-03-26 14:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <5F0021EEA434D511BE7300D0B7B6AB530EE480BB@mail2.ggn.hcltech.com>
2004-03-26 11:16 ` Large memory application exhuasts buffers during write Christoph Hellwig
2004-03-26 1:20 Robin Holt
2004-03-26 10:47 ` Denis Vlasenko
2004-03-26 10:49 ` Robin Holt
2004-03-26 10:57 ` Christoph Hellwig
2004-03-26 14:09 ` John Stoffel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®