Re: Memory usage per memory zone

[Wu Fengguang <fengguang.wu@intel.com>]

On Thu, Mar 12, 2009 at 10:17:44AM +0200, jack marrow wrote:
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >>  http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
> >
> > The cached files seem to be the memory killer:
> 
> I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
> find no oom kill. Oh well :)

Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?

> I thought the kernel dropped caches if a program needs the ram?

Sure, but something is unreclaimable... Maybe some process is taking a
lot of shared memory(shm)? What's the output of `lsof`?

> >
> > MemTotal:      1034496 kB
> > MemFree:         95600 kB
> > Buffers:         49916 kB
> > Cached:         761544 kB
> > SwapCached:          0 kB
> > Active:          80484 kB
> > Inactive:       749960 kB
> > HighTotal:      131008 kB
> > HighFree:        68480 kB
> > LowTotal:       903488 kB
> > LowFree:         27120 kB
> > SwapTotal:     2040212 kB
> > SwapFree:      2039780 kB
> > Dirty:               4 kB
> > Writeback:           0 kB
> > Mapped:          32636 kB
> > Slab:            93856 kB
> > CommitLimit:   2557460 kB
> > Committed_AS:   129980 kB
> > PageTables:       1800 kB
> > VmallocTotal:   106488 kB
> > VmallocUsed:      3372 kB
> > VmallocChunk:   102616 kB
> > HugePages_Total:     0
> > HugePages_Free:      0
> > Hugepagesize:     2048 kB
> >
> > Is upgrading the kernel an option for you?
> 
> No :(
> 
> I think shoving some more ram in the box is the best doable option.
> Would this help here?

There have been huge amounts of change sets in mm area since 2.6.9...

> To do that I need to say "look at how much cache we are using for
> files, that cache is in the high mem zone (look here) so let's put
> some more ram in". Does the cache always live in the high mem zone?

Both highmem and normal zones will be used for caches.

Thanks,
Fengguang