Re: Memory usage per memory zone

[jack marrow <jackmarrow2@gmail.com>]

2009/3/11 Thomas Schoebel-Theuer <thomas.schoebel-theuer@1und1.de>:
> Am Mittwoch, 11. März 2009 11:41:43 schrieb jack marrow:
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> Hi Jack,
>
> my experience with plain old 2.6.24 on 32bit _production_ boxes was that under
> heavy load and after >30days uptime we saw an sudden inflation of oom-killers
> on some of them until those boxes died. The standard kernel statistics about
> memory looked much the same as yours (and I suspect they could have been
> wrong or at least misleading, but I have neither checked nor tried to fix).
>
>> is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> Try the attached experimental hack which can provide you with some insight
> about whats really going on in the _physical_ memory. Since it does not
> allocate any memory for the purpose of displaying those memory patterns it
> wants to examine, you have to allocate a large enough buffer in userspace.
> Don't use cat, but something like dd with parameters such as bs=4M (as
> mentioned in the comment). Probably you have to adjust the patch to some
> newer kernel versions, and/or to fix some sysctl table checks if you want to
> get it upstreams (I will not). And, of course, you can visualize more/other
> flags as well.
>
> After gaining some insight with /proc/sys/vm/mempattern and some development
> of further experimental patches which successfully reduced fragmentation (but
> finally only _delayed_ the oom problems without being able to _fundamentally_
> resolve them), the ultimate solution was just to use CONFIG_VMSPLIT_2G or
> even CONFIG_VMSPLIT_1G in order to overcome the artificial shortening of
> zone_normal.
>
> This supports old wisdom that an OS cannot give you resources it just does not
> possess... Just make sure you have enough resources for your working set.
> Thats all.
>
> In the hope of being helpful,
>
> Thomas
>

Thanks a lot for this Thomas. I'm not able to run a patched kernel on
this box, but I'll keep this in mind for future.