negative anon_rss?

negative anon_rss?

[Pádraig Brady]

Sorry for not sending this to linux-mm,
but I can't subscribe at present? Anyway...

I wrote the following script to try and accurately determine
how much RAM a particular program uses:
http://www.pixelbeat.org/scripts/ps_mem.py

A user reported an issue on debian with kernel 2.6.8-2-386
where many processes were being reported as using
a negative amount of memory.

I asked him to run the following:

  (
    echo total rss shared trs - drs -
    for pid in `pidof apache2`; do
        cat /proc/$pid/statm
    done
  ) | column -t

the output of which was:

  total  rss   shared  trs  -  drs   -
  6580   2306  5273    95   0  6485  0
  6580   2313  5273    95   0  6485  0
  6119   1717  5269    95   0  6024  0
  6630   2371  5273    95   0  6535  0
  6735   2503  5273    95   0  6640  0
  6773   2546  5273    95   0  6678  0
  5845   1146  5198    95   0  5750  0

Notice the large values for the shared column.
Also notice that the shared column is larger than rss!?
I had assumed that shared was a subset of rss from
the following (pseudo) code from fs/proc/task_mmu.c::task_statm()

  *total  = mm->total_em
  *shared = get_mm_counter(mm, file_rss)
  *rss    = *shared + get_mm_counter(mm, anon_rss)
  *trs    = mm->end_code - mm->start_code
  *drs    = mm->total_vm - mm->shared_vm

Therefore anon_rss must be negative in the kernel?
So there is a mismatch between pages being marked
as anonymous and anon_rss being updated appropriately?

cheers,
Pádraig.

Re: negative anon_rss?

[Hugh Dickins]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3406 bytes --]

On Wed, 15 Nov 2006, Pádraig Brady wrote:
> Sorry for not sending this to linux-mm,
> but I can't subscribe at present? Anyway...
> 
> I wrote the following script to try and accurately determine
> how much RAM a particular program uses:
> http://www.pixelbeat.org/scripts/ps_mem.py
> 
> A user reported an issue on debian with kernel 2.6.8-2-386
> where many processes were being reported as using
> a negative amount of memory.
> 
> I asked him to run the following:
> 
>   (
>     echo total rss shared trs - drs -
>     for pid in `pidof apache2`; do
>         cat /proc/$pid/statm
>     done
>   ) | column -t
> 
> the output of which was:
> 
>   total  rss   shared  trs  -  drs   -
>   6580   2306  5273    95   0  6485  0
>   6580   2313  5273    95   0  6485  0
>   6119   1717  5269    95   0  6024  0
>   6630   2371  5273    95   0  6535  0
>   6735   2503  5273    95   0  6640  0
>   6773   2546  5273    95   0  6678  0
>   5845   1146  5198    95   0  5750  0
> 
> Notice the large values for the shared column.
> Also notice that the shared column is larger than rss!?
> I had assumed that shared was a subset of rss from
> the following (pseudo) code from fs/proc/task_mmu.c::task_statm()
> 
>   *total  = mm->total_em
>   *shared = get_mm_counter(mm, file_rss)
>   *rss    = *shared + get_mm_counter(mm, anon_rss)
>   *trs    = mm->end_code - mm->start_code
>   *drs    = mm->total_vm - mm->shared_vm
> 
> Therefore anon_rss must be negative in the kernel?
> So there is a mismatch between pages being marked
> as anonymous and anon_rss being updated appropriately?

You're looking at recent source to explain what happened in 2.6.8:
not a good strategy.  2.6.8 didn't even record file_rss and anon_rss
separately.  We split off anon_rss in 2.6.10, and changed the meaning
of "shared" then too - here's my ChangeLog comment justifying it.

From: Hugh Dickins <hugh@veritas.com>
Date: Thu, 28 Oct 2004 01:17:23 +0000 (-0700)
Subject: [PATCH] statm: shared = rss - anon_rss
X-Git-Tag: v2.6.10~147^2~103
X-Git-Url: http://127.0.0.1:1234/?p=.git;a=commitdiff_plain;h=8586febd93ed02668b9d0ad87963610425e44977;hp=da63671a5c506729abb8fe383704c264967c53c0

[PATCH] statm: shared = rss - anon_rss

The third "shared" field of /proc/$pid/statm in 2.4 was a count of pages in
the mm whose page_count is more than 1 (oddly, including pages shared just
with swapcache).  That's too costly to calculate each time, so 2.6 changed
it to the total file-backed extent.  But Andrea knows apps and users
surprised when (rss - shared) goes negative: we need to provide an rss-like
statistic, close to the 2.4 interpretation.

Something that's quick and easy to maintain accurately is mm->anon_rss, the
count of anonymous pages in the mm.  Then shared = rss - anon_rss gives a
pretty good and meaningful approximation to 2.4's intention: wli confirms
that this will be useful to Oracle too.

Where to show it?  I think it's best to treat this as a bugfix and show it
in the third field of /proc/$pid/statm, after resident, as before - there's
no evidence that the total file-backed extent was found useful.

Albert would like other fields to revert to page counts, but that's a lot
harder: if mprotect can change the category of a page, then it can't be
accounted as simply as this.  Only go that route if real need shown.

Hugh