Re: Conveying memory pressure to userspace?

["Martin J. Bligh" <mbligh@mbligh.org>]

Bas Westerbaan wrote:
> Hello,
> 
> Quite a lot of userspace applications cache.  Firefox caches pages;
> mySQL caches queries;  libc' free (like practically all other
> userspace allocators) won't directly return the first byte of memory
> freed, etc.
> 
> These applications are unaware of memory pressure.  While the kernel
> is trying its best to to free memory by for instance dropping,
> possibly more valuable caches, userspace is left blissfully unaware.
> 
> Obviously this isn't a really big problem, given that we've still got
> swap to swap out those rarely used caches, except for when the caches
> aren't _that_ rarely used and of which the backing store (eg.
> precomputed values) might be faster than the disk to swap back the
> pages from.
> 
> A solution would be to either
> 
>  a) let the application make the kernel aware of pages that, when in
> memory pressure, may be dropped.  This would be tricky to implement
> for the userspace: it's hard to avoid an application to race into a
> dropped page.  However, the kernel can directly free a page from
> userspace, which makes it use full when under real pressure.  This in
> contrast to
> b) letting the application register itself with a cache share
> priority.  The application (and other aware applications) would then
> be able to query how fair they are at the moment proportional to their
> cache share priority.  Freeing would still be completely in their own
> hands.
> 
> 
> The only relevant related matter I could find were madvise and mincore.
> 
> With madvise pages can be marked to be unnecessary and these should be
> swapped out earlier.  With mincore one can determine whether pages are
> resident (not cached).  This would make an existing alternative to
> solution a.  However, this doesn't eliminate the writes to the swap
> and polling everytime before accessing a cache isn't really pretty.
> 
> I did consider guessing the memory pressure by looking at
> /proc/meminfo, but I think it isn't that accurate.

The prev_priority field in the zoneinfo stuff is more useful for
memory pressure. I'm playing with making a blocking callback that
can wake someone up when this gets down to a certain priority level
(prio=12 => everything's rosy, prio=0 => we're in deep shit).

> Before hacking something together (and being uncertain about the
> thoroughness with which I searched for existing work, sorry), I would
> like your thoughts on this.
> 
> Please CC me, I'm not in the list.
> 
>  Bas
>