Re: [PATCH] module, fix percpu reserved memory exhaustion

[Tejun Heo <tj@kernel.org>]

Hello, Rusty.

On Sat, Jan 12, 2013 at 11:36:09AM +1030, Rusty Russell wrote:
> > I've been thinking about that.  The problem is that at the same time the kvm
> > problem occurs I'm attempting to load a debug module that I've written to debug
> > some cpu timer issues that allocates a large amount of percpu data (~.5K/cpu).
> > While extending PERCPU_MODULE_RESERVE to 10k might work now, it might not work
> > tomorrow if I have the need to increase the size of my log buffer.
> 
> Well, it looks like PERCPU_MODULE_RESERVE is actually very generous; it
> could easily be halved.  I guess this is because dynamic per-cpu data is
> now such a nice alternative (thanks to Tejun).

Heh, thanks. :)

There are two percpu constants that I basically pulled out of my ass.

* PERCPU_MODULE_RESERVE: It's used on architectures with data model
  which restricts the distance between code and data.  e.g. x86_64
  genereates code which can't address full 64bit for static data and
  as percpu addressing depends on the regular address generation to
  derive percpu addresses, the percpu offsets need to be restricted
  somehow.  Percpu allocator currently achieves this by reserving
  certain amount in the first percpu chunk which is guaranteed to have
  very low (starting from 0) percpu address space offsets.

  On those archs, this becomes the hard limit for all module static
  percpu areas.  Maybe 8k is way too high and we can go with 4k.
  Worth a try I guess.

* PERCPU_DYNAMIC_RESERVE: The amount of space at the end of the first
  chunk used for dynamic allocation.  This isn't as critical as the
  above and the only reason it exists is because the first chunk is
  often embedded in the kernel linear address space instead of vmalloc
  area and the former is often mapped with larger page table size than
  the latter.  The goal of PERCPU_DYNAMIC_RESERVE is to have just
  enough space to cover the usual percpu allocations without wasting
  too much space to minimize impact on TLB pressure.

  The current number isn't entirely made up.  I did some testing with
  some random config.  Don't know how it holds up now.  It probably
  can be increased given that we developed quite a few percpu dynamic
  allocations since then.

  It would be awesome to have a way to somehow dynamically adjust this
  one; unfortunately, that would require persistent data across
  boots. :(

Thanks.

-- 
tejun