[RFC] do_execve() perf improvement opportunity?

[RFC] do_execve() perf improvement opportunity?

[cutaway]

do_execve() on EVERY entry/exit allocates and frees a structure pointed to
by the 'bprm' variable.

I'm thinking it may be possible to very cheaply cache a pointer to the last
allocation here rather than freeing it and just recycle it for the next exec
saving a trip through the slab machanism.

On x86 I'm pretty sure this could be done very racy and lockless (other than
XCHG's implied locks).  Other architectures that don't have an implied
locking instruction might need a hard lock of some sort.

Something along the lines of (pseudocode):

static volatile struct blahblah *p = NULL;

/* ...before the exec... */
bprm = NULL
xchg(bprm, p)
if (bprm == NULL) kmalloc like it is now

/*
  blah, blah, blah...exec triage blob as it exists today
*/

/* ...after the exec...*/
xchg(bprm, p) /* cache what we just used */
if (bprm)     /* Maybe free someone else's if it was still available */
    kfree(bprm);

For things that proceed mostly sequentially like a lot of shell scripts,
Linux builds, etc this simple minded high-speed low-drag, single structure
racy implementation might provide a nice gain for minimal cost.

Re: [RFC] do_execve() perf improvement opportunity?

[Rik van Riel]

On Tue, 21 Jun 2005 cutaway@bellsouth.net wrote:

> I'm thinking it may be possible to very cheaply cache a pointer to the 
> last allocation here rather than freeing it and just recycle it for the 
> next exec saving a trip through the slab machanism.

Note that the slab mechanism can do allocations locally
on each CPU in an SMP system, while your pointer would
need some cross-CPU synchronisation.  Also, you could
end up using the bprm from a CPU on a remote NUMA node,
instead of a local piece of memory.

Still, it would be interesting/educational to know if your
optimisation makes a difference on single CPU systems.

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: [RFC] do_execve() perf improvement opportunity?

[cutaway]

Rik, it would certainly fail miserably on any architecture where different
CPU's have any CPU local memory under kernel managment.

Anything resembling the old homogeneous Hydra scheme should be fine though I
would think.

NUMA is odd enough that the kernel is littered with conditionals for it, one
more wouldn't be a big deal ;->

I'll try to code this up and benchmark it and see if there's anything
measurable.  If there is, this sort of simple minded "cache the last one"
scheme might be applicable elsewhere too - pipes, maybe net packets, etc.
It looks like Slab already sort of "caches the last one" on the different
granularities, but it takes a bit more code to get to the point where it
finally figures out it can give you back a cached one.

Maybe there's something to be gained by having an internal special case
allocator for limited numbers of small things (like 32, 64, 128, 256 maybe
where a bit scan instruction or two can trivially find you an empty slot)?
Where the allocator degenerates to just setting a flag byte on the smaller
slices and generating the pointer from a bit index.


----- Original Message ----- 
From: "Rik van Riel" <riel@redhat.com>
To: <cutaway@bellsouth.net>
Cc: <linux-kernel@vger.kernel.org>
Sent: Tuesday, June 21, 2005 09:56
Subject: Re: [RFC] do_execve() perf improvement opportunity?


> On Tue, 21 Jun 2005 cutaway@bellsouth.net wrote:
>
> > I'm thinking it may be possible to very cheaply cache a pointer to the
> > last allocation here rather than freeing it and just recycle it for the
> > next exec saving a trip through the slab machanism.
>
> Note that the slab mechanism can do allocations locally
> on each CPU in an SMP system, while your pointer would
> need some cross-CPU synchronisation.  Also, you could
> end up using the bprm from a CPU on a remote NUMA node,
> instead of a local piece of memory.
>
> Still, it would be interesting/educational to know if your
> optimisation makes a difference on single CPU systems.

Re: [RFC] do_execve() perf improvement opportunity?

[Rik van Riel]

On Tue, 21 Jun 2005 cutaway@bellsouth.net wrote:

> I'll try to code this up and benchmark it and see if there's anything 
> measurable.  If there is, this sort of simple minded "cache the last 
> one" scheme might be applicable elsewhere too - pipes, maybe net 
> packets, etc. It looks like Slab already sort of "caches the last one" 
> on the different granularities, but it takes a bit more code to get to 
> the point where it finally figures out it can give you back a cached 
> one.

The thing is, that code may well be in cache already, while
a cache miss on a piece of data from another CPU is really
really expensive on SMP systems.

I suspect you may be able to get more performance gains
from inserting prefetches in strategic places than from
cutting out a bit of code.

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics