slab: Add transfer_objects() function

slab: Add transfer_objects() function

[Christoph Lameter]

transfer_objects() can be used to transfer objects between various object
caches of the slab allocator. It is currently only used during __cache_alloc() to
retrieve elements from the shared array. We will be using it soon to transfer
elements from the alien caches to the remote shared array.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.16-rc6-mm2/mm/slab.c
===================================================================
--- linux-2.6.16-rc6-mm2.orig/mm/slab.c	2006-03-21 14:52:47.000000000 -0800
+++ linux-2.6.16-rc6-mm2/mm/slab.c	2006-03-21 14:53:40.000000000 -0800
@@ -901,6 +901,30 @@ static struct array_cache *alloc_arrayca
 	return nc;
 }
 
+/*
+ * Transfer objects in one arraycache to another.
+ * Locking must be handled by the caller.
+ *
+ * Return the number of entries transferred.
+ */
+static int transfer_objects(struct array_cache *to,
+		struct array_cache *from, int max)
+{
+	/* Figure out how many entries to transfer */
+	int nr = min(min(from->avail, max), to->limit - to->avail);
+
+	if (!nr)
+		return 0;
+
+	memcpy(to->entry + to->avail, from->entry + from->avail -nr,
+			sizeof(void *) *nr);
+
+	from->avail -= nr;
+	to->avail += nr;
+	to->touched = 1;
+	return nr;
+}
+
 #ifdef CONFIG_NUMA
 static void *__cache_alloc_node(struct kmem_cache *, gfp_t, int);
 static void *alternate_node_alloc(struct kmem_cache *, gfp_t);
@@ -2684,20 +2708,10 @@ retry:
 	BUG_ON(ac->avail > 0 || !l3);
 	spin_lock(&l3->list_lock);
 
-	if (l3->shared) {
-		struct array_cache *shared_array = l3->shared;
-		if (shared_array->avail) {
-			if (batchcount > shared_array->avail)
-				batchcount = shared_array->avail;
-			shared_array->avail -= batchcount;
-			ac->avail = batchcount;
-			memcpy(ac->entry,
-			       &(shared_array->entry[shared_array->avail]),
-			       sizeof(void *) * batchcount);
-			shared_array->touched = 1;
-			goto alloc_done;
-		}
-	}
+	/* See if we can refill from the shared array */
+	if (l3->shared && transfer_objects(ac, l3->shared, batchcount))
+		goto alloc_done;
+
 	while (batchcount > 0) {
 		struct list_head *entry;
 		struct slab *slabp;

slab: Bypass free lists for __drain_alien_cache()

[Christoph Lameter]

__drain_alien_cache() currently drains objects by freeing them to the 
(remote) freelists of the original node. However, each node also has a 
shared list containing objects to be used on any processor of that node. 
We can avoid a number of remote node accesses by copying the pointers to 
the free objects directly into the remote shared array.

Depends on the earlier patch that introduces the transfer_objects() function.

And while we are at it: Skip alien draining if the alien cache spinlock is
already taken.

Kiran reported that this is a performance benefit.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.16-rc6-mm2/mm/slab.c
===================================================================
--- linux-2.6.16-rc6-mm2.orig/mm/slab.c	2006-03-21 14:53:40.000000000 -0800
+++ linux-2.6.16-rc6-mm2/mm/slab.c	2006-03-21 14:55:09.000000000 -0800
@@ -974,6 +974,13 @@ static void __drain_alien_cache(struct k
 
 	if (ac->avail) {
 		spin_lock(&rl3->list_lock);
+		/*
+		 * Stuff objects into the remote nodes shared array first.
+		 * That way we could avoid the overhead of putting the objects
+		 * into the free lists and getting them back later.
+		 */
+		transfer_objects(rl3->shared, ac, ac->limit);
+
 		free_block(cachep, ac->entry, ac->avail, node);
 		ac->avail = 0;
 		spin_unlock(&rl3->list_lock);
@@ -989,8 +996,8 @@ static void reap_alien(struct kmem_cache
 
 	if (l3->alien) {
 		struct array_cache *ac = l3->alien[node];
-		if (ac && ac->avail) {
-			spin_lock_irq(&ac->lock);
+
+		if (ac && ac->avail && spin_trylock_irq(&ac->lock)) {
 			__drain_alien_cache(cachep, ac, node);
 			spin_unlock_irq(&ac->lock);
 		}

Re: slab: Add transfer_objects() function

[Andrew Morton]

Christoph Lameter <clameter@sgi.com> wrote:
>
> +static int transfer_objects(struct array_cache *to,
> +		struct array_cache *from, int max)

Does this ever get called if !CONFIG_NUMA?

If not, can we provide a non-numa version which just goes BUG and saves
some text?

Re: slab: Add transfer_objects() function

[Christoph Lameter]

On Tue, 21 Mar 2006, Andrew Morton wrote:

> Christoph Lameter <clameter@sgi.com> wrote:
> >
> > +static int transfer_objects(struct array_cache *to,
> > +		struct array_cache *from, int max)
> 
> Does this ever get called if !CONFIG_NUMA?

Yes. See later in the patch.