Re: rcu-refcount stacker performance

[serue@us.ibm.com]

Quoting Paul E. McKenney (paulmck@us.ibm.com):
> On Thu, Jul 14, 2005 at 08:44:50AM -0500, serue@us.ibm.com wrote:
> > Quoting Paul E. McKenney (paulmck@us.ibm.com):
> > > My guess is that the reference count is indeed costing you quite a
> > > bit.  I glance quickly at the patch, and most of the uses seem to
> > > be of the form:
> > > 
> > > 	increment ref count
> > > 	rcu_read_lock()
> > > 	do something
> > > 	rcu_read_unlock()
> > > 	decrement ref count
> > > 
> > > Can't these cases rely solely on rcu_read_lock()?  Why do you also
> > > need to increment the reference count in these cases?
> > 
> > The problem is on module unload: is it possible for CPU1 to be
> > on "do something", and sleep, and, while it sleeps, CPU2 does
> > rmmod(lsm), so that by the time CPU1 stops sleeping, the code it
> > is executing has been freed?
> 
> OK, but in the above case, "do something" cannot be sleeping, since
> it is under rcu_read_lock().

Oh, but that's not quite what the code is doing, rather it is doing:

	rcu_read_lock
	while get next element from list
		inc element.refcount
		rcu_read_unlock
		do something
		rcu_read_lock
		dec refcount
	rcu_read_unlock

What I plan to try next is:

	rcu_read_lock
	while get next element from list
		if (element->owning_module->state != LIVE)
			continue
		rcu_read_unlock
		do something
		rcu_read_lock
	rcu_read_unlock

> > Because stacker won't remove the lsm from the list of modules
> > until mod->exit() is executed, and module_free(mod) happens
> > immediately after that, the above scenario seems possible.
> 
> Right, if you have some other code path that sleeps (outside of
> rcu_read_lock(), right?), then you need the reference count for that
> code path.  But the code paths that do not sleep should be able to
> dispense with the reference count, reducing the cache-line traffic.

Most if not all of the codepaths can sleep, however.  So unfortunately
that doesn't seem a feasible solution.  That's why I'm hoping there is
something inherent in the module unload code that I can take advantage
of to forego my own refcounting.

thanks,
-serge