Re: [PATCH] Fix SUNRPC wakeup/execute race condition

[Trond Myklebust <trond.myklebust@fys.uio.no>]

On Sun, 2006-11-05 at 02:04 +0100, Christophe Saout wrote:
> The sunrpc scheduler contains a race condition that can let an RPC
> task end up being neither running nor on any wait queue. The race takes
> place between rpc_make_runnable (called from rpc_wake_up_task) and
> __rpc_execute under the following condition:
> 
> First __rpc_execute calls tk_action which puts the task on some wait
> queue. The task is dequeued by another process before __rpc_execute
> continues its execution. While executing rpc_make_runnable exactly after
> setting the task `running' bit and before clearing the `queued' bit
> __rpc_execute picks up execution, clears `running' and subsequently
> both functions fall through, both under the false assumption somebody
> else took the job.
> 
> Swapping rpc_test_and_set_running with rpc_clear_queued in
> rpc_make_runnable fixes that hole. The reordering hopefully doesn't
> introduce some new race condition, in fact the only possible one is
> already correctly detected and handled in __rpc_execute.
> 
> Bug noticed on a 4-way x86_64 system under XEN with an NFSv4 server
> on the same physical machine, apparently one of the few ways to hit
> this race condition at all.
> 
> Cc: Trond Myklebust <trond.myklebust@fys.uio.no>
> Cc: J. Bruce Fields <bfields@citi.umich.edu>
> Cc: Olaf Kirch <okir@monad.swb.de>
> Signed-off-by: Christophe Saout <christophe@saout.de>
> 
> --- linux-2.6.18/net/sunrpc/sched.c	2006-09-20 05:42:06.000000000 +0200
> +++ linux/net/sunrpc/sched.c	2006-11-04 20:38:56.000000000 +0100
> @@ -302,12 +302,9 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
>   */
>  static void rpc_make_runnable(struct rpc_task *task)
>  {
> -	int do_ret;
> -
>  	BUG_ON(task->tk_timeout_fn);
> -	do_ret = rpc_test_and_set_running(task);
>  	rpc_clear_queued(task);
> -	if (do_ret)
> +	if (rpc_test_and_set_running(task))
>  		return;
>  	if (RPC_IS_ASYNC(task)) {
>  		int status;

This fix looks wrong to me. If we've made it to 'rpc_make_runnable',
then the rpc_task will have already been removed from the
rpc_wait_queue.
The only question left is "who is responsible for waking up the
synchronous task / setting up the asynchronous workqueue item?".

Cheers,
  Trond