mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Trond Myklebust <trond.myklebust@fys.uio.no>
To: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Cc: "'Andrew Morton'" <akpm@osdl.org>,
	linux-aio@kvack.org, "'xb'" <xavier.bru@bull.net>,
	linux-kernel@vger.kernel.org,
	"'Zach Brown'" <zach.brown@oracle.com>
Subject: RE: 2.6.18.4: flush_workqueue calls mutex_lock in interrupt environment
Date: Fri, 15 Dec 2006 08:49:21 -0500	[thread overview]
Message-ID: <1166190561.5761.12.camel@lade.trondhjem.org> (raw)
In-Reply-To: <000001c72013$2a1a90c0$b481030a@amr.corp.intel.com>

On Thu, 2006-12-14 at 22:35 -0800, Chen, Kenneth W wrote:
> Chen, Kenneth wrote on Thursday, December 14, 2006 5:59 PM
> > > It seems utterly insane to have aio_complete() flush a workqueue. That
> > > function has to be called from a number of different environments,
> > > including non-sleep tolerant environments.
> > > 
> > > For instance it means that directIO on NFS will now cause the rpciod
> > > workqueues to call flush_workqueue(aio_wq), thus slowing down all RPC
> > > activity.
> > 
> > The bug appears to be somewhere else, somehow the ref count on ioctx is
> > all messed up.
> > 
> > In aio_complete, __put_ioctx() should not be invoked because ref count
> > on ioctx is supposedly more than 2, aio_complete decrement it once and
> > should return without invoking the free function.

This makes absolutely no sense whatsoever. If the refcount is 'always
supposedo to be more than 2' then why would you need a refcount at all
in aio_complete?

> > The real freeing ioctx should be coming from exit_aio() or io_destroy(),
> > in which case both wait until no further pending AIO request via
> > wait_for_all_aios().
> 
> Ah, I think I see the bug: it must be a race between io_destroy() and
> aio_complete().  A possible scenario:
> 
> cpu0                               cpu1
> io_destroy                         aio_complete
>   wait_for_all_aios {                __aio_put_req
>      ...                                 ctx->reqs_active--;
>      if (!ctx->reqs_active)
>         return;
>   }
>   ...
>   put_ioctx(ioctx)
> 
>                                      put_ioctx(ctx);
>                                         bam! Bug trigger!
> 
> AIO finished on cpu1 and while in the middle of aio_complete, cpu0 starts
> io_destroy sequence, sees no pending AIO, went ahead decrement the ref
> count on ioctx.  At a later point in aio_complete, the put_ioctx decrement
> last ref count and calls the ioctx freeing function and there it triggered
> the bug warning.
> 
> A simple fix would be to access ctx->reqs_active inside ctx spin lock in wait_for_all_aios().  At the mean time, I would like to
> remove ref counting
> for each iocb because we already performing ref count using reqs_active. This
> would also prevent similar buggy code in the future.
> 
> 
> Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
> 
> --- ./fs/aio.c.orig	2006-11-29 13:57:37.000000000 -0800
> +++ ./fs/aio.c	2006-12-14 20:45:14.000000000 -0800
> @@ -298,17 +298,23 @@ static void wait_for_all_aios(struct kio
>  	struct task_struct *tsk = current;
>  	DECLARE_WAITQUEUE(wait, tsk);
>  
> +	spin_lock_irq(&ctx->ctx_lock);
>  	if (!ctx->reqs_active)
> -		return;
> +		goto out;
>  
>  	add_wait_queue(&ctx->wait, &wait);
>  	set_task_state(tsk, TASK_UNINTERRUPTIBLE);
>  	while (ctx->reqs_active) {
> +		spin_unlock_irq(&ctx->ctx_lock);
>  		schedule();
>  		set_task_state(tsk, TASK_UNINTERRUPTIBLE);
> +		spin_lock_irq(&ctx->ctx_lock);
>  	}
>  	__set_task_state(tsk, TASK_RUNNING);
>  	remove_wait_queue(&ctx->wait, &wait);
> +
> +out:
> +	spin_unlock_irq(&ctx->ctx_lock);
>  }
>  
>  /* wait_on_sync_kiocb:
> @@ -425,7 +431,6 @@ static struct kiocb fastcall *__aio_get_
>  	ring = kmap_atomic(ctx->ring_info.ring_pages[0], KM_USER0);
>  	if (ctx->reqs_active < aio_ring_avail(&ctx->ring_info, ring)) {
>  		list_add(&req->ki_list, &ctx->active_reqs);
> -		get_ioctx(ctx);
>  		ctx->reqs_active++;
>  		okay = 1;
>  	}
> @@ -538,8 +543,6 @@ int fastcall aio_put_req(struct kiocb *r
>  	spin_lock_irq(&ctx->ctx_lock);
>  	ret = __aio_put_req(ctx, req);
>  	spin_unlock_irq(&ctx->ctx_lock);
> -	if (ret)
> -		put_ioctx(ctx);
>  	return ret;
>  }
>  
> @@ -795,8 +798,7 @@ static int __aio_run_iocbs(struct kioctx
>  		 */
>  		iocb->ki_users++;       /* grab extra reference */
>  		aio_run_iocb(iocb);
> -		if (__aio_put_req(ctx, iocb))  /* drop extra ref */
> -			put_ioctx(ctx);
> +		__aio_put_req(ctx, iocb);
>   	}
>  	if (!list_empty(&ctx->run_list))
>  		return 1;
> @@ -942,7 +944,6 @@ int fastcall aio_complete(struct kiocb *
>  	struct io_event	*event;
>  	unsigned long	flags;
>  	unsigned long	tail;
> -	int		ret;
>  
>  	/*
>  	 * Special case handling for sync iocbs:
> @@ -1011,18 +1012,12 @@ int fastcall aio_complete(struct kiocb *
>  	pr_debug("%ld retries: %zd of %zd\n", iocb->ki_retried,
>  		iocb->ki_nbytes - iocb->ki_left, iocb->ki_nbytes);
>  put_rq:
> -	/* everything turned out well, dispose of the aiocb. */
> -	ret = __aio_put_req(ctx, iocb);
> -
>  	spin_unlock_irqrestore(&ctx->ctx_lock, flags);
>  
>  	if (waitqueue_active(&ctx->wait))
>  		wake_up(&ctx->wait);
>  
> -	if (ret)
> -		put_ioctx(ctx);
> -
> -	return ret;
> +	return aio_put_req(iocb);
>  }

New race: You are now calling wake_up(&ctx->wait) before you've called
aio_put_req(). You also still have the exact same race as before since
aio_put_req() also calls put_ioctx() if ret != 0...

Trond


  reply	other threads:[~2006-12-15 13:49 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2006-12-15  6:35 Chen, Kenneth W
2006-12-15 13:49 ` Trond Myklebust [this message]
2006-12-15 14:01   ` Trond Myklebust
2006-12-15 18:58     ` 2.6.18.4: flush_workqueue calls mutex_lock in interruptenvironment Chen, Kenneth W
     [not found] <20061214172010.a3810c9d.akpm@osdl.org>
2006-12-15  1:58 ` 2.6.18.4: flush_workqueue calls mutex_lock in interrupt environment Chen, Kenneth W
  -- strict thread matches above, loose matches on Subject: below --
2006-12-13  7:25 xb
2006-12-13  8:02 ` Arjan van de Ven
2006-12-13 12:41   ` Trond Myklebust

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1166190561.5761.12.camel@lade.trondhjem.org \
    --to=trond.myklebust@fys.uio.no \
    --cc=akpm@osdl.org \
    --cc=kenneth.w.chen@intel.com \
    --cc=linux-aio@kvack.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=xavier.bru@bull.net \
    --cc=zach.brown@oracle.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®