mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH] target: fix deadlock on unload
@ 2014-06-23 17:42 Mikulas Patocka
  2014-06-27  4:18 ` Nicholas A. Bellinger
  0 siblings, 1 reply; 4+ messages in thread
From: Mikulas Patocka @ 2014-06-23 17:42 UTC (permalink / raw)
  To: Nicholas A. Bellinger; +Cc: linux-scsi, target-devel, linux-kernel

target: fix deadlock on unload

On uniprocessor preemptible kernel, target core deadlocks on unload. The
following events happen:
* iscsit_del_np is called
* it calls send_sig(SIGINT, np->np_thread, 1);
* the scheduler switches to the np_thread
* the np_thread is woken up, it sees that kthread_should_stop() returns
  false, so it doesn't terminate
* the np_thread clears signals with flush_signals(current); and goes back
  to sleep in iscsit_accept_np
* the scheduler switches back to iscsit_del_np
* iscsit_del_np calls kthread_stop(np->np_thread);
* the np_thread is waiting in iscsit_accept_np and it doesn't respond to
  kthread_stop

The deadlock could be resolved if the administrator sends SIGINT signal to
the np_thread with killall -INT iscsi_np

The reproducible deadlock was introduced in commit
db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
racy even before.

This patch fixes the problem. Using kthread_should_stop to stop the
np_thread is unreliable, so we test np_thread_state instead. If
np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Cc: stable@vger.kernel.org

---
 drivers/target/iscsi/iscsi_target_login.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

Index: linux-3.14.8/drivers/target/iscsi/iscsi_target_login.c
===================================================================
--- linux-3.14.8.orig/drivers/target/iscsi/iscsi_target_login.c	2014-06-19 00:10:44.634113370 +0200
+++ linux-3.14.8/drivers/target/iscsi/iscsi_target_login.c	2014-06-23 19:05:44.887437402 +0200
@@ -1196,7 +1196,7 @@ old_sess_out:
 static int __iscsi_target_login_thread(struct iscsi_np *np)
 {
 	u8 *buffer, zero_tsih = 0;
-	int ret = 0, rc, stop;
+	int ret = 0, rc;
 	struct iscsi_conn *conn = NULL;
 	struct iscsi_login *login;
 	struct iscsi_portal_group *tpg = NULL;
@@ -1210,6 +1210,9 @@ static int __iscsi_target_login_thread(s
 	if (np->np_thread_state == ISCSI_NP_THREAD_RESET) {
 		np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
 		complete(&np->np_restart_comp);
+	} else if (np->np_thread_state == ISCSI_NP_THREAD_SHUTDOWN) {
+		spin_unlock_bh(&np->np_thread_lock);
+		goto exit;
 	} else {
 		np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
 	}
@@ -1402,10 +1405,8 @@ old_sess_out:
 	}
 
 out:
-	stop = kthread_should_stop();
-	/* Wait for another socket.. */
-	if (!stop)
-		return 1;
+	return 1;
+
 exit:
 	iscsi_stop_login_thread_timer(np);
 	spin_lock_bh(&np->np_thread_lock);
@@ -1422,7 +1423,7 @@ int iscsi_target_login_thread(void *arg)
 
 	allow_signal(SIGINT);
 
-	while (!kthread_should_stop()) {
+	while (1) {
 		ret = __iscsi_target_login_thread(np);
 		/*
 		 * We break and exit here unless another sock_accept() call

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] target: fix deadlock on unload
  2014-06-23 17:42 [PATCH] target: fix deadlock on unload Mikulas Patocka
@ 2014-06-27  4:18 ` Nicholas A. Bellinger
  2014-06-27 12:35   ` Mikulas Patocka
  0 siblings, 1 reply; 4+ messages in thread
From: Nicholas A. Bellinger @ 2014-06-27  4:18 UTC (permalink / raw)
  To: Mikulas Patocka; +Cc: linux-scsi, target-devel, linux-kernel

Hi Mikulas,

On Mon, 2014-06-23 at 13:42 -0400, Mikulas Patocka wrote:
> target: fix deadlock on unload
> 
> On uniprocessor preemptible kernel, target core deadlocks on unload. The
> following events happen:
> * iscsit_del_np is called
> * it calls send_sig(SIGINT, np->np_thread, 1);
> * the scheduler switches to the np_thread
> * the np_thread is woken up, it sees that kthread_should_stop() returns
>   false, so it doesn't terminate
> * the np_thread clears signals with flush_signals(current); and goes back
>   to sleep in iscsit_accept_np
> * the scheduler switches back to iscsit_del_np
> * iscsit_del_np calls kthread_stop(np->np_thread);
> * the np_thread is waiting in iscsit_accept_np and it doesn't respond to
>   kthread_stop
> 
> The deadlock could be resolved if the administrator sends SIGINT signal to
> the np_thread with killall -INT iscsi_np
> 
> The reproducible deadlock was introduced in commit
> db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
> racy even before.
> 
> This patch fixes the problem. Using kthread_should_stop to stop the
> np_thread is unreliable, so we test np_thread_state instead. If
> np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.
> 
> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> Cc: stable@vger.kernel.org
> 

Apologies for the delayed response..

Applied to target-pending/master and including in the next -rc3 PULL
request.

Also FYI, I've added '3.12+' to the stable tag to match how far back
commit db6077fd0 has been included in stable.

Thanks,

--nab


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] target: fix deadlock on unload
  2014-06-27  4:18 ` Nicholas A. Bellinger
@ 2014-06-27 12:35   ` Mikulas Patocka
  2014-06-28  6:51     ` Nicholas A. Bellinger
  0 siblings, 1 reply; 4+ messages in thread
From: Mikulas Patocka @ 2014-06-27 12:35 UTC (permalink / raw)
  To: Nicholas A. Bellinger; +Cc: linux-scsi, target-devel, linux-kernel



On Thu, 26 Jun 2014, Nicholas A. Bellinger wrote:

> Hi Mikulas,
> 
> On Mon, 2014-06-23 at 13:42 -0400, Mikulas Patocka wrote:
> > target: fix deadlock on unload
> > 
> > On uniprocessor preemptible kernel, target core deadlocks on unload. The
> > following events happen:
> > * iscsit_del_np is called
> > * it calls send_sig(SIGINT, np->np_thread, 1);
> > * the scheduler switches to the np_thread
> > * the np_thread is woken up, it sees that kthread_should_stop() returns
> >   false, so it doesn't terminate
> > * the np_thread clears signals with flush_signals(current); and goes back
> >   to sleep in iscsit_accept_np
> > * the scheduler switches back to iscsit_del_np
> > * iscsit_del_np calls kthread_stop(np->np_thread);
> > * the np_thread is waiting in iscsit_accept_np and it doesn't respond to
> >   kthread_stop
> > 
> > The deadlock could be resolved if the administrator sends SIGINT signal to
> > the np_thread with killall -INT iscsi_np
> > 
> > The reproducible deadlock was introduced in commit
> > db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
> > racy even before.
> > 
> > This patch fixes the problem. Using kthread_should_stop to stop the
> > np_thread is unreliable, so we test np_thread_state instead. If
> > np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.
> > 
> > Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> > Cc: stable@vger.kernel.org
> > 
> 
> Apologies for the delayed response..
> 
> Applied to target-pending/master and including in the next -rc3 PULL
> request.
> 
> Also FYI, I've added '3.12+' to the stable tag to match how far back
> commit db6077fd0 has been included in stable.
> 
> Thanks,
> 
> --nab

Hi

I think db6077fd0 should be backported to stable kernels beginning with 
3.10 (because they set np->np_thread = NULL in 
__iscsi_target_login_thread). The current 3.10-stable branch misses this 
patch.


This patch for unload deadlock should be backported to all stable kernels 
(because unload is racy there), but because of different code, we should 
make a different patch for old stable branches.

For example in 3.4.95, __iscsi_target_login_thread contains this code:
        spin_lock_bh(&np->np_thread_lock);
        if (np->np_thread_state == ISCSI_NP_THREAD_RESET) {
                np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
                complete(&np->np_restart_comp);
        } else {
                np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
        }
        spin_unlock_bh(&np->np_thread_lock);
If the state is ISCSI_NP_THREAD_SHUTDOWN, the above piece of code will 
change it to ISCSI_NP_THREAD_ACTIVE and open the same kthread_should_stop 
race.

Mikulas

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] target: fix deadlock on unload
  2014-06-27 12:35   ` Mikulas Patocka
@ 2014-06-28  6:51     ` Nicholas A. Bellinger
  0 siblings, 0 replies; 4+ messages in thread
From: Nicholas A. Bellinger @ 2014-06-28  6:51 UTC (permalink / raw)
  To: Mikulas Patocka; +Cc: linux-scsi, target-devel, linux-kernel

On Fri, 2014-06-27 at 08:35 -0400, Mikulas Patocka wrote:
> 
> On Thu, 26 Jun 2014, Nicholas A. Bellinger wrote:
> 
> > Hi Mikulas,
> > 
> > On Mon, 2014-06-23 at 13:42 -0400, Mikulas Patocka wrote:
> > > target: fix deadlock on unload
> > > 
> > > On uniprocessor preemptible kernel, target core deadlocks on unload. The
> > > following events happen:
> > > * iscsit_del_np is called
> > > * it calls send_sig(SIGINT, np->np_thread, 1);
> > > * the scheduler switches to the np_thread
> > > * the np_thread is woken up, it sees that kthread_should_stop() returns
> > >   false, so it doesn't terminate
> > > * the np_thread clears signals with flush_signals(current); and goes back
> > >   to sleep in iscsit_accept_np
> > > * the scheduler switches back to iscsit_del_np
> > > * iscsit_del_np calls kthread_stop(np->np_thread);
> > > * the np_thread is waiting in iscsit_accept_np and it doesn't respond to
> > >   kthread_stop
> > > 
> > > The deadlock could be resolved if the administrator sends SIGINT signal to
> > > the np_thread with killall -INT iscsi_np
> > > 
> > > The reproducible deadlock was introduced in commit
> > > db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
> > > racy even before.
> > > 
> > > This patch fixes the problem. Using kthread_should_stop to stop the
> > > np_thread is unreliable, so we test np_thread_state instead. If
> > > np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.
> > > 
> > > Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> > > Cc: stable@vger.kernel.org
> > > 
> > 
> > Apologies for the delayed response..
> > 
> > Applied to target-pending/master and including in the next -rc3 PULL
> > request.
> > 
> > Also FYI, I've added '3.12+' to the stable tag to match how far back
> > commit db6077fd0 has been included in stable.
> > 
> > Thanks,
> > 
> > --nab
> 
> Hi
> 
> I think db6077fd0 should be backported to stable kernels beginning with 
> 3.10 (because they set np->np_thread = NULL in 
> __iscsi_target_login_thread). The current 3.10-stable branch misses this 
> patch.
> 
> 
> This patch for unload deadlock should be backported to all stable kernels 
> (because unload is racy there), but because of different code, we should 
> make a different patch for old stable branches.
> 
> For example in 3.4.95, __iscsi_target_login_thread contains this code:
>         spin_lock_bh(&np->np_thread_lock);
>         if (np->np_thread_state == ISCSI_NP_THREAD_RESET) {
>                 np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
>                 complete(&np->np_restart_comp);
>         } else {
>                 np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
>         }
>         spin_unlock_bh(&np->np_thread_lock);
> If the state is ISCSI_NP_THREAD_SHUTDOWN, the above piece of code will 
> change it to ISCSI_NP_THREAD_ACTIVE and open the same kthread_should_stop 
> race.
> 

Ok, dropping the 'v3.12+' stable tag from this patch, and will include
it in a PATCH-v3.10.y series together with db6077fd0 once Greg-KH
attempts to queue it up.

As for v3.4.x, care to send along a separate patch with your Tested-by +
Signed-off-by..?

Thanks for the extra stable info!

--nab


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2014-06-28  6:51 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-06-23 17:42 [PATCH] target: fix deadlock on unload Mikulas Patocka
2014-06-27  4:18 ` Nicholas A. Bellinger
2014-06-27 12:35   ` Mikulas Patocka
2014-06-28  6:51     ` Nicholas A. Bellinger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®