Stopping Kernel Threads at module unload time

Stopping Kernel Threads at module unload time

[Aritz Bastida]

Hello

I've got some questions about kernel threads.I am writing a module
which spawns some kernel threads, which would be removed when the
module unloads. For that purpose i call kthread_stop() at module
unload time. When issuing rmmod on the module, it deadlocks at that
point (in the call to kthread_stop), and never returns.

In the thread main function the code was something like this (it's of
course simplified).

thread_main()
{
     while( ! kthread_should_stop())
     {
           .............
           wait_event_interruptible(stop_wq, kthread_should_stop() );
     }

     return 0;
}

So if kthread_stop() first sets the thread "closing flag", and then
calls wake_up_process(), the thread should wake up, see he should
stop, and
end the loop. That doesnt actually never happen.

I have also tried what it is done in kernel/sched.c to finish:

	/* wait for kthread_stop */
	set_current_state(TASK_INTERRUPTIBLE);
	while (!kthread_should_stop()) {	
		schedule();
		set_current_state(TASK_INTERRUPTIBLE);
	}
	__set_current_state(TASK_RUNNING);
                return 0;

I have ensured it actually arrives to that point by using printks, but
when the thread goes to sleep, it does never wake up again, so the
call to kthread_stop() lasts forever.

I dont know why this happens. Is the module cleanup code in the
context of a user process just like system calls? Can that code sleep?
If it can't sleep then the answer would be quite easy: kthread_stop()
wakes up the processes and then waits for the threads to finish (on
call wait_for_completion), but doesnt actually let them execute,
because it cannot sleep, so it deadlocks.

So I would be grateful if anyone can help me in this matter.
Regards

Aritz

Re: Stopping Kernel Threads at module unload time

[linux-os (Dick Johnson)]

On Wed, 9 Nov 2005, Aritz Bastida wrote:

> Hello
>
> I've got some questions about kernel threads.I am writing a module
> which spawns some kernel threads, which would be removed when the
> module unloads. For that purpose i call kthread_stop() at module
> unload time. When issuing rmmod on the module, it deadlocks at that
> point (in the call to kthread_stop), and never returns.
>
> In the thread main function the code was something like this (it's of
> course simplified).
>
> thread_main()
> {
>     while( ! kthread_should_stop())
>     {
>           .............
>           wait_event_interruptible(stop_wq, kthread_should_stop() );
>     }
>
>     return 0;
> }
>
> So if kthread_stop() first sets the thread "closing flag", and then
> calls wake_up_process(), the thread should wake up, see he should
> stop, and
> end the loop. That doesnt actually never happen.
>
> I have also tried what it is done in kernel/sched.c to finish:
>
> 	/* wait for kthread_stop */
> 	set_current_state(TASK_INTERRUPTIBLE);
> 	while (!kthread_should_stop()) {
> 		schedule();
> 		set_current_state(TASK_INTERRUPTIBLE);
> 	}
> 	__set_current_state(TASK_RUNNING);
>                return 0;
>
> I have ensured it actually arrives to that point by using printks, but
> when the thread goes to sleep, it does never wake up again, so the
> call to kthread_stop() lasts forever.
>
> I dont know why this happens. Is the module cleanup code in the
> context of a user process just like system calls? Can that code sleep?
> If it can't sleep then the answer would be quite easy: kthread_stop()
> wakes up the processes and then waits for the threads to finish (on
> call wait_for_completion), but doesnt actually let them execute,
> because it cannot sleep, so it deadlocks.
>
> So I would be grateful if anyone can help me in this matter.
> Regards
>
> Aritz


Here is code that actually works. There have been some kernel
API changes that convert some of this stuff to macros. Nevertheless.
This stuff works.


void make_daemon(const char *dev, int32_t prior)
{
#ifdef OLD_DAEMON              // kernel 2.4.x
     exit_files(current);
     daemonize();
     spin_lock_irq(&current->sigmask_lock);
     sigemptyset(&current->blocked);
     sigdelset(&current->blocked, SIGTERM);
     recalc_sigpending(current);
     spin_unlock_irq(&current->sigmask_lock);
     task_lock(current);
     current->nice = prior;
     strcpy(current->comm, dev);
     task_unlock(current);
#else				// kernel 2.6.x
     daemonize("%s", dev);
     allow_signal(SIGTERM);
     set_user_nice(current, prior);
#endif
//
//  Destroy process privs just in case somebody is able to hack this
//  into a shell.
//
     task_lock(current);
     current->uid  = current->gid  = current->sgid = current->fsgid =
     current->euid = current->egid = current->suid = current->fsuid = 666;
     task_unlock(current);
}

Note above that the task is already running inside the kernel and
has access to all your data. You don't need it to be 'root' so that
possible back-door gets closed.


//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
//
//    This is the kernel thread. It handles things that should not
//    be handled in an interrupt. It is awakened from the interrupt
//    service routine.
//
static int32_t local_thread(void *unused)
{
     make_daemon(task_name, PRIOR);
     set_current_state(TASK_INTERRUPTIBLE);	// For the following sleep
     schedule_timeout(1);                        // Let insmod complete
     set_current_state(TASK_RUNNING);		// Remember to restore
     __asm__ __volatile__("thread:\n.global thread\n");
     for(;;)				// Do forever.......
     {
         set_current_state(TASK_INTERRUPTIBLE);
         if(signal_pending(current))
             complete_and_exit(&info->quit, 0);	// This is how you get out
         interruptible_sleep_on(&info->twait);	// Sleep until something to do
         set_current_state(TASK_RUNNING);

         do_stuff(.....);    // Whatever the thread needs to do....

     }
     return 0;
}


  This is how the kernel thread is started.

     info->pid = kernel_thread(local_thread, NULL, CLONE_FS|CLONE_FILES);


This is how the kernel thread is stopped.

     if(info->pid)
     {
         (void)kill_proc(info->pid, SIGTERM, 1);
         wait_for_completion(&info->quit);
     }


Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Stopping Kernel Threads at module unload time

[Aritz Bastida]

Hello,

> This is how the kernel thread is stopped.
>
>      if(info->pid)
>      {
>          (void)kill_proc(info->pid, SIGTERM, 1);
>          wait_for_completion(&info->quit);
>      }
>

I actually would prefer to do it with the new kernel thread API.
So, to create the thread:   kthread_create
bind it to a cpu:                  kthread_bind
stop it:                                kthread_stop

Now, if I call kthread_stop() in module unload time, does that code
run in user process context just like system calls do? That is
important, because if it cannot sleep, it would deadlock.

Re: Stopping Kernel Threads at module unload time

[linux-os (Dick Johnson)]

On Wed, 9 Nov 2005, Aritz Bastida wrote:

> Hello,
>
>> This is how the kernel thread is stopped.
>>
>>      if(info->pid)
>>      {
>>          (void)kill_proc(info->pid, SIGTERM, 1);
>>          wait_for_completion(&info->quit);
>>      }
>>
>
> I actually would prefer to do it with the new kernel thread API.
> So, to create the thread:   kthread_create
> bind it to a cpu:                  kthread_bind
> stop it:                                kthread_stop
>

Sure your original question was about how to properly stop
a kernel thread prior to module removal. This, I showed you.
Now you say you "prefer" to do it some other way. Good luck.
Unless your code runs exactly like the prototype that the
new macros were written for, you may have fix the macros.

> Now, if I call kthread_stop() in module unload time, does that code
> run in user process context just like system calls do? That is
> important, because if it cannot sleep, it would deadlock.
>

Not relevent. You have apparently made up your mind that you
need to do it "your" way. Fine.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Stopping Kernel Threads at module unload time

[Aritz Bastida]

>
> > Now, if I call kthread_stop() in module unload time, does that code
> > run in user process context just like system calls do? That is
> > important, because if it cannot sleep, it would deadlock.
> >
>
> Not relevent. You have apparently made up your mind that you
> need to do it "your" way. Fine.

No, no, that is not MY way. That is the 2.6 way.
 (see http://lwn.net/Articles/64444/).

With the old API (kernel_thread) there was no way (at least, direct
way) to bind a thread to a cpu, which i actually need. And this is not
MY own approach , it is used throught the kernel:

* migration thread at sched.c:
           http://lxr.linux.no/source/kernel/sched.c#L4196

* ksoftirqd thread at softirq.c:
           http://lxr.linux.no/source/kernel/softirq.c#L350

* worker thread used in workqueues:
           http://lxr.linux.no/source/kernel/workqueue.c#L182

...and so on

All these threads are standard and well debugged, and all use the API
which I was talking about. Of course, as they are included in the
official kernel sources, none of them is stopped at module unload
time.

Thus, all I want to know is why kthread_stop() cant be called at
module unload time, if the cleanup code can sleep and if there is a
workaround for that.

I hope my questions are clearer now.
Thanks for your help

Aritz

Re: Stopping Kernel Threads at module unload time

[linux-os (Dick Johnson)]

On Wed, 9 Nov 2005, Aritz Bastida wrote:

>>
>>> Now, if I call kthread_stop() in module unload time, does that code
>>> run in user process context just like system calls do? That is
>>> important, because if it cannot sleep, it would deadlock.
>>>
>>
>> Not relevent. You have apparently made up your mind that you
>> need to do it "your" way. Fine.
>
> No, no, that is not MY way. That is the 2.6 way.
> (see http://lwn.net/Articles/64444/).
>
> With the old API (kernel_thread) there was no way (at least, direct
> way) to bind a thread to a cpu, which i actually need. And this is not
> MY own approach , it is used throught the kernel:
>
> * migration thread at sched.c:
>           http://lxr.linux.no/source/kernel/sched.c#L4196
>
> * ksoftirqd thread at softirq.c:
>           http://lxr.linux.no/source/kernel/softirq.c#L350
>
> * worker thread used in workqueues:
>           http://lxr.linux.no/source/kernel/workqueue.c#L182
>
> ...and so on
>
> All these threads are standard and well debugged, and all use the API
> which I was talking about. Of course, as they are included in the
> official kernel sources, none of them is stopped at module unload
> time.
>
> Thus, all I want to know is why kthread_stop() cant be called at
> module unload time, if the cleanup code can sleep and if there is a
> workaround for that.
>
> I hope my questions are clearer now.
> Thanks for your help

Then if you are going to use the macros to bind to a certain CPU
when you start up a kernel thread, then you need to use the
complimenting macros to run them down.

You need to use the for_each_cpu() stuff to pick up each thread.
Then you may actually get to talk to the right process.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Stopping Kernel Threads at module unload time

[Arnd Bergmann]

On Middeweken 09 November 2005 17:56, Aritz Bastida wrote:
> Now, if I call kthread_stop() in module unload time, does that code
> run in user process context just like system calls do? That is
> important, because if it cannot sleep, it would deadlock.

Yes, it runs in the context of the delete_module system call.
I think it's more likely that you're not returning from your
thread loop.

Please post a URL for your module source code so we can see
what goes wrong there.

	Arnd <><

Re: Stopping Kernel Threads at module unload time

[Aritz Bastida]

2005/11/9, Arnd Bergmann <arnd@arndb.de>:
> On Middeweken 09 November 2005 17:56, Aritz Bastida wrote:
> > Now, if I call kthread_stop() in module unload time, does that code
> > run in user process context just like system calls do? That is
> > important, because if it cannot sleep, it would deadlock.
>
> Yes, it runs in the context of the delete_module system call.
> I think it's more likely that you're not returning from your
> thread loop.
>
> Please post a URL for your module source code so we can see
> what goes wrong there.
>
>         Arnd <><

Than you very much Arnd!
You solved my problem. hehe

I began to write a test module for showing you this and have just
realized about the problem. As I create as many threads as CPUs, I
have to delete them all when finishing.

I killed them like this:

        /* We don't need the distraction of CPUs appearing and vanishing. */
        lock_cpu_hotplug();
        for_each_online_cpu(cpu) {
                p = per_cpu(ksensord_info, cpu);
                kthread_stop(p);
        }
        unlock_cpu_hotplug();

I locked the cpu hotplug lock to protect the for_each_online_cpu()
code in case a cpu appears/vanishes, so I am actually calling
kthread_stop() in an atomic context, so it wakes up the process, but
dont let it run!

This is quite a subtle error, but of course it's my complete fault :P
May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of
errors be acknowledged more easily.

Thank you for your help
Regards

Aritz Bastida

Re: Stopping Kernel Threads at module unload time

[Aritz Bastida]

Thank you very much Arnd!
You solved my problem. hehe

I began to write a test module for showing you this and have just
realized about the problem. As I create as many threads as CPUs, I
have to delete them all when finishing.

I killed them like this:

        /* We don't need the distraction of CPUs appearing and vanishing. */
        lock_cpu_hotplug();
        for_each_online_cpu(cpu) {
                p = per_cpu(ksensord_info, cpu);
                kthread_stop(p);
        }
        unlock_cpu_hotplug();

I locked the cpu hotplug lock to protect the for_each_online_cpu()
code in case a cpu appears/vanishes, so I am actually calling
kthread_stop() in an atomic context, so it wakes up the process, but
dont let it run!

This is quite a subtle error, but of course it's my complete fault :P
May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of
errors be acknowledged more easily.

Thank you for your help
Regards

Aritz Bastida

PS:  Sorry Arnd, I forgot adding the linux-kernel address to the message

Re: Stopping Kernel Threads at module unload time

[linux-os (Dick Johnson)]

On Wed, 9 Nov 2005, Aritz Bastida wrote:

> 2005/11/9, Arnd Bergmann <arnd@arndb.de>:
>> On Middeweken 09 November 2005 17:56, Aritz Bastida wrote:
>>> Now, if I call kthread_stop() in module unload time, does that code
>>> run in user process context just like system calls do? That is
>>> important, because if it cannot sleep, it would deadlock.
>>
>> Yes, it runs in the context of the delete_module system call.
>> I think it's more likely that you're not returning from your
>> thread loop.
>>
>> Please post a URL for your module source code so we can see
>> what goes wrong there.
>>
>>         Arnd <><
>
> Than you very much Arnd!
> You solved my problem. hehe
>
> I began to write a test module for showing you this and have just
> realized about the problem. As I create as many threads as CPUs, I
> have to delete them all when finishing.
>
> I killed them like this:
>
>        /* We don't need the distraction of CPUs appearing and vanishing. */
>        lock_cpu_hotplug();
>        for_each_online_cpu(cpu) {
>                p = per_cpu(ksensord_info, cpu);
>                kthread_stop(p);
>        }
>        unlock_cpu_hotplug();
>
> I locked the cpu hotplug lock to protect the for_each_online_cpu()
> code in case a cpu appears/vanishes, so I am actually calling
> kthread_stop() in an atomic context, so it wakes up the process, but
> dont let it run!
>
> This is quite a subtle error, but of course it's my complete fault :P
> May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of
> errors be acknowledged more easily.
>
> Thank you for your help
> Regards
>
> Aritz Bastida


Ah, yes.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.