Re: [PATCH 1/3] sched: add sched_task_call()

[Josh Poimboeuf <jpoimboe@redhat.com>]

On Mon, Feb 16, 2015 at 09:44:36PM +0100, Peter Zijlstra wrote:
> On Mon, Feb 16, 2015 at 12:52:34PM -0600, Josh Poimboeuf wrote:
> > +++ b/kernel/sched/core.c
> > @@ -1338,6 +1338,23 @@ void kick_process(struct task_struct *p)
> >  EXPORT_SYMBOL_GPL(kick_process);
> >  #endif /* CONFIG_SMP */
> >  
> > +/***
> > + * sched_task_call - call a function with a task's state locked
> > + *
> > + * The task is guaranteed to remain either active or inactive during the
> > + * function call.
> > + */
> > +void sched_task_call(sched_task_call_func_t func, struct task_struct *p,
> > +		     void *data)
> > +{
> > +	unsigned long flags;
> > +	struct rq *rq;
> > +
> > +	rq = task_rq_lock(p, &flags);
> > +	func(p, data);
> > +	task_rq_unlock(rq, p, &flags);
> > +}
> 
> Yeah, I think not. We're so not going to allow running random code under
> rq->lock and p->pi_lock.

Yeah, I can understand that.  I definitely want to avoid touching the
scheduler code.  Basically I'm trying to find a way to atomically do the
following:

if (task is sleeping) {
	walk the stack
	if (certain set of functions isn't on the stack)
		set (or clear) a thread flag for the task
}

Any ideas on how I can achieve that?  So far my ideas are:

1. Use task_rq_lock() -- but rq_lock is internal to sched code.

2. Use wait_task_inactive() -- I could call it twice, with the stack
   checking in between, and use ncsw to ensure that it didn't reschedule
   in the mean time.  But this still seems racy.  i.e., I think the task
   could start running after the second call to wait_task_inactive()
   returns but before setting the thread flag.  Not sure if that's a
   realistic race condition or not.

3. Use set_cpus_allowed() to temporarily pin the task to its current
   CPU, and then call smp_call_function_single() to run the above
   critical section on that CPU.  I'm not sure if there's a race-free
   way to do it but it's a lot more disruptive than I'd like...

Any ideas or guidance would be greatly appreciated!

-- 
Josh