Re: "Dying CPU not properly vacated" splat

[Valentin Schneider <vschneid@redhat.com>]

On 25/04/22 10:33, Paul E. McKenney wrote:
> On Mon, Apr 25, 2022 at 05:15:13PM +0100, Valentin Schneider wrote:
>>
>> Hi Paul,
>>
>> On 21/04/22 12:38, Paul E. McKenney wrote:
>> > Hello!
>> >
>> > The rcutorture TREE03 scenario got the following splat, which appears
>> > to be a one-off, or if not, having an MTBF in the thousands of hours,
>> > even assuming that it is specific to TREE03.  (If it is not specific to
>> > TREE03, we are talking tens of thousands of hours of rcutorture runtime.)
>> >
>> > So just in case this rings any bells or there are some diagnostics I
>> > should add in case this ever happens again.  ;-)
>>
>> There should be a dump of the enqueued tasks right after the snippet you've
>> sent, any chance you could share that if it's there? That should tell us
>> which tasks are potentially misbehaving.
>
> And now that I know to look for them, there they are!  Thank you!!!
>
> CPU7 enqueued tasks (2 total):
>  pid: 52, name: migration/7
>  pid: 135, name: rcu_torture_rea
> smpboot: CPU 7 is now offline
>
> So what did rcu_torture_reader() do wrong here?  ;-)
>

So on teardown, CPUHP_AP_SCHED_WAIT_EMPTY->sched_cpu_wait_empty() waits for
the rq to be empty. Tasks must *not* be enqueued onto that CPU after that
step has been run - if there are per-CPU tasks bound to that CPU, they must
be unbound in their respective hotplug callback.

For instance for workqueue.c, we have workqueue_offline_cpu() as a hotplug
callback which invokes unbind_workers(cpu), the interesting bit being:

                for_each_pool_worker(worker, pool) {
                        kthread_set_per_cpu(worker->task, -1);
                        WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
                }

The rcu_torture_reader() kthreads aren't bound to any particular CPU are
they? I can't find any code that would indicate they are - and in that case
it means we have a problem with is_cpu_allowed() or related.