Re: [PATCH] sched/deadline: fix inactive_task_timer splat with CONFIG_PREEMPT_RT

[Valentin Schneider <vschneid@redhat.com>]

On 10/01/23 14:27, Paul E. McKenney wrote:
> On Tue, Jan 10, 2023 at 05:52:03PM -0300, Wander Lairson Costa wrote:
>> On Mon, Jan 9, 2023 at 10:40 PM Paul E. McKenney <paulmck@kernel.org> wrote:
>> >
>> > On Wed, Jan 04, 2023 at 03:17:01PM -0300, Wander Lairson Costa wrote:
>> > > inactive_task_timer() executes in interrupt (atomic) context. It calls
>> > > put_task_struct(), which indirectly acquires sleeping locks under
>> > > PREEMPT_RT.
>> > >
>> > > Below is an example of a splat that happened in a test environment:
>> > >
>> > >  CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W ---------
>> > >  Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012
>> > >  Call Trace:
>> > >  dump_stack_lvl+0x57/0x7d
>> > >  mark_lock_irq.cold+0x33/0xba
>> > >  ? stack_trace_save+0x4b/0x70
>> > >  ? save_trace+0x55/0x150
>> > >  mark_lock+0x1e7/0x400
>> > >  mark_usage+0x11d/0x140
>> > >  __lock_acquire+0x30d/0x930
>> > >  lock_acquire.part.0+0x9c/0x210
>> > >  ? refill_obj_stock+0x3d/0x3a0
>> > >  ? rcu_read_lock_sched_held+0x3f/0x70
>> > >  ? trace_lock_acquire+0x38/0x140
>> > >  ? lock_acquire+0x30/0x80
>> > >  ? refill_obj_stock+0x3d/0x3a0
>> > >  rt_spin_lock+0x27/0xe0
>> > >  ? refill_obj_stock+0x3d/0x3a0
>> > >  refill_obj_stock+0x3d/0x3a0
>> > >  ? inactive_task_timer+0x1ad/0x340
>> > >  kmem_cache_free+0x357/0x560
>> > >  inactive_task_timer+0x1ad/0x340
>> > >  ? switched_from_dl+0x2d0/0x2d0
>> > >  __run_hrtimer+0x8a/0x1a0
>> > >  __hrtimer_run_queues+0x91/0x130
>> > >  hrtimer_interrupt+0x10f/0x220
>> > >  __sysvec_apic_timer_interrupt+0x7b/0xd0
>> > >  sysvec_apic_timer_interrupt+0x4f/0xd0
>> > >  ? asm_sysvec_apic_timer_interrupt+0xa/0x20
>> > >  asm_sysvec_apic_timer_interrupt+0x12/0x20
>> > >  RIP: 0033:0x7fff196bf6f5
>> > >
>> > > Instead of calling put_task_struct() directly, we defer it using
>> > > call_rcu(). A more natural approach would use a workqueue, but since
>> > > in PREEMPT_RT, we can't allocate dynamic memory from atomic context,
>> > > the code would become more complex because we would need to put the
>> > > work_struct instance in the task_struct and initialize it when we
>> > > allocate a new task_struct.
>> > >
>> > > Signed-off-by: Wander Lairson Costa <wander@redhat.com>
>> > > Cc: Paul McKenney <paulmck@kernel.org>
>> > > Cc: Thomas Gleixner <tglx@linutronix.de>
>> > > ---
>> > >  kernel/sched/build_policy.c |  1 +
>> > >  kernel/sched/deadline.c     | 24 +++++++++++++++++++++++-
>> > >  2 files changed, 24 insertions(+), 1 deletion(-)
>> > >
>> > > diff --git a/kernel/sched/build_policy.c b/kernel/sched/build_policy.c
>> > > index d9dc9ab3773f..f159304ee792 100644
>> > > --- a/kernel/sched/build_policy.c
>> > > +++ b/kernel/sched/build_policy.c
>> > > @@ -28,6 +28,7 @@
>> > >  #include <linux/suspend.h>
>> > >  #include <linux/tsacct_kern.h>
>> > >  #include <linux/vtime.h>
>> > > +#include <linux/rcupdate.h>
>> > >
>> > >  #include <uapi/linux/sched/types.h>
>> > >
>> > > diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
>> > > index 9ae8f41e3372..ab9301d4cc24 100644
>> > > --- a/kernel/sched/deadline.c
>> > > +++ b/kernel/sched/deadline.c
>> > > @@ -1405,6 +1405,13 @@ static void update_curr_dl(struct rq *rq)
>> > >       }
>> > >  }
>> > >
>> > > +static void delayed_put_task_struct(struct rcu_head *rhp)
>> > > +{
>> > > +     struct task_struct *task = container_of(rhp, struct task_struct, rcu);
>> > > +
>> > > +     __put_task_struct(task);
>> >
>> > Please note that BH is disabled here.  Don't you therefore
>> > need to schedule a workqueue handler?  Perhaps directly from
>> > inactive_task_timer(), or maybe from this point.  If the latter, one
>> > way to skip the extra step is to use queue_rcu_work().
>> >
>>
>> My initial work was using a workqueue [1,2]. However, I realized I
>> could reach a much simpler code with call_rcu().
>> I am afraid my ignorance doesn't allow me to get your point. Does
>> disabling softirq imply atomic context?
>
> Given that this problem occurred in PREEMPT_RT, I am assuming that the
> appropriate definition of "atomic context" is "cannot call schedule()".
> And you are in fact not permitted to call schedule() from a bh-disabled
> region.
>
> This also means that you cannot acquire a non-raw spinlock in a
> bh-disabled region of code in a PREEMPT_RT kernel, because doing
> so can invoke schedule.
>

But per the PREEMPT_RT lock "replacement", non-raw spinlocks end up
invoking schedule_rtlock(), which should be safe vs BH disabled
(local_lock() + rcu_read_lock()):

  6991436c2b5d ("sched/core: Provide a scheduling point for RT locks")

Unless I'm missing something else?