Re: [PATCH] locking/rtmutex: Always use trylock in rt_mutex_trylock()

[Waiman Long <llong@redhat.com>]

On 10/7/24 10:50 AM, Peter Zijlstra wrote:
> On Wed, Oct 02, 2024 at 01:54:16PM -0400, Waiman Long wrote:
>> On 10/2/24 05:37, Peter Zijlstra wrote:
>>> On Thu, Sep 26, 2024 at 11:13:15AM -0400, Waiman Long wrote:
>>>> One reason to use a trylock is to avoid a ABBA deadlock in case we need
>>>> to use a locking sequence that is not in the expected locking order. That
>>>> requires the use of trylock all the ways in the abnormal locking
>>>> sequence. Unfortunately, this is not the case for rt_mutex_trylock() as
>>>> it uses a raw_spin_lock_irqsave() to acquire the lock->wait_lock. That
>>>> will cause a lockdep splat like the following in a PREEMPT_RT kernel:
>>>>
>>>> [   63.695668] -> #0 (&lock->wait_lock){-...}-{2:2}:
>>>> [   63.695674]        check_prev_add+0x1bd/0x1310
>>>> [   63.695678]        validate_chain+0x6cf/0x7c0
>>>> [   63.695682]        __lock_acquire+0x879/0xf40
>>>> [   63.695686]        lock_acquire.part.0+0xfa/0x2d0
>>>> [   63.695690]        _raw_spin_lock_irqsave+0x46/0x90
>>>> [   63.695695]        rt_mutex_slowtrylock+0x3f/0xb0
>>>> [   63.695699]        rt_spin_trylock+0x13/0xc0
>>>> [   63.695702]        rmqueue_pcplist+0x5b/0x180
>>>> [   63.695705]        rmqueue+0xb01/0x1040
>>>> [   63.695708]        get_page_from_freelist+0x1d0/0xa00
>>>> [   63.695712]        __alloc_pages_noprof+0x19a/0x450
>>>> [   63.695716]        alloc_pages_mpol_noprof+0xaf/0x1e0
>>>> [   63.695721]        stack_depot_save_flags+0x4db/0x520
>>>> [   63.695727]        kasan_save_stack+0x3f/0x50
>>>> [   63.695731]        __kasan_record_aux_stack+0x8e/0xa0
>>>> [   63.695736]        task_work_add+0x1ad/0x240
>>>> [   63.695741]        sched_tick+0x1c7/0x500
>>>> [   63.695744]        update_process_times+0xf1/0x130
>>>> [   63.695750]        tick_nohz_handler+0xf7/0x230
>>>> [   63.695754]        __hrtimer_run_queues+0x13b/0x7b0
>>>> [   63.695758]        hrtimer_interrupt+0x1c2/0x350
>>>> [   63.695763]        __sysvec_apic_timer_interrupt+0xdb/0x340
>>>> [   63.695770]        sysvec_apic_timer_interrupt+0x9c/0xd0
>>>> [   63.695774]        asm_sysvec_apic_timer_interrupt+0x1a/0x20
>>>> [   63.695780]        __asan_load8+0x8/0xa0
>>>> [   63.695784]        mas_wr_end_piv+0x28/0x2c0
>>>> [   63.695789]        mas_preallocate+0x3a8/0x680
>>>> [   63.695793]        vma_shrink+0x180/0x3f0
>>>> [   63.695799]        shift_arg_pages+0x219/0x2c0
>>>> [   63.695804]        setup_arg_pages+0x343/0x5e0
>>>> [   63.695807]        load_elf_binary+0x5ac/0x15d0
>>>> [   63.695813]        search_binary_handler+0x125/0x370
>>>> [   63.695818]        exec_binprm+0xc9/0x3d0
>>>> [   63.695821]        bprm_execve+0x103/0x230
>>>> [   63.695824]        kernel_execve+0x187/0x1c0
>>>> [   63.695828]        call_usermodehelper_exec_async+0x145/0x1e0
>>>> [   63.695832]        ret_from_fork+0x31/0x60
>>>> [   63.695836]        ret_from_fork_asm+0x1a/0x30
>>>> [   63.695840]
>>>> [   63.695840] other info that might help us debug this:
>>>> [   63.695840]
>>>> [   63.695842] Chain exists of:
>>>> [   63.695842]   &lock->wait_lock --> &p->pi_lock --> &rq->__lock
>>>> [   63.695842]
>>>> [   63.695850]  Possible unsafe locking scenario:
>>>> [   63.695850]
>>>> [   63.695851]        CPU0                    CPU1
>>>> [   63.695852]        ----                    ----
>>>> [   63.695854]   lock(&rq->__lock);
>>>> [   63.695857]                                lock(&p->pi_lock);
>>>> [   63.695861]                                lock(&rq->__lock);
>>>> [   63.695864]   lock(&lock->wait_lock);
>>>> [   63.695868]
>>>> [   63.695868]  *** DEADLOCK ***
>>>>
>>>> Fix it by using raw_spin_trylock_irqsave() instead.
>>> That truncated lockdep report doesn't really explain anything. Please
>>> just transcribe the full lockdep report into something small and
>>> coherent.
>> I was trying to show where the offending call is coming from. I will send a
>> v2 with a condensed lockdep splat.
> No no no... explain the actual problem.
>
> Is the problem that:
>
> 	sched_tick()
> 	  task_tick_mm_cid()
> 	    task_work_add()
> 	      kasan_save_stack()
> 	        idiotic crap while holding rq->__lock ?
>
> Because afaict that is completely insane. And has nothing to do with
> rtmutex.
>
> We are not going to change rtmutex because instrumentation shit is shit.

Yes, it is because of KASAN that causes page allocation while holding 
the rq->__lock. Maybe we can blame KASAN for this. It is actually not a 
problem for non-PREEMPT_RT kernel because only trylock is being used. 
However, we don't use trylock all the way when rt_spin_trylock() is 
being used with PREEMPT_RT Kernel. This is certainly a problem that we 
need to fix as there may be other similar case not involving rq->__lock 
lurking somewhere.

Cheers,
Longman

>