Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

["Chen, Yu C" <yu.c.chen@intel.com>]

On 7/2/2025 5:05 AM, Jirka Hladky wrote:
> Hi Chenyu,
> 
> I have the first
> 
> if (unlikely(!p->mm)) {
> trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm, p->flags);
> 
> log from dual socket AMD Milanx 7573X server:
> 
> [74615.841401] # entries-in-buffer/entries-written: 1/1   #P:128
> [74615.841401] #
> [74615.841401] #                                _-----=> irqs-off/BH-disabled
> [74615.841401] #                               / _----=> need-resched
> [74615.841401] #                              | / _---=> hardirq/softirq
> [74615.841401] #                              || / _--=> preempt-depth
> [74615.841401] #                              ||| / _-=> migrate-disable
> [74615.841401] #                              |||| /     delay
> [74615.841401] #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
> [74615.841401] #              | |         |   |||||     |         |
> [74615.841401]      migration/6-54      [006] d..5. 58590.368364:
> __migrate_swap_task: !! (2939199 stress-ng-sem) flags=40004c
> 
> debug code caught a task with p->mm == NULL during __migrate_swap_task().
> 
> Trace shows:
>    * It was a stress-ng-sem process (PID 2939199),
>    * That process was in the process of exiting (PF_EXITING | PF_EXITPIDONE),

Thanks Jirka. So this race condition is related to task exit.

>    * And was a workqueue worker (PF_WQ_WORKER),
>    * It explains why p->mm had already been freed
> 
> I'm curious as to why the command is stress-ng-sem, and the flag
> indicates it is a workqueue worker.

The flag 0x40004c is
PF_EXITING PF_POSTCOREDUMP PF_FORKNOEXEC PF_RANDOMIZE
according to latest
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/sched.h?h=v6.16-rc4

thanks,
Chenyu>
> More jobs are still running. I will keep an eye on them and report the
> other cases as they finish.
> 
> Jirka
> 
> On Tue, Jul 1, 2025 at 5:10 PM Jirka Hladky <jhladky@redhat.com> wrote:
>>
>> Hi Chenyu and Libo,
>>
>> I agree, let's wait for the results. Hopefully, we can fix the root
>> cause based on the debug messages.
>>
>> There was a problem with the infrastructure over the weekend, so I had
>> to restart the jobs yesterday. I should have results tomorrow.
>>
>> Stay tuned.
>> Jirka
>>
>>
>> On Tue, Jul 1, 2025 at 4:59 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
>>>
>>> Hi Libo,
>>>
>>> On 7/1/2025 3:32 PM, Libo Chen wrote:
>>>> Hi Chenyu,
>>>>
>>>> On 6/27/25 00:33, Chen, Yu C wrote:
>>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
>>>>>> Hi Jirka,
>>>>>>
>>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
>>>>>>> Hi Chen and all,
>>>>>>>
>>>>>>> we have now verified that the following commit causes a kernel panic
>>>>>>> discussed in this thread:
>>>>>>>
>>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
>>>>>>>
>>>>>>> Reverting this commit fixes the issue.
>>>>>>>
>>>>>>> I'm happy to help debug this further or test a proposed fix.
>>>>>>>
>>>>>>
>>>>>> Thanks very much for your report, it seems that there is a
>>>>>> race condition that when the swap task candidate was chosen,
>>>>>> but its mm_struct get released due to task exit, then later
>>>>>> when doing the task swaping, the p->mm is NULL which caused
>>>>>> the problem:
>>>>>>
>>>>>> CPU0                                   CPU1
>>>>>> :
>>>>>> ...
>>>>>> task_numa_migrate
>>>>>>      task_numa_find_cpu
>>>>>>       task_numa_compare
>>>>>>         # a normal task p is chosen
>>>>>>         env->best_task = p
>>>>>>
>>>>>>                                           # p exit:
>>>>>>                                           exit_signals(p);
>>>>>>                                              p->flags |= PF_EXITING
>>>>>>                                           exit_mm
>>>>>>                                              p->mm = NULL;
>>>>>>
>>>>>>       migrate_swap_stop
>>>>>>         __migrate_swap_task((arg->src_task, arg->dst_cpu)
>>>>>>          count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
>>>>>>
>>>>>> Could you please help check if the following debug patch works,
>>>>>
>>>>> Attached the patch:
>>>>>
>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>> index 8988d38d46a3..82fc966b390c 100644
>>>>> --- a/kernel/sched/core.c
>>>>> +++ b/kernel/sched/core.c
>>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
>>>>>    {
>>>>>        __schedstat_inc(p->stats.numa_task_swapped);
>>>>>        count_vm_numa_event(NUMA_TASK_SWAP);
>>>>> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>>>> +    if (unlikely(!p->mm)) {
>>>>
>>>> I am starting to wonder if we should keep this check and add a big fat warning
>>>> like  "there is a bug here, please report it!" rather than brick the kernel.
>>>> A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
>>>> so it's a tradeoff I guess. What do you think?
>>>>
>>>
>>> I agree with you that adding something like WARN_ON_ONCE()
>>> could help, and it seems that the kernel panic is not expected
>>> in this scenario because this feature is a statistic calculation
>>> rather than the critical logic. Maybe waiting for Jirka's feedback
>>> to decide the next step.
>>>
>>> thanks,
>>> Chenyu
>>>
>>>
>>>
>>>>
>>>> Thanks,
>>>> Libo
>>>>
>>>>> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
>>>>> +                p->flags);
>>>>> +    } else {
>>>>> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
>>>>> +    }
>>>>>
>>>>>        if (task_on_rq_queued(p)) {
>>>>>            struct rq *src_rq, *dst_rq;
>>>>
>>>
>>
>>
>> --
>> -Jirka
> 
> 
>