Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()

[Moshe Shemesh <moshe@nvidia.com>]

On 11/3/2025 12:45 PM, Usama Arif wrote:
> On 15/10/2025 12:13, Moshe Shemesh wrote:
>>
>>
>> On 10/9/2025 3:42 PM, Breno Leitao wrote:
>>> Hello,
>>>
>>> I am seeing a crash in some production host in function
>>> mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
>>> that doesn't seem to be addressable. I am seeing this on 6.13, but,
>>> looking at the upstream code, the function is the same.
>>>
>>> Unfortunately I am not able to reproduce this on upstream kernel easily.
>>> Host is running ConnectX-7.
>>>
>>> Here is the quick stack of the problem:
>>>
>>>      Unable to handle kernel paging request at virtual address 00000000213afe58
>>>
>>>      #0  string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
>>>      #1  string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
>>>      #2  vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
>>>      #3  snprintf (lib/vsprintf.c:2983:6)
>>>
>>> Looking further, I found this code:
>>>
>>>           snprintf(tmp, sizeof(tmp), str_frmt->string,
>>>                    str_frmt->params[0],
>>>                    str_frmt->params[1],
>>>                    str_frmt->params[2],
>>>                    str_frmt->params[3],
>>>                    str_frmt->params[4],
>>>                    str_frmt->params[5],
>>>                    str_frmt->params[6]);
>>>
>>>
>>> and the str_frmt has the following content:
>>>
>>>      *(struct tracer_string_format *)0xffff100026547260 = {
>>>      .string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
>>>      .params = (int [7]){ 557514328, 3 },
>>>      .num_of_params = (int)2,
>>>      .last_param_num = (int)2,
>>>      .event_id = (u8)3,
>>>      .tmsn = (u32)5201,
>>>      .hlist = (struct hlist_node){
>>>          .next = (struct hlist_node *)0xffff0009f63ce078,
>>>          .pprev = (struct hlist_node **)0xffff0004123ec8d8,
>>>      },
>>>      .list = (struct list_head){
>>>          .next = (struct list_head *)0xdead000000000100,
>>>          .prev = (struct list_head *)0xdead000000000122,
>>>      },
>>>      .timestamp = (u32)22,
>>>      .lost = (bool)0,
>>>      }
>>>
>>>
>>> My understanding that we are printf %s with params[0], which is 557514328 (aka
>>> 0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
>>> is invalid, and crash.
>>>
>>> Is this a known issue?
>>>
>>
>> Not a known issue, not expected, thanks for reporting.
>> We will send patch to protect from such crash.
>> Please send FW version it was detected on.
> 
> Hello!
> 
> I work with Breno and just following up on his behalf while he is away. Just wanted to check
> if there was an update on the patch?
> 

Hi, patch is currently in internal review.

Thanks,
Moshe.

> Thanks
> Usama
>>
>> Thanks,
>> Moshe.
>>
>>> Thanks
>>> --breno
>>>
>>
>