Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()

[Moshe Shemesh <moshe@nvidia.com>]

On 10/9/2025 3:42 PM, Breno Leitao wrote:
> Hello,
> 
> I am seeing a crash in some production host in function
> mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
> that doesn't seem to be addressable. I am seeing this on 6.13, but,
> looking at the upstream code, the function is the same.
> 
> Unfortunately I am not able to reproduce this on upstream kernel easily.
> Host is running ConnectX-7.
> 
> Here is the quick stack of the problem:
> 
> 	Unable to handle kernel paging request at virtual address 00000000213afe58
> 
> 	#0  string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
> 	#1  string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
> 	#2  vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
> 	#3  snprintf (lib/vsprintf.c:2983:6)
> 
> Looking further, I found this code:
> 
>          snprintf(tmp, sizeof(tmp), str_frmt->string,
>                   str_frmt->params[0],
>                   str_frmt->params[1],
>                   str_frmt->params[2],
>                   str_frmt->params[3],
>                   str_frmt->params[4],
>                   str_frmt->params[5],
>                   str_frmt->params[6]);
> 
> 
> and the str_frmt has the following content:
> 
> 	*(struct tracer_string_format *)0xffff100026547260 = {
> 	.string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
> 	.params = (int [7]){ 557514328, 3 },
> 	.num_of_params = (int)2,
> 	.last_param_num = (int)2,
> 	.event_id = (u8)3,
> 	.tmsn = (u32)5201,
> 	.hlist = (struct hlist_node){
> 		.next = (struct hlist_node *)0xffff0009f63ce078,
> 		.pprev = (struct hlist_node **)0xffff0004123ec8d8,
> 	},
> 	.list = (struct list_head){
> 		.next = (struct list_head *)0xdead000000000100,
> 		.prev = (struct list_head *)0xdead000000000122,
> 	},
> 	.timestamp = (u32)22,
> 	.lost = (bool)0,
> 	}
> 
> 
> My understanding that we are printf %s with params[0], which is 557514328 (aka
> 0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
> is invalid, and crash.
> 
> Is this a known issue?
> 

Not a known issue, not expected, thanks for reporting.
We will send patch to protect from such crash.
Please send FW version it was detected on.

Thanks,
Moshe.

> Thanks
> --breno
>