* mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()
@ 2025-10-09 12:42 Breno Leitao
2025-10-15 9:13 ` Moshe Shemesh
0 siblings, 1 reply; 6+ messages in thread
From: Breno Leitao @ 2025-10-09 12:42 UTC (permalink / raw)
To: saeedm, itayavr; +Cc: netdev, linux-kernel, dcostantino, kuba
Hello,
I am seeing a crash in some production host in function
mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
that doesn't seem to be addressable. I am seeing this on 6.13, but,
looking at the upstream code, the function is the same.
Unfortunately I am not able to reproduce this on upstream kernel easily.
Host is running ConnectX-7.
Here is the quick stack of the problem:
Unable to handle kernel paging request at virtual address 00000000213afe58
#0 string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
#1 string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
#2 vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
#3 snprintf (lib/vsprintf.c:2983:6)
Looking further, I found this code:
snprintf(tmp, sizeof(tmp), str_frmt->string,
str_frmt->params[0],
str_frmt->params[1],
str_frmt->params[2],
str_frmt->params[3],
str_frmt->params[4],
str_frmt->params[5],
str_frmt->params[6]);
and the str_frmt has the following content:
*(struct tracer_string_format *)0xffff100026547260 = {
.string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
.params = (int [7]){ 557514328, 3 },
.num_of_params = (int)2,
.last_param_num = (int)2,
.event_id = (u8)3,
.tmsn = (u32)5201,
.hlist = (struct hlist_node){
.next = (struct hlist_node *)0xffff0009f63ce078,
.pprev = (struct hlist_node **)0xffff0004123ec8d8,
},
.list = (struct list_head){
.next = (struct list_head *)0xdead000000000100,
.prev = (struct list_head *)0xdead000000000122,
},
.timestamp = (u32)22,
.lost = (bool)0,
}
My understanding that we are printf %s with params[0], which is 557514328 (aka
0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
is invalid, and crash.
Is this a known issue?
Thanks
--breno
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()
2025-10-09 12:42 mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace() Breno Leitao
@ 2025-10-15 9:13 ` Moshe Shemesh
2025-10-15 12:26 ` Breno Leitao
2025-11-03 10:45 ` Usama Arif
0 siblings, 2 replies; 6+ messages in thread
From: Moshe Shemesh @ 2025-10-15 9:13 UTC (permalink / raw)
To: Breno Leitao, saeedm, itayavr; +Cc: netdev, linux-kernel, dcostantino, kuba
On 10/9/2025 3:42 PM, Breno Leitao wrote:
> Hello,
>
> I am seeing a crash in some production host in function
> mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
> that doesn't seem to be addressable. I am seeing this on 6.13, but,
> looking at the upstream code, the function is the same.
>
> Unfortunately I am not able to reproduce this on upstream kernel easily.
> Host is running ConnectX-7.
>
> Here is the quick stack of the problem:
>
> Unable to handle kernel paging request at virtual address 00000000213afe58
>
> #0 string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
> #1 string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
> #2 vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
> #3 snprintf (lib/vsprintf.c:2983:6)
>
> Looking further, I found this code:
>
> snprintf(tmp, sizeof(tmp), str_frmt->string,
> str_frmt->params[0],
> str_frmt->params[1],
> str_frmt->params[2],
> str_frmt->params[3],
> str_frmt->params[4],
> str_frmt->params[5],
> str_frmt->params[6]);
>
>
> and the str_frmt has the following content:
>
> *(struct tracer_string_format *)0xffff100026547260 = {
> .string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
> .params = (int [7]){ 557514328, 3 },
> .num_of_params = (int)2,
> .last_param_num = (int)2,
> .event_id = (u8)3,
> .tmsn = (u32)5201,
> .hlist = (struct hlist_node){
> .next = (struct hlist_node *)0xffff0009f63ce078,
> .pprev = (struct hlist_node **)0xffff0004123ec8d8,
> },
> .list = (struct list_head){
> .next = (struct list_head *)0xdead000000000100,
> .prev = (struct list_head *)0xdead000000000122,
> },
> .timestamp = (u32)22,
> .lost = (bool)0,
> }
>
>
> My understanding that we are printf %s with params[0], which is 557514328 (aka
> 0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
> is invalid, and crash.
>
> Is this a known issue?
>
Not a known issue, not expected, thanks for reporting.
We will send patch to protect from such crash.
Please send FW version it was detected on.
Thanks,
Moshe.
> Thanks
> --breno
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()
2025-10-15 9:13 ` Moshe Shemesh
@ 2025-10-15 12:26 ` Breno Leitao
2025-10-16 15:39 ` Moshe Shemesh
2025-11-03 10:45 ` Usama Arif
1 sibling, 1 reply; 6+ messages in thread
From: Breno Leitao @ 2025-10-15 12:26 UTC (permalink / raw)
To: Moshe Shemesh; +Cc: saeedm, itayavr, netdev, linux-kernel, dcostantino, kuba
Hello Moshe,
On Wed, Oct 15, 2025 at 12:13:29PM +0300, Moshe Shemesh wrote:
> On 10/9/2025 3:42 PM, Breno Leitao wrote:
> >
> > My understanding that we are printf %s with params[0], which is 557514328 (aka
> > 0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
> > is invalid, and crash.
> >
> > Is this a known issue?
>
> Not a known issue, not expected, thanks for reporting.
> We will send patch to protect from such crash.
Thanks. how do you plan to protect it? I understand that the string is
coming from the firmware and the kernel is just using it in snprintf. Is
it right?
> Please send FW version it was detected on.
`ethtool -i` outputs:
firmware-version: 28.44.2506 (FB_0000000030)
Thanks for your answer,
--breno
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()
2025-10-15 12:26 ` Breno Leitao
@ 2025-10-16 15:39 ` Moshe Shemesh
0 siblings, 0 replies; 6+ messages in thread
From: Moshe Shemesh @ 2025-10-16 15:39 UTC (permalink / raw)
To: Breno Leitao; +Cc: saeedm, itayavr, netdev, linux-kernel, dcostantino, kuba
On 10/15/2025 3:26 PM, Breno Leitao wrote:
>
> Hello Moshe,
>
> On Wed, Oct 15, 2025 at 12:13:29PM +0300, Moshe Shemesh wrote:
>> On 10/9/2025 3:42 PM, Breno Leitao wrote:
>>>
>>> My understanding that we are printf %s with params[0], which is 557514328 (aka
>>> 0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
>>> is invalid, and crash.
>>>
>>> Is this a known issue?
>>
>> Not a known issue, not expected, thanks for reporting.
>> We will send patch to protect from such crash.
>
> Thanks. how do you plan to protect it? I understand that the string is
> coming from the firmware and the kernel is just using it in snprintf. Is
> it right?
Yes, but FW should not use %s. Driver can check it.
>
>> Please send FW version it was detected on.
>
> `ethtool -i` outputs:
> firmware-version: 28.44.2506 (FB_0000000030)
Thanks.
>
> Thanks for your answer,
> --breno
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()
2025-10-15 9:13 ` Moshe Shemesh
2025-10-15 12:26 ` Breno Leitao
@ 2025-11-03 10:45 ` Usama Arif
2025-11-04 12:09 ` Moshe Shemesh
1 sibling, 1 reply; 6+ messages in thread
From: Usama Arif @ 2025-11-03 10:45 UTC (permalink / raw)
To: Moshe Shemesh, Breno Leitao, saeedm, itayavr
Cc: netdev, linux-kernel, dcostantino, kuba
On 15/10/2025 12:13, Moshe Shemesh wrote:
>
>
> On 10/9/2025 3:42 PM, Breno Leitao wrote:
>> Hello,
>>
>> I am seeing a crash in some production host in function
>> mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
>> that doesn't seem to be addressable. I am seeing this on 6.13, but,
>> looking at the upstream code, the function is the same.
>>
>> Unfortunately I am not able to reproduce this on upstream kernel easily.
>> Host is running ConnectX-7.
>>
>> Here is the quick stack of the problem:
>>
>> Unable to handle kernel paging request at virtual address 00000000213afe58
>>
>> #0 string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
>> #1 string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
>> #2 vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
>> #3 snprintf (lib/vsprintf.c:2983:6)
>>
>> Looking further, I found this code:
>>
>> snprintf(tmp, sizeof(tmp), str_frmt->string,
>> str_frmt->params[0],
>> str_frmt->params[1],
>> str_frmt->params[2],
>> str_frmt->params[3],
>> str_frmt->params[4],
>> str_frmt->params[5],
>> str_frmt->params[6]);
>>
>>
>> and the str_frmt has the following content:
>>
>> *(struct tracer_string_format *)0xffff100026547260 = {
>> .string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
>> .params = (int [7]){ 557514328, 3 },
>> .num_of_params = (int)2,
>> .last_param_num = (int)2,
>> .event_id = (u8)3,
>> .tmsn = (u32)5201,
>> .hlist = (struct hlist_node){
>> .next = (struct hlist_node *)0xffff0009f63ce078,
>> .pprev = (struct hlist_node **)0xffff0004123ec8d8,
>> },
>> .list = (struct list_head){
>> .next = (struct list_head *)0xdead000000000100,
>> .prev = (struct list_head *)0xdead000000000122,
>> },
>> .timestamp = (u32)22,
>> .lost = (bool)0,
>> }
>>
>>
>> My understanding that we are printf %s with params[0], which is 557514328 (aka
>> 0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
>> is invalid, and crash.
>>
>> Is this a known issue?
>>
>
> Not a known issue, not expected, thanks for reporting.
> We will send patch to protect from such crash.
> Please send FW version it was detected on.
Hello!
I work with Breno and just following up on his behalf while he is away. Just wanted to check
if there was an update on the patch?
Thanks
Usama
>
> Thanks,
> Moshe.
>
>> Thanks
>> --breno
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()
2025-11-03 10:45 ` Usama Arif
@ 2025-11-04 12:09 ` Moshe Shemesh
0 siblings, 0 replies; 6+ messages in thread
From: Moshe Shemesh @ 2025-11-04 12:09 UTC (permalink / raw)
To: Usama Arif, Breno Leitao, saeedm, itayavr
Cc: netdev, linux-kernel, dcostantino, kuba
On 11/3/2025 12:45 PM, Usama Arif wrote:
> On 15/10/2025 12:13, Moshe Shemesh wrote:
>>
>>
>> On 10/9/2025 3:42 PM, Breno Leitao wrote:
>>> Hello,
>>>
>>> I am seeing a crash in some production host in function
>>> mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
>>> that doesn't seem to be addressable. I am seeing this on 6.13, but,
>>> looking at the upstream code, the function is the same.
>>>
>>> Unfortunately I am not able to reproduce this on upstream kernel easily.
>>> Host is running ConnectX-7.
>>>
>>> Here is the quick stack of the problem:
>>>
>>> Unable to handle kernel paging request at virtual address 00000000213afe58
>>>
>>> #0 string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
>>> #1 string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
>>> #2 vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
>>> #3 snprintf (lib/vsprintf.c:2983:6)
>>>
>>> Looking further, I found this code:
>>>
>>> snprintf(tmp, sizeof(tmp), str_frmt->string,
>>> str_frmt->params[0],
>>> str_frmt->params[1],
>>> str_frmt->params[2],
>>> str_frmt->params[3],
>>> str_frmt->params[4],
>>> str_frmt->params[5],
>>> str_frmt->params[6]);
>>>
>>>
>>> and the str_frmt has the following content:
>>>
>>> *(struct tracer_string_format *)0xffff100026547260 = {
>>> .string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
>>> .params = (int [7]){ 557514328, 3 },
>>> .num_of_params = (int)2,
>>> .last_param_num = (int)2,
>>> .event_id = (u8)3,
>>> .tmsn = (u32)5201,
>>> .hlist = (struct hlist_node){
>>> .next = (struct hlist_node *)0xffff0009f63ce078,
>>> .pprev = (struct hlist_node **)0xffff0004123ec8d8,
>>> },
>>> .list = (struct list_head){
>>> .next = (struct list_head *)0xdead000000000100,
>>> .prev = (struct list_head *)0xdead000000000122,
>>> },
>>> .timestamp = (u32)22,
>>> .lost = (bool)0,
>>> }
>>>
>>>
>>> My understanding that we are printf %s with params[0], which is 557514328 (aka
>>> 0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
>>> is invalid, and crash.
>>>
>>> Is this a known issue?
>>>
>>
>> Not a known issue, not expected, thanks for reporting.
>> We will send patch to protect from such crash.
>> Please send FW version it was detected on.
>
> Hello!
>
> I work with Breno and just following up on his behalf while he is away. Just wanted to check
> if there was an update on the patch?
>
Hi, patch is currently in internal review.
Thanks,
Moshe.
> Thanks
> Usama
>>
>> Thanks,
>> Moshe.
>>
>>> Thanks
>>> --breno
>>>
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-11-04 12:11 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-10-09 12:42 mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace() Breno Leitao
2025-10-15 9:13 ` Moshe Shemesh
2025-10-15 12:26 ` Breno Leitao
2025-10-16 15:39 ` Moshe Shemesh
2025-11-03 10:45 ` Usama Arif
2025-11-04 12:09 ` Moshe Shemesh
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®