vhost: linux-next: kernel crash at vhost_dev_cleanup/kfree

vhost: linux-next: kernel crash at vhost_dev_cleanup/kfree

[JAEHOON KIM]

Dear Jason Wang,

I would like to kindly report a kernel crash issue on our s390x server 
which seems to be related to the following patch.
--------------------------------------------------------------------------------------------------------------------------
   commit 7918bb2d19c9 ("vhost: basic in order support")
https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/commit/?id=7918bb2d19c9
--------------------------------------------------------------------------------------------------------------------------

This patch landed in linux-next between July 16th and 17th. Since then,  
kernel crash have been observed during stress testing.
The issue can be confirmed using the following command:
-------------------------------------------
   stress-ng --dev 1 -t 10s
-------------------------------------------

Crash log and call stack are as follows.
Additionally, this crash appears similar to the issue discussed in the 
following thread:
https://lore.kernel.org/kvm/bvjomrplpsjklglped5pmwttzmljigasdafjiizt2sfmytc5rr@ljpu455kx52j/

[ 5413.029569] Unable to handle kernel pointer dereference in virtual 
kernel address space
[ 5413.029573] Failing address: 00000328856e8000 TEID: 00000328856e8803
[ 5413.029576] Fault in home space mode while using kernel ASCE.
[ 5413.029580] AS:0000000371fdc007 R3:0000000000000024
[ 5413.029607] Oops: 003b ilc:3 [#1]SMP
   .......
[ 5413.029655] CPU: 23 UID: 0 PID: 2339 Comm: stress-ng-dev Not tainted 
6.16.0-rc6-10099-g60a66ed35d6b #63 NONE
[ 5413.029659] Hardware name: IBM 3906 M05 780 (LPAR)
[ 5413.029662] Krnl PSW : 0704e00180000000 0000032714b9f156 
(kfree+0x66/0x340)
[ 5413.029673]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 
PM:0 RI:0 EA:3
[ 5413.029677] Krnl GPRS: 0000000000000002 0000008c056e8000 
0000262500000000 0000000085bf4610
[ 5413.029681]            0000000085bf4660 0000000085bf4618 
0000032716402270 0000032694e0391a
[ 5413.029683]            0000032716402290 0000032714720000 
00000328856e8000 0000262500000000
[ 5413.029685]            000003ff8312cfa8 0000000000000000 
000023015ba00000 000002a71e8d3ba8
[ 5413.029697] Krnl Code: 0000032714b9f146: e3e060080008 ag %r14,8(%r6)
[ 5413.029697]            0000032714b9f14c: ec1e06b93a59 risbgn 
%r1,%r14,6,185,58
[ 5413.029697]           #0000032714b9f152: b90800a1 agr %r10,%r1
[ 5413.029697]           >0000032714b9f156: e320a0080004 lg      %r2,8(%r10)
[ 5413.029697]            0000032714b9f15c: a7210001 tmll    %r2,1
[ 5413.029697]            0000032714b9f160: a77400e0 brc 7,0000032714b9f320
[ 5413.029697]            0000032714b9f164: c004000000ca brcl 
0,0000032714b9f2f8
[ 5413.029697]            0000032714b9f16a: 95f5a030 cli 48(%r10),245
[ 5413.029738] Call Trace:
[ 5413.029741]  [<0000032714b9f156>] kfree+0x66/0x340
[ 5413.029747]  [<0000032694e0391a>] vhost_dev_free_iovecs+0x9a/0xc0 [vhost]
[ 5413.029757]  [<0000032694e05406>] vhost_dev_cleanup+0xb6/0x210 [vhost]
[ 5413.029763]  [<000003269507000a>] vhost_vsock_dev_release+0x1aa/0x1e0 
[vhost_vsock]
[ 5413.029768]  [<0000032714c16ece>] __fput+0xee/0x2e0
[ 5413.029774]  [<00000327148c0488>] task_work_run+0x88/0xd0
[ 5413.029783]  [<00000327148977aa>] do_exit+0x18a/0x4e0
[ 5413.029786]  [<0000032714897cf0>] do_group_exit+0x40/0xc0
[ 5413.029789]  [<0000032714897dce>] __s390x_sys_exit_group+0x2e/0x30
[ 5413.029792]  [<00000327156519c6>] __do_syscall+0x136/0x340
[ 5413.029797]  [<000003271565d5de>] system_call+0x6e/0x90
[ 5413.029802] Last Breaking-Event-Address:
[ 5413.029803]  [<0000032694e03914>] vhost_dev_free_iovecs+0x94/0xc0 [vhost]
[ 5413.029811] Kernel panic - not syncing: Fatal exception: panic_on_oops


Best regards,
Jaehoon Kim

Re: vhost: linux-next: kernel crash at vhost_dev_cleanup/kfree

[Jason Wang]

On Tue, Jul 29, 2025 at 5:25 AM JAEHOON KIM <jhkim@linux.ibm.com> wrote:
>
>
> Dear Jason Wang,
>
> I would like to kindly report a kernel crash issue on our s390x server
> which seems to be related to the following patch.
> --------------------------------------------------------------------------------------------------------------------------
>    commit 7918bb2d19c9 ("vhost: basic in order support")
> https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/commit/?id=7918bb2d19c9
> --------------------------------------------------------------------------------------------------------------------------
>
> This patch landed in linux-next between July 16th and 17th. Since then,
> kernel crash have been observed during stress testing.
> The issue can be confirmed using the following command:
> -------------------------------------------
>    stress-ng --dev 1 -t 10s
> -------------------------------------------

Right, I forgot to initialize vq->nheads in vhost_dev_init().

I've posted a fix here:

https://lore.kernel.org/virtualization/20250729073916.80647-1-jasowang@redhat.com/T/#u

Thanks

>
> Crash log and call stack are as follows.
> Additionally, this crash appears similar to the issue discussed in the
> following thread:
> https://lore.kernel.org/kvm/bvjomrplpsjklglped5pmwttzmljigasdafjiizt2sfmytc5rr@ljpu455kx52j/
>
> [ 5413.029569] Unable to handle kernel pointer dereference in virtual
> kernel address space
> [ 5413.029573] Failing address: 00000328856e8000 TEID: 00000328856e8803
> [ 5413.029576] Fault in home space mode while using kernel ASCE.
> [ 5413.029580] AS:0000000371fdc007 R3:0000000000000024
> [ 5413.029607] Oops: 003b ilc:3 [#1]SMP
>    .......
> [ 5413.029655] CPU: 23 UID: 0 PID: 2339 Comm: stress-ng-dev Not tainted
> 6.16.0-rc6-10099-g60a66ed35d6b #63 NONE
> [ 5413.029659] Hardware name: IBM 3906 M05 780 (LPAR)
> [ 5413.029662] Krnl PSW : 0704e00180000000 0000032714b9f156
> (kfree+0x66/0x340)
> [ 5413.029673]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2
> PM:0 RI:0 EA:3
> [ 5413.029677] Krnl GPRS: 0000000000000002 0000008c056e8000
> 0000262500000000 0000000085bf4610
> [ 5413.029681]            0000000085bf4660 0000000085bf4618
> 0000032716402270 0000032694e0391a
> [ 5413.029683]            0000032716402290 0000032714720000
> 00000328856e8000 0000262500000000
> [ 5413.029685]            000003ff8312cfa8 0000000000000000
> 000023015ba00000 000002a71e8d3ba8
> [ 5413.029697] Krnl Code: 0000032714b9f146: e3e060080008 ag %r14,8(%r6)
> [ 5413.029697]            0000032714b9f14c: ec1e06b93a59 risbgn
> %r1,%r14,6,185,58
> [ 5413.029697]           #0000032714b9f152: b90800a1 agr %r10,%r1
> [ 5413.029697]           >0000032714b9f156: e320a0080004 lg      %r2,8(%r10)
> [ 5413.029697]            0000032714b9f15c: a7210001 tmll    %r2,1
> [ 5413.029697]            0000032714b9f160: a77400e0 brc 7,0000032714b9f320
> [ 5413.029697]            0000032714b9f164: c004000000ca brcl
> 0,0000032714b9f2f8
> [ 5413.029697]            0000032714b9f16a: 95f5a030 cli 48(%r10),245
> [ 5413.029738] Call Trace:
> [ 5413.029741]  [<0000032714b9f156>] kfree+0x66/0x340
> [ 5413.029747]  [<0000032694e0391a>] vhost_dev_free_iovecs+0x9a/0xc0 [vhost]
> [ 5413.029757]  [<0000032694e05406>] vhost_dev_cleanup+0xb6/0x210 [vhost]
> [ 5413.029763]  [<000003269507000a>] vhost_vsock_dev_release+0x1aa/0x1e0
> [vhost_vsock]
> [ 5413.029768]  [<0000032714c16ece>] __fput+0xee/0x2e0
> [ 5413.029774]  [<00000327148c0488>] task_work_run+0x88/0xd0
> [ 5413.029783]  [<00000327148977aa>] do_exit+0x18a/0x4e0
> [ 5413.029786]  [<0000032714897cf0>] do_group_exit+0x40/0xc0
> [ 5413.029789]  [<0000032714897dce>] __s390x_sys_exit_group+0x2e/0x30
> [ 5413.029792]  [<00000327156519c6>] __do_syscall+0x136/0x340
> [ 5413.029797]  [<000003271565d5de>] system_call+0x6e/0x90
> [ 5413.029802] Last Breaking-Event-Address:
> [ 5413.029803]  [<0000032694e03914>] vhost_dev_free_iovecs+0x94/0xc0 [vhost]
> [ 5413.029811] Kernel panic - not syncing: Fatal exception: panic_on_oops
>
>
> Best regards,
> Jaehoon Kim
>