* 6.10/bisected/regression - Since commit e356d321d024 in the kernel log appears the message "MES failed to respond to msg=MISC (WAIT_REG_MEM)" which were never seen before
@ 2024-07-20 17:08 Mikhail Gavrilov
2024-07-22 8:43 ` Christian König
0 siblings, 1 reply; 5+ messages in thread
From: Mikhail Gavrilov @ 2024-07-20 17:08 UTC (permalink / raw)
To: amd-gfx list, dri-devel, Christian König, Deucher,
Alexander, mukul.joshi, Linux List Kernel Mailing,
Linux regressions mailing list
[-- Attachment #1: Type: text/plain, Size: 9929 bytes --]
Hi,
I spotted "MES failed to respond to msg=MISC (WAIT_REG_MEM)" messages
in my kernel log since 6.10-rc5.
After this message, usually follow "[drm:amdgpu_mes_reg_write_reg_wait
[amdgpu]] *ERROR* failed to reg_write_reg_wait".
[ 8972.590502] input: Noble FoKus Mystique (AVRCP) as
/devices/virtual/input/input21
[ 9964.748433] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748433] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748494] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748476] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748479] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748661] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9964.748770] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9977.224893] Bluetooth: hci0: ACL packet for unknown connection handle 3837
[ 9980.347061] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.347077] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349859] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349859] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
msg=MISC (WAIT_REG_MEM)
[ 9980.349870] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349868] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349870] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349890] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349865] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349865] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349867] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349867] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349869] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
failed to reg_write_reg_wait
[10037.250083] Bluetooth: hci0: ACL packet for unknown connection handle 3837
[12054.238867] workqueue: gc_worker [nf_conntrack] hogged CPU for
>10000us 1027 times, consider switching to WQ_UNBOUND
[12851.087896] fossilize_repla (45968) used greatest stack depth:
17440 bytes left
Unfortunately, it is not easily reproducible.
Usually it appears when I play several hours in the game "STAR WARS
Jedi: Survivor".
So it is why I bisected it so long.
git bisect start
# status: waiting for both good and bad commits
# bad: [f2661062f16b2de5d7b6a5c42a9a5c96326b8454] Linux 6.10-rc5
git bisect bad f2661062f16b2de5d7b6a5c42a9a5c96326b8454
# good: [50736169ecc8387247fe6a00932852ce7b057083] Merge tag
'for-6.10-rc4-tag' of
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
git bisect good 50736169ecc8387247fe6a00932852ce7b057083
# bad: [d4ba3313e84dfcdeb92a13434a2d02aad5e973e1] Merge tag
'loongarch-fixes-6.10-2' of
git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
git bisect bad d4ba3313e84dfcdeb92a13434a2d02aad5e973e1
# good: [264efe488fd82cf3145a3dc625f394c61db99934] Merge tag
'ovl-fixes-6.10-rc5' of
git://git.kernel.org/pub/scm/linux/kernel/git/overlayfs/vfs
git bisect good 264efe488fd82cf3145a3dc625f394c61db99934
# bad: [35bb670d65fc0f80c62383ab4f2544cec85ac57a] Merge tag
'scsi-fixes' of
git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
git bisect bad 35bb670d65fc0f80c62383ab4f2544cec85ac57a
# good: [f0d576f840153392d04b2d52cf3adab8f62e8cb6] drm/amdgpu: fix
UBSAN warning in kv_dpm.c
git bisect good f0d576f840153392d04b2d52cf3adab8f62e8cb6
# bad: [07e06189c5ea7ffe897d12b546c918380d3bffb1] Merge tag
'amd-drm-fixes-6.10-2024-06-19' of
https://gitlab.freedesktop.org/agd5f/linux into drm-fixes
git bisect bad 07e06189c5ea7ffe897d12b546c918380d3bffb1
# bad: [ed5a4484f074aa2bfb1dad99ff3628ea8da4acdc] drm/amdgpu: init TA
fw for psp v14
git bisect bad ed5a4484f074aa2bfb1dad99ff3628ea8da4acdc
# bad: [e356d321d0240663a09b139fa3658ddbca163e27] drm/amdgpu: cleanup
MES11 command submission
git bisect bad e356d321d0240663a09b139fa3658ddbca163e27
# first bad commit: [e356d321d0240663a09b139fa3658ddbca163e27]
drm/amdgpu: cleanup MES11 command submission
Author: Christian König <christian.koenig@amd.com>
Date: Fri May 31 10:56:00 2024 +0200
drm/amdgpu: cleanup MES11 command submission
The approach of having a separate WB slot for each submission doesn't
really work well and for example breaks GPU reset.
Use a status query packet for the fence update instead since those
should always succeed we can use the fence of the original packet to
signal the state of the operation.
While at it cleanup the coding style.
Fixes: eef016ba8986 ("drm/amdgpu/mes11: Use a separate fence per
transaction")
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
And I can confirm after reverting e356d321d024 I played the whole day,
and the "MES failed to respond" error message does not appear anymore.
My hardware specs are: https://linux-hardware.org/?probe=78d8c680db
Christian, can you look into it, please?
--
Best Regards,
Mike Gavrilov.
[-- Attachment #2: dmesg.zip --]
[-- Type: application/zip, Size: 57513 bytes --]
[-- Attachment #3: .config.zip --]
[-- Type: application/zip, Size: 66515 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 6.10/bisected/regression - Since commit e356d321d024 in the kernel log appears the message "MES failed to respond to msg=MISC (WAIT_REG_MEM)" which were never seen before
2024-07-20 17:08 6.10/bisected/regression - Since commit e356d321d024 in the kernel log appears the message "MES failed to respond to msg=MISC (WAIT_REG_MEM)" which were never seen before Mikhail Gavrilov
@ 2024-07-22 8:43 ` Christian König
2024-07-22 21:33 ` Alex Deucher
0 siblings, 1 reply; 5+ messages in thread
From: Christian König @ 2024-07-22 8:43 UTC (permalink / raw)
To: Mikhail Gavrilov, amd-gfx list, dri-devel, Christian König,
Deucher, Alexander, mukul.joshi, Linux List Kernel Mailing,
Linux regressions mailing list
That's a known issue and we are already working on it.
Regards,
Christian.
Am 20.07.24 um 19:08 schrieb Mikhail Gavrilov:
> Hi,
> I spotted "MES failed to respond to msg=MISC (WAIT_REG_MEM)" messages
> in my kernel log since 6.10-rc5.
> After this message, usually follow "[drm:amdgpu_mes_reg_write_reg_wait
> [amdgpu]] *ERROR* failed to reg_write_reg_wait".
>
> [ 8972.590502] input: Noble FoKus Mystique (AVRCP) as
> /devices/virtual/input/input21
> [ 9964.748433] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748433] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748494] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748476] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748479] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748661] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9964.748770] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9977.224893] Bluetooth: hci0: ACL packet for unknown connection handle 3837
> [ 9980.347061] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.347077] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349859] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349859] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> msg=MISC (WAIT_REG_MEM)
> [ 9980.349870] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349868] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349870] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349890] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349865] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349865] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349867] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349867] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349869] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> failed to reg_write_reg_wait
> [10037.250083] Bluetooth: hci0: ACL packet for unknown connection handle 3837
> [12054.238867] workqueue: gc_worker [nf_conntrack] hogged CPU for
>> 10000us 1027 times, consider switching to WQ_UNBOUND
> [12851.087896] fossilize_repla (45968) used greatest stack depth:
> 17440 bytes left
>
> Unfortunately, it is not easily reproducible.
> Usually it appears when I play several hours in the game "STAR WARS
> Jedi: Survivor".
> So it is why I bisected it so long.
>
> git bisect start
> # status: waiting for both good and bad commits
> # bad: [f2661062f16b2de5d7b6a5c42a9a5c96326b8454] Linux 6.10-rc5
> git bisect bad f2661062f16b2de5d7b6a5c42a9a5c96326b8454
> # good: [50736169ecc8387247fe6a00932852ce7b057083] Merge tag
> 'for-6.10-rc4-tag' of
> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
> git bisect good 50736169ecc8387247fe6a00932852ce7b057083
> # bad: [d4ba3313e84dfcdeb92a13434a2d02aad5e973e1] Merge tag
> 'loongarch-fixes-6.10-2' of
> git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
> git bisect bad d4ba3313e84dfcdeb92a13434a2d02aad5e973e1
> # good: [264efe488fd82cf3145a3dc625f394c61db99934] Merge tag
> 'ovl-fixes-6.10-rc5' of
> git://git.kernel.org/pub/scm/linux/kernel/git/overlayfs/vfs
> git bisect good 264efe488fd82cf3145a3dc625f394c61db99934
> # bad: [35bb670d65fc0f80c62383ab4f2544cec85ac57a] Merge tag
> 'scsi-fixes' of
> git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
> git bisect bad 35bb670d65fc0f80c62383ab4f2544cec85ac57a
> # good: [f0d576f840153392d04b2d52cf3adab8f62e8cb6] drm/amdgpu: fix
> UBSAN warning in kv_dpm.c
> git bisect good f0d576f840153392d04b2d52cf3adab8f62e8cb6
> # bad: [07e06189c5ea7ffe897d12b546c918380d3bffb1] Merge tag
> 'amd-drm-fixes-6.10-2024-06-19' of
> https://gitlab.freedesktop.org/agd5f/linux into drm-fixes
> git bisect bad 07e06189c5ea7ffe897d12b546c918380d3bffb1
> # bad: [ed5a4484f074aa2bfb1dad99ff3628ea8da4acdc] drm/amdgpu: init TA
> fw for psp v14
> git bisect bad ed5a4484f074aa2bfb1dad99ff3628ea8da4acdc
> # bad: [e356d321d0240663a09b139fa3658ddbca163e27] drm/amdgpu: cleanup
> MES11 command submission
> git bisect bad e356d321d0240663a09b139fa3658ddbca163e27
> # first bad commit: [e356d321d0240663a09b139fa3658ddbca163e27]
> drm/amdgpu: cleanup MES11 command submission
>
> Author: Christian König <christian.koenig@amd.com>
> Date: Fri May 31 10:56:00 2024 +0200
>
> drm/amdgpu: cleanup MES11 command submission
>
> The approach of having a separate WB slot for each submission doesn't
> really work well and for example breaks GPU reset.
>
> Use a status query packet for the fence update instead since those
> should always succeed we can use the fence of the original packet to
> signal the state of the operation.
>
> While at it cleanup the coding style.
>
> Fixes: eef016ba8986 ("drm/amdgpu/mes11: Use a separate fence per
> transaction")
> Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
> Signed-off-by: Christian König <christian.koenig@amd.com>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
>
> And I can confirm after reverting e356d321d024 I played the whole day,
> and the "MES failed to respond" error message does not appear anymore.
>
> My hardware specs are: https://linux-hardware.org/?probe=78d8c680db
>
> Christian, can you look into it, please?
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 6.10/bisected/regression - Since commit e356d321d024 in the kernel log appears the message "MES failed to respond to msg=MISC (WAIT_REG_MEM)" which were never seen before
2024-07-22 8:43 ` Christian König
@ 2024-07-22 21:33 ` Alex Deucher
2024-07-24 17:16 ` Mikhail Gavrilov
0 siblings, 1 reply; 5+ messages in thread
From: Alex Deucher @ 2024-07-22 21:33 UTC (permalink / raw)
To: Christian König
Cc: Mikhail Gavrilov, amd-gfx list, dri-devel, Christian König,
Deucher, Alexander, mukul.joshi, Linux List Kernel Mailing,
Linux regressions mailing list
On Mon, Jul 22, 2024 at 4:50 AM Christian König
<ckoenig.leichtzumerken@gmail.com> wrote:
>
> That's a known issue and we are already working on it.
Do either of these patches help?
https://patchwork.freedesktop.org/patch/605437/
https://patchwork.freedesktop.org/patch/605201/
Alex
>
> Regards,
> Christian.
>
> Am 20.07.24 um 19:08 schrieb Mikhail Gavrilov:
> > Hi,
> > I spotted "MES failed to respond to msg=MISC (WAIT_REG_MEM)" messages
> > in my kernel log since 6.10-rc5.
> > After this message, usually follow "[drm:amdgpu_mes_reg_write_reg_wait
> > [amdgpu]] *ERROR* failed to reg_write_reg_wait".
> >
> > [ 8972.590502] input: Noble FoKus Mystique (AVRCP) as
> > /devices/virtual/input/input21
> > [ 9964.748433] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748433] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748434] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748494] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748493] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748476] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748479] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748477] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748478] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748661] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9964.748770] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9977.224893] Bluetooth: hci0: ACL packet for unknown connection handle 3837
> > [ 9980.347061] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.347077] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349857] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349859] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349858] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349859] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to
> > msg=MISC (WAIT_REG_MEM)
> > [ 9980.349870] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349868] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349870] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349890] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349865] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349865] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349866] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349867] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349867] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349869] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [ 9980.349871] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR*
> > failed to reg_write_reg_wait
> > [10037.250083] Bluetooth: hci0: ACL packet for unknown connection handle 3837
> > [12054.238867] workqueue: gc_worker [nf_conntrack] hogged CPU for
> >> 10000us 1027 times, consider switching to WQ_UNBOUND
> > [12851.087896] fossilize_repla (45968) used greatest stack depth:
> > 17440 bytes left
> >
> > Unfortunately, it is not easily reproducible.
> > Usually it appears when I play several hours in the game "STAR WARS
> > Jedi: Survivor".
> > So it is why I bisected it so long.
> >
> > git bisect start
> > # status: waiting for both good and bad commits
> > # bad: [f2661062f16b2de5d7b6a5c42a9a5c96326b8454] Linux 6.10-rc5
> > git bisect bad f2661062f16b2de5d7b6a5c42a9a5c96326b8454
> > # good: [50736169ecc8387247fe6a00932852ce7b057083] Merge tag
> > 'for-6.10-rc4-tag' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
> > git bisect good 50736169ecc8387247fe6a00932852ce7b057083
> > # bad: [d4ba3313e84dfcdeb92a13434a2d02aad5e973e1] Merge tag
> > 'loongarch-fixes-6.10-2' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
> > git bisect bad d4ba3313e84dfcdeb92a13434a2d02aad5e973e1
> > # good: [264efe488fd82cf3145a3dc625f394c61db99934] Merge tag
> > 'ovl-fixes-6.10-rc5' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/overlayfs/vfs
> > git bisect good 264efe488fd82cf3145a3dc625f394c61db99934
> > # bad: [35bb670d65fc0f80c62383ab4f2544cec85ac57a] Merge tag
> > 'scsi-fixes' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
> > git bisect bad 35bb670d65fc0f80c62383ab4f2544cec85ac57a
> > # good: [f0d576f840153392d04b2d52cf3adab8f62e8cb6] drm/amdgpu: fix
> > UBSAN warning in kv_dpm.c
> > git bisect good f0d576f840153392d04b2d52cf3adab8f62e8cb6
> > # bad: [07e06189c5ea7ffe897d12b546c918380d3bffb1] Merge tag
> > 'amd-drm-fixes-6.10-2024-06-19' of
> > https://gitlab.freedesktop.org/agd5f/linux into drm-fixes
> > git bisect bad 07e06189c5ea7ffe897d12b546c918380d3bffb1
> > # bad: [ed5a4484f074aa2bfb1dad99ff3628ea8da4acdc] drm/amdgpu: init TA
> > fw for psp v14
> > git bisect bad ed5a4484f074aa2bfb1dad99ff3628ea8da4acdc
> > # bad: [e356d321d0240663a09b139fa3658ddbca163e27] drm/amdgpu: cleanup
> > MES11 command submission
> > git bisect bad e356d321d0240663a09b139fa3658ddbca163e27
> > # first bad commit: [e356d321d0240663a09b139fa3658ddbca163e27]
> > drm/amdgpu: cleanup MES11 command submission
> >
> > Author: Christian König <christian.koenig@amd.com>
> > Date: Fri May 31 10:56:00 2024 +0200
> >
> > drm/amdgpu: cleanup MES11 command submission
> >
> > The approach of having a separate WB slot for each submission doesn't
> > really work well and for example breaks GPU reset.
> >
> > Use a status query packet for the fence update instead since those
> > should always succeed we can use the fence of the original packet to
> > signal the state of the operation.
> >
> > While at it cleanup the coding style.
> >
> > Fixes: eef016ba8986 ("drm/amdgpu/mes11: Use a separate fence per
> > transaction")
> > Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
> > Signed-off-by: Christian König <christian.koenig@amd.com>
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> >
> > And I can confirm after reverting e356d321d024 I played the whole day,
> > and the "MES failed to respond" error message does not appear anymore.
> >
> > My hardware specs are: https://linux-hardware.org/?probe=78d8c680db
> >
> > Christian, can you look into it, please?
> >
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 6.10/bisected/regression - Since commit e356d321d024 in the kernel log appears the message "MES failed to respond to msg=MISC (WAIT_REG_MEM)" which were never seen before
2024-07-22 21:33 ` Alex Deucher
@ 2024-07-24 17:16 ` Mikhail Gavrilov
2024-08-02 21:29 ` Mikhail Gavrilov
0 siblings, 1 reply; 5+ messages in thread
From: Mikhail Gavrilov @ 2024-07-24 17:16 UTC (permalink / raw)
To: Alex Deucher
Cc: Christian König, amd-gfx list, dri-devel,
Christian König, Deucher, Alexander, mukul.joshi,
Linux List Kernel Mailing, Linux regressions mailing list
[-- Attachment #1: Type: text/plain, Size: 805 bytes --]
On Tue, Jul 23, 2024 at 2:34 AM Alex Deucher <alexdeucher@gmail.com> wrote:
> Do either of these patches help?
> https://patchwork.freedesktop.org/patch/605437/
Unfortunately, this patch didn't help. Please see the attached kernel log.
> https://patchwork.freedesktop.org/patch/605201/
For which kernel is this patch intended? The patch is not applied on
top of d67978318827.
mikhail@primary-ws ~/p/g/linux-3 (master)> git reset d67978318827 --hard
HEAD is now at d67978318827 Merge tag 'x86_cpu_for_v6.11_rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
mikhail@primary-ws ~/p/g/linux-3 (master)> git apply
drm-amdgpu-mes-fix-mes-ring-buffer-overflow.patch
error: drivers/gpu/drm/amd/amdgpu/mes_v12_0.c: No such file or directory
--
Best Regards,
Mike Gavrilov.
[-- Attachment #2: 6.10.0-d67978318827-revert-c1385c1f0ba3-set-sched_hw_submission.zip --]
[-- Type: application/zip, Size: 53553 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 6.10/bisected/regression - Since commit e356d321d024 in the kernel log appears the message "MES failed to respond to msg=MISC (WAIT_REG_MEM)" which were never seen before
2024-07-24 17:16 ` Mikhail Gavrilov
@ 2024-08-02 21:29 ` Mikhail Gavrilov
0 siblings, 0 replies; 5+ messages in thread
From: Mikhail Gavrilov @ 2024-08-02 21:29 UTC (permalink / raw)
To: Alex Deucher
Cc: Christian König, amd-gfx list, dri-devel,
Christian König, Deucher, Alexander, mukul.joshi,
Linux List Kernel Mailing, Linux regressions mailing list
On Wed, Jul 24, 2024 at 10:16 PM Mikhail Gavrilov
<mikhail.v.gavrilov@gmail.com> wrote:
> > https://patchwork.freedesktop.org/patch/605201/
> For which kernel is this patch intended? The patch is not applied on
> top of d67978318827.
I am able to apply this patch on top of e4fc196f5ba3 and the issue is gone.
Tested-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
--
Best Regards,
Mike Gavrilov.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-08-02 21:29 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-07-20 17:08 6.10/bisected/regression - Since commit e356d321d024 in the kernel log appears the message "MES failed to respond to msg=MISC (WAIT_REG_MEM)" which were never seen before Mikhail Gavrilov
2024-07-22 8:43 ` Christian König
2024-07-22 21:33 ` Alex Deucher
2024-07-24 17:16 ` Mikhail Gavrilov
2024-08-02 21:29 ` Mikhail Gavrilov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®