hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Thomas Tai]

Hi Michael,

We see an issue with the mainline kernel on the Azure Gen 2 VM when trying to induce a kernel panic with sysrq commands. The VM would hang with soft lockup. A similar issue happens when executing kexec on the VM. This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios boot are fine.

git bisect identifies the issue is cased by the commit 20ee2ae8c5899 ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ). However, reverting the commit would cause the frame buffer not to work on the Gen2 VM. 

Do you have any hints on what caused this issue?

To reproduce the issue with kdump:
- Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
- echo 1 > /proc/sys/kernel/sysrq
- echo c > /proc/sysrq-trigger

To reproduce the issue with executing kexec: 
- Install mainline kernel on Azure Gen 2 VM and use kexec
- sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-line="$( cat /proc/cmdline )"
- sudo kexec -e

Thank you,
Thomas

--- soft lockup log---
[    1.690032] efifb: probing for efifb
[    1.693989] efifb: framebuffer at 0x40000000, using 3072k, total 3072k
[    1.700996] efifb: mode is 1024x768x32, linelength=4096, pages=1
[    1.706999] efifb: scrolling: redraw
[    1.710981] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.716806] Console: switching to colour frame buffer device 128x48
[   29.176649] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [swapper/0:1]
[   29.176655] Modules linked in:
[   29.176658] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.13.0+ #1
[   29.176661] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
[   29.176663] RIP: 0010:fast_imageblit.isra.0+0x2de/0x460
[   29.176669] Code: 07 49 21 f1 41 83 e1 01 46 8b 4c 8c 40 44 89 09 41 89 d1 4c 8d 79 08 41 c0 e9 06 49 21 f1 41 83 e1 03 46 8b 4c 8c 40 45 89 0e <41> 89 d1 4c 8d 71 0c 41 c0 e9 05 49 21 f1 41 83 e1 07 46 8b 4c 8c
[   29.176671] RSP: 0018:ffffc900000437a8 EFLAGS: 00010246
[   29.176673] RAX: 0000000000000400 RBX: 0000000000000005 RCX: ffffc900006bb140
[   29.176674] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888012146600
[   29.176675] RBP: 0000000000000080 R08: ffffc900006bb000 R09: 0000000000000000
[   29.176676] R10: 0000000000000001 R11: ffff888012146580 R12: 0000000000001000
[   29.176677] R13: ffff88801214658a R14: ffffc900006bb144 R15: ffffc900006bb148
[   29.176678] FS:  0000000000000000(0000) GS:ffff888026a00000(0000) knlGS:0000000000000000
[   29.176680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   29.176681] CR2: 00007f8bc160c238 CR3: 000000002ac38002 CR4: 00000000003706b0
[   29.176683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   29.176684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   29.176685] Call Trace:
[   29.176687]  <IRQ>
[   29.176690]  ? watchdog_timer_fn+0x220/0x2a0
[   29.176695]  ? __pfx_watchdog_timer_fn+0x10/0x10
[   29.176697]  ? __hrtimer_run_queues+0x112/0x2b0
[   29.176702]  ? hrtimer_interrupt+0x108/0x270
[   29.176704]  ? sched_clock_cpu+0x60/0x1a0
[   29.176708]  ? __sysvec_hyperv_stimer0+0x32/0x60
[   29.176712]  ? sysvec_hyperv_stimer0+0x70/0x90
[   29.176715]  </IRQ>
[   29.176716]  <TASK>
[   29.176716]  ? asm_sysvec_hyperv_stimer0+0x1a/0x20
[   29.176722]  ? fast_imageblit.isra.0+0x2de/0x460
[   29.176724]  cfb_imageblit+0x433/0x470
[   29.176726]  bit_putcs+0x291/0x570
[   29.176731]  ? __pfx_bit_putcs+0x10/0x10
[   29.176733]  fbcon_putcs+0x139/0x1a0
[   29.176735]  do_update_region+0xf1/0x110
[   29.176740]  redraw_screen+0x22f/0x290
[   29.176743]  do_bind_con_driver.isra.0+0x2ab/0x3d0
[   29.176745]  do_take_over_console+0x3a/0x50
[   29.176747]  do_fbcon_takeover+0x5c/0xe0
[   29.176749]  fbcon_fb_registered+0x4f/0x70
[   29.176751]  do_register_framebuffer+0x1bc/0x2a0
[   29.176755]  devm_register_framebuffer+0x28/0x90
[   29.176757]  efifb_probe+0x544/0x720
[   29.176760]  platform_probe+0x43/0xb0
[   29.176763]  really_probe+0xd9/0x390
[   29.176767]  ? __pfx___device_attach_driver+0x10/0x10
[   29.176769]  __driver_probe_device+0x78/0x160
[   29.176771]  driver_probe_device+0x1e/0xa0
[   29.176773]  __device_attach_driver+0x99/0x130
[   29.176775]  bus_for_each_drv+0x98/0xf0
[   29.176777]  __device_attach+0xbc/0x1f0
[   29.176779]  bus_probe_device+0x8d/0xb0
[   29.176781]  device_add+0x4f1/0x6e0
[   29.176785]  platform_device_add+0xfa/0x260
[   29.176787]  sysfb_init+0x109/0x120
[   29.176791]  ? __pfx_sysfb_init+0x10/0x10
[   29.176793]  do_one_initcall+0x5b/0x330
[   29.176796]  do_initcalls+0xac/0x130
[   29.176800]  kernel_init_freeable+0x134/0x1e0
[   29.176802]  ? __pfx_kernel_init+0x10/0x10
[   29.176806]  kernel_init+0x1a/0x1d0
[   29.176808]  ret_from_fork+0x34/0x50
[   29.176813]  ? __pfx_kernel_init+0x10/0x10
[   29.176815]  ret_from_fork_asm+0x1a/0x30
[   29.176819]  </TASK>
[  432.306986] fb0: EFI VGA frame buffer device

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Thomas Tai]

> -----Original Message-----
> From: Thomas Tai <thomas.tai@oracle.com>
> Sent: Thursday, January 30, 2025 1:37 PM
> To: mhkelley58@gmail.com; haiyangz@microsoft.com; wei.liu@kernel.org;
> decui@microsoft.com; drawat.floss@gmail.com; javierm@redhat.com; Helge
> Deller <deller@gmx.de>; daniel@ffwll.ch; airlied@gmail.com;
> tzimmermann@suse.de
> Cc: dri-devel@lists.freedesktop.org; linux-fbdev@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-hyperv@vger.kernel.org; Thomas Tai
> <thomas.tai@oracle.com>
> Subject: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
> executing kexec

Sorry for the typo in the subject title. It should have been 'hyperv_fb soft lockup on Azure Gen2 VM when taking kdump or executing kexec'

Thomas

> 
> Hi Michael,
> 
> We see an issue with the mainline kernel on the Azure Gen 2 VM when
> trying to induce a kernel panic with sysrq commands. The VM would hang
> with soft lockup. A similar issue happens when executing kexec on the VM.
> This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios
> boot are fine.
> 
> git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> However, reverting the commit would cause the frame buffer not to work
> on the Gen2 VM.
> 
> Do you have any hints on what caused this issue?
> 
> To reproduce the issue with kdump:
> - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> - echo 1 > /proc/sys/kernel/sysrq
> - echo c > /proc/sysrq-trigger
> 
> To reproduce the issue with executing kexec:
> - Install mainline kernel on Azure Gen 2 VM and use kexec
> - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> line="$( cat /proc/cmdline )"
> - sudo kexec -e
> 
> Thank you,
> Thomas
> 
> --- soft lockup log---
> [    1.690032] efifb: probing for efifb
> [    1.693989] efifb: framebuffer at 0x40000000, using 3072k, total 3072k
> [    1.700996] efifb: mode is 1024x768x32, linelength=4096, pages=1
> [    1.706999] efifb: scrolling: redraw
> [    1.710981] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
> [    1.716806] Console: switching to colour frame buffer device 128x48
> [   29.176649] watchdog: BUG: soft lockup - CPU#0 stuck for 26s!
> [swapper/0:1]
> [   29.176655] Modules linked in:
> [   29.176658] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.13.0+ #1
> [   29.176661] Hardware name: Microsoft Corporation Virtual
> Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
> [   29.176663] RIP: 0010:fast_imageblit.isra.0+0x2de/0x460
> [   29.176669] Code: 07 49 21 f1 41 83 e1 01 46 8b 4c 8c 40 44 89 09 41 89 d1
> 4c 8d 79 08 41 c0 e9 06 49 21 f1 41 83 e1 03 46 8b 4c 8c 40 45 89 0e <41> 89
> d1 4c 8d 71 0c 41 c0 e9 05 49 21 f1 41 83 e1 07 46 8b 4c 8c
> [   29.176671] RSP: 0018:ffffc900000437a8 EFLAGS: 00010246
> [   29.176673] RAX: 0000000000000400 RBX: 0000000000000005 RCX:
> ffffc900006bb140
> [   29.176674] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> ffff888012146600
> [   29.176675] RBP: 0000000000000080 R08: ffffc900006bb000 R09:
> 0000000000000000
> [   29.176676] R10: 0000000000000001 R11: ffff888012146580 R12:
> 0000000000001000
> [   29.176677] R13: ffff88801214658a R14: ffffc900006bb144 R15:
> ffffc900006bb148
> [   29.176678] FS:  0000000000000000(0000) GS:ffff888026a00000(0000)
> knlGS:0000000000000000
> [   29.176680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   29.176681] CR2: 00007f8bc160c238 CR3: 000000002ac38002 CR4:
> 00000000003706b0
> [   29.176683] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   29.176684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   29.176685] Call Trace:
> [   29.176687]  <IRQ>
> [   29.176690]  ? watchdog_timer_fn+0x220/0x2a0
> [   29.176695]  ? __pfx_watchdog_timer_fn+0x10/0x10
> [   29.176697]  ? __hrtimer_run_queues+0x112/0x2b0
> [   29.176702]  ? hrtimer_interrupt+0x108/0x270
> [   29.176704]  ? sched_clock_cpu+0x60/0x1a0
> [   29.176708]  ? __sysvec_hyperv_stimer0+0x32/0x60
> [   29.176712]  ? sysvec_hyperv_stimer0+0x70/0x90
> [   29.176715]  </IRQ>
> [   29.176716]  <TASK>
> [   29.176716]  ? asm_sysvec_hyperv_stimer0+0x1a/0x20
> [   29.176722]  ? fast_imageblit.isra.0+0x2de/0x460
> [   29.176724]  cfb_imageblit+0x433/0x470
> [   29.176726]  bit_putcs+0x291/0x570
> [   29.176731]  ? __pfx_bit_putcs+0x10/0x10
> [   29.176733]  fbcon_putcs+0x139/0x1a0
> [   29.176735]  do_update_region+0xf1/0x110
> [   29.176740]  redraw_screen+0x22f/0x290
> [   29.176743]  do_bind_con_driver.isra.0+0x2ab/0x3d0
> [   29.176745]  do_take_over_console+0x3a/0x50
> [   29.176747]  do_fbcon_takeover+0x5c/0xe0
> [   29.176749]  fbcon_fb_registered+0x4f/0x70
> [   29.176751]  do_register_framebuffer+0x1bc/0x2a0
> [   29.176755]  devm_register_framebuffer+0x28/0x90
> [   29.176757]  efifb_probe+0x544/0x720
> [   29.176760]  platform_probe+0x43/0xb0
> [   29.176763]  really_probe+0xd9/0x390
> [   29.176767]  ? __pfx___device_attach_driver+0x10/0x10
> [   29.176769]  __driver_probe_device+0x78/0x160
> [   29.176771]  driver_probe_device+0x1e/0xa0
> [   29.176773]  __device_attach_driver+0x99/0x130
> [   29.176775]  bus_for_each_drv+0x98/0xf0
> [   29.176777]  __device_attach+0xbc/0x1f0
> [   29.176779]  bus_probe_device+0x8d/0xb0
> [   29.176781]  device_add+0x4f1/0x6e0
> [   29.176785]  platform_device_add+0xfa/0x260
> [   29.176787]  sysfb_init+0x109/0x120
> [   29.176791]  ? __pfx_sysfb_init+0x10/0x10
> [   29.176793]  do_one_initcall+0x5b/0x330
> [   29.176796]  do_initcalls+0xac/0x130
> [   29.176800]  kernel_init_freeable+0x134/0x1e0
> [   29.176802]  ? __pfx_kernel_init+0x10/0x10
> [   29.176806]  kernel_init+0x1a/0x1d0
> [   29.176808]  ret_from_fork+0x34/0x50
> [   29.176813]  ? __pfx_kernel_init+0x10/0x10
> [   29.176815]  ret_from_fork_asm+0x1a/0x30
> [   29.176819]  </TASK>
> [  432.306986] fb0: EFI VGA frame buffer device

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30, 2025 10:50 AM
> 
> Sorry for the typo in the subject title. It should have been 'hyperv_fb soft lockup on
> Azure Gen2 VM when taking kdump or executing kexec'
> 
> Thomas
> 
> >
> > Hi Michael,
> >
> > We see an issue with the mainline kernel on the Azure Gen 2 VM when
> > trying to induce a kernel panic with sysrq commands. The VM would hang
> > with soft lockup. A similar issue happens when executing kexec on the VM.
> > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios
> > boot are fine.
> >
> > git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > However, reverting the commit would cause the frame buffer not to work
> > on the Gen2 VM.
> >
> > Do you have any hints on what caused this issue?
> >
> > To reproduce the issue with kdump:
> > - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> > - echo 1 > /proc/sys/kernel/sysrq
> > - echo c > /proc/sysrq-trigger
> >
> > To reproduce the issue with executing kexec:
> > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> > line="$( cat /proc/cmdline )"
> > - sudo kexec -e
> >
> > Thank you,
> > Thomas

I will take a look, but it might be early next week before I can do so.

It looks like your soft lockup log below is from the kdump kernel (or the newly
kexec'ed kernel). Can you confirm? Also, this looks like a subset of the full log.
Do you have the full serial console log that you could email to me?  Seeing
everything might be helpful. Of course, I'll try to repro the problem myself as
well.

Thanks,

Michael

> >
> > --- soft lockup log---
> > [    1.690032] efifb: probing for efifb
> > [    1.693989] efifb: framebuffer at 0x40000000, using 3072k, total 3072k
> > [    1.700996] efifb: mode is 1024x768x32, linelength=4096, pages=1
> > [    1.706999] efifb: scrolling: redraw
> > [    1.710981] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
> > [    1.716806] Console: switching to colour frame buffer device 128x48
> > [   29.176649] watchdog: BUG: soft lockup - CPU#0 stuck for 26s!
> > [swapper/0:1]
> > [   29.176655] Modules linked in:
> > [   29.176658] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.13.0+ #1
> > [   29.176661] Hardware name: Microsoft Corporation Virtual
> > Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
> > [   29.176663] RIP: 0010:fast_imageblit.isra.0+0x2de/0x460
> > [   29.176669] Code: 07 49 21 f1 41 83 e1 01 46 8b 4c 8c 40 44 89 09 41 89 d1
> > 4c 8d 79 08 41 c0 e9 06 49 21 f1 41 83 e1 03 46 8b 4c 8c 40 45 89 0e <41> 89
> > d1 4c 8d 71 0c 41 c0 e9 05 49 21 f1 41 83 e1 07 46 8b 4c 8c
> > [   29.176671] RSP: 0018:ffffc900000437a8 EFLAGS: 00010246
> > [   29.176673] RAX: 0000000000000400 RBX: 0000000000000005 RCX:
> > ffffc900006bb140
> > [   29.176674] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> > ffff888012146600
> > [   29.176675] RBP: 0000000000000080 R08: ffffc900006bb000 R09:
> > 0000000000000000
> > [   29.176676] R10: 0000000000000001 R11: ffff888012146580 R12:
> > 0000000000001000
> > [   29.176677] R13: ffff88801214658a R14: ffffc900006bb144 R15:
> > ffffc900006bb148
> > [   29.176678] FS:  0000000000000000(0000) GS:ffff888026a00000(0000)
> > knlGS:0000000000000000
> > [   29.176680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   29.176681] CR2: 00007f8bc160c238 CR3: 000000002ac38002 CR4:
> > 00000000003706b0
> > [   29.176683] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   29.176684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   29.176685] Call Trace:
> > [   29.176687]  <IRQ>
> > [   29.176690]  ? watchdog_timer_fn+0x220/0x2a0
> > [   29.176695]  ? __pfx_watchdog_timer_fn+0x10/0x10
> > [   29.176697]  ? __hrtimer_run_queues+0x112/0x2b0
> > [   29.176702]  ? hrtimer_interrupt+0x108/0x270
> > [   29.176704]  ? sched_clock_cpu+0x60/0x1a0
> > [   29.176708]  ? __sysvec_hyperv_stimer0+0x32/0x60
> > [   29.176712]  ? sysvec_hyperv_stimer0+0x70/0x90
> > [   29.176715]  </IRQ>
> > [   29.176716]  <TASK>
> > [   29.176716]  ? asm_sysvec_hyperv_stimer0+0x1a/0x20
> > [   29.176722]  ? fast_imageblit.isra.0+0x2de/0x460
> > [   29.176724]  cfb_imageblit+0x433/0x470
> > [   29.176726]  bit_putcs+0x291/0x570
> > [   29.176731]  ? __pfx_bit_putcs+0x10/0x10
> > [   29.176733]  fbcon_putcs+0x139/0x1a0
> > [   29.176735]  do_update_region+0xf1/0x110
> > [   29.176740]  redraw_screen+0x22f/0x290
> > [   29.176743]  do_bind_con_driver.isra.0+0x2ab/0x3d0
> > [   29.176745]  do_take_over_console+0x3a/0x50
> > [   29.176747]  do_fbcon_takeover+0x5c/0xe0
> > [   29.176749]  fbcon_fb_registered+0x4f/0x70
> > [   29.176751]  do_register_framebuffer+0x1bc/0x2a0
> > [   29.176755]  devm_register_framebuffer+0x28/0x90
> > [   29.176757]  efifb_probe+0x544/0x720
> > [   29.176760]  platform_probe+0x43/0xb0
> > [   29.176763]  really_probe+0xd9/0x390
> > [   29.176767]  ? __pfx___device_attach_driver+0x10/0x10
> > [   29.176769]  __driver_probe_device+0x78/0x160
> > [   29.176771]  driver_probe_device+0x1e/0xa0
> > [   29.176773]  __device_attach_driver+0x99/0x130
> > [   29.176775]  bus_for_each_drv+0x98/0xf0
> > [   29.176777]  __device_attach+0xbc/0x1f0
> > [   29.176779]  bus_probe_device+0x8d/0xb0
> > [   29.176781]  device_add+0x4f1/0x6e0
> > [   29.176785]  platform_device_add+0xfa/0x260
> > [   29.176787]  sysfb_init+0x109/0x120
> > [   29.176791]  ? __pfx_sysfb_init+0x10/0x10
> > [   29.176793]  do_one_initcall+0x5b/0x330
> > [   29.176796]  do_initcalls+0xac/0x130
> > [   29.176800]  kernel_init_freeable+0x134/0x1e0
> > [   29.176802]  ? __pfx_kernel_init+0x10/0x10
> > [   29.176806]  kernel_init+0x1a/0x1d0
> > [   29.176808]  ret_from_fork+0x34/0x50
> > [   29.176813]  ? __pfx_kernel_init+0x10/0x10
> > [   29.176815]  ret_from_fork_asm+0x1a/0x30
> > [   29.176819]  </TASK>
> > [  432.306986] fb0: EFI VGA frame buffer device

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Thomas Tai]

[-- Attachment #1: Type: text/plain, Size: 7469 bytes --]

> -----Original Message-----
> From: Michael Kelley <mhklinux@outlook.com>
> Sent: Thursday, January 30, 2025 3:20 PM
> To: Thomas Tai <thomas.tai@oracle.com>; mhkelley58@gmail.com;
> haiyangz@microsoft.com; wei.liu@kernel.org; decui@microsoft.com;
> drawat.floss@gmail.com; javierm@redhat.com; Helge Deller
> <deller@gmx.de>; daniel@ffwll.ch; airlied@gmail.com;
> tzimmermann@suse.de
> Cc: dri-devel@lists.freedesktop.org; linux-fbdev@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-hyperv@vger.kernel.org
> Subject: RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
> executing kexec
> 
> From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30,
> 2025 10:50 AM
> >
> > Sorry for the typo in the subject title. It should have been 'hyperv_fb soft
> lockup on
> > Azure Gen2 VM when taking kdump or executing kexec'
> >
> > Thomas
> >
> > >
> > > Hi Michael,
> > >
> > > We see an issue with the mainline kernel on the Azure Gen 2 VM when
> > > trying to induce a kernel panic with sysrq commands. The VM would
> hang
> > > with soft lockup. A similar issue happens when executing kexec on the
> VM.
> > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with
> bios
> > > boot are fine.
> > >
> > > git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > However, reverting the commit would cause the frame buffer not to
> work
> > > on the Gen2 VM.
> > >
> > > Do you have any hints on what caused this issue?
> > >
> > > To reproduce the issue with kdump:
> > > - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> > > - echo 1 > /proc/sys/kernel/sysrq
> > > - echo c > /proc/sysrq-trigger
> > >
> > > To reproduce the issue with executing kexec:
> > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> > > line="$( cat /proc/cmdline )"
> > > - sudo kexec -e
> > >
> > > Thank you,
> > > Thomas
> 
> I will take a look, but it might be early next week before I can do so.
> 

Thank you, Michael for your help!

> It looks like your soft lockup log below is from the kdump kernel (or the
> newly
> kexec'ed kernel). Can you confirm? Also, this looks like a subset of the full
> log.

Yes, the soft lockup log below is from the kdump kernel. 

> Do you have the full serial console log that you could email to me?  Seeing
> everything might be helpful. Of course, I'll try to repro the problem myself
> as
> well.

I have attached the complete bootup and kdump kernel log.

File: bootup_and_kdump.log
Line 1 ... 984 (bootup log)
Line 990       (kdump kernel booting up)
Line 1351      (soft lockup)  

Thank you,
Thomas

> 
> Thanks,
> 
> Michael
> 
> > >
> > > --- soft lockup log---
> > > [    1.690032] efifb: probing for efifb
> > > [    1.693989] efifb: framebuffer at 0x40000000, using 3072k, total 3072k
> > > [    1.700996] efifb: mode is 1024x768x32, linelength=4096, pages=1
> > > [    1.706999] efifb: scrolling: redraw
> > > [    1.710981] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
> > > [    1.716806] Console: switching to colour frame buffer device 128x48
> > > [   29.176649] watchdog: BUG: soft lockup - CPU#0 stuck for 26s!
> > > [swapper/0:1]
> > > [   29.176655] Modules linked in:
> > > [   29.176658] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.13.0+
> #1
> > > [   29.176661] Hardware name: Microsoft Corporation Virtual
> > > Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
> > > [   29.176663] RIP: 0010:fast_imageblit.isra.0+0x2de/0x460
> > > [   29.176669] Code: 07 49 21 f1 41 83 e1 01 46 8b 4c 8c 40 44 89 09 41 89
> d1
> > > 4c 8d 79 08 41 c0 e9 06 49 21 f1 41 83 e1 03 46 8b 4c 8c 40 45 89 0e <41>
> 89
> > > d1 4c 8d 71 0c 41 c0 e9 05 49 21 f1 41 83 e1 07 46 8b 4c 8c
> > > [   29.176671] RSP: 0018:ffffc900000437a8 EFLAGS: 00010246
> > > [   29.176673] RAX: 0000000000000400 RBX: 0000000000000005 RCX:
> > > ffffc900006bb140
> > > [   29.176674] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> > > ffff888012146600
> > > [   29.176675] RBP: 0000000000000080 R08: ffffc900006bb000 R09:
> > > 0000000000000000
> > > [   29.176676] R10: 0000000000000001 R11: ffff888012146580 R12:
> > > 0000000000001000
> > > [   29.176677] R13: ffff88801214658a R14: ffffc900006bb144 R15:
> > > ffffc900006bb148
> > > [   29.176678] FS:  0000000000000000(0000) GS:ffff888026a00000(0000)
> > > knlGS:0000000000000000
> > > [   29.176680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   29.176681] CR2: 00007f8bc160c238 CR3: 000000002ac38002 CR4:
> > > 00000000003706b0
> > > [   29.176683] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   29.176684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   29.176685] Call Trace:
> > > [   29.176687]  <IRQ>
> > > [   29.176690]  ? watchdog_timer_fn+0x220/0x2a0
> > > [   29.176695]  ? __pfx_watchdog_timer_fn+0x10/0x10
> > > [   29.176697]  ? __hrtimer_run_queues+0x112/0x2b0
> > > [   29.176702]  ? hrtimer_interrupt+0x108/0x270
> > > [   29.176704]  ? sched_clock_cpu+0x60/0x1a0
> > > [   29.176708]  ? __sysvec_hyperv_stimer0+0x32/0x60
> > > [   29.176712]  ? sysvec_hyperv_stimer0+0x70/0x90
> > > [   29.176715]  </IRQ>
> > > [   29.176716]  <TASK>
> > > [   29.176716]  ? asm_sysvec_hyperv_stimer0+0x1a/0x20
> > > [   29.176722]  ? fast_imageblit.isra.0+0x2de/0x460
> > > [   29.176724]  cfb_imageblit+0x433/0x470
> > > [   29.176726]  bit_putcs+0x291/0x570
> > > [   29.176731]  ? __pfx_bit_putcs+0x10/0x10
> > > [   29.176733]  fbcon_putcs+0x139/0x1a0
> > > [   29.176735]  do_update_region+0xf1/0x110
> > > [   29.176740]  redraw_screen+0x22f/0x290
> > > [   29.176743]  do_bind_con_driver.isra.0+0x2ab/0x3d0
> > > [   29.176745]  do_take_over_console+0x3a/0x50
> > > [   29.176747]  do_fbcon_takeover+0x5c/0xe0
> > > [   29.176749]  fbcon_fb_registered+0x4f/0x70
> > > [   29.176751]  do_register_framebuffer+0x1bc/0x2a0
> > > [   29.176755]  devm_register_framebuffer+0x28/0x90
> > > [   29.176757]  efifb_probe+0x544/0x720
> > > [   29.176760]  platform_probe+0x43/0xb0
> > > [   29.176763]  really_probe+0xd9/0x390
> > > [   29.176767]  ? __pfx___device_attach_driver+0x10/0x10
> > > [   29.176769]  __driver_probe_device+0x78/0x160
> > > [   29.176771]  driver_probe_device+0x1e/0xa0
> > > [   29.176773]  __device_attach_driver+0x99/0x130
> > > [   29.176775]  bus_for_each_drv+0x98/0xf0
> > > [   29.176777]  __device_attach+0xbc/0x1f0
> > > [   29.176779]  bus_probe_device+0x8d/0xb0
> > > [   29.176781]  device_add+0x4f1/0x6e0
> > > [   29.176785]  platform_device_add+0xfa/0x260
> > > [   29.176787]  sysfb_init+0x109/0x120
> > > [   29.176791]  ? __pfx_sysfb_init+0x10/0x10
> > > [   29.176793]  do_one_initcall+0x5b/0x330
> > > [   29.176796]  do_initcalls+0xac/0x130
> > > [   29.176800]  kernel_init_freeable+0x134/0x1e0
> > > [   29.176802]  ? __pfx_kernel_init+0x10/0x10
> > > [   29.176806]  kernel_init+0x1a/0x1d0
> > > [   29.176808]  ret_from_fork+0x34/0x50
> > > [   29.176813]  ? __pfx_kernel_init+0x10/0x10
> > > [   29.176815]  ret_from_fork_asm+0x1a/0x30
> > > [   29.176819]  </TASK>
> > > [  432.306986] fb0: EFI VGA frame buffer device


[-- Attachment #2: bootup_and_kdump.log --]
[-- Type: application/octet-stream, Size: 182487 bytes --]

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30, 2025 12:44 PM
> 
> > -----Original Message-----
> > From: Michael Kelley <mhklinux@outlook.com>
> > Sent: Thursday, January 30, 2025 3:20 PM
> > To: Thomas Tai <thomas.tai@oracle.com>; mhkelley58@gmail.com;
> > haiyangz@microsoft.com; wei.liu@kernel.org; decui@microsoft.com;
> > drawat.floss@gmail.com; javierm@redhat.com; Helge Deller
> > <deller@gmx.de>; daniel@ffwll.ch; airlied@gmail.com;
> > tzimmermann@suse.de
> > Cc: dri-devel@lists.freedesktop.org; linux-fbdev@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-hyperv@vger.kernel.org
> > Subject: RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
> > executing kexec
> >
> > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30,
> > 2025 10:50 AM
> > >
> > > Sorry for the typo in the subject title. It should have been 'hyperv_fb soft lockup on
> > > Azure Gen2 VM when taking kdump or executing kexec'
> > >
> > > Thomas
> > >
> > > >
> > > > Hi Michael,
> > > >
> > > > We see an issue with the mainline kernel on the Azure Gen 2 VM when
> > > > trying to induce a kernel panic with sysrq commands. The VM would hang
> > > > with soft lockup. A similar issue happens when executing kexec on the VM.
> > > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios
> > > > boot are fine.
> > > >
> > > > git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> > > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > > However, reverting the commit would cause the frame buffer not to work
> > > > on the Gen2 VM.
> > > >
> > > > Do you have any hints on what caused this issue?
> > > >
> > > > To reproduce the issue with kdump:
> > > > - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> > > > - echo 1 > /proc/sys/kernel/sysrq
> > > > - echo c > /proc/sysrq-trigger
> > > >
> > > > To reproduce the issue with executing kexec:
> > > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> > > > line="$( cat /proc/cmdline )"
> > > > - sudo kexec -e
> > > >
> > > > Thank you,
> > > > Thomas
> >
> > I will take a look, but it might be early next week before I can do so.
> >
> 
> Thank you, Michael for your help!
> 
> > It looks like your soft lockup log below is from the kdump kernel (or the newly
> > kexec'ed kernel). Can you confirm? Also, this looks like a subset of the full log.
> 
> Yes, the soft lockup log below is from the kdump kernel.
> 
> > Do you have the full serial console log that you could email to me?  Seeing
> > everything might be helpful. Of course, I'll try to repro the problem myself
> > as well.
> 
> I have attached the complete bootup and kdump kernel log.
> 
> File: bootup_and_kdump.log
> Line 1 ... 984 (bootup log)
> Line 990       (kdump kernel booting up)
> Line 1351      (soft lockup)
> 
> Thank you,
> Thomas
> 

I have reproduced the problem in an Azure VM running Oracle Linux
9.4 with the 6.13.0 kernel. Interestingly, the problem does not occur
in a VM running on a locally installed Hyper-V with Ubuntu 20.04 and
the 6.13.0 kernel. There are several differences in the two
environments:  the version of Hyper-V, the VM configuration, the Linux
distro, and the .config file used to build the 6.13.0 kernel. I'll try to
figure out what make the difference, and then the root cause.

Michael

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: Michael Kelley <mhklinux@outlook.com>
> 
> From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30, 2025 12:44 PM
> >
> > > -----Original Message-----
> > > From: Michael Kelley <mhklinux@outlook.com> Sent: Thursday, January 30, 2025 3:20 PM
> > >
> > > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30,
> > > 2025 10:50 AM
> > > >
> > > > Sorry for the typo in the subject title. It should have been 'hyperv_fb soft lockup on
> > > > Azure Gen2 VM when taking kdump or executing kexec'
> > > >
> > > > Thomas
> > > >
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > > We see an issue with the mainline kernel on the Azure Gen 2 VM when
> > > > > trying to induce a kernel panic with sysrq commands. The VM would hang
> > > > > with soft lockup. A similar issue happens when executing kexec on the VM.
> > > > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios
> > > > > boot are fine.
> > > > >
> > > > > git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> > > > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > > > However, reverting the commit would cause the frame buffer not to work
> > > > > on the Gen2 VM.
> > > > >
> > > > > Do you have any hints on what caused this issue?
> > > > >
> > > > > To reproduce the issue with kdump:
> > > > > - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> > > > > - echo 1 > /proc/sys/kernel/sysrq
> > > > > - echo c > /proc/sysrq-trigger
> > > > >
> > > > > To reproduce the issue with executing kexec:
> > > > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> > > > > line="$( cat /proc/cmdline )"
> > > > > - sudo kexec -e
> > > > >
> > > > > Thank you,
> > > > > Thomas
> > >
> > > I will take a look, but it might be early next week before I can do so.
> > >
> >
> > Thank you, Michael for your help!
> >
> > > It looks like your soft lockup log below is from the kdump kernel (or the newly
> > > kexec'ed kernel). Can you confirm? Also, this looks like a subset of the full log.
> >
> > Yes, the soft lockup log below is from the kdump kernel.
> >
> > > Do you have the full serial console log that you could email to me?  Seeing
> > > everything might be helpful. Of course, I'll try to repro the problem myself
> > > as well.
> >
> > I have attached the complete bootup and kdump kernel log.
> >
> > File: bootup_and_kdump.log
> > Line 1 ... 984 (bootup log)
> > Line 990       (kdump kernel booting up)
> > Line 1351      (soft lockup)
> >
> > Thank you,
> > Thomas
> >
> 
> I have reproduced the problem in an Azure VM running Oracle Linux
> 9.4 with the 6.13.0 kernel. Interestingly, the problem does not occur
> in a VM running on a locally installed Hyper-V with Ubuntu 20.04 and
> the 6.13.0 kernel. There are several differences in the two
> environments:  the version of Hyper-V, the VM configuration, the Linux
> distro, and the .config file used to build the 6.13.0 kernel. I'll try to
> figure out what make the difference, and then the root cause.
> 

This has been a real bear to investigate. :-(  The key observation
is that with older kernel versions, the efifb driver does *not* try
to load when running in the kdump kernel, and everything works.
In newer kernels, the efifb driver *does* try to load, and it appears
to hang.  (Actually, it is causing the VM to run very slowly. More on
that in a minute.)

I've bisected the kernel again, compensating for the fact that commit
20ee2ae8c5899 is needed to make the Hyper-V frame buffer work. With
that compensation, the actual problematic commit is 2bebc3cd4870
(Revert "firmware/sysfb: Clear screen_info state after consuming it").
Doing the revert causes screen_info.orig_video_isVGA to retain its value
of 0x70 (VIDEO_TYPE_EFI), which the kdump kernel picks up, causing it
to load the efifb driver.

Then the question is why the efifb driver doesn't work in the kdump
kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
on the Oracle Linux 9.4 system, and transferred the kernel image binary
and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
efifb driver is loaded as part of the kdump kernel, and it doesn't cause
any problems. But there's an interesting difference. In the Oracle Linux
9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
difference is due to differences in how the screen_info variable gets
setup in the two VMs.

When the normal kernel starts in a freshly booted VM, Hyper-V provides
the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
FB driver or Hyper-V DRM driver has initialized, Linux has picked a
different MMIO address range and told Hyper-V to use the new
address range (which often starts at 0x40900000). A kexec does *not*
reset Hyper-V's transition to the new range, so when the efifb driver
tries to use the framebuffer at 0x40000000, the accesses trap to
Hyper-V and probably fail or timeout (I'm not sure of the details). After
the guest does some number of these bad references, Hyper-V considers
itself to be under attack from an ill-behaving guest, and throttles the
guest so that it doesn't run for a few seconds. The throttling repeats,
and results in extremely slow running in the kdump kernel.

Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
as stored in screen_info.lfb_base gets updated to be 0x40900000. I
haven't fully debugged how that happens. But with that update, the
efifb driver is using the updated framebuffer address and it works. On
the Oracle Linux 9.4 system, that update doesn't appear to happen,
and the problem occurs.

This in an interim update on the problem. I'm still investigating how
screen_info.lfb_base is set in the kdump kernel, and why it is different
in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
well understood, we can contemplate how to fix the problem. Undoing
the revert that is commit 2bebc3cd4870 doesn't seem like the solution
since the original code there was reported to cause many other issues.
The solution focus will likely be on how to ensure the kdump kernel gets
the correct framebuffer address so the efifb driver works, since the
framebuffer address changing is a quirk of Hyper-V behavior.

If anyone else has insight into what's going on here, please chime in.
What I've learned so far is still somewhat tentative.

Michael

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Saurabh Singh Sengar]

Thanks Michael, for the analysis. 

I have tried the kdump steps on Oracle 9.4, 6.13.0 kernel as well. Although I couldn't see
the soft lockup issue I see some other VMBus failures. But I agree the bootup is extremely
slow, which should be due to same reason.

My system is having newer UEFI version, wondering if the latest UEFI version
(UEFI Release v4.1 08/23/2024) causing this difference in behaviour.

Relevant part of the logs:
---------------------------------------------------------
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
[  982.948352] sysrq: Trigger a crash
[  982.949553] Kernel panic - not syncing: sysrq triggered crash
[  982.951515] CPU: 31 UID: 0 PID: 6938 Comm: bash Kdump: loaded Not tainted 6.13.0 #1
[  982.954115] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 08/23/2024
[  982.957641] Call Trace:
[  982.958508]  <TASK>
[  982.959251]  panic+0x37e/0x3b0
[  982.960373]  ? _printk+0x64/0x90
[  982.961452]  sysrq_handle_crash+0x1a/0x20
[  982.962840]  __handle_sysrq+0x9b/0x190
[  982.964145]  write_sysrq_trigger+0x5f/0x80
[  982.965578]  proc_reg_write+0x59/0xb0
[  982.966905]  vfs_write+0x111/0x470
[  982.968004]  ? __count_memcg_events+0xbf/0x150
[  982.969432]  ? count_memcg_events.constprop.0+0x26/0x50
[  982.971190]  ksys_write+0x6e/0xf0
[  982.972307]  do_syscall_64+0x62/0x180
[  982.973438]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  982.975102] RIP: 0033:0x7f3d570fdbd7
[  982.976421] Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[  982.982893] RSP: 002b:00007fff6d613c48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  982.985424] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d570fdbd7
[  982.987613] RDX: 0000000000000002 RSI: 000056362a928470 RDI: 0000000000000001
[  982.989774] RBP: 000056362a928470 R08: 0000000000000000 R09: 00007f3d571b0d40
[  982.992109] R10: 00007f3d571b0c40 R11: 0000000000000246 R12: 0000000000000002
[  982.994321] R13: 00007f3d571fa780 R14: 0000000000000002 R15: 00007f3d571f59e0
[  982.996461]  </TASK>
[  982.998317] Kernel Offset: 0x10c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    0.000000] Linux version 6.13.0 (lisatest@lisa--505-e0-n0) (gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2.0.1), GNU ld version 2.35.2-54.0.1.el9) #1 SMP PREEMPT_DYNAMIC Thu Feb  6 10:05:27 UTC 2025
[    0.000000] Command line: elfcorehdr=0xd000000 BOOT_IMAGE=(hd0,gpt1)/vmlinuz-6.13.0 ro console=tty0 console=ttyS0,115200n8 rd.lvm.vg=rootvg irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 acpi_no_memhotplug transparent_hugepage=never nokaslr hest_disable novmcoredd cma=0 hugetlb_cma=0 iommu=off disable_cpu_apicid=0
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000c0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000d0e00b0-0x000000002cffffff] usable
[    0.000000] BIOS-e820: [mem 0x000000003eead000-0x000000003eeb3fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003ff41000-0x000000003ffc8fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003ffc9000-0x000000003fffafff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000003fffb000-0x000000003fffefff] ACPI NVS
[    0.000000] random: crng init done

<snip>

[    0.928063] Console: switching to colour frame buffer device 128x48
[   13.391297] fb0: EFI VGA frame buffer device

<snip>

[  590.199511] hv_netvsc 7c1e527c-2980-7c1e-527c-29807c1e527c (unnamed net_device) (uninitialized): VF slot 1 added
[  595.120270] Console: switching to colour dummy device 80x25
[  605.203700] hyperv_fb: Time out on waiting vram location ack
[  605.206161] iounmap: bad address 0000000005f4dac5
[  605.207740] CPU: 0 UID: 0 PID: 30 Comm: kworker/u4:2 Not tainted 6.13.0 #1
[  605.209984] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 08/23/2024
[  605.213869] Workqueue: async async_run_entry_fn
[  605.215601] Call Trace:
[  605.216382]  <TASK>
[  605.217123]  dump_stack_lvl+0x66/0x90
[  605.218184]  hvfb_putmem+0x32/0x110 [hyperv_fb]
[  605.219646]  hvfb_probe+0x27f/0x360 [hyperv_fb]
[  605.221120]  vmbus_probe+0x3d/0xa0 [hv_vmbus]
[  605.222623]  really_probe+0xd9/0x390
[  605.223779]  __driver_probe_device+0x78/0x160
[  605.225213]  driver_probe_device+0x1e/0xa0
[  605.226591]  __driver_attach_async_helper+0x5e/0xe0
[  605.228166]  async_run_entry_fn+0x34/0x130
[  605.229681]  process_one_work+0x187/0x3b0
[  605.231075]  worker_thread+0x24e/0x360
[  605.232376]  ? __pfx_worker_thread+0x10/0x10
[  605.233758]  kthread+0xd3/0x100
[  605.234805]  ? __pfx_kthread+0x10/0x10
[  605.236053]  ret_from_fork+0x34/0x50
[  605.237251]  ? __pfx_kthread+0x10/0x10
[  605.238519]  ret_from_fork_asm+0x1a/0x30
[  605.239833]  </TASK>
[  605.240855] hv_vmbus: probe failed for device 5620e0c7-8062-4dce-aeb7-520c7ef76171 (-110)
[  605.243404] hyperv_fb 5620e0c7-8062-4dce-aeb7-520c7ef76171: probe with driver hyperv_fb failed with error -110
[  605.254672] hv_vmbus: registering driver hv_pci




- Saurabh

> -----Original Message-----
> From: Michael Kelley <mhklinux@outlook.com>
> Sent: 07 February 2025 02:30
> To: Michael Kelley <mhklinux@outlook.com>; Thomas Tai
> <thomas.tai@oracle.com>; mhkelley58@gmail.com; Haiyang Zhang
> <haiyangz@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
> <decui@microsoft.com>; drawat.floss@gmail.com; javierm@redhat.com;
> Helge Deller <deller@gmx.de>; daniel@ffwll.ch; airlied@gmail.com;
> tzimmermann@suse.de
> Cc: dri-devel@lists.freedesktop.org; linux-fbdev@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-hyperv@vger.kernel.org
> Subject: [EXTERNAL] RE: hyper_bf soft lockup on Azure Gen2 VM when taking
> kdump or executing kexec
> 
> From: Michael Kelley <mhklinux@outlook.com>
> >
> > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30,
> > 2025 12:44 PM
> > >
> > > > -----Original Message-----
> > > > From: Michael Kelley <mhklinux@outlook.com> Sent: Thursday,
> > > > January 30, 2025 3:20 PM
> > > >
> > > > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January
> > > > 30,
> > > > 2025 10:50 AM
> > > > >
> > > > > Sorry for the typo in the subject title. It should have been
> > > > > 'hyperv_fb soft lockup on Azure Gen2 VM when taking kdump or
> executing kexec'
> > > > >
> > > > > Thomas
> > > > >
> > > > > >
> > > > > > Hi Michael,
> > > > > >
> > > > > > We see an issue with the mainline kernel on the Azure Gen 2 VM
> > > > > > when trying to induce a kernel panic with sysrq commands. The
> > > > > > VM would hang with soft lockup. A similar issue happens when
> executing kexec on the VM.
> > > > > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1
> > > > > > VMs with bios boot are fine.
> > > > > >
> > > > > > git bisect identifies the issue is cased by the commit
> > > > > > 20ee2ae8c5899
> > > > > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > > > > However, reverting the commit would cause the frame buffer not
> > > > > > to work on the Gen2 VM.
> > > > > >
> > > > > > Do you have any hints on what caused this issue?
> > > > > >
> > > > > > To reproduce the issue with kdump:
> > > > > > - Install mainline kernel on an Azure Gen 2 VM and trigger a
> > > > > > kdump
> > > > > > - echo 1 > /proc/sys/kernel/sysrq
> > > > > > - echo c > /proc/sysrq-trigger
> > > > > >
> > > > > > To reproduce the issue with executing kexec:
> > > > > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > > > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img
> > > > > > --command- line="$( cat /proc/cmdline )"
> > > > > > - sudo kexec -e
> > > > > >
> > > > > > Thank you,
> > > > > > Thomas
> > > >
> > > > I will take a look, but it might be early next week before I can do so.
> > > >
> > >
> > > Thank you, Michael for your help!
> > >
> > > > It looks like your soft lockup log below is from the kdump kernel
> > > > (or the newly kexec'ed kernel). Can you confirm? Also, this looks like a
> subset of the full log.
> > >
> > > Yes, the soft lockup log below is from the kdump kernel.
> > >
> > > > Do you have the full serial console log that you could email to
> > > > me?  Seeing everything might be helpful. Of course, I'll try to
> > > > repro the problem myself as well.
> > >
> > > I have attached the complete bootup and kdump kernel log.
> > >
> > > File: bootup_and_kdump.log
> > > Line 1 ... 984 (bootup log)
> > > Line 990       (kdump kernel booting up)
> > > Line 1351      (soft lockup)
> > >
> > > Thank you,
> > > Thomas
> > >
> >
> > I have reproduced the problem in an Azure VM running Oracle Linux
> > 9.4 with the 6.13.0 kernel. Interestingly, the problem does not occur
> > in a VM running on a locally installed Hyper-V with Ubuntu 20.04 and
> > the 6.13.0 kernel. There are several differences in the two
> > environments:  the version of Hyper-V, the VM configuration, the Linux
> > distro, and the .config file used to build the 6.13.0 kernel. I'll try
> > to figure out what make the difference, and then the root cause.
> >
> 
> This has been a real bear to investigate. :-(  The key observation is that with
> older kernel versions, the efifb driver does *not* try to load when running in
> the kdump kernel, and everything works.
> In newer kernels, the efifb driver *does* try to load, and it appears to hang.
> (Actually, it is causing the VM to run very slowly. More on that in a minute.)
> 
> I've bisected the kernel again, compensating for the fact that commit
> 20ee2ae8c5899 is needed to make the Hyper-V frame buffer work. With that
> compensation, the actual problematic commit is 2bebc3cd4870 (Revert
> "firmware/sysfb: Clear screen_info state after consuming it").
> Doing the revert causes screen_info.orig_video_isVGA to retain its value of
> 0x70 (VIDEO_TYPE_EFI), which the kdump kernel picks up, causing it to load
> the efifb driver.
> 
> Then the question is why the efifb driver doesn't work in the kdump kernel.
> Actually, it *does* work in many cases. I built the 6.13.0 kernel on the Oracle
> Linux 9.4 system, and transferred the kernel image binary and module
> binaries to an Ubuntu 20.04 VM in Azure. In that VM, the efifb driver is
> loaded as part of the kdump kernel, and it doesn't cause any problems. But
> there's an interesting difference. In the Oracle Linux
> 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on the
> Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This difference is
> due to differences in how the screen_info variable gets setup in the two VMs.
> 
> When the normal kernel starts in a freshly booted VM, Hyper-V provides the
> EFI framebuffer at 0x40000000, and it works. But after the Hyper-V FB driver
> or Hyper-V DRM driver has initialized, Linux has picked a different MMIO
> address range and told Hyper-V to use the new address range (which often
> starts at 0x40900000). A kexec does *not* reset Hyper-V's transition to the
> new range, so when the efifb driver tries to use the framebuffer at
> 0x40000000, the accesses trap to Hyper-V and probably fail or timeout (I'm
> not sure of the details). After the guest does some number of these bad
> references, Hyper-V considers itself to be under attack from an ill-behaving
> guest, and throttles the guest so that it doesn't run for a few seconds. The
> throttling repeats, and results in extremely slow running in the kdump kernel.
> 
> Somehow in the Ubuntu 20.04 VM, the location of the frame buffer as stored
> in screen_info.lfb_base gets updated to be 0x40900000. I haven't fully
> debugged how that happens. But with that update, the efifb driver is using
> the updated framebuffer address and it works. On the Oracle Linux 9.4
> system, that update doesn't appear to happen, and the problem occurs.
> 
> This in an interim update on the problem. I'm still investigating how
> screen_info.lfb_base is set in the kdump kernel, and why it is different in the
> Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is well
> understood, we can contemplate how to fix the problem. Undoing the revert
> that is commit 2bebc3cd4870 doesn't seem like the solution since the original
> code there was reported to cause many other issues.
> The solution focus will likely be on how to ensure the kdump kernel gets the
> correct framebuffer address so the efifb driver works, since the framebuffer
> address changing is a quirk of Hyper-V behavior.
> 
> If anyone else has insight into what's going on here, please chime in.
> What I've learned so far is still somewhat tentative.
> 
> Michael

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: Michael Kelley <mhklinux@outlook.com> Sent: Thursday, February 6, 2025 1:00 PM
> 
> From: Michael Kelley <mhklinux@outlook.com>
> >
> > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30, 2025 12:44 PM
> > >
> > > > -----Original Message-----
> > > > From: Michael Kelley <mhklinux@outlook.com> Sent: Thursday, January 30, 2025 3:20 PM
> > > >
> > > > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30,
> > > > 2025 10:50 AM
> > > > >
> > > > > Sorry for the typo in the subject title. It should have been 'hyperv_fb soft lockup on
> > > > > Azure Gen2 VM when taking kdump or executing kexec'
> > > > >
> > > > > Thomas
> > > > >
> > > > > >
> > > > > > Hi Michael,
> > > > > >
> > > > > > We see an issue with the mainline kernel on the Azure Gen 2 VM when
> > > > > > trying to induce a kernel panic with sysrq commands. The VM would hang
> > > > > > with soft lockup. A similar issue happens when executing kexec on the VM.
> > > > > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios
> > > > > > boot are fine.
> > > > > >
> > > > > > git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> > > > > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > > > > However, reverting the commit would cause the frame buffer not to work
> > > > > > on the Gen2 VM.
> > > > > >
> > > > > > Do you have any hints on what caused this issue?
> > > > > >
> > > > > > To reproduce the issue with kdump:
> > > > > > - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> > > > > > - echo 1 > /proc/sys/kernel/sysrq
> > > > > > - echo c > /proc/sysrq-trigger
> > > > > >
> > > > > > To reproduce the issue with executing kexec:
> > > > > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > > > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> > > > > > line="$( cat /proc/cmdline )"
> > > > > > - sudo kexec -e
> > > > > >
> > > > > > Thank you,
> > > > > > Thomas
> > > >
> > > > I will take a look, but it might be early next week before I can do so.
> > > >
> > >
> > > Thank you, Michael for your help!
> > >
> > > > It looks like your soft lockup log below is from the kdump kernel (or the newly
> > > > kexec'ed kernel). Can you confirm? Also, this looks like a subset of the full log.
> > >
> > > Yes, the soft lockup log below is from the kdump kernel.
> > >
> > > > Do you have the full serial console log that you could email to me?  Seeing
> > > > everything might be helpful. Of course, I'll try to repro the problem myself
> > > > as well.
> > >
> > > I have attached the complete bootup and kdump kernel log.
> > >
> > > File: bootup_and_kdump.log
> > > Line 1 ... 984 (bootup log)
> > > Line 990       (kdump kernel booting up)
> > > Line 1351      (soft lockup)
> > >
> > > Thank you,
> > > Thomas
> > >
> >
> > I have reproduced the problem in an Azure VM running Oracle Linux
> > 9.4 with the 6.13.0 kernel. Interestingly, the problem does not occur
> > in a VM running on a locally installed Hyper-V with Ubuntu 20.04 and
> > the 6.13.0 kernel. There are several differences in the two
> > environments:  the version of Hyper-V, the VM configuration, the Linux
> > distro, and the .config file used to build the 6.13.0 kernel. I'll try to
> > figure out what make the difference, and then the root cause.
> >
> 
> This has been a real bear to investigate. :-(  The key observation
> is that with older kernel versions, the efifb driver does *not* try
> to load when running in the kdump kernel, and everything works.
> In newer kernels, the efifb driver *does* try to load, and it appears
> to hang.  (Actually, it is causing the VM to run very slowly. More on
> that in a minute.)
> 
> I've bisected the kernel again, compensating for the fact that commit
> 20ee2ae8c5899 is needed to make the Hyper-V frame buffer work. With
> that compensation, the actual problematic commit is 2bebc3cd4870
> (Revert "firmware/sysfb: Clear screen_info state after consuming it").
> Doing the revert causes screen_info.orig_video_isVGA to retain its value
> of 0x70 (VIDEO_TYPE_EFI), which the kdump kernel picks up, causing it
> to load the efifb driver.
> 
> Then the question is why the efifb driver doesn't work in the kdump
> kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
> on the Oracle Linux 9.4 system, and transferred the kernel image binary
> and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
> efifb driver is loaded as part of the kdump kernel, and it doesn't cause
> any problems. But there's an interesting difference. In the Oracle Linux
> 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
> the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
> difference is due to differences in how the screen_info variable gets
> setup in the two VMs.
> 
> When the normal kernel starts in a freshly booted VM, Hyper-V provides
> the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
> FB driver or Hyper-V DRM driver has initialized, Linux has picked a
> different MMIO address range and told Hyper-V to use the new
> address range (which often starts at 0x40900000). A kexec does *not*
> reset Hyper-V's transition to the new range, so when the efifb driver
> tries to use the framebuffer at 0x40000000, the accesses trap to
> Hyper-V and probably fail or timeout (I'm not sure of the details). After
> the guest does some number of these bad references, Hyper-V considers
> itself to be under attack from an ill-behaving guest, and throttles the
> guest so that it doesn't run for a few seconds. The throttling repeats,
> and results in extremely slow running in the kdump kernel.
> 
> Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
> as stored in screen_info.lfb_base gets updated to be 0x40900000. I
> haven't fully debugged how that happens. But with that update, the
> efifb driver is using the updated framebuffer address and it works. On
> the Oracle Linux 9.4 system, that update doesn't appear to happen,
> and the problem occurs.
> 
> This in an interim update on the problem. I'm still investigating how
> screen_info.lfb_base is set in the kdump kernel, and why it is different
> in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
> well understood, we can contemplate how to fix the problem. Undoing
> the revert that is commit 2bebc3cd4870 doesn't seem like the solution
> since the original code there was reported to cause many other issues.
> The solution focus will likely be on how to ensure the kdump kernel gets
> the correct framebuffer address so the efifb driver works, since the
> framebuffer address changing is a quirk of Hyper-V behavior.
> 
> If anyone else has insight into what's going on here, please chime in.
> What I've learned so far is still somewhat tentative.
> 

Here's what is happening. On Ubuntu 20.04, the kdump image is
loaded into crash memory using the kexec command. Ubuntu 20.04
has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1,
and per the kexec man page, it defaults to using the older kexec_load()
system call. When using kexec_load(), the contents to be loaded into
crash memory is constructed in user space by the kexec command.
The kexec command gets the "screen_info" settings, including the
physical address of the frame buffer, via the FBIOGET_FSCREENINFO
ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself
with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns
the updated framebuffer address. So the efifb driver loads and runs
correctly.

On Oracle Linux 9.4, the kdump image is also loaded with the
kexec command, but from kexec-tools package version 
kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than
the version on Ubuntu 20.04. This newer kexec defaults to using the
newer kexec_file_load() system call. This system call gets the
framebuffer address from the screen_info variable in the kernel, which
has not been updated to reflect the new framebuffer address. Hence
in the kdump kernel, the efifb driver uses the old framebuffer address,
and hence the problem.

To further complicate matters, the kexec on Oracle Linux 9.4 seems to
have a bug when the -c option forces the use of kexec_load() instead
of kexec_file_load(). As an experiment, I modified the kdumpctl shell
script to add the "-c" option to kexec, but in that case the value "0x0"
is passed as the framebuffer address, which is wrong. Furthermore,
the " screen_info.orig_video_isVGA" value (which I mentioned earlier
in connection with commit 2bebc3cd4870) is also set to 0, so the
kdump kernel no longer thinks it has an EFI framebuffer. Hence the
efifb driver isn't loaded, and the kdump works, though for the wrong
reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4
VM, then kdump works as expected, with the efifb driver being loaded
and using the correct framebuffer address. So something is going wrong
with kexec 2.0.28 in how it sets up the screen_info when the -c option
is used. I'll leave the debugging of the kexec bug to someone else.

I'm still thinking about alternatives to fix this mess. Please chime
in if you have suggestions.

Michael

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: Saurabh Singh Sengar <ssengar@microsoft.com> Sent: Friday, February 7, 2025 6:06 AM
> 
> Thanks Michael, for the analysis.
> 
> I have tried the kdump steps on Oracle 9.4, 6.13.0 kernel as well. Although I couldn't see
> the soft lockup issue I see some other VMBus failures. But I agree the bootup is
> extremely slow, which should be due to same reason.

Yes, I would also think it is the same underlying reason.

> 
> My system is having newer UEFI version, wondering if the latest UEFI version
> (UEFI Release v4.1 08/23/2024) causing this difference in behaviour.

I've seen both the original behavior that Thomas Tai reported, as well as
the extremely slow behavior. In my experiments, it seems to depend on
the Azure V size being used, though I didn't fully investigate. Originally I
was using a DS5_v2 VM (which is what Thomas was using) and saw the
same "soft lockup" as Thomas. Then I moved to a D8ds_v5 VM, which
is somewhat cheaper, and was seeing the very slow behavior.

See my separate email from this morning with a full explanation of the
root cause.

Michael

> 
> Relevant part of the logs:
> ---------------------------------------------------------
> echo 1 > /proc/sys/kernel/sysrq
> echo c > /proc/sysrq-trigger
> [  982.948352] sysrq: Trigger a crash
> [  982.949553] Kernel panic - not syncing: sysrq triggered crash
> [  982.951515] CPU: 31 UID: 0 PID: 6938 Comm: bash Kdump: loaded Not tainted
> 6.13.0 #1
> [  982.954115] Hardware name: Microsoft Corporation Virtual Machine/Virtual
> Machine, BIOS Hyper-V UEFI Release v4.1 08/23/2024
> [  982.957641] Call Trace:
> [  982.958508]  <TASK>
> [  982.959251]  panic+0x37e/0x3b0
> [  982.960373]  ? _printk+0x64/0x90
> [  982.961452]  sysrq_handle_crash+0x1a/0x20
> [  982.962840]  __handle_sysrq+0x9b/0x190
> [  982.964145]  write_sysrq_trigger+0x5f/0x80
> [  982.965578]  proc_reg_write+0x59/0xb0
> [  982.966905]  vfs_write+0x111/0x470
> [  982.968004]  ? __count_memcg_events+0xbf/0x150
> [  982.969432]  ? count_memcg_events.constprop.0+0x26/0x50
> [  982.971190]  ksys_write+0x6e/0xf0
> [  982.972307]  do_syscall_64+0x62/0x180
> [  982.973438]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [  982.975102] RIP: 0033:0x7f3d570fdbd7
> [  982.976421] Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa
> 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3
> 48 83 ec 28 48 89 54 24 18 48 89 74 24
> [  982.982893] RSP: 002b:00007fff6d613c48 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000001
> [  982.985424] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d570fdbd7
> [  982.987613] RDX: 0000000000000002 RSI: 000056362a928470 RDI:
> 0000000000000001
> [  982.989774] RBP: 000056362a928470 R08: 0000000000000000 R09:
> 00007f3d571b0d40
> [  982.992109] R10: 00007f3d571b0c40 R11: 0000000000000246 R12:
> 0000000000000002
> [  982.994321] R13: 00007f3d571fa780 R14: 0000000000000002 R15:
> 00007f3d571f59e0
> [  982.996461]  </TASK>
> [  982.998317] Kernel Offset: 0x10c00000 from 0xffffffff81000000 (relocation range:
> 0xffffffff80000000-0xffffffffbfffffff)
> [    0.000000] Linux version 6.13.0 (lisatest@lisa--505-e0-n0) (gcc (GCC) 11.5.0
> 20240719 (Red Hat 11.5.0-2.0.1), GNU ld version 2.35.2-54.0.1.el9) #1 SMP
> PREEMPT_DYNAMIC Thu Feb  6 10:05:27 UTC 2025
> [    0.000000] Command line: elfcorehdr=0xd000000
> BOOT_IMAGE=(hd0,gpt1)/vmlinuz-6.13.0 ro console=tty0 console=ttyS0,115200n8
> rd.lvm.vg=rootvg irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off
> numa=off udev.children-max=2 panic=10 acpi_no_memhotplug
> transparent_hugepage=never nokaslr hest_disable novmcoredd cma=0 hugetlb_cma=0
> iommu=off disable_cpu_apicid=0
> [    0.000000] BIOS-provided physical RAM map:
> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009ffff] usable
> [    0.000000] BIOS-e820: [mem 0x00000000000c0000-0x00000000000fffff] reserved
> [    0.000000] BIOS-e820: [mem 0x000000000d0e00b0-0x000000002cffffff] usable
> [    0.000000] BIOS-e820: [mem 0x000000003eead000-0x000000003eeb3fff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000003ff41000-0x000000003ffc8fff] reserved
> [    0.000000] BIOS-e820: [mem 0x000000003ffc9000-0x000000003fffafff] ACPI data
> [    0.000000] BIOS-e820: [mem 0x000000003fffb000-0x000000003fffefff] ACPI NVS
> [    0.000000] random: crng init done
> 
> <snip>
> 
> [    0.928063] Console: switching to colour frame buffer device 128x48
> [   13.391297] fb0: EFI VGA frame buffer device
> 
> <snip>
> 
> [  590.199511] hv_netvsc 7c1e527c-2980-7c1e-527c-29807c1e527c (unnamed
> net_device) (uninitialized): VF slot 1 added
> [  595.120270] Console: switching to colour dummy device 80x25
> [  605.203700] hyperv_fb: Time out on waiting vram location ack
> [  605.206161] iounmap: bad address 0000000005f4dac5
> [  605.207740] CPU: 0 UID: 0 PID: 30 Comm: kworker/u4:2 Not tainted 6.13.0 #1
> [  605.209984] Hardware name: Microsoft Corporation Virtual Machine/Virtual
> Machine, BIOS Hyper-V UEFI Release v4.1 08/23/2024
> [  605.213869] Workqueue: async async_run_entry_fn
> [  605.215601] Call Trace:
> [  605.216382]  <TASK>
> [  605.217123]  dump_stack_lvl+0x66/0x90
> [  605.218184]  hvfb_putmem+0x32/0x110 [hyperv_fb]
> [  605.219646]  hvfb_probe+0x27f/0x360 [hyperv_fb]
> [  605.221120]  vmbus_probe+0x3d/0xa0 [hv_vmbus]
> [  605.222623]  really_probe+0xd9/0x390
> [  605.223779]  __driver_probe_device+0x78/0x160
> [  605.225213]  driver_probe_device+0x1e/0xa0
> [  605.226591]  __driver_attach_async_helper+0x5e/0xe0
> [  605.228166]  async_run_entry_fn+0x34/0x130
> [  605.229681]  process_one_work+0x187/0x3b0
> [  605.231075]  worker_thread+0x24e/0x360
> [  605.232376]  ? __pfx_worker_thread+0x10/0x10
> [  605.233758]  kthread+0xd3/0x100
> [  605.234805]  ? __pfx_kthread+0x10/0x10
> [  605.236053]  ret_from_fork+0x34/0x50
> [  605.237251]  ? __pfx_kthread+0x10/0x10
> [  605.238519]  ret_from_fork_asm+0x1a/0x30
> [  605.239833]  </TASK>
> [  605.240855] hv_vmbus: probe failed for device 5620e0c7-8062-4dce-aeb7-
> 520c7ef76171 (-110)
> [  605.243404] hyperv_fb 5620e0c7-8062-4dce-aeb7-520c7ef76171: probe with
> driver hyperv_fb failed with error -110
> [  605.254672] hv_vmbus: registering driver hv_pci
> 
> 
> 
> 
> - Saurabh
> 
> > -----Original Message-----
> > From: Michael Kelley <mhklinux@outlook.com>
> > Sent: 07 February 2025 02:30
> > To: Michael Kelley <mhklinux@outlook.com>; Thomas Tai
> > <thomas.tai@oracle.com>; mhkelley58@gmail.com; Haiyang Zhang
> > <haiyangz@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
> > <decui@microsoft.com>; drawat.floss@gmail.com; javierm@redhat.com;
> > Helge Deller <deller@gmx.de>; daniel@ffwll.ch; airlied@gmail.com;
> > tzimmermann@suse.de
> > Cc: dri-devel@lists.freedesktop.org; linux-fbdev@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-hyperv@vger.kernel.org
> > Subject: [EXTERNAL] RE: hyper_bf soft lockup on Azure Gen2 VM when taking
> > kdump or executing kexec
> >
> > From: Michael Kelley <mhklinux@outlook.com>
> > >
> > > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January 30,
> > > 2025 12:44 PM
> > > >
> > > > > -----Original Message-----
> > > > > From: Michael Kelley <mhklinux@outlook.com> Sent: Thursday,
> > > > > January 30, 2025 3:20 PM
> > > > >
> > > > > From: Thomas Tai <thomas.tai@oracle.com> Sent: Thursday, January
> > > > > 30,
> > > > > 2025 10:50 AM
> > > > > >
> > > > > > Sorry for the typo in the subject title. It should have been
> > > > > > 'hyperv_fb soft lockup on Azure Gen2 VM when taking kdump or
> > executing kexec'
> > > > > >
> > > > > > Thomas
> > > > > >
> > > > > > >
> > > > > > > Hi Michael,
> > > > > > >
> > > > > > > We see an issue with the mainline kernel on the Azure Gen 2 VM
> > > > > > > when trying to induce a kernel panic with sysrq commands. The
> > > > > > > VM would hang with soft lockup. A similar issue happens when
> > executing kexec on the VM.
> > > > > > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1
> > > > > > > VMs with bios boot are fine.
> > > > > > >
> > > > > > > git bisect identifies the issue is cased by the commit
> > > > > > > 20ee2ae8c5899
> > > > > > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > > > > > However, reverting the commit would cause the frame buffer not
> > > > > > > to work on the Gen2 VM.
> > > > > > >
> > > > > > > Do you have any hints on what caused this issue?
> > > > > > >
> > > > > > > To reproduce the issue with kdump:
> > > > > > > - Install mainline kernel on an Azure Gen 2 VM and trigger a
> > > > > > > kdump
> > > > > > > - echo 1 > /proc/sys/kernel/sysrq
> > > > > > > - echo c > /proc/sysrq-trigger
> > > > > > >
> > > > > > > To reproduce the issue with executing kexec:
> > > > > > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > > > > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img
> > > > > > > --command- line="$( cat /proc/cmdline )"
> > > > > > > - sudo kexec -e
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Thomas
> > > > >
> > > > > I will take a look, but it might be early next week before I can do so.
> > > > >
> > > >
> > > > Thank you, Michael for your help!
> > > >
> > > > > It looks like your soft lockup log below is from the kdump kernel
> > > > > (or the newly kexec'ed kernel). Can you confirm? Also, this looks like a
> > subset of the full log.
> > > >
> > > > Yes, the soft lockup log below is from the kdump kernel.
> > > >
> > > > > Do you have the full serial console log that you could email to
> > > > > me?  Seeing everything might be helpful. Of course, I'll try to
> > > > > repro the problem myself as well.
> > > >
> > > > I have attached the complete bootup and kdump kernel log.
> > > >
> > > > File: bootup_and_kdump.log
> > > > Line 1 ... 984 (bootup log)
> > > > Line 990       (kdump kernel booting up)
> > > > Line 1351      (soft lockup)
> > > >
> > > > Thank you,
> > > > Thomas
> > > >
> > >
> > > I have reproduced the problem in an Azure VM running Oracle Linux
> > > 9.4 with the 6.13.0 kernel. Interestingly, the problem does not occur
> > > in a VM running on a locally installed Hyper-V with Ubuntu 20.04 and
> > > the 6.13.0 kernel. There are several differences in the two
> > > environments:  the version of Hyper-V, the VM configuration, the Linux
> > > distro, and the .config file used to build the 6.13.0 kernel. I'll try
> > > to figure out what make the difference, and then the root cause.
> > >
> >
> > This has been a real bear to investigate. :-(  The key observation is that with
> > older kernel versions, the efifb driver does *not* try to load when running in
> > the kdump kernel, and everything works.
> > In newer kernels, the efifb driver *does* try to load, and it appears to hang.
> > (Actually, it is causing the VM to run very slowly. More on that in a minute.)
> >
> > I've bisected the kernel again, compensating for the fact that commit
> > 20ee2ae8c5899 is needed to make the Hyper-V frame buffer work. With that
> > compensation, the actual problematic commit is 2bebc3cd4870 (Revert
> > "firmware/sysfb: Clear screen_info state after consuming it").
> > Doing the revert causes screen_info.orig_video_isVGA to retain its value of
> > 0x70 (VIDEO_TYPE_EFI), which the kdump kernel picks up, causing it to load
> > the efifb driver.
> >
> > Then the question is why the efifb driver doesn't work in the kdump kernel.
> > Actually, it *does* work in many cases. I built the 6.13.0 kernel on the Oracle
> > Linux 9.4 system, and transferred the kernel image binary and module
> > binaries to an Ubuntu 20.04 VM in Azure. In that VM, the efifb driver is
> > loaded as part of the kdump kernel, and it doesn't cause any problems. But
> > there's an interesting difference. In the Oracle Linux
> > 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on the
> > Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This difference is
> > due to differences in how the screen_info variable gets setup in the two VMs.
> >
> > When the normal kernel starts in a freshly booted VM, Hyper-V provides the
> > EFI framebuffer at 0x40000000, and it works. But after the Hyper-V FB driver
> > or Hyper-V DRM driver has initialized, Linux has picked a different MMIO
> > address range and told Hyper-V to use the new address range (which often
> > starts at 0x40900000). A kexec does *not* reset Hyper-V's transition to the
> > new range, so when the efifb driver tries to use the framebuffer at
> > 0x40000000, the accesses trap to Hyper-V and probably fail or timeout (I'm
> > not sure of the details). After the guest does some number of these bad
> > references, Hyper-V considers itself to be under attack from an ill-behaving
> > guest, and throttles the guest so that it doesn't run for a few seconds. The
> > throttling repeats, and results in extremely slow running in the kdump kernel.
> >
> > Somehow in the Ubuntu 20.04 VM, the location of the frame buffer as stored
> > in screen_info.lfb_base gets updated to be 0x40900000. I haven't fully
> > debugged how that happens. But with that update, the efifb driver is using
> > the updated framebuffer address and it works. On the Oracle Linux 9.4
> > system, that update doesn't appear to happen, and the problem occurs.
> >
> > This in an interim update on the problem. I'm still investigating how
> > screen_info.lfb_base is set in the kdump kernel, and why it is different in the
> > Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is well
> > understood, we can contemplate how to fix the problem. Undoing the revert
> > that is commit 2bebc3cd4870 doesn't seem like the solution since the original
> > code there was reported to cause many other issues.
> > The solution focus will likely be on how to ensure the kdump kernel gets the
> > correct framebuffer address so the efifb driver works, since the framebuffer
> > address changing is a quirk of Hyper-V behavior.
> >
> > If anyone else has insight into what's going on here, please chime in.
> > What I've learned so far is still somewhat tentative.
> >
> > Michael

Re: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[thomas.tai]

<snip>

>> Then the question is why the efifb driver doesn't work in the kdump
>> kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
>> on the Oracle Linux 9.4 system, and transferred the kernel image binary
>> and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
>> efifb driver is loaded as part of the kdump kernel, and it doesn't cause
>> any problems. But there's an interesting difference. In the Oracle Linux
>> 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
>> the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
>> difference is due to differences in how the screen_info variable gets
>> setup in the two VMs.
>>
>> When the normal kernel starts in a freshly booted VM, Hyper-V provides
>> the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
>> FB driver or Hyper-V DRM driver has initialized, Linux has picked a
>> different MMIO address range and told Hyper-V to use the new
>> address range (which often starts at 0x40900000). A kexec does *not*
>> reset Hyper-V's transition to the new range, so when the efifb driver
>> tries to use the framebuffer at 0x40000000, the accesses trap to
>> Hyper-V and probably fail or timeout (I'm not sure of the details). After
>> the guest does some number of these bad references, Hyper-V considers
>> itself to be under attack from an ill-behaving guest, and throttles the
>> guest so that it doesn't run for a few seconds. The throttling repeats,
>> and results in extremely slow running in the kdump kernel.
>>
>> Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
>> as stored in screen_info.lfb_base gets updated to be 0x40900000. I
>> haven't fully debugged how that happens. But with that update, the
>> efifb driver is using the updated framebuffer address and it works. On
>> the Oracle Linux 9.4 system, that update doesn't appear to happen,
>> and the problem occurs.
>>
>> This in an interim update on the problem. I'm still investigating how
>> screen_info.lfb_base is set in the kdump kernel, and why it is different
>> in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
>> well understood, we can contemplate how to fix the problem. Undoing
>> the revert that is commit 2bebc3cd4870 doesn't seem like the solution
>> since the original code there was reported to cause many other issues.
>> The solution focus will likely be on how to ensure the kdump kernel gets
>> the correct framebuffer address so the efifb driver works, since the
>> framebuffer address changing is a quirk of Hyper-V behavior.
>>
>> If anyone else has insight into what's going on here, please chime in.
>> What I've learned so far is still somewhat tentative.
>>
> Here's what is happening. On Ubuntu 20.04, the kdump image is
> loaded into crash memory using the kexec command. Ubuntu 20.04
> has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1,
> and per the kexec man page, it defaults to using the older kexec_load()
> system call. When using kexec_load(), the contents to be loaded into
> crash memory is constructed in user space by the kexec command.
> The kexec command gets the "screen_info" settings, including the
> physical address of the frame buffer, via the FBIOGET_FSCREENINFO
> ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself
> with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns
> the updated framebuffer address. So the efifb driver loads and runs
> correctly.
>
> On Oracle Linux 9.4, the kdump image is also loaded with the
> kexec command, but from kexec-tools package version
> kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than
> the version on Ubuntu 20.04. This newer kexec defaults to using the
> newer kexec_file_load() system call. This system call gets the
> framebuffer address from the screen_info variable in the kernel, which
> has not been updated to reflect the new framebuffer address. Hence
> in the kdump kernel, the efifb driver uses the old framebuffer address,
> and hence the problem.
>
> To further complicate matters, the kexec on Oracle Linux 9.4 seems to
> have a bug when the -c option forces the use of kexec_load() instead
> of kexec_file_load(). As an experiment, I modified the kdumpctl shell
> script to add the "-c" option to kexec, but in that case the value "0x0"
> is passed as the framebuffer address, which is wrong. Furthermore,
> the " screen_info.orig_video_isVGA" value (which I mentioned earlier
> in connection with commit 2bebc3cd4870) is also set to 0, so the
> kdump kernel no longer thinks it has an EFI framebuffer. Hence the
> efifb driver isn't loaded, and the kdump works, though for the wrong
> reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4
> VM, then kdump works as expected, with the efifb driver being loaded
> and using the correct framebuffer address. So something is going wrong
> with kexec 2.0.28 in how it sets up the screen_info when the -c option
> is used. I'll leave the debugging of the kexec bug to someone else.
Hi Michael,

Do you think we need to handle Azure Gen2 VM differently in the kexec?

Or should we change the kexec_file_load() system call to

retrieve the correct framebuffer address?


Thank you,
Thomas
> I'm still thinking about alternatives to fix this mess. Please chime
> in if you have suggestions.
>
> Michael

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: thomas.tai@oracle.com <thomas.tai@oracle.com> Sent: Monday, February 10, 2025 7:08 AM
> 
> <snip>
> 
> >> Then the question is why the efifb driver doesn't work in the kdump
> >> kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
> >> on the Oracle Linux 9.4 system, and transferred the kernel image binary
> >> and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
> >> efifb driver is loaded as part of the kdump kernel, and it doesn't cause
> >> any problems. But there's an interesting difference. In the Oracle Linux
> >> 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
> >> the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
> >> difference is due to differences in how the screen_info variable gets
> >> setup in the two VMs.
> >>
> >> When the normal kernel starts in a freshly booted VM, Hyper-V provides
> >> the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
> >> FB driver or Hyper-V DRM driver has initialized, Linux has picked a
> >> different MMIO address range and told Hyper-V to use the new
> >> address range (which often starts at 0x40900000). A kexec does *not*
> >> reset Hyper-V's transition to the new range, so when the efifb driver
> >> tries to use the framebuffer at 0x40000000, the accesses trap to
> >> Hyper-V and probably fail or timeout (I'm not sure of the details). After
> >> the guest does some number of these bad references, Hyper-V considers
> >> itself to be under attack from an ill-behaving guest, and throttles the
> >> guest so that it doesn't run for a few seconds. The throttling repeats,
> >> and results in extremely slow running in the kdump kernel.
> >>
> >> Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
> >> as stored in screen_info.lfb_base gets updated to be 0x40900000. I
> >> haven't fully debugged how that happens. But with that update, the
> >> efifb driver is using the updated framebuffer address and it works. On
> >> the Oracle Linux 9.4 system, that update doesn't appear to happen,
> >> and the problem occurs.
> >>
> >> This in an interim update on the problem. I'm still investigating how
> >> screen_info.lfb_base is set in the kdump kernel, and why it is different
> >> in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
> >> well understood, we can contemplate how to fix the problem. Undoing
> >> the revert that is commit 2bebc3cd4870 doesn't seem like the solution
> >> since the original code there was reported to cause many other issues.
> >> The solution focus will likely be on how to ensure the kdump kernel gets
> >> the correct framebuffer address so the efifb driver works, since the
> >> framebuffer address changing is a quirk of Hyper-V behavior.
> >>
> >> If anyone else has insight into what's going on here, please chime in.
> >> What I've learned so far is still somewhat tentative.
> >>
> > Here's what is happening. On Ubuntu 20.04, the kdump image is
> > loaded into crash memory using the kexec command. Ubuntu 20.04
> > has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1,
> > and per the kexec man page, it defaults to using the older kexec_load()
> > system call. When using kexec_load(), the contents to be loaded into
> > crash memory is constructed in user space by the kexec command.
> > The kexec command gets the "screen_info" settings, including the
> > physical address of the frame buffer, via the FBIOGET_FSCREENINFO
> > ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself
> > with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns
> > the updated framebuffer address. So the efifb driver loads and runs
> > correctly.
> >
> > On Oracle Linux 9.4, the kdump image is also loaded with the
> > kexec command, but from kexec-tools package version
> > kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than
> > the version on Ubuntu 20.04. This newer kexec defaults to using the
> > newer kexec_file_load() system call. This system call gets the
> > framebuffer address from the screen_info variable in the kernel, which
> > has not been updated to reflect the new framebuffer address. Hence
> > in the kdump kernel, the efifb driver uses the old framebuffer address,
> > and hence the problem.
> >
> > To further complicate matters, the kexec on Oracle Linux 9.4 seems to
> > have a bug when the -c option forces the use of kexec_load() instead
> > of kexec_file_load(). As an experiment, I modified the kdumpctl shell
> > script to add the "-c" option to kexec, but in that case the value "0x0"
> > is passed as the framebuffer address, which is wrong. Furthermore,
> > the " screen_info.orig_video_isVGA" value (which I mentioned earlier
> > in connection with commit 2bebc3cd4870) is also set to 0, so the
> > kdump kernel no longer thinks it has an EFI framebuffer. Hence the
> > efifb driver isn't loaded, and the kdump works, though for the wrong
> > reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4
> > VM, then kdump works as expected, with the efifb driver being loaded
> > and using the correct framebuffer address. So something is going wrong
> > with kexec 2.0.28 in how it sets up the screen_info when the -c option
> > is used. I'll leave the debugging of the kexec bug to someone else.
>
> Hi Michael,
> 
> Do you think we need to handle Azure Gen2 VM differently in the kexec?
> 
> Or should we change the kexec_file_load() system call to retrieve the correct
> framebuffer address?

I'm thinking there may be a fix in the Hyper-V FB and Hyper-V DRM drivers.
Commit c25a19afb81c may also be a cause of the problem -- see precursor
commit 3cb73bc3fa2a, which describes exactly the problem. I still need to
do some testing, but without that commit, kdump won't detect that it has
an EFI framebuffer, won't load the efifb driver, and so won't encounter the
problem. But we probably need to get Thomas Zimmerman to weigh in on
the implications of reverting c25a19afb81c.

There's one additional variation of the problem. Assume the Hyper-V FB
driver is loaded (for example) during boot and moves the framebuffer. Then
system runs kexec as part of arming kdump during the boot sequence.
The most recent location of the framebuffer (and whether it is an EFI framebuffer)
gets picked at the time kexec runs, and is stored in the crash kernel memory area.
But what if the framebuffer later moves, perhaps because the Hyper-V FB driver
is unbound? The crash kernel memory area doesn’t get updated and kdump
could still have the wrong framebuffer address. This anomaly argues for the
commit 3cb73bc3fa2a approach of just ensuring that the efifb driver doesn't
load. Of course that approach means that the kdump kernel *must* contain
either the Hyper-V FB or Hyper-V DRM driver in order to work on a system
with only a framebuffer for text output. The efifb driver won't work. But
perhaps that's OK.

Changing kexec (or the invoking script) to special case Hyper-V Gen 2 VMs and
always use kexec_load() instead of kexec_file_load() sounds like a big hack
to me.  And with that approach, you give up the ability to enforce loading only
properly signed kdump images. This is something kexec_file_load() provides
that kexec_load() doesn't, and is one of the main reasons that kexec_file_load()
was added.

Whether the kexec_file_load() system call could be enhanced to get the
frame buffer information from the /dev/fb0 device, I'm not sure. That might
be a reasonable approach, though it still has the problem that the framebuffer
address could change *after* kexec_file_load() runs.

Anyway, that's a dump of my current thoughts. I haven't reached a final
conclusion or recommendation yet. Comments from others on the
thread are welcome.

Michael

> 
> Thank you,
> Thomas
>
> > I'm still thinking about alternatives to fix this mess. Please chime
> > in if you have suggestions.
> >
> > Michael

Re: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Maxim Levitsky]

On Mon, 2025-02-10 at 21:35 +0000, Michael Kelley wrote:
> From: thomas.tai@oracle.com <thomas.tai@oracle.com> Sent: Monday, February 10, 2025 7:08 AM
> > <snip>
> > 
> > > > Then the question is why the efifb driver doesn't work in the kdump
> > > > kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
> > > > on the Oracle Linux 9.4 system, and transferred the kernel image binary
> > > > and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
> > > > efifb driver is loaded as part of the kdump kernel, and it doesn't cause
> > > > any problems. But there's an interesting difference. In the Oracle Linux
> > > > 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
> > > > the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
> > > > difference is due to differences in how the screen_info variable gets
> > > > setup in the two VMs.
> > > > 
> > > > When the normal kernel starts in a freshly booted VM, Hyper-V provides
> > > > the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
> > > > FB driver or Hyper-V DRM driver has initialized, Linux has picked a
> > > > different MMIO address range and told Hyper-V to use the new
> > > > address range (which often starts at 0x40900000). A kexec does *not*
> > > > reset Hyper-V's transition to the new range, so when the efifb driver
> > > > tries to use the framebuffer at 0x40000000, the accesses trap to
> > > > Hyper-V and probably fail or timeout (I'm not sure of the details). After
> > > > the guest does some number of these bad references, Hyper-V considers
> > > > itself to be under attack from an ill-behaving guest, and throttles the
> > > > guest so that it doesn't run for a few seconds. The throttling repeats,
> > > > and results in extremely slow running in the kdump kernel.
> > > > 
> > > > Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
> > > > as stored in screen_info.lfb_base gets updated to be 0x40900000. I
> > > > haven't fully debugged how that happens. But with that update, the
> > > > efifb driver is using the updated framebuffer address and it works. On
> > > > the Oracle Linux 9.4 system, that update doesn't appear to happen,
> > > > and the problem occurs.
> > > > 
> > > > This in an interim update on the problem. I'm still investigating how
> > > > screen_info.lfb_base is set in the kdump kernel, and why it is different
> > > > in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
> > > > well understood, we can contemplate how to fix the problem. Undoing
> > > > the revert that is commit 2bebc3cd4870 doesn't seem like the solution
> > > > since the original code there was reported to cause many other issues.
> > > > The solution focus will likely be on how to ensure the kdump kernel gets
> > > > the correct framebuffer address so the efifb driver works, since the
> > > > framebuffer address changing is a quirk of Hyper-V behavior.
> > > > 
> > > > If anyone else has insight into what's going on here, please chime in.
> > > > What I've learned so far is still somewhat tentative.
> > > > 
> > > Here's what is happening. On Ubuntu 20.04, the kdump image is
> > > loaded into crash memory using the kexec command. Ubuntu 20.04
> > > has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1,
> > > and per the kexec man page, it defaults to using the older kexec_load()
> > > system call. When using kexec_load(), the contents to be loaded into
> > > crash memory is constructed in user space by the kexec command.
> > > The kexec command gets the "screen_info" settings, including the
> > > physical address of the frame buffer, via the FBIOGET_FSCREENINFO
> > > ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself
> > > with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns
> > > the updated framebuffer address. So the efifb driver loads and runs
> > > correctly.
> > > 
> > > On Oracle Linux 9.4, the kdump image is also loaded with the
> > > kexec command, but from kexec-tools package version
> > > kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than
> > > the version on Ubuntu 20.04. This newer kexec defaults to using the
> > > newer kexec_file_load() system call. This system call gets the
> > > framebuffer address from the screen_info variable in the kernel, which
> > > has not been updated to reflect the new framebuffer address. Hence
> > > in the kdump kernel, the efifb driver uses the old framebuffer address,
> > > and hence the problem.
> > > 
> > > To further complicate matters, the kexec on Oracle Linux 9.4 seems to
> > > have a bug when the -c option forces the use of kexec_load() instead
> > > of kexec_file_load(). As an experiment, I modified the kdumpctl shell
> > > script to add the "-c" option to kexec, but in that case the value "0x0"
> > > is passed as the framebuffer address, which is wrong. Furthermore,
> > > the " screen_info.orig_video_isVGA" value (which I mentioned earlier
> > > in connection with commit 2bebc3cd4870) is also set to 0, so the
> > > kdump kernel no longer thinks it has an EFI framebuffer. Hence the
> > > efifb driver isn't loaded, and the kdump works, though for the wrong
> > > reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4
> > > VM, then kdump works as expected, with the efifb driver being loaded
> > > and using the correct framebuffer address. So something is going wrong
> > > with kexec 2.0.28 in how it sets up the screen_info when the -c option
> > > is used. I'll leave the debugging of the kexec bug to someone else.
> > 
> > Hi Michael,
> > 
> > Do you think we need to handle Azure Gen2 VM differently in the kexec?
> > 
> > Or should we change the kexec_file_load() system call to retrieve the correct
> > framebuffer address?
> 
> I'm thinking there may be a fix in the Hyper-V FB and Hyper-V DRM drivers.
> Commit c25a19afb81c may also be a cause of the problem -- see precursor
> commit 3cb73bc3fa2a, which describes exactly the problem. I still need to
> do some testing, but without that commit, kdump won't detect that it has
> an EFI framebuffer, won't load the efifb driver, and so won't encounter the
> problem. But we probably need to get Thomas Zimmerman to weigh in on
> the implications of reverting c25a19afb81c.
> 
> There's one additional variation of the problem. Assume the Hyper-V FB
> driver is loaded (for example) during boot and moves the framebuffer. Then
> system runs kexec as part of arming kdump during the boot sequence.
> The most recent location of the framebuffer (and whether it is an EFI framebuffer)
> gets picked at the time kexec runs, and is stored in the crash kernel memory area.
> But what if the framebuffer later moves, perhaps because the Hyper-V FB driver
> is unbound? The crash kernel memory area doesn’t get updated and kdump
> could still have the wrong framebuffer address. This anomaly argues for the
> commit 3cb73bc3fa2a approach of just ensuring that the efifb driver doesn't
> load. Of course that approach means that the kdump kernel *must* contain
> either the Hyper-V FB or Hyper-V DRM driver in order to work on a system
> with only a framebuffer for text output. The efifb driver won't work. But
> perhaps that's OK.
> 
> Changing kexec (or the invoking script) to special case Hyper-V Gen 2 VMs and
> always use kexec_load() instead of kexec_file_load() sounds like a big hack
> to me.  And with that approach, you give up the ability to enforce loading only
> properly signed kdump images. This is something kexec_file_load() provides
> that kexec_load() doesn't, and is one of the main reasons that kexec_file_load()
> was added.
> 
> Whether the kexec_file_load() system call could be enhanced to get the
> frame buffer information from the /dev/fb0 device, I'm not sure. That might
> be a reasonable approach, though it still has the problem that the framebuffer
> address could change *after* kexec_file_load() runs.
> 
> Anyway, that's a dump of my current thoughts. I haven't reached a final
> conclusion or recommendation yet. Comments from others on the
> thread are welcome.

Hi!

Asking because I also had to do some digging in this area:

Do you think that the kernel can *ask* the hypervisor where the framebuffer is instead
of relying on bios, the bootloader and/or kexec to somehow provide this information?

If hyperv doesn't provide this API, how hard it would be in your opinion to provide it?

I am asking because, I also had to debug a RHEL downstream issue where a slightly botched backport
ensured that the first stage of the compressed uefi boot image, stopped passing the 'screen_info'
to the second stage (the kernel itself), and as a result of this, the second stage stopped
loading simplefb, and as a result of *this*, the PCI driver started to try to use the framebuffer
range for its own use which failed and resulted in a cryptic error.

If the kernel was to just issue some form of a hypercall to ask the hypervisor where the framebuffer currently is,
we could avoid a whole class of bugs similar to this.
What do you think?

Best regards,
	Maxim Levitsky


> 
> Michael
> 
> > Thank you,
> > Thomas
> > 
> > > I'm still thinking about alternatives to fix this mess. Please chime
> > > in if you have suggestions.
> > > 
> > > Michael

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: Maxim Levitsky <mlevitsk@redhat.com> Sent: Monday, February 10, 2025 3:57 PM
> 
> On Mon, 2025-02-10 at 21:35 +0000, Michael Kelley wrote:
> > From: thomas.tai@oracle.com <thomas.tai@oracle.com> Sent: Monday, February 10, 2025 7:08 AM
> > > <snip>
> > >
> > > > > Then the question is why the efifb driver doesn't work in the kdump
> > > > > kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
> > > > > on the Oracle Linux 9.4 system, and transferred the kernel image binary
> > > > > and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
> > > > > efifb driver is loaded as part of the kdump kernel, and it doesn't cause
> > > > > any problems. But there's an interesting difference. In the Oracle Linux
> > > > > 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
> > > > > the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
> > > > > difference is due to differences in how the screen_info variable gets
> > > > > setup in the two VMs.
> > > > >
> > > > > When the normal kernel starts in a freshly booted VM, Hyper-V provides
> > > > > the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
> > > > > FB driver or Hyper-V DRM driver has initialized, Linux has picked a
> > > > > different MMIO address range and told Hyper-V to use the new
> > > > > address range (which often starts at 0x40900000). A kexec does *not*
> > > > > reset Hyper-V's transition to the new range, so when the efifb driver
> > > > > tries to use the framebuffer at 0x40000000, the accesses trap to
> > > > > Hyper-V and probably fail or timeout (I'm not sure of the details). After
> > > > > the guest does some number of these bad references, Hyper-V considers
> > > > > itself to be under attack from an ill-behaving guest, and throttles the
> > > > > guest so that it doesn't run for a few seconds. The throttling repeats,
> > > > > and results in extremely slow running in the kdump kernel.
> > > > >
> > > > > Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
> > > > > as stored in screen_info.lfb_base gets updated to be 0x40900000. I
> > > > > haven't fully debugged how that happens. But with that update, the
> > > > > efifb driver is using the updated framebuffer address and it works. On
> > > > > the Oracle Linux 9.4 system, that update doesn't appear to happen,
> > > > > and the problem occurs.
> > > > >
> > > > > This in an interim update on the problem. I'm still investigating how
> > > > > screen_info.lfb_base is set in the kdump kernel, and why it is different
> > > > > in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
> > > > > well understood, we can contemplate how to fix the problem. Undoing
> > > > > the revert that is commit 2bebc3cd4870 doesn't seem like the solution
> > > > > since the original code there was reported to cause many other issues.
> > > > > The solution focus will likely be on how to ensure the kdump kernel gets
> > > > > the correct framebuffer address so the efifb driver works, since the
> > > > > framebuffer address changing is a quirk of Hyper-V behavior.
> > > > >
> > > > > If anyone else has insight into what's going on here, please chime in.
> > > > > What I've learned so far is still somewhat tentative.
> > > > >
> > > > Here's what is happening. On Ubuntu 20.04, the kdump image is
> > > > loaded into crash memory using the kexec command. Ubuntu 20.04
> > > > has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1,
> > > > and per the kexec man page, it defaults to using the older kexec_load()
> > > > system call. When using kexec_load(), the contents to be loaded into
> > > > crash memory is constructed in user space by the kexec command.
> > > > The kexec command gets the "screen_info" settings, including the
> > > > physical address of the frame buffer, via the FBIOGET_FSCREENINFO
> > > > ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself
> > > > with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns
> > > > the updated framebuffer address. So the efifb driver loads and runs
> > > > correctly.
> > > >
> > > > On Oracle Linux 9.4, the kdump image is also loaded with the
> > > > kexec command, but from kexec-tools package version
> > > > kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than
> > > > the version on Ubuntu 20.04. This newer kexec defaults to using the
> > > > newer kexec_file_load() system call. This system call gets the
> > > > framebuffer address from the screen_info variable in the kernel, which
> > > > has not been updated to reflect the new framebuffer address. Hence
> > > > in the kdump kernel, the efifb driver uses the old framebuffer address,
> > > > and hence the problem.
> > > >
> > > > To further complicate matters, the kexec on Oracle Linux 9.4 seems to
> > > > have a bug when the -c option forces the use of kexec_load() instead
> > > > of kexec_file_load(). As an experiment, I modified the kdumpctl shell
> > > > script to add the "-c" option to kexec, but in that case the value "0x0"
> > > > is passed as the framebuffer address, which is wrong. Furthermore,
> > > > the " screen_info.orig_video_isVGA" value (which I mentioned earlier
> > > > in connection with commit 2bebc3cd4870) is also set to 0, so the
> > > > kdump kernel no longer thinks it has an EFI framebuffer. Hence the
> > > > efifb driver isn't loaded, and the kdump works, though for the wrong
> > > > reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4
> > > > VM, then kdump works as expected, with the efifb driver being loaded
> > > > and using the correct framebuffer address. So something is going wrong
> > > > with kexec 2.0.28 in how it sets up the screen_info when the -c option
> > > > is used. I'll leave the debugging of the kexec bug to someone else.
> > >
> > > Hi Michael,
> > >
> > > Do you think we need to handle Azure Gen2 VM differently in the kexec?
> > >
> > > Or should we change the kexec_file_load() system call to retrieve the correct
> > > framebuffer address?
> >
> > I'm thinking there may be a fix in the Hyper-V FB and Hyper-V DRM drivers.
> > Commit c25a19afb81c may also be a cause of the problem -- see precursor
> > commit 3cb73bc3fa2a, which describes exactly the problem. I still need to
> > do some testing, but without that commit, kdump won't detect that it has
> > an EFI framebuffer, won't load the efifb driver, and so won't encounter the
> > problem. But we probably need to get Thomas Zimmerman to weigh in on
> > the implications of reverting c25a19afb81c.
> >
> > There's one additional variation of the problem. Assume the Hyper-V FB
> > driver is loaded (for example) during boot and moves the framebuffer. Then
> > system runs kexec as part of arming kdump during the boot sequence.
> > The most recent location of the framebuffer (and whether it is an EFI framebuffer)
> > gets picked at the time kexec runs, and is stored in the crash kernel memory area.
> > But what if the framebuffer later moves, perhaps because the Hyper-V FB driver
> > is unbound? The crash kernel memory area doesn’t get updated and kdump
> > could still have the wrong framebuffer address. This anomaly argues for the
> > commit 3cb73bc3fa2a approach of just ensuring that the efifb driver doesn't
> > load. Of course that approach means that the kdump kernel *must* contain
> > either the Hyper-V FB or Hyper-V DRM driver in order to work on a system
> > with only a framebuffer for text output. The efifb driver won't work. But
> > perhaps that's OK.
> >
> > Changing kexec (or the invoking script) to special case Hyper-V Gen 2 VMs and
> > always use kexec_load() instead of kexec_file_load() sounds like a big hack
> > to me.  And with that approach, you give up the ability to enforce loading only
> > properly signed kdump images. This is something kexec_file_load() provides
> > that kexec_load() doesn't, and is one of the main reasons that kexec_file_load()
> > was added.
> >
> > Whether the kexec_file_load() system call could be enhanced to get the
> > frame buffer information from the /dev/fb0 device, I'm not sure. That might
> > be a reasonable approach, though it still has the problem that the framebuffer
> > address could change *after* kexec_file_load() runs.
> >
> > Anyway, that's a dump of my current thoughts. I haven't reached a final
> > conclusion or recommendation yet. Comments from others on the
> > thread are welcome.
> 
> Hi!
> 
> Asking because I also had to do some digging in this area:
> 
> Do you think that the kernel can *ask* the hypervisor where the framebuffer is instead
> of relying on bios, the bootloader and/or kexec to somehow provide this information?
> 
> If hyperv doesn't provide this API, how hard it would be in your opinion to provide it?
> 
> I am asking because, I also had to debug a RHEL downstream issue where a slightly
> botched backport
> ensured that the first stage of the compressed uefi boot image, stopped passing the
> 'screen_info'
> to the second stage (the kernel itself), and as a result of this, the second stage stopped
> loading simplefb, and as a result of *this*, the PCI driver started to try to use the
> framebuffer
> range for its own use which failed and resulted in a cryptic error.
> 
> If the kernel was to just issue some form of a hypercall to ask the hypervisor where the
> framebuffer currently is,
> we could avoid a whole class of bugs similar to this.
> What do you think?
> 

I'm not aware of a way to ask Hyper-V about the framebuffer location.
I had not previously thought about such a possibility, so it's worth
thinking through. Here's how I see it: The issue is with generic drivers like
efifb (and others) that are hardcoded to read screen_info.lfb_base to
find the framebuffer. So the proposed new hypercall would need to be
made relatively early during boot, and it would update screen_info.lfb_base
to reflect the current location of the framebuffer. Hypercalls can only be
made after the setup in hyperv_init() is done. Fortunately, that's probably
before any framebuffer driver would read screen_info.lfb_base, though I'm
not completely sure.

Another factor is that the Hyper-V framebuffer is provided by the QEMU
equivalent that's embedded in the overall Hyper-V host, and not by
the hypervisor itself. The framebuffer is a VMBus device. So the Hyper-V
people would probably want getting the framebuffer location to be a
VMBus message to the framebuffer device, not a hypercall. And the VMBus
machinery isn't setup up until later -- too late, in fact, to change
screen_info.lfb_base before some generic driver reads it. So that's likely
to be a problem with the idea, though I'm speculating on what the 
Hyper-V folks would say.

The last factor is getting Hyper-V to add the feature. Somebody on the
Microsoft side would need to carry that request to the Hyper-V team.
I'm former Microsoft, but retired 1+ years ago, so I'm now just an unpaid
hobbyist contributing to the kernel because I enjoy the challenge. :-) But
I no longer have the Microsoft insider connection to the Hyper-V team.
From my past experience, getting such features added is hard, and takes
a long time (years?) to get implemented and rolled out across the Azure
fleet, unless there's some critical issue that needs to be addressed. This
kdump problem probably doesn't reach that level of criticality.

So I'm not super optimistic about the idea. But maybe your thinking
is different from what I've laid out. I'm happy to hear further discussion.

Michael

RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

[Michael Kelley]

From: Michael Kelley <mhklinux@outlook.com> Sent: Monday, February 10, 2025 1:36 PM
> 
> From: thomas.tai@oracle.com <thomas.tai@oracle.com> Sent: Monday, February 10, 2025 7:08 AM
> >
> > <snip>
> >
> > >> Then the question is why the efifb driver doesn't work in the kdump
> > >> kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
> > >> on the Oracle Linux 9.4 system, and transferred the kernel image binary
> > >> and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
> > >> efifb driver is loaded as part of the kdump kernel, and it doesn't cause
> > >> any problems. But there's an interesting difference. In the Oracle Linux
> > >> 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
> > >> the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
> > >> difference is due to differences in how the screen_info variable gets
> > >> setup in the two VMs.
> > >>
> > >> When the normal kernel starts in a freshly booted VM, Hyper-V provides
> > >> the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
> > >> FB driver or Hyper-V DRM driver has initialized, Linux has picked a
> > >> different MMIO address range and told Hyper-V to use the new
> > >> address range (which often starts at 0x40900000). A kexec does *not*
> > >> reset Hyper-V's transition to the new range, so when the efifb driver
> > >> tries to use the framebuffer at 0x40000000, the accesses trap to
> > >> Hyper-V and probably fail or timeout (I'm not sure of the details). After
> > >> the guest does some number of these bad references, Hyper-V considers
> > >> itself to be under attack from an ill-behaving guest, and throttles the
> > >> guest so that it doesn't run for a few seconds. The throttling repeats,
> > >> and results in extremely slow running in the kdump kernel.
> > >>
> > >> Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
> > >> as stored in screen_info.lfb_base gets updated to be 0x40900000. I
> > >> haven't fully debugged how that happens. But with that update, the
> > >> efifb driver is using the updated framebuffer address and it works. On
> > >> the Oracle Linux 9.4 system, that update doesn't appear to happen,
> > >> and the problem occurs.
> > >>
> > >> This in an interim update on the problem. I'm still investigating how
> > >> screen_info.lfb_base is set in the kdump kernel, and why it is different
> > >> in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
> > >> well understood, we can contemplate how to fix the problem. Undoing
> > >> the revert that is commit 2bebc3cd4870 doesn't seem like the solution
> > >> since the original code there was reported to cause many other issues.
> > >> The solution focus will likely be on how to ensure the kdump kernel gets
> > >> the correct framebuffer address so the efifb driver works, since the
> > >> framebuffer address changing is a quirk of Hyper-V behavior.
> > >>
> > >> If anyone else has insight into what's going on here, please chime in.
> > >> What I've learned so far is still somewhat tentative.
> > >>
> > > Here's what is happening. On Ubuntu 20.04, the kdump image is
> > > loaded into crash memory using the kexec command. Ubuntu 20.04
> > > has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1,
> > > and per the kexec man page, it defaults to using the older kexec_load()
> > > system call. When using kexec_load(), the contents to be loaded into
> > > crash memory is constructed in user space by the kexec command.
> > > The kexec command gets the "screen_info" settings, including the
> > > physical address of the frame buffer, via the FBIOGET_FSCREENINFO
> > > ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself
> > > with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns
> > > the updated framebuffer address. So the efifb driver loads and runs
> > > correctly.
> > >
> > > On Oracle Linux 9.4, the kdump image is also loaded with the
> > > kexec command, but from kexec-tools package version
> > > kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than
> > > the version on Ubuntu 20.04. This newer kexec defaults to using the
> > > newer kexec_file_load() system call. This system call gets the
> > > framebuffer address from the screen_info variable in the kernel, which
> > > has not been updated to reflect the new framebuffer address. Hence
> > > in the kdump kernel, the efifb driver uses the old framebuffer address,
> > > and hence the problem.
> > >
> > > To further complicate matters, the kexec on Oracle Linux 9.4 seems to
> > > have a bug when the -c option forces the use of kexec_load() instead
> > > of kexec_file_load(). As an experiment, I modified the kdumpctl shell
> > > script to add the "-c" option to kexec, but in that case the value "0x0"
> > > is passed as the framebuffer address, which is wrong. Furthermore,
> > > the " screen_info.orig_video_isVGA" value (which I mentioned earlier
> > > in connection with commit 2bebc3cd4870) is also set to 0, so the
> > > kdump kernel no longer thinks it has an EFI framebuffer. Hence the
> > > efifb driver isn't loaded, and the kdump works, though for the wrong
> > > reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4
> > > VM, then kdump works as expected, with the efifb driver being loaded
> > > and using the correct framebuffer address. So something is going wrong
> > > with kexec 2.0.28 in how it sets up the screen_info when the -c option
> > > is used. I'll leave the debugging of the kexec bug to someone else.
> >
> > Hi Michael,
> >
> > Do you think we need to handle Azure Gen2 VM differently in the kexec?
> >
> > Or should we change the kexec_file_load() system call to retrieve the correct
> > framebuffer address?
> 
> I'm thinking there may be a fix in the Hyper-V FB and Hyper-V DRM drivers.
> Commit c25a19afb81c may also be a cause of the problem -- see precursor
> commit 3cb73bc3fa2a, which describes exactly the problem. I still need to
> do some testing, but without that commit, kdump won't detect that it has
> an EFI framebuffer, won't load the efifb driver, and so won't encounter the
> problem. But we probably need to get Thomas Zimmerman to weigh in on
> the implications of reverting c25a19afb81c.
> 
> There's one additional variation of the problem. Assume the Hyper-V FB
> driver is loaded (for example) during boot and moves the framebuffer. Then
> system runs kexec as part of arming kdump during the boot sequence.
> The most recent location of the framebuffer (and whether it is an EFI framebuffer)
> gets picked at the time kexec runs, and is stored in the crash kernel memory area.
> But what if the framebuffer later moves, perhaps because the Hyper-V FB driver
> is unbound? The crash kernel memory area doesn’t get updated and kdump
> could still have the wrong framebuffer address. This anomaly argues for the
> commit 3cb73bc3fa2a approach of just ensuring that the efifb driver doesn't
> load. Of course that approach means that the kdump kernel *must* contain
> either the Hyper-V FB or Hyper-V DRM driver in order to work on a system
> with only a framebuffer for text output. The efifb driver won't work. But
> perhaps that's OK.
> 
> Changing kexec (or the invoking script) to special case Hyper-V Gen 2 VMs and
> always use kexec_load() instead of kexec_file_load() sounds like a big hack
> to me.  And with that approach, you give up the ability to enforce loading only
> properly signed kdump images. This is something kexec_file_load() provides
> that kexec_load() doesn't, and is one of the main reasons that kexec_file_load()
> was added.
> 
> Whether the kexec_file_load() system call could be enhanced to get the
> frame buffer information from the /dev/fb0 device, I'm not sure. That might
> be a reasonable approach, though it still has the problem that the framebuffer
> address could change *after* kexec_file_load() runs.
> 
> Anyway, that's a dump of my current thoughts. I haven't reached a final
> conclusion or recommendation yet. Comments from others on the
> thread are welcome.
> 

Here is some additional information about the cause, and a resolution, of
this issue. I noticed that when a Gen 2 VM in Azure first boots up, the MMIO
space assigned to the hyperv_fb driver is 0x40800000 (or something similar,
depending on what virtual PCI devices, like the Mellanox NIC, are present in
the VM). This address is different from the original base address for the
framebuffer, which is 0x40000000. But interestingly, if the hyperv_fb driver
is unbound, and then rebound to the device, the MMIO space reverts to the
original base address of 0x40000000. If kexec is run at that point to load the
kdump kernel, a subsequent kdump would work correctly.

Furthermore, I realized this morning that if the original kernel is configured
to use the hyperv_drm driver instead of hyperv_fb, the MMIO space assigned
to the hyperv_drm driver during first boot is the original base address of
0x40000000, which again would mean that kdump works correctly. So the
problem is confined to hyperv_fb.

The difference is in how hyperv_fb handles removing conflicting framebuffers vs.
how hyperv_drm does it -- i.e., the call to aperture_remove_conflicting_devices()
or aperture_remove_all_conflicting_devices(). Hyperv_drm makes the call
to remove the conflicting devices *before* calling vmbus_allocate_mmio(), while
hyperv_fb does it *after*. As a result, when hyperv_fb allocates MMIO, the
conflicting framebuffer driver (such as efifb, or simplefb) still occupies the
original MMIO space (0x40000000), and hyperv_fb can't get it. But later,
when doing an unbind/bind sequence, that space has been vacated, and
hyperv_fb gets it when doing the rebind.

My solution is to make minor changes to hvfb_getmem() so the conflicting
framebuffers are removed before allocating MMIO space. A quick test shows
that hyperv_fb then gets MMIO address 0x40000000 on the first boot, so
then kdump should not encounter any problems. In other words, let's avoid
having hyperv_fb move the framebuffer address, which is what is causing
the kdump problem.

Arguably, there's a small downside to this change in that if hvfb_getmem()
subsequently encounters an error and fails, there's no way to restore
the efifb or simplefb device that was removed. But that problem is present
in hyperv_drm, and is there even now in hyperv_fb because errors can
occur during hvfb_probe() after hvfb_getmem() is called. So I don't think
the downside is significant. The comments at the top of file
drivers/video/aperture.c strongly imply that removing the conflicting
devices *before* allocating MMIO space is the right thing to do.

I still need to try various configurations and do further testing to confirm
that everything works properly for Gen 1 VMs, and when the video
resolution has been changed either as a Hyper-V setting on the VM,
or on the Linux kernel boot line. There might still be some corner
cases where the framebuffer gets moved -- I'm not sure. But even if
that’s the case, I think this change makes things better overall.

I'll work on a patch once I'm satisfied that nothing else goes wrong.
It will probably be mid next week before I submit that patch.

Michael