reserve_mem command line option causing a reboot but not by the kernel

reserve_mem command line option causing a reboot but not by the kernel

[Steven Rostedt]

This is mostly just an FYI,

I upgraded both my server and my workstation to 6.12-rc1 to try out the
new reserve_mem and my persistent trace instance. My server which is a
Dell PowerEdge T430 (bought here: https://www.newegg.com/dell-poweredge-t430-tower/p/2NS-0008-70EW5)
worked flawlessly. I enabled tracing, did a reboot and it contained the
trace on the following bootup. This used the following command line:

  reserve_mem=112M:2097152:trace trace_instance=boot_mapped@trace,console

Then I tried it out on my workstation. This is a machine I built years
ago. It has an American Megatrends Inc. BIOS (no UEFI). Since it only
has 8 CPUs, I had the command line of:

  reserve_mem=20M:2097152:trace trace_instance=boot_mapped@trace,console

Rebooted, and right after grub (legacy grub) loaded the kernel, the
machine rebooted again. I removed the "reserve_mem" option, and the
kernel booted fine with the rest of the options.

I started debugging this, but something else happened. When the system
rebooted, I picked an older kernel. But it rebooted too! That is,
booting a kernel without the "reserve_mem" code but the "reserve_mem"
option also had this issue! When installing, I did an "update_grub"
that added the command line to every kernel. It rebooted on any kernel
that had "reserve_mem" as one of the kernel command line options. I
tried kernels down to 6.6.

I then renamed the option in the 6.12-rc1 kernel as "reserve_mm" and
tried again. It booted fine! Although I found that my BIOS on this
machine does not keep memory consistent across reboots so the option is
useless.

The strange thing here is why "reserve_mem" on the kernel command line
causes the machine to reboot? It reboots very early. I even hooked up a
serial and enabled earlyprintk, and it doesn't ever print anything when
the reboot happens.

I'm wondering if this has something to do with legacy grub? Perhaps it
is parsing the kernel command line parameter too and does something
special with "reserve_mem"? My server is using grub2, and so are most
of the other machines I tested on. The ones without grub2 had extlinux,
which worked fine too.

I don't have time to debug further. Maybe I need to look at the legacy
grub code. I just found this interesting and decided to share. Perhaps
someone else might hit this too?

-- Steve

Re: reserve_mem command line option causing a reboot but not by the kernel

[Steven Rostedt]

On Sun, 6 Oct 2024 13:32:33 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> I don't have time to debug further. Maybe I need to look at the legacy
> grub code. I just found this interesting and decided to share. Perhaps
> someone else might hit this too?

I found that one of my old test machines also uses legacy grub, and it too
has the same issue.

It appears that grub does indeed search for "mem=" (without looking for
something before it) in the kernel command line.

  https://github.com/jezze/grub-legacy/blob/master/stage2/boot.c#L275

Thus, "reserve_mem=" (or any other "mem=" option) cannot be used with legacy
grub installs :-p

-- Steve

Re: reserve_mem command line option causing a reboot but not by the kernel

[Steven Rostedt]

On Tue, 8 Oct 2024 13:22:59 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> It appears that grub does indeed search for "mem=" (without looking for
> something before it) in the kernel command line.
> 
>   https://github.com/jezze/grub-legacy/blob/master/stage2/boot.c#L275
> 
> Thus, "reserve_mem=" (or any other "mem=" option) cannot be used with legacy
> grub installs :-p

But it only looks at the first instance of "mem=". So I added "testmem=16G"
to the command line (giving lots of memory to fool grub, and the kernel
fixes it after the decompression) before the "reserve_mem", and now I can
use "reserve_mem" on this test box (and it preserves the memory across
boots!).

           <...>-2378    [000] dN.1.   124.665568: clear_local_APIC <-disable_local_APIC
           <...>-2378    [000] dN.1.   124.665569: mcheck_cpu_clear <-native_stop_other_cpus
           <...>-2378    [000] dN.1.   124.665569: mce_intel_feature_clear <-native_stop_other_cpus
           <...>-2378    [000] dN.1.   124.665569: lmce_supported <-mce_intel_feature_clear
           <...>-2378    [000] dN.1.   124.665570: lapic_shutdown <-native_machine_shutdown
           <...>-2378    [000] dN.1.   124.665570: clear_local_APIC <-lapic_shutdown
           <...>-2378    [000] dN.1.   124.665570: restore_boot_irq_mode <-native_machine_shutdown
           <...>-2378    [000] dN.1.   124.665571: native_restore_boot_irq_mode <-native_machine_shutdown
           <...>-2378    [000] dN.1.   124.665571: disconnect_bsp_APIC <-native_machine_shutdown
           <...>-2378    [000] dN.1.   124.665572: hpet_disable <-native_machine_shutdown
           <...>-2378    [000] dN.1.   124.665572: iommu_shutdown_noop <-native_machine_shutdown
           <...>-2378    [000] dN.1.   124.665572: native_machine_emergency_restart <-__do_sys_reboot
           <...>-2378    [000] dN.1.   124.665572: tboot_shutdown <-native_machine_emergency_restart
           <...>-2378    [000] dN.1.   124.665573: machine_real_restart <-native_machine_emergency_restart
           <...>-2378    [000] dN.1.   124.665573: _raw_spin_lock <-machine_real_restart
           <...>-2378    [000] dN.1.   124.665574: preempt_count_add <-_raw_spin_lock
           <...>-2378    [000] dN.2.   124.665574: rtc_cmos_write <-machine_real_restart
           <...>-2378    [000] dN.2.   124.665576: _raw_spin_unlock <-machine_real_restart
           <...>-2378    [000] dN.2.   124.665577: preempt_count_sub <-_raw_spin_unlock
           <...>-2378    [000] dN.1.   124.665577: load_trampoline_pgtable <-machine_real_restart
           <...>-2378    [000] dN.1.   124.665578: __flush_tlb_all <-machine_real_restart
           <...>-2378    [000] dN.1.   124.665578: native_flush_tlb_global <-__flush_tlb_all

But this machine did find a bug in the code where I need to send out a fix for.

-- Steve