[PATCH 0/2] Use hibernate callbacks when shutting down the system

[PATCH 0/2] Use hibernate callbacks when shutting down the system

[Mario Limonciello]

OEM systems that ship with Linux preloaded need to go through energy
certifications that match regulatory bodies in the regions that they
will ship.

If any of those certifications don't pass then OEMs might not be able
to ship systems preloaded in applicable regions.

Multiple models of systems are reported to fail in Linux but pass
for Windows on the exact same hardware.

By looking at a breakdown of power consumption across devices the
issue is that some devices aren't turned off when user puts the
system into S5 (shutdown).

A few years ago when I first root caused this I submitted patches
to PCI core to change the PCI policy at shutdown [1].

But through various iterations I've come up with a better approach.
Instead of changing policy in each subsystem we can instead
change our shutdown command to run hibernate callbacks for all hardware.

In order to accomplish that the last 17 kernel cycles I've been slowly
drippping changes into the hibernate code paths of all subsystems to make
sure they behave as intended.

Those are all merged as of kernel 7.4-rc1, and thus I am now ready to
start using the hibernate codepath at shutdown. This series adds that
support.

I'm not naïve; there is definitely risk involved with this change as it
will essentially affect anything running Linux that gets turned off.

To that I will say that I have been running, rebasing and iterating this
series on my personal development tree ever since December 2023.
Various rolling distro community members have also been testing iterations
of these patches in their hardware and distros.

I've also put a few guard rails in place:

1. Don't mandate hibernate support. If not compiled with it, just use
   previous shutdown paths.
2. If hibernate callback fail, emit a `pr_emerg()` message but fallback
   to previous shutdown paths.
3. Offer a kernel command line option `shutdown=legacy` that any affected
   user could use to roll back to previous behavior while we root cause
   any issues.

I did split out the kernel command line option to a second commit.  If we
later decide that we want to keep this behavior but drop the rollback knob
we can revert that commit.

Link: https://lore.kernel.org/linux-pci/20231213182656.6165-1-mario.limonciello@amd.com/#t

Signed-off-by: Mario Limonciello <superm1@kernel.org>
---
Mario Limonciello (2):
      PM: Use hibernation callbacks for system power off
      PM: Add shutdown=legacy command line option

 Documentation/admin-guide/kernel-parameters.txt |  4 ++++
 Documentation/power/shutdown-debugging.rst      |  7 ++++++-
 kernel/reboot.c                                 | 16 ++++++++++++++++
 3 files changed, 26 insertions(+), 1 deletion(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261006-superm1-s4-at-s5-f55b441d6c5b

Best regards,
--  
Mario Limonciello <superm1@kernel.org>

[PATCH 1/2] PM: Use hibernation callbacks for system power off

[Mario Limonciello]

During system power-off, device_shutdown() invokes the PCI bus shutdown
callback for each PCI device. pci_device_shutdown() first calls
pm_runtime_resume(), which may return a runtime-suspended device to D0
before invoking its driver's ->shutdown() callback. Consequently, devices
may remain in D0, consuming power or causing spurious wakeups.

Devices should instead remain in a low-power state until power is removed.
ACPI 6.5 section 7.4.2.5 states that all devices are in D3 in S4. Section
7.4.2.6 defines S5 as similar to S4 except that OSPM saves no context, so
reusing the device hibernation power-off callbacks for S5 is appropriate.

For SYSTEM_POWER_OFF, run the device PM core's suspend sequence with
PMSG_POWEROFF. When CONFIG_HIBERNATE_CALLBACKS is enabled, this invokes the
hibernation ->poweroff*() callbacks. Skip device_shutdown() after a
successful sequence so the PCI shutdown callback does not resume devices to
D0 and undo the transition. If callback support is unavailable or either
phase fails, fall back to device_shutdown() so power-off can continue.

Tested-by: Eric Naim <dnaim@cachyos.org>
Tested-by: Denis Benato <benato.denis96@gmail.com>
Tested-by: Bartu Alev <bartualev@gmail.com>
Tested-by: Merthan Karakaş <m3rthn.k@gmail.com>
Link: https://lore.kernel.org/linux-pci/20231213182656.6165-1-mario.limonciello@amd.com/
Link: https://lore.kernel.org/linux-pci/20250506041934.1409302-1-superm1@kernel.org/
Signed-off-by: Mario Limonciello <superm1@kernel.org>
---
 kernel/reboot.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/reboot.c b/kernel/reboot.c
index 9a84401adde5a..1ad8629fde4d9 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -14,6 +14,7 @@
 #include <linux/kexec.h>
 #include <linux/kmsg_dump.h>
 #include <linux/rcupdate.h>
+#include <linux/pm.h>
 #include <linux/reboot.h>
 #include <linux/sched/signal.h>
 #include <linux/suspend.h>
@@ -315,6 +316,13 @@ static void kernel_shutdown_prepare(enum system_states state)
 		(state == SYSTEM_HALT) ? SYS_HALT : SYS_POWER_OFF, NULL);
 	system_state = state;
 	usermodehelper_disable();
+#ifdef CONFIG_HIBERNATE_CALLBACKS
+	if (state == SYSTEM_POWER_OFF) {
+		if (!dpm_suspend_start(PMSG_POWEROFF) && !dpm_suspend_end(PMSG_POWEROFF))
+			return;
+		pr_emerg("Failed to power off devices, using shutdown instead.\n");
+	}
+#endif
 	device_shutdown();
 }
 /**

-- 
2.53.0

[PATCH 2/2] PM: Add shutdown=legacy command line option

[Mario Limonciello]

Powering off now uses the hibernation callback flow when it is available.
Add an escape valve for systems that fail with this flow.

The shutdown=legacy option restores the device_shutdown() path. Document
it in the kernel parameter reference and the shutdown debugging guide.

Tested-by: Bartu Alev <bartualev@gmail.com>
Tested-by: Merthan Karakaş <m3rthn.k@gmail.com>
Signed-off-by: Mario Limonciello <superm1@kernel.org>
---
Bartu specifically tested his ASUS FA507 with this series.  His laptop
got really hot in S5 before the series.  He also intentionally used
this debug knob to verify the problem comes back.

Merthan tested on his ASUS laptop and it fixed system getting hot at
S5.
---
 Documentation/admin-guide/kernel-parameters.txt |  4 ++++
 Documentation/power/shutdown-debugging.rst      |  7 ++++++-
 kernel/reboot.c                                 | 10 +++++++++-
 3 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd94..16d584bdbb73d 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6981,6 +6981,10 @@ Kernel parameters
 	shapers=	[NET]
 			Maximal number of shapers.
 
+	shutdown=legacy [KNL]
+			Use the legacy device shutdown callbacks when powering off
+			the system instead of the hibernation power-off callbacks.
+
 	show_lapic=	[APIC,X86] Advanced Programmable Interrupt Controller
 			Limit apic dumping. The parameter defines the maximal
 			number of local apics being dumped. Also it is possible
diff --git a/Documentation/power/shutdown-debugging.rst b/Documentation/power/shutdown-debugging.rst
index c510122e0bbc2..254cd3e0b5907 100644
--- a/Documentation/power/shutdown-debugging.rst
+++ b/Documentation/power/shutdown-debugging.rst
@@ -33,7 +33,7 @@ some potential options include:
 
 Kernel Command-line Parameters
 ==============================
-Add these parameters to your kernel command line:
+Add these parameters to your kernel command line to capture shutdown logs:
 
 * ``printk.always_kmsg_dump=Y``
 	* Forces the kernel to dump the entire message buffer to pstore during
@@ -41,6 +41,11 @@ Add these parameters to your kernel command line:
 * ``efi_pstore.pstore_disable=N``
 	* For EFI-based systems, ensures the EFI backend is active
 
+If the system fails while running the hibernation power-off callbacks, add
+``shutdown=legacy`` to use the legacy device shutdown callbacks instead. This
+can be used to work around the failure and confirm which callback flow caused
+it.
+
 Userspace Interaction and Log Retrieval
 =======================================
 On the next boot after a hang, pstore logs will be available in the pstore
diff --git a/kernel/reboot.c b/kernel/reboot.c
index 1ad8629fde4d9..6c80356b9a85f 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -70,6 +70,7 @@ struct sys_off_handler {
  * of that.
  */
 static bool poweroff_fallback_to_halt;
+static bool shutdown_legacy;
 
 /*
  * Temporary stub that prevents linkage failure while we're in process
@@ -317,7 +318,7 @@ static void kernel_shutdown_prepare(enum system_states state)
 	system_state = state;
 	usermodehelper_disable();
 #ifdef CONFIG_HIBERNATE_CALLBACKS
-	if (state == SYSTEM_POWER_OFF) {
+	if (state == SYSTEM_POWER_OFF && !shutdown_legacy) {
 		if (!dpm_suspend_start(PMSG_POWEROFF) && !dpm_suspend_end(PMSG_POWEROFF))
 			return;
 		pr_emerg("Failed to power off devices, using shutdown instead.\n");
@@ -1112,6 +1113,13 @@ static ssize_t hw_protection_store(struct kobject *kobj,
 static struct kobj_attribute hw_protection_attr = __ATTR_RW(hw_protection);
 #endif
 
+static int __init shutdown_setup(char *str)
+{
+	shutdown_legacy = true;
+	return 1;
+}
+__setup("shutdown=legacy", shutdown_setup);
+
 static int __init reboot_setup(char *str)
 {
 	for (;;) {

-- 
2.53.0

Re: [PATCH 2/2] PM: Add shutdown=legacy command line option

[Randy Dunlap]

On 11/2/26 5:54 AM, Mario Limonciello wrote:
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 0b3c140babd94..16d584bdbb73d 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -6981,6 +6981,10 @@ Kernel parameters
>  	shapers=	[NET]
>  			Maximal number of shapers.
>  
> +	shutdown=legacy [KNL]
> +			Use the legacy device shutdown callbacks when powering off
> +			the system instead of the hibernation power-off callbacks.
> +
>  	show_lapic=	[APIC,X86] Advanced Programmable Interrupt Controller
>  			Limit apic dumping. The parameter defines the maximal
>  			number of local apics being dumped. Also it is possible

Alphabetical order, please. I.e., add shutdown= after show_lapic=.

-- 
~Randy

Re: [PATCH 2/2] PM: Add shutdown=legacy command line option

[Mario Limonciello]

On 11/2/26 11:17, Randy Dunlap wrote:
> 
> 
> On 11/2/26 5:54 AM, Mario Limonciello wrote:
>> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
>> index 0b3c140babd94..16d584bdbb73d 100644
>> --- a/Documentation/admin-guide/kernel-parameters.txt
>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>> @@ -6981,6 +6981,10 @@ Kernel parameters
>>   	shapers=	[NET]
>>   			Maximal number of shapers.
>>   
>> +	shutdown=legacy [KNL]
>> +			Use the legacy device shutdown callbacks when powering off
>> +			the system instead of the hibernation power-off callbacks.
>> +
>>   	show_lapic=	[APIC,X86] Advanced Programmable Interrupt Controller
>>   			Limit apic dumping. The parameter defines the maximal
>>   			number of local apics being dumped. Also it is possible
> 
> Alphabetical order, please. I.e., add shutdown= after show_lapic=.
> 

Ack, thanks, will change in v2.