[PATCH 0/2] Use hibernate callbacks when shutting down the system

[PATCH 0/2] Use hibernate callbacks when shutting down the system

[Mario Limonciello]

OEM systems that ship with Linux preloaded need to go through energy
certifications that match regulatory bodies in the regions that they
will ship.

If any of those certifications don't pass then OEMs might not be able
to ship systems preloaded in applicable regions.

Multiple models of systems are reported to fail in Linux but pass
for Windows on the exact same hardware.

By looking at a breakdown of power consumption across devices the
issue is that some devices aren't turned off when user puts the
system into S5 (shutdown).

A few years ago when I first root caused this I submitted patches
to PCI core to change the PCI policy at shutdown [1].

But through various iterations I've come up with a better approach.
Instead of changing policy in each subsystem we can instead
change our shutdown command to run hibernate callbacks for all hardware.

In order to accomplish that the last 17 kernel cycles I've been slowly
drippping changes into the hibernate code paths of all subsystems to make
sure they behave as intended.

Those are all merged as of kernel 7.4-rc1, and thus I am now ready to
start using the hibernate codepath at shutdown. This series adds that
support.

I'm not naïve; there is definitely risk involved with this change as it
will essentially affect anything running Linux that gets turned off.

To that I will say that I have been running, rebasing and iterating this
series on my personal development tree ever since December 2023.
Various rolling distro community members have also been testing iterations
of these patches in their hardware and distros.

I've also put a few guard rails in place:

1. Don't mandate hibernate support. If not compiled with it, just use
   previous shutdown paths.
2. If hibernate callback fail, emit a `pr_emerg()` message but fallback
   to previous shutdown paths.
3. Offer a kernel command line option `shutdown=legacy` that any affected
   user could use to roll back to previous behavior while we root cause
   any issues.

I did split out the kernel command line option to a second commit.  If we
later decide that we want to keep this behavior but drop the rollback knob
we can revert that commit.

Link: https://lore.kernel.org/linux-pci/20231213182656.6165-1-mario.limonciello@amd.com/#t

Signed-off-by: Mario Limonciello <superm1@kernel.org>
---
Mario Limonciello (2):
      PM: Use hibernation callbacks for system power off
      PM: Add shutdown=legacy command line option

 Documentation/admin-guide/kernel-parameters.txt |  4 ++++
 Documentation/power/shutdown-debugging.rst      |  7 ++++++-
 kernel/reboot.c                                 | 16 ++++++++++++++++
 3 files changed, 26 insertions(+), 1 deletion(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261006-superm1-s4-at-s5-f55b441d6c5b

Best regards,
--  
Mario Limonciello <superm1@kernel.org>

[PATCH 1/2] PM: Use hibernation callbacks for system power off

[Mario Limonciello]

During system power-off, device_shutdown() invokes the PCI bus shutdown
callback for each PCI device. pci_device_shutdown() first calls
pm_runtime_resume(), which may return a runtime-suspended device to D0
before invoking its driver's ->shutdown() callback. Consequently, devices
may remain in D0, consuming power or causing spurious wakeups.

Devices should instead remain in a low-power state until power is removed.
ACPI 6.5 section 7.4.2.5 states that all devices are in D3 in S4. Section
7.4.2.6 defines S5 as similar to S4 except that OSPM saves no context, so
reusing the device hibernation power-off callbacks for S5 is appropriate.

For SYSTEM_POWER_OFF, run the device PM core's suspend sequence with
PMSG_POWEROFF. When CONFIG_HIBERNATE_CALLBACKS is enabled, this invokes the
hibernation ->poweroff*() callbacks. Skip device_shutdown() after a
successful sequence so the PCI shutdown callback does not resume devices to
D0 and undo the transition. If callback support is unavailable or either
phase fails, fall back to device_shutdown() so power-off can continue.

Tested-by: Eric Naim <dnaim@cachyos.org>
Tested-by: Denis Benato <benato.denis96@gmail.com>
Tested-by: Bartu Alev <bartualev@gmail.com>
Tested-by: Merthan Karakaş <m3rthn.k@gmail.com>
Link: https://lore.kernel.org/linux-pci/20231213182656.6165-1-mario.limonciello@amd.com/
Link: https://lore.kernel.org/linux-pci/20250506041934.1409302-1-superm1@kernel.org/
Signed-off-by: Mario Limonciello <superm1@kernel.org>
---
 kernel/reboot.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/reboot.c b/kernel/reboot.c
index 9a84401adde5a..1ad8629fde4d9 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -14,6 +14,7 @@
 #include <linux/kexec.h>
 #include <linux/kmsg_dump.h>
 #include <linux/rcupdate.h>
+#include <linux/pm.h>
 #include <linux/reboot.h>
 #include <linux/sched/signal.h>
 #include <linux/suspend.h>
@@ -315,6 +316,13 @@ static void kernel_shutdown_prepare(enum system_states state)
 		(state == SYSTEM_HALT) ? SYS_HALT : SYS_POWER_OFF, NULL);
 	system_state = state;
 	usermodehelper_disable();
+#ifdef CONFIG_HIBERNATE_CALLBACKS
+	if (state == SYSTEM_POWER_OFF) {
+		if (!dpm_suspend_start(PMSG_POWEROFF) && !dpm_suspend_end(PMSG_POWEROFF))
+			return;
+		pr_emerg("Failed to power off devices, using shutdown instead.\n");
+	}
+#endif
 	device_shutdown();
 }
 /**

-- 
2.53.0

[PATCH 2/2] PM: Add shutdown=legacy command line option

[Mario Limonciello]

Powering off now uses the hibernation callback flow when it is available.
Add an escape valve for systems that fail with this flow.

The shutdown=legacy option restores the device_shutdown() path. Document
it in the kernel parameter reference and the shutdown debugging guide.

Tested-by: Bartu Alev <bartualev@gmail.com>
Tested-by: Merthan Karakaş <m3rthn.k@gmail.com>
Signed-off-by: Mario Limonciello <superm1@kernel.org>
---
Bartu specifically tested his ASUS FA507 with this series.  His laptop
got really hot in S5 before the series.  He also intentionally used
this debug knob to verify the problem comes back.

Merthan tested on his ASUS laptop and it fixed system getting hot at
S5.
---
 Documentation/admin-guide/kernel-parameters.txt |  4 ++++
 Documentation/power/shutdown-debugging.rst      |  7 ++++++-
 kernel/reboot.c                                 | 10 +++++++++-
 3 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd94..16d584bdbb73d 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6981,6 +6981,10 @@ Kernel parameters
 	shapers=	[NET]
 			Maximal number of shapers.
 
+	shutdown=legacy [KNL]
+			Use the legacy device shutdown callbacks when powering off
+			the system instead of the hibernation power-off callbacks.
+
 	show_lapic=	[APIC,X86] Advanced Programmable Interrupt Controller
 			Limit apic dumping. The parameter defines the maximal
 			number of local apics being dumped. Also it is possible
diff --git a/Documentation/power/shutdown-debugging.rst b/Documentation/power/shutdown-debugging.rst
index c510122e0bbc2..254cd3e0b5907 100644
--- a/Documentation/power/shutdown-debugging.rst
+++ b/Documentation/power/shutdown-debugging.rst
@@ -33,7 +33,7 @@ some potential options include:
 
 Kernel Command-line Parameters
 ==============================
-Add these parameters to your kernel command line:
+Add these parameters to your kernel command line to capture shutdown logs:
 
 * ``printk.always_kmsg_dump=Y``
 	* Forces the kernel to dump the entire message buffer to pstore during
@@ -41,6 +41,11 @@ Add these parameters to your kernel command line:
 * ``efi_pstore.pstore_disable=N``
 	* For EFI-based systems, ensures the EFI backend is active
 
+If the system fails while running the hibernation power-off callbacks, add
+``shutdown=legacy`` to use the legacy device shutdown callbacks instead. This
+can be used to work around the failure and confirm which callback flow caused
+it.
+
 Userspace Interaction and Log Retrieval
 =======================================
 On the next boot after a hang, pstore logs will be available in the pstore
diff --git a/kernel/reboot.c b/kernel/reboot.c
index 1ad8629fde4d9..6c80356b9a85f 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -70,6 +70,7 @@ struct sys_off_handler {
  * of that.
  */
 static bool poweroff_fallback_to_halt;
+static bool shutdown_legacy;
 
 /*
  * Temporary stub that prevents linkage failure while we're in process
@@ -317,7 +318,7 @@ static void kernel_shutdown_prepare(enum system_states state)
 	system_state = state;
 	usermodehelper_disable();
 #ifdef CONFIG_HIBERNATE_CALLBACKS
-	if (state == SYSTEM_POWER_OFF) {
+	if (state == SYSTEM_POWER_OFF && !shutdown_legacy) {
 		if (!dpm_suspend_start(PMSG_POWEROFF) && !dpm_suspend_end(PMSG_POWEROFF))
 			return;
 		pr_emerg("Failed to power off devices, using shutdown instead.\n");
@@ -1112,6 +1113,13 @@ static ssize_t hw_protection_store(struct kobject *kobj,
 static struct kobj_attribute hw_protection_attr = __ATTR_RW(hw_protection);
 #endif
 
+static int __init shutdown_setup(char *str)
+{
+	shutdown_legacy = true;
+	return 1;
+}
+__setup("shutdown=legacy", shutdown_setup);
+
 static int __init reboot_setup(char *str)
 {
 	for (;;) {

-- 
2.53.0

Re: [PATCH 2/2] PM: Add shutdown=legacy command line option

[Randy Dunlap]

On 11/2/26 5:54 AM, Mario Limonciello wrote:
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 0b3c140babd94..16d584bdbb73d 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -6981,6 +6981,10 @@ Kernel parameters
>  	shapers=	[NET]
>  			Maximal number of shapers.
>  
> +	shutdown=legacy [KNL]
> +			Use the legacy device shutdown callbacks when powering off
> +			the system instead of the hibernation power-off callbacks.
> +
>  	show_lapic=	[APIC,X86] Advanced Programmable Interrupt Controller
>  			Limit apic dumping. The parameter defines the maximal
>  			number of local apics being dumped. Also it is possible

Alphabetical order, please. I.e., add shutdown= after show_lapic=.

-- 
~Randy

Re: [PATCH 2/2] PM: Add shutdown=legacy command line option

[Mario Limonciello]

On 11/2/26 11:17, Randy Dunlap wrote:
> 
> 
> On 11/2/26 5:54 AM, Mario Limonciello wrote:
>> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
>> index 0b3c140babd94..16d584bdbb73d 100644
>> --- a/Documentation/admin-guide/kernel-parameters.txt
>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>> @@ -6981,6 +6981,10 @@ Kernel parameters
>>   	shapers=	[NET]
>>   			Maximal number of shapers.
>>   
>> +	shutdown=legacy [KNL]
>> +			Use the legacy device shutdown callbacks when powering off
>> +			the system instead of the hibernation power-off callbacks.
>> +
>>   	show_lapic=	[APIC,X86] Advanced Programmable Interrupt Controller
>>   			Limit apic dumping. The parameter defines the maximal
>>   			number of local apics being dumped. Also it is possible
> 
> Alphabetical order, please. I.e., add shutdown= after show_lapic=.
> 

Ack, thanks, will change in v2.

Re: [PATCH 0/2] Use hibernate callbacks when shutting down the system

[Rafael J. Wysocki (Intel)]

On Mon, Nov 2, 2026 at 2:54 PM Mario Limonciello <superm1@kernel.org> wrote:
>
> OEM systems that ship with Linux preloaded need to go through energy
> certifications that match regulatory bodies in the regions that they
> will ship.
>
> If any of those certifications don't pass then OEMs might not be able
> to ship systems preloaded in applicable regions.
>
> Multiple models of systems are reported to fail in Linux but pass
> for Windows on the exact same hardware.
>
> By looking at a breakdown of power consumption across devices the
> issue is that some devices aren't turned off when user puts the
> system into S5 (shutdown).
>
> A few years ago when I first root caused this I submitted patches
> to PCI core to change the PCI policy at shutdown [1].
>
> But through various iterations I've come up with a better approach.
> Instead of changing policy in each subsystem we can instead
> change our shutdown command to run hibernate callbacks for all hardware.
>
> In order to accomplish that the last 17 kernel cycles I've been slowly
> drippping changes into the hibernate code paths of all subsystems to make
> sure they behave as intended.
>
> Those are all merged as of kernel 7.4-rc1, and thus I am now ready to
> start using the hibernate codepath at shutdown. This series adds that
> support.
>
> I'm not naïve; there is definitely risk involved with this change as it
> will essentially affect anything running Linux that gets turned off.
>
> To that I will say that I have been running, rebasing and iterating this
> series on my personal development tree ever since December 2023.
> Various rolling distro community members have also been testing iterations
> of these patches in their hardware and distros.
>
> I've also put a few guard rails in place:
>
> 1. Don't mandate hibernate support. If not compiled with it, just use
>    previous shutdown paths.
> 2. If hibernate callback fail, emit a `pr_emerg()` message but fallback
>    to previous shutdown paths.
> 3. Offer a kernel command line option `shutdown=legacy` that any affected
>    user could use to roll back to previous behavior while we root cause
>    any issues.
>
> I did split out the kernel command line option to a second commit.  If we
> later decide that we want to keep this behavior but drop the rollback knob
> we can revert that commit.
>
> Link: https://lore.kernel.org/linux-pci/20231213182656.6165-1-mario.limonciello@amd.com/#t

I have a couple of concerns regarding this, after looking at the patches.

First off, dpm_suspend_start() seems to be called during shutdown
while user space is running.  That is risky because at least some
things in the power-off code path assume that user space has been
frozen.  I think that freezing tasks, along the lines of what system
suspend does, before running dpm_suspend_start(), is requisite or you
will get into trouble with people who run fuzzers across shutdown or
similar.

The other concern is that the "legacy" shutdown should be the default
or people will get confused.  Arguably, it is not even necessary to
try to change that in many cases.

Of course, this means that the new shutdown arrangement will not be
used to start with, so there needs to be a plan to migrate users to
this new order and I would start with the systems where it actually is
an improvement.

I would try to add the shutdown= option upfront in the first patch and
make it take one of two options, "classic" (the regular shutdown
callbacks) or "pm-aware" (hibernation callbacks etc) with the first
one being the default at least to start with.

Then, you can add a DMI quirk table or similar with a list of systems
where "pm-aware" works better than "classic" and expand its use going
forward.  My expectation would be that, at one point in time, the
majority of x86 platforms using ACPI would be better off with
"pm-aware" than with "classic", but I'm not sure that this point in
time is today.  Beyond that, it is really hard to say how cards will
fall.

Re: [PATCH 0/2] Use hibernate callbacks when shutting down the system

[Mario Limonciello]

On 11/6/26 10:57, Rafael J. Wysocki (Intel) wrote:
> On Mon, Nov 2, 2026 at 2:54 PM Mario Limonciello <superm1@kernel.org> wrote:
>>
>> OEM systems that ship with Linux preloaded need to go through energy
>> certifications that match regulatory bodies in the regions that they
>> will ship.
>>
>> If any of those certifications don't pass then OEMs might not be able
>> to ship systems preloaded in applicable regions.
>>
>> Multiple models of systems are reported to fail in Linux but pass
>> for Windows on the exact same hardware.
>>
>> By looking at a breakdown of power consumption across devices the
>> issue is that some devices aren't turned off when user puts the
>> system into S5 (shutdown).
>>
>> A few years ago when I first root caused this I submitted patches
>> to PCI core to change the PCI policy at shutdown [1].
>>
>> But through various iterations I've come up with a better approach.
>> Instead of changing policy in each subsystem we can instead
>> change our shutdown command to run hibernate callbacks for all hardware.
>>
>> In order to accomplish that the last 17 kernel cycles I've been slowly
>> drippping changes into the hibernate code paths of all subsystems to make
>> sure they behave as intended.
>>
>> Those are all merged as of kernel 7.4-rc1, and thus I am now ready to
>> start using the hibernate codepath at shutdown. This series adds that
>> support.
>>
>> I'm not naïve; there is definitely risk involved with this change as it
>> will essentially affect anything running Linux that gets turned off.
>>
>> To that I will say that I have been running, rebasing and iterating this
>> series on my personal development tree ever since December 2023.
>> Various rolling distro community members have also been testing iterations
>> of these patches in their hardware and distros.
>>
>> I've also put a few guard rails in place:
>>
>> 1. Don't mandate hibernate support. If not compiled with it, just use
>>     previous shutdown paths.
>> 2. If hibernate callback fail, emit a `pr_emerg()` message but fallback
>>     to previous shutdown paths.
>> 3. Offer a kernel command line option `shutdown=legacy` that any affected
>>     user could use to roll back to previous behavior while we root cause
>>     any issues.
>>
>> I did split out the kernel command line option to a second commit.  If we
>> later decide that we want to keep this behavior but drop the rollback knob
>> we can revert that commit.
>>
>> Link: https://lore.kernel.org/linux-pci/20231213182656.6165-1-mario.limonciello@amd.com/#t
> 
> I have a couple of concerns regarding this, after looking at the patches.
> 
> First off, dpm_suspend_start() seems to be called during shutdown
> while user space is running.  That is risky because at least some
> things in the power-off code path assume that user space has been
> frozen.  I think that freezing tasks, along the lines of what system
> suspend does, before running dpm_suspend_start(), is requisite or you
> will get into trouble with people who run fuzzers across shutdown or
> similar.

Hmm.  I'll see about adding that to this path, thanks.

> 
> The other concern is that the "legacy" shutdown should be the default
> or people will get confused.  Arguably, it is not even necessary to
> try to change that in many cases.
> 
> Of course, this means that the new shutdown arrangement will not be
> used to start with, so there needs to be a plan to migrate users to
> this new order and I would start with the systems where it actually is
> an improvement.

It's improved every system I've tried and measured; but the level of 
improvement comes down to what hardware is contained.

The most demonstrable improvement comes from systems with dGPUs.

> 
> I would try to add the shutdown= option upfront in the first patch and
> make it take one of two options, "classic" (the regular shutdown
> callbacks) or "pm-aware" (hibernation callbacks etc) with the first
> one being the default at least to start with.
> 

OK.

> Then, you can add a DMI quirk table or similar with a list of systems
> where "pm-aware" works better than "classic" and expand its use going
> forward.  My expectation would be that, at one point in time, the
> majority of x86 platforms using ACPI would be better off with
> "pm-aware" than with "classic", but I'm not sure that this point in
> time is today.  Beyond that, it is really hard to say how cards will
> fall.

How would you feel about a heuristic to turn it on for anything with > 1 
GPU/accelerator for now?  I think that's the biggest bang for the buck 
right now.

But deciding when/how to turn it on for more systems than that I think 
is going to require a wider call for testing.