mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec
@ 2026-10-28  0:14 Jez Ng via B4 Relay
  2026-10-28  0:14 ` [PATCH 1/2] thermal: intel: hfi: Disable HFI on shutdown and kexec Jez Ng via B4 Relay
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Jez Ng via B4 Relay @ 2026-10-28  0:14 UTC (permalink / raw)
  To: Rafael J. Wysocki, Daniel Lezcano, Zhang Rui, Lukasz Luba,
	Ricardo Neri, Len Brown, Stanislaw Gruszka
  Cc: Rafael J. Wysocki, linux-pm, linux-kernel, stable, Andrew James,
	John Ranzie, Jez Ng

HFI hardware writes updates to a table at a physical address the kernel
programs. Nothing disables HFI on shutdown, and since commit b33f3d2677b8
("thermal: intel: hfi: Enable HFI only when required") a kernel that
starts without thermal netlink clients does not touch HFI at all. So
after a kexec, hardware keeps writing into the previous kernel's table,
which the new kernel treats as free memory.

We hit this on dual-socket Sapphire Rapids hosts (Xeon Platinum 8462Y+)
that boot a Debian 6.1 kernel and kexec into a 7.0 kernel. The new
kernel crashed in find_inode_fast() when the inode hash table landed on
one of the old table pages. Reading MSR 0x17d0 on the running 7.0
kernel showed both packages still pointing at the 6.1 kernel's tables,
and the corruption had the HFI table layout: a timestamp, a changed
flag, then one row per CPU.

Patch 1 disables HFI in a syscore shutdown callback, which covers
reboot, power off and kexec, and keeps it disabled afterwards: on kexec,
user space runs until machine_shutdown(), so a new thermal netlink
client could otherwise enable HFI again. Patch 2 makes a starting kernel
disable HFI if the previous kernel left it enabled. Patch 2 can only
narrow the window: hardware can still write between kernel start and
the moment the package's first CPU reaches the thermal online callback,
which was about 30 seconds into boot on these hosts.

This is the kexec counterpart of the hibernation corruption fixed by
commit 97566d09fd02 ("thermal: intel: hfi: Add syscore callbacks for
system-wide PM"), which disables HFI for suspend and hibernation but not
for shutdown.

Tested on one of the hosts above with both patches applied to this base,
alternating 100 kexecs (kexec_file_load + reboot(LINUX_REBOOT_CMD_KEXEC))
between the patched kernel and an unpatched 7.0 kernel. Before each
kexec, a thermal netlink subscriber enabled HFI on both packages. All
100 passed:

 - Patch 1: after each of the 50 kexecs out of the patched kernel, the
   unpatched kernel read HW_FEEDBACK_CONFIG = 0 on both packages.

 - Patch 2: after each of the 50 kexecs into the patched kernel, it
   logged "HFI left enabled on package N, disabled" for both packages
   and read HW_FEEDBACK_CONFIG = 0.

A test-only 5 second sleep after the shutdown callback, with a thermal
netlink client subscribing during it, showed that patch 1 keeps HFI
disabled. Without the hfi_shutdown_done check, the next kernel found HFI
enabled again.

Thanks to Andrew James and John Ranzie for reviewing.

The investigation and testing were done via Opus 5.5, as were the
drafting of the patches and changelogs. I reviewed all of it.

---
Jez Ng (2):
      thermal: intel: hfi: Disable HFI on shutdown and kexec
      thermal: intel: hfi: Disable HFI left enabled by a previous kernel

 drivers/thermal/intel/intel_hfi.c | 71 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 63 insertions(+), 8 deletions(-)
---
base-commit: 70f331d89564126fdcaaa37f25194fe391d8f3fa
change-id: 20261027-hfi-kexec-e7c4b7f584be

Best regards,
--  
Jez Ng <jez@sfcompute.com>



^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 1/2] thermal: intel: hfi: Disable HFI on shutdown and kexec
  2026-10-28  0:14 [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Jez Ng via B4 Relay
@ 2026-10-28  0:14 ` Jez Ng via B4 Relay
  2026-11-19  5:42   ` Ricardo Neri
  2026-10-28  0:14 ` [PATCH 2/2] thermal: intel: hfi: Disable HFI left enabled by a previous kernel Jez Ng via B4 Relay
  2026-11-13 15:26 ` [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Rafael J. Wysocki (Intel)
  2 siblings, 1 reply; 7+ messages in thread
From: Jez Ng via B4 Relay @ 2026-10-28  0:14 UTC (permalink / raw)
  To: Rafael J. Wysocki, Daniel Lezcano, Zhang Rui, Lukasz Luba,
	Ricardo Neri, Len Brown, Stanislaw Gruszka
  Cc: Rafael J. Wysocki, linux-pm, linux-kernel, stable, Andrew James,
	John Ranzie, Jez Ng

From: Jez Ng <jez@sfcompute.com>

HFI is enabled per package by writing the physical address of a table to
MSR_IA32_HW_FEEDBACK_PTR and setting the enable bit in
MSR_IA32_HW_FEEDBACK_CONFIG. From then on, hardware writes performance and
energy efficiency updates to that table on its own.

Nothing disables HFI when the kernel shuts down. After a kexec, hardware
keeps writing to the old kernel's table, which is now ordinary free memory
in the new kernel. The new kernel does not reprogram HFI unless a thermal
netlink client subscribes to events, so the writes continue indefinitely.

This was observed on dual-socket Sapphire Rapids systems (Xeon Platinum
8462Y+) that boot a Debian 6.1 kernel and kexec into a 7.0 kernel. Both
packages kept writing HFI updates (a timestamp, a changed flag and one
row per CPU) into the old kernel's table pages. When the inode hash table
landed on one of those pages, the new kernel oopsed in find_inode_fast().
Disabling HFI before kexec made the writes stop.

Add a syscore shutdown callback that disables HFI on every package with an
online CPU. syscore_shutdown() runs on reboot, power off and kexec, after
CPU hotplug is disabled and while the other CPUs are still online, so each
package can be reached with smp_call_function_single().

Do not enable HFI again once the callback has run. On kexec, user space
keeps running on the other CPUs until machine_shutdown() stops them, and
kernel_kexec() enables CPU hotplug again after syscore_shutdown(). A new
thermal netlink client, or the first CPU of a package coming online, would
otherwise enable HFI after it was disabled.

Skip packages where HFI is already disabled. hfi_disable() waits up to
2 ms for hardware to set HFI_UPDATED, which a package that is not
running HFI may never do.

Fixes: ab09b0744a99 ("thermal: intel: hfi: Enable notification interrupt")
Cc: stable@vger.kernel.org
Reviewed-by: Andrew James <ajames@sfcompute.com>
Reviewed-by: John Ranzie <jranzie@sfcompute.com>
Signed-off-by: Jez Ng <jez@sfcompute.com>
---
 drivers/thermal/intel/intel_hfi.c | 50 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 48 insertions(+), 2 deletions(-)

diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c
index 3273b8fe3d4d..8ff094eccbaf 100644
--- a/drivers/thermal/intel/intel_hfi.c
+++ b/drivers/thermal/intel/intel_hfi.c
@@ -161,6 +161,7 @@ static DEFINE_PER_CPU(struct hfi_cpu_info, hfi_cpu_info) = { .index = -1 };
 
 static int max_hfi_instances;
 static int hfi_clients_nr;
+static bool hfi_shutdown_done;
 static struct hfi_instance *hfi_instances;
 
 static struct hfi_features hfi_features;
@@ -398,6 +399,19 @@ static void hfi_disable(void)
 	}
 }
 
+/* Caller must hold hfi_instance_lock. */
+static bool hfi_disable_if_enabled(void)
+{
+	u64 msr_val;
+
+	rdmsrq(MSR_IA32_HW_FEEDBACK_CONFIG, msr_val);
+	if (!(msr_val & HW_FEEDBACK_CONFIG_HFI_ENABLE_BIT))
+		return false;
+
+	hfi_disable();
+	return true;
+}
+
 /**
  * intel_hfi_online() - Enable HFI on @cpu
  * @cpu:	CPU in which the HFI will be enabled
@@ -483,7 +497,8 @@ void intel_hfi_online(unsigned int cpu)
 	 * Enable this HFI instance if this is its first online CPU and
 	 * there are user-space clients of thermal events.
 	 */
-	if (cpumask_weight(hfi_instance->cpus) == 1 && hfi_clients_nr > 0) {
+	if (cpumask_weight(hfi_instance->cpus) == 1 && hfi_clients_nr > 0 &&
+	    !hfi_shutdown_done) {
 		hfi_set_hw_table(hfi_instance);
 		hfi_enable();
 	}
@@ -593,6 +608,11 @@ static void hfi_disable_instance(void *ptr)
 	hfi_disable();
 }
 
+static void hfi_shutdown_instance(void *ptr)
+{
+	hfi_disable_if_enabled();
+}
+
 static void hfi_syscore_resume(void *data)
 {
 	/* This code runs only on the boot CPU. */
@@ -612,9 +632,35 @@ static int hfi_syscore_suspend(void *data)
 	return 0;
 }
 
+static void hfi_syscore_shutdown(void *data)
+{
+	struct hfi_instance *hfi_instance;
+	int i;
+
+	/*
+	 * Hardware keeps writing to the HFI table after a kexec, into memory
+	 * that the next kernel owns. Disable HFI in every package, and keep it
+	 * disabled: user space runs until machine_shutdown() stops the other
+	 * CPUs, and kexec enables CPU hotplug again, so a new thermal netlink
+	 * client or a CPU coming online could otherwise enable it.
+	 */
+	mutex_lock(&hfi_instance_lock);
+	hfi_shutdown_done = true;
+	for (i = 0; i < max_hfi_instances; i++) {
+		hfi_instance = &hfi_instances[i];
+		if (cpumask_empty(hfi_instance->cpus))
+			continue;
+
+		smp_call_function_single(cpumask_any(hfi_instance->cpus),
+					 hfi_shutdown_instance, hfi_instance, true);
+	}
+	mutex_unlock(&hfi_instance_lock);
+}
+
 static const struct syscore_ops hfi_pm_ops = {
 	.resume = hfi_syscore_resume,
 	.suspend = hfi_syscore_suspend,
+	.shutdown = hfi_syscore_shutdown,
 };
 
 static struct syscore hfi_pm = {
@@ -649,7 +695,7 @@ static int hfi_thermal_notify(struct notifier_block *nb, unsigned long state,
 		break;
 	}
 
-	if (!func)
+	if (!func || hfi_shutdown_done)
 		goto out;
 
 	for (i = 0; i < max_hfi_instances; i++) {

-- 
2.39.5



^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 2/2] thermal: intel: hfi: Disable HFI left enabled by a previous kernel
  2026-10-28  0:14 [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Jez Ng via B4 Relay
  2026-10-28  0:14 ` [PATCH 1/2] thermal: intel: hfi: Disable HFI on shutdown and kexec Jez Ng via B4 Relay
@ 2026-10-28  0:14 ` Jez Ng via B4 Relay
  2026-11-19  5:42   ` Ricardo Neri
  2026-11-13 15:26 ` [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Rafael J. Wysocki (Intel)
  2 siblings, 1 reply; 7+ messages in thread
From: Jez Ng via B4 Relay @ 2026-10-28  0:14 UTC (permalink / raw)
  To: Rafael J. Wysocki, Daniel Lezcano, Zhang Rui, Lukasz Luba,
	Ricardo Neri, Len Brown, Stanislaw Gruszka
  Cc: Rafael J. Wysocki, linux-pm, linux-kernel, stable, Andrew James,
	John Ranzie, Jez Ng

From: Jez Ng <jez@sfcompute.com>

Since commit b33f3d2677b8 ("thermal: intel: hfi: Enable HFI only when
required"), the kernel only programs HFI when a thermal netlink client
subscribes to events. Before that, the first CPU of each package to come
online wrote a fresh table address and enabled HFI, which replaced any
setup a previous kernel left behind.

Now, if the previous kernel left HFI enabled, for example across kexec,
nothing touches it. Hardware keeps writing updates into the previous
kernel's table, which is free memory in this kernel and can be allocated
for anything. On dual-socket Sapphire Rapids systems kexec'ing from a 6.1
kernel into 7.0, these writes corrupted the inode hash table and caused
oopses in find_inode_fast().

When the first CPU of a package comes online, check
MSR_IA32_HW_FEEDBACK_CONFIG and disable HFI if it is already enabled. If
a client subscribes later, HFI is re-enabled with this kernel's table as
before.

This narrows the window but cannot close it. Hardware may still write
to the old table until the package's first CPU reaches the thermal
online callback, which runs from a device initcall. On the systems
above that was about 30 seconds after this kernel started, 21 of which
were spent in acpi_init(). In testing there, hardware wrote the table
once right after HFI was enabled and not again over several minutes,
idle or with every CPU busy, but the rate under other loads is
unknown. The complete fix is for the previous kernel to disable HFI on
shutdown, as "thermal: intel: hfi: Disable HFI on shutdown and kexec"
does; this covers previous kernels that do not.

Also update the kernel-doc of intel_hfi_online(), which still described
it as always enabling HFI.

Fixes: b33f3d2677b8 ("thermal: intel: hfi: Enable HFI only when required")
Cc: stable@vger.kernel.org
Reviewed-by: Andrew James <ajames@sfcompute.com>
Reviewed-by: John Ranzie <jranzie@sfcompute.com>
Signed-off-by: Jez Ng <jez@sfcompute.com>
---
 drivers/thermal/intel/intel_hfi.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c
index 8ff094eccbaf..b97f8113930c 100644
--- a/drivers/thermal/intel/intel_hfi.c
+++ b/drivers/thermal/intel/intel_hfi.c
@@ -413,13 +413,14 @@ static bool hfi_disable_if_enabled(void)
 }
 
 /**
- * intel_hfi_online() - Enable HFI on @cpu
- * @cpu:	CPU in which the HFI will be enabled
+ * intel_hfi_online() - Set up HFI on @cpu
+ * @cpu:	CPU that is coming online
  *
- * Enable the HFI to be used in @cpu. The HFI is enabled at the package
- * level. The first CPU in the package to come online does the full HFI
- * initialization. Subsequent CPUs will just link themselves to the HFI
- * instance of their package.
+ * The HFI is enabled at the package level. The first CPU in the package to
+ * come online does the full HFI initialization, which includes disabling HFI
+ * if a previous kernel left it enabled. HFI is enabled only when there are
+ * thermal netlink clients. Subsequent CPUs will just link themselves to the
+ * HFI instance of their package.
  *
  * This function is called before enabling the thermal vector in the local APIC
  * in order to ensure that @cpu has an associated HFI instance when it receives
@@ -462,6 +463,14 @@ void intel_hfi_online(unsigned int cpu)
 	if (hfi_instance->hdr)
 		goto enable;
 
+	/*
+	 * A previous kernel may have left HFI enabled across kexec. Hardware
+	 * would keep writing to that kernel's table, in memory this kernel now
+	 * owns.
+	 */
+	if (hfi_disable_if_enabled())
+		pr_info("HFI left enabled on package %u, disabled\n", pkg_id);
+
 	/*
 	 * Hardware is programmed with the physical address of the first page
 	 * frame of the table. Hence, the allocated memory must be page-aligned.

-- 
2.39.5



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec
  2026-10-28  0:14 [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Jez Ng via B4 Relay
  2026-10-28  0:14 ` [PATCH 1/2] thermal: intel: hfi: Disable HFI on shutdown and kexec Jez Ng via B4 Relay
  2026-10-28  0:14 ` [PATCH 2/2] thermal: intel: hfi: Disable HFI left enabled by a previous kernel Jez Ng via B4 Relay
@ 2026-11-13 15:26 ` Rafael J. Wysocki (Intel)
  2026-11-19  5:45   ` Ricardo Neri
  2 siblings, 1 reply; 7+ messages in thread
From: Rafael J. Wysocki (Intel) @ 2026-11-13 15:26 UTC (permalink / raw)
  To: jez, Ricardo Neri
  Cc: Daniel Lezcano, Zhang Rui, Lukasz Luba, Len Brown,
	Rafael J. Wysocki, linux-pm, linux-kernel, Andrew James,
	John Ranzie

On Wed, Oct 28, 2026 at 1:20 AM Jez Ng via B4 Relay
<devnull+jez.sfcompute.com@kernel.org> wrote:
>
> HFI hardware writes updates to a table at a physical address the kernel
> programs. Nothing disables HFI on shutdown, and since commit b33f3d2677b8
> ("thermal: intel: hfi: Enable HFI only when required") a kernel that
> starts without thermal netlink clients does not touch HFI at all. So
> after a kexec, hardware keeps writing into the previous kernel's table,
> which the new kernel treats as free memory.
>
> We hit this on dual-socket Sapphire Rapids hosts (Xeon Platinum 8462Y+)
> that boot a Debian 6.1 kernel and kexec into a 7.0 kernel. The new
> kernel crashed in find_inode_fast() when the inode hash table landed on
> one of the old table pages. Reading MSR 0x17d0 on the running 7.0
> kernel showed both packages still pointing at the 6.1 kernel's tables,
> and the corruption had the HFI table layout: a timestamp, a changed
> flag, then one row per CPU.
>
> Patch 1 disables HFI in a syscore shutdown callback, which covers
> reboot, power off and kexec, and keeps it disabled afterwards: on kexec,
> user space runs until machine_shutdown(), so a new thermal netlink
> client could otherwise enable HFI again. Patch 2 makes a starting kernel
> disable HFI if the previous kernel left it enabled. Patch 2 can only
> narrow the window: hardware can still write between kernel start and
> the moment the package's first CPU reaches the thermal online callback,
> which was about 30 seconds into boot on these hosts.
>
> This is the kexec counterpart of the hibernation corruption fixed by
> commit 97566d09fd02 ("thermal: intel: hfi: Add syscore callbacks for
> system-wide PM"), which disables HFI for suspend and hibernation but not
> for shutdown.
>
> Tested on one of the hosts above with both patches applied to this base,
> alternating 100 kexecs (kexec_file_load + reboot(LINUX_REBOOT_CMD_KEXEC))
> between the patched kernel and an unpatched 7.0 kernel. Before each
> kexec, a thermal netlink subscriber enabled HFI on both packages. All
> 100 passed:
>
>  - Patch 1: after each of the 50 kexecs out of the patched kernel, the
>    unpatched kernel read HW_FEEDBACK_CONFIG = 0 on both packages.
>
>  - Patch 2: after each of the 50 kexecs into the patched kernel, it
>    logged "HFI left enabled on package N, disabled" for both packages
>    and read HW_FEEDBACK_CONFIG = 0.

Ricardo, can you please have a look at this series and let me know
what you think about it?

> A test-only 5 second sleep after the shutdown callback, with a thermal
> netlink client subscribing during it, showed that patch 1 keeps HFI
> disabled. Without the hfi_shutdown_done check, the next kernel found HFI
> enabled again.
>
> Thanks to Andrew James and John Ranzie for reviewing.
>
> The investigation and testing were done via Opus 5.5, as were the
> drafting of the patches and changelogs. I reviewed all of it.
>
> ---
> Jez Ng (2):
>       thermal: intel: hfi: Disable HFI on shutdown and kexec
>       thermal: intel: hfi: Disable HFI left enabled by a previous kernel
>
>  drivers/thermal/intel/intel_hfi.c | 71 ++++++++++++++++++++++++++++++++++-----
>  1 file changed, 63 insertions(+), 8 deletions(-)
> ---
> base-commit: 70f331d89564126fdcaaa37f25194fe391d8f3fa
> change-id: 20261027-hfi-kexec-e7c4b7f584be
>
> Best regards,
> --
> Jez Ng <jez@sfcompute.com>
>
>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 2/2] thermal: intel: hfi: Disable HFI left enabled by a previous kernel
  2026-10-28  0:14 ` [PATCH 2/2] thermal: intel: hfi: Disable HFI left enabled by a previous kernel Jez Ng via B4 Relay
@ 2026-11-19  5:42   ` Ricardo Neri
  0 siblings, 0 replies; 7+ messages in thread
From: Ricardo Neri @ 2026-11-19  5:42 UTC (permalink / raw)
  To: jez
  Cc: Rafael J. Wysocki, Daniel Lezcano, Zhang Rui, Lukasz Luba,
	Len Brown, Stanislaw Gruszka, Rafael J. Wysocki, linux-pm,
	linux-kernel, stable, Andrew James, John Ranzie

On Wed, Oct 28, 2026 at 12:14:46AM +0000, Jez Ng via B4 Relay wrote:
> From: Jez Ng <jez@sfcompute.com>

[ ... ]

> 
> When the first CPU of a package comes online, check
> MSR_IA32_HW_FEEDBACK_CONFIG and disable HFI if it is already enabled. If
> a client subscribes later, HFI is re-enabled with this kernel's table as
> before.

This is not true on all processors. Some ignore later writes to
MSR_IA32_HW_FEEDBACK_PTR and, when HFI is enabled again, keep writing to
the table programmed by the first kernel after reset. Perhaps this
sentence can be qualified?

Also, would you be willing to check whether your Sapphire Rapids systems
honor later writes to MSR_IA32_HW_FEEDBACK_PTR?

[ ... ]

  
> +	/*
> +	 * A previous kernel may have left HFI enabled across kexec. Hardware
> +	 * would keep writing to that kernel's table, in memory this kernel now
> +	 * owns.
> +	 */
> +	if (hfi_disable_if_enabled())
> +		pr_info("HFI left enabled on package %u, disabled\n", pkg_id);
> +

By the time this message is printed, memory corruption may already have
happened. Perhaps this should be a pr_warn() that mentions the potential
corruption explicitly?

As you acknowledge in the changelog, this code runs rather late during
boot. HFI could be disabled much earlier, from intel_init_thermal(),
which runs at CPU identify time: from start_kernel() for the boot CPU
and from smp_init() for the APs. This also works when the next kernel
has CONFIG_INTEL_HFI_THERMAL=n.

Maybe something like this in therm_throt.c or intel_hfi.h, called first
thing in intel_init_thermal():

static void intel_hfi_disable_stale(struct cpuinfo_x86 *c)
{
	u64 msr_val;

	if (!cpu_has(c, X86_FEATURE_HFI))
		return;

	if (IS_ENABLED(CONFIG_INTEL_HFI_THERMAL) && system_state >= SYSTEM_RUNNING)
		return;

	rdmsrq(MSR_IA32_HW_FEEDBACK_CONFIG, msr_val);
	if (!(msr_val & HW_FEEDBACK_CONFIG_HFI_ENABLE_BIT))
		return;

	wrmsrq(MSR_IA32_HW_FEEDBACK_CONFIG,
	       msr_val & ~HW_FEEDBACK_CONFIG_HFI_ENABLE_BIT);
	pr_warn("CPU%d: HFI was enabled by a previous kernel; disabled it. Memory may be corrupted.\n",
		smp_processor_id());
}

intel_init_thermal() also runs when a CPU comes online after boot and
on resume. Such a CPU must not disable an HFI instance that the driver
enabled for the other CPUs of its package. intel_hfi_disable_stale()
therefore checks system_state. Before SYSTEM_RUNNING, no client can have
asked for HFI. SYSTEM_BOOTING would not work: the APs come up during
smp_init() with SYSTEM_SCHEDULING.

I tried this on a Panther Lake-H system (one package, 16 CPUs). I
disabled the shutdown path of patch 1 and subscribed a thermal netlink
client to leave HFI enabled across kexec. The boot CPU disabled HFI
before smp_init() and acpi_init(). With your patch 2 alone, HFI remained
enabled 2.9 seconds longer. You should see a larger difference on your
systems, since 21 of their 30 seconds were spent in acpi_init().

A package with no CPU online at boot, for instance with maxcpus= or
nr_cpus=, keeps writing to the previous kernel's table until one of its
CPUs comes online, if ever. The check you added in intel_hfi_online() is
still needed to catch such a package when it comes online. With the
earlier check, that is the only case in which your check can find HFI
enabled. Only the syscore shutdown callback, in the previous kernel,
protects the time before that. Could the comment above your check and
the changelog say so?

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 1/2] thermal: intel: hfi: Disable HFI on shutdown and kexec
  2026-10-28  0:14 ` [PATCH 1/2] thermal: intel: hfi: Disable HFI on shutdown and kexec Jez Ng via B4 Relay
@ 2026-11-19  5:42   ` Ricardo Neri
  0 siblings, 0 replies; 7+ messages in thread
From: Ricardo Neri @ 2026-11-19  5:42 UTC (permalink / raw)
  To: jez
  Cc: Rafael J. Wysocki, Daniel Lezcano, Zhang Rui, Lukasz Luba,
	Len Brown, Stanislaw Gruszka, Rafael J. Wysocki, linux-pm,
	linux-kernel, stable, Andrew James, John Ranzie

On Wed, Oct 28, 2026 at 12:14:45AM +0000, Jez Ng via B4 Relay wrote:
> From: Jez Ng <jez@sfcompute.com>

[ ... ]
  
> +static void hfi_syscore_shutdown(void *data)
> +{
> +	struct hfi_instance *hfi_instance;
> +	int i;
> +
> +	/*
> +	 * Hardware keeps writing to the HFI table after a kexec, into memory
> +	 * that the next kernel owns. Disable HFI in every package, and keep it
> +	 * disabled: user space runs until machine_shutdown() stops the other
> +	 * CPUs, and kexec enables CPU hotplug again, so a new thermal netlink
> +	 * client or a CPU coming online could otherwise enable it.
> +	 */
> +	mutex_lock(&hfi_instance_lock);
> +	hfi_shutdown_done = true;
> +	for (i = 0; i < max_hfi_instances; i++) {
> +		hfi_instance = &hfi_instances[i];
> +		if (cpumask_empty(hfi_instance->cpus))
> +			continue;
> +
> +		smp_call_function_single(cpumask_any(hfi_instance->cpus),
> +					 hfi_shutdown_instance, hfi_instance, true);
> +	}
> +	mutex_unlock(&hfi_instance_lock);

Using guard(mutex)(&hfi_instance_lock) would read more simply here.
However, this patch is tagged for stable, and kernels older than v6.5,
such as 6.1, do not have guard(). It is probably better to leave it as
is.

I tested this patch once on a Panther Lake-H client system (one
package, 16 CPUs), together with patch 2. With a thermal netlink client
subscribed, a kexec left HW_FEEDBACK_CONFIG at 0 in the new kernel.

Reviewed-by: Ricardo Neri ricardo.neri-calderon@linux.intel.com
Tested-by: Ricardo Neri ricardo.neri-calderon@linux.intel.com

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec
  2026-11-13 15:26 ` [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Rafael J. Wysocki (Intel)
@ 2026-11-19  5:45   ` Ricardo Neri
  0 siblings, 0 replies; 7+ messages in thread
From: Ricardo Neri @ 2026-11-19  5:45 UTC (permalink / raw)
  To: Rafael J. Wysocki (Intel)
  Cc: jez, Daniel Lezcano, Zhang Rui, Lukasz Luba, Len Brown,
	Rafael J. Wysocki, linux-pm, linux-kernel, Andrew James,
	John Ranzie

On Fri, Nov 13, 2026 at 04:26:06PM +0100, Rafael J. Wysocki (Intel) wrote:
> On Wed, Oct 28, 2026 at 1:20 AM Jez Ng via B4 Relay
> <devnull+jez.sfcompute.com@kernel.org> wrote:
> >
> > HFI hardware writes updates to a table at a physical address the kernel
> > programs. Nothing disables HFI on shutdown, and since commit b33f3d2677b8
> > ("thermal: intel: hfi: Enable HFI only when required") a kernel that
> > starts without thermal netlink clients does not touch HFI at all. So
> > after a kexec, hardware keeps writing into the previous kernel's table,
> > which the new kernel treats as free memory.
> >
> > We hit this on dual-socket Sapphire Rapids hosts (Xeon Platinum 8462Y+)
> > that boot a Debian 6.1 kernel and kexec into a 7.0 kernel. The new
> > kernel crashed in find_inode_fast() when the inode hash table landed on
> > one of the old table pages. Reading MSR 0x17d0 on the running 7.0
> > kernel showed both packages still pointing at the 6.1 kernel's tables,
> > and the corruption had the HFI table layout: a timestamp, a changed
> > flag, then one row per CPU.
> >
> > Patch 1 disables HFI in a syscore shutdown callback, which covers
> > reboot, power off and kexec, and keeps it disabled afterwards: on kexec,
> > user space runs until machine_shutdown(), so a new thermal netlink
> > client could otherwise enable HFI again. Patch 2 makes a starting kernel
> > disable HFI if the previous kernel left it enabled. Patch 2 can only
> > narrow the window: hardware can still write between kernel start and
> > the moment the package's first CPU reaches the thermal online callback,
> > which was about 30 seconds into boot on these hosts.
> >
> > This is the kexec counterpart of the hibernation corruption fixed by
> > commit 97566d09fd02 ("thermal: intel: hfi: Add syscore callbacks for
> > system-wide PM"), which disables HFI for suspend and hibernation but not
> > for shutdown.
> >
> > Tested on one of the hosts above with both patches applied to this base,
> > alternating 100 kexecs (kexec_file_load + reboot(LINUX_REBOOT_CMD_KEXEC))
> > between the patched kernel and an unpatched 7.0 kernel. Before each
> > kexec, a thermal netlink subscriber enabled HFI on both packages. All
> > 100 passed:
> >
> >  - Patch 1: after each of the 50 kexecs out of the patched kernel, the
> >    unpatched kernel read HW_FEEDBACK_CONFIG = 0 on both packages.
> >
> >  - Patch 2: after each of the 50 kexecs into the patched kernel, it
> >    logged "HFI left enabled on package N, disabled" for both packages
> >    and read HW_FEEDBACK_CONFIG = 0.
> 
> Ricardo, can you please have a look at this series and let me know
> what you think about it?

Hi Rafael, Jez,

These patches fix a real problem, and I think they can be applied
once we close on my comments on patch 2.

This series does not cover processors that ignore a write to
MSR_IA32_HW_FEEDBACK_PTR after the first one. On those processors, the
HFI table of the first kernel after reset must be preserved across
kexec and reused by the next kernel. I have work in progress for that,
based on Kexec Handover (KHO), and plan to send it as a follow-up to
this series.

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-19  5:37 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-28  0:14 [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Jez Ng via B4 Relay
2026-10-28  0:14 ` [PATCH 1/2] thermal: intel: hfi: Disable HFI on shutdown and kexec Jez Ng via B4 Relay
2026-11-19  5:42   ` Ricardo Neri
2026-10-28  0:14 ` [PATCH 2/2] thermal: intel: hfi: Disable HFI left enabled by a previous kernel Jez Ng via B4 Relay
2026-11-19  5:42   ` Ricardo Neri
2026-11-13 15:26 ` [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec Rafael J. Wysocki (Intel)
2026-11-19  5:45   ` Ricardo Neri

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®