[PATCH] x86/resctrl: Require force for L3 CAT on early Skylake-X

[PATCH] x86/resctrl: Require force for L3 CAT on early Skylake-X

[Alexander Graf]

When L3 CAT gets enabled with rdt=l3cat on Skylake-X stepping 3,
consecutive kexec boots hang for me during secondary socket bringup.
Cold boots on the same machine work.

We already disable L3 CAT by default on Skylake-X and Cascade Lake and
enable it only when the rdt=l3cat option is passed on the kernel command
line. I want to enable l3cat on Cascade Lake, but preserve a single kernel
command line across both Skylake and Cascade Lake systems.

To enable this, guard l3cat on Skylake-X steppings below 4 behind a force
option. That way, safe (but potentially imprecise) options continue to be
guarded by the normal rdt command line, but combinations that impact
system stability also require force in addition. With this, I can have a
single, safe kernel command line across both environments.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 .../admin-guide/kernel-parameters.txt         |  6 ++++
 arch/x86/kernel/cpu/resctrl/core.c            | 31 ++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index e75344f4e0cd..54f8a9ea665a 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6458,6 +6458,12 @@ Kernel parameters
 			perf telemetry monitoring associated with guid 0x12345
 			is enabled use:
 				rdt=!energy,perf:0x12345
+			On Skylake-X stepping < 4, rdt=l3cat is ignored
+			unless the bare token "force" is also given, e.g.:
+				rdt=force,l3cat
+			because L3 CAT there can hang a later kexec boot.
+			The token may appear anywhere in any rdt= parameter.
+			"!force" is ignored.
 
 	reboot=		[KNL]
 			Format (x86 or x86_64):
diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 55214d6fdc49..c4eb157b8b72 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -833,6 +833,8 @@ static struct rdt_options rdt_options[]  __ro_after_init = {
 };
 #define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
 
+static bool rdt_force_raw __initdata;
+
 static int __init set_rdt_options(char *str)
 {
 	struct rdt_options *o;
@@ -842,6 +844,10 @@ static int __init set_rdt_options(char *str)
 	if (*str == '=')
 		str++;
 	while ((tok = strsep(&str, ",")) != NULL) {
+		if (!strcmp(tok, "force")) {
+			rdt_force_raw = true;
+			continue;
+		}
 		force_off = *tok == '!';
 		if (force_off)
 			tok++;
@@ -989,6 +995,26 @@ static __init bool get_rdt_mon_resources(void)
 	return !rdt_get_l3_mon_config(r);
 }
 
+/*
+ * Disable a feature that plain rdt= enable requests must not re-enable.
+ * Only rdt=force lets an explicit enable request override this.
+ */
+static __init void rdt_force_off_hard(int idx)
+{
+	struct rdt_options *o = &rdt_options[idx];
+
+	o->force_off = true;
+	if (!o->force_on)
+		return;
+
+	if (rdt_force_raw) {
+		pr_info("rdt=force: keeping %s enabled on this CPU\n", o->name);
+	} else {
+		pr_info("Ignoring rdt=%s: unsafe on this CPU, see rdt=force\n", o->name);
+		o->force_on = false;
+	}
+}
+
 static __init void __check_quirks_intel(void)
 {
 	switch (boot_cpu_data.x86_vfm) {
@@ -998,7 +1024,10 @@ static __init void __check_quirks_intel(void)
 		break;
 	case INTEL_SKYLAKE_X:
 		if (boot_cpu_data.x86_stepping <= 4)
-			set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat");
+			set_rdt_options("!cmt,!mbmtotal,!mbmlocal");
+		/* L3 CAT on these steppings can hang a later kexec boot. */
+		if (boot_cpu_data.x86_stepping < 4)
+			rdt_force_off_hard(RDT_FLAG_L3_CAT);
 		else
 			set_rdt_options("!l3cat");
 		fallthrough;

base-commit: e767a4ea70a3992c37ed604157d32f0dfbf9b1e3
-- 
2.47.1

RE: [PATCH] x86/resctrl: Require force for L3 CAT on early Skylake-X

[Luck, Tony]

> When L3 CAT gets enabled with rdt=l3cat on Skylake-X stepping 3,
> consecutive kexec boots hang for me during secondary socket bringup.
> Cold boots on the same machine work.

This is troubling. I thought that the problems on Skylake were simply that in some circumstances
the CLOSID cache mask would not be honored so that data would spill into cache ways that
were supposed to be reserved for other users. I did not expect that the system could hang
with L3 cat enabled.

Your fix seems quite complex.  Perhaps something like this?

	case INTEL_SKYLAKE_X:
		if (boot_cpu_data.x86_stepping == 3)
			setup_clear_cpu_cap(X86_FEATURE_CAT_L3);
		if (boot_cpu_data.x86_stepping <= 4)
			set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat");
		else
			set_rdt_options("!l3cat");
		fallthrough;

-Tony

Re: [PATCH] x86/resctrl: Require force for L3 CAT on early Skylake-X

[Reinette Chatre]

Hi Alexander,

On 10/14/26 1:36 AM, Alexander Graf wrote:
> When L3 CAT gets enabled with rdt=l3cat on Skylake-X stepping 3,
> consecutive kexec boots hang for me during secondary socket bringup.
> Cold boots on the same machine work.

Any insight into where the hang occurs?

What is state of resctrl fs when kexec runs? Is it mounted (with which options)?

Does the boot hang whether resctrl fs is mounted or not?

If resctrl is mounted at time of kexec, has it been set up with different groups?
Does kexec task run from a particular group?
 
Does it make a difference if resctrl fs is unmounted before kexec? As I understand
kexec takes system from resctrl without giving opportunity to reset. There should
be no accumulation of state but system may boot up with settings left over from
previous run. Unmounting resctrl would give opportunity to reset state so that
new kernel can start with clean slate.

What does "during secondary socket bringup" mean exactly? For example, is
the hang when first CPU of second socket is onlined and found to be hanging
in the resctrl online handler?

Thank you.

Reinette

Re: [PATCH] x86/resctrl: Require force for L3 CAT on early Skylake-X

[Graf (AWS), Alexander]

Hi Reinette and Tony,

On 14.10.26 22:44, Reinette Chatre wrote:
> Hi Alexander,
>
> On 10/14/26 1:36 AM, Alexander Graf wrote:
>> When L3 CAT gets enabled with rdt=l3cat on Skylake-X stepping 3,
>> consecutive kexec boots hang for me during secondary socket bringup.
>> Cold boots on the same machine work.
> Any insight into where the hang occurs?
>
> What is state of resctrl fs when kexec runs? Is it mounted (with which options)?
>
> Does the boot hang whether resctrl fs is mounted or not?
>
> If resctrl is mounted at time of kexec, has it been set up with different groups?
> Does kexec task run from a particular group?
>   
> Does it make a difference if resctrl fs is unmounted before kexec? As I understand
> kexec takes system from resctrl without giving opportunity to reset. There should
> be no accumulation of state but system may boot up with settings left over from
> previous run. Unmounting resctrl would give opportunity to reset state so that
> new kernel can start with clean slate.
>
> What does "during secondary socket bringup" mean exactly? For example, is
> the hang when first CPU of second socket is onlined and found to be hanging
> in the resctrl online handler?


I ran a few more experiments. This seems un- or if anything only mildly 
related to kexec. The real culprit is the microcode version. In my 
environment, we boot with older microcode (0x1000159), then hot load the 
latest microcode (0x1000191) and then kexec. In that flow, any write to 
IA32_L3_QOS_MASK_n stalls the socket that write happens on. I'm 
collecting a bit more data - including microcode version bisects, early 
load, etc - and give you a full rundown tomorrow or on Monday, depending 
on how quickly it finishes.


Alex

Re: [PATCH] x86/resctrl: Require force for L3 CAT on early Skylake-X

[Luck, Tony]

On Thu, Oct 15, 2026 at 04:58:46PM +0000, Graf (AWS), Alexander wrote:
> I ran a few more experiments. This seems un- or if anything only mildly 
> related to kexec. The real culprit is the microcode version. In my 
> environment, we boot with older microcode (0x1000159), then hot load the 
> latest microcode (0x1000191) and then kexec. In that flow, any write to 
> IA32_L3_QOS_MASK_n stalls the socket that write happens on. I'm 
> collecting a bit more data - including microcode version bisects, early 
> load, etc - and give you a full rundown tomorrow or on Monday, depending 
> on how quickly it finishes.

Alexander,

I don't think that an upstream generic kernel solution is the right
path for your very system-specific issue.

Can't you fix your deployment scripts to look at system stepping and
use that to choose the kernel boot options? I presume that you already
have checks for CPU model number since there are lots of different
options appropriate on a per-model basis. So adding a stepping check
may be easier than battling upstream Linux submission process.

-Tony

Re: [PATCH] x86/resctrl: Require force for L3 CAT on early Skylake-X

[Graf (AWS), Alexander]

Hi Tony,

On 15.10.26 19:11, Luck, Tony wrote:
> On Thu, Oct 15, 2026 at 04:58:46PM +0000, Graf (AWS), Alexander wrote:
>> I ran a few more experiments. This seems un- or if anything only mildly
>> related to kexec. The real culprit is the microcode version. In my
>> environment, we boot with older microcode (0x1000159), then hot load the
>> latest microcode (0x1000191) and then kexec. In that flow, any write to
>> IA32_L3_QOS_MASK_n stalls the socket that write happens on. I'm
>> collecting a bit more data - including microcode version bisects, early
>> load, etc - and give you a full rundown tomorrow or on Monday, depending
>> on how quickly it finishes.
> Alexander,
>
> I don't think that an upstream generic kernel solution is the right
> path for your very system-specific issue.
>
> Can't you fix your deployment scripts to look at system stepping and
> use that to choose the kernel boot options? I presume that you already
> have checks for CPU model number since there are lots of different
> options appropriate on a per-model basis. So adding a stepping check
> may be easier than battling upstream Linux submission process.


I can hack around it for my specific environment. The reason I'm 
submitting this is because I expect that I won't be the only person 
falling into that pitfall. I expect that early microcode loading (which 
is distro default) would show the same failure without kexec. I want to 
save the next person that runs into it the same painful debugging 
exercise :).


Alex