From: Babu Moger <babu.moger@amd.com>
To: tony.luck@intel.com, reinette.chatre@intel.com,
ben.horgan@arm.com, Dave.Martin@arm.com, james.morse@arm.com,
bp@alien8.de
Cc: corbet@lwn.net, skhan@linuxfoundation.org, rdunlap@infradead.org,
x86@kernel.org, fustini@kernel.org, tglx@kernel.org,
mingo@redhat.com, dave.hansen@linux.intel.com, hpa@zytor.com,
fenghuay@nvidia.com, rppt@kernel.org, arnd@arndb.de,
kmehltretter@gmail.com, dapeng1.mi@linux.intel.com,
elver@google.com, enelsonmoore@gmail.com, jlayton@kernel.org,
seanjc@google.com, ewanhai-oc@zhaoxin.com,
binbin.wu@linux.intel.com, nikunj@amd.com, ashish.kalra@amd.com,
peterz@infradead.org, superm1@kernel.org,
thomas.lendacky@amd.com, prathyushi.nangia@amd.com,
kim.phillips@amd.com, ricardo.neri-calderon@linux.intel.com,
darwi@linutronix.de, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v7 13/19] x86,fs/resctrl: Add CPU hotplug support for kernel mode
Date: Mon, 16 Nov 2026 14:08:01 -0600 [thread overview]
Message-ID: <79684ccf-9b77-4c2c-8f86-0fee306a8745@amd.com> (raw)
In-Reply-To: <a46c34d2cb1cf11daaece5ef49f7e7936082d026.1794437052.git.babu.moger@amd.com>
On 11/11/26 16:44, Babu Moger wrote:
> The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode allows the CPU's CLOSID
> and/or RMID association to change when transitioning from user mode to
> kernel mode. The mode is enabled or disabled on a per-CPU basis through a
> backing resource group. CPUs that come online are brought into a state
> consistent with the active kernel mode, while CPUs that go offline are
> removed from its CPU mask.
>
> Integrate RESCTRL_GLOBAL_ENABLE_PER_CPU mode with CPU hotplug handling so
> that CPUs automatically join the active kernel mode when they come online
> and are removed from it when they go offline. Also, ensure the register
> state remains in a consistent reset state across kexec boot transitions.
> This ensures that kernel mode behavior remains consistent across CPU
> hotplug events and kexec transitions.
>
> Introduce resctrl_kmode_global_online_cpu() and
> resctrl_kmode_global_offline_cpu(), and hook them into the existing
> resctrl_online_cpu() and resctrl_offline_cpu() paths. Also update
> clear_closid_rmid() to reset MSR_IA32_PQR_PLZA_ASSOC register across kexec
> transitions.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v7: Changelog update.
> Code comment update.
> Added changes to reset the PLZA register during the bootup.
>
> v6: Update the changelog in non-bugfix form.
> Update the code to remove rdtgroup->kmode.
> Introduce new wrapper resctrl_configure_kmode_global() to consolidate
> architecture calls.
>
> v5: Added resctrl_kmode_offline_cpu() to disable the kernel mode
> association and update kmode_cpu_mask when an associated CPU goes
> offline.
> This addresses sashiko comments:
> https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=11
> https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=10
> Also moved the patch earlier than patch 10.
> [RESEND PATCH v4 10/15] fs/resctrl: Reset the kernel mode binding when an rdtgroup is removed
> That seemed much more clear.
>
> v4: New patch in the series. Patch taken from:
> https://lore.kernel.org/lkml/20260611111706.1981788-5-qinyuntan@linux.alibaba.com/
> Updated the code to enable kernel mode by default whenever a CPU comes
> online when one of global-assign mode is enabled.
> ---
> arch/x86/kernel/cpu/resctrl/core.c | 3 ++
> fs/resctrl/rdtgroup.c | 56 ++++++++++++++++++++++++++++++
> 2 files changed, 59 insertions(+)
>
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 14530c11bbd7..d7faf30c2885 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -704,6 +704,9 @@ static void clear_closid_rmid(int cpu)
> state->cur_closid = RESCTRL_RESERVED_CLOSID;
> state->cur_rmid = RESCTRL_RESERVED_RMID;
> wrmsrq(MSR_IA32_PQR_ASSOC, val.q);
> +
> + if (boot_cpu_has(X86_FEATURE_PLZA))
> + wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, 0);
> }
>
Sashiko reported:
https://sashiko.dev/#/patchset/cover.1794437052.git.babu.moger%40amd.com?part=13
=============================================================
commit 87e9e9cf13d544344f9797647860bcba9ffab37d
Author: Babu Moger <babu.moger@amd.com>
x86,fs/resctrl: Add CPU hotplug support for kernel mode
This commit integrates RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode with CPU
hotplug handling and resets MSR_IA32_PQR_PLZA_ASSOC in clear_closid_rmid().
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c
b/arch/x86/kernel/cpu/resctrl/core.c
> index fc59d66f77de1..33965b760aed0 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -735,6 +735,9 @@ static void clear_closid_rmid(int cpu)
> state->cur_closid = RESCTRL_RESERVED_CLOSID;
> state->cur_rmid = RESCTRL_RESERVED_RMID;
> wrmsrq(MSR_IA32_PQR_ASSOC, val.q);
> +
> + if (boot_cpu_has(X86_FEATURE_PLZA))
> + wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, 0);
> }
Does zeroing MSR_IA32_PQR_PLZA_ASSOC in clear_closid_rmid() clobber the
domain's active PLZA configuration during CPU hotplug?
When a CPU goes offline while RESCTRL_GLOBAL_ENABLE_PER_CPU is active,
resctrl_arch_offline_cpu() first runs resctrl_offline_cpu() ->
resctrl_kmode_global_offline_cpu(), which clears plza_en while
preserving rmid, rmid_en, closid, and closid_en in
MSR_IA32_PQR_PLZA_ASSOC (or leaves the MSR untouched if the CPU was not
in kmode_cpu_mask). Immediately afterward, resctrl_arch_offline_cpu()
calls clear_closid_rmid() and writes 0 to MSR_IA32_PQR_PLZA_ASSOC:
arch/x86/kernel/cpu/resctrl/core.c:resctrl_arch_offline_cpu() {
...
resctrl_offline_cpu(cpu);
mutex_lock(&domain_list_lock);
for_each_capable_rdt_resource(r)
domain_remove_cpu(cpu, r);
mutex_unlock(&domain_list_lock);
clear_closid_rmid(cpu);
...
}
That overwrites the MSR value just programmed by
resctrl_kmode_global_offline_cpu() and zeroes rmid, rmid_en, closid, and
closid_en on the offlined CPU while other CPUs in the same QoS domain
still have the active kernel mode configuration programmed, violating
the hardware requirement documented for union msr_pqr_plza_assoc that
all fields except plza_en must be programmed identically across every
CPU in a QoS domain. During CPU online, resctrl_arch_online_cpu()
likewise calls clear_closid_rmid() after domain_add_cpu() and before
resctrl_online_cpu() takes rdtgroup_mutex to program the active
configuration.
=======================================================================
Yes. This is an issue. We cannot write 0 to MSR_IA32_PQR_PLZA_ASSOC in
hotplug handler.
Couple of options:
1. Introduce the reboot notifier for resctrl and clear
MSR_IA32_PQR_PLZA_ASSOC. Reboot notifier is called during kexec reboot.
or
2. Just clear the PLZA_EN bit in clear_closid_rmid().
if (boot_cpu_has(X86_FEATURE_PLZA))
msr_clear_bit(MSR_IA32_PQR_PLZA_ASSOC, PLZA_EN);
Thoughts?
Thanks
Babu
next prev parent reply other threads:[~2026-11-16 20:08 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 22:43 [PATCH v7 00/19] x86,fs/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem Babu Moger
2026-11-11 22:43 ` [PATCH v7 01/19] x86/cpufeatures: Support Privilege Level Zero Association (PLZA) Babu Moger
2026-11-11 22:43 ` [PATCH v7 02/19] x86/resctrl: Add PLZA support to command-line options Babu Moger
2026-11-11 22:43 ` [PATCH v7 03/19] x86/resctrl: Add PLZA MSR definition and field layout Babu Moger
2026-11-11 22:43 ` [PATCH v7 04/19] fs/resctrl: Introduce kernel mode enum Babu Moger
2026-11-16 20:07 ` Babu Moger
2026-11-11 22:43 ` [PATCH v7 05/19] arm,x86,fs/resctrl: Introduce arch hook to program global per-CPU kernel mode Babu Moger
2026-11-11 22:43 ` [PATCH v7 06/19] fs/resctrl: Introduce data structures for kernel mode configuration Babu Moger
2026-11-11 22:44 ` [PATCH v7 07/19] fs/resctrl: Introduce resctrl_set_kmode_support() to register supported modes Babu Moger
2026-11-11 22:44 ` [PATCH v7 08/19] x86/resctrl: Register RESCTRL_GLOBAL_ENABLE_PER_CPU when PLZA is available Babu Moger
2026-11-11 22:44 ` [PATCH v7 09/19] fs/resctrl: Rearrange top-level files in info directory Babu Moger
2026-11-11 22:44 ` [PATCH v7 10/19] fs/resctrl: Add interface to display kernel mode status Babu Moger
2026-11-11 22:44 ` [PATCH v7 11/19] fs/resctrl: Add support for hidden resource group files Babu Moger
2026-11-11 22:44 ` [PATCH v7 12/19] fs/resctrl: Introduce kmode_cpus/kmode_cpus_list per rdtgroup Babu Moger
2026-11-11 22:44 ` [PATCH v7 13/19] x86,fs/resctrl: Add CPU hotplug support for kernel mode Babu Moger
2026-11-16 20:08 ` Babu Moger [this message]
2026-11-11 22:44 ` [PATCH v7 14/19] fs/resctrl: Deactivate kernel mode when the group is removed Babu Moger
2026-11-11 22:44 ` [PATCH v7 15/19] fs/resctrl: Extend info/kernel_mode to configure the kernel mode Babu Moger
2026-11-16 20:09 ` Babu Moger
2026-11-11 22:44 ` [PATCH v7 16/19] fs/resctrl: Reject mode changes for groups backing " Babu Moger
2026-11-11 22:44 ` [PATCH v7 17/19] fs/resctrl: Control visibility of global per-CPU kernel mode group files Babu Moger
2026-11-16 20:10 ` Babu Moger
2026-11-11 22:44 ` [PATCH v7 18/19] fs/resctrl: Allow userspace to write kmode_cpus/kmode_cpus_list Babu Moger
2026-11-11 22:44 ` [PATCH v7 19/19] fs/resctrl: Add kernel mode documentation and examples Babu Moger
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=79684ccf-9b77-4c2c-8f86-0fee306a8745@amd.com \
--to=babu.moger@amd.com \
--cc=Dave.Martin@arm.com \
--cc=arnd@arndb.de \
--cc=ashish.kalra@amd.com \
--cc=ben.horgan@arm.com \
--cc=binbin.wu@linux.intel.com \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=dapeng1.mi@linux.intel.com \
--cc=darwi@linutronix.de \
--cc=dave.hansen@linux.intel.com \
--cc=elver@google.com \
--cc=enelsonmoore@gmail.com \
--cc=ewanhai-oc@zhaoxin.com \
--cc=fenghuay@nvidia.com \
--cc=fustini@kernel.org \
--cc=hpa@zytor.com \
--cc=james.morse@arm.com \
--cc=jlayton@kernel.org \
--cc=kim.phillips@amd.com \
--cc=kmehltretter@gmail.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=nikunj@amd.com \
--cc=peterz@infradead.org \
--cc=prathyushi.nangia@amd.com \
--cc=rdunlap@infradead.org \
--cc=reinette.chatre@intel.com \
--cc=ricardo.neri-calderon@linux.intel.com \
--cc=rppt@kernel.org \
--cc=seanjc@google.com \
--cc=skhan@linuxfoundation.org \
--cc=superm1@kernel.org \
--cc=tglx@kernel.org \
--cc=thomas.lendacky@amd.com \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®