[PATCH 0/6] Intel Performance Limit Reasons support

[PATCH 0/6] Intel Performance Limit Reasons support

[Tero Kristo]

Hello,

This series adds support for the Intel PLR (Performance Limit Reasons).
PLR provides status information for Intel HW performance events,
exported via debugfs within the existing TPMI hierarchy. The status bits
are sticky, and they can be cleared by writing "0" to the corresponding
debugfs files.

-Tero

[PATCH 1/6] platform/x86/intel/tpmi: Add support for performance limit reasons

[Tero Kristo]

Add TPMI ID 0x0C (Perf Limit Reasons) to the list of supported TPMI IDs.

Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
 drivers/platform/x86/intel/tpmi.c | 2 ++
 include/linux/intel_tpmi.h        | 1 +
 2 files changed, 3 insertions(+)

diff --git a/drivers/platform/x86/intel/tpmi.c b/drivers/platform/x86/intel/tpmi.c
index 6c0cbccd80bb..c2ef2cd587ba 100644
--- a/drivers/platform/x86/intel/tpmi.c
+++ b/drivers/platform/x86/intel/tpmi.c
@@ -577,6 +577,8 @@ static const char *intel_tpmi_name(enum intel_tpmi_id id)
 		return "uncore";
 	case TPMI_ID_SST:
 		return "sst";
+	case TPMI_ID_PLR:
+		return "plr";
 	default:
 		return NULL;
 	}
diff --git a/include/linux/intel_tpmi.h b/include/linux/intel_tpmi.h
index 1e880cb0f454..a88ac937d2c2 100644
--- a/include/linux/intel_tpmi.h
+++ b/include/linux/intel_tpmi.h
@@ -21,6 +21,7 @@ enum intel_tpmi_id {
 	TPMI_ID_PEM = 1,	/* Power and Perf excursion Monitor */
 	TPMI_ID_UNCORE = 2,	/* Uncore Frequency Scaling */
 	TPMI_ID_SST = 5,	/* Speed Select Technology */
+	TPMI_ID_PLR = 0xc,	/* Performance Limit Reasons */
 	TPMI_CONTROL_ID = 0x80,	/* Special ID for getting feature status */
 	TPMI_INFO_ID = 0x81,	/* Special ID for PCI BDF and Package ID information */
 };
-- 
2.43.1

[PATCH 2/6] platform/x86/intel/tpmi: Add API to get debugfs root

[Tero Kristo]

Add new API to get the debugfs root directory for TPMI. This allows any
TPMI devices to add their own debugfs items under the same directory
structure.

Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
 drivers/platform/x86/intel/tpmi.c | 9 +++++++++
 include/linux/intel_tpmi.h        | 1 +
 2 files changed, 10 insertions(+)

diff --git a/drivers/platform/x86/intel/tpmi.c b/drivers/platform/x86/intel/tpmi.c
index c2ef2cd587ba..83e8b1fe53b3 100644
--- a/drivers/platform/x86/intel/tpmi.c
+++ b/drivers/platform/x86/intel/tpmi.c
@@ -357,6 +357,15 @@ int tpmi_get_feature_status(struct auxiliary_device *auxdev,
 }
 EXPORT_SYMBOL_NS_GPL(tpmi_get_feature_status, INTEL_TPMI);
 
+struct dentry *tpmi_get_debugfs_dir(struct auxiliary_device *auxdev)
+{
+	struct intel_vsec_device *intel_vsec_dev = dev_to_ivdev(auxdev->dev.parent);
+	struct intel_tpmi_info *tpmi_info = auxiliary_get_drvdata(&intel_vsec_dev->auxdev);
+
+	return tpmi_info->dbgfs_dir;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_debugfs_dir, INTEL_TPMI);
+
 static int tpmi_pfs_dbg_show(struct seq_file *s, void *unused)
 {
 	struct intel_tpmi_info *tpmi_info = s->private;
diff --git a/include/linux/intel_tpmi.h b/include/linux/intel_tpmi.h
index a88ac937d2c2..ff480b47ae64 100644
--- a/include/linux/intel_tpmi.h
+++ b/include/linux/intel_tpmi.h
@@ -54,4 +54,5 @@ struct resource *tpmi_get_resource_at_index(struct auxiliary_device *auxdev, int
 int tpmi_get_resource_count(struct auxiliary_device *auxdev);
 int tpmi_get_feature_status(struct auxiliary_device *auxdev, int feature_id, bool *read_blocked,
 			    bool *write_blocked);
+struct dentry *tpmi_get_debugfs_dir(struct auxiliary_device *auxdev);
 #endif
-- 
2.43.1

[PATCH 3/6] platform/x86/intel: TPMI domain id and CPU mapping

[Tero Kristo]

From: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>

Each TPMI power domain includes a group of CPUs. Several power
management settings in this case applicable to a group of CPUs.
There can be several power domains in a CPU package. So, provide
interfaces for:
- Get power domain id for a Linux CPU
- Get mask of Linux CPUs in a power domain

Hardware Punit uses different CPU numbering, which is not based on
APIC (Advanced Programmable Interrupt Controller) CPU numbering.
The Linux CPU numbering is based on APIC CPU numbering. Some PM features
like Intel Speed Select, the CPU core mask provided by the hardware is
based on the Punit CPU numbering. To use the core mask, this mask
needs to be converted to a Linux CPUs mask. So, provide interfaces for:
- Convert to a Linux CPU number from a Punit CPU number
- Convert to a Punit CPU number from a Linux CPU number

On each CPU online, MSR 0x54 is used to read the mapping and stores in
a per cpu array. Create a hash for faster searching of a Linux CPU number
from a Punit CPU number.

Signed-off-by: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
[tero.kristo: minor updates]
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
 drivers/platform/x86/intel/Kconfig            |   4 +
 drivers/platform/x86/intel/Makefile           |   3 +
 .../platform/x86/intel/tpmi_power_domains.c   | 236 ++++++++++++++++++
 .../platform/x86/intel/tpmi_power_domains.h   |  19 ++
 4 files changed, 262 insertions(+)
 create mode 100644 drivers/platform/x86/intel/tpmi_power_domains.c
 create mode 100644 drivers/platform/x86/intel/tpmi_power_domains.h

diff --git a/drivers/platform/x86/intel/Kconfig b/drivers/platform/x86/intel/Kconfig
index e9dc0c021029..e97a97355d5a 100644
--- a/drivers/platform/x86/intel/Kconfig
+++ b/drivers/platform/x86/intel/Kconfig
@@ -192,10 +192,14 @@ config INTEL_SMARTCONNECT
 	  This driver checks to determine whether the device has Intel Smart
 	  Connect enabled, and if so disables it.
 
+config INTEL_TPMI_POWER_DOMAINS
+	tristate
+
 config INTEL_TPMI
 	tristate "Intel Topology Aware Register and PM Capsule Interface (TPMI)"
 	depends on INTEL_VSEC
 	depends on X86_64
+	select INTEL_TPMI_POWER_DOMAINS
 	help
 	  The Intel Topology Aware Register and PM Capsule Interface (TPMI),
 	  provides enumerable MMIO interface for power management features.
diff --git a/drivers/platform/x86/intel/Makefile b/drivers/platform/x86/intel/Makefile
index c1d5fe05e3f3..10437e56027d 100644
--- a/drivers/platform/x86/intel/Makefile
+++ b/drivers/platform/x86/intel/Makefile
@@ -53,6 +53,9 @@ obj-$(CONFIG_INTEL_PUNIT_IPC)		+= intel_punit_ipc.o
 intel_vsec_tpmi-y			:= tpmi.o
 obj-$(CONFIG_INTEL_TPMI)		+= intel_vsec_tpmi.o
 
+intel_tpmi_power_domains-y		:= tpmi_power_domains.o
+obj-$(CONFIG_INTEL_TPMI_POWER_DOMAINS)	+= intel_tpmi_power_domains.o
+
 # Intel Uncore drivers
 intel-rst-y				:= rst.o
 obj-$(CONFIG_INTEL_RST)			+= intel-rst.o
diff --git a/drivers/platform/x86/intel/tpmi_power_domains.c b/drivers/platform/x86/intel/tpmi_power_domains.c
new file mode 100644
index 000000000000..40f994814248
--- /dev/null
+++ b/drivers/platform/x86/intel/tpmi_power_domains.c
@@ -0,0 +1,236 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Mapping of TPMI power domains CPU mapping
+ *
+ * Copyright (c) 2024, Intel Corporation.
+ * All Rights Reserved.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/cleanup.h>
+#include <linux/cpuhotplug.h>
+#include <linux/cpumask.h>
+#include <linux/errno.h>
+#include <linux/export.h>
+#include <linux/hashtable.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/overflow.h>
+#include <linux/slab.h>
+#include <linux/topology.h>
+#include <linux/types.h>
+
+#include <asm/cpu_device_id.h>
+#include <asm/intel-family.h>
+#include <asm/msr.h>
+
+#include "tpmi_power_domains.h"
+
+#define MSR_PM_LOGICAL_ID       0x54
+
+/*
+ * Struct of MSR 0x54
+ * [15:11] PM_DOMAIN_ID
+ * [10:3] MODULE_ID (aka IDI_AGENT_ID)
+ * [2:0] LP_ID
+ * For Atom:
+ *   [2] Always 0
+ *   [1:0] core ID within module
+ * For Core
+ *   [2:1] Always 0
+ *   [0] thread ID
+ */
+
+#define LP_ID_MASK		GENMASK_ULL(2, 0)
+#define MODULE_ID_MASK		GENMASK_ULL(10, 3)
+#define PM_DOMAIN_ID_MASK	GENMASK_ULL(15, 11)
+
+/**
+ * struct tpmi_cpu_info - Mapping information for a CPU
+ * @hnode: Used to add mapping information to hash list
+ * @linux_cpu:	Linux CPU number
+ * @pkg_id: Package ID of this CPU
+ * @punit_thread_id: Punit thread id of this CPU
+ * @punit_core_id: Punit core id
+ * @punit_domain_id: Power domain id from Punit
+ *
+ * Structure to store mapping information for a Linux CPU
+ * to a Punit core, thread and power domain.
+ */
+struct tpmi_cpu_info {
+	struct hlist_node hnode;
+	int linux_cpu;
+	u8 pkg_id;
+	u8 punit_thread_id;
+	u8 punit_core_id;
+	u8 punit_domain_id;
+};
+
+static DEFINE_PER_CPU(struct tpmi_cpu_info, tpmi_cpu_info);
+
+/* The dynamically assigned cpu hotplug state to free later */
+static enum cpuhp_state tpmi_hp_state __read_mostly;
+
+#define MAX_POWER_DOMAINS	8
+
+static cpumask_t *tpmi_power_domain_mask;
+
+/* Lock to protect tpmi_power_domain_mask and tpmi_cpu_hash */
+static DEFINE_MUTEX(tpmi_lock);
+
+static const struct x86_cpu_id tpmi_cpu_ids[] = {
+	X86_MATCH_INTEL_FAM6_MODEL(GRANITERAPIDS_X,	NULL),
+	X86_MATCH_INTEL_FAM6_MODEL(ATOM_CRESTMONT_X,	NULL),
+	X86_MATCH_INTEL_FAM6_MODEL(ATOM_CRESTMONT,	NULL),
+	X86_MATCH_INTEL_FAM6_MODEL(GRANITERAPIDS_D,	NULL),
+	{}
+};
+MODULE_DEVICE_TABLE(x86cpu, tpmi_cpu_ids);
+
+static DECLARE_HASHTABLE(tpmi_cpu_hash, 8);
+
+static bool tpmi_domain_is_valid(struct tpmi_cpu_info *info)
+{
+	return info->pkg_id < topology_max_packages() &&
+		info->punit_domain_id < MAX_POWER_DOMAINS;
+}
+
+int tpmi_get_linux_cpu_number(int package_id, int domain_id, int punit_core_id)
+{
+	struct tpmi_cpu_info *info;
+	int ret = -EINVAL;
+
+	guard(mutex)(&tpmi_lock);
+	hash_for_each_possible(tpmi_cpu_hash, info, hnode, punit_core_id) {
+		if (info->punit_domain_id == domain_id && info->pkg_id == package_id) {
+			ret = info->linux_cpu;
+			break;
+		}
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_linux_cpu_number, INTEL_TPMI_POWER_DOMAIN);
+
+int tpmi_get_punit_core_number(int cpu_no)
+{
+	if (cpu_no >= num_possible_cpus())
+		return -EINVAL;
+
+	return per_cpu(tpmi_cpu_info, cpu_no).punit_core_id;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_punit_core_number, INTEL_TPMI_POWER_DOMAIN);
+
+int tpmi_get_power_domain_id(int cpu_no)
+{
+	if (cpu_no >= num_possible_cpus())
+		return -EINVAL;
+
+	return per_cpu(tpmi_cpu_info, cpu_no).punit_domain_id;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_power_domain_id, INTEL_TPMI_POWER_DOMAIN);
+
+cpumask_t *tpmi_get_power_domain_mask(int cpu_no)
+{
+	struct tpmi_cpu_info *info;
+	cpumask_t *mask;
+	int index;
+
+	if (cpu_no >= num_possible_cpus())
+		return NULL;
+
+	info = &per_cpu(tpmi_cpu_info, cpu_no);
+	if (!tpmi_domain_is_valid(info))
+		return NULL;
+
+	index = info->pkg_id * MAX_POWER_DOMAINS + info->punit_domain_id;
+	guard(mutex)(&tpmi_lock);
+	mask = &tpmi_power_domain_mask[index];
+
+	return mask;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_power_domain_mask, INTEL_TPMI_POWER_DOMAIN);
+
+static int tpmi_get_logical_id(unsigned int cpu, struct tpmi_cpu_info *info)
+{
+	u64 data;
+	int ret;
+
+	ret = rdmsrl_safe(MSR_PM_LOGICAL_ID, &data);
+	if (ret)
+		return ret;
+
+	info->punit_domain_id = FIELD_GET(PM_DOMAIN_ID_MASK, data);
+	if (info->punit_domain_id >= MAX_POWER_DOMAINS)
+		return -EINVAL;
+
+	info->punit_thread_id = FIELD_GET(LP_ID_MASK, data);
+	info->punit_core_id = FIELD_GET(MODULE_ID_MASK, data);
+	info->pkg_id = topology_physical_package_id(cpu);
+	info->linux_cpu = cpu;
+
+	return 0;
+}
+
+static int tpmi_cpu_online(unsigned int cpu)
+{
+	struct tpmi_cpu_info *info = &per_cpu(tpmi_cpu_info, cpu);
+	int ret, index;
+
+	/* Don't fail CPU online for some bad mapping of CPUs */
+	ret = tpmi_get_logical_id(cpu, info);
+	if (ret)
+		return 0;
+
+	index = info->pkg_id * MAX_POWER_DOMAINS + info->punit_domain_id;
+
+	guard(mutex)(&tpmi_lock);
+	cpumask_set_cpu(cpu, &tpmi_power_domain_mask[index]);
+	hash_add(tpmi_cpu_hash, &info->hnode, info->punit_core_id);
+
+	return 0;
+}
+
+static int __init tpmi_init(void)
+{
+	const struct x86_cpu_id *id;
+	u64 data;
+	int ret;
+
+	id = x86_match_cpu(tpmi_cpu_ids);
+	if (!id)
+		return -ENODEV;
+
+	/* Check for MSR 0x54 presence */
+	ret = rdmsrl_safe(MSR_PM_LOGICAL_ID, &data);
+	if (ret)
+		return ret;
+
+	tpmi_power_domain_mask = kcalloc(size_mul(topology_max_packages(), MAX_POWER_DOMAINS),
+					 sizeof(*tpmi_power_domain_mask), GFP_KERNEL);
+	if (!tpmi_power_domain_mask)
+		return -ENOMEM;
+
+	ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
+				"platform/x86/tpmi_power_domains:online",
+				tpmi_cpu_online, NULL);
+	if (ret < 0) {
+		kfree(tpmi_power_domain_mask);
+		return ret;
+	}
+
+	tpmi_hp_state = ret;
+
+	return 0;
+}
+module_init(tpmi_init)
+
+static void __exit tpmi_exit(void)
+{
+	cpuhp_remove_state(tpmi_hp_state);
+	kfree(tpmi_power_domain_mask);
+}
+module_exit(tpmi_exit)
+
+MODULE_DESCRIPTION("TPMI Power Domains Mapping");
+MODULE_LICENSE("GPL");
diff --git a/drivers/platform/x86/intel/tpmi_power_domains.h b/drivers/platform/x86/intel/tpmi_power_domains.h
new file mode 100644
index 000000000000..0c2154bd941f
--- /dev/null
+++ b/drivers/platform/x86/intel/tpmi_power_domains.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Mapping of TPMI power domain and CPUs
+ *
+ * Copyright (c) 2024, Intel Corporation.
+ * All rights reserved.
+ */
+
+#ifndef _TPMI_POWER_DOMAINS_H_
+#define _TPMI_POWER_DOMAINS_H_
+
+#include <linux/cpumask.h>
+
+int tpmi_get_linux_cpu_number(int package_id, int die_id, int punit_core_id);
+int tpmi_get_punit_core_number(int cpu_no);
+int tpmi_get_power_domain_id(int cpu_no);
+cpumask_t *tpmi_get_power_domain_mask(int cpu_no);
+
+#endif
-- 
2.43.1

[PATCH 4/6] platform/x86/intel/tpmi: Add new auxiliary driver for performance limits

[Tero Kristo]

Add new auxiliary driver that exposes the SoC performance limit reasons
via debugfs interface.

Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
 drivers/platform/x86/intel/Kconfig          |   7 +
 drivers/platform/x86/intel/Makefile         |   1 +
 drivers/platform/x86/intel/intel_plr_tpmi.c | 205 ++++++++++++++++++++
 3 files changed, 213 insertions(+)
 create mode 100644 drivers/platform/x86/intel/intel_plr_tpmi.c

diff --git a/drivers/platform/x86/intel/Kconfig b/drivers/platform/x86/intel/Kconfig
index e97a97355d5a..ad50bbabec61 100644
--- a/drivers/platform/x86/intel/Kconfig
+++ b/drivers/platform/x86/intel/Kconfig
@@ -209,6 +209,13 @@ config INTEL_TPMI
 	  To compile this driver as a module, choose M here: the module will
 	  be called intel_vsec_tpmi.
 
+config INTEL_PLR_TPMI
+        tristate "Intel SoC TPMI Power Limit Reasons driver"
+        depends on INTEL_TPMI
+        help
+          This driver provides the TPMI power limit reasons status information
+          via debugfs files.
+
 config INTEL_TURBO_MAX_3
 	bool "Intel Turbo Boost Max Technology 3.0 enumeration driver"
 	depends on X86_64 && SCHED_MC_PRIO
diff --git a/drivers/platform/x86/intel/Makefile b/drivers/platform/x86/intel/Makefile
index 10437e56027d..74db065c82d6 100644
--- a/drivers/platform/x86/intel/Makefile
+++ b/drivers/platform/x86/intel/Makefile
@@ -52,6 +52,7 @@ obj-$(CONFIG_INTEL_PUNIT_IPC)		+= intel_punit_ipc.o
 # TPMI drivers
 intel_vsec_tpmi-y			:= tpmi.o
 obj-$(CONFIG_INTEL_TPMI)		+= intel_vsec_tpmi.o
+obj-$(CONFIG_INTEL_PLR_TPMI)		+= intel_plr_tpmi.o
 
 intel_tpmi_power_domains-y		:= tpmi_power_domains.o
 obj-$(CONFIG_INTEL_TPMI_POWER_DOMAINS)	+= intel_tpmi_power_domains.o
diff --git a/drivers/platform/x86/intel/intel_plr_tpmi.c b/drivers/platform/x86/intel/intel_plr_tpmi.c
new file mode 100644
index 000000000000..5fd45dd3c396
--- /dev/null
+++ b/drivers/platform/x86/intel/intel_plr_tpmi.c
@@ -0,0 +1,205 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Performance Limit Reasons via TPMI
+ *
+ * Copyright (c) 2024, Intel Corporation.
+ * All Rights Reserved.
+ */
+
+#include <linux/array_size.h>
+#include <linux/auxiliary_bus.h>
+#include <linux/bitmap.h>
+#include <linux/debugfs.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/gfp_types.h>
+#include <linux/intel_tpmi.h>
+#include <linux/io.h>
+#include <linux/kstrtox.h>
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/seq_file.h>
+#include <linux/sprintf.h>
+#include <linux/types.h>
+
+#define PLR_HEADER		0x00
+#define PLR_DIE_LEVEL		0x18
+
+#define PLR_INVALID		GENMASK_ULL(63, 0)
+
+struct tpmi_plr_die {
+	void __iomem *base;
+};
+
+struct tpmi_plr {
+	struct dentry *dbgfs_dir;
+	struct tpmi_plr_die *die_info;
+	int num_dies;
+};
+
+static const char * const plr_coarse_reasons[] = {
+	"FREQUENCY",
+	"CURRENT",
+	"POWER",
+	"THERMAL",
+	"PLATFORM",
+	"MCP",
+	"RAS",
+	"MISC",
+	"QOS",
+	"DFC",
+};
+
+static u64 plr_read(struct tpmi_plr_die *plr_die, int offset)
+{
+	return readq(plr_die->base + offset);
+}
+
+static void plr_write(u64 val, struct tpmi_plr_die *plr_die, int offset)
+{
+	writeq(val, plr_die->base + offset);
+}
+
+static void plr_print_bits(struct seq_file *s, u64 val, int bits)
+{
+	const unsigned long mask[] = { BITMAP_FROM_U64(val) };
+	int bit;
+
+	for_each_set_bit(bit, mask, bits) {
+		if (bit >= ARRAY_SIZE(plr_coarse_reasons))
+			seq_printf(s, " UNKNOWN(%d)", bit);
+		else
+			seq_printf(s, " %s", plr_coarse_reasons[bit]);
+	}
+
+	if (!val)
+		seq_puts(s, " none");
+
+	seq_putc(s, '\n');
+}
+
+static int plr_status_show(struct seq_file *s, void *unused)
+{
+	struct tpmi_plr_die *plr_die = s->private;
+	u64 val;
+
+	val = plr_read(plr_die, PLR_DIE_LEVEL);
+	seq_puts(s, "cpus");
+	plr_print_bits(s, val, 32);
+
+	return 0;
+}
+
+static ssize_t plr_status_write(struct file *filp, const char __user *ubuf,
+				size_t count, loff_t *ppos)
+{
+	struct seq_file *s = filp->private_data;
+	struct tpmi_plr_die *plr_die = s->private;
+	bool val;
+	int ret;
+
+	ret = kstrtobool_from_user(ubuf, count, &val);
+	if (ret)
+		return ret;
+
+	if (val != 0)
+		return -EINVAL;
+
+	plr_write(0, plr_die, PLR_DIE_LEVEL);
+
+	return count;
+}
+DEFINE_SHOW_STORE_ATTRIBUTE(plr_status);
+
+static int intel_plr_probe(struct auxiliary_device *auxdev, const struct auxiliary_device_id *id)
+{
+	struct intel_tpmi_plat_info *plat_info;
+	struct dentry *dentry;
+	int i, num_resources;
+	struct resource *res;
+	struct tpmi_plr *plr;
+	void __iomem *base;
+	char name[16];
+	int err;
+
+	plat_info = tpmi_get_platform_data(auxdev);
+	if (!plat_info)
+		return dev_err_probe(&auxdev->dev, -EINVAL, "No platform info\n");
+
+	dentry = tpmi_get_debugfs_dir(auxdev);
+	if (!dentry)
+		return dev_err_probe(&auxdev->dev, -ENODEV, "No TPMI debugfs directory.\n");
+
+	num_resources = tpmi_get_resource_count(auxdev);
+	if (!num_resources)
+		return -EINVAL;
+
+	plr = devm_kzalloc(&auxdev->dev, sizeof(*plr), GFP_KERNEL);
+	if (!plr)
+		return -ENOMEM;
+
+	plr->die_info = devm_kcalloc(&auxdev->dev, num_resources, sizeof(*plr->die_info),
+				     GFP_KERNEL);
+	if (!plr->die_info)
+		return -ENOMEM;
+
+	plr->num_dies = num_resources;
+	plr->dbgfs_dir = debugfs_create_dir("plr", dentry);
+
+	for (i = 0; i < num_resources; i++) {
+		res = tpmi_get_resource_at_index(auxdev, i);
+		if (!res) {
+			err = dev_err_probe(&auxdev->dev, -EINVAL, "No resource\n");
+			goto err;
+		}
+
+		base = devm_ioremap_resource(&auxdev->dev, res);
+		if (IS_ERR(base)) {
+			err = PTR_ERR(base);
+			goto err;
+		}
+
+		plr->die_info[i].base = base;
+
+		if (plr_read(&plr->die_info[i], PLR_HEADER) == PLR_INVALID)
+			continue;
+
+		snprintf(name, sizeof(name), "domain%d", i);
+
+		dentry = debugfs_create_dir(name, plr->dbgfs_dir);
+		debugfs_create_file("status", 0444, dentry, &plr->die_info[i],
+				    &plr_status_fops);
+	}
+
+	auxiliary_set_drvdata(auxdev, plr);
+
+	return 0;
+
+err:
+	debugfs_remove_recursive(plr->dbgfs_dir);
+	return err;
+}
+
+static void intel_plr_remove(struct auxiliary_device *auxdev)
+{
+	struct tpmi_plr *plr = auxiliary_get_drvdata(auxdev);
+
+	debugfs_remove_recursive(plr->dbgfs_dir);
+}
+
+static const struct auxiliary_device_id intel_plr_id_table[] = {
+	{ .name = "intel_vsec.tpmi-plr" },
+	{}
+};
+MODULE_DEVICE_TABLE(auxiliary, intel_plr_id_table);
+
+static struct auxiliary_driver intel_plr_aux_driver = {
+	.id_table       = intel_plr_id_table,
+	.remove         = intel_plr_remove,
+	.probe          = intel_plr_probe,
+};
+module_auxiliary_driver(intel_plr_aux_driver);
+
+MODULE_IMPORT_NS(INTEL_TPMI);
+MODULE_DESCRIPTION("Intel TPMI PLR Driver");
+MODULE_LICENSE("GPL");
-- 
2.43.1

[PATCH 5/6] platform/x86/intel/tpmi/plr: Add support for the plr mailbox

[Tero Kristo]

Add support for reading fine grained power limit reasons via the PLR
mailbox.

Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
 drivers/platform/x86/intel/intel_plr_tpmi.c | 159 +++++++++++++++++++-
 1 file changed, 155 insertions(+), 4 deletions(-)

diff --git a/drivers/platform/x86/intel/intel_plr_tpmi.c b/drivers/platform/x86/intel/intel_plr_tpmi.c
index 5fd45dd3c396..c597c6052875 100644
--- a/drivers/platform/x86/intel/intel_plr_tpmi.c
+++ b/drivers/platform/x86/intel/intel_plr_tpmi.c
@@ -8,6 +8,7 @@
 
 #include <linux/array_size.h>
 #include <linux/auxiliary_bus.h>
+#include <linux/bitfield.h>
 #include <linux/bitmap.h>
 #include <linux/debugfs.h>
 #include <linux/device.h>
@@ -15,26 +16,50 @@
 #include <linux/gfp_types.h>
 #include <linux/intel_tpmi.h>
 #include <linux/io.h>
+#include <linux/iopoll.h>
 #include <linux/kstrtox.h>
+#include <linux/lockdep.h>
 #include <linux/module.h>
 #include <linux/mod_devicetable.h>
+#include <linux/mutex.h>
 #include <linux/seq_file.h>
 #include <linux/sprintf.h>
 #include <linux/types.h>
 
+#include "tpmi_power_domains.h"
+
 #define PLR_HEADER		0x00
+#define PLR_MAILBOX_INTERFACE	0x08
+#define PLR_MAILBOX_DATA	0x10
 #define PLR_DIE_LEVEL		0x18
 
+#define PLR_MODULE_ID_MASK	GENMASK_ULL(19, 12)
+#define PLR_RUN_BUSY		BIT_ULL(63)
+
+#define PLR_COMMAND_WRITE	1
+
 #define PLR_INVALID		GENMASK_ULL(63, 0)
 
+#define PLR_TIMEOUT_US		5
+#define PLR_TIMEOUT_MAX_US	1000
+
+#define PLR_COARSE_REASON_BITS	32
+
+struct tpmi_plr;
+
 struct tpmi_plr_die {
 	void __iomem *base;
+	struct mutex lock; /* Protect access to PLR mailbox */
+	int package_id;
+	int die_id;
+	struct tpmi_plr *plr;
 };
 
 struct tpmi_plr {
 	struct dentry *dbgfs_dir;
 	struct tpmi_plr_die *die_info;
 	int num_dies;
+	struct auxiliary_device *auxdev;
 };
 
 static const char * const plr_coarse_reasons[] = {
@@ -50,6 +75,39 @@ static const char * const plr_coarse_reasons[] = {
 	"DFC",
 };
 
+static const char * const plr_fine_reasons[] = {
+	"FREQUENCY_CDYN0",
+	"FREQUENCY_CDYN1",
+	"FREQUENCY_CDYN2",
+	"FREQUENCY_CDYN3",
+	"FREQUENCY_CDYN4",
+	"FREQUENCY_CDYN5",
+	"FREQUENCY_FCT",
+	"FREQUENCY_PCS_TRL",
+	"CURRENT_MTPMAX",
+	"POWER_FAST_RAPL",
+	"POWER_PKG_PL1_MSR_TPMI",
+	"POWER_PKG_PL1_MMIO",
+	"POWER_PKG_PL1_PCS",
+	"POWER_PKG_PL2_MSR_TPMI",
+	"POWER_PKG_PL2_MMIO",
+	"POWER_PKG_PL2_PCS",
+	"POWER_PLATFORM_PL1_MSR_TPMI",
+	"POWER_PLATFORM_PL1_MMIO",
+	"POWER_PLATFORM_PL1_PCS",
+	"POWER_PLATFORM_PL2_MSR_TPMI",
+	"POWER_PLATFORM_PL2_MMIO",
+	"POWER_PLATFORM_PL2_PCS",
+	"UNKNOWN(22)",
+	"THERMAL_PER_CORE",
+	"DFC_UFS",
+	"PLATFORM_PROCHOT",
+	"PLATFORM_HOT_VR",
+	"UNKNOWN(27)",
+	"UNKNOWN(28)",
+	"MISC_PCS_PSTATE",
+};
+
 static u64 plr_read(struct tpmi_plr_die *plr_die, int offset)
 {
 	return readq(plr_die->base + offset);
@@ -60,16 +118,68 @@ static void plr_write(u64 val, struct tpmi_plr_die *plr_die, int offset)
 	writeq(val, plr_die->base + offset);
 }
 
+static int plr_read_cpu_status(struct tpmi_plr_die *plr_die, int cpu,
+			       u64 *status)
+{
+	u64 regval;
+	int ret;
+
+	lockdep_assert_held(&plr_die->lock);
+
+	regval = FIELD_PREP(PLR_MODULE_ID_MASK, tpmi_get_punit_core_number(cpu));
+	regval |= PLR_RUN_BUSY;
+
+	plr_write(regval, plr_die, PLR_MAILBOX_INTERFACE);
+
+	ret = readq_poll_timeout(plr_die->base + PLR_MAILBOX_INTERFACE, regval,
+				 !(regval & PLR_RUN_BUSY), PLR_TIMEOUT_US,
+				 PLR_TIMEOUT_MAX_US);
+	if (ret)
+		return ret;
+
+	*status = plr_read(plr_die, PLR_MAILBOX_DATA);
+
+	return 0;
+}
+
+static int plr_clear_cpu_status(struct tpmi_plr_die *plr_die, int cpu)
+{
+	u64 regval;
+
+	lockdep_assert_held(&plr_die->lock);
+
+	regval = FIELD_PREP(PLR_MODULE_ID_MASK, tpmi_get_punit_core_number(cpu));
+	regval |= PLR_RUN_BUSY | PLR_COMMAND_WRITE;
+
+	plr_write(0, plr_die, PLR_MAILBOX_DATA);
+
+	plr_write(regval, plr_die, PLR_MAILBOX_INTERFACE);
+
+	return readq_poll_timeout(plr_die->base + PLR_MAILBOX_INTERFACE, regval,
+				  !(regval & PLR_RUN_BUSY), PLR_TIMEOUT_US,
+				  PLR_TIMEOUT_MAX_US);
+}
+
 static void plr_print_bits(struct seq_file *s, u64 val, int bits)
 {
 	const unsigned long mask[] = { BITMAP_FROM_U64(val) };
-	int bit;
+	const char *str;
+	int bit, index;
 
 	for_each_set_bit(bit, mask, bits) {
-		if (bit >= ARRAY_SIZE(plr_coarse_reasons))
-			seq_printf(s, " UNKNOWN(%d)", bit);
+		if (bit < PLR_COARSE_REASON_BITS) {
+			if (bit < ARRAY_SIZE(plr_coarse_reasons))
+				str = plr_coarse_reasons[bit];
+		} else {
+			index = bit - PLR_COARSE_REASON_BITS;
+			if (index < ARRAY_SIZE(plr_fine_reasons))
+				str = plr_fine_reasons[index];
+		}
+
+		if (str)
+			seq_printf(s, " %s", str);
 		else
-			seq_printf(s, " %s", plr_coarse_reasons[bit]);
+			seq_printf(s, " UNKNOWN(%d)", bit);
 	}
 
 	if (!val)
@@ -81,12 +193,33 @@ static void plr_print_bits(struct seq_file *s, u64 val, int bits)
 static int plr_status_show(struct seq_file *s, void *unused)
 {
 	struct tpmi_plr_die *plr_die = s->private;
+	int ret;
 	u64 val;
 
 	val = plr_read(plr_die, PLR_DIE_LEVEL);
 	seq_puts(s, "cpus");
 	plr_print_bits(s, val, 32);
 
+	guard(mutex)(&plr_die->lock);
+
+	for (int cpu = 0; cpu < nr_cpu_ids; cpu++) {
+		if (plr_die->die_id != tpmi_get_power_domain_id(cpu))
+			continue;
+
+		if (plr_die->package_id != topology_physical_package_id(cpu))
+			continue;
+
+		seq_printf(s, "cpu%d", cpu);
+		ret = plr_read_cpu_status(plr_die, cpu, &val);
+		if (ret) {
+			dev_err(&plr_die->plr->auxdev->dev, "Failed to read PLR for cpu %d, ret=%d\n",
+				cpu, ret);
+			return ret;
+		}
+
+		plr_print_bits(s, val, 64);
+	}
+
 	return 0;
 }
 
@@ -107,6 +240,18 @@ static ssize_t plr_status_write(struct file *filp, const char __user *ubuf,
 
 	plr_write(0, plr_die, PLR_DIE_LEVEL);
 
+	guard(mutex)(&plr_die->lock);
+
+	for (int cpu = 0; cpu < nr_cpu_ids; cpu++) {
+		if (plr_die->die_id != tpmi_get_power_domain_id(cpu))
+			continue;
+
+		if (plr_die->package_id != topology_physical_package_id(cpu))
+			continue;
+
+		plr_clear_cpu_status(plr_die, cpu);
+	}
+
 	return count;
 }
 DEFINE_SHOW_STORE_ATTRIBUTE(plr_status);
@@ -145,6 +290,7 @@ static int intel_plr_probe(struct auxiliary_device *auxdev, const struct auxilia
 
 	plr->num_dies = num_resources;
 	plr->dbgfs_dir = debugfs_create_dir("plr", dentry);
+	plr->auxdev = auxdev;
 
 	for (i = 0; i < num_resources; i++) {
 		res = tpmi_get_resource_at_index(auxdev, i);
@@ -160,6 +306,10 @@ static int intel_plr_probe(struct auxiliary_device *auxdev, const struct auxilia
 		}
 
 		plr->die_info[i].base = base;
+		plr->die_info[i].package_id = plat_info->package_id;
+		plr->die_info[i].die_id = i;
+		plr->die_info[i].plr = plr;
+		mutex_init(&plr->die_info[i].lock);
 
 		if (plr_read(&plr->die_info[i], PLR_HEADER) == PLR_INVALID)
 			continue;
@@ -201,5 +351,6 @@ static struct auxiliary_driver intel_plr_aux_driver = {
 module_auxiliary_driver(intel_plr_aux_driver);
 
 MODULE_IMPORT_NS(INTEL_TPMI);
+MODULE_IMPORT_NS(INTEL_TPMI_POWER_DOMAIN);
 MODULE_DESCRIPTION("Intel TPMI PLR Driver");
 MODULE_LICENSE("GPL");
-- 
2.43.1

[PATCH 6/6] doc: TPMI: Add entry for Performance Limit Reasons

[Tero Kristo]

Describe the new 'plr' (Performance Limit Reasons) directory contents under
the main TPMI debugfs folder.

Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
 Documentation/ABI/testing/debugfs-tpmi | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/Documentation/ABI/testing/debugfs-tpmi b/Documentation/ABI/testing/debugfs-tpmi
index 597f0475fe6e..4daa9ecd7918 100644
--- a/Documentation/ABI/testing/debugfs-tpmi
+++ b/Documentation/ABI/testing/debugfs-tpmi
@@ -29,3 +29,12 @@ Example:
 echo 0,0x20,0xff > mem_write
 echo 1,64,64 > mem_write
 Users:		Debugging, any user space test suite
+
+What:		/sys/kernel/debug/tpmi-<n>/plr/domain<n>/status
+Date:		Aug 2024
+KernelVersion:	6.11
+Contact:	Tero Kristo <tero.kristo@linux.intel.com>
+Description:
+Shows the currently active Performance Limit Reasons for die level and the
+individual CPUs under the die. The contents of this file are sticky, and
+clearing all the statuses can be done by writing "0\n" to this file.
-- 
2.43.1

Re: [PATCH 3/6] platform/x86/intel: TPMI domain id and CPU mapping

[srinivas pandruvada]

On Mon, 2024-05-27 at 16:29 +0300, Tero Kristo wrote:
> From: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
> 
> Each TPMI power domain includes a group of CPUs. Several power
> management settings in this case applicable to a group of CPUs.
> There can be several power domains in a CPU package. So, provide
> interfaces for:
> - Get power domain id for a Linux CPU
> - Get mask of Linux CPUs in a power domain
> 
> Hardware Punit uses different CPU numbering, which is not based on
> APIC (Advanced Programmable Interrupt Controller) CPU numbering.
> The Linux CPU numbering is based on APIC CPU numbering. Some PM
> features
> like Intel Speed Select, the CPU core mask provided by the hardware
> is
> based on the Punit CPU numbering. To use the core mask, this mask
> needs to be converted to a Linux CPUs mask. So, provide interfaces
> for:
> - Convert to a Linux CPU number from a Punit CPU number
> - Convert to a Punit CPU number from a Linux CPU number
> 
> On each CPU online, MSR 0x54 is used to read the mapping and stores
> in
> a per cpu array. Create a hash for faster searching of a Linux CPU
> number
> from a Punit CPU number.
> 
> Signed-off-by: Srinivas Pandruvada
> <srinivas.pandruvada@linux.intel.com>
> [tero.kristo: minor updates]
> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
> ---
>  drivers/platform/x86/intel/Kconfig            |   4 +
>  drivers/platform/x86/intel/Makefile           |   3 +
>  .../platform/x86/intel/tpmi_power_domains.c   | 236
> ++++++++++++++++++
>  .../platform/x86/intel/tpmi_power_domains.h   |  19 ++
>  4 files changed, 262 insertions(+)
>  create mode 100644 drivers/platform/x86/intel/tpmi_power_domains.c
>  create mode 100644 drivers/platform/x86/intel/tpmi_power_domains.h
> 
> diff --git a/drivers/platform/x86/intel/Kconfig
> b/drivers/platform/x86/intel/Kconfig
> index e9dc0c021029..e97a97355d5a 100644
> --- a/drivers/platform/x86/intel/Kconfig
> +++ b/drivers/platform/x86/intel/Kconfig
> @@ -192,10 +192,14 @@ config INTEL_SMARTCONNECT
>           This driver checks to determine whether the device has
> Intel Smart
>           Connect enabled, and if so disables it.
>  
> +config INTEL_TPMI_POWER_DOMAINS
> +       tristate
> +
>  config INTEL_TPMI
>         tristate "Intel Topology Aware Register and PM Capsule
> Interface (TPMI)"
>         depends on INTEL_VSEC
>         depends on X86_64
> +       select INTEL_TPMI_POWER_DOMAINS
>         help
>           The Intel Topology Aware Register and PM Capsule Interface
> (TPMI),
>           provides enumerable MMIO interface for power management
> features.
> diff --git a/drivers/platform/x86/intel/Makefile
> b/drivers/platform/x86/intel/Makefile
> index c1d5fe05e3f3..10437e56027d 100644
> --- a/drivers/platform/x86/intel/Makefile
> +++ b/drivers/platform/x86/intel/Makefile
> @@ -53,6 +53,9 @@ obj-$(CONFIG_INTEL_PUNIT_IPC)         +=
> intel_punit_ipc.o
>  intel_vsec_tpmi-y                      := tpmi.o
>  obj-$(CONFIG_INTEL_TPMI)               += intel_vsec_tpmi.o
>  
> +intel_tpmi_power_domains-y             := tpmi_power_domains.o
> +obj-$(CONFIG_INTEL_TPMI_POWER_DOMAINS) += intel_tpmi_power_domains.o
> +
>  # Intel Uncore drivers
>  intel-rst-y                            := rst.o
>  obj-$(CONFIG_INTEL_RST)                        += intel-rst.o
> diff --git a/drivers/platform/x86/intel/tpmi_power_domains.c
> b/drivers/platform/x86/intel/tpmi_power_domains.c
> new file mode 100644
> index 000000000000..40f994814248
> --- /dev/null
> +++ b/drivers/platform/x86/intel/tpmi_power_domains.c
> @@ -0,0 +1,236 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Mapping of TPMI power domains CPU mapping
> + *
> + * Copyright (c) 2024, Intel Corporation.
> + * All Rights Reserved.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/cleanup.h>
> +#include <linux/cpuhotplug.h>
> +#include <linux/cpumask.h>
> +#include <linux/errno.h>
> +#include <linux/export.h>
> +#include <linux/hashtable.h>
> +#include <linux/module.h>
> +#include <linux/mutex.h>
> +#include <linux/overflow.h>
> +#include <linux/slab.h>
> +#include <linux/topology.h>
> +#include <linux/types.h>
> +
> +#include <asm/cpu_device_id.h>
> +#include <asm/intel-family.h>
> +#include <asm/msr.h>
> +
> +#include "tpmi_power_domains.h"
> +
> +#define MSR_PM_LOGICAL_ID       0x54
> +
> +/*
> + * Struct of MSR 0x54
> + * [15:11] PM_DOMAIN_ID
> + * [10:3] MODULE_ID (aka IDI_AGENT_ID)
> + * [2:0] LP_ID
> + * For Atom:
> + *   [2] Always 0
> + *   [1:0] core ID within module
> + * For Core
> + *   [2:1] Always 0
> + *   [0] thread ID
> + */
> +
> +#define LP_ID_MASK             GENMASK_ULL(2, 0)
> +#define MODULE_ID_MASK         GENMASK_ULL(10, 3)
> +#define PM_DOMAIN_ID_MASK      GENMASK_ULL(15, 11)
> +
> +/**
> + * struct tpmi_cpu_info - Mapping information for a CPU
> + * @hnode: Used to add mapping information to hash list
> + * @linux_cpu: Linux CPU number
> + * @pkg_id: Package ID of this CPU
> + * @punit_thread_id: Punit thread id of this CPU
> + * @punit_core_id: Punit core id
> + * @punit_domain_id: Power domain id from Punit
> + *
> + * Structure to store mapping information for a Linux CPU
> + * to a Punit core, thread and power domain.
> + */
> +struct tpmi_cpu_info {
> +       struct hlist_node hnode;
> +       int linux_cpu;
> +       u8 pkg_id;
> +       u8 punit_thread_id;
> +       u8 punit_core_id;
> +       u8 punit_domain_id;
> +};
> +
> +static DEFINE_PER_CPU(struct tpmi_cpu_info, tpmi_cpu_info);
> +
> +/* The dynamically assigned cpu hotplug state to free later */
> +static enum cpuhp_state tpmi_hp_state __read_mostly;
> +
> +#define MAX_POWER_DOMAINS      8
> +
> +static cpumask_t *tpmi_power_domain_mask;
> +
> +/* Lock to protect tpmi_power_domain_mask and tpmi_cpu_hash */
> +static DEFINE_MUTEX(tpmi_lock);
> +
> +static const struct x86_cpu_id tpmi_cpu_ids[] = {
> +       X86_MATCH_INTEL_FAM6_MODEL(GRANITERAPIDS_X,     NULL),
> +       X86_MATCH_INTEL_FAM6_MODEL(ATOM_CRESTMONT_X,    NULL),
> +       X86_MATCH_INTEL_FAM6_MODEL(ATOM_CRESTMONT,      NULL),
> +       X86_MATCH_INTEL_FAM6_MODEL(GRANITERAPIDS_D,     NULL),
> +       {}
> +};

From 6.10 kernel we are supposed to use
X86_MATCH_VFM() macro.

It is better to either change here or apply a patch on top.

Thanks,
Srinivas


> +MODULE_DEVICE_TABLE(x86cpu, tpmi_cpu_ids);
> +
> +static DECLARE_HASHTABLE(tpmi_cpu_hash, 8);
> +
> +static bool tpmi_domain_is_valid(struct tpmi_cpu_info *info)
> +{
> +       return info->pkg_id < topology_max_packages() &&
> +               info->punit_domain_id < MAX_POWER_DOMAINS;
> +}
> +
> +int tpmi_get_linux_cpu_number(int package_id, int domain_id, int
> punit_core_id)
> +{
> +       struct tpmi_cpu_info *info;
> +       int ret = -EINVAL;
> +
> +       guard(mutex)(&tpmi_lock);
> +       hash_for_each_possible(tpmi_cpu_hash, info, hnode,
> punit_core_id) {
> +               if (info->punit_domain_id == domain_id && info-
> >pkg_id == package_id) {
> +                       ret = info->linux_cpu;
> +                       break;
> +               }
> +       }
> +
> +       return ret;
> +}
> +EXPORT_SYMBOL_NS_GPL(tpmi_get_linux_cpu_number,
> INTEL_TPMI_POWER_DOMAIN);
> +
> +int tpmi_get_punit_core_number(int cpu_no)
> +{
> +       if (cpu_no >= num_possible_cpus())
> +               return -EINVAL;
> +
> +       return per_cpu(tpmi_cpu_info, cpu_no).punit_core_id;
> +}
> +EXPORT_SYMBOL_NS_GPL(tpmi_get_punit_core_number,
> INTEL_TPMI_POWER_DOMAIN);
> +
> +int tpmi_get_power_domain_id(int cpu_no)
> +{
> +       if (cpu_no >= num_possible_cpus())
> +               return -EINVAL;
> +
> +       return per_cpu(tpmi_cpu_info, cpu_no).punit_domain_id;
> +}
> +EXPORT_SYMBOL_NS_GPL(tpmi_get_power_domain_id,
> INTEL_TPMI_POWER_DOMAIN);
> +
> +cpumask_t *tpmi_get_power_domain_mask(int cpu_no)
> +{
> +       struct tpmi_cpu_info *info;
> +       cpumask_t *mask;
> +       int index;
> +
> +       if (cpu_no >= num_possible_cpus())
> +               return NULL;
> +
> +       info = &per_cpu(tpmi_cpu_info, cpu_no);
> +       if (!tpmi_domain_is_valid(info))
> +               return NULL;
> +
> +       index = info->pkg_id * MAX_POWER_DOMAINS + info-
> >punit_domain_id;
> +       guard(mutex)(&tpmi_lock);
> +       mask = &tpmi_power_domain_mask[index];
> +
> +       return mask;
> +}
> +EXPORT_SYMBOL_NS_GPL(tpmi_get_power_domain_mask,
> INTEL_TPMI_POWER_DOMAIN);
> +
> +static int tpmi_get_logical_id(unsigned int cpu, struct
> tpmi_cpu_info *info)
> +{
> +       u64 data;
> +       int ret;
> +
> +       ret = rdmsrl_safe(MSR_PM_LOGICAL_ID, &data);
> +       if (ret)
> +               return ret;
> +
> +       info->punit_domain_id = FIELD_GET(PM_DOMAIN_ID_MASK, data);
> +       if (info->punit_domain_id >= MAX_POWER_DOMAINS)
> +               return -EINVAL;
> +
> +       info->punit_thread_id = FIELD_GET(LP_ID_MASK, data);
> +       info->punit_core_id = FIELD_GET(MODULE_ID_MASK, data);
> +       info->pkg_id = topology_physical_package_id(cpu);
> +       info->linux_cpu = cpu;
> +
> +       return 0;
> +}
> +
> +static int tpmi_cpu_online(unsigned int cpu)
> +{
> +       struct tpmi_cpu_info *info = &per_cpu(tpmi_cpu_info, cpu);
> +       int ret, index;
> +
> +       /* Don't fail CPU online for some bad mapping of CPUs */
> +       ret = tpmi_get_logical_id(cpu, info);
> +       if (ret)
> +               return 0;
> +
> +       index = info->pkg_id * MAX_POWER_DOMAINS + info-
> >punit_domain_id;
> +
> +       guard(mutex)(&tpmi_lock);
> +       cpumask_set_cpu(cpu, &tpmi_power_domain_mask[index]);
> +       hash_add(tpmi_cpu_hash, &info->hnode, info->punit_core_id);
> +
> +       return 0;
> +}
> +
> +static int __init tpmi_init(void)
> +{
> +       const struct x86_cpu_id *id;
> +       u64 data;
> +       int ret;
> +
> +       id = x86_match_cpu(tpmi_cpu_ids);
> +       if (!id)
> +               return -ENODEV;
> +
> +       /* Check for MSR 0x54 presence */
> +       ret = rdmsrl_safe(MSR_PM_LOGICAL_ID, &data);
> +       if (ret)
> +               return ret;
> +
> +       tpmi_power_domain_mask =
> kcalloc(size_mul(topology_max_packages(), MAX_POWER_DOMAINS),
> +                                       
> sizeof(*tpmi_power_domain_mask), GFP_KERNEL);
> +       if (!tpmi_power_domain_mask)
> +               return -ENOMEM;
> +
> +       ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
> +                               "platform/x86/tpmi_power_domains:onli
> ne",
> +                               tpmi_cpu_online, NULL);
> +       if (ret < 0) {
> +               kfree(tpmi_power_domain_mask);
> +               return ret;
> +       }
> +
> +       tpmi_hp_state = ret;
> +
> +       return 0;
> +}
> +module_init(tpmi_init)
> +
> +static void __exit tpmi_exit(void)
> +{
> +       cpuhp_remove_state(tpmi_hp_state);
> +       kfree(tpmi_power_domain_mask);
> +}
> +module_exit(tpmi_exit)
> +
> +MODULE_DESCRIPTION("TPMI Power Domains Mapping");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/platform/x86/intel/tpmi_power_domains.h
> b/drivers/platform/x86/intel/tpmi_power_domains.h
> new file mode 100644
> index 000000000000..0c2154bd941f
> --- /dev/null
> +++ b/drivers/platform/x86/intel/tpmi_power_domains.h
> @@ -0,0 +1,19 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Mapping of TPMI power domain and CPUs
> + *
> + * Copyright (c) 2024, Intel Corporation.
> + * All rights reserved.
> + */
> +
> +#ifndef _TPMI_POWER_DOMAINS_H_
> +#define _TPMI_POWER_DOMAINS_H_
> +
> +#include <linux/cpumask.h>
> +
> +int tpmi_get_linux_cpu_number(int package_id, int die_id, int
> punit_core_id);
> +int tpmi_get_punit_core_number(int cpu_no);
> +int tpmi_get_power_domain_id(int cpu_no);
> +cpumask_t *tpmi_get_power_domain_mask(int cpu_no);
> +
> +#endif

[PATCH v2 3/6] platform/x86/intel: TPMI domain id and CPU mapping

[Tero Kristo]

From: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>

Each TPMI power domain includes a group of CPUs. Several power
management settings in this case applicable to a group of CPUs.
There can be several power domains in a CPU package. So, provide
interfaces for:
- Get power domain id for a Linux CPU
- Get mask of Linux CPUs in a power domain

Hardware Punit uses different CPU numbering, which is not based on
APIC (Advanced Programmable Interrupt Controller) CPU numbering.
The Linux CPU numbering is based on APIC CPU numbering. Some PM features
like Intel Speed Select, the CPU core mask provided by the hardware is
based on the Punit CPU numbering. To use the core mask, this mask
needs to be converted to a Linux CPUs mask. So, provide interfaces for:
- Convert to a Linux CPU number from a Punit CPU number
- Convert to a Punit CPU number from a Linux CPU number

On each CPU online, MSR 0x54 is used to read the mapping and stores in
a per cpu array. Create a hash for faster searching of a Linux CPU number
from a Punit CPU number.

Signed-off-by: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
[tero.kristo: minor updates]
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
---
v2:
  * changed to use X86_MATCH_VFM() instead of X86_MATCH_INTEL_FAM6_MODEL()

 drivers/platform/x86/intel/Kconfig            |   4 +
 drivers/platform/x86/intel/Makefile           |   3 +
 .../platform/x86/intel/tpmi_power_domains.c   | 236 ++++++++++++++++++
 .../platform/x86/intel/tpmi_power_domains.h   |  19 ++
 4 files changed, 262 insertions(+)
 create mode 100644 drivers/platform/x86/intel/tpmi_power_domains.c
 create mode 100644 drivers/platform/x86/intel/tpmi_power_domains.h

diff --git a/drivers/platform/x86/intel/Kconfig b/drivers/platform/x86/intel/Kconfig
index e9dc0c021029..e97a97355d5a 100644
--- a/drivers/platform/x86/intel/Kconfig
+++ b/drivers/platform/x86/intel/Kconfig
@@ -192,10 +192,14 @@ config INTEL_SMARTCONNECT
 	  This driver checks to determine whether the device has Intel Smart
 	  Connect enabled, and if so disables it.
 
+config INTEL_TPMI_POWER_DOMAINS
+	tristate
+
 config INTEL_TPMI
 	tristate "Intel Topology Aware Register and PM Capsule Interface (TPMI)"
 	depends on INTEL_VSEC
 	depends on X86_64
+	select INTEL_TPMI_POWER_DOMAINS
 	help
 	  The Intel Topology Aware Register and PM Capsule Interface (TPMI),
 	  provides enumerable MMIO interface for power management features.
diff --git a/drivers/platform/x86/intel/Makefile b/drivers/platform/x86/intel/Makefile
index c1d5fe05e3f3..10437e56027d 100644
--- a/drivers/platform/x86/intel/Makefile
+++ b/drivers/platform/x86/intel/Makefile
@@ -53,6 +53,9 @@ obj-$(CONFIG_INTEL_PUNIT_IPC)		+= intel_punit_ipc.o
 intel_vsec_tpmi-y			:= tpmi.o
 obj-$(CONFIG_INTEL_TPMI)		+= intel_vsec_tpmi.o
 
+intel_tpmi_power_domains-y		:= tpmi_power_domains.o
+obj-$(CONFIG_INTEL_TPMI_POWER_DOMAINS)	+= intel_tpmi_power_domains.o
+
 # Intel Uncore drivers
 intel-rst-y				:= rst.o
 obj-$(CONFIG_INTEL_RST)			+= intel-rst.o
diff --git a/drivers/platform/x86/intel/tpmi_power_domains.c b/drivers/platform/x86/intel/tpmi_power_domains.c
new file mode 100644
index 000000000000..40f994814248
--- /dev/null
+++ b/drivers/platform/x86/intel/tpmi_power_domains.c
@@ -0,0 +1,236 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Mapping of TPMI power domains CPU mapping
+ *
+ * Copyright (c) 2024, Intel Corporation.
+ * All Rights Reserved.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/cleanup.h>
+#include <linux/cpuhotplug.h>
+#include <linux/cpumask.h>
+#include <linux/errno.h>
+#include <linux/export.h>
+#include <linux/hashtable.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/overflow.h>
+#include <linux/slab.h>
+#include <linux/topology.h>
+#include <linux/types.h>
+
+#include <asm/cpu_device_id.h>
+#include <asm/intel-family.h>
+#include <asm/msr.h>
+
+#include "tpmi_power_domains.h"
+
+#define MSR_PM_LOGICAL_ID       0x54
+
+/*
+ * Struct of MSR 0x54
+ * [15:11] PM_DOMAIN_ID
+ * [10:3] MODULE_ID (aka IDI_AGENT_ID)
+ * [2:0] LP_ID
+ * For Atom:
+ *   [2] Always 0
+ *   [1:0] core ID within module
+ * For Core
+ *   [2:1] Always 0
+ *   [0] thread ID
+ */
+
+#define LP_ID_MASK		GENMASK_ULL(2, 0)
+#define MODULE_ID_MASK		GENMASK_ULL(10, 3)
+#define PM_DOMAIN_ID_MASK	GENMASK_ULL(15, 11)
+
+/**
+ * struct tpmi_cpu_info - Mapping information for a CPU
+ * @hnode: Used to add mapping information to hash list
+ * @linux_cpu:	Linux CPU number
+ * @pkg_id: Package ID of this CPU
+ * @punit_thread_id: Punit thread id of this CPU
+ * @punit_core_id: Punit core id
+ * @punit_domain_id: Power domain id from Punit
+ *
+ * Structure to store mapping information for a Linux CPU
+ * to a Punit core, thread and power domain.
+ */
+struct tpmi_cpu_info {
+	struct hlist_node hnode;
+	int linux_cpu;
+	u8 pkg_id;
+	u8 punit_thread_id;
+	u8 punit_core_id;
+	u8 punit_domain_id;
+};
+
+static DEFINE_PER_CPU(struct tpmi_cpu_info, tpmi_cpu_info);
+
+/* The dynamically assigned cpu hotplug state to free later */
+static enum cpuhp_state tpmi_hp_state __read_mostly;
+
+#define MAX_POWER_DOMAINS	8
+
+static cpumask_t *tpmi_power_domain_mask;
+
+/* Lock to protect tpmi_power_domain_mask and tpmi_cpu_hash */
+static DEFINE_MUTEX(tpmi_lock);
+
+static const struct x86_cpu_id tpmi_cpu_ids[] = {
+	X86_MATCH_VFM(INTEL_GRANITERAPIDS_X,	NULL),
+	X86_MATCH_VFM(INTEL_ATOM_CRESTMONT_X,	NULL),
+	X86_MATCH_VFM(INTEL_ATOM_CRESTMONT,	NULL),
+	X86_MATCH_VFM(INTEL_GRANITERAPIDS_D,	NULL),
+	{}
+};
+MODULE_DEVICE_TABLE(x86cpu, tpmi_cpu_ids);
+
+static DECLARE_HASHTABLE(tpmi_cpu_hash, 8);
+
+static bool tpmi_domain_is_valid(struct tpmi_cpu_info *info)
+{
+	return info->pkg_id < topology_max_packages() &&
+		info->punit_domain_id < MAX_POWER_DOMAINS;
+}
+
+int tpmi_get_linux_cpu_number(int package_id, int domain_id, int punit_core_id)
+{
+	struct tpmi_cpu_info *info;
+	int ret = -EINVAL;
+
+	guard(mutex)(&tpmi_lock);
+	hash_for_each_possible(tpmi_cpu_hash, info, hnode, punit_core_id) {
+		if (info->punit_domain_id == domain_id && info->pkg_id == package_id) {
+			ret = info->linux_cpu;
+			break;
+		}
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_linux_cpu_number, INTEL_TPMI_POWER_DOMAIN);
+
+int tpmi_get_punit_core_number(int cpu_no)
+{
+	if (cpu_no >= num_possible_cpus())
+		return -EINVAL;
+
+	return per_cpu(tpmi_cpu_info, cpu_no).punit_core_id;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_punit_core_number, INTEL_TPMI_POWER_DOMAIN);
+
+int tpmi_get_power_domain_id(int cpu_no)
+{
+	if (cpu_no >= num_possible_cpus())
+		return -EINVAL;
+
+	return per_cpu(tpmi_cpu_info, cpu_no).punit_domain_id;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_power_domain_id, INTEL_TPMI_POWER_DOMAIN);
+
+cpumask_t *tpmi_get_power_domain_mask(int cpu_no)
+{
+	struct tpmi_cpu_info *info;
+	cpumask_t *mask;
+	int index;
+
+	if (cpu_no >= num_possible_cpus())
+		return NULL;
+
+	info = &per_cpu(tpmi_cpu_info, cpu_no);
+	if (!tpmi_domain_is_valid(info))
+		return NULL;
+
+	index = info->pkg_id * MAX_POWER_DOMAINS + info->punit_domain_id;
+	guard(mutex)(&tpmi_lock);
+	mask = &tpmi_power_domain_mask[index];
+
+	return mask;
+}
+EXPORT_SYMBOL_NS_GPL(tpmi_get_power_domain_mask, INTEL_TPMI_POWER_DOMAIN);
+
+static int tpmi_get_logical_id(unsigned int cpu, struct tpmi_cpu_info *info)
+{
+	u64 data;
+	int ret;
+
+	ret = rdmsrl_safe(MSR_PM_LOGICAL_ID, &data);
+	if (ret)
+		return ret;
+
+	info->punit_domain_id = FIELD_GET(PM_DOMAIN_ID_MASK, data);
+	if (info->punit_domain_id >= MAX_POWER_DOMAINS)
+		return -EINVAL;
+
+	info->punit_thread_id = FIELD_GET(LP_ID_MASK, data);
+	info->punit_core_id = FIELD_GET(MODULE_ID_MASK, data);
+	info->pkg_id = topology_physical_package_id(cpu);
+	info->linux_cpu = cpu;
+
+	return 0;
+}
+
+static int tpmi_cpu_online(unsigned int cpu)
+{
+	struct tpmi_cpu_info *info = &per_cpu(tpmi_cpu_info, cpu);
+	int ret, index;
+
+	/* Don't fail CPU online for some bad mapping of CPUs */
+	ret = tpmi_get_logical_id(cpu, info);
+	if (ret)
+		return 0;
+
+	index = info->pkg_id * MAX_POWER_DOMAINS + info->punit_domain_id;
+
+	guard(mutex)(&tpmi_lock);
+	cpumask_set_cpu(cpu, &tpmi_power_domain_mask[index]);
+	hash_add(tpmi_cpu_hash, &info->hnode, info->punit_core_id);
+
+	return 0;
+}
+
+static int __init tpmi_init(void)
+{
+	const struct x86_cpu_id *id;
+	u64 data;
+	int ret;
+
+	id = x86_match_cpu(tpmi_cpu_ids);
+	if (!id)
+		return -ENODEV;
+
+	/* Check for MSR 0x54 presence */
+	ret = rdmsrl_safe(MSR_PM_LOGICAL_ID, &data);
+	if (ret)
+		return ret;
+
+	tpmi_power_domain_mask = kcalloc(size_mul(topology_max_packages(), MAX_POWER_DOMAINS),
+					 sizeof(*tpmi_power_domain_mask), GFP_KERNEL);
+	if (!tpmi_power_domain_mask)
+		return -ENOMEM;
+
+	ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
+				"platform/x86/tpmi_power_domains:online",
+				tpmi_cpu_online, NULL);
+	if (ret < 0) {
+		kfree(tpmi_power_domain_mask);
+		return ret;
+	}
+
+	tpmi_hp_state = ret;
+
+	return 0;
+}
+module_init(tpmi_init)
+
+static void __exit tpmi_exit(void)
+{
+	cpuhp_remove_state(tpmi_hp_state);
+	kfree(tpmi_power_domain_mask);
+}
+module_exit(tpmi_exit)
+
+MODULE_DESCRIPTION("TPMI Power Domains Mapping");
+MODULE_LICENSE("GPL");
diff --git a/drivers/platform/x86/intel/tpmi_power_domains.h b/drivers/platform/x86/intel/tpmi_power_domains.h
new file mode 100644
index 000000000000..0c2154bd941f
--- /dev/null
+++ b/drivers/platform/x86/intel/tpmi_power_domains.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Mapping of TPMI power domain and CPUs
+ *
+ * Copyright (c) 2024, Intel Corporation.
+ * All rights reserved.
+ */
+
+#ifndef _TPMI_POWER_DOMAINS_H_
+#define _TPMI_POWER_DOMAINS_H_
+
+#include <linux/cpumask.h>
+
+int tpmi_get_linux_cpu_number(int package_id, int die_id, int punit_core_id);
+int tpmi_get_punit_core_number(int cpu_no);
+int tpmi_get_power_domain_id(int cpu_no);
+cpumask_t *tpmi_get_power_domain_mask(int cpu_no);
+
+#endif
-- 
2.43.1

Re: [PATCH v2 3/6] platform/x86/intel: TPMI domain id and CPU mapping

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

On Tue, 28 May 2024, Tero Kristo wrote:

> From: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
> 
> Each TPMI power domain includes a group of CPUs. Several power
> management settings in this case applicable to a group of CPUs.
> There can be several power domains in a CPU package. So, provide
> interfaces for:
> - Get power domain id for a Linux CPU
> - Get mask of Linux CPUs in a power domain
> 
> Hardware Punit uses different CPU numbering, which is not based on
> APIC (Advanced Programmable Interrupt Controller) CPU numbering.
> The Linux CPU numbering is based on APIC CPU numbering. Some PM features
> like Intel Speed Select, the CPU core mask provided by the hardware is
> based on the Punit CPU numbering. To use the core mask, this mask
> needs to be converted to a Linux CPUs mask. So, provide interfaces for:
> - Convert to a Linux CPU number from a Punit CPU number
> - Convert to a Punit CPU number from a Linux CPU number
> 
> On each CPU online, MSR 0x54 is used to read the mapping and stores in
> a per cpu array. Create a hash for faster searching of a Linux CPU number
> from a Punit CPU number.
> 
> Signed-off-by: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
> [tero.kristo: minor updates]
> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
> ---
> v2:
>   * changed to use X86_MATCH_VFM() instead of X86_MATCH_INTEL_FAM6_MODEL()

I've applied this v2 + the other patches from v1 series now to review-ilpo 
branch.

For the record, I removed "All rights reserved." lines from the patches 
while applying. I asking first (privately) a permission from Tero whether 
it's okay with him I remove those lines.

-- 
 i.

Re: [PATCH v2 3/6] platform/x86/intel: TPMI domain id and CPU mapping

[Tero Kristo]

On Fri, 2024-05-31 at 16:45 +0300, Ilpo Järvinen wrote:
> On Tue, 28 May 2024, Tero Kristo wrote:
> 
> > From: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
> > 
> > Each TPMI power domain includes a group of CPUs. Several power
> > management settings in this case applicable to a group of CPUs.
> > There can be several power domains in a CPU package. So, provide
> > interfaces for:
> > - Get power domain id for a Linux CPU
> > - Get mask of Linux CPUs in a power domain
> > 
> > Hardware Punit uses different CPU numbering, which is not based on
> > APIC (Advanced Programmable Interrupt Controller) CPU numbering.
> > The Linux CPU numbering is based on APIC CPU numbering. Some PM
> > features
> > like Intel Speed Select, the CPU core mask provided by the hardware
> > is
> > based on the Punit CPU numbering. To use the core mask, this mask
> > needs to be converted to a Linux CPUs mask. So, provide interfaces
> > for:
> > - Convert to a Linux CPU number from a Punit CPU number
> > - Convert to a Punit CPU number from a Linux CPU number
> > 
> > On each CPU online, MSR 0x54 is used to read the mapping and stores
> > in
> > a per cpu array. Create a hash for faster searching of a Linux CPU
> > number
> > from a Punit CPU number.
> > 
> > Signed-off-by: Srinivas Pandruvada
> > <srinivas.pandruvada@linux.intel.com>
> > [tero.kristo: minor updates]
> > Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> > Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > Signed-off-by: Tero Kristo <tero.kristo@linux.intel.com>
> > ---
> > v2:
> >   * changed to use X86_MATCH_VFM() instead of
> > X86_MATCH_INTEL_FAM6_MODEL()
> 
> I've applied this v2 + the other patches from v1 series now to
> review-ilpo 
> branch.
> 
> For the record, I removed "All rights reserved." lines from the
> patches 
> while applying. I asking first (privately) a permission from Tero
> whether 
> it's okay with him I remove those lines.
> 

Thanks Ilpo!

-Tero