[Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues

[Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues

[Dapeng Mi]

This patch series fixes several x86 LBR sampling issues discovered during
review and debugging. Most of them are related to BPF LBR snapshotting
support.

The series originally targeted the BPF stack out-of-bounds access issue,
but additional problems were found during debugging and code review,
including:
- PMU and LBR being re-enabled unconditionally at BPF snapshotting exit,
  even when they had already been disabled before BPF snapshotting on both
  AMD and Intel.
- LBR not being fully frozen in the PMI handler on Intel, which allows
  branches in the PMI path to pollute the BPF LBR snapshot.
- BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on
  Intel.
- PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on
  Intel when both overflow in the same PMI.
- LBR counter events reporting zero counters on Intel when multiple
  LBR counter events overflow in the same PMI.


Patch layout:
- Patch 01: Add a sanity check to ensure the LBR stack does not exceed
  MAX_LBR_ENTRIES.
- Patch 02: Validate that hardware LBR data has actually been stored in
  the XSAVE area before parsing it.
- Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on
  AMD and Intel separately.
- Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue
  in perf_snapshot_branch_stack() for AMD and Intel separately.
- Patch 09: Freeze LBR throughout the PMI processing period and ensure
  all LBR events observe a consistent LBR stack in the PMI handler.
- Patch 10: Add counted bounds tracking for perf_branch_stack{}.
- Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshotting to avoid
  overwriting the default LBR stack of other LBR events.
- Patch 13: Ensure perf_snapshot_branch_stack() snapshots the live
  hardware LBR data instead of the cached PEBS LBR stack.
- Patch 14: Refresh LBR stack in PMI only when LBR users are active for
  AMD.
- Patch 15: Prevent PEBS LBR data from overwriting non-PEBS LBR data when
  both overflow in the same PMI.
- Patch 16: Remove dead lbr_pebs_users and relative code.
- Patch 17-18: Ensure LBR counters are not dropped when multiple LBR
  counter events overflow in the same PMI.
- Patch 19: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to
  avoid LBR stack out-of-bounds access.
- Patch 20: Fix the shared LBR stack depth truncation issue.


Changes in v6:
- Patch 01: Move LBR stack depth validation into common
  init_hw_perf_events(). (Peter)
- Patch 10: New patch, add __counted_by() tracking for flexible array
  entries[] of perf_branch_stack{}. (Peter)
- Patch 14: Refresh LBR stack only when LBR users are active for AMD.
  (Peter)
- Patch 15: Refactor code to ensure no duplicated LBR stack reading.
  (Peter)
- Patch 16: New patch, remove dead lbr_pebs_users.

Changes in v5:
- Patch 02: Clear xsave->header.xfeatures before calling the xsaves
  instructuion to read LBR states.
- Patch 05-18: New patches to fix newly discovered issues described above.

Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds
  MAX_LBR_ENTRIES (Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack
  for BPF path (Sashiko).

Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude
  MAX_LBR_ENTRIES (Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers,
  which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS
  events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).

Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
  https://lore.kernel.org/all/eb6c7bda-0dcf-4fac-ad69-9f5b24abc21b@linux.intel.com/

History:
v5: https://lore.kernel.org/all/20261029043914.1395147-1-dapeng1.mi@linux.intel.com/
v4: https://lore.kernel.org/all/20261021050711.4043703-1-dapeng1.mi@linux.intel.com/
v3: https://lore.kernel.org/all/20261019070836.2972403-1-dapeng1.mi@linux.intel.com/
v2: https://lore.kernel.org/all/cover.1792064123.git.sandipan.das@amd.com/
v1: https://lore.kernel.org/all/20261007181002.4005262-1-sandipan.das@amd.com/

Dapeng Mi (19):
  perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
  perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
  perf/x86/intel/lbr: Limit branch snapshots to valid entries
  perf/x86: Add lbr_enabled to track LBR state
  perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
  perf/x86/intel: Add helper to enable PMU without LBR
  perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
  perf/x86/intel: Keep LBR frozen throughout PMI handling
  perf: Add counted-bounds tracking for perf_branch_stack
  perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
  perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
  perf/x86/intel: Always snapshot live hardware LBR state for BPF
  perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active
  perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
  perf/x86/intel: Remove dead lbr_pebs_users
  perf/x86/intel: Clear raw LBR counters in the LBR stack
  perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
  perf/x86/intel: Pass LBR stack size into store_lbr()
  perf/core: Don't overwrite shared LBR stack depth

Sandipan Das (1):
  perf/x86/amd/lbr: Limit branch snapshots to valid entries

 arch/powerpc/perf/core-book3s.c |  43 +++++---
 arch/x86/events/amd/brs.c       |   7 +-
 arch/x86/events/amd/core.c      |  21 ++--
 arch/x86/events/amd/lbr.c       |  48 +++++----
 arch/x86/events/core.c          |   8 +-
 arch/x86/events/intel/core.c    | 130 ++++++++++++++++++++----
 arch/x86/events/intel/ds.c      |  20 ++--
 arch/x86/events/intel/lbr.c     | 170 +++++++++++++++++++-------------
 arch/x86/events/perf_event.h    |  45 +++++++--
 drivers/perf/arm_brbe.c         |   5 +
 include/linux/perf_event.h      |  15 ++-
 kernel/events/core.c            |   8 +-
 12 files changed, 365 insertions(+), 155 deletions(-)


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.34.1

[Patch v6 01/20] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES

[Dapeng Mi]

Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing
out-of-bounds access in the cached LBR stack handling path.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/core.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 4e344fb1ae3e..82d9b0e5ad1c 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)
 	if (!x86_pmu.config_mask)
 		x86_pmu.config_mask = X86_RAW_EVENT_MASK;
 
+	if (x86_pmu.lbr_nr > MAX_LBR_ENTRIES) {
+		pr_warn("LBR entries %d exceeds max %d, disable LBR.\n",
+			x86_pmu.lbr_nr, MAX_LBR_ENTRIES);
+		x86_pmu.lbr_nr = 0;
+	}
+
 	perf_events_lapic_init();
 	register_nmi_handler(NMI_LOCAL, perf_event_nmi_handler, 0, "PMI");
 
-- 
2.34.1

[Patch v6 02/20] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack

[Dapeng Mi]

intel_pmu_arch_lbr_read_xsave() doesn't check if the LBR data is present
in the xsave area. If LBR data is in the INIT state, the LBR data may not
be really written into the xsave area.

Add an explicit LBR XSAVE state presence check before updating the cached
LBR stack. Also clear the cached LBR stack when intel_pmu_lbr_reset() is
called, in addition to resetting the hardware LBR register file, to prevent
stale LBR data from being reused after a context switch or similar
transition.

Fixes: c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/lbr.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 0b446d8912e4..a2e4ba7e1a4f 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
+	cpuc->lbr_stack.nr = 0;
+
 	if (!x86_pmu.lbr_nr)
 		return;
 
@@ -1010,9 +1012,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
 		intel_pmu_store_lbr(cpuc, NULL);
 		return;
 	}
+	xsave->header.xfeatures = 0;
 	xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
 
-	intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+	if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
+		intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+	} else {
+		/* Clear the LBR stack to avoid leaking stale entries. */
+		cpuc->lbr_stack.nr = 0;
+	}
 }
 
 void intel_pmu_lbr_read(void)
-- 
2.34.1

[Patch v6 03/20] perf/x86/amd/lbr: Limit branch snapshots to valid entries

[Dapeng Mi]

From: Sandipan Das <sandipan.das@amd.com>

The branch snapshot callback used by bpf_get_branch_snapshot() copies
up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
last LBR read kept fewer branches. The entries past lbr_stack.nr are
invalid, but they end up reaching BPF programs.

Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
amd_pmu_lbr_read() skips reading the registers, so that a stale count
is never reused.

Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/amd/core.c | 2 +-
 arch/x86/events/amd/lbr.c  | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 49b6b8fce566..bf631f60757c 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 	cpuc = this_cpu_ptr(&cpu_hw_events);
 
 	amd_pmu_lbr_read();
-	cnt = min(cnt, x86_pmu.lbr_nr);
+	cnt = min(cnt, cpuc->lbr_stack.nr);
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 
 	amd_pmu_v2_enable_all(0);
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index a55646fcb846..6c93b8042c73 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)
 	struct branch_entry entry;
 	int out = 0, idx, i;
 
-	if (!cpuc->lbr_users)
+	if (!cpuc->lbr_users) {
+		cpuc->lbr_stack.nr = 0;
 		return;
+	}
 
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		entry.from.full	= amd_pmu_lbr_get_from(i);
-- 
2.34.1

[Patch v6 04/20] perf/x86/intel/lbr: Limit branch snapshots to valid entries

[Dapeng Mi]

The branch snapshot callback used by bpf_get_branch_snapshot() always
copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
last LBR read kept fewer branches. The entries past lbr_stack.nr hold
invalid branches which end up reaching BPF programs.

Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
no active LBR events or LBR hardware is owned by guest. So the stale or
guest LBR stack data can't be leaked.

For the case where all LBR events are PEBS events, hardware LBR reads are
currently skipped; that can leave stale LBR stack data from previous PEBS
sampling in place and may be inconsistent with the intended semantics of
bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.

Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
Co-developed-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 2 +-
 arch/x86/events/intel/lbr.c  | 9 +++++++--
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 0a34d674df59..5dfb9e9156e8 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
 	intel_pmu_lbr_read();
-	cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr);
+	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
 
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 	intel_pmu_enable_all(0);
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index a2e4ba7e1a4f..1ae853f7ac1a 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -1027,14 +1027,19 @@ void intel_pmu_lbr_read(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
+	if (!cpuc->lbr_users || vlbr_exclude_host()) {
+		/* Reset lbr_stack.nr to prevent stale data leakage. */
+		cpuc->lbr_stack.nr = 0;
+		return;
+	}
+
 	/*
 	 * Don't read when all LBRs users are using adaptive PEBS.
 	 *
 	 * This could be smarter and actually check the event,
 	 * but this simple approach seems to work for now.
 	 */
-	if (!cpuc->lbr_users || vlbr_exclude_host() ||
-	    cpuc->lbr_users == cpuc->lbr_pebs_users)
+	if (cpuc->lbr_users == cpuc->lbr_pebs_users)
 		return;
 
 	x86_pmu.lbr_read(cpuc);
-- 
2.34.1

[Patch v6 05/20] perf/x86: Add lbr_enabled to track LBR state

[Dapeng Mi]

Add a lbr_enabled flag to track whether LBR is enabled. This will be
used by a subsequent change to avoid blindly re-enabling LBR in
perf_snapshot_branch_stack() without checking whether LBR has already
been disabled.

Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/amd/lbr.c    | 2 ++
 arch/x86/events/intel/core.c | 5 +++++
 arch/x86/events/intel/lbr.c  | 2 ++
 arch/x86/events/perf_event.h | 9 +++++++++
 4 files changed, 18 insertions(+)

diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 6c93b8042c73..400c94ecded0 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -399,6 +399,8 @@ void amd_pmu_lbr_enable_all(void)
 	if (!cpuc->lbr_users || !x86_pmu.lbr_nr)
 		return;
 
+	cpuc->lbr_enabled = 1;
+
 	/* Set hardware branch filter */
 	if (cpuc->lbr_select) {
 		lbr_select = cpuc->lbr_sel->config & LBR_SELECT_MASK;
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 5dfb9e9156e8..e99b311c7b8d 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4001,6 +4001,11 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	 * It needs to be restored when leaving the handler.
 	 */
 	pmu_enabled = cpuc->enabled;
+
+	/* If pmu version < 4, LBR freeze disables LBR. */
+	if (x86_pmu.version < 4)
+		cpuc->lbr_enabled = 0;
+
 	/*
 	 * In general, the early ACK is only applied for old platforms.
 	 * For the big core starts from Haswell, the late ACK should be
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 1ae853f7ac1a..ee724b28d90c 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -132,6 +132,8 @@ static void __intel_pmu_lbr_enable(bool pmi)
 	if (pmi && x86_pmu.version >= 4)
 		return;
 
+	cpuc->lbr_enabled = 1;
+
 	/*
 	 * No need to reprogram LBR_SELECT in a PMI, as it
 	 * did not change.
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index e274802ef062..a4aab8130e92 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -473,6 +473,7 @@ struct cpu_hw_events {
 	unsigned long		active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
 	unsigned long		dirty[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
 	int			enabled;
+	int			lbr_enabled;
 
 	int			n_events; /* the # of events in the below arrays */
 	int			n_added;  /* the # last events in the below arrays;
@@ -1676,8 +1677,11 @@ int amd_pmu_lbr_hw_config(struct perf_event *event);
 
 static __always_inline void __amd_pmu_lbr_disable(void)
 {
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	u64 dbg_ctl, dbg_extn_cfg;
 
+	cpuc->lbr_enabled = 0;
+
 	rdmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg);
 	wrmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg & ~DBG_EXTN_CFG_LBRV2EN);
 
@@ -1829,13 +1833,18 @@ static __always_inline void __intel_pmu_pebs_disable_all(void)
 
 static __always_inline void __intel_pmu_arch_lbr_disable(void)
 {
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+	cpuc->lbr_enabled = 0;
 	wrmsrq(MSR_ARCH_LBR_CTL, 0);
 }
 
 static __always_inline void __intel_pmu_lbr_disable(void)
 {
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	u64 debugctl;
 
+	cpuc->lbr_enabled = 0;
 	rdmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
 	debugctl &= ~(DEBUGCTLMSR_LBR | DEBUGCTLMSR_FREEZE_LBRS_ON_PMI);
 	wrmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
-- 
2.34.1

[Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled

[Dapeng Mi]

amd_pmu_v2_snapshot_branch_stack() currently re-enables the PMU and LBR
unconditionally even when they were already disabled in PMI context
This can cause events such as cycles and instructions to be counted
while the PMI handler is running, which is not intended.

Only restore the PMU and LBR on the BPF path if they were originally
enabled.

Cc: Sandipan Das <sandipan.das@amd.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/amd/core.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index bf631f60757c..851d8a938f52 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 {
 	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/*
 	 * The sequence of steps to freeze LBR should be completely inlined
 	 * and contain no branches to minimize contamination of LBR snapshot
 	 */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	amd_pmu_core_disable_all();
 	__amd_pmu_lbr_disable();
 
-	cpuc = this_cpu_ptr(&cpu_hw_events);
-
 	amd_pmu_lbr_read();
 	cnt = min(cnt, cpuc->lbr_stack.nr);
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 
-	amd_pmu_v2_enable_all(0);
+	if (lbr_enabled)
+		amd_pmu_lbr_enable_all();
+	if (pmu_enabled)
+		amd_pmu_core_enable_all();
 	local_irq_restore(flags);
 
 	return cnt;
-- 
2.34.1

[Patch v6 07/20] perf/x86/intel: Add helper to enable PMU without LBR

[Dapeng Mi]

Currently, __intel_pmu_enable_all() also re-enables LBR in addition to
the PMU. Split out the PMU-only path into a new helper,
___intel_pmu_enable_all(), so the PMU can be enabled independently. This
helper will be used by a subsequent patch.

Also remove the unused added argument from __intel_pmu_enable_all().

No functional change intended.

Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index e99b311c7b8d..9d8398da1989 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2783,13 +2783,11 @@ static __always_inline void intel_pmu_disable_all(void)
 	intel_pmu_lbr_disable_all();
 }
 
-static void __intel_pmu_enable_all(int added, bool pmi)
+static void ___intel_pmu_enable_all(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
 
-	intel_pmu_lbr_enable_all(pmi);
-
 	if (cpuc->fixed_ctrl_val != cpuc->active_fixed_ctrl_val) {
 		wrmsrq(MSR_ARCH_PERFMON_FIXED_CTR_CTRL, cpuc->fixed_ctrl_val);
 		cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val;
@@ -2809,10 +2807,16 @@ static void __intel_pmu_enable_all(int added, bool pmi)
 	}
 }
 
+static void __intel_pmu_enable_all(bool pmi)
+{
+	intel_pmu_lbr_enable_all(pmi);
+	___intel_pmu_enable_all();
+}
+
 static void intel_pmu_enable_all(int added)
 {
 	static_call_cond(x86_pmu_pebs_enable_all)();
-	__intel_pmu_enable_all(added, false);
+	__intel_pmu_enable_all(false);
 }
 
 static noinline int
@@ -4056,7 +4060,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	/* Only restore PMU state when it's active. See x86_pmu_disable(). */
 	cpuc->enabled = pmu_enabled;
 	if (pmu_enabled)
-		__intel_pmu_enable_all(0, true);
+		__intel_pmu_enable_all(true);
 	intel_bts_enable_local();
 
 	/*
-- 
2.34.1

[Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled

[Dapeng Mi]

__intel_pmu_snapshot_branch_stack() currently re-enables the PMU and LBR
unconditionally even when they were already disabled in PMI context.
This can cause events such as cycles and instructions to be counted
while the PMI handler is running, which is not intended.

Only restore the PMU and LBR on the BPF path if they were originally
enabled.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 38 ++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 9d8398da1989..739c454827ac 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
 
 static noinline int
 __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
-				  unsigned int cnt, unsigned long flags)
+				  unsigned int cnt, unsigned long flags,
+				  int pmu_enabled, int lbr_enabled)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
 
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
-	intel_pmu_enable_all(0);
+
+	/*
+	 * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
+	 * This path may be entered from the PMI handler through
+	 * bpf_overflow_handler().
+	 *
+	 * If PMU version < 4, LBR is disabled by the LBR freeze mechanism,
+	 * LBR can't be re-enabled here.
+	 * If PMU version >= 4, LBR is only frozen instead of disabled,
+	 * LBR should be re-enabled here.
+	 */
+	if (lbr_enabled)
+		intel_pmu_lbr_enable_all(false);
+	if (pmu_enabled)
+		___intel_pmu_enable_all();
+
 	local_irq_restore(flags);
 	return cnt;
 }
@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 static int
 intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }
 
 static int
 intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_arch_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }
 
 /*
-- 
2.34.1

[Patch v6 09/20] perf/x86/intel: Keep LBR frozen throughout PMI handling

[Dapeng Mi]

For the platforms which has 4+ PMU version, streamline LBR freeze is
supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once
a PMU counter overflow. However, the current PMI handler clears the LBR
freeze bit too early, allowing branch operations that occur during PMI
handling to be recorded into the LBR stack.

This can pollute or overwrite the intended branch snapshot, especially
when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler().
In that case, branches recorded while processing the PMI are mixed into
the original LBR data.

Defer unfreezing until after PMI handling completes so the LBR remains
frozen for the entire PMI path and no branch activity from the handler
is recorded.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Fixes: 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 739c454827ac..ab6261ab54b6 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4062,7 +4062,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	loops = 0;
 again:
 	intel_pmu_lbr_read();
-	intel_pmu_ack_status(status);
+
+	/*
+	 * Do not clear the LBR frozen status here; keep LBR frozen
+	 * throughout PMI handling.
+	 */
+	intel_pmu_ack_status(status & ~GLOBAL_STATUS_LBRS_FROZEN);
+
 	if (++loops > 100) {
 		static bool warned;
 
@@ -4081,10 +4087,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	 * Repeat if there is more work to be done:
 	 */
 	status = intel_pmu_get_status();
-	if (status)
+	if (status & ~GLOBAL_STATUS_LBRS_FROZEN)
 		goto again;
 
 done:
+	if (status & GLOBAL_STATUS_LBRS_FROZEN)
+		intel_pmu_ack_status(status & GLOBAL_STATUS_LBRS_FROZEN);
+
 	if (mid_ack)
 		apic_write(APIC_LVTPC, APIC_DM_NMI);
 	/* Only restore PMU state when it's active. See x86_pmu_disable(). */
-- 
2.34.1

[Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack

[Dapeng Mi]

struct perf_branch_stack uses a trailing flexible array member,
entries[], whose valid length is tracked by nr. When this structure
is embedded as a value member in cpu_hw_events, the compiler can treat
the trailing flexible array as having a fixed zero-length layout under
-fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds
diagnostics when entries[] is indexed.

Annotate entries[] with __counted_by(nr) so the compiler and sanitizer
track the array length from the nr field. Because a structure containing
a counted flexible array cannot be embedded as a value member, switch
the per-CPU LBR stack backing storage to a complete __lbr_stack[] array
and add a helper cpu_lbr_stack() to recover the struct perf_branch_stack*
view for x86 and powerpc.

Also initialize the branch-stack count before populating entries[] to
avoid false-positive bounds checks while the stack is being filled.

No functional change intended.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Cc: Sandipan Das <sandipan.das@amd.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: "Christophe Leroy (CS GROUP)" <chleroy@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/powerpc/perf/core-book3s.c | 43 +++++++++++-----
 arch/x86/events/amd/brs.c       |  7 +--
 arch/x86/events/amd/core.c      |  6 +--
 arch/x86/events/amd/lbr.c       | 44 +++++++++-------
 arch/x86/events/core.c          |  2 +-
 arch/x86/events/intel/core.c    |  5 +-
 arch/x86/events/intel/ds.c      | 12 +++--
 arch/x86/events/intel/lbr.c     | 89 +++++++++++++++++----------------
 arch/x86/events/perf_event.h    | 19 ++++++-
 drivers/perf/arm_brbe.c         |  5 ++
 include/linux/perf_event.h      |  2 +-
 11 files changed, 144 insertions(+), 90 deletions(-)

diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index 10ded865583c..978fd39b586b 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -30,6 +30,8 @@
 #define BHRB_PREDICTION		0x0000000000000001
 #define BHRB_EA			0xFFFFFFFFFFFFFFFCUL
 
+#define LBR_STACK_SIZE		struct_size_t(struct perf_branch_stack, entries, BHRB_MAX_ENTRIES)
+
 struct cpu_hw_events {
 	int n_events;
 	int n_percpu;
@@ -54,8 +56,7 @@ struct cpu_hw_events {
 	u64				bhrb_filter;	/* BHRB HW branch filter */
 	unsigned int			bhrb_users;
 	void				*bhrb_context;
-	struct	perf_branch_stack	bhrb_stack;
-	struct	perf_branch_entry	bhrb_entries[BHRB_MAX_ENTRIES];
+	u64				__lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
 	u64				ic_init;
 
 	/* Store the PMC values */
@@ -177,6 +178,20 @@ static bool regs_use_siar(struct pt_regs *regs)
 	return ((TRAP(regs) == INTERRUPT_PERFMON) && regs->result);
 }
 
+static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc)
+{
+	return (struct perf_branch_stack *)cpuc->__lbr_stack;
+}
+
+static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+{
+	/*
+	 * Initialize the branch-stack count before populating entries[]
+	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+	 */
+	cpu_lbr_stack(cpuc)->nr = nr;
+}
+
 /*
  * Things that are specific to 64-bit implementations.
  */
@@ -493,6 +508,8 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
 
 	r_index = 0;
 	u_index = 0;
+
+	cpu_init_lbr_stack(cpuhw, ppmu->bhrb_nr);
 	while (r_index < ppmu->bhrb_nr) {
 		/* Assembly read function */
 		val = read_bhrb(r_index++);
@@ -541,9 +558,9 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
 				/* Target branches use two entries
 				 * (ie. computed gotos/XL form)
 				 */
-				cpuhw->bhrb_entries[u_index].to = addr;
-				cpuhw->bhrb_entries[u_index].mispred = pred;
-				cpuhw->bhrb_entries[u_index].predicted = ~pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].to = addr;
+				cpu_lbr_stack(cpuhw)->entries[u_index].mispred = pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].predicted = ~pred;
 
 				/* Get from address in next entry */
 				val = read_bhrb(r_index++);
@@ -554,22 +571,22 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
 					r_index--;
 					addr = 0;
 				}
-				cpuhw->bhrb_entries[u_index].from = addr;
+				cpu_lbr_stack(cpuhw)->entries[u_index].from = addr;
 			} else {
 				/* Branches to immediate field 
 				   (ie I or B form) */
-				cpuhw->bhrb_entries[u_index].from = addr;
-				cpuhw->bhrb_entries[u_index].to =
+				cpu_lbr_stack(cpuhw)->entries[u_index].from = addr;
+				cpu_lbr_stack(cpuhw)->entries[u_index].to =
 					power_pmu_bhrb_to(addr);
-				cpuhw->bhrb_entries[u_index].mispred = pred;
-				cpuhw->bhrb_entries[u_index].predicted = ~pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].mispred = pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].predicted = ~pred;
 			}
 			u_index++;
 
 		}
 	}
-	cpuhw->bhrb_stack.nr = u_index;
-	cpuhw->bhrb_stack.hw_idx = -1ULL;
+	cpu_lbr_stack(cpuhw)->nr = u_index;
+	cpu_lbr_stack(cpuhw)->hw_idx = -1ULL;
 	return;
 }
 
@@ -2332,7 +2349,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,
 			struct cpu_hw_events *cpuhw;
 			cpuhw = this_cpu_ptr(&cpu_hw_events);
 			power_pmu_bhrb_read(event, cpuhw);
-			perf_sample_save_brstack(&data, event, &cpuhw->bhrb_stack, NULL);
+			perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuhw), NULL);
 		}
 
 		if (event->attr.sample_type & PERF_SAMPLE_DATA_SRC &&
diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c
index 54b13faba116..1906d1257646 100644
--- a/arch/x86/events/amd/brs.c
+++ b/arch/x86/events/amd/brs.c
@@ -282,7 +282,6 @@ void amd_brs_drain(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	struct perf_event *event = cpuc->events[0];
-	struct perf_branch_entry *br = cpuc->lbr_entries;
 	union amd_debug_extn_cfg cfg;
 	u32 i, nr = 0, num, tos, start;
 	u32 shift = 64 - boot_cpu_data.x86_virt_bits;
@@ -317,6 +316,8 @@ void amd_brs_drain(void)
 
 	num = tos - start + 1;
 
+	cpu_init_lbr_stack(cpuc, num);
+
 	/*
 	 * BRS is only one pass (saturation) from MSROFF to depth-1
 	 * MSROFF wraps to zero when buffer is full
@@ -343,7 +344,7 @@ void amd_brs_drain(void)
 		if (!amd_brs_match_plm(event, from, to))
 			continue;
 
-		br[nr] = (struct perf_branch_entry){
+		cpu_lbr_stack(cpuc)->entries[nr] = (struct perf_branch_entry){
 			.from	= from,
 			.to	= to,
 		};
@@ -351,7 +352,7 @@ void amd_brs_drain(void)
 	}
 empty:
 	/* Record number of sampled branches */
-	cpuc->lbr_stack.nr = nr;
+	cpu_lbr_stack(cpuc)->nr = nr;
 }
 
 /*
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 851d8a938f52..160102a46b18 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -947,8 +947,8 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 	__amd_pmu_lbr_disable();
 
 	amd_pmu_lbr_read();
-	cnt = min(cnt, cpuc->lbr_stack.nr);
-	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+	cnt = min(cnt, cpu_lbr_stack(cpuc)->nr);
+	memcpy(entries, cpu_lbr_stack(cpuc)->entries, sizeof(struct perf_branch_entry) * cnt);
 
 	if (lbr_enabled)
 		amd_pmu_lbr_enable_all();
@@ -1020,7 +1020,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
 		if (!x86_perf_event_set_period(event))
 			continue;
 
-		perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+		perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
 
 		perf_event_overflow(event, &data, regs);
 	}
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 400c94ecded0..3e4e5e984649 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -104,15 +104,18 @@ static void amd_pmu_lbr_filter(void)
 	bool compress = false;
 	bool fused_only = false;
 	u64 from, to;
+	int lbr_nr;
 
 	/* If sampling all branches, there is nothing to filter */
 	if (((br_sel & X86_BR_ALL) == X86_BR_ALL) &&
 	    ((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
 		fused_only = true;
 
-	for (i = 0; i < cpuc->lbr_stack.nr; i++) {
-		from = cpuc->lbr_entries[i].from;
-		to = cpuc->lbr_entries[i].to;
+	for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+		struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
+
+		from = be->from;
+		to = be->to;
 		type = branch_type_fused(from, to, 0, &offset);
 
 		/*
@@ -121,37 +124,41 @@ static void amd_pmu_lbr_filter(void)
 		 * actual branch
 		 */
 		if (offset) {
-			cpuc->lbr_entries[i].from += offset;
+			be->from += offset;
 			if (fused_only)
 				continue;
 		}
 
 		/* If type does not correspond, then discard */
 		if (type == X86_BR_NONE || (br_sel & type) != type ||
-		    (!(br_sel & X86_BR_KERNEL) && kernel_ip(cpuc->lbr_entries[i].from))) {
-			cpuc->lbr_entries[i].from = 0;	/* mark invalid */
+		    (!(br_sel & X86_BR_KERNEL) && kernel_ip(be->from))) {
+			be->from = 0;	/* mark invalid */
 			compress = true;
 		}
 
 		if ((br_sel & X86_BR_TYPE_SAVE) == X86_BR_TYPE_SAVE)
-			cpuc->lbr_entries[i].type = common_branch_type(type);
+			be->type = common_branch_type(type);
 	}
 
 	if (!compress)
 		return;
 
 	/* Remove all invalid entries */
-	for (i = 0; i < cpuc->lbr_stack.nr; ) {
-		if (!cpuc->lbr_entries[i].from) {
+	lbr_nr = cpu_lbr_stack(cpuc)->nr;
+	for (i = 0; i < lbr_nr; ) {
+		if (!cpu_lbr_stack(cpuc)->entries[i].from) {
 			j = i;
-			while (++j < cpuc->lbr_stack.nr)
-				cpuc->lbr_entries[j - 1] = cpuc->lbr_entries[j];
-			cpuc->lbr_stack.nr--;
-			if (!cpuc->lbr_entries[i].from)
+			while (++j < lbr_nr) {
+				cpu_lbr_stack(cpuc)->entries[j - 1] =
+					cpu_lbr_stack(cpuc)->entries[j];
+			}
+			lbr_nr--;
+			if (!cpu_lbr_stack(cpuc)->entries[i].from)
 				continue;
 		}
 		i++;
 	}
+	cpu_lbr_stack(cpuc)->nr = lbr_nr;
 }
 
 static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {
@@ -164,15 +171,16 @@ static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {
 void amd_pmu_lbr_read(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
-	struct perf_branch_entry *br = cpuc->lbr_entries;
 	struct branch_entry entry;
 	int out = 0, idx, i;
 
 	if (!cpuc->lbr_users) {
-		cpuc->lbr_stack.nr = 0;
+		cpu_lbr_stack(cpuc)->nr = 0;
 		return;
 	}
 
+	cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		entry.from.full	= amd_pmu_lbr_get_from(i);
 		entry.to.full	= amd_pmu_lbr_get_to(i);
@@ -204,7 +212,7 @@ void amd_pmu_lbr_read(void)
 		 */
 		idx = (entry.to.split.valid << 1) | entry.to.split.spec;
 
-		br[out] = (struct perf_branch_entry){
+		cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
 			.from		= sign_ext_branch_ip(entry.from.split.ip),
 			.to		= sign_ext_branch_ip(entry.to.split.ip),
 			.mispred	= entry.from.split.mispredict,
@@ -214,13 +222,13 @@ void amd_pmu_lbr_read(void)
 		out++;
 	}
 
-	cpuc->lbr_stack.nr = out;
+	cpu_lbr_stack(cpuc)->nr = out;
 
 	/*
 	 * Internal register renaming always ensures that LBR From[0] and
 	 * LBR To[0] always represent the TOS
 	 */
-	cpuc->lbr_stack.hw_idx = 0;
+	cpu_lbr_stack(cpuc)->hw_idx = 0;
 
 	/* Perform further software filtering */
 	amd_pmu_lbr_filter();
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 82d9b0e5ad1c..a7bdc6f370ec 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)
 
 		perf_sample_data_init(&data, 0, last_period);
 
-		perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+		perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
 
 		perf_event_overflow(event, &data, regs);
 	}
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index ab6261ab54b6..42d4d525a729 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2827,9 +2827,10 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
 	intel_pmu_lbr_read();
-	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
+	cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
 
-	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+	memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+	       sizeof(struct perf_branch_entry) * cnt);
 
 	/*
 	 * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 8444670cee4a..332754e1da4f 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2010,8 +2010,7 @@ void intel_pmu_pebs_disable_all(void)
 static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
-	unsigned long from = cpuc->lbr_entries[0].from;
-	unsigned long old_to, to = cpuc->lbr_entries[0].to;
+	unsigned long from, old_to, to;
 	unsigned long ip = regs->ip;
 	int is_64bit = 0;
 	void *kaddr;
@@ -2026,7 +2025,12 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
 	/*
 	 * No LBR entry, no basic block, no rewinding
 	 */
-	if (!cpuc->lbr_stack.nr || !from || !to)
+	if (!cpu_lbr_stack(cpuc)->nr)
+		return 0;
+
+	from = cpu_lbr_stack(cpuc)->entries[0].from;
+	to = cpu_lbr_stack(cpuc)->entries[0].to;
+	if (!from || !to)
 		return 0;
 
 	/*
@@ -2318,7 +2322,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,
 	if (x86_pmu.intel_cap.pebs_format >= 3)
 		setup_pebs_time(event, data, pebs->tsc);
 
-	perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
+	perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
 }
 
 static void adaptive_pebs_save_regs(struct pt_regs *regs,
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index ee724b28d90c..62f2f4b9cfde 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -195,7 +195,7 @@ void intel_pmu_lbr_reset(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
-	cpuc->lbr_stack.nr = 0;
+	cpu_lbr_stack(cpuc)->nr = 0;
 
 	if (!x86_pmu.lbr_nr)
 		return;
@@ -744,10 +744,10 @@ void intel_pmu_lbr_disable_all(void)
 void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
 {
 	unsigned long mask = x86_pmu.lbr_nr - 1;
-	struct perf_branch_entry *br = cpuc->lbr_entries;
 	u64 tos = intel_pmu_lbr_tos();
 	int i;
 
+	cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		unsigned long lbr_idx = (tos - i) & mask;
 		union {
@@ -760,14 +760,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
 
 		rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
 
-		*br = (struct perf_branch_entry){
+		cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
 			.from	= msr_lastbranch.from,
 			.to	= msr_lastbranch.to,
 		};
-		br++;
 	}
-	cpuc->lbr_stack.nr = i;
-	cpuc->lbr_stack.hw_idx = tos;
+	cpu_lbr_stack(cpuc)->nr = i;
+	cpu_lbr_stack(cpuc)->hw_idx = tos;
 }
 
 /*
@@ -779,7 +778,6 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 {
 	bool need_info = false, call_stack = false;
 	unsigned long mask = x86_pmu.lbr_nr - 1;
-	struct perf_branch_entry *br = cpuc->lbr_entries;
 	u64 tos = intel_pmu_lbr_tos();
 	int i;
 	int out = 0;
@@ -791,6 +789,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 			call_stack = true;
 	}
 
+	cpu_init_lbr_stack(cpuc, num);
 	for (i = 0; i < num; i++) {
 		unsigned long lbr_idx = (tos - i) & mask;
 		u64 from, to, mis = 0, pred = 0, in_tx = 0, abort = 0;
@@ -851,7 +850,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		if (abort && x86_pmu.lbr_double_abort && out > 0)
 			out--;
 
-		br[out] = (struct perf_branch_entry){
+		cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
 			.from		= from,
 			.to		= to,
 			.mispred	= mis,
@@ -862,8 +861,8 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		};
 		out++;
 	}
-	cpuc->lbr_stack.nr = out;
-	cpuc->lbr_stack.hw_idx = tos;
+	cpu_lbr_stack(cpuc)->nr = out;
+	cpu_lbr_stack(cpuc)->hw_idx = tos;
 }
 
 static DEFINE_STATIC_KEY_FALSE(x86_lbr_mispred);
@@ -907,15 +906,14 @@ static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LB
 static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 				struct lbr_entry *entries)
 {
-	struct perf_branch_entry *e;
 	struct lbr_entry *lbr;
 	u64 from, to, info;
 	bool mispred;
 	int i;
 
+	cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		lbr = entries ? &entries[i] : NULL;
-		e = &cpuc->lbr_entries[i];
 
 		from = rdlbr_from(i, lbr);
 		/*
@@ -929,7 +927,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 
 		mispred = get_lbr_mispred(info);
 
-		*e = (struct perf_branch_entry){
+		cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
 			.from		= from,
 			.to		= to,
 			.mispred	= mispred,
@@ -939,18 +937,18 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 			.cycles		= get_lbr_cycles(info),
 			.type		= get_lbr_br_type(info),
 			/*
-			 * Leverage the reserved field of
-			 * cpuc->lbr_entries[i] to temporarily store the
-			 * branch counters information. The later code will
-			 * decide what content can be disclosed to the perf
-			 * tool. Pleae see intel_pmu_lbr_counters_reorder().
+			 * Leverage the reserved field of LBR entries to
+			 * temporarily store the branch counters information.
+			 * The later code will decide what content can be
+			 * disclosed to the perf tool.
+			 * Pleae see intel_pmu_lbr_counters_reorder().
 			 */
 			.reserved	= (info >> LBR_INFO_BR_CNTR_OFFSET) &
 					  LBR_INFO_BR_CNTR_FULL_MASK,
 		};
 	}
 
-	cpuc->lbr_stack.nr = i;
+	cpu_lbr_stack(cpuc)->nr = i;
 }
 
 /*
@@ -976,15 +974,15 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,
 
 	WARN_ON_ONCE(!pos);
 
-	for (i = 0; i < cpuc->lbr_stack.nr; i++) {
-		src = cpuc->lbr_entries[i].reserved;
+	for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+		src = cpu_lbr_stack(cpuc)->entries[i].reserved;
 		dst = 0;
 		for (j = 0; j < pos; j++) {
 			cnt = (src >> (order[j] * LBR_INFO_BR_CNTR_BITS)) & LBR_INFO_BR_CNTR_MASK;
 			dst |= cnt << j * LBR_INFO_BR_CNTR_BITS;
 		}
 		cpuc->lbr_counters[i] = dst;
-		cpuc->lbr_entries[i].reserved = 0;
+		cpu_lbr_stack(cpuc)->entries[i].reserved = 0;
 	}
 }
 
@@ -994,11 +992,11 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 {
 	if (is_branch_counters_group(event)) {
 		intel_pmu_lbr_counters_reorder(cpuc, event);
-		perf_sample_save_brstack(data, event, &cpuc->lbr_stack, cpuc->lbr_counters);
+		perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 		return;
 	}
 
-	perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
+	perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
 }
 
 static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc)
@@ -1021,7 +1019,7 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
 		intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
 	} else {
 		/* Clear the LBR stack to avoid leaking stale entries. */
-		cpuc->lbr_stack.nr = 0;
+		cpu_lbr_stack(cpuc)->nr = 0;
 	}
 }
 
@@ -1031,7 +1029,7 @@ void intel_pmu_lbr_read(void)
 
 	if (!cpuc->lbr_users || vlbr_exclude_host()) {
 		/* Reset lbr_stack.nr to prevent stale data leakage. */
-		cpuc->lbr_stack.nr = 0;
+		cpu_lbr_stack(cpuc)->nr = 0;
 		return;
 	}
 
@@ -1231,21 +1229,23 @@ static const int arch_lbr_br_type_map[ARCH_LBR_BR_TYPE_MAP_MAX] = {
 static void
 intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
 {
-	u64 from, to;
 	int br_sel = cpuc->br_sel;
 	int i, j, type, from_plm, to_plm;
 	bool compress = false;
+	u64 from, to;
+	int lbr_nr;
 
 	/* if sampling all branches, then nothing to filter */
 	if (((br_sel & X86_BR_ALL) == X86_BR_ALL) &&
 	    ((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
 		return;
 
-	for (i = 0; i < cpuc->lbr_stack.nr; i++) {
+	for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+		struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
 
-		from = cpuc->lbr_entries[i].from;
-		to = cpuc->lbr_entries[i].to;
-		type = cpuc->lbr_entries[i].type;
+		from = be->from;
+		to = be->to;
+		type = be->type;
 
 		/*
 		 * Parse the branch type recorded in LBR_x_INFO MSR.
@@ -1257,9 +1257,9 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
 			to_plm = kernel_ip(to) ? X86_BR_KERNEL : X86_BR_USER;
 			type = arch_lbr_br_type_map[type] | to_plm;
 		} else
-			type = branch_type(from, to, cpuc->lbr_entries[i].abort);
+			type = branch_type(from, to, be->abort);
 		if (type != X86_BR_NONE && (br_sel & X86_BR_ANYTX)) {
-			if (cpuc->lbr_entries[i].in_tx)
+			if (be->in_tx)
 				type |= X86_BR_IN_TX;
 			else
 				type |= X86_BR_NO_TX;
@@ -1273,31 +1273,34 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
 		 */
 		if (type == X86_BR_NONE || (br_sel & type) != type ||
 		    (!(br_sel & X86_BR_KERNEL) && (from_plm & X86_BR_KERNEL))) {
-			cpuc->lbr_entries[i].from = 0;
+			be->from = 0;
 			compress = true;
 		}
 
 		if ((br_sel & X86_BR_TYPE_SAVE) == X86_BR_TYPE_SAVE)
-			cpuc->lbr_entries[i].type = common_branch_type(type);
+			be->type = common_branch_type(type);
 	}
 
 	if (!compress)
 		return;
 
 	/* remove all entries with from=0 */
-	for (i = 0; i < cpuc->lbr_stack.nr; ) {
-		if (!cpuc->lbr_entries[i].from) {
+	lbr_nr = cpu_lbr_stack(cpuc)->nr;
+	for (i = 0; i < lbr_nr; ) {
+		if (!cpu_lbr_stack(cpuc)->entries[i].from) {
 			j = i;
-			while (++j < cpuc->lbr_stack.nr) {
-				cpuc->lbr_entries[j-1] = cpuc->lbr_entries[j];
+			while (++j < lbr_nr) {
+				cpu_lbr_stack(cpuc)->entries[j-1] =
+					cpu_lbr_stack(cpuc)->entries[j];
 				cpuc->lbr_counters[j-1] = cpuc->lbr_counters[j];
 			}
-			cpuc->lbr_stack.nr--;
-			if (!cpuc->lbr_entries[i].from)
+			lbr_nr--;
+			if (!cpu_lbr_stack(cpuc)->entries[i].from)
 				continue;
 		}
 		i++;
 	}
+	cpu_lbr_stack(cpuc)->nr = lbr_nr;
 }
 
 void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
@@ -1307,9 +1310,9 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
 	/* Cannot get TOS for large PEBS and Arch LBR */
 	if (cpu_feature_enabled(X86_FEATURE_ARCH_LBR) ||
 	    (cpuc->n_pebs == cpuc->n_large_pebs))
-		cpuc->lbr_stack.hw_idx = -1ULL;
+		cpu_lbr_stack(cpuc)->hw_idx = -1ULL;
 	else
-		cpuc->lbr_stack.hw_idx = intel_pmu_lbr_tos();
+		cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
 
 	intel_pmu_store_lbr(cpuc, lbr);
 	intel_pmu_lbr_filter(cpuc);
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index a4aab8130e92..e3d625a6d7eb 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -447,6 +447,8 @@ struct intel_excl_cntrs {
 struct x86_perf_task_context;
 #define MAX_LBR_ENTRIES		32
 
+#define LBR_STACK_SIZE		struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES)
+
 enum {
 	LBR_FORMAT_32		= 0x00,
 	LBR_FORMAT_LIP		= 0x01,
@@ -525,8 +527,7 @@ struct cpu_hw_events {
 	 */
 	int				lbr_users;
 	int				lbr_pebs_users;
-	struct perf_branch_stack	lbr_stack;
-	struct perf_branch_entry	lbr_entries[MAX_LBR_ENTRIES];
+	u64				__lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
 	u64				lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
 	union {
 		struct er_account		*lbr_sel;
@@ -588,6 +589,20 @@ struct cpu_hw_events {
 	struct pmu			*pmu;
 };
 
+static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc)
+{
+	return (struct perf_branch_stack *)cpuc->__lbr_stack;
+}
+
+static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+{
+	/*
+	 * Initialize the branch-stack count before populating entries[]
+	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+	 */
+	cpu_lbr_stack(cpuc)->nr = nr;
+}
+
 #define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) {	\
 	{ .idxmsk64 = (n) },		\
 	.code = (c),			\
diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c
index 073d14584741..e2b32b239de1 100644
--- a/drivers/perf/arm_brbe.c
+++ b/drivers/perf/arm_brbe.c
@@ -763,6 +763,11 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 
 	prepare_event_branch_type_mask(branch_sample_type, event_type_mask);
 
+	/*
+	 * Initialize the branch_stack->nr before populating entries[]
+	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+	 */
+	branch_stack->nr = nr_hw;
 	for (int bank = 0; bank < nr_banks; bank++) {
 		int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
 		int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index eb3f7bab498c..19de3f1cbd6e 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)
 struct perf_branch_stack {
 	u64				nr;
 	u64				hw_idx;
-	struct perf_branch_entry	entries[];
+	struct perf_branch_entry	entries[] __counted_by(nr);
 };
 
 struct task_struct;
-- 
2.34.1

[Patch v6 11/20] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks

[Dapeng Mi]

The LBR read helpers currently assume the default stack,
cpuc->lbr_stack, and therefore cannot be reused for other LBR
stacks that may be in use. Extend the *_lbr_read() helpers to take
an explicit LBR stack argument, so they can operate on any LBR stack
instead of only the default one.

Update the related callers, including intel_pmu_store_lbr() and
intel_pmu_lbr_filter(), to pass the appropriate stack explicitly.

No function change intended.

Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/amd/brs.c    |  2 +-
 arch/x86/events/amd/lbr.c    |  2 +-
 arch/x86/events/intel/core.c |  4 +-
 arch/x86/events/intel/lbr.c  | 89 ++++++++++++++++++++----------------
 arch/x86/events/perf_event.h | 16 ++++---
 5 files changed, 64 insertions(+), 49 deletions(-)

diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c
index 1906d1257646..475e3699d3ca 100644
--- a/arch/x86/events/amd/brs.c
+++ b/arch/x86/events/amd/brs.c
@@ -316,7 +316,7 @@ void amd_brs_drain(void)
 
 	num = tos - start + 1;
 
-	cpu_init_lbr_stack(cpuc, num);
+	cpu_init_lbr_stack(cpu_lbr_stack(cpuc), num);
 
 	/*
 	 * BRS is only one pass (saturation) from MSROFF to depth-1
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 3e4e5e984649..95898a1d9901 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -179,7 +179,7 @@ void amd_pmu_lbr_read(void)
 		return;
 	}
 
-	cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+	cpu_init_lbr_stack(cpu_lbr_stack(cpuc), x86_pmu.lbr_nr);
 
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		entry.from.full	= amd_pmu_lbr_get_from(i);
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 42d4d525a729..b7045c05b736 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2826,7 +2826,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
-	intel_pmu_lbr_read();
+	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 	cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
 
 	memcpy(entries, cpu_lbr_stack(cpuc)->entries,
@@ -4062,7 +4062,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
-	intel_pmu_lbr_read();
+	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 
 	/*
 	 * Do not clear the LBR frozen status here; keep LBR frozen
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 62f2f4b9cfde..979b094ae228 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -105,7 +105,9 @@
 
 #define ARCH_LBR_CTL_MASK			0x7f000e
 
-static void intel_pmu_lbr_filter(struct cpu_hw_events *cpuc);
+static void intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
+				 struct perf_branch_stack *lbr_stack,
+				 u64 *lbr_counters);
 
 static __always_inline bool is_lbr_call_stack_bit_set(u64 config)
 {
@@ -741,13 +743,14 @@ void intel_pmu_lbr_disable_all(void)
 	}
 }
 
-void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
+void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc,
+			   struct perf_branch_stack *lbr_stack)
 {
 	unsigned long mask = x86_pmu.lbr_nr - 1;
 	u64 tos = intel_pmu_lbr_tos();
 	int i;
 
-	cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+	cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		unsigned long lbr_idx = (tos - i) & mask;
 		union {
@@ -760,13 +763,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
 
 		rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
 
-		cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
+		lbr_stack->entries[i] = (struct perf_branch_entry){
 			.from	= msr_lastbranch.from,
 			.to	= msr_lastbranch.to,
 		};
 	}
-	cpu_lbr_stack(cpuc)->nr = i;
-	cpu_lbr_stack(cpuc)->hw_idx = tos;
+	lbr_stack->nr = i;
+	lbr_stack->hw_idx = tos;
 }
 
 /*
@@ -774,7 +777,8 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
  * is the same as the linear address, allowing us to merge the LIP and EIP
  * LBR formats.
  */
-void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
+void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
+			   struct perf_branch_stack *lbr_stack)
 {
 	bool need_info = false, call_stack = false;
 	unsigned long mask = x86_pmu.lbr_nr - 1;
@@ -789,7 +793,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 			call_stack = true;
 	}
 
-	cpu_init_lbr_stack(cpuc, num);
+	cpu_init_lbr_stack(lbr_stack, num);
 	for (i = 0; i < num; i++) {
 		unsigned long lbr_idx = (tos - i) & mask;
 		u64 from, to, mis = 0, pred = 0, in_tx = 0, abort = 0;
@@ -850,7 +854,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		if (abort && x86_pmu.lbr_double_abort && out > 0)
 			out--;
 
-		cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
+		lbr_stack->entries[out] = (struct perf_branch_entry){
 			.from		= from,
 			.to		= to,
 			.mispred	= mis,
@@ -861,8 +865,8 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		};
 		out++;
 	}
-	cpu_lbr_stack(cpuc)->nr = out;
-	cpu_lbr_stack(cpuc)->hw_idx = tos;
+	lbr_stack->nr = out;
+	lbr_stack->hw_idx = tos;
 }
 
 static DEFINE_STATIC_KEY_FALSE(x86_lbr_mispred);
@@ -903,17 +907,19 @@ static __always_inline u16 get_lbr_cycles(u64 info)
 
 static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBR_INFO_BR_CNTR_BITS);
 
-static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
-				struct lbr_entry *entries)
+static void intel_pmu_store_lbr(struct lbr_entry *entries,
+				struct perf_branch_stack *lbr_stack)
 {
+	struct perf_branch_entry *e;
 	struct lbr_entry *lbr;
 	u64 from, to, info;
 	bool mispred;
 	int i;
 
-	cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+	cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		lbr = entries ? &entries[i] : NULL;
+		e = &lbr_stack->entries[i];
 
 		from = rdlbr_from(i, lbr);
 		/*
@@ -927,7 +933,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 
 		mispred = get_lbr_mispred(info);
 
-		cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
+		*e = (struct perf_branch_entry){
 			.from		= from,
 			.to		= to,
 			.mispred	= mispred,
@@ -948,7 +954,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 		};
 	}
 
-	cpu_lbr_stack(cpuc)->nr = i;
+	lbr_stack->nr = i;
 }
 
 /*
@@ -999,37 +1005,41 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 	perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
 }
 
-static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc)
+static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,
+				    struct perf_branch_stack *lbr_stack)
 {
-	intel_pmu_store_lbr(cpuc, NULL);
+	intel_pmu_store_lbr(NULL, lbr_stack);
 }
 
-static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
+static void
+intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,
+			      struct perf_branch_stack *lbr_stack)
 {
 	struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;
 
 	if (!xsave) {
-		intel_pmu_store_lbr(cpuc, NULL);
+		intel_pmu_store_lbr(NULL, lbr_stack);
 		return;
 	}
 	xsave->header.xfeatures = 0;
 	xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
 
 	if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
-		intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+		intel_pmu_store_lbr(xsave->lbr.entries, lbr_stack);
 	} else {
 		/* Clear the LBR stack to avoid leaking stale entries. */
-		cpu_lbr_stack(cpuc)->nr = 0;
+		lbr_stack->nr = 0;
 	}
 }
 
-void intel_pmu_lbr_read(void)
+void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
+			u64 *lbr_counters)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
 	if (!cpuc->lbr_users || vlbr_exclude_host()) {
-		/* Reset lbr_stack.nr to prevent stale data leakage. */
-		cpu_lbr_stack(cpuc)->nr = 0;
+		/* Reset lbr_stack->nr to prevent stale data leakage. */
+		lbr_stack->nr = 0;
 		return;
 	}
 
@@ -1042,9 +1052,9 @@ void intel_pmu_lbr_read(void)
 	if (cpuc->lbr_users == cpuc->lbr_pebs_users)
 		return;
 
-	x86_pmu.lbr_read(cpuc);
+	x86_pmu.lbr_read(cpuc, lbr_stack);
 
-	intel_pmu_lbr_filter(cpuc);
+	intel_pmu_lbr_filter(cpuc, lbr_stack, lbr_counters);
 }
 
 /*
@@ -1227,7 +1237,9 @@ static const int arch_lbr_br_type_map[ARCH_LBR_BR_TYPE_MAP_MAX] = {
  * in PERF_SAMPLE_BRANCH_STACK sample may vary.
  */
 static void
-intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
+intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
+		     struct perf_branch_stack *lbr_stack,
+		     u64 *lbr_counters)
 {
 	int br_sel = cpuc->br_sel;
 	int i, j, type, from_plm, to_plm;
@@ -1240,8 +1252,8 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
 	    ((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
 		return;
 
-	for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
-		struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
+	for (i = 0; i < lbr_stack->nr; i++) {
+		struct perf_branch_entry *be = &lbr_stack->entries[i];
 
 		from = be->from;
 		to = be->to;
@@ -1285,22 +1297,21 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
 		return;
 
 	/* remove all entries with from=0 */
-	lbr_nr = cpu_lbr_stack(cpuc)->nr;
+	lbr_nr = lbr_stack->nr;
 	for (i = 0; i < lbr_nr; ) {
-		if (!cpu_lbr_stack(cpuc)->entries[i].from) {
+		if (!lbr_stack->entries[i].from) {
 			j = i;
 			while (++j < lbr_nr) {
-				cpu_lbr_stack(cpuc)->entries[j-1] =
-					cpu_lbr_stack(cpuc)->entries[j];
-				cpuc->lbr_counters[j-1] = cpuc->lbr_counters[j];
+				lbr_stack->entries[j-1] = lbr_stack->entries[j];
+				lbr_counters[j-1] = lbr_counters[j];
 			}
 			lbr_nr--;
-			if (!cpu_lbr_stack(cpuc)->entries[i].from)
+			if (!lbr_stack->entries[i].from)
 				continue;
 		}
 		i++;
 	}
-	cpu_lbr_stack(cpuc)->nr = lbr_nr;
+	lbr_stack->nr = lbr_nr;
 }
 
 void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
@@ -1314,8 +1325,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
 	else
 		cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
 
-	intel_pmu_store_lbr(cpuc, lbr);
-	intel_pmu_lbr_filter(cpuc);
+	intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
+	intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 }
 
 /*
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index e3d625a6d7eb..fa0af528cd1e 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -594,13 +594,13 @@ static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_eve
 	return (struct perf_branch_stack *)cpuc->__lbr_stack;
 }
 
-static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+static __always_inline void cpu_init_lbr_stack(struct perf_branch_stack *lbr_stack, int nr)
 {
 	/*
 	 * Initialize the branch-stack count before populating entries[]
 	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
 	 */
-	cpu_lbr_stack(cpuc)->nr = nr;
+	lbr_stack->nr = nr;
 }
 
 #define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) {	\
@@ -1207,7 +1207,8 @@ struct x86_pmu {
 	unsigned int	lbr_counters:4;
 
 	void		(*lbr_reset)(void);
-	void		(*lbr_read)(struct cpu_hw_events *cpuc);
+	void		(*lbr_read)(struct cpu_hw_events *cpuc,
+				    struct perf_branch_stack *lbr_stack);
 	void		(*lbr_save)(void *ctx);
 	void		(*lbr_restore)(void *ctx);
 
@@ -2007,11 +2008,14 @@ void intel_pmu_lbr_enable_all(bool pmi);
 
 void intel_pmu_lbr_disable_all(void);
 
-void intel_pmu_lbr_read(void);
+void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
+			u64 *lbr_counters);
 
-void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc,
+			   struct perf_branch_stack *lbr_stack);
 
-void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
+			   struct perf_branch_stack *lbr_stack);
 
 void intel_pmu_lbr_save(void *ctx);
 
-- 
2.34.1

[Patch v6 12/20] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting

[Dapeng Mi]

BPF LBR snapshotting currently reuses the default LBR stack
(cpuc->__lbr_stack) when storing sampled data. This is unsafe when the
BPF helper runs in PMI context via bpf_overflow_handler(): it can
overwrite the cached LBR state for active PEBS LBR events, causing
incorrect LBR data to be reported to userspace. When the PEBS and BPF
stacks have different depths, this can also lead to out-of-bounds reads
when the LBR stack is later exported.

Use a dedicated LBR stack for BPF snapshotting so the BPF path no
longer clobbers the shared LBR state used by other perf events.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b7045c05b736..a3aaa98108cd 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2819,17 +2819,25 @@ static void intel_pmu_enable_all(int added)
 	__intel_pmu_enable_all(false);
 }
 
+struct lbr_snapshot {
+	u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
+	u64 counters[MAX_LBR_ENTRIES];
+};
+static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot);
+
 static noinline int
 __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 				  unsigned int cnt, unsigned long flags,
 				  int pmu_enabled, int lbr_enabled)
 {
-	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+	struct lbr_snapshot *bpf = this_cpu_ptr(&lbr_snapshot);
+	struct perf_branch_stack *lbr_stack =
+		(struct perf_branch_stack *)bpf->__lbr_stack;
 
-	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
-	cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
+	intel_pmu_lbr_read(lbr_stack, bpf->counters);
+	cnt = min_t(unsigned int, cnt, lbr_stack->nr);
 
-	memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+	memcpy(entries, lbr_stack->entries,
 	       sizeof(struct perf_branch_entry) * cnt);
 
 	/*
-- 
2.34.1

[Patch v6 13/20] perf/x86/intel: Always snapshot live hardware LBR state for BPF

[Dapeng Mi]

__intel_pmu_snapshot_branch_stack() is intended to capture the current
live hardware LBR state for the BPF path. However, that snapshot is
currently skipped when all LBR users are PEBS events.

BPF LBR snapshotting now uses its own dedicated LBR stack, so there is
no longer a risk of clobbering the shared default LBR stack. As a
result, always snapshot the live hardware LBR state for the BPF path.

Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 7 ++++++-
 arch/x86/events/intel/lbr.c  | 9 ---------
 2 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index a3aaa98108cd..fbde874af618 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4070,7 +4070,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
-	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+	/*
+	 * No need to read the LBR stack when all LBR users are PEBS events,
+	 * since the PEBS handler updates it.
+	 */
+	if (cpuc->lbr_users != cpuc->lbr_pebs_users)
+		intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 
 	/*
 	 * Do not clear the LBR frozen status here; keep LBR frozen
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 979b094ae228..27e4be839926 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -1043,15 +1043,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
 		return;
 	}
 
-	/*
-	 * Don't read when all LBRs users are using adaptive PEBS.
-	 *
-	 * This could be smarter and actually check the event,
-	 * but this simple approach seems to work for now.
-	 */
-	if (cpuc->lbr_users == cpuc->lbr_pebs_users)
-		return;
-
 	x86_pmu.lbr_read(cpuc, lbr_stack);
 
 	intel_pmu_lbr_filter(cpuc, lbr_stack, lbr_counters);
-- 
2.34.1

[Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active

[Dapeng Mi]

Refreshing the LBR stack in the PMI handler is unnecessary when no LBR
users are active. This adds avoidable overhead and hurts performance.

Limit the LBR stack refresh to the PMI path only when LBR users are
present.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Cc: "Sandipan Das <sandipan.das@amd.com>"
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/amd/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 160102a46b18..f0425e369837 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -988,7 +988,8 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
 
 	/* Read branch records */
 	if (x86_pmu.lbr_nr) {
-		amd_pmu_lbr_read();
+		if (cpuc->lbr_users > 0)
+			amd_pmu_lbr_read();
 		status &= ~GLOBAL_STATUS_LBRS_FROZEN;
 	}
 
-- 
2.34.1

[Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed

[Dapeng Mi]

PEBS and non-PEBS events share the same LBR stack in cpuc->__lbr_stack.
In the PMI handler, the non-PEBS path refreshes the LBR state before the
PEBS buffer is drained.

When PEBS and non-PEBS LBR events overflow in the same PMI, draining the
PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot.
This leaves stale LBR data from the PEBS record in place, so userspace
can see the wrong branch stack instead of the live LBR state at PMI
entry.

Track whether the LBR state has already been refreshed with a
lbr_refreshed flag. If PEBS events overwrite the shared LBR stack before
the non-PEBS LBR event is processed, clear the flag so the live LBR state
is refreshed again. Remove the unconditional LBR refresh at PMI entry and
initialize the flag as false so the stack is refreshed only when an active
non-PEBS LBR event requires it. Since the LBR state is frozen while the
PMI is being handled, it only needs to be read once for multiple non-PEBS
LBR events in the same interrupt.

Fixes: c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 40 +++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 10 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index fbde874af618..b267c68f9467 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3864,8 +3864,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 {
 	struct perf_sample_data data;
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
-	int bit;
+	bool lbr_refreshed = false;
 	int handled = 0;
+	int bit;
 
 	inc_perf_irq_stat();
 
@@ -3906,6 +3907,15 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 	if (__test_and_clear_bit(GLOBAL_STATUS_BUFFER_OVF_BIT, (unsigned long *)&status)) {
 		u64 pebs_enabled = cpuc->pebs_enabled;
 
+		/*
+		 * Prior to adaptive PEBS, PEBS does not support LBR snapshotting.
+		 * Refresh the LBR stack explicitly when LBR users are present.
+		 */
+		if (cpuc->lbr_users > 0 && !x86_pmu.intel_cap.pebs_baseline) {
+			intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+			lbr_refreshed = true;
+		}
+
 		handled += x86_pmu_handle_guest_pebs(regs, &data);
 		handled += static_call(x86_pmu_drain_pebs)(regs, &data);
 		/* Ensure no "suspicious NMI" warning for empty PEBS buffer. */
@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 		 *
 		 * The PEBS buffer has to be drained before handling the A-PMI
 		 */
-		if (is_pebs_counter_event_group(event))
+		if (is_pebs_counter_event_group(event)) {
+			/*
+			 * PEBS counter sampling is only supported on adaptive PEBS
+			 * and arch-PEBS, both of which snapshot the LBR stack.
+			 * No explicit LBR refresh is needed here.
+			 */
 			static_call(x86_pmu_drain_pebs)(regs, &data);
+			/*
+			 * Draining the PEBS buffer can overwrite the shared
+			 * LBR stack, so mark lbr_refreshed as false.
+			 */
+			lbr_refreshed = false;
+		}
 
 		last_period = event->hw.last_period;
 
@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 
 		perf_sample_data_init(&data, 0, last_period);
 
-		if (has_branch_stack(event))
+		if (has_branch_stack(event)) {
+			if (!lbr_refreshed) {
+				intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
+						   cpuc->lbr_counters);
+				lbr_refreshed = true;
+			}
 			intel_pmu_lbr_save_brstack(&data, cpuc, event);
+		}
 
 		x86_pmu_clear_perf_regs(regs);
 		x86_pmu_update_perf_regs(event, &data, regs, false);
@@ -4070,13 +4097,6 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
-	/*
-	 * No need to read the LBR stack when all LBR users are PEBS events,
-	 * since the PEBS handler updates it.
-	 */
-	if (cpuc->lbr_users != cpuc->lbr_pebs_users)
-		intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
-
 	/*
 	 * Do not clear the LBR frozen status here; keep LBR frozen
 	 * throughout PMI handling.
-- 
2.34.1

[Patch v6 16/20] perf/x86/intel: Remove dead lbr_pebs_users

[Dapeng Mi]

The lbr_pebs_users accounting is no longer used, so remove the dead
field and the related code paths that maintain it.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/lbr.c  | 5 -----
 arch/x86/events/perf_event.h | 1 -
 2 files changed, 6 deletions(-)

diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 27e4be839926..3500cf3518f4 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -617,8 +617,6 @@ void intel_pmu_lbr_add(struct perf_event *event)
 	 * be 'new'. Conversely, a new event can get installed through the
 	 * context switch path for the first time.
 	 */
-	if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
-		cpuc->lbr_pebs_users++;
 	perf_sched_cb_inc(event->pmu);
 	if (!cpuc->lbr_users++ && !event->total_time_running)
 		intel_pmu_lbr_reset();
@@ -688,11 +686,8 @@ void intel_pmu_lbr_del(struct perf_event *event)
 	if (event->hw.flags & PERF_X86_EVENT_LBR_SELECT)
 		cpuc->lbr_select = 0;
 
-	if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
-		cpuc->lbr_pebs_users--;
 	cpuc->lbr_users--;
 	WARN_ON_ONCE(cpuc->lbr_users < 0);
-	WARN_ON_ONCE(cpuc->lbr_pebs_users < 0);
 	perf_sched_cb_dec(event->pmu);
 
 	/*
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index fa0af528cd1e..0213e2a8efff 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -526,7 +526,6 @@ struct cpu_hw_events {
 	 * Intel LBR bits
 	 */
 	int				lbr_users;
-	int				lbr_pebs_users;
 	u64				__lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
 	u64				lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
 	union {
-- 
2.34.1

[Patch v6 17/20] perf/x86/intel: Clear raw LBR counters in the LBR stack

[Dapeng Mi]

Raw LBR counters are currently stored in the reserved fields of the LBR
stack even when no LBR event needs them.

If an LBR counter event is present, it clears the raw LBR counters while
processing the stack, but those counters can be leaked to userspace when
no LBR event requires them. Clear the raw LBR counters when no LBR event
uses them.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c |  1 +
 arch/x86/events/intel/lbr.c  | 13 +++++++++++++
 arch/x86/events/perf_event.h |  2 ++
 3 files changed, 16 insertions(+)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b267c68f9467..707bff285ed7 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2835,6 +2835,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 		(struct perf_branch_stack *)bpf->__lbr_stack;
 
 	intel_pmu_lbr_read(lbr_stack, bpf->counters);
+	intel_pmu_lbr_clear_counters(lbr_stack);
 	cnt = min_t(unsigned int, cnt, lbr_stack->nr);
 
 	memcpy(entries, lbr_stack->entries,
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 3500cf3518f4..fbc053ebbeaf 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -987,6 +987,18 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,
 	}
 }
 
+void intel_pmu_lbr_clear_counters(struct perf_branch_stack *lbr_stack)
+{
+	int i;
+
+	if (!(x86_pmu.flags & PMU_FL_BR_CNTR))
+		return;
+
+	/* reserved field stores raw branch counters, clear it. */
+	for (i = 0; i < lbr_stack->nr; i++)
+		lbr_stack->entries[i].reserved = 0;
+}
+
 void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 				struct cpu_hw_events *cpuc,
 				struct perf_event *event)
@@ -997,6 +1009,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 		return;
 	}
 
+	intel_pmu_lbr_clear_counters(cpu_lbr_stack(cpuc));
 	perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
 }
 
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 0213e2a8efff..a4e53c99eaf5 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1984,6 +1984,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
 
 void intel_pebs_init(void);
 
+void intel_pmu_lbr_clear_counters(struct perf_branch_stack *lbr_stack);
+
 void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 				struct cpu_hw_events *cpuc,
 				struct perf_event *event);
-- 
2.34.1

[Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler

[Dapeng Mi]

When handling an LBR counter event, the raw LBR counters in the reserved
field of each LBR entry are cleared after the stack is processed and
reported to userspace. This prevents raw counter data from leaking.

However, if multiple events overflow in the same PMI, an LBR counter
event may not be the first one processed. In that case, the raw LBR
counters may already have been cleared by the previous LBR event, so the
later LBR counter event cannot read the correct values.

Refill the LBR stack before processing an LBR counter event when multiple
events are handled in a single PMI. This restores the correct raw LBR
counter data for the event in this corner case.

This situation is rare, the performance impact from extra refill should
be negligible.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 707bff285ed7..f4d4613fb9f7 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3866,6 +3866,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 	struct perf_sample_data data;
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	bool lbr_refreshed = false;
+	bool lbr_counter_cleared = false;
 	int handled = 0;
 	int bit;
 
@@ -4036,12 +4037,20 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 		perf_sample_data_init(&data, 0, last_period);
 
 		if (has_branch_stack(event)) {
-			if (!lbr_refreshed) {
+			/*
+			 * The raw LBR counters stored in the reserved field of
+			 * the LBR entries were cleared during processing of the
+			 * previous LBR event, so refresh the LBR stack again if
+			 * this event needs the LBR counters.
+			 */
+			if (!lbr_refreshed ||
+			    (lbr_counter_cleared && is_branch_counters_group(event))) {
 				intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
 						   cpuc->lbr_counters);
 				lbr_refreshed = true;
 			}
 			intel_pmu_lbr_save_brstack(&data, cpuc, event);
+			lbr_counter_cleared = true;
 		}
 
 		x86_pmu_clear_perf_regs(regs);
-- 
2.34.1

[Patch v6 19/20] perf/x86/intel: Pass LBR stack size into store_lbr()

[Dapeng Mi]

intel_pmu_store_lbr() assumes the LBR stack always contains
x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth.
This can result in out-of-bounds reads when the stack is shallower than
the hardware maximum.

Add a lbr_nr parameter to the helper so the code only processes the
available entries and avoids overrunning the LBR buffer.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/ds.c   |  8 ++++----
 arch/x86/events/intel/lbr.c  | 18 ++++++++++--------
 arch/x86/events/perf_event.h |  2 +-
 3 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 332754e1da4f..95a2166f5a47 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2595,12 +2595,12 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,
 
 	if (format_group & PEBS_DATACFG_LBRS) {
 		struct lbr_entry *lbr = next_record;
-		int num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT)
+		unsigned int num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT)
 					& 0xff) + 1;
 		next_record = next_record + num_lbr * sizeof(struct lbr_entry);
 
 		if (has_branch_stack(event)) {
-			intel_pmu_store_pebs_lbrs(lbr);
+			intel_pmu_store_pebs_lbrs(lbr, num_lbr);
 			intel_pmu_lbr_save_brstack(data, cpuc, event);
 		}
 	}
@@ -2781,7 +2781,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
 	if (header->lbr) {
 		struct arch_pebs_lbr_header *lbr_header = next_record;
 		struct lbr_entry *lbr;
-		int num_lbr;
+		unsigned int num_lbr;
 
 		next_record = lbr_header + 1;
 		lbr = next_record;
@@ -2792,7 +2792,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
 		next_record += num_lbr * sizeof(struct lbr_entry);
 
 		if (has_branch_stack(event)) {
-			intel_pmu_store_pebs_lbrs(lbr);
+			intel_pmu_store_pebs_lbrs(lbr, num_lbr);
 			intel_pmu_lbr_save_brstack(data, cpuc, event);
 		}
 	}
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index fbc053ebbeaf..ce6aa35e20fb 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -902,7 +902,7 @@ static __always_inline u16 get_lbr_cycles(u64 info)
 
 static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBR_INFO_BR_CNTR_BITS);
 
-static void intel_pmu_store_lbr(struct lbr_entry *entries,
+static void intel_pmu_store_lbr(struct lbr_entry *entries, int lbr_nr,
 				struct perf_branch_stack *lbr_stack)
 {
 	struct perf_branch_entry *e;
@@ -911,8 +911,8 @@ static void intel_pmu_store_lbr(struct lbr_entry *entries,
 	bool mispred;
 	int i;
 
-	cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
-	for (i = 0; i < x86_pmu.lbr_nr; i++) {
+	cpu_init_lbr_stack(lbr_stack, lbr_nr);
+	for (i = 0; i < lbr_nr; i++) {
 		lbr = entries ? &entries[i] : NULL;
 		e = &lbr_stack->entries[i];
 
@@ -1016,7 +1016,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,
 				    struct perf_branch_stack *lbr_stack)
 {
-	intel_pmu_store_lbr(NULL, lbr_stack);
+	intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
 }
 
 static void
@@ -1026,14 +1026,15 @@ intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,
 	struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;
 
 	if (!xsave) {
-		intel_pmu_store_lbr(NULL, lbr_stack);
+		intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
 		return;
 	}
 	xsave->header.xfeatures = 0;
 	xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
 
 	if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
-		intel_pmu_store_lbr(xsave->lbr.entries, lbr_stack);
+		intel_pmu_store_lbr(xsave->lbr.entries,
+				    x86_pmu.lbr_nr, lbr_stack);
 	} else {
 		/* Clear the LBR stack to avoid leaking stale entries. */
 		lbr_stack->nr = 0;
@@ -1313,7 +1314,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
 	lbr_stack->nr = lbr_nr;
 }
 
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
@@ -1324,7 +1325,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
 	else
 		cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
 
-	intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
+	intel_pmu_store_lbr(lbr, min(lbr_nr, x86_pmu.lbr_nr),
+			    cpu_lbr_stack(cpuc));
 	intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 }
 
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index a4e53c99eaf5..1b3fa997d89d 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1980,7 +1980,7 @@ void intel_pmu_pebs_late_setup(struct cpu_hw_events *cpuc);
 
 void intel_pmu_drain_pebs_buffer(void);
 
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr);
 
 void intel_pebs_init(void);
 
-- 
2.34.1

[Patch v6 20/20] perf/core: Don't overwrite shared LBR stack depth

[Dapeng Mi]

perf_sample_save_brstack() currently clamps the shared LBR stack depth
to min(event->attr.sample_max_stack, brs->nr). This is unsafe because
the global LBR stack is shared across all LBR events, and a small
sample_max_stack from one event can shrink the depth for all others in
the same PMI.

As a result, later events may output a shorter LBR stack than requested
even when they do not require a small sample depth. Do not overwrite the
shared LBR stack depth in perf_sample_save_brstack(); instead, keep the
per-event limit in a local variable so each event is constrained
independently without mutating global state.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 include/linux/perf_event.h | 13 ++++++++++---
 kernel/events/core.c       |  8 ++++----
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 19de3f1cbd6e..b9a4a821c7c6 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -1424,12 +1424,19 @@ static inline bool has_branch_stack(struct perf_event *event)
 	return event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK;
 }
 
+static inline u64 max_lbr_stack_depth(struct perf_event *event,
+				      struct perf_branch_stack *brs)
+{
+	return min_t(u16, event->attr.sample_max_stack, brs->nr);
+}
+
 static inline void perf_sample_save_brstack(struct perf_sample_data *data,
 					    struct perf_event *event,
 					    struct perf_branch_stack *brs,
 					    u64 *brs_cntr)
 {
 	int size = sizeof(u64); /* nr */
+	u64 lbr_nr;
 
 	if (!has_branch_stack(event))
 		return;
@@ -1439,9 +1446,9 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
 	if (branch_sample_hw_index(event))
 		size += sizeof(u64);
 
-	brs->nr = min_t(u16, event->attr.sample_max_stack, brs->nr);
+	lbr_nr = max_lbr_stack_depth(event, brs);
 
-	size += brs->nr * sizeof(struct perf_branch_entry);
+	size += lbr_nr * sizeof(struct perf_branch_entry);
 
 	/*
 	 * The extension space for counters is appended after the
@@ -1449,7 +1456,7 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
 	 * of events of each branch.
 	 */
 	if (brs_cntr)
-		size += brs->nr * sizeof(u64);
+		size += lbr_nr * sizeof(u64);
 
 	data->br_stack = brs;
 	data->br_stack_cntr = brs_cntr;
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 457d75ea7142..db2613c2e1a2 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8462,12 +8462,12 @@ void perf_output_sample(struct perf_output_handle *handle,
 
 	if (sample_type & PERF_SAMPLE_BRANCH_STACK) {
 		if (data->br_stack) {
+			u64 lbr_nr = max_lbr_stack_depth(event, data->br_stack);
 			size_t size;
 
-			size = data->br_stack->nr
-			     * sizeof(struct perf_branch_entry);
+			size = lbr_nr * sizeof(struct perf_branch_entry);
 
-			perf_output_put(handle, data->br_stack->nr);
+			perf_output_put(handle, lbr_nr);
 			if (branch_sample_hw_index(event))
 				perf_output_put(handle, data->br_stack->hw_idx);
 			perf_output_copy(handle, data->br_stack->entries, size);
@@ -8476,7 +8476,7 @@ void perf_output_sample(struct perf_output_handle *handle,
 			 * right after the struct perf_branch_stack.
 			 */
 			if (data->br_stack_cntr) {
-				size = data->br_stack->nr * sizeof(u64);
+				size = lbr_nr * sizeof(u64);
 				perf_output_copy(handle, data->br_stack_cntr, size);
 			}
 		} else {
-- 
2.34.1

Re: [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues

[Mi, Dapeng]

Sashiko reported an issue on the patch 15/20 "perf/x86/intel: Refresh LBR
stack for non-PEBS LBR events when needed",
https://lore.kernel.org/all/sashiko-outbox-200872@kernel.org/.

It looks a new version is needed to fix the issue. Thanks.

On 11/6/2026 12:32 PM, Dapeng Mi wrote:
> This patch series fixes several x86 LBR sampling issues discovered during
> review and debugging. Most of them are related to BPF LBR snapshotting
> support.
>
> The series originally targeted the BPF stack out-of-bounds access issue,
> but additional problems were found during debugging and code review,
> including:
> - PMU and LBR being re-enabled unconditionally at BPF snapshotting exit,
>   even when they had already been disabled before BPF snapshotting on both
>   AMD and Intel.
> - LBR not being fully frozen in the PMI handler on Intel, which allows
>   branches in the PMI path to pollute the BPF LBR snapshot.
> - BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on
>   Intel.
> - PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on
>   Intel when both overflow in the same PMI.
> - LBR counter events reporting zero counters on Intel when multiple
>   LBR counter events overflow in the same PMI.
>
>
> Patch layout:
> - Patch 01: Add a sanity check to ensure the LBR stack does not exceed
>   MAX_LBR_ENTRIES.
> - Patch 02: Validate that hardware LBR data has actually been stored in
>   the XSAVE area before parsing it.
> - Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on
>   AMD and Intel separately.
> - Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue
>   in perf_snapshot_branch_stack() for AMD and Intel separately.
> - Patch 09: Freeze LBR throughout the PMI processing period and ensure
>   all LBR events observe a consistent LBR stack in the PMI handler.
> - Patch 10: Add counted bounds tracking for perf_branch_stack{}.
> - Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshotting to avoid
>   overwriting the default LBR stack of other LBR events.
> - Patch 13: Ensure perf_snapshot_branch_stack() snapshots the live
>   hardware LBR data instead of the cached PEBS LBR stack.
> - Patch 14: Refresh LBR stack in PMI only when LBR users are active for
>   AMD.
> - Patch 15: Prevent PEBS LBR data from overwriting non-PEBS LBR data when
>   both overflow in the same PMI.
> - Patch 16: Remove dead lbr_pebs_users and relative code.
> - Patch 17-18: Ensure LBR counters are not dropped when multiple LBR
>   counter events overflow in the same PMI.
> - Patch 19: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to
>   avoid LBR stack out-of-bounds access.
> - Patch 20: Fix the shared LBR stack depth truncation issue.
>
>
> Changes in v6:
> - Patch 01: Move LBR stack depth validation into common
>   init_hw_perf_events(). (Peter)
> - Patch 10: New patch, add __counted_by() tracking for flexible array
>   entries[] of perf_branch_stack{}. (Peter)
> - Patch 14: Refresh LBR stack only when LBR users are active for AMD.
>   (Peter)
> - Patch 15: Refactor code to ensure no duplicated LBR stack reading.
>   (Peter)
> - Patch 16: New patch, remove dead lbr_pebs_users.
>
> Changes in v5:
> - Patch 02: Clear xsave->header.xfeatures before calling the xsaves
>   instructuion to read LBR states.
> - Patch 05-18: New patches to fix newly discovered issues described above.
>
> Changes in v4:
> - Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds
>   MAX_LBR_ENTRIES (Sashiko).
> - Patch 05: New patch, which always snapshots live hardware LBR stack
>   for BPF path (Sashiko).
>
> Changes in v3:
> - Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude
>   MAX_LBR_ENTRIES (Internal Sashiko).
> - Patch 03: Clean cached LBR stack when reseting hardware LBR registers,
>   which prevents stale LBR stack leakage.
> - Patch 04: Don't reset cached LBR stack if all LBR events are PEBS
>   events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).
>
> Changes in v2:
> - Include a fix for Intel LBR.
> - Add Tested-by from Dapeng to the Intel fix.
>   https://lore.kernel.org/all/eb6c7bda-0dcf-4fac-ad69-9f5b24abc21b@linux.intel.com/
>
> History:
> v5: https://lore.kernel.org/all/20261029043914.1395147-1-dapeng1.mi@linux.intel.com/
> v4: https://lore.kernel.org/all/20261021050711.4043703-1-dapeng1.mi@linux.intel.com/
> v3: https://lore.kernel.org/all/20261019070836.2972403-1-dapeng1.mi@linux.intel.com/
> v2: https://lore.kernel.org/all/cover.1792064123.git.sandipan.das@amd.com/
> v1: https://lore.kernel.org/all/20261007181002.4005262-1-sandipan.das@amd.com/
>
> Dapeng Mi (19):
>   perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
>   perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
>   perf/x86/intel/lbr: Limit branch snapshots to valid entries
>   perf/x86: Add lbr_enabled to track LBR state
>   perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
>   perf/x86/intel: Add helper to enable PMU without LBR
>   perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
>   perf/x86/intel: Keep LBR frozen throughout PMI handling
>   perf: Add counted-bounds tracking for perf_branch_stack
>   perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
>   perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
>   perf/x86/intel: Always snapshot live hardware LBR state for BPF
>   perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active
>   perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
>   perf/x86/intel: Remove dead lbr_pebs_users
>   perf/x86/intel: Clear raw LBR counters in the LBR stack
>   perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
>   perf/x86/intel: Pass LBR stack size into store_lbr()
>   perf/core: Don't overwrite shared LBR stack depth
>
> Sandipan Das (1):
>   perf/x86/amd/lbr: Limit branch snapshots to valid entries
>
>  arch/powerpc/perf/core-book3s.c |  43 +++++---
>  arch/x86/events/amd/brs.c       |   7 +-
>  arch/x86/events/amd/core.c      |  21 ++--
>  arch/x86/events/amd/lbr.c       |  48 +++++----
>  arch/x86/events/core.c          |   8 +-
>  arch/x86/events/intel/core.c    | 130 ++++++++++++++++++++----
>  arch/x86/events/intel/ds.c      |  20 ++--
>  arch/x86/events/intel/lbr.c     | 170 +++++++++++++++++++-------------
>  arch/x86/events/perf_event.h    |  45 +++++++--
>  drivers/perf/arm_brbe.c         |   5 +
>  include/linux/perf_event.h      |  15 ++-
>  kernel/events/core.c            |   8 +-
>  12 files changed, 365 insertions(+), 155 deletions(-)
>
>
> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4