[PATCH v7 0/8] drm/panthor: Performance counter implementation with single manual client support

[PATCH v7 0/8] drm/panthor: Performance counter implementation with single manual client support

[Lukas Zapolskas]

Hello,

This patch set implements initial support for performance counter
sampling in Panthor, as a follow-up for Adrián Larumbe's patch
set [1]. In response to follow-up comments on v6, uAPI features that
were not being populated in the original series are now dropped, leaving
only the features necessary for the initial implementation.

The Mali performance counters are controlled via a global firmware
interface to point to a buffer that contains samples of either 64 or 128
32-bit counters per block instance. Each of the block types has its own
32-bit enable mask, allowing for finer-grained control of which counters
are emitted. The size of the sample is dictated by the sum of the
FIRMWARE_SIZE and the HARDWARE_SIZE fields, but neither the FW nor the
HW provide sufficient information about what block types are present at
what offsets. This information is architectural, and may change between
different major versions, as well as different firmware revisions.

To work around the instability of the counter block layout, the kernel
maintains its own offset table per GPU, and provides that via
additional block headers inserted into the data. This enables old
userspace clients to interpret a buffer copied over directly from hardware,
for users like libGPUCounters[2] embedded in an executable to retain some level of
support for counter sampling without being updated.

The following performance counter implementation supports a single
userspace explicitly requesting samples via ioctl, and managing the
consumed samples via a ring buffer and a set of control indices. The
user client passes in the resources to create a session, at which point
they can explicitly enable or disable sampling via the
PERF_SESSION_START/PERF_SESSION_STOP ioctls.

Samples are request via an explicit PERF_SESSION_SAMPLE ioctl. When
the request round trips back to the KMD, an eventfd is signaled for the
user indicating that the sample is complete and can be found in the
shared user/kernel ring buffer.

Existing performance counter workflows, such as those in game
engines, and user-space power models/governor implementations
require the ability to simultaneously obtain counter data on the same
driver. Though this is not enabled out of the box, handling for multiple
sessions sampling at different cadences is added via the sample
accumulation handling.

Mesa MR: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35024

v7:
- Move the perf commands into their own ioctls
- Moved session tracking to the pfile
- Fixed padding on uAPI structs
- Removed the sampler abstraction
- Moved new registers to a separate commit
- Drop the offset argument for session creation
- Drop the count of enabled clients, using the session_list directly
- Removed the r-b's due to significant changes in the patches
- Added check for CAP_PERFMON on session setup
- Fixed architectural handling of prfcnt_en
- Link to v6: https://lore.kernel.org/dri-devel/20251215171453.2506348-1-lukas.zapolskas@arm.com/

v6:
- xa_erase -> xa_delete to fix up dangling session issues
- Move setting user data into request callback to avoid setting it on error
- Embed the enable mask into the sampler
- Added helper for sample_enable and sampler_disable to eliminate duplicate checks
- Drop PM reference on session enable failure
- Drop the panthor_perf_em_zero function
- Added R-b's for the first several patches
- Added added reading of the PRFCNT_FEATURES register on supporting architectures
- link to v5: https://lore.kernel.org/dri-devel/cover.1753449448.git.lukas.zapolskas@arm.com/

v5:
- Started re-using perf info size fields instead of
  recomputing the size where needed
- Removed panthor_file pointer to the drm_file
- Fixed ordering of subsystem unplug on init failure
- Using the kernel struct size to allocate memory for user-passed
  uAPI struct.
- Inlined panthor_perf_sampler_{suspend,resume} into
  panthor_perf_{suspend,resume}
- Inlined
- Updated all callers of CIRC_SPACE_TO_END to use CIRC_SPACE
  for correct ring buffer wraparound.
- Free the session and sampler enable maps on termination
- Drop the return values from panthor_perf_{suspend,resume}
- Update userdata and end timestamp on accumulation
- Removed the ptdev checks on suspend and resume paths
- Link to v4: https://lore.kernel.org/dri-devel/cover.1747148172.git.lukas.zapolskas@arm.com/

v4:
- Added sample size to the uAPI.
- Clarified the bit-to-counter mapping for enable masks.
- Fixed IRQ handling: the PERFCNT_THRESHOLD and PERFCNT_OVERFLOW
  interrupts can be handled by checking the difference between the
  REQ and ACK bits, whereas PERFCNT_SAMPLE needs external data to
  validate.
- FW ring buffer indices are now only wrapped when reading the buffer
  and are otherwise left in their pre-wrapped form.
- Accumulation index is now bumped after the first copy.
- All insert and extract index reads now use the proper, full-width
  type.
- L2 slices are now computed via a macro to extract the relevant
  bits from the MEM_FEATURES register. This macro was moved from
  the uAPI due to changes in the register making it unstable.
- Consistently take the sampler lock to check if a sample has been
  requested.
- Link to v3: https://lore.kernel.org/dri-devel/cover.1743517880.git.lukas.zapolskas@arm.com/

v3:
- Fixed offset issues into FW ring buffer
- Fixed sparse shader core handling
- Added pre- and post- reset handlers
- Added module param to control size of FW ring buffer
- Clarified naming on sampler functions
- Added error logging for PERF_SETUP
- Link to v2: https://lore.kernel.org/dri-devel/20241211165024.490748-1-lukas.zapolskas@arm.com/

RFC v2:
- Link to v1: https://lore.kernel.org/lkml/20240305165820.585245-1-adrian.larumbe@collabora.com/T/#m67d1f89614fe35dc0560e8304d6731eb1a6942b6

[1]: https://lore.kernel.org/lkml/20240305165820.585245-1-adrian.larumbe@collabora.com/T/#m67d1f89614fe35dc0560e8304d6731eb1a6942b6
[2]: https://github.com/ARM-software/libGPUCounters

base commit: e48123c607a0

---
Adrián Larumbe (1):
      drm/panthor: Implement sample handling

Lukas Zapolskas (7):
      drm/panthor: Add performance counter uAPI
      drm/panthor: Add register definitions for perf
      drm/panthor: Add DEV_QUERY.PERF_INFO handling
      drm/panthor: Add panthor perf initialization and termination
      drm/panthor: Introduce sampling sessions to handle userspace clients
      drm/panthor: Add suspend, resume and reset handling
      drm/panthor: Expose the panthor perf ioctls

 drivers/gpu/drm/panthor/Makefile           |    1 +
 drivers/gpu/drm/panthor/panthor_device.c   |   17 +-
 drivers/gpu/drm/panthor/panthor_device.h   |   12 +-
 drivers/gpu/drm/panthor/panthor_drv.c      |  134 +-
 drivers/gpu/drm/panthor/panthor_fw.c       |    5 +
 drivers/gpu/drm/panthor/panthor_fw.h       |    9 +-
 drivers/gpu/drm/panthor/panthor_gpu.c      |    5 +
 drivers/gpu/drm/panthor/panthor_gpu.h      |    2 +
 drivers/gpu/drm/panthor/panthor_gpu_regs.h |    4 +
 drivers/gpu/drm/panthor/panthor_perf.c     | 1953 ++++++++++++++++++++++++++++
 drivers/gpu/drm/panthor/panthor_perf.h     |   40 +
 include/uapi/drm/panthor_drm.h             |  370 ++++++
 12 files changed, 2547 insertions(+), 5 deletions(-)
---
base-commit: 4b7a49d66c8398a7984dff4ac54d1acc6b3a58df
change-id: 20260529-panthor-performance-counters-594dea247c2b

Best regards,
--  
Lukas Zapolskas <lukas.zapolskas@arm.com>

[PATCH v7 1/8] drm/panthor: Add performance counter uAPI

[Lukas Zapolskas]

This patch extends the DEV_QUERY ioctl to return information about the
performance counter setup for userspace, and introduces the new
DRM_PANTHOR_PERF_SESSION ioctls in order to allow for the sampling of
performance counters.

The new design is inspired by the perf aux ringbuffer [0], with the
insert and extract indices being mapped to userspace, allowing
multiple samples to be exposed at any given time. The buffer is
annotated to include sample metadata and block metadata inline with
the elements they describe.

Userspace is responsible for passing in resources for samples to be
exposed, including the event file descriptor for notification of new
sample availability, the ringbuffer BO to store samples, and the
control BO.

[0]: https://docs.kernel.org/userspace-api/perf_ring_buffer.html

Co-developed-by: Mihail Atanassov <mihail.atanassov@arm.com>
Signed-off-by: Mihail Atanassov <mihail.atanassov@arm.com>
Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 include/uapi/drm/panthor_drm.h | 370 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 370 insertions(+)

diff --git a/include/uapi/drm/panthor_drm.h b/include/uapi/drm/panthor_drm.h
index a2ff0f4ec691..e45f75b76245 100644
--- a/include/uapi/drm/panthor_drm.h
+++ b/include/uapi/drm/panthor_drm.h
@@ -154,6 +154,23 @@ enum drm_panthor_ioctl_id {
 	 * This is useful for imported BOs.
 	 */
 	DRM_PANTHOR_BO_QUERY_INFO,
+
+	/** @DRM_PANTHOR_PERF_SESSION_CREATE: Create a performance counter session. */
+	DRM_PANTHOR_PERF_SESSION_CREATE,
+
+	/** @DRM_PANTHOR_PERF_SESSION_START: Start a performance counter session. */
+	DRM_PANTHOR_PERF_SESSION_START,
+
+	/** @DRM_PANTHOR_PERF_SESSION_SAMPLE: Request a manual sample on a performance counter
+	 * session.
+	 */
+	DRM_PANTHOR_PERF_SESSION_SAMPLE,
+
+	/** @DRM_PANTHOR_PERF_SESSION_STOP: Stop a performance counter session. */
+	DRM_PANTHOR_PERF_SESSION_STOP,
+
+	/** @DRM_PANTHOR_PERF_SESSION_DESTROY: Destroy a performance counter session. */
+	DRM_PANTHOR_PERF_SESSION_DESTROY,
 };
 
 /**
@@ -256,6 +273,9 @@ enum drm_panthor_dev_query_type {
 
 	/** @DRM_PANTHOR_DEV_QUERY_MMU_INFO: Query MMU information. */
 	DRM_PANTHOR_DEV_QUERY_MMU_INFO,
+
+	/** @DRM_PANTHOR_DEV_QUERY_PERF_INFO: Query performance counter interface information. */
+	DRM_PANTHOR_DEV_QUERY_PERF_INFO,
 };
 
 /**
@@ -517,6 +537,93 @@ struct drm_panthor_group_priorities_info {
 	__u8 pad[3];
 };
 
+/**
+ * enum drm_panthor_perf_block_type - Performance counter supported block types.
+ */
+enum drm_panthor_perf_block_type {
+	/** @DRM_PANTHOR_PERF_BLOCK_METADATA: Internal use only. */
+	DRM_PANTHOR_PERF_BLOCK_METADATA = 0,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_FW: The FW counter block. */
+	DRM_PANTHOR_PERF_BLOCK_FW,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_CSHW: The CSHW counter block. */
+	DRM_PANTHOR_PERF_BLOCK_CSHW,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_TILER: The tiler counter block. */
+	DRM_PANTHOR_PERF_BLOCK_TILER,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_MEMSYS: A memsys counter block. */
+	DRM_PANTHOR_PERF_BLOCK_MEMSYS,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_SHADER: A shader core counter block. */
+	DRM_PANTHOR_PERF_BLOCK_SHADER,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_FIRST: Internal use only. */
+	DRM_PANTHOR_PERF_BLOCK_FIRST = DRM_PANTHOR_PERF_BLOCK_FW,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_LAST: Internal use only. */
+	DRM_PANTHOR_PERF_BLOCK_LAST = DRM_PANTHOR_PERF_BLOCK_SHADER,
+
+	/** @DRM_PANTHOR_PERF_BLOCK_MAX: Internal use only. */
+	DRM_PANTHOR_PERF_BLOCK_MAX = DRM_PANTHOR_PERF_BLOCK_LAST + 1,
+};
+
+/**
+ * struct drm_panthor_perf_info - Performance counter interface information
+ *
+ * Structure grouping all queryable information relating to the performance counter
+ * interfaces.
+ */
+struct drm_panthor_perf_info {
+	/**
+	 * @counters_per_block: The number of 8-byte counters available in a block.
+	 */
+	__u32 counters_per_block;
+
+	/**
+	 * @sample_header_size: The size of the header struct available at the beginning
+	 * of every sample.
+	 */
+	__u32 sample_header_size;
+
+	/**
+	 * @block_header_size: The size of the header struct inline with the counters for a
+	 * single block.
+	 */
+	__u32 block_header_size;
+
+	/**
+	 * @sample_size: The size of a fully annotated sample, starting with a sample header
+	 *               of size @sample_header_size bytes, and all available blocks for the current
+	 *               configuration, each comprised of @counters_per_block 64-bit counters and
+	 *               a block header of @block_header_size bytes.
+	 *
+	 *               The user must use this field to allocate size for the ring buffer. In
+	 *               the case of new blocks being added, an old userspace can always use
+	 *               this field and ignore any blocks it does not know about.
+	 */
+	__u32 sample_size;
+
+	/** @fw_blocks: Number of FW blocks available. */
+	__u32 fw_blocks;
+
+	/** @cshw_blocks: Number of CSHW blocks available. */
+	__u32 cshw_blocks;
+
+	/** @tiler_blocks: Number of tiler blocks available. */
+	__u32 tiler_blocks;
+
+	/** @memsys_blocks: Number of memsys blocks available. */
+	__u32 memsys_blocks;
+
+	/** @shader_blocks: Number of shader core blocks available. */
+	__u32 shader_blocks;
+
+	/** @pad1: MBZ. */
+	__u32 pad1;
+};
+
 /**
  * struct drm_panthor_dev_query - Arguments passed to DRM_PANTHOR_IOCTL_DEV_QUERY
  */
@@ -1270,6 +1377,259 @@ struct drm_panthor_bo_query_info {
 	__u32 pad;
 };
 
+/**
+ * DOC: Performance counter decoding in userspace.
+ *
+ * Each sample will be exposed to userspace in the following manner:
+ *
+ * +--------+--------+------------------------+--------+-------------------------+-----+
+ * | Sample | Block  |        Block           | Block  |         Block           | ... |
+ * | header | header |        counters        | header |         counters        |     |
+ * +--------+--------+------------------------+--------+-------------------------+-----+
+ *
+ * Each sample starts with a &struct drm_panthor_perf_sample_header containing
+ * timestamps, sample flags and the command's user data.
+ *
+ * Each block starts with a &struct drm_panthor_perf_block_header identifying its
+ * type and index, followed by counters_per_block 64-bit counters, as reported by
+ * &struct drm_panthor_perf_info.
+ */
+
+/**
+ * struct drm_panthor_perf_block_header - Header present before every block in the
+ * sample ringbuffer.
+ */
+struct drm_panthor_perf_block_header {
+	/** @block_type: Type of the block. */
+	__u8 block_type;
+
+	/** @block_idx: Block index. */
+	__u8 block_idx;
+
+	/** @pad0: MBZ. */
+	__u8 pad0[6];
+};
+
+/**
+ * enum drm_panthor_perf_sample_flags - Sample-wide events that occurred over the sampling
+ * period.
+ */
+enum drm_panthor_perf_sample_flags {
+	/**
+	 * @DRM_PANTHOR_PERF_SAMPLE_OVERFLOW: This sample contains overflows due to the duration
+	 * of the sampling period.
+	 */
+	DRM_PANTHOR_PERF_SAMPLE_OVERFLOW = 1 << 0,
+};
+
+/**
+ * struct drm_panthor_perf_sample_header - Header present before every sample.
+ */
+struct drm_panthor_perf_sample_header {
+	/**
+	 * @timestamp_start_ns: Earliest timestamp that values in this sample represent, in
+	 * nanoseconds. Derived from CLOCK_MONOTONIC_RAW.
+	 */
+	__u64 timestamp_start_ns;
+
+	/**
+	 * @timestamp_end_ns: Latest timestamp that values in this sample represent, in
+	 * nanoseconds. Derived from CLOCK_MONOTONIC_RAW.
+	 */
+	__u64 timestamp_end_ns;
+
+	/** @flags: Current sample flags, combination of drm_panthor_perf_sample_flags. */
+	__u32 flags;
+
+	/** @pad: MBZ. */
+	__u32 pad;
+
+	/**
+	 * @user_data: User data provided as part of the command that triggered this sample.
+	 */
+	__u64 user_data;
+};
+
+/**
+ * enum drm_panthor_perf_counter_set - The counter set to be requested from the hardware.
+ *
+ * The hardware supports a single performance counter set at a time, so requesting any set other
+ * than the primary may fail if another process is sampling at the same time.
+ *
+ * If in doubt, the primary counter set has the most commonly used counters and requires no
+ * additional permissions to open.
+ */
+enum drm_panthor_perf_counter_set {
+	/**
+	 * @DRM_PANTHOR_PERF_SET_PRIMARY: The default set configured on the hardware.
+	 *
+	 * This is the only set for which all counters in all blocks are defined.
+	 */
+	DRM_PANTHOR_PERF_SET_PRIMARY,
+
+	/** @DRM_PANTHOR_PERF_SET_SECONDARY: The secondary performance counter set. */
+	DRM_PANTHOR_PERF_SET_SECONDARY,
+
+	/** @DRM_PANTHOR_PERF_SET_TERTIARY: The tertiary performance counter set.*/
+	DRM_PANTHOR_PERF_SET_TERTIARY,
+};
+
+/**
+ * struct drm_panthor_perf_ringbuf_control - Struct used to map in the ring buffer control indices
+ *                                           into memory shared between user and kernel.
+ *
+ */
+struct drm_panthor_perf_ringbuf_control {
+	/**
+	 * @extract_idx: Number of samples consumed or discarded by userspace. Only
+	 *               modifiable by userspace.
+	 */
+	__u64 extract_idx;
+
+	/**
+	 * @insert_idx: Number of samples emitted by the kernel. Only modifiable by
+	 *              the kernel.
+	 */
+	__u64 insert_idx;
+};
+
+/**
+ * struct drm_panthor_perf_session_create - Arguments passed to DRM_PANTHOR_PERF_SESSION_CREATE.
+ */
+struct drm_panthor_perf_session_create {
+	/**
+	 * @block_set: Set of performance counter blocks, member of
+	 *             enum drm_panthor_perf_counter_set.
+	 *
+	 * This is a global configuration and only one set can be active at a time. If
+	 * another client has already requested a counter set, any further requests
+	 * for a different counter set will fail and return an -EBUSY.
+	 *
+	 * If the requested set does not exist, the request will fail and return an -EINVAL.
+	 *
+	 * Some sets have additional requirements to be enabled, and the setup request will
+	 * fail with an -EACCES if these requirements are not satisfied.
+	 */
+	__u8 block_set;
+
+	/** @pad: MBZ. */
+	__u8 pad[7];
+
+	/** @fd: eventfd for signalling the availability of a new sample. */
+	__u32 fd;
+
+	/** @ringbuf_handle: Handle to the BO to write perf counter sample to. */
+	__u32 ringbuf_handle;
+
+	/**
+	 * @control_handle: Handle to the BO containing a contiguous 16 byte range, used for the
+	 * insert and extract indices for the ringbuffer.
+	 */
+	__u32 control_handle;
+
+	/**
+	 * @sample_slots: The number of slots available in the userspace-provided BO. Must be
+	 * a power of 2.
+	 *
+	 * If sample_slots * sample_size does not match the BO size, the setup request will fail.
+	 */
+	__u32 sample_slots;
+
+	/**
+	 * @fw_enable_mask: Bitmask of counters to request from the FW counter block. Any bits
+	 * past the first drm_panthor_perf_info.counters_per_block bits will be ignored. Bit 0
+	 * corresponds to counter 0.
+	 */
+	__u64 fw_enable_mask[2];
+
+	/**
+	 * @cshw_enable_mask: Bitmask of counters to request from the CSHW counter block. Any bits
+	 * past the first drm_panthor_perf_info.counters_per_block bits will be ignored. Bit 0
+	 * corresponds to counter 0.
+	 */
+	__u64 cshw_enable_mask[2];
+
+	/**
+	 * @tiler_enable_mask: Bitmask of counters to request from the tiler counter block. Any
+	 * bits past the first drm_panthor_perf_info.counters_per_block bits will be ignored. Bit
+	 * 0 corresponds to counter 0.
+	 */
+	__u64 tiler_enable_mask[2];
+
+	/**
+	 * @memsys_enable_mask: Bitmask of counters to request from the memsys counter blocks. Any
+	 * bits past the first drm_panthor_perf_info.counters_per_block bits will be ignored. Bit 0
+	 * corresponds to counter 0.
+	 */
+	__u64 memsys_enable_mask[2];
+
+	/**
+	 * @shader_enable_mask: Bitmask of counters to request from the shader core counter blocks.
+	 * Any bits past the first drm_panthor_perf_info.counters_per_block bits will be ignored.
+	 * Bit 0 corresponds to counter 0.
+	 */
+	__u64 shader_enable_mask[2];
+};
+
+/**
+ * struct drm_panthor_perf_session_start - Arguments passed to DRM_PANTHOR_PERF_SESSION_START.
+ */
+struct drm_panthor_perf_session_start {
+	/** @handle: Performance counter session handle. */
+	__u32 handle;
+
+	/** @pad0: MBZ. */
+	__u32 pad0;
+
+	/**
+	 * @user_data: User provided data that will be attached to automatic samples collected
+	 * until the next DRM_PANTHOR_PERF_SESSION_SAMPLE.
+	 */
+	__u64 user_data;
+};
+
+/**
+ * struct drm_panthor_perf_session_sample - Arguments passed to DRM_PANTHOR_PERF_SESSION_SAMPLE.
+ */
+struct drm_panthor_perf_session_sample {
+	/** @handle: Performance counter session handle. */
+	__u32 handle;
+
+	/** @pad0: MBZ. */
+	__u32 pad0;
+
+	/** @user_data: User provided data that will be attached to the sample.*/
+	__u64 user_data;
+};
+
+/**
+ * struct drm_panthor_perf_session_stop - Arguments passed to DRM_PANTHOR_PERF_SESSION_STOP.
+ */
+struct drm_panthor_perf_session_stop {
+	/** @handle: Performance counter session handle. */
+	__u32 handle;
+
+	/** @pad0: MBZ. */
+	__u32 pad0;
+
+	/**
+	 * @user_data: User provided data that will be attached to the automatic sample collected
+	 * at the end of this sampling session.
+	 */
+	__u64 user_data;
+};
+
+/**
+ * struct drm_panthor_perf_session_destroy - Arguments passed to DRM_PANTHOR_PERF_SESSION_DESTROY.
+ */
+struct drm_panthor_perf_session_destroy {
+	/** @handle: Performance counter session handle. */
+	__u32 handle;
+
+	/** @pad0: MBZ. */
+	__u32 pad0;
+};
+
 /**
  * DRM_IOCTL_PANTHOR() - Build a Panthor IOCTL number
  * @__access: Access type. Must be R, W or RW.
@@ -1320,6 +1680,16 @@ enum {
 		DRM_IOCTL_PANTHOR(WR, BO_SYNC, bo_sync),
 	DRM_IOCTL_PANTHOR_BO_QUERY_INFO =
 		DRM_IOCTL_PANTHOR(WR, BO_QUERY_INFO, bo_query_info),
+	DRM_IOCTL_PANTHOR_PERF_SESSION_CREATE =
+		DRM_IOCTL_PANTHOR(WR, PERF_SESSION_CREATE, perf_session_create),
+	DRM_IOCTL_PANTHOR_PERF_SESSION_START =
+		DRM_IOCTL_PANTHOR(WR, PERF_SESSION_START, perf_session_start),
+	DRM_IOCTL_PANTHOR_PERF_SESSION_SAMPLE =
+		DRM_IOCTL_PANTHOR(WR, PERF_SESSION_SAMPLE, perf_session_sample),
+	DRM_IOCTL_PANTHOR_PERF_SESSION_STOP =
+		DRM_IOCTL_PANTHOR(WR, PERF_SESSION_STOP, perf_session_stop),
+	DRM_IOCTL_PANTHOR_PERF_SESSION_DESTROY =
+		DRM_IOCTL_PANTHOR(WR, PERF_SESSION_DESTROY, perf_session_destroy),
 };
 
 #if defined(__cplusplus)

-- 
2.43.0

[PATCH v7 2/8] drm/panthor: Add register definitions for perf

[Lukas Zapolskas]

The perfcnt_size register, needed to size the FW-side ring buffer, is
comprised out of two fields, FIRMWARE_SIZE and HARDWARE_SIZE, indicating
the size of the corresponding components of a single sample. The total
size of a single sample in that ring buffer is the sum of the two.

The perfcnt_config register configures the size of the ring buffer, the
counter set requested by the user, and whether the optional metadata block
is required.

For newer FW revisions, the perfcnt_features register reports the
presence of an optional metadata block.

GPUs with arch major 11 or greater also report the number of performance
counters in a single counter block via the GPU_PRFCNT_FEATURES.

Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 drivers/gpu/drm/panthor/panthor_fw.h       | 9 ++++++++-
 drivers/gpu/drm/panthor/panthor_gpu_regs.h | 4 ++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/panthor/panthor_fw.h b/drivers/gpu/drm/panthor/panthor_fw.h
index a99a9b6f4825..124e2b40b15f 100644
--- a/drivers/gpu/drm/panthor/panthor_fw.h
+++ b/drivers/gpu/drm/panthor/panthor_fw.h
@@ -198,8 +198,12 @@ struct panthor_fw_global_control_iface {
 	u32 output_va;
 	u32 group_num;
 	u32 group_stride;
+#define GLB_PERFCNT_FW_SIZE(x) ((((x) >> 16) << 8))
+#define GLB_PERFCNT_HW_SIZE(x) (((x) & GENMASK(15, 0)) << 8)
 	u32 perfcnt_size;
 	u32 instr_features;
+#define PERFCNT_FEATURES_MD_SIZE(x) (((x) & GENMASK(3, 0)) << 8)
+	u32 perfcnt_features;
 };
 
 struct panthor_fw_global_input_iface {
@@ -208,7 +212,7 @@ struct panthor_fw_global_input_iface {
 #define GLB_CFG_ALLOC_EN			BIT(2)
 #define GLB_CFG_POWEROFF_TIMER			BIT(3)
 #define GLB_PROTM_ENTER				BIT(4)
-#define GLB_PERFCNT_EN				BIT(5)
+#define GLB_PERFCNT_ENABLE			BIT(5)
 #define GLB_PERFCNT_SAMPLE			BIT(6)
 #define GLB_COUNTER_EN				BIT(7)
 #define GLB_PING				BIT(8)
@@ -248,6 +252,9 @@ struct panthor_fw_global_input_iface {
 	u64 perfcnt_base;
 	u32 perfcnt_extract;
 	u32 reserved3[3];
+#define GLB_PERFCNT_CONFIG_SIZE(x) ((x) & GENMASK(7, 0))
+#define GLB_PERFCNT_CONFIG_SET(x) (((x) & GENMASK(1, 0)) << 8)
+#define GLB_PERFCNT_METADATA_ENABLE BIT(10)
 	u32 perfcnt_config;
 	u32 perfcnt_csg_select;
 	u32 perfcnt_fw_enable;
diff --git a/drivers/gpu/drm/panthor/panthor_gpu_regs.h b/drivers/gpu/drm/panthor/panthor_gpu_regs.h
index 4c5b953796e4..2f0877068e44 100644
--- a/drivers/gpu/drm/panthor/panthor_gpu_regs.h
+++ b/drivers/gpu/drm/panthor/panthor_gpu_regs.h
@@ -23,6 +23,7 @@
 #define GPU_TILER_FEATURES				0xC
 #define GPU_MEM_FEATURES				0x10
 #define   GROUPS_L2_COHERENT				BIT(0)
+#define   GPU_MEM_FEATURES_L2_SLICES(x)			((((x) & GENMASK(11, 8)) >> 8) + 1)
 
 #define GPU_MMU_FEATURES				0x14
 #define  GPU_MMU_FEATURES_VA_BITS(x)			((x) & GENMASK(7, 0))
@@ -69,6 +70,9 @@
 #define   GPU_FEATURES_RAY_INTERSECTION			BIT(2)
 #define   GPU_FEATURES_RAY_TRAVERSAL			BIT(5)
 
+#define GPU_PRFCNT_FEATURES				0x68
+#define   PRFCNT_FEATURES_COUNTER_BLOCK_SIZE(x)		(((x) & GENMASK(7, 0)) << 8)
+
 #define GPU_TIMESTAMP_OFFSET				0x88
 #define GPU_CYCLE_COUNT					0x90
 #define GPU_TIMESTAMP					0x98

-- 
2.43.0

[PATCH v7 3/8] drm/panthor: Add DEV_QUERY.PERF_INFO handling

[Lukas Zapolskas]

This change adds the IOCTL to query data about the performance counter
setup. Some of this data was available via previous DEV_QUERY calls,
for instance for GPU info, but exposing it via PERF_INFO
minimizes the overhead of creating a single session to just the one
aggregate IOCTL, and removes the dependence on architectural details of
the counter layout.

This required reading one of the GPU registers, which is exposed via
a new helper in the gpu header.

The current handling covers GPUs from the Mali-Gx10 up to and including
the Mali-G1.

Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 drivers/gpu/drm/panthor/Makefile         |  1 +
 drivers/gpu/drm/panthor/panthor_device.c |  7 +++
 drivers/gpu/drm/panthor/panthor_device.h |  3 ++
 drivers/gpu/drm/panthor/panthor_drv.c    | 10 +++-
 drivers/gpu/drm/panthor/panthor_gpu.c    |  5 ++
 drivers/gpu/drm/panthor/panthor_gpu.h    |  2 +
 drivers/gpu/drm/panthor/panthor_perf.c   | 87 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/panthor/panthor_perf.h   | 15 ++++++
 8 files changed, 129 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/panthor/Makefile b/drivers/gpu/drm/panthor/Makefile
index dd15d52a88ba..c65fff290720 100644
--- a/drivers/gpu/drm/panthor/Makefile
+++ b/drivers/gpu/drm/panthor/Makefile
@@ -10,6 +10,7 @@ panthor-y := \
 	panthor_heap.o \
 	panthor_hw.o \
 	panthor_mmu.o \
+	panthor_perf.o \
 	panthor_pwr.o \
 	panthor_sched.o
 
diff --git a/drivers/gpu/drm/panthor/panthor_device.c b/drivers/gpu/drm/panthor/panthor_device.c
index 9687c59de350..2877d47c6717 100644
--- a/drivers/gpu/drm/panthor/panthor_device.c
+++ b/drivers/gpu/drm/panthor/panthor_device.c
@@ -24,6 +24,7 @@
 #include "panthor_gpu.h"
 #include "panthor_hw.h"
 #include "panthor_mmu.h"
+#include "panthor_perf.h"
 #include "panthor_pwr.h"
 #include "panthor_sched.h"
 
@@ -287,6 +288,10 @@ int panthor_device_init(struct panthor_device *ptdev)
 	if (ret)
 		goto err_unplug_fw;
 
+	ret = panthor_perf_init(ptdev);
+	if (ret)
+		goto err_unplug_sched;
+
 	panthor_gem_init(ptdev);
 
 	/* Now that everything is initialized, we can enable the reset work. */
@@ -305,6 +310,8 @@ int panthor_device_init(struct panthor_device *ptdev)
 
 err_disable_autosuspend:
 	pm_runtime_dont_use_autosuspend(ptdev->base.dev);
+
+err_unplug_sched:
 	panthor_sched_unplug(ptdev);
 
 err_unplug_fw:
diff --git a/drivers/gpu/drm/panthor/panthor_device.h b/drivers/gpu/drm/panthor/panthor_device.h
index b55a3f9edd41..d93b182a3d9b 100644
--- a/drivers/gpu/drm/panthor/panthor_device.h
+++ b/drivers/gpu/drm/panthor/panthor_device.h
@@ -174,6 +174,9 @@ struct panthor_device {
 	/** @pwr: Power control management data. */
 	struct panthor_pwr *pwr;
 
+	/** @perf_info: Performance counter interface information. */
+	struct drm_panthor_perf_info perf_info;
+
 	/** @gpu: GPU management data. */
 	struct panthor_gpu *gpu;
 
diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c
index 8cdba0c1a14b..b6b55b0f2ff5 100644
--- a/drivers/gpu/drm/panthor/panthor_drv.c
+++ b/drivers/gpu/drm/panthor/panthor_drv.c
@@ -182,7 +182,8 @@ panthor_get_uobj_array(const struct drm_panthor_obj_array *in, u32 min_stride,
 		 PANTHOR_UOBJ_DECL(struct drm_panthor_queue_submit, syncs), \
 		 PANTHOR_UOBJ_DECL(struct drm_panthor_queue_create, ringbuf_size), \
 		 PANTHOR_UOBJ_DECL(struct drm_panthor_vm_bind_op, syncs), \
-		 PANTHOR_UOBJ_DECL(struct drm_panthor_bo_sync_op, size))
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_bo_sync_op, size), \
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_info, shader_blocks))
 
 /**
  * PANTHOR_UOBJ_SET() - Copy a kernel object to a user object.
@@ -959,6 +960,10 @@ static int panthor_ioctl_dev_query(struct drm_device *ddev, void *data, struct d
 			args->size = sizeof(ptdev->mmu_info);
 			return 0;
 
+		case DRM_PANTHOR_DEV_QUERY_PERF_INFO:
+			args->size = sizeof(ptdev->perf_info);
+			return 0;
+
 		default:
 			return -EINVAL;
 		}
@@ -992,6 +997,9 @@ static int panthor_ioctl_dev_query(struct drm_device *ddev, void *data, struct d
 	case DRM_PANTHOR_DEV_QUERY_MMU_INFO:
 		return PANTHOR_UOBJ_SET(args->pointer, args->size, ptdev->mmu_info);
 
+	case DRM_PANTHOR_DEV_QUERY_PERF_INFO:
+		return PANTHOR_UOBJ_SET(args->pointer, args->size, ptdev->perf_info);
+
 	default:
 		return -EINVAL;
 	}
diff --git a/drivers/gpu/drm/panthor/panthor_gpu.c b/drivers/gpu/drm/panthor/panthor_gpu.c
index a383b04f101e..726a70b74346 100644
--- a/drivers/gpu/drm/panthor/panthor_gpu.c
+++ b/drivers/gpu/drm/panthor/panthor_gpu.c
@@ -508,3 +508,8 @@ int panthor_gpu_coherency_init(struct panthor_device *ptdev)
 	drm_err(&ptdev->base, "Coherency not supported by the device");
 	return -ENOTSUPP;
 }
+
+u32 panthor_gpu_get_prfcnt_features(struct panthor_device *ptdev)
+{
+	return gpu_read(ptdev->gpu->iomem, GPU_PRFCNT_FEATURES);
+}
diff --git a/drivers/gpu/drm/panthor/panthor_gpu.h b/drivers/gpu/drm/panthor/panthor_gpu.h
index f615feb05609..46a6ca98cbb1 100644
--- a/drivers/gpu/drm/panthor/panthor_gpu.h
+++ b/drivers/gpu/drm/panthor/panthor_gpu.h
@@ -60,4 +60,6 @@ u64 panthor_gpu_get_cycle_count(struct panthor_device *ptdev);
 
 int panthor_gpu_coherency_init(struct panthor_device *ptdev);
 
+u32 panthor_gpu_get_prfcnt_features(struct panthor_device *ptdev);
+
 #endif
diff --git a/drivers/gpu/drm/panthor/panthor_perf.c b/drivers/gpu/drm/panthor/panthor_perf.c
new file mode 100644
index 000000000000..cbfe1ca5872d
--- /dev/null
+++ b/drivers/gpu/drm/panthor/panthor_perf.c
@@ -0,0 +1,87 @@
+// SPDX-License-Identifier: GPL-2.0 or MIT
+/* Copyright 2023 Collabora Ltd */
+/* Copyright 2025 Arm ltd. */
+
+#include <linux/bitops.h>
+#include <drm/panthor_drm.h>
+
+#include "panthor_device.h"
+#include "panthor_fw.h"
+#include "panthor_gpu.h"
+#include "panthor_gpu_regs.h"
+#include "panthor_perf.h"
+
+/**
+ * PANTHOR_PERF_COUNTERS_PER_BLOCK - On CSF architectures pre-11.x, the number of counters
+ * per block was hardcoded to be 64. Arch 11.0 onwards supports the PRFCNT_FEATURES GPU register,
+ * which indicates the same information.
+ */
+#define PANTHOR_PERF_COUNTERS_PER_BLOCK (64)
+
+/**
+ * PANTHOR_HW_COUNTER_SIZE - The size of a hardware counter in the FW ring buffer.
+ */
+#define PANTHOR_HW_COUNTER_SIZE (sizeof(u32))
+
+struct panthor_perf_counter_block {
+	struct drm_panthor_perf_block_header header;
+	u64 counters[];
+};
+
+static size_t get_annotated_block_size(size_t counters_per_block)
+{
+	return struct_size_t(struct panthor_perf_counter_block, counters, counters_per_block);
+}
+
+static size_t session_get_sample_size(const struct drm_panthor_perf_info *const info)
+{
+	const size_t block_size = get_annotated_block_size(info->counters_per_block);
+	const size_t block_nr = info->cshw_blocks + info->fw_blocks +
+		info->tiler_blocks + info->memsys_blocks + info->shader_blocks;
+
+	return info->sample_header_size + (block_size * block_nr);
+}
+
+static void panthor_perf_info_init(struct panthor_device *const ptdev)
+{
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(ptdev);
+	struct drm_panthor_perf_info *const perf_info = &ptdev->perf_info;
+
+	if (GPU_ARCH_MAJOR(ptdev->gpu_info.gpu_id) < 11) {
+		perf_info->counters_per_block = PANTHOR_PERF_COUNTERS_PER_BLOCK;
+	} else {
+		u32 features = panthor_gpu_get_prfcnt_features(ptdev);
+
+		perf_info->counters_per_block = PRFCNT_FEATURES_COUNTER_BLOCK_SIZE(features) /
+			PANTHOR_HW_COUNTER_SIZE;
+	}
+
+	perf_info->sample_header_size = sizeof(struct drm_panthor_perf_sample_header);
+	perf_info->block_header_size = sizeof(struct drm_panthor_perf_block_header);
+
+	if (GLB_PERFCNT_FW_SIZE(glb_iface->control->perfcnt_size))
+		perf_info->fw_blocks = 1;
+
+	perf_info->cshw_blocks = 1;
+	perf_info->tiler_blocks = 1;
+	perf_info->memsys_blocks = GPU_MEM_FEATURES_L2_SLICES(ptdev->gpu_info.mem_features);
+	perf_info->shader_blocks = hweight64(ptdev->gpu_info.shader_present);
+
+	perf_info->sample_size = session_get_sample_size(perf_info);
+}
+
+/**
+ * panthor_perf_init - Initialize the performance counter subsystem.
+ * @ptdev: Panthor device
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int panthor_perf_init(struct panthor_device *ptdev)
+{
+	if (!ptdev)
+		return -EINVAL;
+
+	panthor_perf_info_init(ptdev);
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/panthor/panthor_perf.h b/drivers/gpu/drm/panthor/panthor_perf.h
new file mode 100644
index 000000000000..3c32c24c164c
--- /dev/null
+++ b/drivers/gpu/drm/panthor/panthor_perf.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 or MIT */
+/* Copyright 2025 Collabora Ltd */
+/* Copyright 2025 Arm ltd. */
+
+#ifndef __PANTHOR_PERF_H__
+#define __PANTHOR_PERF_H__
+
+#include <linux/types.h>
+
+struct panthor_device;
+
+int panthor_perf_init(struct panthor_device *ptdev);
+
+#endif /* __PANTHOR_PERF_H__ */
+

-- 
2.43.0

[PATCH v7 4/8] drm/panthor: Add panthor perf initialization and termination

[Lukas Zapolskas]

Added the panthor_perf system initialization and unplug code for
subsystem-level handling in subsequent patches. Userspace session
handling is done at the pfile level, via a pool of sessions.

Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 drivers/gpu/drm/panthor/panthor_device.c |  2 +
 drivers/gpu/drm/panthor/panthor_device.h |  9 +++-
 drivers/gpu/drm/panthor/panthor_drv.c    |  9 ++++
 drivers/gpu/drm/panthor/panthor_perf.c   | 85 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/panthor/panthor_perf.h   |  6 +++
 5 files changed, 110 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/panthor/panthor_device.c b/drivers/gpu/drm/panthor/panthor_device.c
index 2877d47c6717..535383ef629f 100644
--- a/drivers/gpu/drm/panthor/panthor_device.c
+++ b/drivers/gpu/drm/panthor/panthor_device.c
@@ -95,6 +95,7 @@ void panthor_device_unplug(struct panthor_device *ptdev)
 	/* Now, try to cleanly shutdown the GPU before the device resources
 	 * get reclaimed.
 	 */
+	panthor_perf_unplug(ptdev);
 	panthor_sched_unplug(ptdev);
 	panthor_fw_unplug(ptdev);
 	panthor_mmu_unplug(ptdev);
@@ -310,6 +311,7 @@ int panthor_device_init(struct panthor_device *ptdev)
 
 err_disable_autosuspend:
 	pm_runtime_dont_use_autosuspend(ptdev->base.dev);
+	panthor_perf_unplug(ptdev);
 
 err_unplug_sched:
 	panthor_sched_unplug(ptdev);
diff --git a/drivers/gpu/drm/panthor/panthor_device.h b/drivers/gpu/drm/panthor/panthor_device.h
index d93b182a3d9b..085254a60fc4 100644
--- a/drivers/gpu/drm/panthor/panthor_device.h
+++ b/drivers/gpu/drm/panthor/panthor_device.h
@@ -29,10 +29,11 @@ struct panthor_hw;
 struct panthor_job;
 struct panthor_mmu;
 struct panthor_fw;
-struct panthor_perfcnt;
 struct panthor_pwr;
+struct panthor_perf;
 struct panthor_vm;
 struct panthor_vm_pool;
+struct panthor_perf_pool;
 
 /**
  * struct panthor_soc_data - Panthor SoC Data
@@ -323,6 +324,9 @@ struct panthor_device {
 		atomic_t recovery_needed;
 	} pm;
 
+	/** @perf: Performance counter management data. */
+	struct panthor_perf *perf;
+
 	/** @profile_mask: User-set profiling flags for job accounting. */
 	u32 profile_mask;
 
@@ -377,6 +381,9 @@ struct panthor_file {
 	/** @groups: Scheduling group pool attached to this file. */
 	struct panthor_group_pool *groups;
 
+	/** @sessions: Perf session pool attached to this file. */
+	struct panthor_perf_pool *sessions;
+
 	/** @stats: cycle and timestamp measures for job execution. */
 	struct panthor_gpu_usage stats;
 };
diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c
index b6b55b0f2ff5..3e62142d7ebd 100644
--- a/drivers/gpu/drm/panthor/panthor_drv.c
+++ b/drivers/gpu/drm/panthor/panthor_drv.c
@@ -37,6 +37,7 @@
 #include "panthor_gpu.h"
 #include "panthor_heap.h"
 #include "panthor_mmu.h"
+#include "panthor_perf.h"
 #include "panthor_sched.h"
 
 /**
@@ -1635,9 +1636,16 @@ panthor_open(struct drm_device *ddev, struct drm_file *file)
 	if (ret)
 		goto err_destroy_vm_pool;
 
+	ret = panthor_perf_pool_create(pfile);
+	if (ret)
+		goto err_destroy_group_pool;
+
 	file->driver_priv = pfile;
 	return 0;
 
+err_destroy_group_pool:
+	panthor_group_pool_destroy(pfile);
+
 err_destroy_vm_pool:
 	panthor_vm_pool_destroy(pfile);
 
@@ -1651,6 +1659,7 @@ panthor_postclose(struct drm_device *ddev, struct drm_file *file)
 {
 	struct panthor_file *pfile = file->driver_priv;
 
+	panthor_perf_pool_destroy(pfile);
 	panthor_group_pool_destroy(pfile);
 	panthor_vm_pool_destroy(pfile);
 
diff --git a/drivers/gpu/drm/panthor/panthor_perf.c b/drivers/gpu/drm/panthor/panthor_perf.c
index cbfe1ca5872d..ba466e22c50d 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.c
+++ b/drivers/gpu/drm/panthor/panthor_perf.c
@@ -3,7 +3,9 @@
 /* Copyright 2025 Arm ltd. */
 
 #include <linux/bitops.h>
+#include <drm/drm_managed.h>
 #include <drm/panthor_drm.h>
+#include <drm/drm_print.h>
 
 #include "panthor_device.h"
 #include "panthor_fw.h"
@@ -23,11 +25,26 @@
  */
 #define PANTHOR_HW_COUNTER_SIZE (sizeof(u32))
 
+#define MAX_SESSIONS_PER_FILE 1
+
 struct panthor_perf_counter_block {
 	struct drm_panthor_perf_block_header header;
 	u64 counters[];
 };
 
+struct panthor_perf {
+	/** @info: Pointer to the previously initialized perf info. */
+	struct drm_panthor_perf_info *info;
+};
+
+struct panthor_perf_pool {
+	/** @session_range: The number of sessions supported at a time. */
+	struct xa_limit session_range;
+
+	/** @xa: Xarray used to manage session handles. */
+	struct xarray xa;
+};
+
 static size_t get_annotated_block_size(size_t counters_per_block)
 {
 	return struct_size_t(struct panthor_perf_counter_block, counters, counters_per_block);
@@ -74,14 +91,82 @@ static void panthor_perf_info_init(struct panthor_device *const ptdev)
  * panthor_perf_init - Initialize the performance counter subsystem.
  * @ptdev: Panthor device
  *
+ * The performance counters require the FW interface to be available to setup the
+ * sampling ringbuffers, so this must be called only after FW is initialized.
+ *
  * Return: 0 on success, negative error code on failure.
  */
 int panthor_perf_init(struct panthor_device *ptdev)
 {
+	struct panthor_perf *perf;
+	int ret = 0;
+
 	if (!ptdev)
 		return -EINVAL;
 
 	panthor_perf_info_init(ptdev);
 
+	perf = drmm_kzalloc(&ptdev->base, sizeof(*perf), GFP_KERNEL);
+	if (!perf)
+		return -ENOMEM;
+
+	drm_info(&ptdev->base, "Performance counter subsystem initialized");
+
+	perf->info = &ptdev->perf_info;
+	ptdev->perf = perf;
+
+	return ret;
+}
+
+/**
+ * panthor_perf_unplug - Terminate the performance counter subsystem.
+ * @ptdev: Panthor device.
+ *
+ * Release the performance counter hardware resources. The control structure is
+ * released with the DRM device.
+ */
+void panthor_perf_unplug(struct panthor_device *ptdev)
+{
+	struct panthor_perf *perf = ptdev->perf;
+
+	if (!perf)
+		return;
+
+	ptdev->perf = NULL;
+}
+
+int panthor_perf_pool_create(struct panthor_file *pfile)
+{
+	struct panthor_perf_pool *pool;
+
+	pool = kzalloc_obj(*pool);
+	if (!pool)
+		return -ENOMEM;
+
+	xa_init_flags(&pool->xa, XA_FLAGS_ALLOC);
+
+	pool->session_range.min = 0;
+	pool->session_range.max = MAX_SESSIONS_PER_FILE - 1;
+
+	pfile->sessions = pool;
+
 	return 0;
 }
+
+void panthor_perf_pool_destroy(struct panthor_file *pfile)
+{
+	struct panthor_perf_pool *pool = pfile->sessions;
+
+	if (IS_ERR_OR_NULL(pool))
+		return;
+
+	if (!xa_empty(&pool->xa)) {
+		drm_err(&pfile->ptdev->base,
+			"Performance counter sessions active when unplugging the driver!");
+	}
+
+	xa_destroy(&pool->xa);
+	kfree(pool);
+
+	pfile->sessions = NULL;
+}
diff --git a/drivers/gpu/drm/panthor/panthor_perf.h b/drivers/gpu/drm/panthor/panthor_perf.h
index 3c32c24c164c..ad01fc88f18f 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.h
+++ b/drivers/gpu/drm/panthor/panthor_perf.h
@@ -8,8 +8,14 @@
 #include <linux/types.h>
 
 struct panthor_device;
+struct panthor_file;
+struct panthor_perf_pool;
+
+int panthor_perf_pool_create(struct panthor_file *pfile);
+void panthor_perf_pool_destroy(struct panthor_file *pfile);
 
 int panthor_perf_init(struct panthor_device *ptdev);
+void panthor_perf_unplug(struct panthor_device *ptdev);
 
 #endif /* __PANTHOR_PERF_H__ */
 

-- 
2.43.0

[PATCH v7 5/8] drm/panthor: Introduce sampling sessions to handle userspace clients

[Lukas Zapolskas]

To allow for combining the requests from multiple userspace clients,
an intermediary layer between the HW/FW interfaces and userspace is
created, containing the information for the counter requests and
tracking of insert and extract indices. Each session starts inactive
and must be explicitly activated via PERF_SESSION_START and
explicitly stopped via PERF_SESSION_STOP. Userspace identifies a
single client with its session ID and the panthor file it is
associated with.

The SAMPLE and STOP commands both produce a single sample when called,
and these samples can be disambiguated via the opaque user data field
passed in the PERF_SESSION uAPI. If this functionality is not desired,
these fields can be kept as zero, as the kernel copies this value into
the corresponding sample without attempting to interpret it.

Currently, only manual sampling sessions are supported, providing
samples when userspace calls PERF_CONTROL_SAMPLE, and only a single
session is allowed at a time. Multiple sessions and periodic sampling
will be enabled in following patches.

No protection is provided against the 32-bit hardware counter
overflows, so for the moment it is up to userspace to ensure that
the counters are sampled at a reasonable frequency.

Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 drivers/gpu/drm/panthor/panthor_perf.c | 657 ++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/panthor/panthor_perf.h |  11 +
 2 files changed, 664 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/panthor/panthor_perf.c b/drivers/gpu/drm/panthor/panthor_perf.c
index ba466e22c50d..76c346981150 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.c
+++ b/drivers/gpu/drm/panthor/panthor_perf.c
@@ -2,10 +2,12 @@
 /* Copyright 2023 Collabora Ltd */
 /* Copyright 2025 Arm ltd. */
 
-#include <linux/bitops.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
 #include <drm/drm_managed.h>
 #include <drm/panthor_drm.h>
 #include <drm/drm_print.h>
+#include <linux/bitops.h>
 
 #include "panthor_device.h"
 #include "panthor_fw.h"
@@ -27,14 +29,130 @@
 
 #define MAX_SESSIONS_PER_FILE 1
 
+/**
+ * PANTHOR_PERF_EM_BITS - Number of bits in a user-facing enable mask. This must correspond
+ *                        to the maximum number of counters available for selection on the newest
+ *                        Mali GPUs (128 as of the Mali-Gx15).
+ */
+#define PANTHOR_PERF_EM_BITS (BITS_PER_TYPE(u64) * 2)
+
+enum panthor_perf_session_state {
+	/** @PANTHOR_PERF_SESSION_STOPPED: The session cannot sample. It starts out stopped. */
+	PANTHOR_PERF_SESSION_STOPPED = 0,
+
+	/** @PANTHOR_PERF_SESSION_STARTING: The session has not yet been started and is in the
+	 *				    process of handling the clearing sample.
+	 */
+	PANTHOR_PERF_SESSION_STARTING,
+
+	/** @PANTHOR_PERF_SESSION_ACTIVE: The session is active and can be used for sampling. */
+	PANTHOR_PERF_SESSION_ACTIVE,
+
+	/** @PANTHOR_PERF_SESSION_STOPPING: The session is currently stopping. After emitting the
+	 *				    next sample, it will transition to inactive.
+	 */
+	PANTHOR_PERF_SESSION_STOPPING,
+};
+
+struct panthor_perf_enable_masks {
+	/**
+	 * @mask: Array of bitmasks indicating the counters userspace requested, where
+	 *        one bit represents a single counter. Used to build the firmware configuration
+	 *        and ensure that userspace clients obtain only the counters they requested.
+	 */
+	unsigned long mask[DRM_PANTHOR_PERF_BLOCK_MAX][BITS_TO_LONGS(PANTHOR_PERF_EM_BITS)];
+};
+
 struct panthor_perf_counter_block {
 	struct drm_panthor_perf_block_header header;
 	u64 counters[];
 };
 
+struct panthor_perf_bo {
+	struct drm_gem_object *obj;
+	void *vaddr;
+};
+
+struct panthor_perf_session {
+	/** @state_lock: Used to protect session state information and overflow bit. */
+	struct mutex state_lock;
+
+	/** @state: The current state of the session. */
+	enum panthor_perf_session_state state;
+
+	/** @overflow: The session is affected by a FW ring buffer overflow. */
+	bool overflow;
+
+	/**  @insert_idx: The insert index in the user-visible buffer for the current session. */
+	u64 insert_idx;
+
+	/**
+	 * @accum_idx: Insert index of the populated slot. When this matches @insert_idx,
+	 *             accumulate into that slot; otherwise populate a new slot.
+	 */
+	u64 accum_idx;
+
+	/** @sample_start_ns: Sample request time, obtained from a monotonic raw clock. */
+	u64 sample_start_ns;
+
+	/**
+	 * @user_data: Opaque handle passed in when starting a session, requesting a sample (for
+	 *             manual sampling sessions only) and when stopping a session. This handle
+	 *             allows the disambiguation of a sample in the ringbuffer.
+	 */
+	u64 user_data;
+
+	/**
+	 * @eventfd: Event file descriptor context used to signal userspace of a new sample
+	 *           being emitted.
+	 */
+	struct eventfd_ctx *eventfd;
+
+	/**
+	 * @enabled_counters: This session's requested counters. Note that these cannot change
+	 *                    for the lifetime of the session.
+	 */
+	struct panthor_perf_enable_masks *enabled_counters;
+
+	/** @ringbuf_slots: Slots in the user-facing ringbuffer. */
+	size_t ringbuf_slots;
+
+	/** @ring_buf: BO and mapping for the userspace ringbuffer. */
+	struct panthor_perf_bo ring_buf;
+
+	/**
+	 * @control_buf: BO and mapping for the insert and extract indices.
+	 */
+	struct panthor_perf_bo control_buf;
+
+	/** @control: The mapped insert and extract indices. */
+	struct drm_panthor_perf_ringbuf_control *control;
+
+	/** @samples: The mapping of the @ring_buf into the kernel's VA space. */
+	u8 *samples;
+
+	/**
+	 * @sessions: The list node tracking all registered sessions.
+	 */
+	struct list_head sessions;
+
+	/**
+	 * @ref: Session reference count. The sample delivery to userspace is asynchronous, meaning
+	 *       the lifetime of the session must extend at least until the sample is exposed to
+	 *       userspace.
+	 */
+	struct kref ref;
+};
+
 struct panthor_perf {
 	/** @info: Pointer to the previously initialized perf info. */
 	struct drm_panthor_perf_info *info;
+
+	/**
+	 * @op_lock: Serializes session operations. Acquire before looking up session
+	 *           handles and before a session's state_lock.
+	 */
+	struct mutex op_lock;
 };
 
 struct panthor_perf_pool {
@@ -43,6 +161,9 @@ struct panthor_perf_pool {
 
 	/** @xa: Xarray used to manage session handles. */
 	struct xarray xa;
+
+	/** @next: Next handle to allocate. */
+	u32 next;
 };
 
 static size_t get_annotated_block_size(size_t counters_per_block)
@@ -87,6 +208,163 @@ static void panthor_perf_info_init(struct panthor_device *const ptdev)
 	perf_info->sample_size = session_get_sample_size(perf_info);
 }
 
+static struct panthor_perf_enable_masks *
+panthor_perf_create_em(struct drm_panthor_perf_session_create *const create_args)
+{
+	struct panthor_perf_enable_masks *const em = kmalloc(sizeof(*em), GFP_KERNEL);
+
+	if (IS_ERR_OR_NULL(em))
+		return em;
+
+	bitmap_from_arr64(em->mask[DRM_PANTHOR_PERF_BLOCK_FW],
+			  create_args->fw_enable_mask, PANTHOR_PERF_EM_BITS);
+	bitmap_from_arr64(em->mask[DRM_PANTHOR_PERF_BLOCK_CSHW],
+			  create_args->cshw_enable_mask, PANTHOR_PERF_EM_BITS);
+	bitmap_from_arr64(em->mask[DRM_PANTHOR_PERF_BLOCK_TILER],
+			  create_args->tiler_enable_mask, PANTHOR_PERF_EM_BITS);
+	bitmap_from_arr64(em->mask[DRM_PANTHOR_PERF_BLOCK_MEMSYS],
+			  create_args->memsys_enable_mask, PANTHOR_PERF_EM_BITS);
+	bitmap_from_arr64(em->mask[DRM_PANTHOR_PERF_BLOCK_SHADER],
+			  create_args->shader_enable_mask, PANTHOR_PERF_EM_BITS);
+
+	return em;
+}
+
+static struct panthor_perf_session *
+session_get(struct panthor_perf_session *session)
+{
+	if (session)
+		kref_get(&session->ref);
+
+	return session;
+}
+
+static int
+panthor_perf_bo_get(struct drm_file *file, u32 handle, size_t expected_size,
+		    struct panthor_perf_bo *bo)
+{
+	struct drm_device *ddev = file->minor->dev;
+	struct drm_gem_object *obj;
+	struct iosys_map map;
+	int ret;
+
+	if (WARN_ON_ONCE(!bo))
+		return -EINVAL;
+
+	obj = drm_gem_object_lookup(file, handle);
+	if (!obj) {
+		drm_err(ddev, "Could not find handle %d!\n", handle);
+		return -EINVAL;
+	}
+
+	if (expected_size && obj->size != PFN_ALIGN(expected_size)) {
+		drm_err(ddev,
+			"Incorrect BO size from userspace: user %zu vs kernel %lu\n",
+			obj->size, PFN_ALIGN(expected_size));
+		ret = -ENOMEM;
+		goto err_put;
+	}
+
+	ret = drm_gem_vmap(obj, &map);
+	if (ret)
+		goto err_put;
+
+	bo->obj = obj;
+	bo->vaddr = map.vaddr;
+
+	return 0;
+
+err_put:
+	drm_gem_object_put(obj);
+	return ret;
+}
+
+static void panthor_perf_bo_put(struct panthor_perf_bo *bo)
+{
+	if (!bo->obj)
+		return;
+
+	if (bo->vaddr) {
+		struct iosys_map map = IOSYS_MAP_INIT_VADDR(bo->vaddr);
+
+		drm_gem_vunmap(bo->obj, &map);
+	}
+
+	drm_gem_object_put(bo->obj);
+
+	memset(bo, 0, sizeof(*bo));
+}
+
+static void session_free(struct kref *ref)
+{
+	struct panthor_perf_session *session = container_of(ref, typeof(*session), ref);
+
+	panthor_perf_bo_put(&session->ring_buf);
+	panthor_perf_bo_put(&session->control_buf);
+
+	kfree(session->enabled_counters);
+
+	eventfd_ctx_put(session->eventfd);
+
+	kfree(session);
+}
+
+static void session_put(struct panthor_perf_session *session)
+{
+	kref_put(&session->ref, session_free);
+}
+
+/**
+ * session_from_handle - Find a session associated with the pool and session_handle.
+ *
+ * @pool: Per-file pool.
+ * @handle: Session handle.
+ *
+ * The reference count of a valid session is increased to ensure it does not disappear
+ * in the window between the XA lock being dropped and the internal session functions
+ * being called.
+ *
+ * Return: valid session pointer or NULL.
+ */
+static struct panthor_perf_session *
+session_from_handle(struct panthor_perf_pool *pool, unsigned long handle)
+{
+	struct panthor_perf_session *session;
+
+	xa_lock(&pool->xa);
+	session = session_get(xa_load(&pool->xa, handle));
+	xa_unlock(&pool->xa);
+
+	return session;
+}
+
+#define DEFINE_SESSION_STATE_CHECK(name, _state)			     \
+	static bool session_is_## name(struct panthor_perf_session *session) \
+	{								     \
+		lockdep_assert_held(&session->state_lock);		     \
+									     \
+		return session->state == (_state);			    \
+	}
+
+DEFINE_SESSION_STATE_CHECK(active, PANTHOR_PERF_SESSION_ACTIVE);
+DEFINE_SESSION_STATE_CHECK(starting, PANTHOR_PERF_SESSION_STARTING);
+DEFINE_SESSION_STATE_CHECK(stopping, PANTHOR_PERF_SESSION_STOPPING);
+
+static bool session_has_overflow(struct panthor_perf_session *session)
+{
+	lockdep_assert_held(&session->state_lock);
+
+	return session->overflow;
+}
+
+static void session_set_state(struct panthor_perf_session *session,
+			      enum panthor_perf_session_state state)
+{
+	lockdep_assert_held(&session->state_lock);
+
+	session->state = state;
+}
+
 /**
  * panthor_perf_init - Initialize the performance counter subsystem.
  * @ptdev: Panthor device
@@ -110,6 +388,10 @@ int panthor_perf_init(struct panthor_device *ptdev)
 	if (!perf)
 		return -ENOMEM;
 
+	ret = drmm_mutex_init(&ptdev->base, &perf->op_lock);
+	if (ret)
+		return ret;
+
 	drm_info(&ptdev->base, "Performance counter subsystem initialized");
 
 	perf->info = &ptdev->perf_info;
@@ -118,6 +400,371 @@ int panthor_perf_init(struct panthor_device *ptdev)
 	return ret;
 }
 
+/**
+ * panthor_perf_session_create - Create a user-visible session.
+ *
+ * @file: DRM file associated with the request.
+ * @create_args: Creation arguments passed in via ioctl.
+ * @sid: Session identifier output.
+ *
+ * Creates a new session associated with the session ID returned. When initialized, the
+ * session must explicitly request sampling to start with a successive call to PERF_SESSION_START.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int panthor_perf_session_create(struct drm_file *file,
+				struct drm_panthor_perf_session_create *create_args,
+				unsigned long *sid)
+{
+	struct panthor_file *pfile = file->driver_priv;
+	struct panthor_device *ptdev = pfile->ptdev;
+	struct panthor_perf_bo ringbuffer = {}, control = {};
+	const size_t slots = create_args->sample_slots;
+	struct panthor_perf_pool *pool = pfile->sessions;
+	struct panthor_perf_session *session;
+	struct panthor_perf_enable_masks *em;
+	size_t user_sample_size;
+	u32 session_id;
+	int ret;
+
+	if (!ptdev->perf)
+		return -EOPNOTSUPP;
+
+	if (!pool)
+		return -EINVAL;
+
+	if (!perfmon_capable())
+		return -EACCES;
+
+	session = kzalloc_obj(*session);
+	if (ZERO_OR_NULL_PTR(session))
+		return -ENOMEM;
+
+	if (!is_power_of_2(slots)) {
+		drm_err(&ptdev->base,
+			"Number of slots is not power-of-2 aligned\n");
+		ret = -EINVAL;
+		goto cleanup_session;
+	}
+
+	user_sample_size = ptdev->perf_info.sample_size * slots;
+
+	ret = panthor_perf_bo_get(file, create_args->ringbuf_handle,
+				  user_sample_size, &ringbuffer);
+	if (ret)
+		goto cleanup_session;
+
+	ret = panthor_perf_bo_get(file, create_args->control_handle, 0, &control);
+	if (ret)
+		goto cleanup_ringbuf;
+
+	session->eventfd = eventfd_ctx_fdget(create_args->fd);
+	if (IS_ERR(session->eventfd)) {
+		drm_err(&ptdev->base, "Invalid eventfd %d!\n", create_args->fd);
+		ret = PTR_ERR_OR_ZERO(session->eventfd) ?: -EINVAL;
+		goto cleanup_control;
+	}
+
+	em = panthor_perf_create_em(create_args);
+	if (IS_ERR_OR_NULL(em)) {
+		ret = -ENOMEM;
+		goto cleanup_eventfd;
+	}
+
+	kref_init(&session->ref);
+	INIT_LIST_HEAD(&session->sessions);
+	mutex_init(&session->state_lock);
+	session->control = control.vaddr;
+	*session->control = (struct drm_panthor_perf_ringbuf_control) { 0 };
+	session->samples = ringbuffer.vaddr;
+	session->enabled_counters = em;
+	session->ring_buf = ringbuffer;
+	session->ringbuf_slots = slots;
+	session->control_buf = control;
+	session->accum_idx = U32_MAX;
+
+	ret = xa_alloc_cyclic(&pfile->sessions->xa, &session_id, session,
+			      pfile->sessions->session_range,
+			      &pfile->sessions->next, GFP_KERNEL);
+	if (ret < 0) {
+		drm_err(&ptdev->base, "File session limit exceeded.\n");
+		ret = -EBUSY;
+		goto cleanup_obj;
+	}
+
+	*sid = session_id;
+	return 0;
+
+cleanup_obj:
+	session_put(session);
+	return ret;
+
+cleanup_eventfd:
+	eventfd_ctx_put(session->eventfd);
+
+cleanup_control:
+	panthor_perf_bo_put(&control);
+
+cleanup_ringbuf:
+	panthor_perf_bo_put(&ringbuffer);
+
+cleanup_session:
+	kfree(session);
+
+	return ret;
+}
+
+static int session_try_stop(struct panthor_perf_session *session, u64 user_data)
+{
+	guard(mutex)(&session->state_lock);
+
+	if (!session_is_active(session))
+		return 0;
+
+	if (session_is_starting(session))
+		return -EBUSY;
+
+	if (session_is_stopping(session))
+		return 1;
+
+	session_set_state(session, PANTHOR_PERF_SESSION_STOPPING);
+	session->user_data = user_data;
+
+	return 1;
+}
+
+static int session_stop(struct panthor_perf *perf, struct panthor_perf_session *session,
+			u64 user_data)
+{
+	int ret;
+
+	ret = session_try_stop(session, user_data);
+	if (ret <= 0)
+		return ret;
+
+	/* TODO Calls to the FW interface will go here in later patches. */
+	return 0;
+}
+
+static int session_try_start(struct panthor_perf_session *session, u64 user_data)
+{
+	guard(mutex)(&session->state_lock);
+
+	if (session_is_active(session))
+		return 0;
+
+	if (session_is_stopping(session))
+		return -EBUSY;
+
+	if (session_is_active(session) || session_is_starting(session))
+		return 0;
+
+	session_set_state(session, PANTHOR_PERF_SESSION_STARTING);
+	session->user_data = user_data;
+
+	return 1;
+}
+
+static int session_start(struct panthor_perf *perf, struct panthor_perf_session *session,
+			 u64 user_data)
+{
+	int ret;
+
+	ret = session_try_start(session, user_data);
+	if (ret <= 0)
+		return ret;
+
+	/* TODO Calls to the FW interface will go here in later patches. */
+	return 0;
+}
+
+static int session_try_sample(struct panthor_perf_session *session, u64 user_data)
+{
+	guard(mutex)(&session->state_lock);
+
+	if (!session_is_active(session))
+		return -EINVAL;
+
+	if (session_is_stopping(session))
+		return -EBUSY;
+
+	if (session_has_overflow(session))
+		return -EACCES;
+
+	session->user_data = user_data;
+
+	return 1;
+}
+
+static int session_sample(struct panthor_perf *perf, struct panthor_perf_session *session,
+			  u64 user_data)
+{
+
+	int ret;
+
+	ret = session_try_sample(session, user_data);
+	if (ret <= 0)
+		return ret;
+
+	/* TODO Calls to the FW interface will go here in later patches. */
+	return 0;
+}
+
+static int session_try_destroy(struct panthor_perf_session *session)
+{
+	guard(mutex)(&session->state_lock);
+
+	if (session_is_active(session))
+		return -EINVAL;
+
+	if (session_is_starting(session) || session_is_stopping(session))
+		return -EBUSY;
+
+	return 0;
+}
+
+/**
+ * panthor_perf_session_destroy - Destroy the session associated with @handle.
+ * @pfile: Open panthor file.
+ * @handle: Session identifier.
+ *
+ * Only stopped sessions can be destroyed. Other session states return an error.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int panthor_perf_session_destroy(struct panthor_file *pfile, u32 handle)
+{
+	struct panthor_perf_session *session;
+	struct panthor_perf *perf = pfile->ptdev->perf;
+	int ret;
+
+	if (!perf)
+		return -EOPNOTSUPP;
+
+	scoped_guard(mutex, &perf->op_lock) {
+		session = xa_load(&pfile->sessions->xa, handle);
+		if (!session)
+			return -EINVAL;
+
+		ret = session_try_destroy(session);
+		if (ret)
+			return ret;
+
+		xa_erase(&pfile->sessions->xa, handle);
+	}
+
+	session_put(session);
+
+	return ret;
+}
+
+/**
+ * panthor_perf_session_start - Start sampling on a stopped session.
+ * @pfile: Open panthor file.
+ * @handle: Session identifier for the desired session.
+ * @user_data: Opaque value passed in from userspace.
+ *
+ * A session counts as stopped when it is created or when it is explicitly stopped after being
+ * started. Starting an active session is treated as a no-op.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int panthor_perf_session_start(struct panthor_file *pfile, u32 handle, u64 user_data)
+{
+	struct panthor_perf_session *session;
+	struct panthor_perf *perf = pfile->ptdev->perf;
+	int err;
+
+	if (!perf)
+		return -EOPNOTSUPP;
+
+	guard(mutex)(&perf->op_lock);
+
+	session = session_from_handle(pfile->sessions, handle);
+	if (IS_ERR_OR_NULL(session))
+		return IS_ERR(session) ? PTR_ERR(session) : -EINVAL;
+
+	err = session_start(perf, session, user_data);
+
+	session_put(session);
+
+	return err;
+}
+
+/**
+ * panthor_perf_session_stop - Stop sampling on an active session.
+ * @pfile: Open panthor file.
+ * @handle: Session identifier for the desired session.
+ * @user_data: An opaque value passed in from userspace.
+ *
+ * A session counts as active when it has been explicitly started via the PERF_SESSION_START
+ * ioctl. Stopping a stopped session is treated as a no-op.
+ *
+ * To ensure data is not lost when sampling is stopping, there must always be at least one slot
+ * available for the final automatic sample, and the stop command will be rejected if there is not.
+ *
+ * The @user_data will always be associated with the final sample.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int panthor_perf_session_stop(struct panthor_file *pfile, unsigned int handle, u64 user_data)
+{
+	struct panthor_perf_session *session;
+	struct panthor_perf *perf = pfile->ptdev->perf;
+	int err;
+
+	if (!perf)
+		return -EOPNOTSUPP;
+
+	guard(mutex)(&perf->op_lock);
+
+	session = session_from_handle(pfile->sessions, handle);
+	if (IS_ERR_OR_NULL(session))
+		return IS_ERR(session) ? PTR_ERR(session) : -EINVAL;
+
+	err = session_stop(perf, session, user_data);
+
+	session_put(session);
+
+	return err;
+}
+
+/**
+ * panthor_perf_session_sample - Request a sample on a manual sampling session.
+ * @pfile: Open panthor file.
+ * @handle: Session identifier for the desired session.
+ * @user_data: An opaque value passed in from userspace.
+ *
+ * Only an active manual sampler is permitted to request samples directly. Failing to meet either
+ * of these conditions will cause the sampling request to be rejected. Requesting a manual sample
+ * with a full ringbuffer will see the request being rejected.
+ *
+ * The @user_data will always be unambiguously associated one-to-one with the resultant sample.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int panthor_perf_session_sample(struct panthor_file *pfile, u32 handle, u64 user_data)
+{
+	struct panthor_perf_session *session;
+	struct panthor_perf *perf = pfile->ptdev->perf;
+	int err;
+
+	if (!perf)
+		return -EOPNOTSUPP;
+
+	guard(mutex)(&perf->op_lock);
+
+	session = session_from_handle(pfile->sessions, handle);
+	if (IS_ERR_OR_NULL(session))
+		return IS_ERR(session) ? PTR_ERR(session) : -EINVAL;
+
+	err = session_sample(perf, session, user_data);
+
+	session_put(session);
+
+	return err;
+}
+
 /**
  * panthor_perf_unplug - Terminate the performance counter subsystem.
  * @ptdev: Panthor device.
@@ -156,13 +803,15 @@ int panthor_perf_pool_create(struct panthor_file *pfile)
 void panthor_perf_pool_destroy(struct panthor_file *pfile)
 {
 	struct panthor_perf_pool *pool = pfile->sessions;
+	unsigned long handle;
+	struct panthor_perf_session *session;
 
 	if (IS_ERR_OR_NULL(pool))
 		return;
 
-	if (!xa_empty(&pool->xa)) {
-		drm_err(&pfile->ptdev->base,
-			"Performance counter sessions active when unplugging the driver!");
+	xa_for_each(&pool->xa, handle, session) {
+		xa_erase(&pool->xa, handle);
+		session_put(session);
 	}
 
 	xa_destroy(&pool->xa);
diff --git a/drivers/gpu/drm/panthor/panthor_perf.h b/drivers/gpu/drm/panthor/panthor_perf.h
index ad01fc88f18f..641a88edd75a 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.h
+++ b/drivers/gpu/drm/panthor/panthor_perf.h
@@ -7,8 +7,11 @@
 
 #include <linux/types.h>
 
+struct drm_file;
+struct drm_panthor_perf_session_create;
 struct panthor_device;
 struct panthor_file;
+struct panthor_perf;
 struct panthor_perf_pool;
 
 int panthor_perf_pool_create(struct panthor_file *pfile);
@@ -17,5 +20,13 @@ void panthor_perf_pool_destroy(struct panthor_file *pfile);
 int panthor_perf_init(struct panthor_device *ptdev);
 void panthor_perf_unplug(struct panthor_device *ptdev);
 
+int panthor_perf_session_create(struct drm_file *file,
+				struct drm_panthor_perf_session_create *setup_args,
+				unsigned long *sid);
+int panthor_perf_session_destroy(struct panthor_file *pfile, unsigned int handle);
+int panthor_perf_session_start(struct panthor_file *pfile, u32 handle, u64 user_data);
+int panthor_perf_session_stop(struct panthor_file *pfile, unsigned int sid, u64 user_data);
+int panthor_perf_session_sample(struct panthor_file *pfile, unsigned int handle, u64 user_data);
+
 #endif /* __PANTHOR_PERF_H__ */
 

-- 
2.43.0

[PATCH v7 6/8] drm/panthor: Implement sample handling

[Lukas Zapolskas]

From: Adrián Larumbe <adrian.larumbe@collabora.com>

panthor_perf aggregates counter and set requests coming from userspace
and mediates interactions with the FW interface, to ensure that user
sessions cannot override the global configuration.

From the top-level interface, the two types of samples are supported:
clearing samples and regular samples. Clearing samples are
a special sample type that allow for the creation of a sampling
baseline, to ensure that a session does not obtain counter data from
before its creation.

Upon receipt of a relevant interrupt, corresponding to one of the
three relevant bits of the GLB_ACK register, panthor_perf takes any
samples that occurred, and, based on the insert and extract indices,
accumulates them to an internal storage buffer after zero-extending
the counters from the 32-bit counters emitted by the hardware to
64-bit counters for internal accumulation.

When the performance counters are enabled, the FW ensures no counter
data is lost when entering and leaving non-counting regions by
producing automatic samples that do not correspond to a
GLB_REQ.PRFCNT_SAMPLE request. Such regions may be per hardware unit,
such as when a shader core powers down, or global. Most of these
events do not directly correspond to session sample requests, so any
intermediary counter data must be stored into a temporary
accumulation buffer.

If there are sessions waiting for a sample, this accumulated buffer
will be taken, and emitted for each waiting client. During this phase,
information like the timestamps of sample request and sample emission,
type of the counter block and block index annotations are added to the
sample header and block headers. If no sessions are waiting for
a sample, this accumulation buffer is kept until the next time a
sample is requested.

Special handling is needed for the PRFCNT_OVERFLOW interrupt, which is
an indication that the internal sample handling rate was insufficient.

A semi-static table maintains the mappings between the offsets from
the base of a FW sample to the corresponding block type. Since the
counter blocks themselves do not contain this information, this needs to
be done in the kernel.

This table also handles the sparseness of the shader cores,
wherein the physical core mask contains holes, but the memory allocated
for it is done based on the position of the most significant bit. In
cases with highly sparse core masks, this means that a lot of shader
counter blocks are empty, and must be skipped.

The number of ring buffer slots is configurable through module param
to allow for a lower memory footprint on memory constrained systems.

Signed-off-by: Adrián Larumbe <adrian.larumbe@collabora.com>
Co-developed-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 drivers/gpu/drm/panthor/panthor_fw.c   |    5 +
 drivers/gpu/drm/panthor/panthor_perf.c | 1075 +++++++++++++++++++++++++++++++-
 drivers/gpu/drm/panthor/panthor_perf.h |    2 +
 3 files changed, 1069 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/panthor/panthor_fw.c b/drivers/gpu/drm/panthor/panthor_fw.c
index 4f1fab66a13b..7d4b253bfe0c 100644
--- a/drivers/gpu/drm/panthor/panthor_fw.c
+++ b/drivers/gpu/drm/panthor/panthor_fw.c
@@ -25,6 +25,7 @@
 #include "panthor_gpu.h"
 #include "panthor_hw.h"
 #include "panthor_mmu.h"
+#include "panthor_perf.h"
 #include "panthor_sched.h"
 #include "panthor_trace.h"
 
@@ -1072,9 +1073,12 @@ static void panthor_fw_init_global_iface(struct panthor_device *ptdev)
 
 	/* Enable interrupts we care about. */
 	glb_iface->input->ack_irq_mask = GLB_CFG_ALLOC_EN |
+					 GLB_PERFCNT_SAMPLE |
 					 GLB_PING |
 					 GLB_CFG_PROGRESS_TIMER |
 					 GLB_CFG_POWEROFF_TIMER |
+					 GLB_PERFCNT_THRESHOLD |
+					 GLB_PERFCNT_OVERFLOW |
 					 GLB_IDLE_EN |
 					 GLB_IDLE;
 
@@ -1122,6 +1126,7 @@ static void panthor_job_irq_handler(struct panthor_irq *pirq, u32 status)
 			duration = U32_MAX;
 		trace_gpu_job_irq(ptdev->base.dev, status, duration);
 	}
+	panthor_perf_report_irq(ptdev, status);
 }
 
 static irqreturn_t panthor_job_irq_threaded_handler(int irq, void *data)
diff --git a/drivers/gpu/drm/panthor/panthor_perf.c b/drivers/gpu/drm/panthor/panthor_perf.c
index 76c346981150..c66ad658a95d 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.c
+++ b/drivers/gpu/drm/panthor/panthor_perf.c
@@ -11,8 +11,11 @@
 
 #include "panthor_device.h"
 #include "panthor_fw.h"
+#include "panthor_fw_regs.h"
+#include "panthor_gem.h"
 #include "panthor_gpu.h"
 #include "panthor_gpu_regs.h"
+#include "panthor_mmu.h"
 #include "panthor_perf.h"
 
 /**
@@ -36,6 +39,76 @@
  */
 #define PANTHOR_PERF_EM_BITS (BITS_PER_TYPE(u64) * 2)
 
+/**
+ * PANTHOR_CTR_TIMESTAMP_LO - The first architecturally mandated counter of every block type
+ *                            contains the low 32-bits of the TIMESTAMP value.
+ */
+#define PANTHOR_CTR_TIMESTAMP_LO (0)
+
+/**
+ * PANTHOR_CTR_TIMESTAMP_HI - The register offset containinig the high 32-bits of the TIMESTAMP
+ *                            value.
+ */
+#define PANTHOR_CTR_TIMESTAMP_HI (1)
+
+/**
+ * PANTHOR_CTR_PRFCNT_EN - The register offset containing the enable mask for the enabled counters
+ *                         that were written to memory.
+ */
+#define PANTHOR_CTR_PRFCNT_EN (2)
+
+/**
+ * PANTHOR_HEADER_COUNTERS - The first four counters of every block type are architecturally
+ *                           defined to be equivalent. The fourth counter is always reserved,
+ *                           and should be zero and as such, does not have a separate define.
+ *
+ *                           These are the only four counters that are the same between different
+ *                           blocks and are consistent between different architectures.
+ */
+#define PANTHOR_HEADER_COUNTERS (4)
+
+/**
+ * PANTHOR_CTR_SAMPLE_REASON - The metadata block has a single value in position three which
+ *                             indicates the reason a sample was taken.
+ */
+#define PANTHOR_CTR_SAMPLE_REASON (3)
+
+/**
+ * PANTHOR_PERF_RINGBUF_SLOTS_MIN - The minimum permitted number of slots in the Panthor perf
+ *                                  ring buffer.
+ */
+#define PANTHOR_PERF_RINGBUF_SLOTS_MIN (16)
+
+/**
+ * PANTHOR_PERF_RINGBUF_SLOTS_MAX - The maximum permitted number of slots in the Panthor perf
+ *                                  ring buffer.
+ */
+#define PANTHOR_PERF_RINGBUF_SLOTS_MAX (256)
+
+static unsigned int perf_ringbuf_slots = 32;
+
+static int perf_ringbuf_slots_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int slots;
+	int ret = kstrtouint(val, 0, &slots);
+
+	if (ret)
+		return ret;
+
+	if (!is_power_of_2(slots))
+		return -EINVAL;
+
+	return param_set_uint_minmax(val, kp, 16, 128);
+}
+
+static const struct kernel_param_ops perf_ringbuf_ops = {
+	.set = perf_ringbuf_slots_set,
+	.get = param_get_uint,
+};
+module_param_cb(perf_ringbuf_slots, &perf_ringbuf_ops, &perf_ringbuf_slots, 0400);
+MODULE_PARM_DESC(perf_ringbuf_slots,
+		 "Power of two slots allocated for the Panthor perf kernel-FW ringbuffer");
+
 enum panthor_perf_session_state {
 	/** @PANTHOR_PERF_SESSION_STOPPED: The session cannot sample. It starts out stopped. */
 	PANTHOR_PERF_SESSION_STOPPED = 0,
@@ -68,12 +141,33 @@ struct panthor_perf_counter_block {
 	u64 counters[];
 };
 
+struct panthor_perf_fw_block {
+	/** @block_count: Number of blocks of the given @type in the FW ringbuffer. */
+	size_t block_count;
+
+	/** @phys_mask: Bitmask of the physically available blocks. */
+	u64 phys_mask;
+};
+
 struct panthor_perf_bo {
 	struct drm_gem_object *obj;
 	void *vaddr;
 };
 
+struct panthor_perf_sample_block_iter {
+	const u8 *src;
+	u8 *dst;
+	size_t offset;
+	size_t data_size;
+	size_t block_size;
+	const struct panthor_perf_counter_block *src_blk;
+	struct panthor_perf_counter_block *dst_blk;
+};
+
 struct panthor_perf_session {
+	/** @state_wq: Wait queue for state transitions driven by the sampling path. */
+	wait_queue_head_t state_wq;
+
 	/** @state_lock: Used to protect session state information and overflow bit. */
 	struct mutex state_lock;
 
@@ -92,6 +186,12 @@ struct panthor_perf_session {
 	 */
 	u64 accum_idx;
 
+	/** @pending_sample: Whether this session has a sample requested on it. */
+	bool pending_sample;
+
+	/** @sample_flags: Accumulated @drm_panthor_perf_sample_flags for the pending sample. */
+	u32 sample_flags;
+
 	/** @sample_start_ns: Sample request time, obtained from a monotonic raw clock. */
 	u64 sample_start_ns;
 
@@ -148,11 +248,88 @@ struct panthor_perf {
 	/** @info: Pointer to the previously initialized perf info. */
 	struct drm_panthor_perf_info *info;
 
+	/** @rb: Kernel BO in the FW AS containing the sample ringbuffer. */
+	struct panthor_kernel_bo *rb;
+
+	/** @sample_slots: Number of sample slots in the FW ringbuffer. */
+	size_t sample_slots;
+
+	/** @em: Combined enable mask for all of the active sessions. */
+	struct panthor_perf_enable_masks *em;
+
 	/**
-	 * @op_lock: Serializes session operations. Acquire before looking up session
-	 *           handles and before a session's state_lock.
+	 * @fw_block_size: The size of a single block in the FW ring buffer, equal to
+	 *                 sizeof(u32) * counters_per_block.
+	 */
+	size_t fw_block_size;
+
+	/**
+	 * @fw_sample_size: The size of a single FW sample in the FW ringbuffer.
+	 */
+	size_t fw_sample_size;
+
+	/**
+	 * @blocks: FW block layout for one sample. Metadata blocks are consumed
+	 *          internally and omitted from userspace samples.
+	 */
+	struct panthor_perf_fw_block blocks[DRM_PANTHOR_PERF_BLOCK_MAX];
+
+	/**
+	 * @sample: Pointer to an upscaled and annotated sample that may be emitted to userspace.
+	 *          This is used both as an intermediate buffer to do the zero-extension of the
+	 *          32-bit counters to 64-bits and as a storage buffer in case of automatic samples.
+	 */
+	u8 *sample;
+
+	/**
+	 * @op_lock: Serializes session operations and global FW configuration. Acquire
+	 *           before looking up session handles and before @session_lock or a
+	 *           session's state_lock. The sample worker must not acquire this lock,
+	 *           so synchronous session operations can wait for sample delivery.
+	 *           Runtime-PM references must be acquired and released outside this
+	 *           lock, as the PM callbacks also acquire it.
 	 */
 	struct mutex op_lock;
+
+	/**
+	 * @session_lock: Lock used to guard the @session_list.
+	 */
+	struct mutex session_lock;
+
+	/** @session_list: List of all sessions. */
+	struct list_head session_list;
+
+	/** @pend_lock: Lock used to guard the requested sample field. */
+	spinlock_t pend_lock;
+
+
+	/** @fw_sample_requested: A sample has been requested. */
+	bool fw_sample_requested;
+
+	/** @set_config: The set that will be configured onto the hardware. */
+	u8 set_config;
+
+	/** @last_fw_sample_time: The CPU-side timestamp of the latest consumed FW sample. */
+	u64 last_fw_sample_time;
+
+	/** @sample_workq: Workqueue to handle and accumulate samples. */
+	struct workqueue_struct *sample_workq;
+
+	/** @sample_work: Worker to consume available FW samples and expose them to the user. */
+	struct work_struct sample_work;
+
+	/**
+	 * @ptdev: Backpointer to the Panthor device, needed to ring the global doorbell and
+	 *         interface with FW.
+	 */
+	struct panthor_device *ptdev;
+};
+
+struct panthor_perf_fw_block_iter {
+	const struct panthor_perf *perf_ctx;
+	enum drm_panthor_perf_block_type type;
+	u32 idx;
+	size_t offset;
 };
 
 struct panthor_perf_pool {
@@ -171,6 +348,33 @@ static size_t get_annotated_block_size(size_t counters_per_block)
 	return struct_size_t(struct panthor_perf_counter_block, counters, counters_per_block);
 }
 
+static void
+sample_block_iter_init(const struct drm_panthor_perf_info *info,
+		       const u8 *src, u8 *dst,
+		       struct panthor_perf_sample_block_iter *iter)
+{
+	const size_t sample_header_size = info->sample_header_size;
+
+	*iter = (struct panthor_perf_sample_block_iter) {
+		.src = src + sample_header_size,
+		.dst = dst + sample_header_size,
+		.data_size = info->sample_size - sample_header_size,
+		.block_size = get_annotated_block_size(info->counters_per_block),
+	};
+}
+
+static bool sample_block_iter_next(struct panthor_perf_sample_block_iter *iter)
+{
+	if (iter->offset >= iter->data_size)
+		return false;
+
+	iter->src_blk = (const struct panthor_perf_counter_block *)(iter->src + iter->offset);
+	iter->dst_blk = (struct panthor_perf_counter_block *)(iter->dst + iter->offset);
+	iter->offset += iter->block_size;
+
+	return true;
+}
+
 static size_t session_get_sample_size(const struct drm_panthor_perf_info *const info)
 {
 	const size_t block_size = get_annotated_block_size(info->counters_per_block);
@@ -239,6 +443,30 @@ session_get(struct panthor_perf_session *session)
 	return session;
 }
 
+static void panthor_perf_em_add(struct panthor_perf_enable_masks *const dst_em,
+				const struct panthor_perf_enable_masks *const src_em)
+{
+	size_t i = 0;
+
+	for (i = DRM_PANTHOR_PERF_BLOCK_FIRST; i <= DRM_PANTHOR_PERF_BLOCK_LAST; i++)
+		bitmap_or(dst_em->mask[i], dst_em->mask[i], src_em->mask[i], PANTHOR_PERF_EM_BITS);
+}
+
+static u64 session_read_extract_idx(struct panthor_perf_session *session)
+{
+	/* Userspace will update their own extract index to indicate that a sample is consumed
+	 * from the ringbuffer, and we must ensure we read the latest value.
+	 */
+	return smp_load_acquire(&session->control->extract_idx);
+}
+
+static void session_write_insert_idx(struct panthor_perf_session *session, u64 idx)
+{
+	/* Userspace needs the insert index to know where to look for the sample. */
+	smp_store_release(&session->control->insert_idx, idx);
+	session->insert_idx = idx;
+}
+
 static int
 panthor_perf_bo_get(struct drm_file *file, u32 handle, size_t expected_size,
 		    struct panthor_perf_bo *bo)
@@ -338,6 +566,277 @@ session_from_handle(struct panthor_perf_pool *pool, unsigned long handle)
 	return session;
 }
 
+static u32 compress_enable_mask(u32 counters, const unsigned long *const src)
+{
+	size_t i;
+	u32 result = 0;
+	const unsigned long ctr_per_bit = counters / BITS_PER_TYPE(u32);
+
+	for_each_set_bit(i, src, counters) {
+		const unsigned long pos = div_u64(i, ctr_per_bit);
+
+		result |= BIT(pos);
+	}
+
+	return result;
+}
+
+static void expand_enable_mask(u32 counters, u32 em, unsigned long *const dst)
+{
+	size_t i;
+	const unsigned long ctrs_per_bit = counters / BITS_PER_TYPE(u32);
+	DECLARE_BITMAP(emb, BITS_PER_TYPE(u32));
+
+	bitmap_from_arr32(emb, &em, BITS_PER_TYPE(u32));
+
+	for_each_set_bit(i, emb, BITS_PER_TYPE(u32))
+		bitmap_set(dst, i * ctrs_per_bit, ctrs_per_bit);
+}
+
+static inline bool
+panthor_perf_fw_block_iter_valid(struct panthor_perf_fw_block_iter *iter)
+{
+	const struct panthor_perf *perf = iter->perf_ctx;
+
+	while (iter->type < DRM_PANTHOR_PERF_BLOCK_MAX) {
+		if (iter->idx < perf->blocks[iter->type].block_count)
+			return true;
+
+		iter->type++;
+		iter->idx = 0;
+	}
+
+	return false;
+}
+
+static void panthor_perf_handle_fw_samples(struct panthor_device *ptdev,
+					   u32 extract_idx, u32 insert_idx)
+{
+	struct panthor_perf *perf = ptdev->perf;
+	const size_t ann_block_size =
+		get_annotated_block_size(ptdev->perf_info.counters_per_block);
+	u32 i;
+
+	for (i = extract_idx; i != insert_idx; i++) {
+		u32 slot = i % perf->sample_slots;
+		u8 *fw_sample = (u8 *)perf->rb->kmap + slot * perf->fw_sample_size;
+		struct panthor_perf_fw_block_iter fw_iter = { .perf_ctx = perf };
+		size_t ann_off = ptdev->perf_info.sample_header_size;
+
+		for (; panthor_perf_fw_block_iter_valid(&fw_iter);
+		     fw_iter.offset += fw_iter.perf_ctx->fw_block_size, fw_iter.idx++) {
+			struct panthor_perf_counter_block *blk =
+				(typeof(blk))(perf->sample + ann_off);
+			u32 *const block = (u32 *)(fw_sample + fw_iter.offset);
+			const u32 prfcnt_en = block[PANTHOR_CTR_PRFCNT_EN];
+
+			if (fw_iter.type == DRM_PANTHOR_PERF_BLOCK_METADATA) {
+				/*
+				 * The host must clear the SAMPLE_REASON to acknowledge it has
+				 * consumed the sample.
+				 */
+				block[PANTHOR_CTR_SAMPLE_REASON] = 0;
+				continue;
+			}
+
+			if (!(perf->blocks[fw_iter.type].phys_mask & BIT_ULL(fw_iter.idx)))
+				continue;
+
+			blk->header = (struct drm_panthor_perf_block_header) {
+				.block_idx = fw_iter.idx,
+				.block_type = fw_iter.type,
+			};
+
+			if (prfcnt_en == 0)
+				goto advance_block;
+
+			/*
+			 * The four header counters must be treated differently, because they are
+			 * not additive. For the fourth, the assignment does not matter, as it
+			 * is reserved and should be zero.
+			 */
+			blk->counters[PANTHOR_CTR_TIMESTAMP_LO] = block[PANTHOR_CTR_TIMESTAMP_LO];
+			blk->counters[PANTHOR_CTR_TIMESTAMP_HI] = block[PANTHOR_CTR_TIMESTAMP_HI];
+			blk->counters[PANTHOR_CTR_PRFCNT_EN] = block[PANTHOR_CTR_PRFCNT_EN];
+
+			/*
+			 * The host must clear PRFCNT_EN to acknowledge it has consumed the sample.
+			 */
+			block[PANTHOR_CTR_PRFCNT_EN] = 0;
+
+			if (prfcnt_en == 1)
+				goto advance_block;
+
+			for (size_t k = PANTHOR_HEADER_COUNTERS;
+			     k < ptdev->perf_info.counters_per_block; k++)
+				blk->counters[k] += block[k];
+
+advance_block:
+			ann_off += ann_block_size;
+		}
+
+		perf->last_fw_sample_time = ktime_get_raw_ns();
+	}
+}
+
+static size_t panthor_perf_get_fw_reported_size(struct panthor_device *ptdev)
+{
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(ptdev);
+
+	size_t fw_size = GLB_PERFCNT_FW_SIZE(glb_iface->control->perfcnt_size);
+	size_t hw_size = GLB_PERFCNT_HW_SIZE(glb_iface->control->perfcnt_size);
+	size_t md_size = PERFCNT_FEATURES_MD_SIZE(glb_iface->control->perfcnt_features);
+
+	return md_size + fw_size + hw_size;
+}
+
+static size_t get_reserved_sc_blocks(struct panthor_device *ptdev)
+{
+	const u64 sc_mask = ptdev->gpu_info.shader_present;
+
+	if (GPU_ARCH_MAJOR(ptdev->gpu_info.gpu_id) < 14)
+		return fls64(sc_mask);
+
+	return ptdev->perf_info.shader_blocks;
+}
+
+#define BLK_MASK(x) GENMASK_ULL((x) - 1, 0)
+
+static u64 get_sc_mask(struct panthor_device *ptdev)
+{
+	const u64 sc_mask = ptdev->gpu_info.shader_present;
+
+	if (GPU_ARCH_MAJOR(ptdev->gpu_info.gpu_id) < 14)
+		return sc_mask;
+
+	return BLK_MASK(hweight64(sc_mask));
+}
+
+static int panthor_perf_setup_fw_buffer(struct panthor_device *ptdev, struct panthor_perf *perf)
+{
+	struct drm_panthor_perf_info *info = &ptdev->perf_info;
+	const size_t block_size = info->counters_per_block * PANTHOR_HW_COUNTER_SIZE;
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(ptdev);
+	const size_t reported_sample_size = panthor_perf_get_fw_reported_size(ptdev);
+	const bool has_metadata = PERFCNT_FEATURES_MD_SIZE(glb_iface->control->perfcnt_features);
+	const struct panthor_perf_fw_block blocks[DRM_PANTHOR_PERF_BLOCK_MAX] = {
+		[DRM_PANTHOR_PERF_BLOCK_METADATA] = {
+			.block_count = has_metadata ? 1 : 0,
+			.phys_mask = has_metadata ? BIT_ULL(0) : 0,
+		},
+		[DRM_PANTHOR_PERF_BLOCK_FW] = {
+			.block_count = info->fw_blocks,
+			.phys_mask = info->fw_blocks ? BLK_MASK(info->fw_blocks) : 0,
+		},
+		[DRM_PANTHOR_PERF_BLOCK_CSHW] = {
+			.block_count = info->cshw_blocks,
+			.phys_mask = BLK_MASK(info->cshw_blocks),
+		},
+		[DRM_PANTHOR_PERF_BLOCK_TILER] = {
+			.block_count = info->tiler_blocks,
+			.phys_mask = BLK_MASK(info->tiler_blocks),
+		},
+		[DRM_PANTHOR_PERF_BLOCK_MEMSYS] = {
+			.block_count = info->memsys_blocks,
+			.phys_mask = BLK_MASK(info->memsys_blocks),
+		},
+		[DRM_PANTHOR_PERF_BLOCK_SHADER] = {
+			.block_count = get_reserved_sc_blocks(ptdev),
+			.phys_mask = get_sc_mask(ptdev),
+		},
+	};
+	size_t offset = 0;
+
+	for (enum drm_panthor_perf_block_type type = 0; type < DRM_PANTHOR_PERF_BLOCK_MAX; type++)
+		offset += blocks[type].block_count * block_size;
+
+	if (offset != reported_sample_size) {
+		drm_warn(&ptdev->base,
+				"Could not align fw and kernel perf buffers, feature disabled.");
+		return -EINVAL;
+	}
+
+	memcpy(perf->blocks, blocks, sizeof(blocks));
+	perf->fw_block_size = block_size;
+	perf->fw_sample_size = offset;
+
+	return 0;
+}
+
+static int panthor_perf_fw_stop_sampling(struct panthor_device *ptdev)
+{
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(ptdev);
+	u32 acked;
+	int ret;
+
+	if (~READ_ONCE(glb_iface->input->req) & GLB_PERFCNT_ENABLE)
+		return 0;
+
+	panthor_fw_update_reqs(glb_iface, req, 0, GLB_PERFCNT_ENABLE);
+	panthor_fw_ring_doorbell(ptdev, CSF_GLB_DOORBELL_ID);
+	ret = panthor_fw_glb_wait_acks(ptdev, GLB_PERFCNT_ENABLE, &acked, 100);
+	if (ret)
+		drm_warn(&ptdev->base, "Could not disable performance counters");
+
+	return ret;
+}
+
+static int panthor_perf_fw_start_sampling(struct panthor_device *ptdev)
+{
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(ptdev);
+	u32 acked;
+	int ret;
+
+	if (READ_ONCE(glb_iface->input->req) & GLB_PERFCNT_ENABLE)
+		return 0;
+
+	/*
+	 * The spec mandates that the host zero the PRFCNT_EXTRACT register before an enable
+	 * operation, and each (re-)enable will require an enable-disable pair to program
+	 * the new changes onto the FW interface.
+	 */
+	WRITE_ONCE(glb_iface->input->perfcnt_extract, 0);
+
+	panthor_fw_update_reqs(glb_iface, req, GLB_PERFCNT_ENABLE, GLB_PERFCNT_ENABLE);
+	panthor_fw_ring_doorbell(ptdev, CSF_GLB_DOORBELL_ID);
+	ret = panthor_fw_glb_wait_acks(ptdev, GLB_PERFCNT_ENABLE, &acked, 100);
+	if (ret)
+		drm_warn(&ptdev->base, "Could not enable performance counters");
+
+	return ret;
+}
+
+static void panthor_perf_fw_write_config(struct panthor_perf *perf)
+{
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(perf->ptdev);
+	const u32 counters = perf->info->counters_per_block;
+	const struct panthor_perf_enable_masks *const em = perf->em;
+	u32 perfcnt_config;
+
+	glb_iface->input->perfcnt_csf_enable =
+		compress_enable_mask(counters, em->mask[DRM_PANTHOR_PERF_BLOCK_CSHW]);
+	glb_iface->input->perfcnt_shader_enable =
+		compress_enable_mask(counters, em->mask[DRM_PANTHOR_PERF_BLOCK_SHADER]);
+	glb_iface->input->perfcnt_mmu_l2_enable =
+		compress_enable_mask(counters, em->mask[DRM_PANTHOR_PERF_BLOCK_MEMSYS]);
+	glb_iface->input->perfcnt_tiler_enable =
+		compress_enable_mask(counters, em->mask[DRM_PANTHOR_PERF_BLOCK_TILER]);
+	glb_iface->input->perfcnt_fw_enable =
+		compress_enable_mask(counters, em->mask[DRM_PANTHOR_PERF_BLOCK_FW]);
+	glb_iface->input->perfcnt_csg_enable = 0;
+
+	WRITE_ONCE(glb_iface->input->perfcnt_as, panthor_vm_as(panthor_fw_vm(perf->ptdev)));
+	WRITE_ONCE(glb_iface->input->perfcnt_base, panthor_kernel_bo_gpuva(perf->rb));
+
+	perfcnt_config = GLB_PERFCNT_CONFIG_SIZE(perf_ringbuf_slots);
+	perfcnt_config |= GLB_PERFCNT_CONFIG_SET(perf->set_config);
+
+	if (PERFCNT_FEATURES_MD_SIZE(glb_iface->control->perfcnt_features))
+		perfcnt_config |= GLB_PERFCNT_METADATA_ENABLE;
+
+	WRITE_ONCE(glb_iface->input->perfcnt_config, perfcnt_config);
+}
+
 #define DEFINE_SESSION_STATE_CHECK(name, _state)			     \
 	static bool session_is_## name(struct panthor_perf_session *session) \
 	{								     \
@@ -349,6 +848,7 @@ session_from_handle(struct panthor_perf_pool *pool, unsigned long handle)
 DEFINE_SESSION_STATE_CHECK(active, PANTHOR_PERF_SESSION_ACTIVE);
 DEFINE_SESSION_STATE_CHECK(starting, PANTHOR_PERF_SESSION_STARTING);
 DEFINE_SESSION_STATE_CHECK(stopping, PANTHOR_PERF_SESSION_STOPPING);
+DEFINE_SESSION_STATE_CHECK(stopped, PANTHOR_PERF_SESSION_STOPPED);
 
 static bool session_has_overflow(struct panthor_perf_session *session)
 {
@@ -365,6 +865,388 @@ static void session_set_state(struct panthor_perf_session *session,
 	session->state = state;
 }
 
+static bool session_has_space(struct panthor_perf_session *session, u32 needed)
+{
+	const u64 insert = session->insert_idx;
+	const u64 extract = session_read_extract_idx(session);
+
+	return insert - extract + needed <= session->ringbuf_slots;
+}
+
+/**
+ * session_accumulate_sample - Accumulate requested counters into an existing sample.
+ * @ptdev: Panthor device.
+ * @session: Perf session.
+ * @session_sample: Start of the destination sample, including its header.
+ * @perf_sample: Start of the intermediate sample, including its header.
+ *
+ * Firmware enable masks select groups of counters. Copy timestamps and accumulate
+ * only the counters requested by this session.
+ */
+static void session_accumulate_sample(struct panthor_device *ptdev,
+				      struct panthor_perf_session *session,
+				      u8 *session_sample, u8 *perf_sample)
+{
+	const struct drm_panthor_perf_info *const perf_info = &ptdev->perf_info;
+	struct panthor_perf_enable_masks *em = session->enabled_counters;
+	struct panthor_perf_sample_block_iter iter;
+	const u32 counters = perf_info->counters_per_block;
+
+	sample_block_iter_init(perf_info, perf_sample, session_sample, &iter);
+
+	while (sample_block_iter_next(&iter)) {
+		DECLARE_BITMAP(enabled_ctrs, PANTHOR_PERF_EM_BITS);
+		const struct panthor_perf_counter_block *src_blk = iter.src_blk;
+		struct panthor_perf_counter_block *dst_blk = iter.dst_blk;
+
+		bitmap_from_arr64(enabled_ctrs, em->mask[dst_blk->header.block_type],
+				  PANTHOR_PERF_EM_BITS);
+		bitmap_clear(enabled_ctrs, 0, PANTHOR_HEADER_COUNTERS);
+
+		dst_blk->counters[PANTHOR_CTR_TIMESTAMP_HI] =
+			src_blk->counters[PANTHOR_CTR_TIMESTAMP_HI];
+		dst_blk->counters[PANTHOR_CTR_TIMESTAMP_LO] =
+			src_blk->counters[PANTHOR_CTR_TIMESTAMP_LO];
+
+		for (size_t ctr_idx = PANTHOR_HEADER_COUNTERS; ctr_idx < counters; ctr_idx++)
+			if (test_bit(ctr_idx, enabled_ctrs))
+				dst_blk->counters[ctr_idx] += src_blk->counters[ctr_idx];
+	}
+}
+
+static void perf_fw_request_sample(struct panthor_perf *perf)
+{
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(perf->ptdev);
+	bool sample;
+
+	scoped_guard(spinlock_irqsave, &perf->pend_lock) {
+		sample = !perf->fw_sample_requested;
+
+		if (sample)
+			perf->fw_sample_requested = true;
+	}
+
+	if (sample) {
+		panthor_fw_toggle_reqs(glb_iface, req, ack, GLB_PERFCNT_SAMPLE);
+		panthor_fw_ring_doorbell(perf->ptdev, CSF_GLB_DOORBELL_ID);
+	}
+}
+
+/**
+ * session_populate_sample - Populate a new userspace sample slot.
+ * @ptdev: Panthor device.
+ * @session: Perf session.
+ * @session_sample: Start of the destination sample, including its header.
+ * @perf_sample: Start of the intermediate sample, including its header.
+ *
+ * Copy the counter blocks and clear counters not requested by this session.
+ * Update each block's PRFCNT_EN value to reflect this session's enable mask.
+ * The sample header is filled when the sample is published.
+ */
+static void session_populate_sample(struct panthor_device *ptdev,
+				    struct panthor_perf_session *session, u8 *session_sample,
+				    u8 *perf_sample)
+{
+	const struct drm_panthor_perf_info *const perf_info = &ptdev->perf_info;
+	struct panthor_perf_sample_block_iter iter;
+	const size_t sample_header_size = perf_info->sample_header_size;
+	const size_t data_size = perf_info->sample_size - sample_header_size;
+	const u32 counters = perf_info->counters_per_block;
+
+	memcpy(session_sample + sample_header_size, perf_sample + sample_header_size, data_size);
+
+	sample_block_iter_init(perf_info, perf_sample, session_sample, &iter);
+
+	while (sample_block_iter_next(&iter)) {
+		DECLARE_BITMAP(em_diff, PANTHOR_PERF_EM_BITS);
+		DECLARE_BITMAP(enabled_ctrs, PANTHOR_PERF_EM_BITS);
+		struct panthor_perf_counter_block *blk = iter.dst_blk;
+		enum drm_panthor_perf_block_type type = blk->header.block_type;
+		unsigned long *blk_em = session->enabled_counters->mask[type];
+		const u32 blk_prfcnt_en = blk->counters[PANTHOR_CTR_PRFCNT_EN];
+
+		bitmap_zero(enabled_ctrs, PANTHOR_PERF_EM_BITS);
+		expand_enable_mask(counters, blk_prfcnt_en, enabled_ctrs);
+		bitmap_andnot(em_diff, enabled_ctrs, blk_em, PANTHOR_PERF_EM_BITS);
+		bitmap_clear(em_diff, 0, PANTHOR_HEADER_COUNTERS);
+
+		blk->counters[PANTHOR_CTR_PRFCNT_EN] = compress_enable_mask(counters, blk_em);
+
+		for (size_t ctr_idx = PANTHOR_HEADER_COUNTERS; ctr_idx < counters; ctr_idx++) {
+			if (test_bit(ctr_idx, em_diff))
+				blk->counters[ctr_idx] = 0;
+		}
+	}
+}
+
+static void session_copy_sample(struct panthor_device *ptdev,
+				struct panthor_perf_session *session, u32 flags)
+{
+	struct panthor_perf *perf = ptdev->perf;
+	const size_t sample_size = ptdev->perf_info.sample_size;
+	const u64 insert_idx = session->insert_idx;
+	u8 *new_sample;
+
+	scoped_guard(mutex, &session->state_lock) {
+		if (flags & DRM_PANTHOR_PERF_SAMPLE_OVERFLOW)
+			session->overflow = true;
+
+		if (session_is_starting(session) ||
+		    (!session_is_active(session) && !session_is_stopping(session)))
+			return;
+	}
+
+	new_sample = session->samples + (insert_idx % session->ringbuf_slots) * sample_size;
+
+	if (session->accum_idx != insert_idx) {
+		session->sample_flags = flags;
+		session_populate_sample(ptdev, session, new_sample, perf->sample);
+		session->accum_idx = insert_idx;
+	} else {
+		session->sample_flags |= flags;
+		session_accumulate_sample(ptdev, session, new_sample, perf->sample);
+	}
+}
+
+static void session_publish_sample(struct panthor_perf *perf,
+				   struct panthor_perf_session *session)
+{
+	const struct drm_panthor_perf_info *const info = &perf->ptdev->perf_info;
+	u8 *sample = session->samples +
+		(session->insert_idx % session->ringbuf_slots) * info->sample_size;
+	struct drm_panthor_perf_sample_header *hdr = (typeof(hdr))sample;
+
+	*hdr = (struct drm_panthor_perf_sample_header) {
+		.user_data = session->user_data,
+		.timestamp_start_ns = session->sample_start_ns,
+		.timestamp_end_ns = perf->last_fw_sample_time,
+		.flags = session->sample_flags,
+	};
+
+	session_write_insert_idx(session, session->insert_idx + 1);
+
+	/* Order sample and insert-index writes before the eventfd notification. */
+	wmb();
+
+	eventfd_signal(session->eventfd);
+}
+
+static void session_emit_sample(struct panthor_perf *perf, struct panthor_perf_session *session)
+{
+	guard(mutex)(&session->state_lock);
+
+	if (session_is_starting(session)) {
+		session_set_state(session, PANTHOR_PERF_SESSION_ACTIVE);
+		goto update_session;
+	}
+
+	if (session_is_stopping(session)) {
+		session_publish_sample(perf, session);
+		goto handle_stopping;
+	}
+
+	if (!session_is_active(session))
+		return;
+
+	if (session->pending_sample) {
+		session->pending_sample = false;
+		session_publish_sample(perf, session);
+		goto update_session;
+	}
+
+	return;
+
+handle_stopping:
+	if (session_is_stopping(session))
+		session_set_state(session, PANTHOR_PERF_SESSION_STOPPED);
+
+update_session:
+	session->sample_flags = 0;
+	session->sample_start_ns = perf->last_fw_sample_time;
+	wake_up_all(&session->state_wq);
+}
+
+static void panthor_perf_sample_worker(struct work_struct *work)
+{
+	struct panthor_perf *perf = container_of(work, struct panthor_perf, sample_work);
+	struct panthor_device *const ptdev = perf->ptdev;
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(ptdev);
+	const u32 extract_idx = READ_ONCE(glb_iface->input->perfcnt_extract);
+	const u32 insert_idx = READ_ONCE(glb_iface->output->perfcnt_insert);
+	const u32 ack = READ_ONCE(glb_iface->output->ack);
+	const u32 req = READ_ONCE(glb_iface->input->req);
+	const u32 events = (req ^ ack) & (GLB_PERFCNT_THRESHOLD | GLB_PERFCNT_OVERFLOW);
+	const u32 flags = events & GLB_PERFCNT_OVERFLOW ? DRM_PANTHOR_PERF_SAMPLE_OVERFLOW : 0;
+
+	/* Defer sample processing until the pending reset completes. */
+	if (panthor_device_reset_is_pending(ptdev)) {
+		queue_work(perf->sample_workq, &perf->sample_work);
+		return;
+	}
+
+	/* A stale notification must not complete a userspace sample request. */
+	if (extract_idx == insert_idx)
+		goto ack_events;
+
+	panthor_perf_handle_fw_samples(ptdev, extract_idx, insert_idx);
+	WRITE_ONCE(glb_iface->input->perfcnt_extract, insert_idx);
+
+	scoped_guard(mutex, &perf->session_lock) {
+		struct panthor_perf_session *session;
+
+		list_for_each_entry(session, &perf->session_list, sessions)
+			session_copy_sample(ptdev, session, flags);
+
+		list_for_each_entry(session, &perf->session_list, sessions)
+			session_emit_sample(perf, session);
+	}
+
+	memset(perf->sample, 0, ptdev->perf_info.sample_size);
+
+ack_events:
+	if (events)
+		panthor_fw_update_reqs(glb_iface, req, ack, events);
+}
+
+void panthor_perf_report_irq(struct panthor_device *ptdev, u32 status)
+{
+	struct panthor_fw_global_iface *glb_iface = panthor_fw_get_glb_iface(ptdev);
+	struct panthor_perf *const perf = ptdev->perf;
+
+	if (!(status & JOB_INT_GLOBAL_IF))
+		return;
+
+	if (!perf)
+		return;
+
+	const u32 ack = READ_ONCE(glb_iface->output->ack);
+	const u32 req = READ_ONCE(glb_iface->input->req);
+	const u32 events = (req ^ ack) & (GLB_PERFCNT_THRESHOLD | GLB_PERFCNT_OVERFLOW);
+	const bool sample_acked = ((req & GLB_PERFCNT_SAMPLE) == (ack & GLB_PERFCNT_SAMPLE));
+	const u32 extract_idx = READ_ONCE(glb_iface->input->perfcnt_extract);
+	const u32 insert_idx = READ_ONCE(glb_iface->output->perfcnt_insert);
+	const bool has_sample = sample_acked && insert_idx != extract_idx;
+
+	if (sample_acked) {
+		guard(spinlock_irqsave)(&perf->pend_lock);
+		perf->fw_sample_requested = false;
+	}
+
+	if (events || has_sample)
+		queue_work(perf->sample_workq, &perf->sample_work);
+}
+
+static int perf_enable(struct panthor_perf *perf)
+{
+	lockdep_assert_held(&perf->op_lock);
+
+	guard(mutex)(&perf->session_lock);
+
+	if (list_empty(&perf->session_list))
+		return 0;
+
+	panthor_perf_fw_write_config(perf);
+	return panthor_perf_fw_start_sampling(perf->ptdev);
+}
+
+static int perf_disable(struct panthor_perf *perf)
+{
+	lockdep_assert_held(&perf->op_lock);
+
+	guard(mutex)(&perf->session_lock);
+
+	if (list_empty(&perf->session_list))
+		return 0;
+
+	return panthor_perf_fw_stop_sampling(perf->ptdev);
+}
+
+static void compute_enable_mask(struct panthor_perf *perf)
+{
+	lockdep_assert_held(&perf->session_lock);
+
+	struct list_head *snode;
+	struct panthor_perf_enable_masks em = {};
+
+	list_for_each(snode, &perf->session_list)
+	{
+		struct panthor_perf_session *session =
+			container_of(snode, typeof(*session), sessions);
+
+		panthor_perf_em_add(&em, session->enabled_counters);
+	}
+
+	*perf->em = em;
+}
+
+static void perf_attach_session(struct panthor_perf *perf, struct panthor_perf_session *session)
+{
+	lockdep_assert_held(&perf->session_lock);
+
+	session_get(session);
+	list_add_tail(&session->sessions, &perf->session_list);
+	compute_enable_mask(perf);
+}
+
+static void perf_detach_session(struct panthor_perf *perf, struct panthor_perf_session *session)
+{
+	lockdep_assert_held(&perf->session_lock);
+
+	list_del_init(&session->sessions);
+	compute_enable_mask(perf);
+	session_put(session);
+}
+
+static int panthor_perf_session_add(struct panthor_perf *perf,
+				    struct panthor_perf_session *session, u8 set)
+{
+	int ret = 0;
+
+	guard(mutex)(&perf->op_lock);
+
+	scoped_guard(mutex, &perf->session_lock) {
+		if (!list_empty(&perf->session_list))
+			return -EBUSY;
+
+		perf->set_config = set;
+	}
+
+	ret = perf_disable(perf);
+	if (ret)
+		return ret;
+
+	scoped_guard(mutex, &perf->session_lock)
+		perf_attach_session(perf, session);
+
+	ret = perf_enable(perf);
+	if (ret)
+		goto drop_session_ref;
+
+	return ret;
+drop_session_ref:
+	scoped_guard(mutex, &perf->session_lock)
+		perf_detach_session(perf, session);
+
+	return ret;
+}
+
+static int panthor_perf_session_remove(struct panthor_perf *perf,
+				       struct panthor_perf_session *session)
+{
+	guard(mutex)(&perf->op_lock);
+
+	/*
+	 * Even if the disable fails,  the session must be removed, so the error
+	 * is ignored.
+	 */
+	perf_disable(perf);
+
+	scoped_guard(mutex, &perf->session_lock)
+		perf_detach_session(perf, session);
+
+	return perf_enable(perf);
+}
+
 /**
  * panthor_perf_init - Initialize the performance counter subsystem.
  * @ptdev: Panthor device
@@ -377,7 +1259,10 @@ static void session_set_state(struct panthor_perf_session *session,
 int panthor_perf_init(struct panthor_device *ptdev)
 {
 	struct panthor_perf *perf;
-	int ret = 0;
+	struct panthor_kernel_bo *bo;
+	struct panthor_perf_enable_masks *em;
+	u8 *sample;
+	int ret;
 
 	if (!ptdev)
 		return -EINVAL;
@@ -392,12 +1277,107 @@ int panthor_perf_init(struct panthor_device *ptdev)
 	if (ret)
 		return ret;
 
+	ret = drmm_mutex_init(&ptdev->base, &perf->session_lock);
+	if (ret)
+		return ret;
+
+	spin_lock_init(&perf->pend_lock);
+
+	ret = panthor_perf_setup_fw_buffer(ptdev, perf);
+	if (ret) {
+		drm_err(&ptdev->base,
+			"Failed to setup FW ring buffer, disabling perf.");
+		return -EOPNOTSUPP;
+	}
+
+	sample = drmm_kzalloc(&ptdev->base, ptdev->perf_info.sample_size, GFP_KERNEL);
+	if (!sample)
+		return -ENOMEM;
+
+	em = drmm_kzalloc(&ptdev->base, sizeof(*em), GFP_KERNEL);
+	if (!em)
+		return -ENOMEM;
+
+	bo = panthor_kernel_bo_create(ptdev, panthor_fw_vm(ptdev),
+				      perf->fw_sample_size * perf_ringbuf_slots,
+				      DRM_PANTHOR_BO_NO_MMAP,
+				      DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC |
+				      DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED,
+				      PANTHOR_VM_KERNEL_AUTO_VA,
+				      "perf_ringbuf");
+
+	if (IS_ERR_OR_NULL(bo))
+		return IS_ERR(bo) ? PTR_ERR(bo) : -ENOMEM;
+
+	ret = panthor_kernel_bo_vmap(bo);
+	if (ret)
+		goto cleanup_bo;
+
+	perf->sample_workq = alloc_workqueue("panthor_perf_wq", WQ_HIGHPRI | WQ_UNBOUND, 1);
+	if (!perf->sample_workq) {
+		ret = -ENOMEM;
+		goto cleanup_vmap;
+	}
+
+	ret = drmm_add_action_or_reset(&ptdev->base, __drmm_workqueue_release,
+				       perf->sample_workq);
+	if (ret)
+		goto cleanup_vmap;
+
+	INIT_WORK(&perf->sample_work, panthor_perf_sample_worker);
+	perf->rb = bo;
+	perf->sample = sample;
+	perf->sample_slots = perf_ringbuf_slots;
+	perf->em = em;
+	perf->last_fw_sample_time = ktime_get_raw_ns();
+
+	INIT_LIST_HEAD(&perf->session_list);
+
 	drm_info(&ptdev->base, "Performance counter subsystem initialized");
 
 	perf->info = &ptdev->perf_info;
+	perf->ptdev = ptdev;
 	ptdev->perf = perf;
 
 	return ret;
+
+cleanup_vmap:
+	panthor_kernel_bo_vunmap(bo);
+
+cleanup_bo:
+	panthor_kernel_bo_destroy(bo);
+
+	return ret;
+}
+
+typedef bool (*session_state_cb)(struct panthor_perf_session *session);
+
+/**
+ * panthor_perf_request_sync - Request synchronous sample causing
+ * @perf: Perf struct
+ * @session: Target session
+ * @cb: Callback for the expected state of the session at the end of a sample.
+ *
+ * Perform a synchronous sample for scenarios where the sample must complete for the user to
+ * unambiguously know the state of the session. The two cases where this may be used are:
+ * - starting a new session (initial samples): any data from before this sample must not show up
+ *   in a new sampling session.
+ * - stopping an existing session: the eventfd may already have been signalled, meaning the
+ *   user may not find out when the session is stopped and continue waiting for more samples.
+ *
+ * Return: 0 on success, -ETIMEDOUT on wait failure
+ */
+static int panthor_perf_request_sync(struct panthor_perf *perf,
+				     struct panthor_perf_session *session, session_state_cb cb)
+{
+	int ret;
+
+	perf_fw_request_sample(perf);
+
+	ret = wait_event_timeout(session->state_wq, cb(session),
+				 msecs_to_jiffies(1000));
+
+	return ret ? 0 : -ETIMEDOUT;
 }
 
 /**
@@ -473,6 +1453,7 @@ int panthor_perf_session_create(struct drm_file *file,
 
 	kref_init(&session->ref);
 	INIT_LIST_HEAD(&session->sessions);
+	init_waitqueue_head(&session->state_wq);
 	mutex_init(&session->state_lock);
 	session->control = control.vaddr;
 	*session->control = (struct drm_panthor_perf_ringbuf_control) { 0 };
@@ -483,18 +1464,32 @@ int panthor_perf_session_create(struct drm_file *file,
 	session->control_buf = control;
 	session->accum_idx = U32_MAX;
 
+	ret = pm_runtime_resume_and_get(ptdev->base.dev);
+	if (ret)
+		goto cleanup_obj;
+
+	ret = panthor_perf_session_add(ptdev->perf, session, create_args->block_set);
+	if (ret)
+		goto cleanup_pm;
+
 	ret = xa_alloc_cyclic(&pfile->sessions->xa, &session_id, session,
-			      pfile->sessions->session_range,
-			      &pfile->sessions->next, GFP_KERNEL);
+			pfile->sessions->session_range,
+			&pfile->sessions->next, GFP_KERNEL);
 	if (ret < 0) {
 		drm_err(&ptdev->base, "File session limit exceeded.\n");
 		ret = -EBUSY;
-		goto cleanup_obj;
+		goto cleanup_tracked;
 	}
 
 	*sid = session_id;
 	return 0;
 
+cleanup_tracked:
+	panthor_perf_session_remove(pfile->ptdev->perf, session);
+
+cleanup_pm:
+	pm_runtime_put(ptdev->base.dev);
+
 cleanup_obj:
 	session_put(session);
 	return ret;
@@ -524,26 +1519,38 @@ static int session_try_stop(struct panthor_perf_session *session, u64 user_data)
 	if (session_is_starting(session))
 		return -EBUSY;
 
+	if (session->pending_sample)
+		return -EBUSY;
+
+	if (!session_has_space(session, 1))
+		return -EBUSY;
+
 	if (session_is_stopping(session))
 		return 1;
 
 	session_set_state(session, PANTHOR_PERF_SESSION_STOPPING);
+	session->sample_start_ns = ktime_get_raw_ns();
 	session->user_data = user_data;
 
 	return 1;
 }
 
+static bool session_is_stopped_locked(struct panthor_perf_session *session)
+{
+	guard(mutex)(&session->state_lock);
+
+	return session_is_stopped(session);
+}
+
 static int session_stop(struct panthor_perf *perf, struct panthor_perf_session *session,
 			u64 user_data)
 {
 	int ret;
-
 	ret = session_try_stop(session, user_data);
 	if (ret <= 0)
 		return ret;
 
-	/* TODO Calls to the FW interface will go here in later patches. */
-	return 0;
+	return panthor_perf_request_sync(perf, session, session_is_stopped_locked);
 }
 
 static int session_try_start(struct panthor_perf_session *session, u64 user_data)
@@ -560,11 +1567,20 @@ static int session_try_start(struct panthor_perf_session *session, u64 user_data
 		return 0;
 
 	session_set_state(session, PANTHOR_PERF_SESSION_STARTING);
+	session->overflow = false;
+	session->sample_start_ns = ktime_get_raw_ns();
 	session->user_data = user_data;
 
 	return 1;
 }
 
+static bool session_is_active_locked(struct panthor_perf_session *session)
+{
+	guard(mutex)(&session->state_lock);
+
+	return session_is_active(session);
+}
+
 static int session_start(struct panthor_perf *perf, struct panthor_perf_session *session,
 			 u64 user_data)
 {
@@ -574,8 +1590,7 @@ static int session_start(struct panthor_perf *perf, struct panthor_perf_session
 	if (ret <= 0)
 		return ret;
 
-	/* TODO Calls to the FW interface will go here in later patches. */
-	return 0;
+	return panthor_perf_request_sync(perf, session, session_is_active_locked);
 }
 
 static int session_try_sample(struct panthor_perf_session *session, u64 user_data)
@@ -591,7 +1606,20 @@ static int session_try_sample(struct panthor_perf_session *session, u64 user_dat
 	if (session_has_overflow(session))
 		return -EACCES;
 
+	if (session->pending_sample)
+		return -EBUSY;
+
+	/*
+	 * Must have at least two slots remaining in the ringbuffer to sample: one for
+	 * the current sample, and one for a stop sample, since a stop command should
+	 * always be acknowledged by taking a final sample and stopping the session.
+	 */
+	if (!session_has_space(session, 2))
+		return -ENOSPC;
+
 	session->user_data = user_data;
+	session->sample_start_ns = ktime_get_raw_ns();
+	session->pending_sample = true;
 
 	return 1;
 }
@@ -599,14 +1627,14 @@ static int session_try_sample(struct panthor_perf_session *session, u64 user_dat
 static int session_sample(struct panthor_perf *perf, struct panthor_perf_session *session,
 			  u64 user_data)
 {
-
 	int ret;
 
 	ret = session_try_sample(session, user_data);
 	if (ret <= 0)
 		return ret;
 
-	/* TODO Calls to the FW interface will go here in later patches. */
+	perf_fw_request_sample(perf);
+
 	return 0;
 }
 
@@ -651,9 +1679,14 @@ int panthor_perf_session_destroy(struct panthor_file *pfile, u32 handle)
 			return ret;
 
 		xa_erase(&pfile->sessions->xa, handle);
+		ret = perf_disable(perf);
+
+		scoped_guard(mutex, &perf->session_lock)
+			perf_detach_session(perf, session);
 	}
 
 	session_put(session);
+	pm_runtime_put_sync(perf->ptdev->base.dev);
 
 	return ret;
 }
@@ -774,11 +1807,22 @@ int panthor_perf_session_sample(struct panthor_file *pfile, u32 handle, u64 user
  */
 void panthor_perf_unplug(struct panthor_device *ptdev)
 {
+	int ret;
 	struct panthor_perf *perf = ptdev->perf;
 
 	if (!perf)
 		return;
 
+	scoped_guard(mutex, &perf->op_lock)
+		ret = perf_disable(perf);
+	if (ret)
+		drm_err(&perf->ptdev->base, "Failed to disable performance counter sampling!");
+
+	disable_work_sync(&perf->sample_work);
+
+	panthor_kernel_bo_vunmap(perf->rb);
+	panthor_kernel_bo_destroy(perf->rb);
+
 	ptdev->perf = NULL;
 }
 
@@ -809,9 +1853,14 @@ void panthor_perf_pool_destroy(struct panthor_file *pfile)
 	if (IS_ERR_OR_NULL(pool))
 		return;
 
+	if (!pfile->ptdev->perf)
+		return;
+
 	xa_for_each(&pool->xa, handle, session) {
 		xa_erase(&pool->xa, handle);
+		panthor_perf_session_remove(pfile->ptdev->perf, session);
 		session_put(session);
+		pm_runtime_put_sync(pfile->ptdev->base.dev);
 	}
 
 	xa_destroy(&pool->xa);
diff --git a/drivers/gpu/drm/panthor/panthor_perf.h b/drivers/gpu/drm/panthor/panthor_perf.h
index 641a88edd75a..0e1172dbae39 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.h
+++ b/drivers/gpu/drm/panthor/panthor_perf.h
@@ -28,5 +28,7 @@ int panthor_perf_session_start(struct panthor_file *pfile, u32 handle, u64 user_
 int panthor_perf_session_stop(struct panthor_file *pfile, unsigned int sid, u64 user_data);
 int panthor_perf_session_sample(struct panthor_file *pfile, unsigned int handle, u64 user_data);
 
+void panthor_perf_report_irq(struct panthor_device *ptdev, u32 status);
+
 #endif /* __PANTHOR_PERF_H__ */
 

-- 
2.43.0

[PATCH v7 7/8] drm/panthor: Add suspend, resume and reset handling

[Lukas Zapolskas]

The sampler must disable and re-enable counter sampling around
suspends, and must re-program the FW interface after a reset to
avoid losing data. Additionally, if a request was pending before the
reset and it was not acked, the reset will have cleared it, so we toggle
the flag and re-request a sample.

Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 drivers/gpu/drm/panthor/panthor_device.c |  8 ++-
 drivers/gpu/drm/panthor/panthor_perf.c   | 83 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/panthor/panthor_perf.h   |  6 +++
 3 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/panthor/panthor_device.c b/drivers/gpu/drm/panthor/panthor_device.c
index 535383ef629f..c842480472ac 100644
--- a/drivers/gpu/drm/panthor/panthor_device.c
+++ b/drivers/gpu/drm/panthor/panthor_device.c
@@ -138,6 +138,7 @@ static void panthor_device_reset_work(struct work_struct *work)
 	if (!drm_dev_enter(&ptdev->base, &cookie))
 		return;
 
+	panthor_perf_pre_reset(ptdev);
 	panthor_sched_pre_reset(ptdev);
 	panthor_fw_pre_reset(ptdev, true);
 	panthor_mmu_pre_reset(ptdev);
@@ -147,6 +148,8 @@ static void panthor_device_reset_work(struct work_struct *work)
 	ret = panthor_fw_post_reset(ptdev);
 	atomic_set(&ptdev->reset.pending, 0);
 	panthor_sched_post_reset(ptdev, ret != 0);
+	if (!ret)
+		panthor_perf_post_reset(ptdev);
 	drm_dev_exit(cookie);
 
 	if (ret) {
@@ -539,8 +542,10 @@ int panthor_device_resume(struct device *dev)
 			ret = panthor_device_resume_hw_components(ptdev);
 		}
 
-		if (!ret)
+		if (!ret) {
 			panthor_sched_resume(ptdev);
+			panthor_perf_resume(ptdev);
+		}
 
 		drm_dev_exit(cookie);
 
@@ -604,6 +609,7 @@ int panthor_device_suspend(struct device *dev)
 		/* We prepare everything as if we were resetting the GPU.
 		 * The end of the reset will happen in the resume path though.
 		 */
+		panthor_perf_suspend(ptdev);
 		panthor_sched_suspend(ptdev);
 		panthor_fw_suspend(ptdev);
 		panthor_mmu_suspend(ptdev);
diff --git a/drivers/gpu/drm/panthor/panthor_perf.c b/drivers/gpu/drm/panthor/panthor_perf.c
index c66ad658a95d..28b02003f9f9 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.c
+++ b/drivers/gpu/drm/panthor/panthor_perf.c
@@ -1798,6 +1798,57 @@ int panthor_perf_session_sample(struct panthor_file *pfile, u32 handle, u64 user
 	return err;
 }
 
+/**
+ * panthor_perf_suspend - Prepare the performance counter subsystem for system suspend.
+ * @ptdev: Panthor device.
+ *
+ * Indicate to the performance counters that the system is suspending.
+ *
+ * This function must not be used to handle MCU power state transitions: just before MCU goes
+ * from on to any inactive state, an automatic sample will be performed by the firmware, and
+ * the performance counter firmware state will be restored on warm boot.
+ *
+ */
+void panthor_perf_suspend(struct panthor_device *ptdev)
+{
+	struct panthor_perf *perf = ptdev->perf;
+	int ret;
+
+	if (!perf)
+		return;
+
+	guard(mutex)(&perf->op_lock);
+
+	ret = perf_disable(perf);
+
+	if (ret)
+		drm_warn(&ptdev->base, "Could not stop sampling before suspend, err = %d", ret);
+}
+
+/**
+ * panthor_perf_resume - Resume the performance counter subsystem after system resumption.
+ * @ptdev: Panthor device.
+ *
+ * Indicate to the performance counters that the system has resumed. This must not be used
+ * to handle MCU state transitions, for the same reasons as detailed in the kerneldoc for
+ * @panthor_perf_suspend.
+ */
+void panthor_perf_resume(struct panthor_device *ptdev)
+{
+	struct panthor_perf *perf = ptdev->perf;
+	int ret;
+
+	if (!perf)
+		return;
+
+	guard(mutex)(&perf->op_lock);
+
+	ret = perf_enable(perf);
+
+	if (ret)
+		drm_warn(&ptdev->base, "Could not resume sampling, err = %d", ret);
+}
+
 /**
  * panthor_perf_unplug - Terminate the performance counter subsystem.
  * @ptdev: Panthor device.
@@ -1868,3 +1919,35 @@ void panthor_perf_pool_destroy(struct panthor_file *pfile)
 
 	pfile->sessions = NULL;
 }
+
+void panthor_perf_pre_reset(struct panthor_device *ptdev)
+{
+	if (!ptdev->perf)
+		return;
+
+	guard(mutex)(&ptdev->perf->op_lock);
+
+	perf_disable(ptdev->perf);
+}
+
+void panthor_perf_post_reset(struct panthor_device *ptdev)
+{
+	bool rerequest = false;
+
+	if (!ptdev->perf)
+		return;
+
+	guard(mutex)(&ptdev->perf->op_lock);
+
+	perf_enable(ptdev->perf);
+
+	scoped_guard(spinlock_irqsave, &ptdev->perf->pend_lock) {
+		rerequest = ptdev->perf->fw_sample_requested;
+
+		if (rerequest)
+			ptdev->perf->fw_sample_requested = false;
+	}
+
+	if (rerequest)
+		perf_fw_request_sample(ptdev->perf);
+}
diff --git a/drivers/gpu/drm/panthor/panthor_perf.h b/drivers/gpu/drm/panthor/panthor_perf.h
index 0e1172dbae39..e4a323fe6f0d 100644
--- a/drivers/gpu/drm/panthor/panthor_perf.h
+++ b/drivers/gpu/drm/panthor/panthor_perf.h
@@ -18,6 +18,8 @@ int panthor_perf_pool_create(struct panthor_file *pfile);
 void panthor_perf_pool_destroy(struct panthor_file *pfile);
 
 int panthor_perf_init(struct panthor_device *ptdev);
+void panthor_perf_suspend(struct panthor_device *ptdev);
+void panthor_perf_resume(struct panthor_device *ptdev);
 void panthor_perf_unplug(struct panthor_device *ptdev);
 
 int panthor_perf_session_create(struct drm_file *file,
@@ -30,5 +32,9 @@ int panthor_perf_session_sample(struct panthor_file *pfile, unsigned int handle,
 
 void panthor_perf_report_irq(struct panthor_device *ptdev, u32 status);
 
+void panthor_perf_pre_reset(struct panthor_device *ptdev);
+
+void panthor_perf_post_reset(struct panthor_device *ptdev);
+
 #endif /* __PANTHOR_PERF_H__ */
 

-- 
2.43.0

[PATCH v7 8/8] drm/panthor: Expose the panthor perf ioctls

[Lukas Zapolskas]

This patch implements the PANTHOR_PERF_SESSION ioctl series.

The minor version is bumped to indicate that the feature is now
supported.

Signed-off-by: Lukas Zapolskas <lukas.zapolskas@arm.com>
---
 drivers/gpu/drm/panthor/panthor_drv.c | 117 +++++++++++++++++++++++++++++++++-
 1 file changed, 115 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c
index 3e62142d7ebd..5ca41cb9c1ba 100644
--- a/drivers/gpu/drm/panthor/panthor_drv.c
+++ b/drivers/gpu/drm/panthor/panthor_drv.c
@@ -184,7 +184,12 @@ panthor_get_uobj_array(const struct drm_panthor_obj_array *in, u32 min_stride,
 		 PANTHOR_UOBJ_DECL(struct drm_panthor_queue_create, ringbuf_size), \
 		 PANTHOR_UOBJ_DECL(struct drm_panthor_vm_bind_op, syncs), \
 		 PANTHOR_UOBJ_DECL(struct drm_panthor_bo_sync_op, size), \
-		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_info, shader_blocks))
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_info, shader_blocks), \
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_session_create, shader_enable_mask), \
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_session_start, user_data), \
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_session_stop, user_data), \
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_session_sample, user_data), \
+		 PANTHOR_UOBJ_DECL(struct drm_panthor_perf_session_destroy, handle))
 
 /**
  * PANTHOR_UOBJ_SET() - Copy a kernel object to a user object.
@@ -1603,6 +1608,103 @@ static int panthor_ioctl_bo_query_info(struct drm_device *ddev, void *data,
 	return 0;
 }
 
+static int panthor_ioctl_perf_session_create(struct drm_device *ddev, void *data,
+					     struct drm_file *file)
+{
+	int ret, cookie;
+	unsigned long sid = 0;
+	struct drm_panthor_perf_session_create *args = data;
+
+	if (!drm_dev_enter(file->minor->dev, &cookie))
+		return -ENODEV;
+
+	if (!mem_is_zero(args->pad, sizeof(args->pad)))
+		return -EINVAL;
+
+	ret = panthor_perf_session_create(file, args, &sid);
+	if (ret)
+		return ret;
+
+	drm_dev_exit(cookie);
+	return sid;
+}
+
+static int panthor_ioctl_perf_session_start(struct drm_device *ddev, void *data,
+					    struct drm_file *file)
+{
+	int cookie, ret;
+	struct panthor_file *pfile = file->driver_priv;
+	struct drm_panthor_perf_session_start *args = data;
+
+	if (!drm_dev_enter(file->minor->dev, &cookie))
+		return -ENODEV;
+
+	if (args->pad0)
+		return -EINVAL;
+
+	ret = panthor_perf_session_start(pfile, args->handle, args->user_data);
+
+	drm_dev_exit(cookie);
+	return ret;
+}
+
+static int panthor_ioctl_perf_session_sample(struct drm_device *ddev, void *data,
+					     struct drm_file *file)
+{
+	int cookie, ret;
+	struct panthor_file *pfile = file->driver_priv;
+	struct drm_panthor_perf_session_sample *args = data;
+
+	if (!drm_dev_enter(file->minor->dev, &cookie))
+		return -ENODEV;
+
+	if (args->pad0)
+		return -EINVAL;
+
+	ret =  panthor_perf_session_sample(pfile, args->handle, args->user_data);
+
+	drm_dev_exit(cookie);
+	return ret;
+}
+
+static int panthor_ioctl_perf_session_stop(struct drm_device *ddev, void *data,
+					   struct drm_file *file)
+{
+	int cookie, ret;
+	struct panthor_file *pfile = file->driver_priv;
+	struct drm_panthor_perf_session_stop *args = data;
+
+	if (!drm_dev_enter(file->minor->dev, &cookie))
+		return -ENODEV;
+
+	if (args->pad0)
+		return -EINVAL;
+
+	ret = panthor_perf_session_stop(pfile, args->handle, args->user_data);
+
+	drm_dev_exit(cookie);
+	return ret;
+}
+
+static int panthor_ioctl_perf_session_destroy(struct drm_device *ddev, void *data,
+					      struct drm_file *file)
+{
+	int cookie, ret;
+	struct panthor_file *pfile = file->driver_priv;
+	struct drm_panthor_perf_session_destroy *args = data;
+
+	if (!drm_dev_enter(file->minor->dev, &cookie))
+		return -ENODEV;
+
+	if (args->pad0)
+		return -EINVAL;
+
+	ret = panthor_perf_session_destroy(pfile, args->handle);
+
+	drm_dev_exit(cookie);
+	return ret;
+}
+
 static int
 panthor_open(struct drm_device *ddev, struct drm_file *file)
 {
@@ -1687,6 +1789,11 @@ static const struct drm_ioctl_desc panthor_drm_driver_ioctls[] = {
 	PANTHOR_IOCTL(SET_USER_MMIO_OFFSET, set_user_mmio_offset, DRM_RENDER_ALLOW),
 	PANTHOR_IOCTL(BO_SYNC, bo_sync, DRM_RENDER_ALLOW),
 	PANTHOR_IOCTL(BO_QUERY_INFO, bo_query_info, DRM_RENDER_ALLOW),
+	PANTHOR_IOCTL(PERF_SESSION_CREATE, perf_session_create, DRM_RENDER_ALLOW),
+	PANTHOR_IOCTL(PERF_SESSION_START, perf_session_start, DRM_RENDER_ALLOW),
+	PANTHOR_IOCTL(PERF_SESSION_SAMPLE, perf_session_sample, DRM_RENDER_ALLOW),
+	PANTHOR_IOCTL(PERF_SESSION_STOP, perf_session_stop, DRM_RENDER_ALLOW),
+	PANTHOR_IOCTL(PERF_SESSION_DESTROY, perf_session_destroy, DRM_RENDER_ALLOW),
 };
 
 static int panthor_mmap(struct file *filp, struct vm_area_struct *vma)
@@ -1807,6 +1914,12 @@ static void panthor_debugfs_init(struct drm_minor *minor)
  * - 1.8 - extends DEV_QUERY_TIMESTAMP_INFO with flags
  * - 1.9 - adds DRM_PANTHOR_DEV_QUERY_MMU_INFO query
  *       - adds DRM_PANTHOR_VM_BIND_OP_MAP_SPARSE flag
+ * - 1.10 - adds DEV_QUERY_PERF_INFO query
+ *        - adds PERF_SESSION_CREATE ioctl
+ *        - adds PERF_SESSION_START ioctl
+ *        - adds PERF_SESSION_SAMPLE ioctl
+ *        - adds PERF_SESSION_STOP ioctl
+ *        - adds PERF_SESSION_DESTROY ioctl
  */
 static const struct drm_driver panthor_drm_driver = {
 	.driver_features = DRIVER_RENDER | DRIVER_GEM | DRIVER_SYNCOBJ |
@@ -1820,7 +1933,7 @@ static const struct drm_driver panthor_drm_driver = {
 	.name = "panthor",
 	.desc = "Panthor DRM driver",
 	.major = 1,
-	.minor = 9,
+	.minor = 10,
 
 	.gem_prime_import_sg_table = panthor_gem_prime_import_sg_table,
 	.gem_prime_import = panthor_gem_prime_import,

-- 
2.43.0