[PATCH 0/4] Exynos 5422: Mali-T628 GPU compute fixes

[PATCH 0/4] Exynos 5422: Mali-T628 GPU compute fixes

[Young-Jun Hong]

These patches make the Mali-T628 MP6 GPU of the Exynos 5422 (Samsung
ARTIK 10, also Odroid-XU3/XU4) usable for OpenCL compute with panfrost
and Mesa Rusticl. They touch three subsystems and can be taken
independently:

  1/4 iommu: the GPU is not cache coherent on this SoC; its LPAE page
      tables must use inner shareable, or the first large GPU buffer
      after boot hangs the SoC.
  2/4 clk: VPLL rates for GPU operating points above 600 MHz. The
      ARTIK 10 device tree is not in mainline, and the mainline 5422
      boards stop at 600 MHz; they can use these rates once higher
      OPPs are added.
  3/4 drm/panfrost: the job affinity names only the shader cores that
      are powered.
  4/4 drm/panfrost (RFC): power both core groups and let a job opt in
      to all of them; the L2 caches of the two groups are not coherent,
      so the default stays on the first group.

1-3 were tested on the board (Linux 6.12.55 based) and cross-compiled
on 7.4-rc1. 4 is compile-tested on 7.4-rc1; the same logic, ported to the
board's 6.12-based panfrost module, was tested with the matching Mesa
change: jobs without the flag stay on the first group, jobs that opt in
use all six cores, and global atomics stay correct.
Background, test programs and results:
https://github.com/nirvanahyj-com/mali-t628-opencl/tree/v1

The analysis, the code and the changelogs were produced with Claude Code
(Anthropic), a coding assistant.

Young-Jun Hong (4):
  iommu/io-pgtable-arm: use inner shareable for Mali LPAE on a
    non-coherent GPU
  clk: samsung: exynos5420: add VPLL rates for 650, 700 and 750 MHz
  drm/panfrost: only put powered shader cores in the job affinity
  drm/panfrost: let jobs opt in to all core groups

 drivers/clk/samsung/clk-exynos5420.c    |  3 +++
 drivers/gpu/drm/panfrost/panfrost_drv.c |  7 ++++++-
 drivers/gpu/drm/panfrost/panfrost_gpu.c | 25 +++++++++++++++----------
 drivers/gpu/drm/panfrost/panfrost_gpu.h |  2 ++
 drivers/gpu/drm/panfrost/panfrost_job.c | 10 ++++++----
 drivers/iommu/io-pgtable-arm.c          |  7 +++++++
 include/uapi/drm/panfrost_drm.h         | 10 ++++++++++
 7 files changed, 49 insertions(+), 15 deletions(-)

-- 
2.47.3

[PATCH 1/4] iommu/io-pgtable-arm: use inner shareable for Mali LPAE on a non-coherent GPU

[Young-Jun Hong]

io-pgtable marks every ARM_MALI_LPAE PTE outer shareable (SH = 2). Outer
shareability is only meaningful when the GPU sits behind a coherent
interconnect, i.e. when the page-table walk is coherent (cfg.coherent_walk,
which panfrost sets from the dma-coherent DT property). The vendor kbase
driver on a non-coherent Exynos 5422 (COHERENCY_NONE) never uses outer
shareable: its PTEs are non-shareable or inner shareable.

On Exynos 5422 (Mali-T628 MP6, ARMv7 LPAE, not IO-coherent) the outer
shareable attribute hangs the whole SoC when a large GPU buffer is first
used under GPU bandwidth load and memory pressure: 0 of 22 runs survived
with SH = 2, 3 of 3 with inner shareable. Non-shareable also avoids the
hang but breaks global atomics between the two core groups, so use inner
shareable for a non-coherent Mali; coherent configurations are unchanged.
Sustained GPU copy bandwidth also rises from 6.3 to 7.7 GB/s.

Tested on Samsung ARTIK 10 (Exynos 5422) with panfrost and Mesa Rusticl:
hang reproducer clean, OpenCL accuracy suite 96/96, no GPU faults over
several days of use.

The analysis, the code and this changelog were produced with Claude
Code (Anthropic), a coding assistant.

Fixes: 7618e4790982 ("iommu/io-pgtable-arm: Improve attribute handling")
Cc: stable@vger.kernel.org
Signed-off-by: Young-Jun Hong <nirvanah@nirvanahyj.com>
---
 drivers/iommu/io-pgtable-arm.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c
index 476c0e25631a..0e7b2edd9ca1 100644
--- a/drivers/iommu/io-pgtable-arm.c
+++ b/drivers/iommu/io-pgtable-arm.c
@@ -565,6 +565,13 @@ static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
 	 */
 	if (prot & IOMMU_CACHE && data->iop.fmt != ARM_MALI_LPAE)
 		pte |= ARM_LPAE_PTE_SH_IS;
+	else if (data->iop.fmt == ARM_MALI_LPAE && !data->iop.cfg.coherent_walk)
+		/*
+		 * A non-coherent Mali must not mark its accesses outer
+		 * shareable; inner shareable keeps atomics consistent across
+		 * its core groups.
+		 */
+		pte |= ARM_LPAE_PTE_SH_IS;
 	else
 		pte |= ARM_LPAE_PTE_SH_OS;
 
-- 
2.47.3

[PATCH 2/4] clk: samsung: exynos5420: add VPLL rates for 650, 700 and 750 MHz

[Young-Jun Hong]

The VPLL drives the G3D (Mali) clock on Exynos 5420/5422. Its rate table
stops at 600 MHz, so GPU OPPs above 600 MHz cannot be reached:
clk_set_rate() silently keeps fout_vpll at 600 MHz.

Add the 650, 700 and 750 MHz rows (VCO 1300/1400/1500 MHz, using the same
P/M/S style as the APLL rows).

Tested on Samsung ARTIK 10 (Exynos 5422): with these rows the G3D devfreq
reaches 700 and 750 MHz, and the OpenCL accuracy suite passes 96/96 at
750 MHz with a board-specific 1.35 V OPP.

The analysis, the code and this changelog were produced with Claude
Code (Anthropic), a coding assistant.

Signed-off-by: Young-Jun Hong <nirvanah@nirvanahyj.com>
---
 drivers/clk/samsung/clk-exynos5420.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/clk/samsung/clk-exynos5420.c b/drivers/clk/samsung/clk-exynos5420.c
index 400a5c59815c..1b4b0ae16765 100644
--- a/drivers/clk/samsung/clk-exynos5420.c
+++ b/drivers/clk/samsung/clk-exynos5420.c
@@ -1451,6 +1451,9 @@ static const struct samsung_pll_rate_table exynos5420_epll_24mhz_tbl[] = {
 };
 
 static const struct samsung_pll_rate_table exynos5420_vpll_24mhz_tbl[] = {
+	PLL_35XX_RATE(24 * MHZ, 750000000U,  125, 2, 1),
+	PLL_35XX_RATE(24 * MHZ, 700000000U,  175, 3, 1),
+	PLL_35XX_RATE(24 * MHZ, 650000000U,  325, 6, 1),
 	PLL_35XX_RATE(24 * MHZ, 600000000U,  200, 2, 2),
 	PLL_35XX_RATE(24 * MHZ, 543000000U,  181, 2, 2),
 	PLL_35XX_RATE(24 * MHZ, 480000000U,  160, 2, 2),
-- 
2.47.3

[PATCH 3/4] drm/panfrost: only put powered shader cores in the job affinity

[Young-Jun Hong]

On GPUs with more than one L2 (core group), panfrost powers only the
first core group (panfrost_get_core_mask()), but the job affinity is
written from shader_present, i.e. it also names the cores of the second,
unpowered group (Mali-T628 MP6 on Exynos 5422: l2_present 0x11, 4+2
cores, affinity 0x3f while only 0x0f is powered).

Write the affinity from the same mask that was powered, so a job never
names cores that are off.

Tested on Samsung ARTIK 10 (Exynos 5422), Mali-T628 MP6.

The analysis, the code and this changelog were produced with Claude
Code (Anthropic), a coding assistant.

Fixes: 6e55d273708b ("drm/panfrost: initial dual core group GPUs support")
Signed-off-by: Young-Jun Hong <nirvanah@nirvanahyj.com>
---
 drivers/gpu/drm/panfrost/panfrost_gpu.c | 6 ++++++
 drivers/gpu/drm/panfrost/panfrost_gpu.h | 1 +
 drivers/gpu/drm/panfrost/panfrost_job.c | 2 +-
 3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/panfrost/panfrost_gpu.c b/drivers/gpu/drm/panfrost/panfrost_gpu.c
index 6081b27829e1..420d224276a5 100644
--- a/drivers/gpu/drm/panfrost/panfrost_gpu.c
+++ b/drivers/gpu/drm/panfrost/panfrost_gpu.c
@@ -452,6 +452,12 @@ static u64 panfrost_get_core_mask(struct panfrost_device *pfdev)
 	return core_mask;
 }
 
+/* cores that panfrost_gpu_power_on() powers */
+u64 panfrost_powered_core_mask(struct panfrost_device *pfdev)
+{
+	return pfdev->features.shader_present & panfrost_get_core_mask(pfdev);
+}
+
 void panfrost_gpu_power_on(struct panfrost_device *pfdev)
 {
 	int ret;
diff --git a/drivers/gpu/drm/panfrost/panfrost_gpu.h b/drivers/gpu/drm/panfrost/panfrost_gpu.h
index b4fef11211d5..78a6270b7fcb 100644
--- a/drivers/gpu/drm/panfrost/panfrost_gpu.h
+++ b/drivers/gpu/drm/panfrost/panfrost_gpu.h
@@ -13,6 +13,7 @@ void panfrost_gpu_fini(struct panfrost_device *pfdev);
 u32 panfrost_gpu_get_latest_flush_id(struct panfrost_device *pfdev);
 
 int panfrost_gpu_soft_reset(struct panfrost_device *pfdev);
+u64 panfrost_powered_core_mask(struct panfrost_device *pfdev);
 void panfrost_gpu_power_on(struct panfrost_device *pfdev);
 void panfrost_gpu_power_off(struct panfrost_device *pfdev);
 void panfrost_gpu_suspend_irq(struct panfrost_device *pfdev);
diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c
index 4ddf4b722ddf..33da86587db7 100644
--- a/drivers/gpu/drm/panfrost/panfrost_job.c
+++ b/drivers/gpu/drm/panfrost/panfrost_job.c
@@ -143,7 +143,7 @@ static void panfrost_job_write_affinity(struct panfrost_device *pfdev,
 	 * Eventually we may need to support tiler only jobs and h/w with
 	 * multiple (2) coherent core groups
 	 */
-	affinity = pfdev->features.shader_present;
+	affinity = panfrost_powered_core_mask(pfdev);
 
 	job_write(pfdev, JS_AFFINITY_NEXT_LO(js), lower_32_bits(affinity));
 	job_write(pfdev, JS_AFFINITY_NEXT_HI(js), upper_32_bits(affinity));
-- 
2.47.3

[PATCH 4/4] drm/panfrost: let jobs opt in to all core groups

[Young-Jun Hong]

Mali-T628 MP6/MP8 (Exynos 5420/5422/5800) have two core groups, each with
its own L2 cache, and the caches are not coherent with each other.
panfrost powers only the first core group, so on Exynos 5422 four of the
six shader cores are used.

Power every core group and let a job ask for all of them with a new
requirement, PANFROST_JD_REQ_ALL_CORE_GROUPS. A job without it keeps
running on the first core group only, so existing userspace, which may
use global atomics or other cross-workgroup communication, is not
affected. Userspace sets the bit for jobs that need no memory coherence
between their work groups. Userspace finds out whether the kernel knows
the bit with a new parameter, DRM_PANFROST_PARAM_SUPPORTED_JD_REQS, which
returns the mask of job requirements the kernel accepts.

RFC: compile-tested on top of 7.4-rc1. The same logic, ported to the
6.12-based panfrost module of an ARTIK 10 (Exynos 5422), was tested on
the board with the matching Mesa change: jobs without the requirement
ran on the first core group only (a compute kernel without atomics was
as fast as one with them), jobs that set it used all six cores (1.43x
faster than the first group alone), and the OpenCL atomics tests passed.
Global atomics from work groups on both core groups lose 25-50 % of the
updates, which is why the default stays on the first group. Open
question: whether powering the second group should depend on a
userspace opt-in as well (power). The Mesa side is posted as a draft
merge request:
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/45971

The analysis, the code and this changelog were produced with Claude
Code (Anthropic), a coding assistant.

Signed-off-by: Young-Jun Hong <nirvanah@nirvanahyj.com>
---
 drivers/gpu/drm/panfrost/panfrost_drv.c |  7 ++++++-
 drivers/gpu/drm/panfrost/panfrost_gpu.c | 23 +++++++++++------------
 drivers/gpu/drm/panfrost/panfrost_gpu.h |  1 +
 drivers/gpu/drm/panfrost/panfrost_job.c |  8 +++++---
 include/uapi/drm/panfrost_drm.h         | 10 ++++++++++
 5 files changed, 33 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c b/drivers/gpu/drm/panfrost/panfrost_drv.c
index 7c348a0bf1ee..1d979f394ef4 100644
--- a/drivers/gpu/drm/panfrost/panfrost_drv.c
+++ b/drivers/gpu/drm/panfrost/panfrost_drv.c
@@ -27,7 +27,8 @@
 #include "panfrost_gpu.h"
 #include "panfrost_perfcnt.h"
 
-#define JOB_REQUIREMENTS (PANFROST_JD_REQ_FS | PANFROST_JD_REQ_CYCLE_COUNT)
+#define JOB_REQUIREMENTS (PANFROST_JD_REQ_FS | PANFROST_JD_REQ_CYCLE_COUNT | \
+			  PANFROST_JD_REQ_ALL_CORE_GROUPS)
 
 static bool unstable_ioctls;
 module_param_unsafe(unstable_ioctls, bool, 0600);
@@ -114,6 +115,10 @@ static int panfrost_ioctl_get_param(struct drm_device *ddev, void *data, struct
 #endif
 		break;
 
+	case DRM_PANFROST_PARAM_SUPPORTED_JD_REQS:
+		param->value = JOB_REQUIREMENTS;
+		break;
+
 	case DRM_PANFROST_PARAM_ALLOWED_JM_CTX_PRIORITIES:
 		param->value = BIT(PANFROST_JM_CTX_PRIORITY_LOW) |
 			       BIT(PANFROST_JM_CTX_PRIORITY_MEDIUM);
diff --git a/drivers/gpu/drm/panfrost/panfrost_gpu.c b/drivers/gpu/drm/panfrost/panfrost_gpu.c
index 420d224276a5..e880769dfd72 100644
--- a/drivers/gpu/drm/panfrost/panfrost_gpu.c
+++ b/drivers/gpu/drm/panfrost/panfrost_gpu.c
@@ -429,7 +429,8 @@ unsigned long long panfrost_timestamp_read(struct panfrost_device *pfdev)
 	return ((u64)hi << 32) | lo;
 }
 
-static u64 panfrost_get_core_mask(struct panfrost_device *pfdev)
+/* cores of the first core group: the cores a job uses by default */
+u64 panfrost_first_core_group_mask(struct panfrost_device *pfdev)
 {
 	u64 core_mask;
 
@@ -437,7 +438,6 @@ static u64 panfrost_get_core_mask(struct panfrost_device *pfdev)
 		return U64_MAX;
 
 	/*
-	 * Only support one core group now.
 	 * ~(l2_present - 1) unsets all bits in l2_present except
 	 * the bottom bit. (l2_present - 2) has all the bits in
 	 * the first core group set. AND them together to generate
@@ -445,39 +445,38 @@ static u64 panfrost_get_core_mask(struct panfrost_device *pfdev)
 	 */
 	core_mask = ~(pfdev->features.l2_present - 1) &
 		     (pfdev->features.l2_present - 2);
-	dev_info_once(pfdev->base.dev, "using only 1st core group (%lu cores from %lu)\n",
+	dev_info_once(pfdev->base.dev, "jobs use the 1st core group (%lu cores from %lu) unless they ask for all\n",
 		      hweight64(core_mask),
 		      hweight64(pfdev->features.shader_present));
 
 	return core_mask;
 }
 
-/* cores that panfrost_gpu_power_on() powers */
+/* cores that panfrost_gpu_power_on() powers: all of them */
 u64 panfrost_powered_core_mask(struct panfrost_device *pfdev)
 {
-	return pfdev->features.shader_present & panfrost_get_core_mask(pfdev);
+	return pfdev->features.shader_present;
 }
 
 void panfrost_gpu_power_on(struct panfrost_device *pfdev)
 {
 	int ret;
 	u32 val;
-	u64 core_mask;
 
 	panfrost_gpu_init_quirks(pfdev);
-	core_mask = panfrost_get_core_mask(pfdev);
+	/* prints which cores jobs use by default */
+	panfrost_first_core_group_mask(pfdev);
 
-	gpu_write(pfdev, L2_PWRON_LO, pfdev->features.l2_present & core_mask);
+	gpu_write(pfdev, L2_PWRON_LO, pfdev->features.l2_present);
 	ret = readl_relaxed_poll_timeout(pfdev->iomem + L2_READY_LO,
-		val, val == (pfdev->features.l2_present & core_mask),
+		val, val == pfdev->features.l2_present,
 		10, 20000);
 	if (ret)
 		dev_err(pfdev->base.dev, "error powering up gpu L2");
 
-	gpu_write(pfdev, SHADER_PWRON_LO,
-		  pfdev->features.shader_present & core_mask);
+	gpu_write(pfdev, SHADER_PWRON_LO, pfdev->features.shader_present);
 	ret = readl_relaxed_poll_timeout(pfdev->iomem + SHADER_READY_LO,
-		val, val == (pfdev->features.shader_present & core_mask),
+		val, val == pfdev->features.shader_present,
 		10, 20000);
 	if (ret)
 		dev_err(pfdev->base.dev, "error powering up gpu shader");
diff --git a/drivers/gpu/drm/panfrost/panfrost_gpu.h b/drivers/gpu/drm/panfrost/panfrost_gpu.h
index 78a6270b7fcb..5c6364e4229a 100644
--- a/drivers/gpu/drm/panfrost/panfrost_gpu.h
+++ b/drivers/gpu/drm/panfrost/panfrost_gpu.h
@@ -14,6 +14,7 @@ u32 panfrost_gpu_get_latest_flush_id(struct panfrost_device *pfdev);
 
 int panfrost_gpu_soft_reset(struct panfrost_device *pfdev);
 u64 panfrost_powered_core_mask(struct panfrost_device *pfdev);
+u64 panfrost_first_core_group_mask(struct panfrost_device *pfdev);
 void panfrost_gpu_power_on(struct panfrost_device *pfdev);
 void panfrost_gpu_power_off(struct panfrost_device *pfdev);
 void panfrost_gpu_suspend_irq(struct panfrost_device *pfdev);
diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c
index 33da86587db7..65634202fd82 100644
--- a/drivers/gpu/drm/panfrost/panfrost_job.c
+++ b/drivers/gpu/drm/panfrost/panfrost_job.c
@@ -139,11 +139,13 @@ static void panfrost_job_write_affinity(struct panfrost_device *pfdev,
 	u64 affinity;
 
 	/*
-	 * Use all cores for now.
-	 * Eventually we may need to support tiler only jobs and h/w with
-	 * multiple (2) coherent core groups
+	 * The L2 caches of the core groups are not coherent with each other:
+	 * only a job that asks for it runs on more than the first core group.
+	 * Eventually we may need to support tiler only jobs.
 	 */
 	affinity = panfrost_powered_core_mask(pfdev);
+	if (!(requirements & PANFROST_JD_REQ_ALL_CORE_GROUPS))
+		affinity &= panfrost_first_core_group_mask(pfdev);
 
 	job_write(pfdev, JS_AFFINITY_NEXT_LO(js), lower_32_bits(affinity));
 	job_write(pfdev, JS_AFFINITY_NEXT_HI(js), upper_32_bits(affinity));
diff --git a/include/uapi/drm/panfrost_drm.h b/include/uapi/drm/panfrost_drm.h
index 7831c48c59c0..e61212494c4a 100644
--- a/include/uapi/drm/panfrost_drm.h
+++ b/include/uapi/drm/panfrost_drm.h
@@ -51,6 +51,15 @@ extern "C" {
 
 #define PANFROST_JD_REQ_FS (1 << 0)
 #define PANFROST_JD_REQ_CYCLE_COUNT (1 << 1)
+/*
+ * The job may run on the shader cores of every core group. On GPUs with
+ * more than one L2 cache (Mali-T628 MP6/MP8) the caches of the core groups
+ * are not coherent with each other, so a job runs on the first core group
+ * only, unless it sets this because its work groups do not need memory
+ * coherence between each other (no global atomics, no cross-workgroup
+ * communication). Ignored on GPUs with one core group.
+ */
+#define PANFROST_JD_REQ_ALL_CORE_GROUPS (1 << 2)
 /**
  * struct drm_panfrost_submit - ioctl argument for submitting commands to the 3D
  * engine.
@@ -236,6 +245,7 @@ enum drm_panfrost_param {
 	DRM_PANFROST_PARAM_SYSTEM_TIMESTAMP_FREQUENCY,
 	DRM_PANFROST_PARAM_ALLOWED_JM_CTX_PRIORITIES,
 	DRM_PANFROST_PARAM_SELECTED_COHERENCY,
+	DRM_PANFROST_PARAM_SUPPORTED_JD_REQS,
 };
 
 enum drm_panfrost_gpu_coherency {
-- 
2.47.3