[PATCH v2 0/6] drm/imagination: Add async VM_BIND with sparse mappings

[PATCH v2 0/6] drm/imagination: Add async VM_BIND with sparse mappings

[Gyeyoung Baek]

Userspace MR:
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43794

This series is for Vulkan sparse binding.

- Patches 1-2 switch the GPUVM to immediate mode.
- Patch 3 moves the existing map and unmap paths onto a single bind
  entry point.
- Patch 4 adds the VM_BIND ioctl.
- Patch 5 adds asynchronous requests.
- Patch 6 adds sparse mappings.

Patches 1-2 were sent separately [1]. They conflict with drm-misc-next,
so they are rebased and included here.

Testing:
Tested on BeagleY-AI.
- The existing IGT pvr_vm_map passes without regressions.
- dEQP-VK.sparse_resources passes for the sparse features the Mesa MR
  above exposes.

[1] https://lore.kernel.org/r/20261011-pvr-gpuvm-immediate-v2-0-744ddc6644c3@gmail.com

Signed-off-by: Gyeyoung Baek <gye976@gmail.com>
---
Changes in v2:
- New patch 3, "Route map and unmap through pvr_vm_bind()".
- Split v1 3/4 into the ioctl (4) and the asynchronous mode (5).
- Bump the driver interface version to 1.2, as 1.1 is already taken by
  another patch.
- Link to v1: https://lore.kernel.org/r/20260817-pvr-vm-bind-v1-0-0a0f21be7d38@gmail.com

---
Gyeyoung Baek (6):
      drm/imagination: Switch the GPUVM to immediate mode
      drm/imagination: Stop locking reservations around map and unmap
      drm/imagination: Route map and unmap through pvr_vm_bind()
      drm/imagination: Add VM_BIND ioctl
      drm/imagination: Add asynchronous VM_BIND
      drm/imagination: Add MAP_SPARSE to VM_BIND

 Documentation/gpu/imagination/uapi.rst     |  15 +
 drivers/gpu/drm/imagination/pvr_device.h   |  10 +
 drivers/gpu/drm/imagination/pvr_drv.c      | 331 ++++++++++++
 drivers/gpu/drm/imagination/pvr_drv.h      |   4 +-
 drivers/gpu/drm/imagination/pvr_fw_meta.c  |   6 +-
 drivers/gpu/drm/imagination/pvr_fw_riscv.c |   6 +-
 drivers/gpu/drm/imagination/pvr_job.c      |   6 +
 drivers/gpu/drm/imagination/pvr_mmu.c      |   2 +-
 drivers/gpu/drm/imagination/pvr_mmu.h      |   1 +
 drivers/gpu/drm/imagination/pvr_queue.c    |   3 +
 drivers/gpu/drm/imagination/pvr_sync.c     | 146 +++++-
 drivers/gpu/drm/imagination/pvr_sync.h     |  13 +
 drivers/gpu/drm/imagination/pvr_vm.c       | 781 ++++++++++++++++++++++-------
 drivers/gpu/drm/imagination/pvr_vm.h       |  63 ++-
 include/uapi/drm/pvr_drm.h                 | 129 +++++
 15 files changed, 1303 insertions(+), 213 deletions(-)
---
base-commit: 2d5a8313d9cf989b8f53a5689a96e891bf99fb2c
change-id: 20260817-pvr-vm-bind-1ce4f09a1774
prerequisite-message-id: <20261015-pvr-unmap-all-loop-v1-1-54c676d41269@gmail.com>
prerequisite-patch-id: 5188d944e90112e4cf2678e75255c7a91434ad09

Best regards,
--  
Gyeyoung Baek <gye976@gmail.com>

[PATCH v2 1/6] drm/imagination: Switch the GPUVM to immediate mode

[Gyeyoung Baek]

Use DRM_GPUVM_IMMEDIATE_MODE so that each GEM's GPUVA list is protected
by its gpuva.lock instead of its dma_resv. This prepares for
asynchronous VM_BIND, whose run_job() will update the GPUVA lists
before signalling its fence. That section imposes three rules:

1. No dma_resv: another thread may hold it while waiting for the job's
   fence, directly or through a dependency.
   -> Protect the lists with gpuva.lock, and take it around
      drm_gpuva_link().

2. No allocation under a lock run_job() takes: reclaim may wait for a
   fence that depends on the job.
   -> Allocate the vm_bo before taking gpuva.lock:
      drm_gpuvm_bo_obtain_locked() becomes drm_gpuvm_bo_create() +
      drm_gpuvm_bo_obtain_prealloc().

3. No freeing of GEM objects: that takes their dma_resv, and under
   gpuva.lock would free the held lock too.
   -> Defer it: drm_gpuva_unlink() and drm_gpuvm_bo_put() become
      drm_gpuva_unlink_defer() and drm_gpuvm_bo_put_deferred(), and
      drm_gpuvm_bo_deferred_cleanup() frees once the locks are dropped.

This also fixes an unlocked GPUVA list update: when a map fails,
pvr_vm_bind_op_fini() drops the last vm_bo reference without holding
the object's dma_resv.

Failing VM_MAP allocations with fail-nth shows
it on a lockdep kernel (real allocation failures under memory pressure
take the same path):

  WARNING: drivers/gpu/drm/drm_gpuvm.c:1661 at drm_gpuvm_bo_put+0x2ec/0x660 [drm_gpuvm]
  Call trace:
   drm_gpuvm_bo_put
   pvr_vm_bind_op_fini
   pvr_vm_map
   pvr_ioctl_vm_map

Fixes: ff5f643de0bf ("drm/imagination: Add GEM and VM related code")
Signed-off-by: Gyeyoung Baek <gye976@gmail.com>
---
 drivers/gpu/drm/imagination/pvr_vm.c | 39 +++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/imagination/pvr_vm.c b/drivers/gpu/drm/imagination/pvr_vm.c
index af1f5526a78..d551d9a4ab4 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.c
+++ b/drivers/gpu/drm/imagination/pvr_vm.c
@@ -214,7 +214,7 @@ static int pvr_vm_bind_op_exec(struct pvr_vm_bind_op *bind_op)
 
 static void pvr_vm_bind_op_fini(struct pvr_vm_bind_op *bind_op)
 {
-	drm_gpuvm_bo_put(bind_op->gpuvm_bo);
+	drm_gpuvm_bo_put_deferred(bind_op->gpuvm_bo);
 
 	kfree(bind_op->new_va);
 	kfree(bind_op->prev_va);
@@ -255,11 +255,11 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 
 	bind_op->type = PVR_VM_BIND_TYPE_MAP;
 
-	dma_resv_lock(obj->resv, NULL);
-	bind_op->gpuvm_bo = drm_gpuvm_bo_obtain_locked(&vm_ctx->gpuvm_mgr, obj);
-	dma_resv_unlock(obj->resv);
-	if (IS_ERR(bind_op->gpuvm_bo))
-		return PTR_ERR(bind_op->gpuvm_bo);
+	bind_op->gpuvm_bo = drm_gpuvm_bo_create(&vm_ctx->gpuvm_mgr, obj);
+	if (!bind_op->gpuvm_bo)
+		return -ENOMEM;
+
+	bind_op->gpuvm_bo = drm_gpuvm_bo_obtain_prealloc(bind_op->gpuvm_bo);
 
 	bind_op->new_va = kzalloc_obj(*bind_op->new_va);
 	bind_op->prev_va = kzalloc_obj(*bind_op->prev_va);
@@ -293,6 +293,8 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 
 err_bind_op_fini:
 	pvr_vm_bind_op_fini(bind_op);
+	/* Drain what fini() deferred. */
+	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
 
 	return err;
 }
@@ -366,7 +368,11 @@ pvr_vm_gpuva_map(struct drm_gpuva_op *op, void *op_ctx)
 		return err;
 
 	drm_gpuva_map(&ctx->vm_ctx->gpuvm_mgr, &ctx->new_va->base, &op->map);
+
+	mutex_lock(&op->map.gem.obj->gpuva.lock);
 	drm_gpuva_link(&ctx->new_va->base, ctx->gpuvm_bo);
+	mutex_unlock(&op->map.gem.obj->gpuva.lock);
+
 	ctx->new_va = NULL;
 
 	return 0;
@@ -396,7 +402,7 @@ pvr_vm_gpuva_unmap(struct drm_gpuva_op *op, void *op_ctx)
 		return err;
 
 	drm_gpuva_unmap(&op->unmap);
-	drm_gpuva_unlink(op->unmap.va);
+	drm_gpuva_unlink_defer(op->unmap.va);
 	kfree(to_pvr_vm_gpuva(op->unmap.va));
 
 	return 0;
@@ -418,6 +424,7 @@ pvr_vm_gpuva_unmap(struct drm_gpuva_op *op, void *op_ctx)
 static int
 pvr_vm_gpuva_remap(struct drm_gpuva_op *op, void *op_ctx)
 {
+	struct drm_gem_object *obj = op->remap.unmap->va->gem.obj;
 	/* The split parts belong to the object of the mapping being split. */
 	struct drm_gpuvm_bo *vm_bo = op->remap.unmap->va->vm_bo;
 	struct pvr_vm_bind_op *ctx = op_ctx;
@@ -435,16 +442,20 @@ pvr_vm_gpuva_remap(struct drm_gpuva_op *op, void *op_ctx)
 	drm_gpuva_remap(&ctx->prev_va->base, &ctx->next_va->base, &op->remap);
 
 	if (op->remap.prev) {
+		mutex_lock(&obj->gpuva.lock);
 		drm_gpuva_link(&ctx->prev_va->base, vm_bo);
+		mutex_unlock(&obj->gpuva.lock);
 		ctx->prev_va = NULL;
 	}
 
 	if (op->remap.next) {
+		mutex_lock(&obj->gpuva.lock);
 		drm_gpuva_link(&ctx->next_va->base, vm_bo);
+		mutex_unlock(&obj->gpuva.lock);
 		ctx->next_va = NULL;
 	}
 
-	drm_gpuva_unlink(op->remap.unmap->va);
+	drm_gpuva_unlink_defer(op->remap.unmap->va);
 	kfree(to_pvr_vm_gpuva(op->remap.unmap->va));
 
 	return 0;
@@ -568,7 +579,8 @@ pvr_vm_create_context(struct pvr_device *pvr_dev, bool is_userspace_context)
 	drm_gem_private_object_init(&pvr_dev->base, &vm_ctx->dummy_gem, 0);
 	drm_gpuvm_init(&vm_ctx->gpuvm_mgr,
 		       is_userspace_context ? "PowerVR-user-VM" : "PowerVR-FW-VM",
-		       0, &pvr_dev->base, &vm_ctx->dummy_gem,
+		       DRM_GPUVM_IMMEDIATE_MODE, &pvr_dev->base,
+		       &vm_ctx->dummy_gem,
 		       0, PVR_PAGE_TABLE_ADDR_SPACE_SIZE, 0, 0, &pvr_vm_gpuva_ops);
 
 	mutex_init(&vm_ctx->lock);
@@ -602,6 +614,7 @@ pvr_vm_context_release(struct kref *ref_count)
 		pvr_fw_object_destroy(vm_ctx->fw_mem_ctx_obj);
 
 	pvr_vm_unmap_all(vm_ctx);
+	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
 
 	pvr_mmu_context_destroy(vm_ctx->mmu_ctx);
 	drm_gem_private_object_fini(&vm_ctx->dummy_gem);
@@ -738,7 +751,9 @@ pvr_vm_map(struct pvr_vm_context *vm_ctx, struct pvr_gem_object *pvr_obj,
 
 err_cleanup:
 	mutex_unlock(&vm_ctx->lock);
+
 	pvr_vm_bind_op_fini(&bind_op);
+	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
 
 	return err;
 }
@@ -820,6 +835,8 @@ pvr_vm_unmap_obj(struct pvr_vm_context *vm_ctx, struct pvr_gem_object *pvr_obj,
 	err = pvr_vm_unmap_obj_locked(vm_ctx, pvr_obj, device_addr, size);
 	mutex_unlock(&vm_ctx->lock);
 
+	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
+
 	return err;
 }
 
@@ -854,6 +871,8 @@ pvr_vm_unmap(struct pvr_vm_context *vm_ctx, u64 device_addr, u64 size)
 
 	mutex_unlock(&vm_ctx->lock);
 
+	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
+
 	return err;
 }
 
@@ -889,6 +908,8 @@ pvr_vm_unmap_all(struct pvr_vm_context *vm_ctx)
 	}
 
 	mutex_unlock(&vm_ctx->lock);
+
+	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
 }
 
 /* Static data areas are determined by firmware. */

-- 
2.43.0

[PATCH v2 2/6] drm/imagination: Stop locking reservations around map and unmap

[Gyeyoung Baek]

The GPUVA lists are now protected by gpuva.lock instead of the
reservations, so map and unmap no longer need to lock the reservations.
Nothing else on that path relies on them.

Signed-off-by: Gyeyoung Baek <gye976@gmail.com>
---
 drivers/gpu/drm/imagination/pvr_vm.c | 44 ------------------------------------
 1 file changed, 44 deletions(-)

diff --git a/drivers/gpu/drm/imagination/pvr_vm.c b/drivers/gpu/drm/imagination/pvr_vm.c
index d551d9a4ab4..12ebf0daf88 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.c
+++ b/drivers/gpu/drm/imagination/pvr_vm.c
@@ -10,7 +10,6 @@
 #include "pvr_rogue_fwif.h"
 #include "pvr_rogue_heap_config.h"
 
-#include <drm/drm_exec.h>
 #include <drm/drm_gem.h>
 #include <drm/drm_gpuvm.h>
 #include <drm/drm_print.h>
@@ -684,16 +683,6 @@ void pvr_destroy_vm_contexts_for_file(struct pvr_file *pvr_file)
 	}
 }
 
-static int
-pvr_vm_lock_extra(struct drm_gpuvm_exec *vm_exec)
-{
-	struct pvr_vm_bind_op *bind_op = vm_exec->extra.priv;
-	struct pvr_gem_object *pvr_obj = bind_op->pvr_obj;
-
-	/* Acquire lock on the GEM object being mapped/unmapped. */
-	return drm_exec_lock_obj(&vm_exec->exec, gem_from_pvr_gem(pvr_obj));
-}
-
 /**
  * pvr_vm_map() - Map a section of physical memory into a section of
  * device-virtual memory.
@@ -721,15 +710,6 @@ pvr_vm_map(struct pvr_vm_context *vm_ctx, struct pvr_gem_object *pvr_obj,
 	   u64 pvr_obj_offset, u64 device_addr, u64 size)
 {
 	struct pvr_vm_bind_op bind_op = {0};
-	struct drm_gpuvm_exec vm_exec = {
-		.vm = &vm_ctx->gpuvm_mgr,
-		.flags = DRM_EXEC_INTERRUPTIBLE_WAIT |
-			 DRM_EXEC_IGNORE_DUPLICATES,
-		.extra = {
-			.fn = pvr_vm_lock_extra,
-			.priv = &bind_op,
-		},
-	};
 
 	int err = pvr_vm_bind_op_map_init(&bind_op, vm_ctx, pvr_obj,
 					  pvr_obj_offset, device_addr,
@@ -741,15 +721,7 @@ pvr_vm_map(struct pvr_vm_context *vm_ctx, struct pvr_gem_object *pvr_obj,
 	pvr_gem_object_get(pvr_obj);
 
 	mutex_lock(&vm_ctx->lock);
-	err = drm_gpuvm_exec_lock(&vm_exec);
-	if (err)
-		goto err_cleanup;
-
 	err = pvr_vm_bind_op_exec(&bind_op);
-
-	drm_gpuvm_exec_unlock(&vm_exec);
-
-err_cleanup:
 	mutex_unlock(&vm_ctx->lock);
 
 	pvr_vm_bind_op_fini(&bind_op);
@@ -782,15 +754,6 @@ pvr_vm_unmap_obj_locked(struct pvr_vm_context *vm_ctx,
 			u64 device_addr, u64 size)
 {
 	struct pvr_vm_bind_op bind_op = {0};
-	struct drm_gpuvm_exec vm_exec = {
-		.vm = &vm_ctx->gpuvm_mgr,
-		.flags = DRM_EXEC_INTERRUPTIBLE_WAIT |
-			 DRM_EXEC_IGNORE_DUPLICATES,
-		.extra = {
-			.fn = pvr_vm_lock_extra,
-			.priv = &bind_op,
-		},
-	};
 
 	int err = pvr_vm_bind_op_unmap_init(&bind_op, vm_ctx, pvr_obj,
 					    device_addr, size);
@@ -799,15 +762,8 @@ pvr_vm_unmap_obj_locked(struct pvr_vm_context *vm_ctx,
 
 	pvr_gem_object_get(pvr_obj);
 
-	err = drm_gpuvm_exec_lock(&vm_exec);
-	if (err)
-		goto err_cleanup;
-
 	err = pvr_vm_bind_op_exec(&bind_op);
 
-	drm_gpuvm_exec_unlock(&vm_exec);
-
-err_cleanup:
 	pvr_vm_bind_op_fini(&bind_op);
 
 	return err;

-- 
2.43.0

[PATCH v2 3/6] drm/imagination: Route map and unmap through pvr_vm_bind()

[Gyeyoung Baek]

Give the VM layer a single kernel-side bind entry point, and move every
existing map and unmap path onto it.

FW mappings and the VM_MAP/VM_UNMAP ioctls each had their own copy of
the same sequence (prepare, lock, execute, clean up). Fold them into one
pvr_vm_bind() taking an array of struct pvr_vm_bind_desc. The VM_BIND
ioctl that follows uses the same entry point.

Before:

  pvr_meta_vm_map() ----+
  pvr_riscv_vm_map() ---+--> pvr_vm_map()
  pvr_ioctl_vm_map() ---+      map_init -> lock -> exec -> unlock -> fini

  pvr_meta_vm_unmap() --+
  pvr_riscv_vm_unmap() -+--> pvr_vm_unmap_obj()
                               unmap_init -> lock -> exec -> unlock -> fini

  pvr_ioctl_vm_unmap() ----> pvr_vm_unmap()
                               unmap_init -> lock -> find -> exec
                                          -> unlock -> fini

After:

  pvr_meta_vm_map() ----+
  pvr_riscv_vm_map() ---+--> pvr_vm_map() ---+
  pvr_ioctl_vm_map() ---+                    |
                                             +--> pvr_vm_bind()
  pvr_meta_vm_unmap() --+                    |      init -> lock -> exec
  pvr_riscv_vm_unmap() -+--> pvr_vm_unmap() -+      -> unlock -> fini
  pvr_ioctl_vm_unmap() -+

- pvr_vm_unmap() sets the exact flag of its descriptor, since VM_UNMAP
  only removes a mapping that exactly matches the range. An unmap
  through pvr_vm_bind() otherwise removes everything in the range,
  because the VM_BIND ioctl that follows takes an arbitrary range, like
  Vulkan sparse unbinding, rather than a previous binding.
- pvr_vm_unmap_all() is the exception and does not go through
  pvr_vm_bind().
  - pvr_vm_bind(): with the asynchronous mode the VM_BIND ioctl gains
    afterwards, a queued request that fails part way through leaves the
    address space in a state nobody can reason about. The VM context is
    then marked unusable, and pvr_vm_bind() refuses it.
  - pvr_vm_unmap_all(): it empties a VM context, so it has to succeed
    whatever state the context is in.
- The only userspace-visible change: VM_UNMAP returns -EINVAL instead of
  -ENOENT for an invalid address or size.

Signed-off-by: Gyeyoung Baek <gye976@gmail.com>
---
 drivers/gpu/drm/imagination/pvr_fw_meta.c  |   6 +-
 drivers/gpu/drm/imagination/pvr_fw_riscv.c |   6 +-
 drivers/gpu/drm/imagination/pvr_vm.c       | 304 +++++++++++++++--------------
 drivers/gpu/drm/imagination/pvr_vm.h       |  46 ++++-
 4 files changed, 205 insertions(+), 157 deletions(-)

diff --git a/drivers/gpu/drm/imagination/pvr_fw_meta.c b/drivers/gpu/drm/imagination/pvr_fw_meta.c
index 38a91dcb66c..a1de51afde1 100644
--- a/drivers/gpu/drm/imagination/pvr_fw_meta.c
+++ b/drivers/gpu/drm/imagination/pvr_fw_meta.c
@@ -542,10 +542,8 @@ pvr_meta_vm_map(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
 static void
 pvr_meta_vm_unmap(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
 {
-	struct pvr_gem_object *pvr_obj = fw_obj->gem;
-
-	pvr_vm_unmap_obj(pvr_dev->kernel_vm_ctx, pvr_obj,
-			 fw_obj->fw_mm_node.start, fw_obj->fw_mm_node.size);
+	pvr_vm_unmap(pvr_dev->kernel_vm_ctx, fw_obj->fw_mm_node.start,
+		     fw_obj->fw_mm_node.size);
 }
 
 static bool
diff --git a/drivers/gpu/drm/imagination/pvr_fw_riscv.c b/drivers/gpu/drm/imagination/pvr_fw_riscv.c
index cd84635bb88..a22b3f99218 100644
--- a/drivers/gpu/drm/imagination/pvr_fw_riscv.c
+++ b/drivers/gpu/drm/imagination/pvr_fw_riscv.c
@@ -149,10 +149,8 @@ pvr_riscv_vm_map(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
 static void
 pvr_riscv_vm_unmap(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
 {
-	struct pvr_gem_object *pvr_obj = fw_obj->gem;
-
-	pvr_vm_unmap_obj(pvr_dev->kernel_vm_ctx, pvr_obj,
-			 fw_obj->fw_mm_node.start, fw_obj->fw_mm_node.size);
+	pvr_vm_unmap(pvr_dev->kernel_vm_ctx, fw_obj->fw_mm_node.start,
+		     fw_obj->fw_mm_node.size);
 }
 
 static bool
diff --git a/drivers/gpu/drm/imagination/pvr_vm.c b/drivers/gpu/drm/imagination/pvr_vm.c
index 12ebf0daf88..2f959cd5e90 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.c
+++ b/drivers/gpu/drm/imagination/pvr_vm.c
@@ -21,6 +21,8 @@
 #include <linux/gfp_types.h>
 #include <linux/kref.h>
 #include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
 #include <linux/stddef.h>
 
 /**
@@ -117,11 +119,6 @@ struct pvr_vm_gpuva {
 
 #define to_pvr_vm_gpuva(va) container_of_const(va, struct pvr_vm_gpuva, base)
 
-enum pvr_vm_bind_type {
-	PVR_VM_BIND_TYPE_MAP,
-	PVR_VM_BIND_TYPE_UNMAP,
-};
-
 /**
  * struct pvr_vm_bind_op - Context of a map/unmap operation.
  */
@@ -132,6 +129,9 @@ struct pvr_vm_bind_op {
 	/** @pvr_obj: Object associated with mapping (map only). */
 	struct pvr_gem_object *pvr_obj;
 
+	/** @exact: See &pvr_vm_bind_desc.exact. */
+	bool exact;
+
 	/**
 	 * @vm_ctx: VM context where the mapping will be created or destroyed.
 	 */
@@ -198,6 +198,11 @@ static int pvr_vm_bind_op_exec(struct pvr_vm_bind_op *bind_op)
 	}
 
 	case PVR_VM_BIND_TYPE_UNMAP:
+		if (bind_op->exact &&
+		    !drm_gpuva_find(&bind_op->vm_ctx->gpuvm_mgr,
+				    bind_op->device_addr, bind_op->size))
+			return -ENOENT;
+
 		return drm_gpuvm_sm_unmap(&bind_op->vm_ctx->gpuvm_mgr,
 					  bind_op, bind_op->device_addr,
 					  bind_op->size);
@@ -229,9 +234,9 @@ static void pvr_vm_bind_op_fini(struct pvr_vm_bind_op *bind_op)
 static int
 pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 			struct pvr_vm_context *vm_ctx,
-			struct pvr_gem_object *pvr_obj, u64 offset,
-			u64 device_addr, u64 size)
+			const struct pvr_vm_bind_desc *desc)
 {
+	struct pvr_gem_object *pvr_obj = desc->obj;
 	struct drm_gem_object *obj = gem_from_pvr_gem(pvr_obj);
 	const bool is_user = vm_ctx != vm_ctx->pvr_dev->kernel_vm_ctx;
 	const u64 pvr_obj_size = pvr_gem_object_size(pvr_obj);
@@ -239,17 +244,19 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 	u64 offset_plus_size;
 	int err;
 
-	if (check_add_overflow(offset, size, &offset_plus_size))
+	if (check_add_overflow(desc->offset, desc->size, &offset_plus_size))
 		return -EINVAL;
 
 	if (is_user &&
-	    !pvr_find_heap_containing(vm_ctx->pvr_dev, device_addr, size)) {
+	    !pvr_find_heap_containing(vm_ctx->pvr_dev, desc->device_addr,
+				      desc->size)) {
 		return -EINVAL;
 	}
 
-	if (!pvr_device_addr_and_size_are_valid(vm_ctx, device_addr, size) ||
-	    offset & ~PAGE_MASK || size & ~PAGE_MASK ||
-	    offset >= pvr_obj_size || offset_plus_size > pvr_obj_size)
+	if (!pvr_device_addr_and_size_are_valid(vm_ctx, desc->device_addr,
+						desc->size) ||
+	    desc->offset & ~PAGE_MASK || desc->size & ~PAGE_MASK ||
+	    desc->offset >= pvr_obj_size || offset_plus_size > pvr_obj_size)
 		return -EINVAL;
 
 	bind_op->type = PVR_VM_BIND_TYPE_MAP;
@@ -275,7 +282,9 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 		goto err_bind_op_fini;
 
 	bind_op->mmu_op_ctx =
-		pvr_mmu_op_context_create(vm_ctx->mmu_ctx, sgt, device_addr, offset, size);
+		pvr_mmu_op_context_create(vm_ctx->mmu_ctx, sgt,
+					  desc->device_addr, desc->offset,
+					  desc->size);
 	err = PTR_ERR_OR_ZERO(bind_op->mmu_op_ctx);
 	if (err) {
 		bind_op->mmu_op_ctx = NULL;
@@ -283,17 +292,16 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 	}
 
 	bind_op->pvr_obj = pvr_obj;
+	pvr_gem_object_get(pvr_obj);
 	bind_op->vm_ctx = vm_ctx;
-	bind_op->device_addr = device_addr;
-	bind_op->size = size;
-	bind_op->offset = offset;
+	bind_op->device_addr = desc->device_addr;
+	bind_op->size = desc->size;
+	bind_op->offset = desc->offset;
 
 	return 0;
 
 err_bind_op_fini:
 	pvr_vm_bind_op_fini(bind_op);
-	/* Drain what fini() deferred. */
-	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
 
 	return err;
 }
@@ -301,12 +309,12 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 static int
 pvr_vm_bind_op_unmap_init(struct pvr_vm_bind_op *bind_op,
 			  struct pvr_vm_context *vm_ctx,
-			  struct pvr_gem_object *pvr_obj,
-			  u64 device_addr, u64 size)
+			  const struct pvr_vm_bind_desc *desc)
 {
 	int err;
 
-	if (!pvr_device_addr_and_size_are_valid(vm_ctx, device_addr, size))
+	if (!pvr_device_addr_and_size_are_valid(vm_ctx, desc->device_addr,
+						desc->size))
 		return -EINVAL;
 
 	bind_op->type = PVR_VM_BIND_TYPE_UNMAP;
@@ -319,17 +327,18 @@ pvr_vm_bind_op_unmap_init(struct pvr_vm_bind_op *bind_op,
 	}
 
 	bind_op->mmu_op_ctx =
-		pvr_mmu_op_context_create(vm_ctx->mmu_ctx, NULL, device_addr, 0, 0);
+		pvr_mmu_op_context_create(vm_ctx->mmu_ctx, NULL,
+					  desc->device_addr, 0, 0);
 	err = PTR_ERR_OR_ZERO(bind_op->mmu_op_ctx);
 	if (err) {
 		bind_op->mmu_op_ctx = NULL;
 		goto err_bind_op_fini;
 	}
 
-	bind_op->pvr_obj = pvr_obj;
 	bind_op->vm_ctx = vm_ctx;
-	bind_op->device_addr = device_addr;
-	bind_op->size = size;
+	bind_op->exact = desc->exact;
+	bind_op->device_addr = desc->device_addr;
+	bind_op->size = desc->size;
 
 	return 0;
 
@@ -683,188 +692,191 @@ void pvr_destroy_vm_contexts_for_file(struct pvr_file *pvr_file)
 	}
 }
 
-/**
- * pvr_vm_map() - Map a section of physical memory into a section of
- * device-virtual memory.
- * @vm_ctx: Target VM context.
- * @pvr_obj: Target PowerVR memory object.
- * @pvr_obj_offset: Offset into @pvr_obj to map from.
- * @device_addr: Virtual device address at the start of the requested mapping.
- * @size: Size of the requested mapping.
- *
- * No handle is returned to represent the mapping. Instead, callers should
- * remember @device_addr and use that as a handle.
- *
- * Return:
- *  * 0 on success,
- *  * -%EINVAL if @device_addr is not a valid page-aligned device-virtual
- *    address; the region specified by @pvr_obj_offset and @size does not fall
- *    entirely within @pvr_obj, or any part of the specified region of @pvr_obj
- *    is not device-virtual page-aligned,
- *  * Any error encountered while performing internal operations required to
- *    destroy the mapping (returned from pvr_vm_gpuva_map or
- *    pvr_vm_gpuva_remap).
- */
-int
-pvr_vm_map(struct pvr_vm_context *vm_ctx, struct pvr_gem_object *pvr_obj,
-	   u64 pvr_obj_offset, u64 device_addr, u64 size)
+static int
+pvr_vm_bind_op_init(struct pvr_vm_bind_op *bind_op,
+		    struct pvr_vm_context *vm_ctx,
+		    const struct pvr_vm_bind_desc *desc)
 {
-	struct pvr_vm_bind_op bind_op = {0};
+	switch (desc->type) {
+	case PVR_VM_BIND_TYPE_MAP:
+		if (WARN_ON(!desc->obj || desc->exact))
+			return -EINVAL;
 
-	int err = pvr_vm_bind_op_map_init(&bind_op, vm_ctx, pvr_obj,
-					  pvr_obj_offset, device_addr,
-					  size);
+		return pvr_vm_bind_op_map_init(bind_op, vm_ctx, desc);
 
-	if (err)
-		return err;
+	case PVR_VM_BIND_TYPE_UNMAP:
+		if (WARN_ON(desc->obj))
+			return -EINVAL;
 
-	pvr_gem_object_get(pvr_obj);
+		return pvr_vm_bind_op_unmap_init(bind_op, vm_ctx, desc);
+	}
 
-	mutex_lock(&vm_ctx->lock);
-	err = pvr_vm_bind_op_exec(&bind_op);
-	mutex_unlock(&vm_ctx->lock);
+	return -EINVAL;
+}
 
-	pvr_vm_bind_op_fini(&bind_op);
-	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
+/**
+ * pvr_vm_bind_ops_free() - Release an array of prepared bind operations.
+ * @bind_ops: Array to release.
+ * @count: Number of prepared entries in @bind_ops.
+ */
+static void pvr_vm_bind_ops_free(struct pvr_vm_bind_op *bind_ops, u32 count)
+{
+	for (u32 i = 0; i < count; i++)
+		pvr_vm_bind_op_fini(&bind_ops[i]);
 
-	return err;
+	kvfree(bind_ops);
 }
 
 /**
- * pvr_vm_unmap_obj_locked() - Unmap an already mapped section of device-virtual
- * memory.
+ * pvr_vm_bind_ops_create() - Prepare an array of bind operations.
  * @vm_ctx: Target VM context.
- * @pvr_obj: Target PowerVR memory object.
- * @device_addr: Virtual device address at the start of the target mapping.
- * @size: Size of the target mapping.
+ * @descs: Descriptions of the operations.
+ * @desc_count: Number of entries in @descs.
  *
- * Return:
- *  * 0 on success,
- *  * -%EINVAL if @device_addr is not a valid page-aligned device-virtual
- *    address,
- *  * Any error encountered while performing internal operations required to
- *    destroy the mapping (returned from pvr_vm_gpuva_unmap or
- *    pvr_vm_gpuva_remap).
+ * Performs every allocation the operations need, so that applying them cannot
+ * fail for want of memory.
  *
- * The vm_ctx->lock must be held when calling this function.
+ * Return: The new array, or an ERR_PTR on failure.
  */
-static int
-pvr_vm_unmap_obj_locked(struct pvr_vm_context *vm_ctx,
-			struct pvr_gem_object *pvr_obj,
-			u64 device_addr, u64 size)
+static struct pvr_vm_bind_op *
+pvr_vm_bind_ops_create(struct pvr_vm_context *vm_ctx,
+		       const struct pvr_vm_bind_desc *descs, u32 desc_count)
 {
-	struct pvr_vm_bind_op bind_op = {0};
-
-	int err = pvr_vm_bind_op_unmap_init(&bind_op, vm_ctx, pvr_obj,
-					    device_addr, size);
-	if (err)
-		return err;
+	struct pvr_vm_bind_op *bind_ops;
+	int err;
 
-	pvr_gem_object_get(pvr_obj);
+	bind_ops = kvzalloc_objs(*bind_ops, desc_count, GFP_KERNEL_ACCOUNT);
+	if (!bind_ops)
+		return ERR_PTR(-ENOMEM);
 
-	err = pvr_vm_bind_op_exec(&bind_op);
+	for (u32 prepared = 0; prepared < desc_count; prepared++) {
+		err = pvr_vm_bind_op_init(&bind_ops[prepared], vm_ctx,
+					  &descs[prepared]);
+		if (err) {
+			pvr_vm_bind_ops_free(bind_ops, prepared);
+			/* Release the vm_bos that fini() deferred. */
+			drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
+			return ERR_PTR(err);
+		}
 
-	pvr_vm_bind_op_fini(&bind_op);
+		cond_resched();
+	}
 
-	return err;
+	return bind_ops;
 }
 
 /**
- * pvr_vm_unmap_obj() - Unmap an already mapped section of device-virtual
- * memory.
+ * pvr_vm_bind() - Apply a batch of bind operations to a VM context.
  * @vm_ctx: Target VM context.
- * @pvr_obj: Target PowerVR memory object.
- * @device_addr: Virtual device address at the start of the target mapping.
- * @size: Size of the target mapping.
+ * @descs: Operations to apply, in array order.
+ * @desc_count: Number of entries in @descs.
+ *
+ * Not atomic: if an operation fails, the earlier ones stay applied.
  *
  * Return:
- *  * 0 on success,
- *  * Any error encountered by pvr_vm_unmap_obj_locked.
+ *  * 0 on success, or
+ *  * A negative error code on failure.
  */
-int
-pvr_vm_unmap_obj(struct pvr_vm_context *vm_ctx, struct pvr_gem_object *pvr_obj,
-		 u64 device_addr, u64 size)
+int pvr_vm_bind(struct pvr_vm_context *vm_ctx,
+		const struct pvr_vm_bind_desc *descs, u32 desc_count)
 {
-	int err;
+	struct pvr_vm_bind_op *bind_ops;
+	int err = 0;
+
+	bind_ops = pvr_vm_bind_ops_create(vm_ctx, descs, desc_count);
+	if (IS_ERR(bind_ops))
+		return PTR_ERR(bind_ops);
 
 	mutex_lock(&vm_ctx->lock);
-	err = pvr_vm_unmap_obj_locked(vm_ctx, pvr_obj, device_addr, size);
+
+	for (u32 i = 0; !err && i < desc_count; i++)
+		err = pvr_vm_bind_op_exec(&bind_ops[i]);
+
 	mutex_unlock(&vm_ctx->lock);
 
+	pvr_vm_bind_ops_free(bind_ops, desc_count);
 	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
 
 	return err;
 }
 
 /**
- * pvr_vm_unmap() - Unmap an already mapped section of device-virtual memory.
+ * pvr_vm_map() - Map part of an object into a VM context.
+ * @vm_ctx: Target VM context.
+ * @pvr_obj: Target PowerVR memory object.
+ * @pvr_obj_offset: Offset into @pvr_obj to map from.
+ * @device_addr: Virtual device address at the start of the requested mapping.
+ * @size: Size of the requested mapping.
+ *
+ * A wrapper around pvr_vm_bind() with a single map operation.
+ *
+ * Return: 0 on success, or a negative error code.
+ */
+int
+pvr_vm_map(struct pvr_vm_context *vm_ctx, struct pvr_gem_object *pvr_obj,
+	   u64 pvr_obj_offset, u64 device_addr, u64 size)
+{
+	const struct pvr_vm_bind_desc desc = {
+		.type = PVR_VM_BIND_TYPE_MAP,
+		.obj = pvr_obj,
+		.offset = pvr_obj_offset,
+		.device_addr = device_addr,
+		.size = size,
+	};
+
+	return pvr_vm_bind(vm_ctx, &desc, 1);
+}
+
+/**
+ * pvr_vm_unmap() - Remove a mapping from a VM context.
  * @vm_ctx: Target VM context.
  * @device_addr: Virtual device address at the start of the target mapping.
  * @size: Size of the target mapping.
  *
+ * A wrapper around pvr_vm_bind() with a single exact unmap operation.
+ *
  * Return:
  *  * 0 on success,
- *  * Any error encountered by drm_gpuva_find,
- *  * Any error encountered by pvr_vm_unmap_obj_locked.
+ *  * -%ENOENT if no mapping starts at @device_addr and spans exactly @size, or
+ *  * Any other error returned by pvr_vm_bind().
  */
 int
 pvr_vm_unmap(struct pvr_vm_context *vm_ctx, u64 device_addr, u64 size)
 {
-	struct pvr_gem_object *pvr_obj;
-	struct drm_gpuva *va;
-	int err;
-
-	mutex_lock(&vm_ctx->lock);
-
-	va = drm_gpuva_find(&vm_ctx->gpuvm_mgr, device_addr, size);
-	if (va) {
-		pvr_obj = gem_to_pvr_gem(va->gem.obj);
-		err = pvr_vm_unmap_obj_locked(vm_ctx, pvr_obj,
-					      va->va.addr, va->va.range);
-	} else {
-		err = -ENOENT;
-	}
-
-	mutex_unlock(&vm_ctx->lock);
-
-	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
-
-	return err;
+	const struct pvr_vm_bind_desc desc = {
+		.type = PVR_VM_BIND_TYPE_UNMAP,
+		.exact = true,
+		.device_addr = device_addr,
+		.size = size,
+	};
+
+	return pvr_vm_bind(vm_ctx, &desc, 1);
 }
 
 /**
  * pvr_vm_unmap_all() - Unmap all mappings associated with a VM context.
  * @vm_ctx: Target VM context.
  *
- * This function ensures that no mappings are left dangling by unmapping them
- * all in order of ascending device-virtual address.
+ * Unmaps the whole device-virtual address space at once.
  */
 void
 pvr_vm_unmap_all(struct pvr_vm_context *vm_ctx)
 {
-	u64 addr = vm_ctx->gpuvm_mgr.mm_start;
-	u64 end = vm_ctx->gpuvm_mgr.mm_start + vm_ctx->gpuvm_mgr.mm_range;
-	u64 va_addr, va_range;
-
-	mutex_lock(&vm_ctx->lock);
-
-	for (; addr < end; addr = va_addr + va_range) {
-		struct pvr_gem_object *pvr_obj;
-		struct drm_gpuva *va;
-
-		va = drm_gpuva_find_first(&vm_ctx->gpuvm_mgr, addr, end - addr);
-		if (!va)
-			break;
-
-		pvr_obj = gem_to_pvr_gem(va->gem.obj);
-		va_addr = va->va.addr;
-		va_range = va->va.range;
+	const struct pvr_vm_bind_desc desc = {
+		.type = PVR_VM_BIND_TYPE_UNMAP,
+		.device_addr = vm_ctx->gpuvm_mgr.mm_start,
+		.size = vm_ctx->gpuvm_mgr.mm_range,
+	};
+	struct pvr_vm_bind_op bind_op = {0};
 
-		WARN_ON(pvr_vm_unmap_obj_locked(vm_ctx, pvr_obj, va_addr, va_range));
-	}
+	if (WARN_ON(pvr_vm_bind_op_unmap_init(&bind_op, vm_ctx, &desc)))
+		return;
 
+	mutex_lock(&vm_ctx->lock);
+	WARN_ON(pvr_vm_bind_op_exec(&bind_op));
 	mutex_unlock(&vm_ctx->lock);
 
+	pvr_vm_bind_op_fini(&bind_op);
 	drm_gpuvm_bo_deferred_cleanup(&vm_ctx->gpuvm_mgr);
 }
 
diff --git a/drivers/gpu/drm/imagination/pvr_vm.h b/drivers/gpu/drm/imagination/pvr_vm.h
index b0528dffa7f..bf6d5e802d9 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.h
+++ b/drivers/gpu/drm/imagination/pvr_vm.h
@@ -38,10 +38,50 @@ struct pvr_vm_context *pvr_vm_create_context(struct pvr_device *pvr_dev,
 int pvr_vm_map(struct pvr_vm_context *vm_ctx,
 	       struct pvr_gem_object *pvr_obj, u64 pvr_obj_offset,
 	       u64 device_addr, u64 size);
-int pvr_vm_unmap_obj(struct pvr_vm_context *vm_ctx,
-		     struct pvr_gem_object *pvr_obj,
-		     u64 device_addr, u64 size);
 int pvr_vm_unmap(struct pvr_vm_context *vm_ctx, u64 device_addr, u64 size);
+
+/**
+ * enum pvr_vm_bind_type - Type of a &struct pvr_vm_bind_desc.
+ * @PVR_VM_BIND_TYPE_MAP: Map @pvr_vm_bind_desc.obj into a device-virtual range.
+ * @PVR_VM_BIND_TYPE_UNMAP: Unmap a device-virtual range.
+ */
+enum pvr_vm_bind_type {
+	PVR_VM_BIND_TYPE_MAP,
+	PVR_VM_BIND_TYPE_UNMAP,
+};
+
+/**
+ * struct pvr_vm_bind_desc - A bind operation, as passed to pvr_vm_bind().
+ */
+struct pvr_vm_bind_desc {
+	/** @type: Map or unmap. */
+	enum pvr_vm_bind_type type;
+
+	/**
+	 * @obj: Object to map. Map only; must be %NULL for an unmap.
+	 * pvr_vm_bind() does not consume the caller's reference.
+	 */
+	struct pvr_gem_object *obj;
+
+	/** @offset: Offset into @obj to begin mapping from. Map only. */
+	u64 offset;
+
+	/**
+	 * @exact: Fail with -%ENOENT unless a single mapping spans exactly
+	 * the range. Unmap only.
+	 */
+	bool exact;
+
+	/** @device_addr: Device-virtual address at the start of the range. */
+	u64 device_addr;
+
+	/** @size: Size of the range. */
+	u64 size;
+};
+
+int pvr_vm_bind(struct pvr_vm_context *vm_ctx,
+		const struct pvr_vm_bind_desc *descs, u32 desc_count);
+
 void pvr_vm_unmap_all(struct pvr_vm_context *vm_ctx);
 
 dma_addr_t pvr_vm_get_page_table_root_addr(struct pvr_vm_context *vm_ctx);

-- 
2.43.0

[PATCH v2 4/6] drm/imagination: Add VM_BIND ioctl

[Gyeyoung Baek]

Add DRM_IOCTL_PVR_VM_BIND, a batched version of VM_MAP and VM_UNMAP.

Problem with VM_MAP and VM_UNMAP:

  vkQueueBindSparse hands over operations in bulk, which VM_MAP and
  VM_UNMAP turn into hundreds of ioctls.

Difference from VM_UNMAP:

  VM_UNMAP only removes a mapping that exactly matches the range.
  VM_BIND's UNMAP removes everything in the range, splitting mappings
  that cross its boundaries, as Vulkan sparse unbinding takes an
  arbitrary range rather than a previous binding.

The flags and sync_ops fields are reserved for asynchronous requests and
must be zero and empty until then, which fixes the layout.

Bump the driver interface minor version to 2.

Signed-off-by: Gyeyoung Baek <gye976@gmail.com>
---
 Documentation/gpu/imagination/uapi.rst |  12 +++
 drivers/gpu/drm/imagination/pvr_drv.c  | 163 +++++++++++++++++++++++++++++++++
 drivers/gpu/drm/imagination/pvr_drv.h  |   4 +-
 include/uapi/drm/pvr_drm.h             |  90 ++++++++++++++++++
 4 files changed, 268 insertions(+), 1 deletion(-)

diff --git a/Documentation/gpu/imagination/uapi.rst b/Documentation/gpu/imagination/uapi.rst
index 7502413d0a9..781d54ec01e 100644
--- a/Documentation/gpu/imagination/uapi.rst
+++ b/Documentation/gpu/imagination/uapi.rst
@@ -87,6 +87,18 @@ VM_MAP and VM_UNMAP
    :identifiers: drm_pvr_ioctl_vm_map_args
                  drm_pvr_ioctl_vm_unmap_args
 
+VM_BIND
+-------
+.. kernel-doc:: include/uapi/drm/pvr_drm.h
+   :doc: PowerVR IOCTL VM_BIND interface
+
+.. kernel-doc:: include/uapi/drm/pvr_drm.h
+   :doc: Flags for VM_BIND operations.
+
+.. kernel-doc:: include/uapi/drm/pvr_drm.h
+   :identifiers: drm_pvr_vm_bind_op
+                 drm_pvr_ioctl_vm_bind_args
+
 CREATE_CONTEXT and DESTROY_CONTEXT
 ----------------------------------
 .. kernel-doc:: include/uapi/drm/pvr_drm.h
diff --git a/drivers/gpu/drm/imagination/pvr_drv.c b/drivers/gpu/drm/imagination/pvr_drv.c
index 2d522a8e46a..8203d5f081d 100644
--- a/drivers/gpu/drm/imagination/pvr_drv.c
+++ b/drivers/gpu/drm/imagination/pvr_drv.c
@@ -1135,6 +1135,168 @@ pvr_ioctl_vm_unmap(struct drm_device *drm_dev, void *raw_args,
 	return err;
 }
 
+/**
+ * pvr_vm_bind_desc_from_uapi() - Translate a userspace bind operation.
+ * @desc: Description to fill in.
+ * @pvr_file: PowerVR file used to resolve buffer object handles.
+ * @uapi_op: Userspace description of the operation.
+ *
+ * Return: 0 on success, or a negative error code.
+ */
+static int
+pvr_vm_bind_desc_from_uapi(struct pvr_vm_bind_desc *desc,
+			   struct pvr_file *pvr_file,
+			   const struct drm_pvr_vm_bind_op *uapi_op)
+{
+	if (uapi_op->flags & ~DRM_PVR_VM_BIND_OP_FLAGS_MASK)
+		return -EINVAL;
+
+	*desc = (struct pvr_vm_bind_desc){
+		.offset = uapi_op->offset,
+		.device_addr = uapi_op->device_addr,
+		.size = uapi_op->size,
+	};
+
+	switch (uapi_op->flags & DRM_PVR_VM_BIND_OP_TYPE_MASK) {
+	case DRM_PVR_VM_BIND_OP_TYPE_MAP:
+		desc->type = PVR_VM_BIND_TYPE_MAP;
+		desc->obj = pvr_gem_object_from_handle(pvr_file, uapi_op->handle);
+		if (!desc->obj)
+			return -ENOENT;
+
+		return 0;
+
+	case DRM_PVR_VM_BIND_OP_TYPE_UNMAP:
+		if (uapi_op->handle || uapi_op->offset)
+			return -EINVAL;
+
+		desc->type = PVR_VM_BIND_TYPE_UNMAP;
+
+		return 0;
+	}
+
+	return -EINVAL;
+}
+
+/**
+ * pvr_vm_bind_descs_release() - Release translated bind operations.
+ * @descs: Array to release.
+ * @count: Number of entries in @descs.
+ */
+static void pvr_vm_bind_descs_release(struct pvr_vm_bind_desc *descs,
+				      u32 count)
+{
+	for (u32 i = 0; i < count; i++) {
+		if (descs[i].obj)
+			pvr_gem_object_put(descs[i].obj);
+	}
+
+	kvfree(descs);
+}
+
+/**
+ * pvr_vm_bind_descs_from_uapi() - Translate userspace VM_BIND operations.
+ * @pvr_file: PowerVR file the request was issued on.
+ * @uapi_ops: Userspace bind operations.
+ * @count: Number of entries in @uapi_ops.
+ *
+ * Return: The new array, to be released with pvr_vm_bind_descs_release(), or
+ * an ERR_PTR on failure.
+ */
+static struct pvr_vm_bind_desc *
+pvr_vm_bind_descs_from_uapi(struct pvr_file *pvr_file,
+			    const struct drm_pvr_vm_bind_op *uapi_ops,
+			    u32 count)
+{
+	struct pvr_vm_bind_desc *descs;
+	int err;
+
+	descs = kvzalloc_objs(*descs, count, GFP_KERNEL_ACCOUNT);
+	if (!descs)
+		return ERR_PTR(-ENOMEM);
+
+	for (u32 i = 0; i < count; i++) {
+		err = pvr_vm_bind_desc_from_uapi(&descs[i], pvr_file,
+						 &uapi_ops[i]);
+		if (err) {
+			pvr_vm_bind_descs_release(descs, i);
+			return ERR_PTR(err);
+		}
+	}
+
+	return descs;
+}
+
+/**
+ * pvr_ioctl_vm_bind() - IOCTL to apply a batch of VM bind operations.
+ * @drm_dev: [IN] DRM device.
+ * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
+ *                 &struct drm_pvr_ioctl_vm_bind_args.
+ * @file: [IN] DRM file private data.
+ *
+ * Called from userspace with %DRM_IOCTL_PVR_VM_BIND.
+ *
+ * Return:
+ *  * 0 on success,
+ *  * -%EINVAL if arguments are invalid,
+ *  * -%ENOENT if an operation refers to an unknown buffer object, or
+ *  * Any other negative error code on failure.
+ */
+static int
+pvr_ioctl_vm_bind(struct drm_device *drm_dev, void *raw_args,
+		  struct drm_file *file)
+{
+	struct drm_pvr_ioctl_vm_bind_args *args = raw_args;
+	struct pvr_file *pvr_file = to_pvr_file(file);
+	struct drm_pvr_vm_bind_op *uapi_ops;
+	struct pvr_vm_bind_desc *descs;
+	struct pvr_vm_context *vm_ctx;
+	int idx;
+	int err;
+
+	if (!drm_dev_enter(drm_dev, &idx))
+		return -EIO;
+
+	if (args->flags || args->sync_ops.count) {
+		err = -EINVAL;
+		goto err_drm_dev_exit;
+	}
+
+	if (!args->ops.count) {
+		err = 0;
+		goto err_drm_dev_exit;
+	}
+
+	vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
+	if (!vm_ctx) {
+		err = -EINVAL;
+		goto err_drm_dev_exit;
+	}
+
+	err = PVR_UOBJ_GET_ARRAY(uapi_ops, &args->ops);
+	if (err)
+		goto err_put_vm_context;
+
+	descs = pvr_vm_bind_descs_from_uapi(pvr_file, uapi_ops, args->ops.count);
+	kvfree(uapi_ops);
+	if (IS_ERR(descs)) {
+		err = PTR_ERR(descs);
+		goto err_put_vm_context;
+	}
+
+	err = pvr_vm_bind(vm_ctx, descs, args->ops.count);
+
+	pvr_vm_bind_descs_release(descs, args->ops.count);
+
+err_put_vm_context:
+	pvr_vm_context_put(vm_ctx);
+
+err_drm_dev_exit:
+	drm_dev_exit(idx);
+
+	return err;
+}
+
 /*
  * pvr_ioctl_submit_job() - IOCTL to submit a job to the GPU
  * @drm_dev: [IN] DRM device.
@@ -1295,6 +1457,7 @@ static const struct drm_ioctl_desc pvr_drm_driver_ioctls[] = {
 	DRM_PVR_IOCTL(CREATE_HWRT_DATASET, create_hwrt_dataset, DRM_RENDER_ALLOW),
 	DRM_PVR_IOCTL(DESTROY_HWRT_DATASET, destroy_hwrt_dataset, DRM_RENDER_ALLOW),
 	DRM_PVR_IOCTL(SUBMIT_JOBS, submit_jobs, DRM_RENDER_ALLOW),
+	DRM_PVR_IOCTL(VM_BIND, vm_bind, DRM_RENDER_ALLOW),
 };
 
 /* clang-format on */
diff --git a/drivers/gpu/drm/imagination/pvr_drv.h b/drivers/gpu/drm/imagination/pvr_drv.h
index 4e058a6fdd5..67317bbf958 100644
--- a/drivers/gpu/drm/imagination/pvr_drv.h
+++ b/drivers/gpu/drm/imagination/pvr_drv.h
@@ -14,9 +14,10 @@
  * Driver interface version:
  *  - 1.0: Initial interface
  *  - 1.1: Add DRM_PVR_DEV_QUERY_FW_INFO_GET query
+ *  - 1.2: Add DRM_IOCTL_PVR_VM_BIND
  */
 #define PVR_DRIVER_MAJOR 1
-#define PVR_DRIVER_MINOR 1
+#define PVR_DRIVER_MINOR 2
 #define PVR_DRIVER_PATCHLEVEL 0
 
 int pvr_get_uobj(u64 usr_ptr, u32 usr_size, u32 min_size, u32 obj_size, void *out);
@@ -61,6 +62,7 @@ int pvr_set_uobj_array(const struct drm_pvr_obj_array *out, u32 min_stride, u32
 #define PVR_UOBJ_MIN_SIZE(_obj_name) _Generic(_obj_name \
 	PVR_UOBJ_DECL(struct drm_pvr_job, hwrt) \
 	PVR_UOBJ_DECL(struct drm_pvr_sync_op, value) \
+	PVR_UOBJ_DECL(struct drm_pvr_vm_bind_op, size) \
 	PVR_UOBJ_DECL(struct drm_pvr_dev_query_gpu_info, num_phantoms) \
 	PVR_UOBJ_DECL(struct drm_pvr_dev_query_runtime_info, cdm_max_local_mem_size_regs) \
 	PVR_UOBJ_DECL(struct drm_pvr_dev_query_quirks, _padding_c) \
diff --git a/include/uapi/drm/pvr_drm.h b/include/uapi/drm/pvr_drm.h
index 1179c1e7d92..55656aa8304 100644
--- a/include/uapi/drm/pvr_drm.h
+++ b/include/uapi/drm/pvr_drm.h
@@ -108,6 +108,7 @@ struct drm_pvr_obj_array {
 #define DRM_IOCTL_PVR_CREATE_HWRT_DATASET PVR_IOCTL(0x0b, DRM_IOWR, create_hwrt_dataset)
 #define DRM_IOCTL_PVR_DESTROY_HWRT_DATASET PVR_IOCTL(0x0c, DRM_IOW, destroy_hwrt_dataset)
 #define DRM_IOCTL_PVR_SUBMIT_JOBS PVR_IOCTL(0x0d, DRM_IOW, submit_jobs)
+#define DRM_IOCTL_PVR_VM_BIND PVR_IOCTL(0x0e, DRM_IOW, vm_bind)
 
 /**
  * DOC: PowerVR IOCTL DEV_QUERY interface
@@ -1309,6 +1310,95 @@ struct drm_pvr_ioctl_submit_jobs_args {
 	struct drm_pvr_obj_array jobs;
 };
 
+/**
+ * DOC: PowerVR IOCTL VM_BIND interface
+ *
+ * %DRM_IOCTL_PVR_VM_BIND applies a batch of map and/or unmap operations to a
+ * single VM context before the IOCTL returns.
+ *
+ * Operations within a request are applied in array order.
+ *
+ * A request that fails part way through may leave the address space in an
+ * undefined state; how much of it was applied is not reported.
+ */
+
+/**
+ * DOC: Flags for VM_BIND operations.
+ *
+ * Set exactly one operation type in &drm_pvr_vm_bind_op.flags.
+ *
+ * .. c:macro:: DRM_PVR_VM_BIND_OP_TYPE_MAP
+ *
+ *    Create a new mapping.
+ *
+ * .. c:macro:: DRM_PVR_VM_BIND_OP_TYPE_UNMAP
+ *
+ *    Remove everything in the range, splitting mappings that cross its
+ *    ends. Unlike %DRM_IOCTL_PVR_VM_UNMAP, the range need not match a
+ *    mapping.
+ *
+ * .. c:macro:: DRM_PVR_VM_BIND_OP_TYPE_MASK
+ *
+ *    Mask used to extract the operation type.
+ */
+#define DRM_PVR_VM_BIND_OP_TYPE_MAP (0u << 28)
+#define DRM_PVR_VM_BIND_OP_TYPE_UNMAP (1u << 28)
+#define DRM_PVR_VM_BIND_OP_TYPE_MASK (0xfu << 28)
+
+#define DRM_PVR_VM_BIND_OP_FLAGS_MASK DRM_PVR_VM_BIND_OP_TYPE_MASK
+
+/**
+ * struct drm_pvr_vm_bind_op - A single VM bind operation.
+ */
+struct drm_pvr_vm_bind_op {
+	/** @flags: [IN] Combination of ``DRM_PVR_VM_BIND_OP_`` flags. */
+	__u32 flags;
+
+	/**
+	 * @handle: [IN] Handle of the target buffer object.
+	 *
+	 * Must be a valid handle returned by %DRM_IOCTL_PVR_CREATE_BO for map
+	 * operations. MBZ for unmap operations.
+	 */
+	__u32 handle;
+
+	/**
+	 * @offset: [IN] Offset into the target buffer object from which to
+	 * begin the mapping. MBZ for unmap operations.
+	 */
+	__u64 offset;
+
+	/**
+	 * @device_addr: [IN] Device-virtual address at the start of the target
+	 * range, aligned to the device page size. For map operations, it must
+	 * also obey the same heap containment rules as %DRM_IOCTL_PVR_VM_MAP.
+	 */
+	__u64 device_addr;
+
+	/** @size: [IN] Size in bytes of the target range. Must be non-zero. */
+	__u64 size;
+};
+
+/**
+ * struct drm_pvr_ioctl_vm_bind_args - Arguments for %DRM_IOCTL_PVR_VM_BIND.
+ */
+struct drm_pvr_ioctl_vm_bind_args {
+	/**
+	 * @vm_context_handle: [IN] Handle for the VM context these operations
+	 * apply to.
+	 */
+	__u32 vm_context_handle;
+
+	/** @flags: [IN] Reserved. MBZ. */
+	__u32 flags;
+
+	/** @ops: [IN] Array of &struct drm_pvr_vm_bind_op to apply. */
+	struct drm_pvr_obj_array ops;
+
+	/** @sync_ops: [IN] Reserved. Must be empty. */
+	struct drm_pvr_obj_array sync_ops;
+};
+
 #if defined(__cplusplus)
 }
 #endif

-- 
2.43.0

[PATCH v2 5/6] drm/imagination: Add asynchronous VM_BIND

[Gyeyoung Baek]

Add DRM_PVR_VM_BIND_ASYNC, which runs a VM_BIND request asynchronously.
sync_ops can only be used with this flag.

Each VM context has one drm_sched queue for bind requests. A request
with the flag set is queued there as a job and runs in submission order.
The job runs on the CPU, so it waits for completion fences rather than
scheduled ones, and syncs the page tables itself.
pvr_mmu_op_context_sync() is made public for this.

A request that fails part way through makes its VM context unusable:
further binds and job submissions on it fail with -ECANCELED.

Converting a wait operation into a fence is split out into
pvr_sync_wait_op_get_fence() and shared with pvr_sync_add_deps_to_job(),
which behaves as before.

Signed-off-by: Gyeyoung Baek <gye976@gmail.com>
---
 Documentation/gpu/imagination/uapi.rst  |   3 +
 drivers/gpu/drm/imagination/pvr_drv.c   | 127 ++++++++++-
 drivers/gpu/drm/imagination/pvr_job.c   |   6 +
 drivers/gpu/drm/imagination/pvr_mmu.c   |   2 +-
 drivers/gpu/drm/imagination/pvr_mmu.h   |   1 +
 drivers/gpu/drm/imagination/pvr_queue.c |   3 +
 drivers/gpu/drm/imagination/pvr_sync.c  | 146 +++++++++++--
 drivers/gpu/drm/imagination/pvr_sync.h  |  13 ++
 drivers/gpu/drm/imagination/pvr_vm.c    | 359 +++++++++++++++++++++++++++++++-
 drivers/gpu/drm/imagination/pvr_vm.h    |   9 +
 include/uapi/drm/pvr_drm.h              |  34 ++-
 11 files changed, 673 insertions(+), 30 deletions(-)

diff --git a/Documentation/gpu/imagination/uapi.rst b/Documentation/gpu/imagination/uapi.rst
index 781d54ec01e..8c96b6bc7a7 100644
--- a/Documentation/gpu/imagination/uapi.rst
+++ b/Documentation/gpu/imagination/uapi.rst
@@ -95,6 +95,9 @@ VM_BIND
 .. kernel-doc:: include/uapi/drm/pvr_drm.h
    :doc: Flags for VM_BIND operations.
 
+.. kernel-doc:: include/uapi/drm/pvr_drm.h
+   :doc: Flags for the VM_BIND ioctl.
+
 .. kernel-doc:: include/uapi/drm/pvr_drm.h
    :identifiers: drm_pvr_vm_bind_op
                  drm_pvr_ioctl_vm_bind_args
diff --git a/drivers/gpu/drm/imagination/pvr_drv.c b/drivers/gpu/drm/imagination/pvr_drv.c
index 8203d5f081d..c35d437aef7 100644
--- a/drivers/gpu/drm/imagination/pvr_drv.c
+++ b/drivers/gpu/drm/imagination/pvr_drv.c
@@ -15,6 +15,7 @@
 #include "pvr_rogue_defs.h"
 #include "pvr_rogue_fwif_client.h"
 #include "pvr_rogue_fwif_shared.h"
+#include "pvr_sync.h"
 #include "pvr_trace.h"
 #include "pvr_vm.h"
 
@@ -26,6 +27,7 @@
 #include <drm/drm_gem.h>
 #include <drm/drm_ioctl.h>
 
+#include <linux/dma-fence.h>
 #include <linux/err.h>
 #include <linux/export.h>
 #include <linux/fs.h>
@@ -1227,6 +1229,84 @@ pvr_vm_bind_descs_from_uapi(struct pvr_file *pvr_file,
 	return descs;
 }
 
+/**
+ * struct pvr_vm_bind_syncs - Translated sync operations of a VM_BIND request.
+ */
+struct pvr_vm_bind_syncs {
+	/** @waits: Fences to wait for before applying the request. */
+	struct dma_fence **waits;
+
+	/** @wait_count: Number of entries in @waits. */
+	u32 wait_count;
+
+	/** @signals: Syncobjs to signal once the request has been applied. */
+	struct xarray signals;
+};
+
+/**
+ * pvr_vm_bind_syncs_release() - Release translated sync operations.
+ * @syncs: Sync operations to release.
+ */
+static void pvr_vm_bind_syncs_release(struct pvr_vm_bind_syncs *syncs)
+{
+	pvr_sync_signal_array_cleanup(&syncs->signals);
+	pvr_sync_put_fences(syncs->waits, syncs->wait_count);
+}
+
+/**
+ * pvr_vm_bind_syncs_from_uapi() - Translate VM_BIND sync operations.
+ * @syncs: Sync operations to fill in, to be released with
+ *         pvr_vm_bind_syncs_release() on success.
+ * @file: DRM file the request was issued on.
+ * @uapi_sync_ops: Userspace sync operations.
+ *
+ * Return: 0 on success, or a negative error code.
+ */
+static int
+pvr_vm_bind_syncs_from_uapi(struct pvr_vm_bind_syncs *syncs,
+			    struct drm_file *file,
+			    const struct drm_pvr_obj_array *uapi_sync_ops)
+{
+	struct drm_pvr_sync_op *ops;
+	struct dma_fence **waits;
+	int err;
+
+	syncs->waits = NULL;
+	syncs->wait_count = 0;
+	xa_init_flags(&syncs->signals, XA_FLAGS_ALLOC);
+
+	if (!uapi_sync_ops->count)
+		return 0;
+
+	err = PVR_UOBJ_GET_ARRAY(ops, uapi_sync_ops);
+	if (err)
+		return err;
+
+	err = pvr_sync_signal_array_collect_ops(&syncs->signals, file,
+						uapi_sync_ops->count, ops);
+	if (err)
+		goto err_syncs_release;
+
+	waits = pvr_sync_get_wait_fences(file, uapi_sync_ops->count, ops,
+					 &syncs->wait_count);
+	if (IS_ERR(waits)) {
+		err = PTR_ERR(waits);
+		goto err_syncs_release;
+	}
+
+	syncs->waits = waits;
+
+	kvfree(ops);
+
+	return 0;
+
+err_syncs_release:
+	pvr_vm_bind_syncs_release(syncs);
+	kvfree(ops);
+
+	return err;
+}
+
 /**
  * pvr_ioctl_vm_bind() - IOCTL to apply a batch of VM bind operations.
  * @drm_dev: [IN] DRM device.
@@ -1248,8 +1328,9 @@ pvr_ioctl_vm_bind(struct drm_device *drm_dev, void *raw_args,
 {
 	struct drm_pvr_ioctl_vm_bind_args *args = raw_args;
 	struct pvr_file *pvr_file = to_pvr_file(file);
-	struct drm_pvr_vm_bind_op *uapi_ops;
+	struct drm_pvr_vm_bind_op *uapi_ops = NULL;
 	struct pvr_vm_bind_desc *descs;
+	struct pvr_vm_bind_syncs syncs;
 	struct pvr_vm_context *vm_ctx;
 	int idx;
 	int err;
@@ -1257,12 +1338,17 @@ pvr_ioctl_vm_bind(struct drm_device *drm_dev, void *raw_args,
 	if (!drm_dev_enter(drm_dev, &idx))
 		return -EIO;
 
-	if (args->flags || args->sync_ops.count) {
+	if (args->flags & ~DRM_PVR_VM_BIND_FLAGS_MASK) {
+		err = -EINVAL;
+		goto err_drm_dev_exit;
+	}
+
+	if (!(args->flags & DRM_PVR_VM_BIND_ASYNC) && args->sync_ops.count) {
 		err = -EINVAL;
 		goto err_drm_dev_exit;
 	}
 
-	if (!args->ops.count) {
+	if (!args->ops.count && !args->sync_ops.count) {
 		err = 0;
 		goto err_drm_dev_exit;
 	}
@@ -1273,9 +1359,11 @@ pvr_ioctl_vm_bind(struct drm_device *drm_dev, void *raw_args,
 		goto err_drm_dev_exit;
 	}
 
-	err = PVR_UOBJ_GET_ARRAY(uapi_ops, &args->ops);
-	if (err)
-		goto err_put_vm_context;
+	if (args->ops.count) {
+		err = PVR_UOBJ_GET_ARRAY(uapi_ops, &args->ops);
+		if (err)
+			goto err_put_vm_context;
+	}
 
 	descs = pvr_vm_bind_descs_from_uapi(pvr_file, uapi_ops, args->ops.count);
 	kvfree(uapi_ops);
@@ -1284,8 +1372,33 @@ pvr_ioctl_vm_bind(struct drm_device *drm_dev, void *raw_args,
 		goto err_put_vm_context;
 	}
 
-	err = pvr_vm_bind(vm_ctx, descs, args->ops.count);
+	/* Empty in synchronous mode, rejected above otherwise. */
+	err = pvr_vm_bind_syncs_from_uapi(&syncs, file, &args->sync_ops);
+	if (err)
+		goto err_descs_release;
+
+	if (args->flags & DRM_PVR_VM_BIND_ASYNC) {
+		struct dma_fence *fence;
+
+		fence = pvr_vm_bind_async(vm_ctx, descs, args->ops.count,
+					  syncs.waits, syncs.wait_count,
+					  file->client_id);
+		if (IS_ERR(fence)) {
+			err = PTR_ERR(fence);
+			goto err_syncs_release;
+		}
+
+		pvr_sync_signal_array_set_fence(&syncs.signals, fence);
+		pvr_sync_signal_array_push_fences(&syncs.signals);
+		dma_fence_put(fence);
+	} else {
+		err = pvr_vm_bind(vm_ctx, descs, args->ops.count);
+	}
+
+err_syncs_release:
+	pvr_vm_bind_syncs_release(&syncs);
 
+err_descs_release:
 	pvr_vm_bind_descs_release(descs, args->ops.count);
 
 err_put_vm_context:
diff --git a/drivers/gpu/drm/imagination/pvr_job.c b/drivers/gpu/drm/imagination/pvr_job.c
index b8a58d81700..04f920aaf12 100644
--- a/drivers/gpu/drm/imagination/pvr_job.c
+++ b/drivers/gpu/drm/imagination/pvr_job.c
@@ -15,6 +15,7 @@
 #include "pvr_stream_defs.h"
 #include "pvr_sync.h"
 #include "pvr_trace.h"
+#include "pvr_vm.h"
 
 #include <drm/drm_exec.h>
 #include <drm/drm_gem.h>
@@ -434,6 +435,11 @@ create_job(struct pvr_device *pvr_dev,
 		goto err_put_job;
 	}
 
+	if (pvr_vm_context_is_unusable(job->ctx->vm_ctx)) {
+		err = -ECANCELED;
+		goto err_put_job;
+	}
+
 	if (args->hwrt.set_handle) {
 		job->hwrt = pvr_hwrt_data_lookup(pvr_file, args->hwrt.set_handle,
 						 args->hwrt.data_index);
diff --git a/drivers/gpu/drm/imagination/pvr_mmu.c b/drivers/gpu/drm/imagination/pvr_mmu.c
index 8ffa80bee97..5e39fa25e58 100644
--- a/drivers/gpu/drm/imagination/pvr_mmu.c
+++ b/drivers/gpu/drm/imagination/pvr_mmu.c
@@ -2017,7 +2017,7 @@ pvr_mmu_op_context_sync_partial(struct pvr_mmu_op_context *op_ctx,
  * that subsequent calls to this function will be no-ops unless @op_ctx is
  * otherwise updated.
  */
-static void
+void
 pvr_mmu_op_context_sync(struct pvr_mmu_op_context *op_ctx)
 {
 	pvr_mmu_op_context_sync_manual(op_ctx, op_ctx->sync_level_required);
diff --git a/drivers/gpu/drm/imagination/pvr_mmu.h b/drivers/gpu/drm/imagination/pvr_mmu.h
index ddf175eb5f9..73475578515 100644
--- a/drivers/gpu/drm/imagination/pvr_mmu.h
+++ b/drivers/gpu/drm/imagination/pvr_mmu.h
@@ -97,6 +97,7 @@ void pvr_mmu_context_destroy(struct pvr_mmu_context *ctx);
 dma_addr_t pvr_mmu_get_root_table_dma_addr(struct pvr_mmu_context *ctx);
 
 void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx);
+void pvr_mmu_op_context_sync(struct pvr_mmu_op_context *op_ctx);
 struct pvr_mmu_op_context *
 pvr_mmu_op_context_create(struct pvr_mmu_context *ctx,
 			  struct sg_table *sgt, u64 device_addr, u64 sgt_offset, u64 size);
diff --git a/drivers/gpu/drm/imagination/pvr_queue.c b/drivers/gpu/drm/imagination/pvr_queue.c
index 24a8645aa91..ac5862b1c52 100644
--- a/drivers/gpu/drm/imagination/pvr_queue.c
+++ b/drivers/gpu/drm/imagination/pvr_queue.c
@@ -759,6 +759,9 @@ static struct dma_fence *pvr_queue_run_job(struct drm_sched_job *sched_job)
 		return dma_fence_get(job->done_fence);
 	}
 
+	if (pvr_vm_context_is_unusable(job->ctx->vm_ctx))
+		return ERR_PTR(-ECANCELED);
+
 	/* The only kind of jobs that can be paired are geometry and fragment, and
 	 * we bail out early if we see a fragment job that's paired with a geometry job.
 	 * Paired jobs must also target the same context and point to the same HWRT.
diff --git a/drivers/gpu/drm/imagination/pvr_sync.c b/drivers/gpu/drm/imagination/pvr_sync.c
index 936f840a522..714b26e776f 100644
--- a/drivers/gpu/drm/imagination/pvr_sync.c
+++ b/drivers/gpu/drm/imagination/pvr_sync.c
@@ -185,6 +185,28 @@ pvr_sync_signal_array_update_fences(struct xarray *array,
 	return 0;
 }
 
+/**
+ * pvr_sync_signal_array_set_fence() - Set the fence of all entries.
+ * @array: Signal array, as filled by pvr_sync_signal_array_collect_ops().
+ * @fence: Fence to set. Must not be %NULL.
+ *
+ * Unlike pvr_sync_signal_array_update_fences(), entries are not filtered by
+ * sync operations. pvr_sync_signal_array_push_fences() installs the fences.
+ */
+void
+pvr_sync_signal_array_set_fence(struct xarray *array, struct dma_fence *fence)
+{
+	struct pvr_sync_signal *sig_sync;
+	unsigned long i;
+
+	xa_for_each(array, i, sig_sync) {
+		struct dma_fence *old_fence = sig_sync->fence;
+
+		sig_sync->fence = dma_fence_get(fence);
+		dma_fence_put(old_fence);
+	}
+}
+
 void
 pvr_sync_signal_array_push_fences(struct xarray *array)
 {
@@ -244,6 +266,40 @@ pvr_sync_add_dep_to_job(struct drm_sched_job *job, struct dma_fence *f)
 	return err;
 }
 
+/*
+ * Resolve a wait operation to the fence it waits on. A fence already set in
+ * @signal_array, if not NULL, takes precedence over the one currently in the
+ * syncobj.
+ */
+static int
+pvr_sync_wait_op_get_fence(struct drm_file *file,
+			   const struct drm_pvr_sync_op *sync_op,
+			   struct xarray *signal_array,
+			   struct dma_fence **fence)
+{
+	struct pvr_sync_signal *sig_sync = NULL;
+	int err;
+
+	err = pvr_check_sync_op(sync_op);
+	if (err)
+		return err;
+
+	if (signal_array)
+		sig_sync = pvr_sync_signal_array_search(signal_array,
+							sync_op->handle,
+							sync_op->value);
+	if (sig_sync) {
+		if (WARN_ON(!sig_sync->fence))
+			return -EINVAL;
+
+		*fence = dma_fence_get(sig_sync->fence);
+		return 0;
+	}
+
+	return drm_syncobj_find_fence(file, sync_op->handle, sync_op->value, 0,
+				      fence);
+}
+
 int
 pvr_sync_add_deps_to_job(struct pvr_file *pvr_file, struct drm_sched_job *job,
 			 u32 sync_op_count,
@@ -256,30 +312,17 @@ pvr_sync_add_deps_to_job(struct pvr_file *pvr_file, struct drm_sched_job *job,
 		return 0;
 
 	for (u32 i = 0; i < sync_op_count; i++) {
-		struct pvr_sync_signal *sig_sync;
 		struct dma_fence *fence;
 
 		if (sync_ops[i].flags & DRM_PVR_SYNC_OP_FLAG_SIGNAL)
 			continue;
 
-		err = pvr_check_sync_op(&sync_ops[i]);
+		err = pvr_sync_wait_op_get_fence(from_pvr_file(pvr_file),
+						 &sync_ops[i], signal_array,
+						 &fence);
 		if (err)
 			return err;
 
-		sig_sync = pvr_sync_signal_array_search(signal_array, sync_ops[i].handle,
-							sync_ops[i].value);
-		if (sig_sync) {
-			if (WARN_ON(!sig_sync->fence))
-				return -EINVAL;
-
-			fence = dma_fence_get(sig_sync->fence);
-		} else {
-			err = drm_syncobj_find_fence(from_pvr_file(pvr_file), sync_ops[i].handle,
-						     sync_ops[i].value, 0, &fence);
-			if (err)
-				return err;
-		}
-
 		err = pvr_sync_add_dep_to_job(job, fence);
 		if (err)
 			return err;
@@ -287,3 +330,74 @@ pvr_sync_add_deps_to_job(struct pvr_file *pvr_file, struct drm_sched_job *job,
 
 	return 0;
 }
+
+/**
+ * pvr_sync_put_fences() - Release an array of fences.
+ * @fences: Array to release. May be %NULL.
+ * @count: Number of entries in @fences.
+ */
+void
+pvr_sync_put_fences(struct dma_fence **fences, u32 count)
+{
+	if (!fences)
+		return;
+
+	for (u32 i = 0; i < count; i++)
+		dma_fence_put(fences[i]);
+
+	kvfree(fences);
+}
+
+/**
+ * pvr_sync_get_wait_fences() - Resolve the wait operations of a request.
+ * @file: DRM file the request was issued on.
+ * @sync_op_count: Number of entries in @sync_ops.
+ * @sync_ops: Sync operations of the request.
+ * @count: Location to store the number of fences returned. Left untouched on
+ *         failure.
+ *
+ * Unlike pvr_sync_add_deps_to_job(), native fences are not replaced by their
+ * scheduled fence: that relies on the FW waiting for completion, which a job
+ * run on the CPU does not get.
+ *
+ * Return: The fences to wait on, to be released with pvr_sync_put_fences(),
+ * %NULL if @sync_op_count is zero, or an ERR_PTR on failure.
+ */
+struct dma_fence **
+pvr_sync_get_wait_fences(struct drm_file *file, u32 sync_op_count,
+			 const struct drm_pvr_sync_op *sync_ops, u32 *count)
+{
+	struct dma_fence **fences;
+	u32 fence_count = 0;
+	int err;
+
+	if (!sync_op_count) {
+		*count = 0;
+		return NULL;
+	}
+
+	fences = kvzalloc_objs(*fences, sync_op_count, GFP_KERNEL_ACCOUNT);
+	if (!fences)
+		return ERR_PTR(-ENOMEM);
+
+	for (u32 i = 0; i < sync_op_count; i++) {
+		if (sync_ops[i].flags & DRM_PVR_SYNC_OP_FLAG_SIGNAL)
+			continue;
+
+		err = pvr_sync_wait_op_get_fence(file, &sync_ops[i], NULL,
+						 &fences[fence_count]);
+		if (err)
+			goto err_put_fences;
+
+		fence_count++;
+	}
+
+	*count = fence_count;
+
+	return fences;
+
+err_put_fences:
+	pvr_sync_put_fences(fences, fence_count);
+
+	return ERR_PTR(err);
+}
diff --git a/drivers/gpu/drm/imagination/pvr_sync.h b/drivers/gpu/drm/imagination/pvr_sync.h
index 48501ad2779..72e34bbfe20 100644
--- a/drivers/gpu/drm/imagination/pvr_sync.h
+++ b/drivers/gpu/drm/imagination/pvr_sync.h
@@ -9,6 +9,9 @@
 /* Forward declaration from <linux/xarray.h>. */
 struct xarray;
 
+/* Forward declaration from <linux/dma-fence.h>. */
+struct dma_fence;
+
 /* Forward declaration from <drm/drm_file.h>. */
 struct drm_file;
 
@@ -72,6 +75,9 @@ pvr_sync_signal_array_update_fences(struct xarray *array,
 				    const struct drm_pvr_sync_op *sync_ops,
 				    struct dma_fence *finished_fence);
 
+void
+pvr_sync_signal_array_set_fence(struct xarray *array, struct dma_fence *fence);
+
 void
 pvr_sync_signal_array_push_fences(struct xarray *array);
 
@@ -81,4 +87,11 @@ pvr_sync_add_deps_to_job(struct pvr_file *pvr_file, struct drm_sched_job *job,
 			 const struct drm_pvr_sync_op *sync_ops,
 			 struct xarray *signal_array);
 
+void
+pvr_sync_put_fences(struct dma_fence **fences, u32 count);
+
+struct dma_fence **
+pvr_sync_get_wait_fences(struct drm_file *file, u32 sync_op_count,
+			 const struct drm_pvr_sync_op *sync_ops, u32 *count);
+
 #endif /* PVR_SYNC_H */
diff --git a/drivers/gpu/drm/imagination/pvr_vm.c b/drivers/gpu/drm/imagination/pvr_vm.c
index 2f959cd5e90..4e58ea8c50c 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.c
+++ b/drivers/gpu/drm/imagination/pvr_vm.c
@@ -13,9 +13,11 @@
 #include <drm/drm_gem.h>
 #include <drm/drm_gpuvm.h>
 #include <drm/drm_print.h>
+#include <drm/gpu_scheduler.h>
 
 #include <linux/bug.h>
 #include <linux/container_of.h>
+#include <linux/dma-fence.h>
 #include <linux/err.h>
 #include <linux/errno.h>
 #include <linux/gfp_types.h>
@@ -24,6 +26,7 @@
 #include <linux/sched.h>
 #include <linux/slab.h>
 #include <linux/stddef.h>
+#include <linux/workqueue.h>
 
 /**
  * DOC: Memory context
@@ -51,6 +54,25 @@ struct pvr_vm_context {
 	/** @lock: Global lock on this VM. */
 	struct mutex lock;
 
+	/**
+	 * @sched: Serialises asynchronous bind requests. Userspace contexts
+	 * only.
+	 */
+	struct drm_gpu_scheduler sched;
+
+	/** @entity: Scheduling entity feeding @sched. */
+	struct drm_sched_entity entity;
+
+	/** @sched_initialised: True if @sched and @entity need tearing down. */
+	bool sched_initialised;
+
+	/**
+	 * @unusable: Set, and never cleared, when an asynchronous bind fails
+	 * part way through. Further binds and jobs on this context then fail
+	 * with -%ECANCELED. Written under @lock, read without it.
+	 */
+	bool unusable;
+
 	/**
 	 * @fw_mem_ctx_obj: Firmware object representing firmware memory
 	 * context.
@@ -73,6 +95,9 @@ struct pvr_vm_context *to_pvr_vm_context(struct drm_gpuvm *gpuvm)
 	return container_of(gpuvm, struct pvr_vm_context, gpuvm_mgr);
 }
 
+static int pvr_vm_bind_sched_init(struct pvr_vm_context *vm_ctx);
+static void pvr_vm_bind_sched_fini(struct pvr_vm_context *vm_ctx);
+
 struct pvr_vm_context *pvr_vm_context_get(struct pvr_vm_context *vm_ctx)
 {
 	if (vm_ctx)
@@ -594,6 +619,15 @@ pvr_vm_create_context(struct pvr_device *pvr_dev, bool is_userspace_context)
 	mutex_init(&vm_ctx->lock);
 	kref_init(&vm_ctx->ref_count);
 
+	if (is_userspace_context) {
+		err = pvr_vm_bind_sched_init(vm_ctx);
+		if (err) {
+			/* Fully set up by now: tear down through release(). */
+			pvr_vm_context_put(vm_ctx);
+			return ERR_PTR(err);
+		}
+	}
+
 	return vm_ctx;
 
 err_page_table_destroy:
@@ -618,6 +652,8 @@ pvr_vm_context_release(struct kref *ref_count)
 	struct pvr_vm_context *vm_ctx =
 		container_of(ref_count, struct pvr_vm_context, ref_count);
 
+	pvr_vm_bind_sched_fini(vm_ctx);
+
 	if (vm_ctx->fw_mem_ctx_obj)
 		pvr_fw_object_destroy(vm_ctx->fw_mem_ctx_obj);
 
@@ -774,8 +810,9 @@ pvr_vm_bind_ops_create(struct pvr_vm_context *vm_ctx,
  * Not atomic: if an operation fails, the earlier ones stay applied.
  *
  * Return:
- *  * 0 on success, or
- *  * A negative error code on failure.
+ *  * 0 on success,
+ *  * -%ECANCELED if @vm_ctx has been left unusable by an earlier failure, or
+ *  * Any other negative error code on failure.
  */
 int pvr_vm_bind(struct pvr_vm_context *vm_ctx,
 		const struct pvr_vm_bind_desc *descs, u32 desc_count)
@@ -789,6 +826,9 @@ int pvr_vm_bind(struct pvr_vm_context *vm_ctx,
 
 	mutex_lock(&vm_ctx->lock);
 
+	if (pvr_vm_context_is_unusable(vm_ctx))
+		err = -ECANCELED;
+
 	for (u32 i = 0; !err && i < desc_count; i++)
 		err = pvr_vm_bind_op_exec(&bind_ops[i]);
 
@@ -853,6 +893,17 @@ pvr_vm_unmap(struct pvr_vm_context *vm_ctx, u64 device_addr, u64 size)
 	return pvr_vm_bind(vm_ctx, &desc, 1);
 }
 
+/**
+ * pvr_vm_context_is_unusable() - Test whether a VM context is unusable.
+ * @vm_ctx: Target VM context.
+ *
+ * Return: %true if @vm_ctx rejects further operations.
+ */
+bool pvr_vm_context_is_unusable(struct pvr_vm_context *vm_ctx)
+{
+	return READ_ONCE(vm_ctx->unusable);
+}
+
 /**
  * pvr_vm_unmap_all() - Unmap all mappings associated with a VM context.
  * @vm_ctx: Target VM context.
@@ -1168,3 +1219,307 @@ pvr_vm_get_fw_mem_context(struct pvr_vm_context *vm_ctx)
 {
 	return vm_ctx->fw_mem_ctx_obj;
 }
+
+/**
+ * struct pvr_vm_bind_job - A queued batch of VM bind operations.
+ */
+struct pvr_vm_bind_job {
+	/** @base: Inherited &drm_sched_job object. */
+	struct drm_sched_job base;
+
+	/** @vm_ctx: VM context targeted by this job. Holds a reference. */
+	struct pvr_vm_context *vm_ctx;
+
+	/** @bind_op_count: Number of entries in @bind_ops. */
+	u32 bind_op_count;
+
+	/** @bind_ops: Prepared bind operations, applied in array order. */
+	struct pvr_vm_bind_op *bind_ops;
+
+	/**
+	 * @cleanup_work: Releases @bind_ops and the reference on @vm_ctx. Not
+	 * done in free_job(): dropping the last VM context reference there
+	 * would call drm_sched_fini(), which flushes the worker it runs on.
+	 */
+	struct work_struct cleanup_work;
+};
+
+#define to_pvr_vm_bind_job(sched_job) \
+	container_of((sched_job), struct pvr_vm_bind_job, base)
+
+static void pvr_vm_bind_job_destroy(struct pvr_vm_bind_job *job)
+{
+	if (!job)
+		return;
+
+	pvr_vm_bind_ops_free(job->bind_ops, job->bind_op_count);
+
+	if (job->vm_ctx) {
+		drm_gpuvm_bo_deferred_cleanup(&job->vm_ctx->gpuvm_mgr);
+		pvr_vm_context_put(job->vm_ctx);
+	}
+
+	kfree(job);
+}
+
+static void pvr_vm_bind_job_cleanup_work(struct work_struct *work)
+{
+	struct pvr_vm_bind_job *job =
+		container_of(work, struct pvr_vm_bind_job, cleanup_work);
+
+	pvr_vm_bind_job_destroy(job);
+}
+
+/*
+ * Runs in the dma-fence signalling path: nothing here may allocate or take a
+ * dma_resv. pvr_vm_bind_job_create() prepares everything beforehand.
+ */
+static struct dma_fence *
+pvr_vm_bind_run_job(struct drm_sched_job *sched_job)
+{
+	struct pvr_vm_bind_job *job = to_pvr_vm_bind_job(sched_job);
+	struct pvr_vm_context *vm_ctx = job->vm_ctx;
+	bool wait = false;
+	int flush_err;
+	int err = 0;
+	bool cookie;
+
+	if (pvr_vm_context_is_unusable(vm_ctx))
+		return ERR_PTR(-ECANCELED);
+
+	cookie = dma_fence_begin_signalling();
+
+	mutex_lock(&vm_ctx->lock);
+
+	/*
+	 * Sync the page tables and flush the MMU before signalling, including
+	 * tables a failed operation has already changed.
+	 */
+	for (u32 i = 0; i < job->bind_op_count; i++) {
+		struct pvr_vm_bind_op *bind_op = &job->bind_ops[i];
+
+		err = pvr_vm_bind_op_exec(bind_op);
+
+		pvr_mmu_op_context_sync(bind_op->mmu_op_ctx);
+		if (bind_op->type == PVR_VM_BIND_TYPE_UNMAP)
+			wait = true;
+
+		if (err)
+			break;
+	}
+
+	flush_err = pvr_mmu_flush_exec(vm_ctx->pvr_dev, wait);
+	if (!err)
+		err = flush_err;
+
+	if (err)
+		WRITE_ONCE(vm_ctx->unusable, true);
+
+	mutex_unlock(&vm_ctx->lock);
+
+	dma_fence_end_signalling(cookie);
+
+	/* NULL completes the job: the page tables are already updated. */
+	return err ? ERR_PTR(err) : NULL;
+}
+
+static enum drm_gpu_sched_stat
+pvr_vm_bind_timedout_job(struct drm_sched_job *sched_job)
+{
+	WARN(1, "VM bind jobs run on a CPU worker and cannot hang\n");
+
+	return DRM_GPU_SCHED_STAT_RESET;
+}
+
+static void pvr_vm_bind_free_job(struct drm_sched_job *sched_job)
+{
+	struct pvr_vm_bind_job *job = to_pvr_vm_bind_job(sched_job);
+
+	drm_sched_job_cleanup(sched_job);
+
+	queue_work(job->vm_ctx->pvr_dev->sched_wq, &job->cleanup_work);
+}
+
+static const struct drm_sched_backend_ops pvr_vm_bind_sched_ops = {
+	.run_job = pvr_vm_bind_run_job,
+	.timedout_job = pvr_vm_bind_timedout_job,
+	.free_job = pvr_vm_bind_free_job,
+};
+
+/**
+ * pvr_vm_bind_sched_init() - Set up the bind scheduler of a VM context.
+ * @vm_ctx: Target VM context.
+ *
+ * Return: 0 on success, or a negative error code.
+ */
+static int pvr_vm_bind_sched_init(struct pvr_vm_context *vm_ctx)
+{
+	struct pvr_device *pvr_dev = vm_ctx->pvr_dev;
+	struct drm_gpu_scheduler *sched = &vm_ctx->sched;
+	const struct drm_sched_init_args sched_args = {
+		.ops = &pvr_vm_bind_sched_ops,
+		.submit_wq = pvr_dev->sched_wq,
+		.num_rqs = 1,
+		.credit_limit = 1,
+		.hang_limit = 0,
+		/* Bind jobs run on a CPU worker and cannot hang. */
+		.timeout = MAX_SCHEDULE_TIMEOUT,
+		.name = "pvr-vm-bind",
+		.dev = from_pvr_device(pvr_dev)->dev,
+	};
+	int err;
+
+	err = drm_sched_init(sched, &sched_args);
+	if (err)
+		return err;
+
+	err = drm_sched_entity_init(&vm_ctx->entity, DRM_SCHED_PRIORITY_KERNEL,
+				    &sched, 1, NULL);
+	if (err)
+		goto err_sched_fini;
+
+	vm_ctx->sched_initialised = true;
+
+	return 0;
+
+err_sched_fini:
+	drm_sched_fini(sched);
+
+	return err;
+}
+
+/**
+ * pvr_vm_bind_sched_fini() - Tear down the bind scheduler of a VM context.
+ * @vm_ctx: Target VM context.
+ *
+ * Waits for all queued bind jobs to be applied before returning.
+ */
+static void pvr_vm_bind_sched_fini(struct pvr_vm_context *vm_ctx)
+{
+	if (!vm_ctx->sched_initialised)
+		return;
+
+	drm_sched_entity_destroy(&vm_ctx->entity);
+	drm_sched_fini(&vm_ctx->sched);
+	vm_ctx->sched_initialised = false;
+}
+
+/**
+ * pvr_vm_bind_job_create() - Prepare a job applying a batch of bind operations.
+ * @vm_ctx: Target VM context.
+ * @descs: Operations to apply, in array order.
+ * @desc_count: Number of entries in @descs.
+ * @deps: Fences to wait for before applying @descs.
+ * @dep_count: Number of entries in @deps.
+ * @client_id: DRM client the job is accounted to.
+ *
+ * Return: The new job, to be queued with pvr_vm_bind_job_push(), or an
+ * ERR_PTR on failure.
+ */
+static struct pvr_vm_bind_job *
+pvr_vm_bind_job_create(struct pvr_vm_context *vm_ctx,
+		       const struct pvr_vm_bind_desc *descs, u32 desc_count,
+		       struct dma_fence **deps, u32 dep_count, u64 client_id)
+{
+	struct pvr_vm_bind_job *job;
+	struct pvr_vm_bind_op *bind_ops;
+	int err;
+
+	job = kzalloc_obj(*job);
+	if (!job)
+		return ERR_PTR(-ENOMEM);
+
+	job->vm_ctx = pvr_vm_context_get(vm_ctx);
+	INIT_WORK(&job->cleanup_work, pvr_vm_bind_job_cleanup_work);
+
+	bind_ops = pvr_vm_bind_ops_create(vm_ctx, descs, desc_count);
+	if (IS_ERR(bind_ops)) {
+		err = PTR_ERR(bind_ops);
+		goto err_destroy_job;
+	}
+	job->bind_ops = bind_ops;
+	job->bind_op_count = desc_count;
+
+	err = drm_sched_job_init(&job->base, &vm_ctx->entity, 1, vm_ctx,
+				 client_id);
+	if (err)
+		goto err_destroy_job;
+
+	for (u32 i = 0; i < dep_count; i++) {
+		err = drm_sched_job_add_dependency(&job->base,
+						   dma_fence_get(deps[i]));
+		if (err)
+			goto err_cleanup_job;
+	}
+
+	return job;
+
+err_cleanup_job:
+	drm_sched_job_cleanup(&job->base);
+
+err_destroy_job:
+	pvr_vm_bind_job_destroy(job);
+
+	return ERR_PTR(err);
+}
+
+/**
+ * pvr_vm_bind_job_push() - Queue a prepared bind job.
+ * @job: Job to queue. The scheduler owns it from now on.
+ *
+ * Return: The fence that signals once @job has been applied, to be released
+ * with dma_fence_put().
+ */
+static struct dma_fence *pvr_vm_bind_job_push(struct pvr_vm_bind_job *job)
+{
+	struct dma_fence *fence;
+
+	drm_sched_job_arm(&job->base);
+
+	/* Pushing may free @job, so take the fence first. */
+	fence = dma_fence_get(&job->base.s_fence->finished);
+
+	drm_sched_entity_push_job(&job->base);
+
+	return fence;
+}
+
+/**
+ * pvr_vm_bind_async() - Queue a batch of bind operations on a VM context.
+ * @vm_ctx: Target VM context.
+ * @descs: Operations to apply, in array order.
+ * @desc_count: Number of entries in @descs.
+ * @deps: Fences to wait for before applying @descs.
+ * @dep_count: Number of entries in @deps.
+ * @client_id: DRM client the request is accounted to.
+ *
+ * Like pvr_vm_bind(), but applied by the VM context scheduler once every
+ * fence in @deps has signalled. The caller keeps its references on @descs and
+ * @deps.
+ *
+ * Return: A fence signalled once the operations have been applied, to be
+ * released with dma_fence_put(), or an ERR_PTR with nothing queued:
+ *  * -%ECANCELED if @vm_ctx has been left unusable by an earlier failure,
+ *  * -%EINVAL if @vm_ctx is the kernel VM context, or
+ *  * Any other negative error code.
+ */
+struct dma_fence *
+pvr_vm_bind_async(struct pvr_vm_context *vm_ctx,
+		  const struct pvr_vm_bind_desc *descs, u32 desc_count,
+		  struct dma_fence **deps, u32 dep_count, u64 client_id)
+{
+	struct pvr_vm_bind_job *job;
+
+	if (pvr_vm_context_is_unusable(vm_ctx))
+		return ERR_PTR(-ECANCELED);
+
+	if (!vm_ctx->sched_initialised)
+		return ERR_PTR(-EINVAL);
+
+	job = pvr_vm_bind_job_create(vm_ctx, descs, desc_count, deps, dep_count,
+				     client_id);
+	if (IS_ERR(job))
+		return ERR_CAST(job);
+
+	return pvr_vm_bind_job_push(job);
+}
diff --git a/drivers/gpu/drm/imagination/pvr_vm.h b/drivers/gpu/drm/imagination/pvr_vm.h
index bf6d5e802d9..df74b405960 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.h
+++ b/drivers/gpu/drm/imagination/pvr_vm.h
@@ -23,6 +23,9 @@ struct pvr_vm_context;
 /* Forward declaration from <uapi/drm/pvr_drm.h> */
 struct drm_pvr_ioctl_get_heap_info_args;
 
+/* Forward declaration from <linux/dma-fence.h> */
+struct dma_fence;
+
 /* Forward declaration from <drm/drm_exec.h> */
 struct drm_exec;
 
@@ -81,9 +84,15 @@ struct pvr_vm_bind_desc {
 
 int pvr_vm_bind(struct pvr_vm_context *vm_ctx,
 		const struct pvr_vm_bind_desc *descs, u32 desc_count);
+struct dma_fence *
+pvr_vm_bind_async(struct pvr_vm_context *vm_ctx,
+		  const struct pvr_vm_bind_desc *descs, u32 desc_count,
+		  struct dma_fence **deps, u32 dep_count, u64 client_id);
 
 void pvr_vm_unmap_all(struct pvr_vm_context *vm_ctx);
 
+bool pvr_vm_context_is_unusable(struct pvr_vm_context *vm_ctx);
+
 dma_addr_t pvr_vm_get_page_table_root_addr(struct pvr_vm_context *vm_ctx);
 struct dma_resv *pvr_vm_get_dma_resv(struct pvr_vm_context *vm_ctx);
 
diff --git a/include/uapi/drm/pvr_drm.h b/include/uapi/drm/pvr_drm.h
index 55656aa8304..57c22f5f215 100644
--- a/include/uapi/drm/pvr_drm.h
+++ b/include/uapi/drm/pvr_drm.h
@@ -1314,12 +1314,20 @@ struct drm_pvr_ioctl_submit_jobs_args {
  * DOC: PowerVR IOCTL VM_BIND interface
  *
  * %DRM_IOCTL_PVR_VM_BIND applies a batch of map and/or unmap operations to a
- * single VM context before the IOCTL returns.
+ * single VM context, synchronously or, with %DRM_PVR_VM_BIND_ASYNC, from a
+ * queue.
  *
- * Operations within a request are applied in array order.
+ * Operations within a request are applied in array order, and queued requests
+ * in submission order per VM context. A synchronous request does not wait for
+ * the queued ones.
  *
  * A request that fails part way through may leave the address space in an
  * undefined state; how much of it was applied is not reported.
+ *
+ * If a queued request fails, its VM context becomes permanently unusable:
+ * further %DRM_IOCTL_PVR_VM_BIND and %DRM_IOCTL_PVR_SUBMIT_JOBS calls on it
+ * fail with -%ECANCELED, and jobs already queued on it complete with an error.
+ * A new VM context has to be created.
  */
 
 /**
@@ -1379,6 +1387,17 @@ struct drm_pvr_vm_bind_op {
 	__u64 size;
 };
 
+/**
+ * DOC: Flags for the VM_BIND ioctl.
+ *
+ * .. c:macro:: DRM_PVR_VM_BIND_ASYNC
+ *
+ *    Queue the request instead of applying it synchronously.
+ */
+#define DRM_PVR_VM_BIND_ASYNC _BITUL(0)
+
+#define DRM_PVR_VM_BIND_FLAGS_MASK DRM_PVR_VM_BIND_ASYNC
+
 /**
  * struct drm_pvr_ioctl_vm_bind_args - Arguments for %DRM_IOCTL_PVR_VM_BIND.
  */
@@ -1389,13 +1408,20 @@ struct drm_pvr_ioctl_vm_bind_args {
 	 */
 	__u32 vm_context_handle;
 
-	/** @flags: [IN] Reserved. MBZ. */
+	/** @flags: [IN] Combination of ``DRM_PVR_VM_BIND_`` flags. */
 	__u32 flags;
 
 	/** @ops: [IN] Array of &struct drm_pvr_vm_bind_op to apply. */
 	struct drm_pvr_obj_array ops;
 
-	/** @sync_ops: [IN] Reserved. Must be empty. */
+	/**
+	 * @sync_ops: [IN] Sync operations applied once to the request as a
+	 * whole: waits before any of @ops, signals after all of them. Must be
+	 * empty unless %DRM_PVR_VM_BIND_ASYNC is set in @flags.
+	 *
+	 * @ops may be empty, in which case the request only places a
+	 * synchronisation point on the queue.
+	 */
 	struct drm_pvr_obj_array sync_ops;
 };
 

-- 
2.43.0

[PATCH v2 6/6] drm/imagination: Add MAP_SPARSE to VM_BIND

[Gyeyoung Baek]

Vulkan sparse residency requires that an access to an unbound part of a
sparse resource does not fault, so a range has to be mappable without
naming any buffer object.

Add DRM_PVR_VM_BIND_OP_MAP_SPARSE, which maps one dummy object over and
over across the range.

To save memory, one dummy is shared by all sparse mappings, but per file
rather than per device: writes to a sparse range land in the dummy, so
per file keeps clients isolated from each other.

The dummy is 2MiB, so pvr_mmu_map() runs once per 2MiB, but the mapping
itself is still done per page.

Signed-off-by: Gyeyoung Baek <gye976@gmail.com>
---
 drivers/gpu/drm/imagination/pvr_device.h | 10 ++++
 drivers/gpu/drm/imagination/pvr_drv.c    | 63 +++++++++++++++++++++++--
 drivers/gpu/drm/imagination/pvr_vm.c     | 81 +++++++++++++++++++++++++++++---
 drivers/gpu/drm/imagination/pvr_vm.h     |  8 ++++
 include/uapi/drm/pvr_drm.h               | 21 +++++++--
 5 files changed, 169 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/imagination/pvr_device.h b/drivers/gpu/drm/imagination/pvr_device.h
index c246122b4ca..c8a7a949fdc 100644
--- a/drivers/gpu/drm/imagination/pvr_device.h
+++ b/drivers/gpu/drm/imagination/pvr_device.h
@@ -380,6 +380,16 @@ struct pvr_file {
 	 */
 	struct xarray hwrt_handles;
 
+	/**
+	 * @sparse_dummy_bo: Object the file's sparse mappings repeat. Pins its
+	 * pages, so it is created on first use, under @sparse_dummy_lock. Per
+	 * file, so that sparse writes stay out of other clients' view.
+	 */
+	struct pvr_gem_object *sparse_dummy_bo;
+
+	/** @sparse_dummy_lock: Serialises the creation of @sparse_dummy_bo. */
+	struct mutex sparse_dummy_lock;
+
 	/**
 	 * @vm_ctx_handles: Array of VM contexts belonging to this file. Array
 	 * members are of type "struct pvr_vm_context *".
diff --git a/drivers/gpu/drm/imagination/pvr_drv.c b/drivers/gpu/drm/imagination/pvr_drv.c
index c35d437aef7..377fa8d936a 100644
--- a/drivers/gpu/drm/imagination/pvr_drv.c
+++ b/drivers/gpu/drm/imagination/pvr_drv.c
@@ -1137,6 +1137,37 @@ pvr_ioctl_vm_unmap(struct drm_device *drm_dev, void *raw_args,
 	return err;
 }
 
+/**
+ * pvr_file_get_sparse_dummy() - Get a file's sparse dummy object.
+ * @pvr_file: Target PowerVR file.
+ *
+ * Return: The object, with a reference for the caller, or an ERR_PTR on
+ * failure.
+ */
+static struct pvr_gem_object *
+pvr_file_get_sparse_dummy(struct pvr_file *pvr_file)
+{
+	struct pvr_gem_object *dummy;
+
+	mutex_lock(&pvr_file->sparse_dummy_lock);
+
+	dummy = pvr_file->sparse_dummy_bo;
+	if (!dummy) {
+		dummy = pvr_vm_sparse_dummy_create(pvr_file->pvr_dev);
+		if (IS_ERR(dummy))
+			goto err_unlock;
+
+		pvr_file->sparse_dummy_bo = dummy;
+	}
+
+	pvr_gem_object_get(dummy);
+
+err_unlock:
+	mutex_unlock(&pvr_file->sparse_dummy_lock);
+
+	return dummy;
+}
+
 /**
  * pvr_vm_bind_desc_from_uapi() - Translate a userspace bind operation.
  * @desc: Description to fill in.
@@ -1150,6 +1181,8 @@ pvr_vm_bind_desc_from_uapi(struct pvr_vm_bind_desc *desc,
 			   struct pvr_file *pvr_file,
 			   const struct drm_pvr_vm_bind_op *uapi_op)
 {
+	const bool sparse = uapi_op->flags & DRM_PVR_VM_BIND_OP_MAP_SPARSE;
+
 	if (uapi_op->flags & ~DRM_PVR_VM_BIND_OP_FLAGS_MASK)
 		return -EINVAL;
 
@@ -1157,19 +1190,35 @@ pvr_vm_bind_desc_from_uapi(struct pvr_vm_bind_desc *desc,
 		.offset = uapi_op->offset,
 		.device_addr = uapi_op->device_addr,
 		.size = uapi_op->size,
+		.sparse = sparse,
 	};
 
 	switch (uapi_op->flags & DRM_PVR_VM_BIND_OP_TYPE_MASK) {
 	case DRM_PVR_VM_BIND_OP_TYPE_MAP:
 		desc->type = PVR_VM_BIND_TYPE_MAP;
-		desc->obj = pvr_gem_object_from_handle(pvr_file, uapi_op->handle);
-		if (!desc->obj)
-			return -ENOENT;
+
+		if (sparse) {
+			struct pvr_gem_object *dummy;
+
+			if (uapi_op->handle || uapi_op->offset)
+				return -EINVAL;
+
+			dummy = pvr_file_get_sparse_dummy(pvr_file);
+			if (IS_ERR(dummy))
+				return PTR_ERR(dummy);
+
+			desc->obj = dummy;
+		} else {
+			desc->obj = pvr_gem_object_from_handle(pvr_file,
+							       uapi_op->handle);
+			if (!desc->obj)
+				return -ENOENT;
+		}
 
 		return 0;
 
 	case DRM_PVR_VM_BIND_OP_TYPE_UNMAP:
-		if (uapi_op->handle || uapi_op->offset)
+		if (uapi_op->handle || uapi_op->offset || sparse)
 			return -EINVAL;
 
 		desc->type = PVR_VM_BIND_TYPE_UNMAP;
@@ -1615,6 +1664,8 @@ pvr_drm_driver_open(struct drm_device *drm_dev, struct drm_file *file)
 
 	INIT_LIST_HEAD(&pvr_file->contexts);
 
+	mutex_init(&pvr_file->sparse_dummy_lock);
+
 	xa_init_flags(&pvr_file->ctx_handles, XA_FLAGS_ALLOC1);
 	xa_init_flags(&pvr_file->free_list_handles, XA_FLAGS_ALLOC1);
 	xa_init_flags(&pvr_file->hwrt_handles, XA_FLAGS_ALLOC1);
@@ -1653,6 +1704,10 @@ pvr_drm_driver_postclose(__always_unused struct drm_device *drm_dev,
 	pvr_destroy_hwrt_datasets_for_file(pvr_file);
 	pvr_destroy_vm_contexts_for_file(pvr_file);
 
+	if (pvr_file->sparse_dummy_bo)
+		pvr_gem_object_put(pvr_file->sparse_dummy_bo);
+	mutex_destroy(&pvr_file->sparse_dummy_lock);
+
 	kfree(pvr_file);
 	file->driver_priv = NULL;
 }
diff --git a/drivers/gpu/drm/imagination/pvr_vm.c b/drivers/gpu/drm/imagination/pvr_vm.c
index 4e58ea8c50c..0c22c63e498 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.c
+++ b/drivers/gpu/drm/imagination/pvr_vm.c
@@ -22,8 +22,10 @@
 #include <linux/errno.h>
 #include <linux/gfp_types.h>
 #include <linux/kref.h>
+#include <linux/minmax.h>
 #include <linux/mutex.h>
 #include <linux/sched.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/stddef.h>
 #include <linux/workqueue.h>
@@ -188,6 +190,9 @@ struct pvr_vm_bind_op {
 	 */
 	struct pvr_vm_gpuva *next_va;
 
+	/** @sparse: See &pvr_vm_bind_desc.sparse. */
+	bool sparse;
+
 	/** @offset: Offset into @pvr_obj to begin mapping from. */
 	u64 offset;
 
@@ -280,11 +285,19 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
 
 	if (!pvr_device_addr_and_size_are_valid(vm_ctx, desc->device_addr,
 						desc->size) ||
-	    desc->offset & ~PAGE_MASK || desc->size & ~PAGE_MASK ||
-	    desc->offset >= pvr_obj_size || offset_plus_size > pvr_obj_size)
+	    desc->offset & ~PAGE_MASK || desc->size & ~PAGE_MASK)
+		return -EINVAL;
+
+	/*
+	 * A sparse mapping repeats @obj over a range wider than it, so only
+	 * ordinary mappings must fit within @obj.
+	 */
+	if (!desc->sparse &&
+	    (desc->offset >= pvr_obj_size || offset_plus_size > pvr_obj_size))
 		return -EINVAL;
 
 	bind_op->type = PVR_VM_BIND_TYPE_MAP;
+	bind_op->sparse = desc->sparse;
 
 	bind_op->gpuvm_bo = drm_gpuvm_bo_create(&vm_ctx->gpuvm_mgr, obj);
 	if (!bind_op->gpuvm_bo)
@@ -373,6 +386,58 @@ pvr_vm_bind_op_unmap_init(struct pvr_vm_bind_op *bind_op,
 	return err;
 }
 
+/**
+ * pvr_vm_sparse_dummy_create() - Create the object sparse mappings repeat.
+ * @pvr_dev: Target PowerVR device.
+ *
+ * It covers a whole level 0 page table, so that each chunk mapped by
+ * pvr_vm_mmu_map_sparse() fills one.
+ *
+ * Return: The new object, or an ERR_PTR on failure.
+ */
+struct pvr_gem_object *pvr_vm_sparse_dummy_create(struct pvr_device *pvr_dev)
+{
+	return pvr_gem_object_create(pvr_dev, SZ_2M,
+				     DRM_PVR_BO_BYPASS_DEVICE_CACHE);
+}
+
+/**
+ * pvr_vm_mmu_map_sparse() - Map a dummy object over and over across a range.
+ * @mmu_op_ctx: Target MMU op context, created for @dummy.
+ * @dummy: Object to repeat.
+ * @device_addr: Virtual device address at the start of the range.
+ * @size: Size of the range.
+ *
+ * Maps @dummy repeatedly, in chunks aligned to its size, which must be a power
+ * of two.
+ *
+ * Return: 0 on success, or a negative error code with nothing left mapped.
+ */
+static int
+pvr_vm_mmu_map_sparse(struct pvr_mmu_op_context *mmu_op_ctx,
+		      struct pvr_gem_object *dummy, u64 device_addr, u64 size)
+{
+	const u64 dummy_size = pvr_gem_object_size(dummy);
+	u64 mapped = 0;
+	int err;
+
+	while (mapped < size) {
+		const u64 addr = device_addr + mapped;
+		const u64 chunk = min(size - mapped,
+				      dummy_size - (addr & (dummy_size - 1)));
+
+		err = pvr_mmu_map(mmu_op_ctx, chunk, dummy->flags, addr);
+		if (err) {
+			pvr_mmu_unmap(mmu_op_ctx, device_addr, mapped);
+			return err;
+		}
+
+		mapped += chunk;
+	}
+
+	return 0;
+}
+
 /**
  * pvr_vm_gpuva_map() - Insert a mapping into a memory context.
  * @op: gpuva op containing the remap details.
@@ -383,7 +448,7 @@ pvr_vm_bind_op_unmap_init(struct pvr_vm_bind_op *bind_op,
  *
  * Return:
  *  * 0 on success, or
- *  * Any error returned by pvr_mmu_map().
+ *  * Any error returned by pvr_mmu_map() or pvr_vm_mmu_map_sparse().
  */
 static int
 pvr_vm_gpuva_map(struct drm_gpuva_op *op, void *op_ctx)
@@ -395,8 +460,12 @@ pvr_vm_gpuva_map(struct drm_gpuva_op *op, void *op_ctx)
 	if ((op->map.gem.offset | op->map.va.range) & ~PVR_DEVICE_PAGE_MASK)
 		return -EINVAL;
 
-	err = pvr_mmu_map(ctx->mmu_op_ctx, op->map.va.range, pvr_gem->flags,
-			  op->map.va.addr);
+	if (ctx->sparse)
+		err = pvr_vm_mmu_map_sparse(ctx->mmu_op_ctx, pvr_gem,
+					    op->map.va.addr, op->map.va.range);
+	else
+		err = pvr_mmu_map(ctx->mmu_op_ctx, op->map.va.range,
+				  pvr_gem->flags, op->map.va.addr);
 	if (err)
 		return err;
 
@@ -741,7 +810,7 @@ pvr_vm_bind_op_init(struct pvr_vm_bind_op *bind_op,
 		return pvr_vm_bind_op_map_init(bind_op, vm_ctx, desc);
 
 	case PVR_VM_BIND_TYPE_UNMAP:
-		if (WARN_ON(desc->obj))
+		if (WARN_ON(desc->obj || desc->sparse))
 			return -EINVAL;
 
 		return pvr_vm_bind_op_unmap_init(bind_op, vm_ctx, desc);
diff --git a/drivers/gpu/drm/imagination/pvr_vm.h b/drivers/gpu/drm/imagination/pvr_vm.h
index df74b405960..68d5ffa5e88 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.h
+++ b/drivers/gpu/drm/imagination/pvr_vm.h
@@ -80,6 +80,12 @@ struct pvr_vm_bind_desc {
 
 	/** @size: Size of the range. */
 	u64 size;
+
+	/**
+	 * @sparse: Repeat @obj over the range rather than covering it once.
+	 * Map only.
+	 */
+	bool sparse;
 };
 
 int pvr_vm_bind(struct pvr_vm_context *vm_ctx,
@@ -93,6 +99,8 @@ void pvr_vm_unmap_all(struct pvr_vm_context *vm_ctx);
 
 bool pvr_vm_context_is_unusable(struct pvr_vm_context *vm_ctx);
 
+struct pvr_gem_object *pvr_vm_sparse_dummy_create(struct pvr_device *pvr_dev);
+
 dma_addr_t pvr_vm_get_page_table_root_addr(struct pvr_vm_context *vm_ctx);
 struct dma_resv *pvr_vm_get_dma_resv(struct pvr_vm_context *vm_ctx);
 
diff --git a/include/uapi/drm/pvr_drm.h b/include/uapi/drm/pvr_drm.h
index 57c22f5f215..a7c2801cc70 100644
--- a/include/uapi/drm/pvr_drm.h
+++ b/include/uapi/drm/pvr_drm.h
@@ -1333,7 +1333,17 @@ struct drm_pvr_ioctl_submit_jobs_args {
 /**
  * DOC: Flags for VM_BIND operations.
  *
- * Set exactly one operation type in &drm_pvr_vm_bind_op.flags.
+ * Set exactly one operation type in &drm_pvr_vm_bind_op.flags, optionally
+ * ORed with modifiers.
+ *
+ * .. c:macro:: DRM_PVR_VM_BIND_OP_MAP_SPARSE
+ *
+ *    Sparsely map a range without naming a buffer object. Only valid with
+ *    %DRM_PVR_VM_BIND_OP_TYPE_MAP.
+ *
+ *    GPU accesses to the range do not fault. Nothing else is guaranteed:
+ *    reads return undefined values, and writes may become visible through
+ *    other sparse mappings.
  *
  * .. c:macro:: DRM_PVR_VM_BIND_OP_TYPE_MAP
  *
@@ -1349,11 +1359,14 @@ struct drm_pvr_ioctl_submit_jobs_args {
  *
  *    Mask used to extract the operation type.
  */
+#define DRM_PVR_VM_BIND_OP_MAP_SPARSE _BITUL(0)
+
 #define DRM_PVR_VM_BIND_OP_TYPE_MAP (0u << 28)
 #define DRM_PVR_VM_BIND_OP_TYPE_UNMAP (1u << 28)
 #define DRM_PVR_VM_BIND_OP_TYPE_MASK (0xfu << 28)
 
-#define DRM_PVR_VM_BIND_OP_FLAGS_MASK DRM_PVR_VM_BIND_OP_TYPE_MASK
+#define DRM_PVR_VM_BIND_OP_FLAGS_MASK \
+	(DRM_PVR_VM_BIND_OP_MAP_SPARSE | DRM_PVR_VM_BIND_OP_TYPE_MASK)
 
 /**
  * struct drm_pvr_vm_bind_op - A single VM bind operation.
@@ -1366,13 +1379,13 @@ struct drm_pvr_vm_bind_op {
 	 * @handle: [IN] Handle of the target buffer object.
 	 *
 	 * Must be a valid handle returned by %DRM_IOCTL_PVR_CREATE_BO for map
-	 * operations. MBZ for unmap operations.
+	 * operations. MBZ for unmap and sparse map operations.
 	 */
 	__u32 handle;
 
 	/**
 	 * @offset: [IN] Offset into the target buffer object from which to
-	 * begin the mapping. MBZ for unmap operations.
+	 * begin the mapping. MBZ for unmap and sparse map operations.
 	 */
 	__u64 offset;
 

-- 
2.43.0