Re: [PATCH v3 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

[Matthew Brost <matthew.brost@intel.com>]

On Tue, Nov 03, 2026 at 12:58:39PM +0000, Matthew Auld wrote:
> On 11/07/2026 03:56, Matthew Brost wrote:
> > Add xe_migrate_copy_defrag(), a copy helper for the upcoming page
> > defragmentation path where a single BO has its backing reallocated at a
> > more beneficial page order and its contents must be relocated from the
> > old pages to the new ones entirely on the GPU.
> > 
> > The contents are relocated in up to two passes:
> > 
> >   - Data pass (always): copy the data pages verbatim. The pages are
> >     mapped without the compression PAT, so the (possibly compressed)
> >     bytes are copied as-is rather than being decompressed and
> >     recompressed, and no CCS aux copy is appended.
> > 
> >   - CCS pass (need_ccs): copy the CCS aux (compression) state. Both the
> >     old and new data pages are mapped with the compression PAT so the
> >     copy engine accesses the aux state indirectly, issuing one
> >     indirect -> indirect XY_CTRL_SURF_COPY_BLT; the data blit is skipped.
> 
> 
> Just to double check, there were no issues with this flow for a compressed
> surface? Not 100% sure if reading from the src with uncompressed PAT will
> zero out the CCS state (update to uncompressed status), such that when you
> do the CCS pass it is already gone. Quite sure it does that for a write, but
> I guess not for a read, so we are fine here?
> 

This isn't my area of expertise, but IIRC I spun on this patch several
times before I finally got it right. In my first few attempts, when I
ran something like 3DMark Wildlife Extreme or WebGL applications that
would trigger the defragmenter, I could see visual corruption on the
screen (i.e., the buffers were clearly being corrupted). I eventually
landed on the approach detailed here and have not seen any visual issues
since. So I think the reads without the compressed PAT are fine, at
least on PTL.

I also have a local update to [1] where I use the compressed PAT, and
the tests seem to pass. I can post an update to this test shortly.

Matt

[1] https://patchwork.freedesktop.org/patch/739244/?series=169723&rev=2

> > 
> > The two passes are issued separately because they require opposite
> > compression PAT settings on the data page mappings (cleared for the
> > verbatim data copy, set for the indirect CCS access), which a single
> > mapping cannot satisfy. Both passes run in order on the same migrate
> > queue and are captured by the returned fence.
> > 
> > The source pages are passed explicitly via an sg_table argument (the
> > old, still populated backing) since they no longer match the BO's
> > current backing. The boolean arguments to __xe_migrate_copy() are folded
> > into struct xe_migrate_copy_flags to keep the growing set of copy
> > variants readable.
> > 
> > Cc: Carlos Santa <carlos.santa@intel.com>
> > Cc: Ryan Neph <ryanneph@google.com>
> > Cc: Christian Koenig <christian.koenig@amd.com>
> > Cc: Huang Rui <ray.huang@amd.com>
> > Cc: Matthew Auld <matthew.auld@intel.com>
> > Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> > Cc: Maxime Ripard <mripard@kernel.org>
> > Cc: Thomas Zimmermann <tzimmermann@suse.de>
> > Cc: David Airlie <airlied@gmail.com>
> > Cc: Simona Vetter <simona@ffwll.ch>
> > Cc: dri-devel@lists.freedesktop.org
> > Cc: linux-kernel@vger.kernel.org
> > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > Assisted-by: GitHub_Copilot:claude-opus-4.8
> > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > ---
> >   drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
> >   drivers/gpu/drm/xe/xe_migrate.h |   8 ++
> >   2 files changed, 128 insertions(+), 16 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
> > index 92d5e81ceac2..305c6e1942fb 100644
> > --- a/drivers/gpu/drm/xe/xe_migrate.c
> > +++ b/drivers/gpu/drm/xe/xe_migrate.c
> > @@ -858,13 +858,39 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
> >   	return flush_flags;
> >   }
> > +/**
> > + * struct xe_migrate_copy_flags - Flags for __xe_migrate_copy()
> > + * @copy_only_ccs: Only copy the CCS aux state, aliasing the destination data
> > + * mapping to the source (used when the data pages are shared between src/dst).
> > + * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
> > + * compression PAT index for the data copy.
> > + * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
> > + * two distinct page sets (old @src_sg pages -> new pages). The data blit is
> > + * skipped (the data was copied by the @defrag_copy pass) and both the source
> > + * and destination data pages are mapped with the compression PAT so the aux
> > + * state is copied indirect -> indirect.
> > + * @defrag_copy: Defrag move pass that copies the data pages between two
> > + * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
> > + * aux state. The data pages are mapped without the compression PAT so the
> > + * (possibly compressed) bytes are copied verbatim rather than being
> > + * decompressed and recompressed, and no CCS aux copy is appended; the aux
> > + * state is migrated separately by the @defrag_ccs pass. The two passes need
> > + * opposite compression PAT settings, which is why they cannot be combined.
> > + */
> > +struct xe_migrate_copy_flags {
> > +	u32 copy_only_ccs : 1;
> > +	u32 is_vram_resolve : 1;
> > +	u32 defrag_ccs : 1;
> > +	u32 defrag_copy : 1;
> > +};
> > +
> >   static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   					   struct xe_bo *src_bo,
> >   					   struct xe_bo *dst_bo,
> >   					   struct ttm_resource *src,
> >   					   struct ttm_resource *dst,
> > -					   bool copy_only_ccs,
> > -					   bool is_vram_resolve)
> > +					   struct sg_table *src_sg,
> > +					   struct xe_migrate_copy_flags flags)
> >   {
> >   	struct xe_gt *gt = m->tile->primary_gt;
> >   	struct xe_device *xe = gt_to_xe(gt);
> > @@ -881,20 +907,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   	bool src_is_vram = mem_type_is_vram(src->mem_type);
> >   	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
> >   	bool type_device = src_bo->ttm.type == ttm_bo_type_device;
> > -	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
> > +	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
> > +		!flags.defrag_copy;
> >   	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
> > -		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
> > -	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
> > +		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
> > +		!flags.defrag_copy;
> > +	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
> > +		!flags.defrag_ccs && !flags.defrag_copy;
> >   	/*
> >   	 * For decompression operation, always use the compression PAT index.
> >   	 * Otherwise, only use the compression PAT index for device memory
> >   	 * when copying from VRAM to system memory.
> >   	 */
> > -	bool use_comp_pat = is_vram_resolve || (type_device &&
> > +	bool use_comp_pat = flags.is_vram_resolve || (type_device &&
> >   			    xe_device_has_flat_ccs(xe) &&
> >   			    GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram);
> > +	/*
> > +	 * The defrag CCS pass migrates the aux state from the old pages
> > +	 * (@src_sg) to the new pages with both sides accessed indirectly, so
> > +	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
> > +	 */
> > +
> >   	/* Copying CCS between two different BOs is not supported yet. */
> >   	if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
> >   		return ERR_PTR(-EINVAL);
> > @@ -903,7 +938,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   		return ERR_PTR(-EINVAL);
> >   	if (!src_is_vram)
> > -		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
> > +		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
> >   	else
> >   		xe_res_first(src, 0, size, &src_it);
> >   	if (!dst_is_vram)
> > @@ -912,7 +947,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   		xe_res_first(dst, 0, size, &dst_it);
> >   	if (copy_system_ccs)
> > -		xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
> > +		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
> > +				xe_bo_ccs_pages_start(src_bo),
> >   				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
> >   				&ccs_it);
> > @@ -942,7 +978,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
> >   					      &src_L0_ofs, &src_L0_pt, 0, 0,
> >   					      avail_pts);
> > -		if (copy_only_ccs) {
> > +		if (flags.copy_only_ccs) {
> >   			dst_L0_ofs = src_L0_ofs;
> >   		} else {
> >   			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
> > @@ -963,7 +999,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   		}
> >   		/* Add copy commands size here */
> > -		batch_size += ((copy_only_ccs) ? 0 : emit_copy_cmd_len(xe)) +
> > +		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
> > +			       emit_copy_cmd_len(xe)) +
> >   			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
> >   		bb = xe_bb_new(gt, batch_size, usm);
> > @@ -975,13 +1012,15 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   		if (src_is_vram && xe_migrate_allow_identity(src_L0, &src_it))
> >   			xe_res_next(&src_it, src_L0);
> >   		else
> > -			emit_pte(m, bb, src_L0_pt, src_is_vram, copy_system_ccs || use_comp_pat,
> > +			emit_pte(m, bb, src_L0_pt, src_is_vram,
> > +				 copy_system_ccs || use_comp_pat || flags.defrag_ccs,
> >   				 &src_it, src_L0, src);
> >   		if (dst_is_vram && xe_migrate_allow_identity(src_L0, &dst_it))
> >   			xe_res_next(&dst_it, src_L0);
> > -		else if (!copy_only_ccs)
> > -			emit_pte(m, bb, dst_L0_pt, dst_is_vram, copy_system_ccs,
> > +		else if (!flags.copy_only_ccs)
> > +			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
> > +				 copy_system_ccs || flags.defrag_ccs,
> >   				 &dst_it, src_L0, dst);
> >   		if (copy_system_ccs)
> > @@ -990,7 +1029,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
> >   		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
> >   		update_idx = bb->len;
> > -		if (!copy_only_ccs)
> > +		if (!flags.copy_only_ccs && !flags.defrag_ccs)
> >   			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
> >   		if (needs_ccs_emit)
> > @@ -1078,7 +1117,69 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
> >   				  struct ttm_resource *dst,
> >   				  bool copy_only_ccs)
> >   {
> > -	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, copy_only_ccs, false);
> > +	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
> > +				 (struct xe_migrate_copy_flags) {
> > +					 .copy_only_ccs = copy_only_ccs,
> > +				 });
> > +}
> > +
> > +/**
> > + * xe_migrate_copy_defrag() - Migrate a BO's backing to a reallocated page set.
> > + * @m: The migration context.
> > + * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
> > + * @src: The source TTM resource (the old backing).
> > + * @dst: The dst TTM resource (the freshly reallocated backing).
> > + * @src_sg: Scatter-gather table of the source (old) backing pages.
> > + * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
> > + *
> > + * Used by the defrag path where a single BO has its backing reallocated at a
> > + * more beneficial page order. The contents are relocated from the old pages
> > + * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
> > + * in up to two passes:
> > + *
> > + *  - Data pass (always): copy the data pages verbatim. The pages are mapped
> > + *    without the compression PAT, so the (possibly compressed) bytes are copied
> > + *    as-is rather than being decompressed and recompressed, and no CCS aux copy
> > + *    is appended.
> > + *
> > + *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
> > + *    and new data pages are mapped with the compression PAT so the copy engine
> > + *    accesses the aux state indirectly, issuing one indirect -> indirect
> > + *    XY_CTRL_SURF_COPY_BLT; the data blit is skipped.
> > + *
> > + * The two passes are issued separately because they require opposite
> > + * compression PAT settings on the data page mappings (cleared for the verbatim
> > + * data copy, set for the indirect CCS access), which a single mapping cannot
> > + * satisfy. Both passes run in order on the same migrate queue.
> > + *
> > + * Return: Pointer to a dma_fence representing the last copy batch, or an error
> > + * pointer on failure.
> > + */
> > +struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
> > +					 struct xe_bo *bo,
> > +					 struct ttm_resource *src,
> > +					 struct ttm_resource *dst,
> > +					 struct sg_table *src_sg,
> > +					 bool need_ccs)
> > +{
> > +	struct dma_fence *fence2, *fence =
> > +		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
> > +				  (struct xe_migrate_copy_flags) {
> > +					 .defrag_copy = true,
> > +					 });
> > +
> > +	if (IS_ERR(fence) || !need_ccs)
> > +		return fence;
> > +
> > +	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
> > +				   (struct xe_migrate_copy_flags) {
> > +				   .defrag_ccs = true,
> > +				   });
> > +	if (IS_ERR(fence2))
> > +		dma_fence_wait(fence, false);
> > +	dma_fence_put(fence);
> > +
> > +	return fence2;
> >   }
> >   /**
> > @@ -1096,7 +1197,10 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
> >   				     struct xe_bo *bo,
> >   				     struct ttm_resource *res)
> >   {
> > -	return __xe_migrate_copy(m, bo, bo, res, res, false, true);
> > +	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
> > +				 (struct xe_migrate_copy_flags) {
> > +					 .is_vram_resolve = true,
> > +				 });
> >   }
> >   /**
> > diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
> > index 965c45889c72..e4f996942a6d 100644
> > --- a/drivers/gpu/drm/xe/xe_migrate.h
> > +++ b/drivers/gpu/drm/xe/xe_migrate.h
> > @@ -11,6 +11,7 @@
> >   struct dma_fence;
> >   struct drm_pagemap_addr;
> >   struct iosys_map;
> > +struct sg_table;
> >   struct ttm_resource;
> >   struct xe_bo;
> > @@ -132,6 +133,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
> >   				  struct ttm_resource *dst,
> >   				  bool copy_only_ccs);
> > +struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
> > +					 struct xe_bo *bo,
> > +					 struct ttm_resource *src,
> > +					 struct ttm_resource *dst,
> > +					 struct sg_table *src_sg,
> > +					 bool need_ccs);
> > +
> >   struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
> >   				     struct xe_bo *bo,
> >   				     struct ttm_resource *res);
>