mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Matthew Brost <matthew.brost@intel.com>
To: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
Cc: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
	"Carlos Santa" <carlos.santa@intel.com>,
	"Ryan Neph" <ryanneph@google.com>,
	"Christian Koenig" <christian.koenig@amd.com>,
	"Huang Rui" <ray.huang@amd.com>,
	"Matthew Auld" <matthew.auld@intel.com>,
	"Maarten Lankhorst" <maarten.lankhorst@linux.intel.com>,
	"Maxime Ripard" <mripard@kernel.org>,
	"Thomas Zimmermann" <tzimmermann@suse.de>,
	"David Airlie" <airlied@gmail.com>,
	"Simona Vetter" <simona@ffwll.ch>,
	linux-kernel@vger.kernel.org,
	"Thomas Hellström" <thomas.hellstrom@linux.intel.com>
Subject: Re: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves
Date: Thu, 19 Nov 2026 13:39:04 -0800	[thread overview]
Message-ID: <av9seBmEpkJn6ubW@gsse-cloud1.jf.intel.com> (raw)
In-Reply-To: <avKDURZLW9pvCFvF@nvishwa1-desk>

On Mon, Nov 09, 2026 at 06:37:05PM -0800, Niranjana Vishwanathapura wrote:
> On Fri, Jul 10, 2026 at 07:55:53PM -0700, Matthew Brost wrote:
> > A defragmentation move relocates a populated tt into a freshly allocated
> > one so that sub-beneficial-order chunks can be upgraded to the device's
> > beneficial order. Previously, if allocation could not complete entirely
> > at beneficial order, the move failed and all progress was discarded.
> > 
> > Instead, once allocation has successfully upgraded part of the object,
> > the remainder of the old tt is harvested to complete the new tt. Thread
> > the BO's old, still-populated tt into the page allocator via a new
> > ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding,
> > any existing chunk already at (or above) the beneficial order is reused
> > directly instead of being reallocated. If allocation later has to fall
> > back below beneficial_order after making partial progress, the remainder
> > of the old tt is harvested instead of failing the move. This preserves
> > the beneficial-order upgrades already made while leaving the rest of the
> > object unchanged, allowing subsequent defragmentation attempts to make
> > additional forward progress.
> > 
> > Harvesting only borrows the pages; they remain owned by the old tt until
> > the move commits. This keeps every unwind path non-destructive: if the
> > (re)allocation or the move later fails, the new tt is torn down without
> > freeing the borrowed pages and the old tt is restored fully intact.
> > Ownership of the shared pages is transferred to the new tt only once the
> > move has committed. ttm_tt_defrag_disown_borrowed() implements both
> > directions of the transfer by clearing a tt's slots for every page it
> > shares with the other tt (same struct page at the same index), so that
> > tt's teardown leaves the shared pages alone and each page is freed
> > exactly once by its owner:
> > 
> > - on the alloc unwind and the failed-move paths, the new tt disowns the
> >  borrowed pages, leaving the old tt as the sole owner;
> > - on the successful-move paths (synchronous teardown and the pipelined
> >  ghost-object teardown), the old tt disowns the now-transferred pages,
> >  leaving the new tt as the sole owner.
> > 
> > Detecting shared pages by same-index struct page identity needs no extra
> > per-tt bookkeeping, since two distinct allocations never alias. This is
> > safe with drivers, such as Xe, that build their own per-tt sg table and
> > DMA-map it independently: each tt maps and unmaps its own sg, so sharing
> > the underlying pages between tts is DMA-clean.
> > 
> > This is a no-op unless ctx->defrag_old_tt is set, which only happens on a
> > defragmentation move.
> > 
> > Cc: Carlos Santa <carlos.santa@intel.com>
> > Cc: Ryan Neph <ryanneph@google.com>
> > Cc: Christian Koenig <christian.koenig@amd.com>
> > Cc: Huang Rui <ray.huang@amd.com>
> > Cc: Matthew Auld <matthew.auld@intel.com>
> > Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> > Cc: Maxime Ripard <mripard@kernel.org>
> > Cc: Thomas Zimmermann <tzimmermann@suse.de>
> > Cc: David Airlie <airlied@gmail.com>
> > Cc: Simona Vetter <simona@ffwll.ch>
> > Cc: dri-devel@lists.freedesktop.org
> > Cc: linux-kernel@vger.kernel.org
> > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > Assisted-by: GitHub_Copilot:claude-opus-4.8
> > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > 
> > ---
> > 
> > v3:
> > - Document the chunk-head invariant read by
> >   ttm_pool_harvest_remaining() (Sashiko)
> > ---
> > drivers/gpu/drm/ttm/ttm_bo.c      |  20 ++-
> > drivers/gpu/drm/ttm/ttm_bo_util.c |   5 +
> > drivers/gpu/drm/ttm/ttm_pool.c    | 235 ++++++++++++++++++++++++++++--
> > drivers/gpu/drm/ttm/ttm_tt.c      |  67 +++++++++
> > include/drm/ttm/ttm_bo.h          |  12 ++
> > include/drm/ttm/ttm_tt.h          |   2 +
> > 6 files changed, 328 insertions(+), 13 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> > index c1629b4c4f0f..ab2e637a017f 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > @@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
> > 		}
> > 
> > 		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
> > +			/*
> > +			 * Let the page allocator harvest already
> > +			 * beneficial-order chunks out of the old tt to back
> > +			 * the new one, rather than reallocating them.
> > +			 */
> > +			ctx->defrag_old_tt = defrag_old_tt;
> > 			ret = ttm_bo_populate(bo, ctx);
> > +			ctx->defrag_old_tt = NULL;
> > 			if (ret) {
> > 				/*
> > 				 * Discharge the freshly allocated backing and
> > @@ -235,6 +242,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
> > 	}
> > 
> > 	if (bo->defrag_old_tt) {
> > +		/*
> > +		 * Successful defrag move: any pages the new tt borrowed from the
> > +		 * old tt are now owned by the new tt. Disown them from the old
> > +		 * tt before it is torn down so they are not freed.
> > +		 */
> > +		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
> > 		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
> > 		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
> > 		bo->defrag_old_tt = NULL;
> > @@ -245,7 +258,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
> > 
> > out_err:
> > 	if (bo->defrag_old_tt) {
> > -		/* Failed defrag move: restore the original backing. */
> > +		/*
> > +		 * Failed defrag move: drop any pages the new tt borrowed from
> > +		 * the old tt (still owned by the old tt) before destroying the
> > +		 * new tt, then restore the original backing.
> > +		 */
> > +		ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
> > 		ttm_bo_tt_destroy(bo);
> > 		bo->ttm = bo->defrag_old_tt;
> > 		bo->defrag_old_tt = NULL;
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > index 771313162c8d..7cb718fc97c0 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > @@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
> > 		 * tt (the move's source) to the ghost so it is kept populated
> > 		 * until @fence signals and then unbound and destroyed, instead
> > 		 * of being torn down synchronously. @bo keeps its new tt.
> > +		 *
> > +		 * Any pages the new tt borrowed from the old tt are now owned by
> > +		 * the new tt; disown them from the old tt so the ghost teardown
> > +		 * leaves them alone.
> > 		 */
> > +		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
> > 		ghost_obj->ttm = bo->defrag_old_tt;
> > 		ghost_obj->defrag_old_tt = NULL;
> > 		bo->defrag_old_tt = NULL;
> > diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> > index 2598d2f33767..52f5e64f05dd 100644
> > --- a/drivers/gpu/drm/ttm/ttm_pool.c
> > +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> > @@ -792,6 +792,148 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
> > 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
> > }
> > 
> > +/*
> > + * Defragmentation page harvesting.
> > + *
> > + * A defrag move relocates a populated tt into a freshly allocated one in order
> > + * to upgrade sub-beneficial-order chunks to the device's beneficial order.
> > + * Chunks in the old tt (ctx->defrag_old_tt) that are ALREADY at the beneficial
> > + * order do not need to be reallocated: their physical pages can simply be
> > + * transferred ("harvested") into the new tt at the same offset, avoiding the
> > + * alloc/free churn that is significant for large buffers.
> > + *
> > + * Harvesting only borrows the pages: the new tt simply references the old tt's
> > + * pages, which remain owned by the old tt. This keeps the unwind paths simple -
> > + * if the (re)allocation or the move later fails, the new tt is torn down without
> > + * freeing the borrowed pages and the old tt is restored fully intact. Ownership
> > + * of the shared pages is transferred to the new tt only once the move has
> > + * committed, via ttm_tt_defrag_disown_borrowed() (see ttm_tt.c).
> > + */
> > +
> > +/*
> > + * Return the allocation order of the old tt chunk whose head sits at @off, or 0
> > + * if @off is not an old chunk head (it falls inside a chunk, or the slot is
> > + * empty / backed up). @cursor monotonically tracks the old chunk boundary and
> > + * must be advanced past every offset that has already been queried; a page
> > + * order may only be read from a chunk head page, hence the lazy walk.
> > + */
> > +static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
> > +					      struct ttm_tt *old_tt,
> > +					      pgoff_t off, pgoff_t *cursor)
> > +{
> > +	struct page *p;
> > +
> > +	while (*cursor < off) {
> > +		p = old_tt->pages[*cursor];
> > +		if (!p || ttm_backup_page_ptr_is_handle(p))
> > +			return 0;
> 
> Assisted-by: LLM
> 

Yes.

> ttm_backup_page_ptr_is_handle() is only check here, but not in the harvest
> function. Probably we won't endup with ptr_is_handle() case during defrag,
> but thisasymmetry makes it bit confusing.
> 

In Xe we only allow defrag in TT placement, so
ttm_backup_page_ptr_is_handle probably could be omitted here.

> > +		*cursor += 1UL << ttm_pool_page_order(pool, p);
> > +	}
> > +
> > +	if (*cursor != off)
> > +		return 0;
> > +
> > +	p = old_tt->pages[off];
> > +	if (!p || ttm_backup_page_ptr_is_handle(p))
> > +		return 0;
> > +
> > +	return ttm_pool_page_order(pool, p);
> > +}
> > +
> > +/*
> > + * Borrow an already beneficial-order chunk from the old tt into @alloc. The
> > + * pages stay owned by the old tt until the move commits.
> > + */
> > +static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
> > +				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
> > +{
> > +	pgoff_t nr = 1UL << order;
> > +	int r;
> > +
> > +	/*
> > +	 * A harvested chunk keeps the (same) bo's caching, so the deferred
> > +	 * caching range is consistent here; flush it before committing, exactly
> > +	 * like the caching-consistent path in ttm_pool_page_allocated().
> > +	 */
> > +	r = ttm_pool_apply_caching(alloc);
> > +	if (r)
> > +		return r;
> > +
> > +	/*
> > +	 * The borrowed chunk's page pointers (consecutive struct pages of one
> > +	 * compound allocation) and dma addresses already sit contiguously in
> > +	 * the old tt at @off. Copy the whole run with memcpy() instead of the
> > +	 * per-page increment loop in ttm_pool_allocated_page_commit(): for a
> > +	 * large defrag the borrowed pages dominate the new tt, so this turns
> > +	 * the populate fill from O(num_pages) individual stores into a handful
> > +	 * of bulk copies.
> > +	 */
> > +	memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
> > +	alloc->pages += nr;
> > +	alloc->remaining_pages -= nr;
> > +
> > +	if (alloc->dma_addr) {
> > +		memcpy(alloc->dma_addr, &old_tt->dma_address[off],
> > +		       nr * sizeof(*alloc->dma_addr));
> > +		alloc->dma_addr += nr;
> > +	}
> > +
> > +	alloc->caching_divide = alloc->pages;
> > +
> > +	return 0;
> > +}
> > +
> > +static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
> > +				      struct ttm_tt *old_tt, pgoff_t off,
> > +				      struct ttm_pool_alloc_state *alloc)
> > +{
> > +	/*
> > +	 * @off is always an old-tt chunk head here, never a tail page, so
> > +	 * reading the per-chunk order below is safe. This holds because:
> > +	 *  - Old chunks are self-aligned: greedy largest-power-of-two fill
> > +	 *    places an order-k chunk at a 2^k-aligned offset, so any
> > +	 *    beneficial-order-aligned offset is a chunk boundary.
> > +	 *  - Fresh defrag allocations are capped at the beneficial order (see
> > +	 *    ttm_pool_iter_acquire_page()), so they advance @off in beneficial
> > +	 *    steps and never overshoot into the middle of a larger old chunk;
> > +	 *    super-beneficial old chunks are always harvested whole by
> > +	 *    ttm_pool_iter_reuse_old().
> > +	 *  - We are only reached once the defrag budget/prealloc is exhausted,
> > +	 *    which in the capped/prealloc config leaves @off beneficial-aligned
> > +	 *    (or, in the sub-beneficial tail, already walking old boundaries).
> > +	 */
> > +	while (alloc->remaining_pages) {
> > +		struct page *p = old_tt->pages[off];
> > +		unsigned int order = ttm_pool_page_order(pool, p);
> > +		pgoff_t nr = 1UL << order;
> > +		int r;
> > +
> > +		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
> > +		if (r)
> > +			return r;
> > +
> > +		off += nr;
> > +	}
> 
> The 'off' being always at chunk head is documented, but not checked/enforced
> in the code. Though currently, we won't have it otherwise, but tomorrow if

I don't know how to enforce this. p->private is the chunk order at the
head page only but an order of 0 is a valid value, i.e., I'm not sure
how to implement 'page is chunk head' in the current code.

> we do for some reason, it will go unnoticed. Probably worth adding a warning
> here.
> 
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> > + * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
> > + * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
> > + * @TTM_POOL_ITER_RETRY: Loop state was adjusted (e.g. order lowered or a chunk
> > + *	harvested); re-enter the loop body for the next order.
> > + * @TTM_POOL_ITER_STOP: The allocation is complete; stop the loop.
> > + *
> > + * Phase helpers return one of these (>= 0) on success, or a negative errno on
> > + * failure (the caller then unwinds via error_free_all).
> > + */
> > +enum ttm_pool_iter_action {
> > +	TTM_POOL_ITER_FILL,
> > +	TTM_POOL_ITER_RETRY,
> > +	TTM_POOL_ITER_STOP,
> 
> NIT...STOP is really not needed here, RETRY will do as the loop is
> expected to end there. But it is fine.
>

Not following this - __ttm_pool_alloc look at TTM_POOL_ITER_RETRY vs
STOP.
 
> > +};
> > +
> > /**
> >  * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
> >  *
> > @@ -806,6 +948,7 @@ struct ttm_pool_alloc_iter {
> > 	const struct ttm_operation_ctx *ctx;
> > 	struct ttm_pool_alloc_state *alloc;
> > 	struct ttm_pool_tt_restore *restore;
> > +	struct ttm_tt *defrag_old_tt;
> > 	unsigned int beneficial_order;
> > 	gfp_t gfp_flags;
> > 
> > @@ -814,9 +957,50 @@ struct ttm_pool_alloc_iter {
> > 	enum ttm_caching page_caching;
> > 	bool allow_pools;
> > 	bool fail_beneficial;
> > +	unsigned int alloc_count;
> > +	pgoff_t old_cursor;
> > 	struct page *p;
> > };
> > 
> > +/*
> > + * Defrag move: reuse an already beneficial-order chunk from the old tt at this
> > + * offset rather than reallocating it. Harvest whatever order the old chunk has
> > + * (>= beneficial order), which may exceed the order the new tt would otherwise
> > + * pick. An order below beneficial_order is only reused for the unaligned tail
> > + * (where there is no benefit to reallocating). Only borrows the pages; the alloc
> > + * unwind path can drop them again.
> > + *
> > + * Returns TTM_POOL_ITER_RETRY when a chunk was harvested, TTM_POOL_ITER_FILL to
> > + * fall through to a fresh allocation, or a negative errno.
> > + */
> > +static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
> > +{
> > +	unsigned int harvest_order;
> > +	pgoff_t off;
> > +	int r;
> > +
> > +	if (!it->defrag_old_tt || !it->beneficial_order)
> > +		return TTM_POOL_ITER_FILL;
> > +
> > +	off = it->tt->num_pages - it->alloc->remaining_pages;
> > +	harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
> > +						  off, &it->old_cursor);
> > +
> > +	if (harvest_order < it->beneficial_order &&
> > +	    it->order >= it->beneficial_order)
> > +		return TTM_POOL_ITER_FILL;
> > +
> > +	it->order = harvest_order;
> > +	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
> > +	if (r)
> > +		return r;
> > +
> > +	it->page_caching = it->tt->caching;
> > +	it->allow_pools = true;
> > +
> > +	return TTM_POOL_ITER_RETRY;
> > +}
> > +
> > /*
> >  * Acquire a single page for the current order, leaving it in @it->p (NULL on
> >  * failure). Tries a same-order pool page, then a fresh system allocation. Fault
> > @@ -859,9 +1043,10 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
> > /*
> >  * The current order could not be satisfied. Lower it and retry, recording a
> >  * sub-optimal backing (so the driver can later re-defrag) when we drop below
> > - * the beneficial order.
> > + * the beneficial order. For a defrag move that has already made partial
> > + * progress, harvest the remaining old pages and stop instead.
> >  *
> > - * Returns 0 to retry the (lowered) order, or a negative errno.
> > + * Returns TTM_POOL_ITER_RETRY, TTM_POOL_ITER_STOP, or a negative errno.
> >  */
> > static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
> > {
> > @@ -871,16 +1056,24 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
> > 	if (!it->order)
> > 		return -ENOMEM;
> > 
> > -	/*
> > -	 * Failing to allocate at the device's beneficial order means we are
> > -	 * about to back this object with a sub-optimal (smaller order) set of
> > -	 * pages. Record it so the driver can later try to defragment the object
> > -	 * back to beneficial order.
> > -	 */
> > 	if (it->fail_beneficial || at_beneficial) {
> > 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
> > -		if (it->ctx->defrag)
> > +
> > +		if (it->alloc_count && it->defrag_old_tt) {
> > +			pgoff_t off = it->tt->num_pages -
> > +				      it->alloc->remaining_pages;
> > +			int r;
> > +
> > +			r = ttm_pool_harvest_remaining(it->pool,
> > +						       it->defrag_old_tt, off,
> > +						       it->alloc);
> > +			if (r)
> > +				return r;
> > +
> > +			return TTM_POOL_ITER_STOP;
> > +		} else if (it->defrag_old_tt) {
> > 			return -ENOMEM;
> > +		}
> > 	}
> > 
> > 	if (it->fail_beneficial)
> > @@ -890,7 +1083,7 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
> > 	it->page_caching = it->tt->caching;
> > 	it->allow_pools = true;
> > 
> > -	return 0;
> > +	return TTM_POOL_ITER_RETRY;
> > }
> > 
> > static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > @@ -904,6 +1097,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > 		.ctx = ctx,
> > 		.alloc = alloc,
> > 		.restore = restore,
> > +		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
> > 		.beneficial_order = ttm_pool_beneficial_order(pool),
> > 		.page_caching = tt->caching,
> > 		.allow_pools = true,
> > @@ -931,13 +1125,22 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
> > 	     alloc->remaining_pages;
> > 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
> > +		/* Reuse an already beneficial-order chunk from the old tt. */
> > +		r = ttm_pool_iter_reuse_old(&it);
> > +		if (r < 0)
> > +			goto error_free_all;
> > +		if (r == TTM_POOL_ITER_RETRY)
> > +			continue;
> > +
> > 		/* Acquire a page (pool / system) for this order. */
> > 		ttm_pool_iter_acquire_page(&it);
> > 		if (!it.p) {
> > 			r = ttm_pool_iter_lower_order(&it);
> > -			if (r)
> > +			if (r < 0)
> > 				goto error_free_all;
> > -			continue;
> > +			if (r == TTM_POOL_ITER_RETRY)
> > +				continue;
> > +			break;	/* TTM_POOL_ITER_STOP */
> > 		}
> > 
> > 		r = ttm_pool_page_allocated(pool, it.order, it.p,
> > @@ -951,6 +1154,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > 			if (r)
> > 				goto error_free_all;
> > 		}
> > +
> > +		it.alloc_count++;
> > 	}
> > 
> > 	r = ttm_pool_apply_caching(alloc);
> > @@ -969,6 +1174,12 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > 	if (tt->restore)
> > 		return r;
> > 
> > +	/*
> > +	 * Drop any pages borrowed from the old tt during a defrag move: they are
> > +	 * still owned by the old tt and must not be freed here.
> > +	 */
> > +	ttm_tt_defrag_disown_borrowed(tt, it.defrag_old_tt);
> > +
> > 	caching_divide = alloc->caching_divide - tt->pages;
> > 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
> > 	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
> > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> > index b645a1818184..2388ff794a59 100644
> > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > @@ -450,6 +450,73 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
> > }
> > EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_unpopulate);
> > 
> > +/**
> > + * ttm_tt_defrag_disown_borrowed() - Drop borrowed defrag pages from a tt
> > + * @tt: The tt to drop the borrowed page references from.
> > + * @src: The tt the pages may have been harvested from, or NULL.
> > + *
> > + * During a defragmentation move the freshly (re)allocated tt may harvest
> > + * already beneficial-order chunks from the BO's old tt, sharing the underlying
> > + * pages instead of reallocating them. Until the move commits these pages stay
> > + * owned by the old tt. Clearing a tt's slots for every page it shares with
> > + * @src (the same struct page at the same index) makes that tt's teardown leave
> > + * the shared pages alone, so they are freed exactly once by their owner.
> > + *
> > + * This is used both to unwind a failed defrag move (disown the borrowed pages
> > + * from the new tt, leaving the old tt intact) and to commit a successful one
> > + * (disown the now-transferred pages from the old tt, leaving them owned by the
> > + * new tt).
> > + */
> > +void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
> > +{
> > +	pgoff_t i = 0, run_start = 0;
> > +	bool in_run = false;
> > +
> > +	if (!tt || !src)
> > +		return;
> > +
> > +	/*
> > +	 * Borrowing happens at chunk granularity: a borrowed chunk shares the
> > +	 * same struct page on both tts across its whole allocation order, while
> > +	 * a non-borrowed chunk differs across its whole order. Stride by @src's
> > +	 * chunk order instead of walking every page, coalescing the borrowed
> > +	 * (matching) chunks into runs and clearing each run with a single
> > +	 * memset rather than a per-page store. @src is a fully populated,
> > +	 * non-dma_alloc tt (the defrag path), so its page order is recorded in
> > +	 * page->private and read via ttm_pool_page_order_nodma().
> > +	 */
> > +	while (i < tt->num_pages) {
> > +		struct page *sp = src->pages[i];
> > +		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
> 
> Here ttm_pool_page_order_nodma() is used where as in harvest funciton,
> ttm_pool_page_order() is used. Though it doesn't matter to Xe, this asymmetry
> makes it bit confusing.
> 

We don't have the pool here, hence we use ttm_pool_page_order_nodma.

Defrag isn't supported on dma_alloc pages, thus is safe.

Matt

> Niranjana
> 
> > +		pgoff_t nr = 1UL << order;
> > +
> > +		if (sp && tt->pages[i] == sp) {
> > +			if (!in_run) {
> > +				run_start = i;
> > +				in_run = true;
> > +			}
> > +		} else if (in_run) {
> > +			memset(&tt->pages[run_start], 0,
> > +			       (i - run_start) * sizeof(*tt->pages));
> > +			if (tt->dma_address)
> > +				memset(&tt->dma_address[run_start], 0,
> > +				       (i - run_start) *
> > +				       sizeof(*tt->dma_address));
> > +			in_run = false;
> > +		}
> > +
> > +		i += nr;
> > +	}
> > +
> > +	if (in_run) {
> > +		memset(&tt->pages[run_start], 0,
> > +		       (i - run_start) * sizeof(*tt->pages));
> > +		if (tt->dma_address)
> > +			memset(&tt->dma_address[run_start], 0,
> > +			       (i - run_start) * sizeof(*tt->dma_address));
> > +	}
> > +}
> > +
> > #ifdef CONFIG_DEBUG_FS
> > 
> > /* Test the shrinker functions and dump the result */
> > diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
> > index f4f8a8c29f04..fcaba5fb8bf7 100644
> > --- a/include/drm/ttm/ttm_bo.h
> > +++ b/include/drm/ttm/ttm_bo.h
> > @@ -211,6 +211,18 @@ struct ttm_operation_ctx {
> > 	 * order.
> > 	 */
> > 	bool defrag;
> > +	/**
> > +	 * @defrag_old_tt: During a defragmentation move, the bo's existing
> > +	 * (populated) tt that is being relocated. The page allocator may
> > +	 * harvest already beneficial-order chunks from this tt to back the new
> > +	 * tt instead of reallocating them, avoiding alloc/free churn for large
> > +	 * buffers. Harvested pages are only borrowed: they stay owned by
> > +	 * @defrag_old_tt until the move commits, at which point
> > +	 * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
> > +	 * transferred pages so they are not freed when the old tt is torn down.
> > +	 * NULL for non-defrag operations.
> > +	 */
> > +	struct ttm_tt *defrag_old_tt;
> > 	/**
> > 	 * @resv: Reservation object to be used together with
> > 	 * @allow_res_evict.
> > diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> > index ce7533677d77..55c35bcf134d 100644
> > --- a/include/drm/ttm/ttm_tt.h
> > +++ b/include/drm/ttm/ttm_tt.h
> > @@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
> >  */
> > void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
> > 
> > +void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
> > +
> > /**
> >  * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
> >  *
> > -- 
> > 2.34.1
> > 

  reply	other threads:[~2026-11-19 21:39 UTC|newest]

Thread overview: 82+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-11  2:55 [PATCH v3 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs Matthew Brost
2026-07-11  2:55 ` [PATCH v3 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order Matthew Brost
2026-11-06  2:55   ` Niranjana Vishwanathapura
2026-11-19 18:47     ` Matthew Brost
2026-07-11  2:55 ` [PATCH v3 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper Matthew Brost
2026-11-05 11:27   ` Matthew Auld
2026-11-06  2:30   ` Niranjana Vishwanathapura
2026-07-11  2:55 ` [PATCH v3 03/33] drm/ttm: Record sub-optimal page order allocations in ttm_tt Matthew Brost
2026-11-06  2:34   ` Niranjana Vishwanathapura
2026-11-06 20:06     ` Matthew Brost
2026-07-11  2:55 ` [PATCH v3 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() Matthew Brost
2026-11-06  2:39   ` Niranjana Vishwanathapura
2026-11-19 19:14     ` Matthew Brost
2026-07-11  2:55 ` [PATCH v3 05/33] drm/ttm: Support defragmentation moves Matthew Brost
2026-11-06  2:51   ` Niranjana Vishwanathapura
2026-11-19 20:15     ` Matthew Brost
2026-07-11  2:55 ` [PATCH v3 06/33] drm/ttm: Add fault injection for beneficial-order allocation failures Matthew Brost
2026-11-06  2:54   ` Niranjana Vishwanathapura
2026-11-19 20:49     ` Matthew Brost
2026-07-11  2:55 ` [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves Matthew Brost
2026-11-10  2:37   ` Niranjana Vishwanathapura
2026-11-19 21:39     ` Matthew Brost [this message]
2026-07-11  2:55 ` [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move Matthew Brost
2026-11-10  2:44   ` Niranjana Vishwanathapura
2026-07-11  2:55 ` [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock Matthew Brost
2026-11-11  0:29   ` Niranjana Vishwanathapura
2026-11-11  7:49   ` Niranjana Vishwanathapura
2026-07-11  2:55 ` [PATCH v3 10/33] drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc() Matthew Brost
2026-11-11  7:31   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 15/33] drm/xe: Track BOs backed at a sub-optimal page order Matthew Brost
2026-11-07  4:49   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure Matthew Brost
2026-11-07  5:02   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies Matthew Brost
2026-11-03 12:58   ` Matthew Auld
2026-11-03 20:08     ` Matthew Brost
2026-11-04 12:37       ` Matthew Auld
2026-11-07  5:07   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 18/33] drm/xe: Handle defrag moves in xe_bo_move() Matthew Brost
2026-11-07  5:12   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves Matthew Brost
2026-11-10  4:25   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 20/33] drm/xe: Add a page defragmentation worker Matthew Brost
2026-11-07  5:27   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 21/33] drm/xe: Add defrag GT stats Matthew Brost
2026-11-09  1:10   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 22/33] drm/xe: Add Kconfig.profile options for BO defrag configuration Matthew Brost
2026-11-09  1:31   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 23/33] drm/xe: Defrag using out-of-lock page preallocation Matthew Brost
2026-11-11  0:39   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 24/33] drm/xe: Add defrag profiling tracepoints Matthew Brost
2026-11-09  4:34   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock Matthew Brost
2026-11-11  7:36   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl Matthew Brost
2026-11-11  7:37   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 27/33] drm/xe: Add IOVA-based xe_res_cursor variant Matthew Brost
2026-11-04 12:23   ` Matthew Auld
2026-11-13  6:14   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 28/33] drm/xe: Use IOVA-based DMA mapping for eligible tt BOs Matthew Brost
2026-11-03 19:00   ` Matthew Auld
2026-11-03 20:21     ` Matthew Brost
2026-11-04 11:39       ` Matthew Auld
2026-11-05  3:04         ` Matthew Brost
2026-11-13  6:28   ` Niranjana Vishwanathapura
2026-11-13 20:09     ` Matthew Brost
2026-11-16  4:10       ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize Matthew Brost
2026-11-14  1:47   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 30/33] drm/xe: Add packed copy-step IOVA mapping for defrag Matthew Brost
2026-11-14  1:52   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies Matthew Brost
2026-11-14  1:57   ` Niranjana Vishwanathapura
2026-11-16  0:41   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job Matthew Brost
2026-11-14  2:02   ` Niranjana Vishwanathapura
2026-07-11  2:56 ` [PATCH v3 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock Matthew Brost
2026-07-11 10:46   ` Christian König
2026-07-13 20:24     ` Matthew Brost
2026-07-14  9:17       ` Christian König
2026-07-14 15:15         ` Matthew Brost
2026-11-16  1:01 ` [PATCH v3 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs Niranjana Vishwanathapura

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av9seBmEpkJn6ubW@gsse-cloud1.jf.intel.com \
    --to=matthew.brost@intel.com \
    --cc=airlied@gmail.com \
    --cc=carlos.santa@intel.com \
    --cc=christian.koenig@amd.com \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maarten.lankhorst@linux.intel.com \
    --cc=matthew.auld@intel.com \
    --cc=mripard@kernel.org \
    --cc=niranjana.vishwanathapura@intel.com \
    --cc=ray.huang@amd.com \
    --cc=ryanneph@google.com \
    --cc=simona@ffwll.ch \
    --cc=thomas.hellstrom@linux.intel.com \
    --cc=tzimmermann@suse.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®