From: Matthew Brost <matthew.brost@intel.com>
To: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
Cc: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
"Carlos Santa" <carlos.santa@intel.com>,
"Ryan Neph" <ryanneph@google.com>,
"Christian Koenig" <christian.koenig@amd.com>,
"Huang Rui" <ray.huang@amd.com>,
"Matthew Auld" <matthew.auld@intel.com>,
"Maarten Lankhorst" <maarten.lankhorst@linux.intel.com>,
"Maxime Ripard" <mripard@kernel.org>,
"Thomas Zimmermann" <tzimmermann@suse.de>,
"David Airlie" <airlied@gmail.com>,
"Simona Vetter" <simona@ffwll.ch>,
linux-kernel@vger.kernel.org,
"Thomas Hellström" <thomas.hellstrom@linux.intel.com>
Subject: Re: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves
Date: Thu, 19 Nov 2026 13:39:04 -0800 [thread overview]
Message-ID: <av9seBmEpkJn6ubW@gsse-cloud1.jf.intel.com> (raw)
In-Reply-To: <avKDURZLW9pvCFvF@nvishwa1-desk>
On Mon, Nov 09, 2026 at 06:37:05PM -0800, Niranjana Vishwanathapura wrote:
> On Fri, Jul 10, 2026 at 07:55:53PM -0700, Matthew Brost wrote:
> > A defragmentation move relocates a populated tt into a freshly allocated
> > one so that sub-beneficial-order chunks can be upgraded to the device's
> > beneficial order. Previously, if allocation could not complete entirely
> > at beneficial order, the move failed and all progress was discarded.
> >
> > Instead, once allocation has successfully upgraded part of the object,
> > the remainder of the old tt is harvested to complete the new tt. Thread
> > the BO's old, still-populated tt into the page allocator via a new
> > ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding,
> > any existing chunk already at (or above) the beneficial order is reused
> > directly instead of being reallocated. If allocation later has to fall
> > back below beneficial_order after making partial progress, the remainder
> > of the old tt is harvested instead of failing the move. This preserves
> > the beneficial-order upgrades already made while leaving the rest of the
> > object unchanged, allowing subsequent defragmentation attempts to make
> > additional forward progress.
> >
> > Harvesting only borrows the pages; they remain owned by the old tt until
> > the move commits. This keeps every unwind path non-destructive: if the
> > (re)allocation or the move later fails, the new tt is torn down without
> > freeing the borrowed pages and the old tt is restored fully intact.
> > Ownership of the shared pages is transferred to the new tt only once the
> > move has committed. ttm_tt_defrag_disown_borrowed() implements both
> > directions of the transfer by clearing a tt's slots for every page it
> > shares with the other tt (same struct page at the same index), so that
> > tt's teardown leaves the shared pages alone and each page is freed
> > exactly once by its owner:
> >
> > - on the alloc unwind and the failed-move paths, the new tt disowns the
> > borrowed pages, leaving the old tt as the sole owner;
> > - on the successful-move paths (synchronous teardown and the pipelined
> > ghost-object teardown), the old tt disowns the now-transferred pages,
> > leaving the new tt as the sole owner.
> >
> > Detecting shared pages by same-index struct page identity needs no extra
> > per-tt bookkeeping, since two distinct allocations never alias. This is
> > safe with drivers, such as Xe, that build their own per-tt sg table and
> > DMA-map it independently: each tt maps and unmaps its own sg, so sharing
> > the underlying pages between tts is DMA-clean.
> >
> > This is a no-op unless ctx->defrag_old_tt is set, which only happens on a
> > defragmentation move.
> >
> > Cc: Carlos Santa <carlos.santa@intel.com>
> > Cc: Ryan Neph <ryanneph@google.com>
> > Cc: Christian Koenig <christian.koenig@amd.com>
> > Cc: Huang Rui <ray.huang@amd.com>
> > Cc: Matthew Auld <matthew.auld@intel.com>
> > Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> > Cc: Maxime Ripard <mripard@kernel.org>
> > Cc: Thomas Zimmermann <tzimmermann@suse.de>
> > Cc: David Airlie <airlied@gmail.com>
> > Cc: Simona Vetter <simona@ffwll.ch>
> > Cc: dri-devel@lists.freedesktop.org
> > Cc: linux-kernel@vger.kernel.org
> > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > Assisted-by: GitHub_Copilot:claude-opus-4.8
> > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> >
> > ---
> >
> > v3:
> > - Document the chunk-head invariant read by
> > ttm_pool_harvest_remaining() (Sashiko)
> > ---
> > drivers/gpu/drm/ttm/ttm_bo.c | 20 ++-
> > drivers/gpu/drm/ttm/ttm_bo_util.c | 5 +
> > drivers/gpu/drm/ttm/ttm_pool.c | 235 ++++++++++++++++++++++++++++--
> > drivers/gpu/drm/ttm/ttm_tt.c | 67 +++++++++
> > include/drm/ttm/ttm_bo.h | 12 ++
> > include/drm/ttm/ttm_tt.h | 2 +
> > 6 files changed, 328 insertions(+), 13 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> > index c1629b4c4f0f..ab2e637a017f 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > @@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
> > }
> >
> > if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
> > + /*
> > + * Let the page allocator harvest already
> > + * beneficial-order chunks out of the old tt to back
> > + * the new one, rather than reallocating them.
> > + */
> > + ctx->defrag_old_tt = defrag_old_tt;
> > ret = ttm_bo_populate(bo, ctx);
> > + ctx->defrag_old_tt = NULL;
> > if (ret) {
> > /*
> > * Discharge the freshly allocated backing and
> > @@ -235,6 +242,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
> > }
> >
> > if (bo->defrag_old_tt) {
> > + /*
> > + * Successful defrag move: any pages the new tt borrowed from the
> > + * old tt are now owned by the new tt. Disown them from the old
> > + * tt before it is torn down so they are not freed.
> > + */
> > + ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
> > ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
> > ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
> > bo->defrag_old_tt = NULL;
> > @@ -245,7 +258,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
> >
> > out_err:
> > if (bo->defrag_old_tt) {
> > - /* Failed defrag move: restore the original backing. */
> > + /*
> > + * Failed defrag move: drop any pages the new tt borrowed from
> > + * the old tt (still owned by the old tt) before destroying the
> > + * new tt, then restore the original backing.
> > + */
> > + ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
> > ttm_bo_tt_destroy(bo);
> > bo->ttm = bo->defrag_old_tt;
> > bo->defrag_old_tt = NULL;
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > index 771313162c8d..7cb718fc97c0 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > @@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
> > * tt (the move's source) to the ghost so it is kept populated
> > * until @fence signals and then unbound and destroyed, instead
> > * of being torn down synchronously. @bo keeps its new tt.
> > + *
> > + * Any pages the new tt borrowed from the old tt are now owned by
> > + * the new tt; disown them from the old tt so the ghost teardown
> > + * leaves them alone.
> > */
> > + ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
> > ghost_obj->ttm = bo->defrag_old_tt;
> > ghost_obj->defrag_old_tt = NULL;
> > bo->defrag_old_tt = NULL;
> > diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> > index 2598d2f33767..52f5e64f05dd 100644
> > --- a/drivers/gpu/drm/ttm/ttm_pool.c
> > +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> > @@ -792,6 +792,148 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
> > return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
> > }
> >
> > +/*
> > + * Defragmentation page harvesting.
> > + *
> > + * A defrag move relocates a populated tt into a freshly allocated one in order
> > + * to upgrade sub-beneficial-order chunks to the device's beneficial order.
> > + * Chunks in the old tt (ctx->defrag_old_tt) that are ALREADY at the beneficial
> > + * order do not need to be reallocated: their physical pages can simply be
> > + * transferred ("harvested") into the new tt at the same offset, avoiding the
> > + * alloc/free churn that is significant for large buffers.
> > + *
> > + * Harvesting only borrows the pages: the new tt simply references the old tt's
> > + * pages, which remain owned by the old tt. This keeps the unwind paths simple -
> > + * if the (re)allocation or the move later fails, the new tt is torn down without
> > + * freeing the borrowed pages and the old tt is restored fully intact. Ownership
> > + * of the shared pages is transferred to the new tt only once the move has
> > + * committed, via ttm_tt_defrag_disown_borrowed() (see ttm_tt.c).
> > + */
> > +
> > +/*
> > + * Return the allocation order of the old tt chunk whose head sits at @off, or 0
> > + * if @off is not an old chunk head (it falls inside a chunk, or the slot is
> > + * empty / backed up). @cursor monotonically tracks the old chunk boundary and
> > + * must be advanced past every offset that has already been queried; a page
> > + * order may only be read from a chunk head page, hence the lazy walk.
> > + */
> > +static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
> > + struct ttm_tt *old_tt,
> > + pgoff_t off, pgoff_t *cursor)
> > +{
> > + struct page *p;
> > +
> > + while (*cursor < off) {
> > + p = old_tt->pages[*cursor];
> > + if (!p || ttm_backup_page_ptr_is_handle(p))
> > + return 0;
>
> Assisted-by: LLM
>
Yes.
> ttm_backup_page_ptr_is_handle() is only check here, but not in the harvest
> function. Probably we won't endup with ptr_is_handle() case during defrag,
> but thisasymmetry makes it bit confusing.
>
In Xe we only allow defrag in TT placement, so
ttm_backup_page_ptr_is_handle probably could be omitted here.
> > + *cursor += 1UL << ttm_pool_page_order(pool, p);
> > + }
> > +
> > + if (*cursor != off)
> > + return 0;
> > +
> > + p = old_tt->pages[off];
> > + if (!p || ttm_backup_page_ptr_is_handle(p))
> > + return 0;
> > +
> > + return ttm_pool_page_order(pool, p);
> > +}
> > +
> > +/*
> > + * Borrow an already beneficial-order chunk from the old tt into @alloc. The
> > + * pages stay owned by the old tt until the move commits.
> > + */
> > +static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
> > + pgoff_t off, struct ttm_pool_alloc_state *alloc)
> > +{
> > + pgoff_t nr = 1UL << order;
> > + int r;
> > +
> > + /*
> > + * A harvested chunk keeps the (same) bo's caching, so the deferred
> > + * caching range is consistent here; flush it before committing, exactly
> > + * like the caching-consistent path in ttm_pool_page_allocated().
> > + */
> > + r = ttm_pool_apply_caching(alloc);
> > + if (r)
> > + return r;
> > +
> > + /*
> > + * The borrowed chunk's page pointers (consecutive struct pages of one
> > + * compound allocation) and dma addresses already sit contiguously in
> > + * the old tt at @off. Copy the whole run with memcpy() instead of the
> > + * per-page increment loop in ttm_pool_allocated_page_commit(): for a
> > + * large defrag the borrowed pages dominate the new tt, so this turns
> > + * the populate fill from O(num_pages) individual stores into a handful
> > + * of bulk copies.
> > + */
> > + memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
> > + alloc->pages += nr;
> > + alloc->remaining_pages -= nr;
> > +
> > + if (alloc->dma_addr) {
> > + memcpy(alloc->dma_addr, &old_tt->dma_address[off],
> > + nr * sizeof(*alloc->dma_addr));
> > + alloc->dma_addr += nr;
> > + }
> > +
> > + alloc->caching_divide = alloc->pages;
> > +
> > + return 0;
> > +}
> > +
> > +static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
> > + struct ttm_tt *old_tt, pgoff_t off,
> > + struct ttm_pool_alloc_state *alloc)
> > +{
> > + /*
> > + * @off is always an old-tt chunk head here, never a tail page, so
> > + * reading the per-chunk order below is safe. This holds because:
> > + * - Old chunks are self-aligned: greedy largest-power-of-two fill
> > + * places an order-k chunk at a 2^k-aligned offset, so any
> > + * beneficial-order-aligned offset is a chunk boundary.
> > + * - Fresh defrag allocations are capped at the beneficial order (see
> > + * ttm_pool_iter_acquire_page()), so they advance @off in beneficial
> > + * steps and never overshoot into the middle of a larger old chunk;
> > + * super-beneficial old chunks are always harvested whole by
> > + * ttm_pool_iter_reuse_old().
> > + * - We are only reached once the defrag budget/prealloc is exhausted,
> > + * which in the capped/prealloc config leaves @off beneficial-aligned
> > + * (or, in the sub-beneficial tail, already walking old boundaries).
> > + */
> > + while (alloc->remaining_pages) {
> > + struct page *p = old_tt->pages[off];
> > + unsigned int order = ttm_pool_page_order(pool, p);
> > + pgoff_t nr = 1UL << order;
> > + int r;
> > +
> > + r = ttm_pool_harvest_page(old_tt, order, off, alloc);
> > + if (r)
> > + return r;
> > +
> > + off += nr;
> > + }
>
> The 'off' being always at chunk head is documented, but not checked/enforced
> in the code. Though currently, we won't have it otherwise, but tomorrow if
I don't know how to enforce this. p->private is the chunk order at the
head page only but an order of 0 is a valid value, i.e., I'm not sure
how to implement 'page is chunk head' in the current code.
> we do for some reason, it will go unnoticed. Probably worth adding a warning
> here.
>
> > +
> > + return 0;
> > +}
> > +
> > +/**
> > + * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
> > + * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
> > + * @TTM_POOL_ITER_RETRY: Loop state was adjusted (e.g. order lowered or a chunk
> > + * harvested); re-enter the loop body for the next order.
> > + * @TTM_POOL_ITER_STOP: The allocation is complete; stop the loop.
> > + *
> > + * Phase helpers return one of these (>= 0) on success, or a negative errno on
> > + * failure (the caller then unwinds via error_free_all).
> > + */
> > +enum ttm_pool_iter_action {
> > + TTM_POOL_ITER_FILL,
> > + TTM_POOL_ITER_RETRY,
> > + TTM_POOL_ITER_STOP,
>
> NIT...STOP is really not needed here, RETRY will do as the loop is
> expected to end there. But it is fine.
>
Not following this - __ttm_pool_alloc look at TTM_POOL_ITER_RETRY vs
STOP.
> > +};
> > +
> > /**
> > * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
> > *
> > @@ -806,6 +948,7 @@ struct ttm_pool_alloc_iter {
> > const struct ttm_operation_ctx *ctx;
> > struct ttm_pool_alloc_state *alloc;
> > struct ttm_pool_tt_restore *restore;
> > + struct ttm_tt *defrag_old_tt;
> > unsigned int beneficial_order;
> > gfp_t gfp_flags;
> >
> > @@ -814,9 +957,50 @@ struct ttm_pool_alloc_iter {
> > enum ttm_caching page_caching;
> > bool allow_pools;
> > bool fail_beneficial;
> > + unsigned int alloc_count;
> > + pgoff_t old_cursor;
> > struct page *p;
> > };
> >
> > +/*
> > + * Defrag move: reuse an already beneficial-order chunk from the old tt at this
> > + * offset rather than reallocating it. Harvest whatever order the old chunk has
> > + * (>= beneficial order), which may exceed the order the new tt would otherwise
> > + * pick. An order below beneficial_order is only reused for the unaligned tail
> > + * (where there is no benefit to reallocating). Only borrows the pages; the alloc
> > + * unwind path can drop them again.
> > + *
> > + * Returns TTM_POOL_ITER_RETRY when a chunk was harvested, TTM_POOL_ITER_FILL to
> > + * fall through to a fresh allocation, or a negative errno.
> > + */
> > +static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
> > +{
> > + unsigned int harvest_order;
> > + pgoff_t off;
> > + int r;
> > +
> > + if (!it->defrag_old_tt || !it->beneficial_order)
> > + return TTM_POOL_ITER_FILL;
> > +
> > + off = it->tt->num_pages - it->alloc->remaining_pages;
> > + harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
> > + off, &it->old_cursor);
> > +
> > + if (harvest_order < it->beneficial_order &&
> > + it->order >= it->beneficial_order)
> > + return TTM_POOL_ITER_FILL;
> > +
> > + it->order = harvest_order;
> > + r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
> > + if (r)
> > + return r;
> > +
> > + it->page_caching = it->tt->caching;
> > + it->allow_pools = true;
> > +
> > + return TTM_POOL_ITER_RETRY;
> > +}
> > +
> > /*
> > * Acquire a single page for the current order, leaving it in @it->p (NULL on
> > * failure). Tries a same-order pool page, then a fresh system allocation. Fault
> > @@ -859,9 +1043,10 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
> > /*
> > * The current order could not be satisfied. Lower it and retry, recording a
> > * sub-optimal backing (so the driver can later re-defrag) when we drop below
> > - * the beneficial order.
> > + * the beneficial order. For a defrag move that has already made partial
> > + * progress, harvest the remaining old pages and stop instead.
> > *
> > - * Returns 0 to retry the (lowered) order, or a negative errno.
> > + * Returns TTM_POOL_ITER_RETRY, TTM_POOL_ITER_STOP, or a negative errno.
> > */
> > static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
> > {
> > @@ -871,16 +1056,24 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
> > if (!it->order)
> > return -ENOMEM;
> >
> > - /*
> > - * Failing to allocate at the device's beneficial order means we are
> > - * about to back this object with a sub-optimal (smaller order) set of
> > - * pages. Record it so the driver can later try to defragment the object
> > - * back to beneficial order.
> > - */
> > if (it->fail_beneficial || at_beneficial) {
> > it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
> > - if (it->ctx->defrag)
> > +
> > + if (it->alloc_count && it->defrag_old_tt) {
> > + pgoff_t off = it->tt->num_pages -
> > + it->alloc->remaining_pages;
> > + int r;
> > +
> > + r = ttm_pool_harvest_remaining(it->pool,
> > + it->defrag_old_tt, off,
> > + it->alloc);
> > + if (r)
> > + return r;
> > +
> > + return TTM_POOL_ITER_STOP;
> > + } else if (it->defrag_old_tt) {
> > return -ENOMEM;
> > + }
> > }
> >
> > if (it->fail_beneficial)
> > @@ -890,7 +1083,7 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
> > it->page_caching = it->tt->caching;
> > it->allow_pools = true;
> >
> > - return 0;
> > + return TTM_POOL_ITER_RETRY;
> > }
> >
> > static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > @@ -904,6 +1097,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > .ctx = ctx,
> > .alloc = alloc,
> > .restore = restore,
> > + .defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
> > .beneficial_order = ttm_pool_beneficial_order(pool),
> > .page_caching = tt->caching,
> > .allow_pools = true,
> > @@ -931,13 +1125,22 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
> > alloc->remaining_pages;
> > it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
> > + /* Reuse an already beneficial-order chunk from the old tt. */
> > + r = ttm_pool_iter_reuse_old(&it);
> > + if (r < 0)
> > + goto error_free_all;
> > + if (r == TTM_POOL_ITER_RETRY)
> > + continue;
> > +
> > /* Acquire a page (pool / system) for this order. */
> > ttm_pool_iter_acquire_page(&it);
> > if (!it.p) {
> > r = ttm_pool_iter_lower_order(&it);
> > - if (r)
> > + if (r < 0)
> > goto error_free_all;
> > - continue;
> > + if (r == TTM_POOL_ITER_RETRY)
> > + continue;
> > + break; /* TTM_POOL_ITER_STOP */
> > }
> >
> > r = ttm_pool_page_allocated(pool, it.order, it.p,
> > @@ -951,6 +1154,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > if (r)
> > goto error_free_all;
> > }
> > +
> > + it.alloc_count++;
> > }
> >
> > r = ttm_pool_apply_caching(alloc);
> > @@ -969,6 +1174,12 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > if (tt->restore)
> > return r;
> >
> > + /*
> > + * Drop any pages borrowed from the old tt during a defrag move: they are
> > + * still owned by the old tt and must not be freed here.
> > + */
> > + ttm_tt_defrag_disown_borrowed(tt, it.defrag_old_tt);
> > +
> > caching_divide = alloc->caching_divide - tt->pages;
> > ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
> > ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
> > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> > index b645a1818184..2388ff794a59 100644
> > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > @@ -450,6 +450,73 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
> > }
> > EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_unpopulate);
> >
> > +/**
> > + * ttm_tt_defrag_disown_borrowed() - Drop borrowed defrag pages from a tt
> > + * @tt: The tt to drop the borrowed page references from.
> > + * @src: The tt the pages may have been harvested from, or NULL.
> > + *
> > + * During a defragmentation move the freshly (re)allocated tt may harvest
> > + * already beneficial-order chunks from the BO's old tt, sharing the underlying
> > + * pages instead of reallocating them. Until the move commits these pages stay
> > + * owned by the old tt. Clearing a tt's slots for every page it shares with
> > + * @src (the same struct page at the same index) makes that tt's teardown leave
> > + * the shared pages alone, so they are freed exactly once by their owner.
> > + *
> > + * This is used both to unwind a failed defrag move (disown the borrowed pages
> > + * from the new tt, leaving the old tt intact) and to commit a successful one
> > + * (disown the now-transferred pages from the old tt, leaving them owned by the
> > + * new tt).
> > + */
> > +void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
> > +{
> > + pgoff_t i = 0, run_start = 0;
> > + bool in_run = false;
> > +
> > + if (!tt || !src)
> > + return;
> > +
> > + /*
> > + * Borrowing happens at chunk granularity: a borrowed chunk shares the
> > + * same struct page on both tts across its whole allocation order, while
> > + * a non-borrowed chunk differs across its whole order. Stride by @src's
> > + * chunk order instead of walking every page, coalescing the borrowed
> > + * (matching) chunks into runs and clearing each run with a single
> > + * memset rather than a per-page store. @src is a fully populated,
> > + * non-dma_alloc tt (the defrag path), so its page order is recorded in
> > + * page->private and read via ttm_pool_page_order_nodma().
> > + */
> > + while (i < tt->num_pages) {
> > + struct page *sp = src->pages[i];
> > + unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
>
> Here ttm_pool_page_order_nodma() is used where as in harvest funciton,
> ttm_pool_page_order() is used. Though it doesn't matter to Xe, this asymmetry
> makes it bit confusing.
>
We don't have the pool here, hence we use ttm_pool_page_order_nodma.
Defrag isn't supported on dma_alloc pages, thus is safe.
Matt
> Niranjana
>
> > + pgoff_t nr = 1UL << order;
> > +
> > + if (sp && tt->pages[i] == sp) {
> > + if (!in_run) {
> > + run_start = i;
> > + in_run = true;
> > + }
> > + } else if (in_run) {
> > + memset(&tt->pages[run_start], 0,
> > + (i - run_start) * sizeof(*tt->pages));
> > + if (tt->dma_address)
> > + memset(&tt->dma_address[run_start], 0,
> > + (i - run_start) *
> > + sizeof(*tt->dma_address));
> > + in_run = false;
> > + }
> > +
> > + i += nr;
> > + }
> > +
> > + if (in_run) {
> > + memset(&tt->pages[run_start], 0,
> > + (i - run_start) * sizeof(*tt->pages));
> > + if (tt->dma_address)
> > + memset(&tt->dma_address[run_start], 0,
> > + (i - run_start) * sizeof(*tt->dma_address));
> > + }
> > +}
> > +
> > #ifdef CONFIG_DEBUG_FS
> >
> > /* Test the shrinker functions and dump the result */
> > diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
> > index f4f8a8c29f04..fcaba5fb8bf7 100644
> > --- a/include/drm/ttm/ttm_bo.h
> > +++ b/include/drm/ttm/ttm_bo.h
> > @@ -211,6 +211,18 @@ struct ttm_operation_ctx {
> > * order.
> > */
> > bool defrag;
> > + /**
> > + * @defrag_old_tt: During a defragmentation move, the bo's existing
> > + * (populated) tt that is being relocated. The page allocator may
> > + * harvest already beneficial-order chunks from this tt to back the new
> > + * tt instead of reallocating them, avoiding alloc/free churn for large
> > + * buffers. Harvested pages are only borrowed: they stay owned by
> > + * @defrag_old_tt until the move commits, at which point
> > + * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
> > + * transferred pages so they are not freed when the old tt is torn down.
> > + * NULL for non-defrag operations.
> > + */
> > + struct ttm_tt *defrag_old_tt;
> > /**
> > * @resv: Reservation object to be used together with
> > * @allow_res_evict.
> > diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> > index ce7533677d77..55c35bcf134d 100644
> > --- a/include/drm/ttm/ttm_tt.h
> > +++ b/include/drm/ttm/ttm_tt.h
> > @@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
> > */
> > void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
> >
> > +void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
> > +
> > /**
> > * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
> > *
> > --
> > 2.34.1
> >
next prev parent reply other threads:[~2026-11-19 21:39 UTC|newest]
Thread overview: 82+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-11 2:55 [PATCH v3 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs Matthew Brost
2026-07-11 2:55 ` [PATCH v3 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order Matthew Brost
2026-11-06 2:55 ` Niranjana Vishwanathapura
2026-11-19 18:47 ` Matthew Brost
2026-07-11 2:55 ` [PATCH v3 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper Matthew Brost
2026-11-05 11:27 ` Matthew Auld
2026-11-06 2:30 ` Niranjana Vishwanathapura
2026-07-11 2:55 ` [PATCH v3 03/33] drm/ttm: Record sub-optimal page order allocations in ttm_tt Matthew Brost
2026-11-06 2:34 ` Niranjana Vishwanathapura
2026-11-06 20:06 ` Matthew Brost
2026-07-11 2:55 ` [PATCH v3 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() Matthew Brost
2026-11-06 2:39 ` Niranjana Vishwanathapura
2026-11-19 19:14 ` Matthew Brost
2026-07-11 2:55 ` [PATCH v3 05/33] drm/ttm: Support defragmentation moves Matthew Brost
2026-11-06 2:51 ` Niranjana Vishwanathapura
2026-11-19 20:15 ` Matthew Brost
2026-07-11 2:55 ` [PATCH v3 06/33] drm/ttm: Add fault injection for beneficial-order allocation failures Matthew Brost
2026-11-06 2:54 ` Niranjana Vishwanathapura
2026-11-19 20:49 ` Matthew Brost
2026-07-11 2:55 ` [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves Matthew Brost
2026-11-10 2:37 ` Niranjana Vishwanathapura
2026-11-19 21:39 ` Matthew Brost [this message]
2026-07-11 2:55 ` [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move Matthew Brost
2026-11-10 2:44 ` Niranjana Vishwanathapura
2026-07-11 2:55 ` [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock Matthew Brost
2026-11-11 0:29 ` Niranjana Vishwanathapura
2026-11-11 7:49 ` Niranjana Vishwanathapura
2026-07-11 2:55 ` [PATCH v3 10/33] drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc() Matthew Brost
2026-11-11 7:31 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 15/33] drm/xe: Track BOs backed at a sub-optimal page order Matthew Brost
2026-11-07 4:49 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure Matthew Brost
2026-11-07 5:02 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies Matthew Brost
2026-11-03 12:58 ` Matthew Auld
2026-11-03 20:08 ` Matthew Brost
2026-11-04 12:37 ` Matthew Auld
2026-11-07 5:07 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 18/33] drm/xe: Handle defrag moves in xe_bo_move() Matthew Brost
2026-11-07 5:12 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves Matthew Brost
2026-11-10 4:25 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 20/33] drm/xe: Add a page defragmentation worker Matthew Brost
2026-11-07 5:27 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 21/33] drm/xe: Add defrag GT stats Matthew Brost
2026-11-09 1:10 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 22/33] drm/xe: Add Kconfig.profile options for BO defrag configuration Matthew Brost
2026-11-09 1:31 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 23/33] drm/xe: Defrag using out-of-lock page preallocation Matthew Brost
2026-11-11 0:39 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 24/33] drm/xe: Add defrag profiling tracepoints Matthew Brost
2026-11-09 4:34 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock Matthew Brost
2026-11-11 7:36 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl Matthew Brost
2026-11-11 7:37 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 27/33] drm/xe: Add IOVA-based xe_res_cursor variant Matthew Brost
2026-11-04 12:23 ` Matthew Auld
2026-11-13 6:14 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 28/33] drm/xe: Use IOVA-based DMA mapping for eligible tt BOs Matthew Brost
2026-11-03 19:00 ` Matthew Auld
2026-11-03 20:21 ` Matthew Brost
2026-11-04 11:39 ` Matthew Auld
2026-11-05 3:04 ` Matthew Brost
2026-11-13 6:28 ` Niranjana Vishwanathapura
2026-11-13 20:09 ` Matthew Brost
2026-11-16 4:10 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize Matthew Brost
2026-11-14 1:47 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 30/33] drm/xe: Add packed copy-step IOVA mapping for defrag Matthew Brost
2026-11-14 1:52 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies Matthew Brost
2026-11-14 1:57 ` Niranjana Vishwanathapura
2026-11-16 0:41 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job Matthew Brost
2026-11-14 2:02 ` Niranjana Vishwanathapura
2026-07-11 2:56 ` [PATCH v3 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock Matthew Brost
2026-07-11 10:46 ` Christian König
2026-07-13 20:24 ` Matthew Brost
2026-07-14 9:17 ` Christian König
2026-07-14 15:15 ` Matthew Brost
2026-11-16 1:01 ` [PATCH v3 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs Niranjana Vishwanathapura
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av9seBmEpkJn6ubW@gsse-cloud1.jf.intel.com \
--to=matthew.brost@intel.com \
--cc=airlied@gmail.com \
--cc=carlos.santa@intel.com \
--cc=christian.koenig@amd.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=intel-xe@lists.freedesktop.org \
--cc=linux-kernel@vger.kernel.org \
--cc=maarten.lankhorst@linux.intel.com \
--cc=matthew.auld@intel.com \
--cc=mripard@kernel.org \
--cc=niranjana.vishwanathapura@intel.com \
--cc=ray.huang@amd.com \
--cc=ryanneph@google.com \
--cc=simona@ffwll.ch \
--cc=thomas.hellstrom@linux.intel.com \
--cc=tzimmermann@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®