[PATCH] drm/vmwgfx: release imported dma-buf when the BO is freed

[Adam Lewis <adam.j.lewis@gmail.com>]

vmw_prime_import_sg_table() creates a BO for a foreign dma-buf through
drm_gem_prime_import_dev(), which attaches to the dma-buf, maps the
attachment and takes a reference on it.  vmw_bo_free() never undoes
this, so each foreign import leaks the attachment, its sg table and the
dma-buf reference, and the exporter's pages stay pinned after the GEM
handle, the DRM file and the importing process are gone.

Any client that can open a render node and obtain a dma-buf from
another exporter (udmabuf, V4L2, another DRM device) can pin memory
without bound.  KWin does so on its own in VMware guests: it wraps
wl_shm client buffers in udmabufs and imports them, leaking gigabytes
within days.

Call drm_prime_gem_destroy() for imported objects before releasing the
GEM object, as amdgpu, radeon and nouveau do.  vmw_bo_free() is TTM's
destroy callback, so TTM is done with the reservation object shared
with the dma-buf by then.

Tested on 7.1.13 and 7.2.7 with a udmabuf import reproducer and on a
Plasma desktop, where KWin's imported client buffers are now released.
The leak was found and the fix written with the help of an LLM coding
assistant.

Fixes: b32233accefff ("drm/vmwgfx: Fix prime import/export")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-fable-5.1
Signed-off-by: Adam Lewis <adam.j.lewis@gmail.com>
---
Notes (dropped on apply):

Security note: per Documentation/process/threat-model.rst an unprivileged
user must not be able to make the system unresponsive through unbounded
resource exhaustion.  On a default systemd desktop every logged-in user
has a render node (0666) and /dev/udmabuf (uaccess ACL), which is all
this needs, and the pinned memory survives the triggering process.  The
bug was found with AI assistance, so per security-bugs.rst it is treated
as public and the reproducer is not attached (available privately).
A CVE request will follow once the fix is in a released kernel.

Environment: VMware Fusion on an Apple M3 Max host, Fedora 44 aarch64
guest, vmwgfx 2.21.0, no swap.  Found on 7.1.12/7.1.13 (Mesa 26.1.8,
KWin 6.7.4), confirmed on 7.2.7 (Mesa 26.2.3, KWin 6.7.5).  The import
and destructor code is unchanged from v7.1 through v7.3-rc5 and current
drm-misc-fixes; b32233accefff went to stable # v6.6+.  Not the same
issue as f739416dc555 ("drm/vmwgfx: drop dma_buf reference on
foreign-fd prime import"), which covers ttm_prime_fd_to_handle()'s
error path and was already in the kernels tested.

How it shows up: KWin wraps every wl_shm client buffer in a udmabuf and
imports it into vmwgfx, for scanout and, when it renders with llvmpipe
because svga reports "No 3D enabled", as a texture.  Every import
leaked, so every client buffer stayed pinned: ~6 GB after a few days on
7.1.  debugfs bufinfo then listed 961 live udmabufs, 954 of them still
attached to vmwgfx, while kwin_wayland held 35 dma-buf fds.  On an
unpatched 7.2.7 boot, nr_foll_pin_released did not move at all in four
hours of desktop use, while MemAvailable fell to 2.8 of 24.5 GB.

Testing:
  - reproducer (udmabuf -> PRIME_FD_TO_HANDLE -> GEM_CLOSE -> close the
    dma-buf -> close the DRM fd): unpatched, the dma-buf outlives the
    process with refcount 1 and vmwgfx still attached; patched (7.1.13
    and 7.2.7), the pins are released as soon as the dma-buf fd is
    closed;
  - imported framebuffer: ADDFB2 + SETPLANE on/off + CLOSEFB through the
    screen-target path on 7.1.13, and ADDFB2 + CLOSEFB on 7.2.7, both
    release cleanly;
  - Plasma desktop on 7.2.7 with the patch, 30 terminal windows opened
    and closed: KWin created 298 udmabufs and 296 were freed (the other
    two were still in use), drm_prime_gem_destroy() ran for 85 imported
    objects, and the pin counters, slab usage and dma-buf census went
    back to their baseline;
  - builds without warnings with W=1 on drm-misc-fixes (e78a9fb).

Two related leaks on the same desktop, to be reported separately:
  - Mesa's kms-dri-sw winsys releases GEM handles with MODE_DESTROY_DUMB,
    which is EACCES on render nodes, so a compositor rendering with
    llvmpipe on a render node leaks the handle of every imported buffer
    (KWin up to 6.7.4; 6.7.5 falls back to the KMS node);
  - vmw_buffer_prime_to_surface_base() creates a GEM handle the caller
    never sees and WARNs for every foreign dma-buf.
With this fix those leaks last until the owning process exits; without
it they pin memory permanently too.

 drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
index 9c7a73c..f226b32 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
@@ -31,6 +31,7 @@
 #include "vmwgfx_resource_priv.h"
 
 #include <drm/ttm/ttm_placement.h>
+#include <drm/drm_prime.h>
 
 /**
  * vmw_bo_free - vmw_bo destructor
@@ -69,6 +70,8 @@ static void vmw_bo_free(struct ttm_buffer_object *bo)
 		vmw_surface_unreference(&vbo->dumb_surface);
 	}
 	WARN_ON(!RB_EMPTY_ROOT(&vbo->res_tree));
+	if (drm_gem_is_imported(&vbo->tbo.base))
+		drm_prime_gem_destroy(&vbo->tbo.base, vbo->tbo.sg);
 	drm_gem_object_release(&vbo->tbo.base);
 	WARN_ON(vbo->dirty);
 	kfree(vbo);

base-commit: e78a9fb7a40c55ec2a70ae39903dde4f898af2ef
-- 
2.55.0