Re: [RFC PATCH v2 0/1] misc: fgds: introduce GPU-File direct I/O via dma-buf

[Pavel Begunkov <asml.silence@gmail.com>]

Hey Li,

On 10/27/26 09:08, Li Wang wrote:
> This patch introduces a driver named fgds that allows moving data between
> a GPU dma-buf and a regular file on NVME without staging it through the
> page cache. This is particularly useful for LLM workloads, such as model
> loading, KV cache offloading, and checkpointing. The fgds device
> corresponds one-to-one with the PCIe GPU on the machine. The usage is
> straightforward: an application first calls ioctl on the corresponding
> fgds device with REGISTER_BUF, passing the target GPU memory buffer
> address (represented as a dma-buf fd). Subsequently, the application can
> perform direct data transfer between a regular file on NVMe and GPU memory
> by invoking ioctl on the fgds device with READ/WRITE, passing the file
> descriptor and the handle returned by REGISTER_BUF.
> 
> This solution allows both applications and the fgds device to work
> seamlessly with GPUs from different vendors without any modifications.
> Furthermore, applications no longer need to call vendor-specific
> proprietary APIs (such as NVIDIA's cuFile API) or install vendor-specific
> kernel modules (such as NVIDIA's nvidia-fs.ko) for different GPU vendors.
> We have tested fgds on NVDIA and AMD GPU cards, and theoretically, it also
> works on GPUs from other vendors that support exporting GPU memory via
> dma-buf. Some performance test results can be found in [1].
> 
> This implementation offers several advantages:
> - Simplicity and Efficiency: It provides a lightweight and intuitive
>    interface. The single ioctl call mimics the parameter structure of
>    pread/pwrite, and its return value semantics are fully compatible with
>    pread/pwrite
> - Isolation and Security: The interface is independent and has a minimal
>    impact scope. This isolation facilitates fine-grained permission control
>    on the device node, enhancing security management
> 
> Changes since v1:
> - Refactor implementation per Christian's feedback, on top of Pavel's
>    DMA-buf support for io_uring [2]

People already replied, but I'm going to elaborate a bit. That series
[2] already exposes a uapi via io_uring, it's device agnostic and all
that. The framework was made to allow multiple users, perhaps folks
have some kernel-internal users in mind, but a completely new
read/write uAPI looks unnecessary, especially an ioctl() based one.
It sounds like [1] is already using io_uring, so it doesn't sound
like a problem that the entire uapi is io_uring based(?).

> - Add validation checks in the XFS I/O submission path to ensure that
>    bios with the REQ_DMABUF flag are only processed via direct I/O, and
>    prevents the kernel from attempting to fall back to buffered I/O, which
>    is incompatible with DMA-buf backed GPU memory regions. While currently
>    supports only XFS, the underlying mechanism is filesystem-agnostic and
>    can be adapted for other filesystems with similar minor changes

xfs support it a it orthogonal to the uapi question, but it should be
implemented in xfs and not side stepped. There is an RFC on the mailing
list from Anuj and Nitesh that is trying to do that.

> - Address the comments from Greg and the Sashiko AI
> 
> [1] https://github.com/Storage-and-OS-for-AI/fgds
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/
>      tag/?h=for-7.4/io_uring-dmabuf-20261023
> 
> Mengmeng Zhao (1):
>    misc: fgds: introduce GPU-File direct I/O via dma-buf
> 
>   .../userspace-api/ioctl/ioctl-number.rst      |   2 +
>   drivers/dma-buf/dma-buf-io.c                  |   3 +
>   drivers/misc/Kconfig                          |  16 +
>   drivers/misc/Makefile                         |   1 +
>   drivers/misc/fgds.c                           | 506 ++++++++++++++++++
>   fs/iomap/buffered-io.c                        |  11 +
>   fs/iomap/direct-io.c                          |  23 +-
>   fs/xfs/xfs_file.c                             |   4 +
>   include/linux/iov_iter.h                      |   4 +
>   include/uapi/linux/fgds.h                     | 124 +++++
>   lib/iov_iter.c                                |   1 +
>   mm/filemap.c                                  |  22 +-
>   12 files changed, 713 insertions(+), 4 deletions(-)
>   create mode 100644 drivers/misc/fgds.c
>   create mode 100644 include/uapi/linux/fgds.h
> 

-- 
Pavel Begunkov