[RFC PATCH v2 0/1] misc: fgds: introduce GPU-File direct I/O via dma-buf

[Li Wang <liwang@kylinos.cn>]

This patch introduces a driver named fgds that allows moving data between 
a GPU dma-buf and a regular file on NVME without staging it through the 
page cache. This is particularly useful for LLM workloads, such as model 
loading, KV cache offloading, and checkpointing. The fgds device 
corresponds one-to-one with the PCIe GPU on the machine. The usage is 
straightforward: an application first calls ioctl on the corresponding 
fgds device with REGISTER_BUF, passing the target GPU memory buffer 
address (represented as a dma-buf fd). Subsequently, the application can 
perform direct data transfer between a regular file on NVMe and GPU memory 
by invoking ioctl on the fgds device with READ/WRITE, passing the file 
descriptor and the handle returned by REGISTER_BUF.

This solution allows both applications and the fgds device to work 
seamlessly with GPUs from different vendors without any modifications. 
Furthermore, applications no longer need to call vendor-specific 
proprietary APIs (such as NVIDIA's cuFile API) or install vendor-specific 
kernel modules (such as NVIDIA's nvidia-fs.ko) for different GPU vendors. 
We have tested fgds on NVDIA and AMD GPU cards, and theoretically, it also 
works on GPUs from other vendors that support exporting GPU memory via 
dma-buf. Some performance test results can be found in [1].

This implementation offers several advantages:
- Simplicity and Efficiency: It provides a lightweight and intuitive 
  interface. The single ioctl call mimics the parameter structure of 
  pread/pwrite, and its return value semantics are fully compatible with 
  pread/pwrite
- Isolation and Security: The interface is independent and has a minimal 
  impact scope. This isolation facilitates fine-grained permission control 
  on the device node, enhancing security management

Changes since v1:
- Refactor implementation per Christian's feedback, on top of Pavel's 
  DMA-buf support for io_uring [2]
- Add validation checks in the XFS I/O submission path to ensure that 
  bios with the REQ_DMABUF flag are only processed via direct I/O, and 
  prevents the kernel from attempting to fall back to buffered I/O, which 
  is incompatible with DMA-buf backed GPU memory regions. While currently 
  supports only XFS, the underlying mechanism is filesystem-agnostic and 
  can be adapted for other filesystems with similar minor changes
- Address the comments from Greg and the Sashiko AI

[1] https://github.com/Storage-and-OS-for-AI/fgds
[2] https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/
    tag/?h=for-7.4/io_uring-dmabuf-20261023

Mengmeng Zhao (1):
  misc: fgds: introduce GPU-File direct I/O via dma-buf

 .../userspace-api/ioctl/ioctl-number.rst      |   2 +
 drivers/dma-buf/dma-buf-io.c                  |   3 +
 drivers/misc/Kconfig                          |  16 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/fgds.c                           | 506 ++++++++++++++++++
 fs/iomap/buffered-io.c                        |  11 +
 fs/iomap/direct-io.c                          |  23 +-
 fs/xfs/xfs_file.c                             |   4 +
 include/linux/iov_iter.h                      |   4 +
 include/uapi/linux/fgds.h                     | 124 +++++
 lib/iov_iter.c                                |   1 +
 mm/filemap.c                                  |  22 +-
 12 files changed, 713 insertions(+), 4 deletions(-)
 create mode 100644 drivers/misc/fgds.c
 create mode 100644 include/uapi/linux/fgds.h

-- 
2.34.1