Re: [RFC PATCH 00/12] Private MMIO support for private assigned dev

[Alexey Kardashevskiy <aik@amd.com>]

Ping?
Also, since there is pushback on 01/12 "dma-buf: Introduce dma_buf_get_pfn_unlocked() kAPI", what is the plan now? Thanks,


On 29/4/25 17:50, Alexey Kardashevskiy wrote:
> 
> 
> On 29/4/25 16:48, Alexey Kardashevskiy wrote:
>> On 8/1/25 01:27, Xu Yilun wrote:
>>> This series is based on an earlier kvm-coco-queue version (v6.12-rc2)
>>
>> Has this been pushed somewhere public? The patchset does not apply on top of v6.12-rc2, for example (I fixed locally).
>> Also, is there somewhere a QEMU tree using this? I am trying to use this new DMA_BUF feature and this require quite some not so obvious plumbing. Thanks,
> 
> 
> More to the point, to make it work, QEMU needs to register VFIO MMIO BAR with KVM_SET_USER_MEMORY_REGION2 which passes slot->guest_memfd to KVM which essentially comes from VFIORegion->mmaps[0].mem->ram_block->guest_memfd. But since you disabled mmap for private MMIO, there is no MR which QEMU would even try registering as KVM memslot and there are many ways to fix it. I took a shortcut and reenabled mmap() but wonder what exactly you did. Makes sense? Thanks,

> 
> 
>>
>>> which includes all basic TDX patches.
>>>
>>> The series is to start the early stage discussion of the private MMIO
>>> handling for Coco-VM, which is part of the Private Device
>>> Assignment (aka TEE-IO, TIO) enabling. There are already some
>>> disscusion about the context of TIO:
>>>
>>> https://lore.kernel.org/linux-coco/173343739517.1074769.13134786548545925484.stgit@dwillia2-xfh.jf.intel.com/
>>> https://lore.kernel.org/all/20240823132137.336874-1-aik@amd.com/
>>>
>>> Private MMIOs are resources owned by Private assigned devices. Like
>>> private memory, they are also not intended to be accessed by host, only
>>> accessible by Coco-VM via some secondary MMUs (e.g. Secure EPT). This
>>> series is for KVM to map these MMIO resources without firstly mapping
>>> into the host. For this purpose, This series uses the FD based MMIO
>>> resources for secure mapping, and the dma-buf is chosen as the FD based
>>> backend, just like guest_memfd for private memory. Patch 6 in this
>>> series has more detailed description.
>>>
>>>
>>> Patch 1 changes dma-buf core, expose a new kAPI for importers to get
>>> dma-buf's PFN without DMA mapping. KVM could use this kAPI to build
>>> GPA -> HPA mapping in KVM MMU.
>>>
>>> Patch 2-4 are from Jason & Vivek, allow vfio-pci to export MMIO
>>> resources as dma-buf. The original series are for native P2P DMA and
>>> focus on p2p DMA mapping opens. I removed these p2p DMA mapping code
>>> just to focus the early stage discussion of private MMIO. The original
>>> series:
>>>
>>> https://lore.kernel.org/all/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/
>>> https://lore.kernel.org/kvm/20240624065552.1572580-1-vivek.kasireddy@intel.com/
>>>
>>> Patch 5 is the implementation of get_pfn() callback for vfio dma-buf
>>> exporter.
>>>
>>> Patch 6-7 is about KVM supports the private MMIO memory slot backed by
>>> vfio dma-buf.
>>>
>>> Patch 8-10 is about how KVM verifies the user provided dma-buf fd
>>> eligible for private MMIO slot.
>>>
>>> Patch 11-12 is the example of how KVM TDX setup the Secure EPT for
>>> private MMIO.
>>>
>>>
>>> TODOs:
>>>
>>> - Follow up the evolving of original VFIO dma-buf series.
>>> - Follow up the evolving of basic TDX patches.
>>>
>>>
>>> Vivek Kasireddy (3):
>>>    vfio: Export vfio device get and put registration helpers
>>>    vfio/pci: Share the core device pointer while invoking feature
>>>      functions
>>>    vfio/pci: Allow MMIO regions to be exported through dma-buf
>>>
>>> Xu Yilun (9):
>>>    dma-buf: Introduce dma_buf_get_pfn_unlocked() kAPI
>>>    vfio/pci: Support get_pfn() callback for dma-buf
>>>    KVM: Support vfio_dmabuf backed MMIO region
>>>    KVM: x86/mmu: Handle page fault for vfio_dmabuf backed MMIO
>>>    vfio/pci: Create host unaccessible dma-buf for private device
>>>    vfio/pci: Export vfio dma-buf specific info for importers
>>>    KVM: vfio_dmabuf: Fetch VFIO specific dma-buf data for sanity check
>>>    KVM: x86/mmu: Export kvm_is_mmio_pfn()
>>>    KVM: TDX: Implement TDX specific private MMIO map/unmap for SEPT
>>>
>>>   Documentation/virt/kvm/api.rst     |   7 +
>>>   arch/x86/include/asm/tdx.h         |   3 +
>>>   arch/x86/kvm/mmu.h                 |   1 +
>>>   arch/x86/kvm/mmu/mmu.c             |  25 ++-
>>>   arch/x86/kvm/mmu/spte.c            |   3 +-
>>>   arch/x86/kvm/vmx/tdx.c             |  57 +++++-
>>>   arch/x86/virt/vmx/tdx/tdx.c        |  52 ++++++
>>>   arch/x86/virt/vmx/tdx/tdx.h        |   3 +
>>>   drivers/dma-buf/dma-buf.c          |  90 ++++++++--
>>>   drivers/vfio/device_cdev.c         |   9 +-
>>>   drivers/vfio/pci/Makefile          |   1 +
>>>   drivers/vfio/pci/dma_buf.c         | 273 +++++++++++++++++++++++++++++
>>>   drivers/vfio/pci/vfio_pci_config.c |  22 ++-
>>>   drivers/vfio/pci/vfio_pci_core.c   |  64 +++++--
>>>   drivers/vfio/pci/vfio_pci_priv.h   |  27 +++
>>>   drivers/vfio/pci/vfio_pci_rdwr.c   |   3 +
>>>   drivers/vfio/vfio_main.c           |   2 +
>>>   include/linux/dma-buf.h            |  13 ++
>>>   include/linux/kvm_host.h           |  25 ++-
>>>   include/linux/vfio.h               |  22 +++
>>>   include/linux/vfio_pci_core.h      |   1 +
>>>   include/uapi/linux/kvm.h           |   1 +
>>>   include/uapi/linux/vfio.h          |  34 +++-
>>>   virt/kvm/Kconfig                   |   6 +
>>>   virt/kvm/Makefile.kvm              |   1 +
>>>   virt/kvm/kvm_main.c                |  32 +++-
>>>   virt/kvm/kvm_mm.h                  |  19 ++
>>>   virt/kvm/vfio_dmabuf.c             | 151 ++++++++++++++++
>>>   28 files changed, 896 insertions(+), 51 deletions(-)
>>>   create mode 100644 drivers/vfio/pci/dma_buf.c
>>>   create mode 100644 virt/kvm/vfio_dmabuf.c
>>>
>>
> 

-- 
Alexey