Re: [PATCH v11 0/8] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes

[Auger Eric <eric.auger@redhat.com>]

Hi Dennis
On 20/07/2016 11:56, Dennis Chen wrote:
> Hi Eric,
> 
> On Tue, Jul 19, 2016 at 12:55:03PM +0000, Eric Auger wrote:
>> This series introduces the msi-iommu api used to:
>>
>> - allocate/free resources for MSI IOMMU mapping
>> - set the MSI iova window aperture
>> - map/unmap physical addresses onto MSI IOVAs.
>> - determine whether an msi needs to be iommu mapped
>> - overwrite an msi_msg PA address with its pre-allocated/mapped IOVA
>>
>> Also a new iommu domain attribute, DOMAIN_ATTR_MSI_GEOMETRY is introduced
>> to report the MSI iova window geometry (aperture and iommu-msi API support).
>>
>> Currently:
>> - iommu driver is supposed to allocate/free MSI mapping resources
>> - VFIO subsystem is supposed to set the MSI IOVA aperture.
>> - The MSI layer is supposed to allocate/free iova mappings and overwrite
>>   msi_msg with IOVA at composition time
>>
>> More details & context can be found at:
>> http://www.linaro.org/blog/core-dump/kvm-pciemsi-passthrough-armarm64/
>>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://github.com/eauger/linux/tree/v4.7-rc7-passthrough-v11
>>
> Why can't I find this new series on your git tree:
> https://git.linaro.org/people/eric.auger/linux.git
you are not looking at the right git repo: see github one above.
> ?
> Also, do I need to download all the 3-part patches to test the PCIe NIC passthru
> as I did on your v9 series?
Yes you need to take the 3 parts. You should have everything that is
needed on the above branch. In case you do not work on Cavium, you
should not cherry-pick
"vfio: pci: HACK! workaround thunderx pci_try_reset_bus crash"

Thanks

Eric
> 
> Thanks,
> Dennis 
>>
>> see part III for wrap-up details.
>>
>> History:
>> v10 -> v11:
>> - no change in the series, just incremented for consistency
>> - added a temporary patch in the branch:
>>   "iommu/iova: FIXUP! validate iova_domain input to put_iova_domain"
>>   originally sent by Nate and adapted for this use case. This is currently
>>   under discussion on the ML. The crash typically occurs in case unsafe
>>   interrupts are discovered while allow_unsafe_interrupts is not set.
>>
>> v9 -> v10:
>> - split error management in iommu_msi_set_aperture
>>
>> v8 -> v9:
>> - rename iommu_domain_msi_geometry programmable flag into iommu_msi_supported
>> - introduce msi_apperture_valid helper and use this instead of is_aperture_set
>>
>> v7 -> v8:
>> - The API is retargetted for MSI: renamed msi-iommu
>>   all "dma-reserved" namings removed
>> - now implemented upon dma-iommu (get, put, init), ie. reuse iova_cookie,
>>   and iova API
>> - msi mapping resources now are guaranteed to exist during the whole iommu
>>   domain's lifetime. No need to lock to garantee the cookie integrity
>> - removed alloc/free_reserved_reserved_iova_domain. We now have a single
>>   function that sets the aperture, looking like iommu_dma_init_domain.
>> - we now use a list instead of an RB-tree
>> - prot is not propagated anymore at domain creation due to the retargetting
>>   for MSI
>> - iommu_domain pointer removed from doorbell_mapping struct
>> - replaced DOMAIN_ATTR_MSI_MAPPING by DOMAIN_ATTR_MSI_GEOMETRY
>>
>> v6 -> v7:
>> - fixed known lock bugs and multiple page sized slots matching
>>   (I only have a single MSI frame made of a single page)
>> - reserved_iova_cookie now pointing to a struct that encapsulates the
>>   iova domain handle + protection attribute passed from VFIO (Alex' req)
>> - 2 new functions exposed: iommu_msi_mapping_translate_msg,
>>   iommu_msi_mapping_desc_to_domain: not sure this is the right location/proto
>>   though
>> - iommu_put_reserved_iova now takes a phys_addr_t
>> - everything now is cleanup on iommu_domain destruction
>>
>> RFC v5 -> patch v6:
>> - split to ease the review process
>> - in dma-reserved-api use a spin lock instead of a mutex (reported by
>>   Jean-Philippe)
>> - revisit iommu_get_reserved_iova API to pass a size parameter upon
>>   Marc's request
>> - Consistently use the page order passed when creating the iova domain.
>> - init reserved_binding_list (reported by Julien)
>>
>> RFC v4 -> RFC v5:
>> - take into account Thomas' comments on MSI related patches
>>   - split "msi: IOMMU map the doorbell address when needed"
>>   - increase readability and add comments
>>   - fix style issues
>>  - split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute"
>>  - platform ITS now advertises IOMMU_CAP_INTR_REMAP
>>  - fix compilation issue with CONFIG_IOMMU API unset
>>  - arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING
>>
>> RFC v3 -> v4:
>> - Move doorbell mapping/unmapping in msi.c
>> - fix ref count issue on set_affinity: in case of a change in the address
>>   the previous address is decremented
>> - doorbell map/unmap now is done on msi composition. Should allow the use
>>   case for platform MSI controllers
>> - create dma-reserved-iommu.h/c exposing/implementing a new API dedicated
>>   to reserved IOVA management (looking like dma-iommu glue)
>> - series reordering to ease the review:
>>   - first part is related to IOMMU
>>   - second related to MSI sub-system
>>   - third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal)
>> - expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO
>>   [this partially addresses Marc's comments on iommu_get/put_single_reserved
>>    size/alignment problematic - which I did not ignore - but I don't know
>>    how much I can do at the moment]
>>
>> RFC v2 -> RFC v3:
>> - should fix wrong handling of some CONFIG combinations:
>>   CONFIG_IOVA, CONFIG_IOMMU_API, CONFIG_PCI_MSI_IRQ_DOMAIN
>> - fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested)
>>
>> PATCH v1 -> RFC v2:
>> - reverted to RFC since it looks more reasonable ;-) the code is split
>>   between VFIO, IOMMU, MSI controller and I am not sure I did the right
>>   choices. Also API need to be further discussed.
>> - iova API usage in arm-smmu.c.
>> - MSI controller natively programs the MSI addr with either the PA or IOVA.
>>   This is not done anymore in vfio-pci driver as suggested by Alex.
>> - check irq remapping capability of the group
>>
>> RFC v1 [2] -> PATCH v1:
>> - use the existing dma map/unmap ioctl interface with a flag to register a
>>   reserved IOVA range. Use the legacy Rb to store this special vfio_dma.
>> - a single reserved IOVA contiguous region now is allowed
>> - use of an RB tree indexed by PA to store allocated reserved slots
>> - use of a vfio_domain iova_domain to manage iova allocation within the
>>   window provided by the userspace
>> - vfio alloc_map/unmap_free take a vfio_group handle
>> - vfio_group handle is cached in vfio_pci_device
>> - add ref counting to bindings
>> - user modality enabled at the end of the series
>>
>>
>> Eric Auger (8):
>>   iommu: Add iommu_domain_msi_geometry and DOMAIN_ATTR_MSI_GEOMETRY
>>   iommu/arm-smmu: initialize the msi geometry and advertise iommu-msi
>>     support
>>   iommu: introduce an msi cookie
>>   iommu/msi-iommu: initialization
>>   iommu/msi-iommu: iommu_msi_[get,put]_doorbell_iova
>>   iommu/msi-iommu: iommu_msi_domain
>>   iommu/msi-iommu: iommu_msi_msg_pa_to_va
>>   iommu/arm-smmu: get/put the msi cookie
>>
>>  drivers/iommu/Kconfig       |   7 +
>>  drivers/iommu/Makefile      |   1 +
>>  drivers/iommu/arm-smmu-v3.c |  18 ++-
>>  drivers/iommu/arm-smmu.c    |  18 ++-
>>  drivers/iommu/iommu.c       |   5 +
>>  drivers/iommu/msi-iommu.c   | 322 ++++++++++++++++++++++++++++++++++++++++++++
>>  include/linux/iommu.h       |  15 +++
>>  include/linux/msi-iommu.h   | 144 ++++++++++++++++++++
>>  8 files changed, 522 insertions(+), 8 deletions(-)
>>  create mode 100644 drivers/iommu/msi-iommu.c
>>  create mode 100644 include/linux/msi-iommu.h
>>
>> -- 
>> 1.9.1
>>
>> _______________________________________________
>> kvmarm mailing list
>> kvmarm@lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
>>
>