Re: [PATCH v2 2/2] vfio/type1: Set IOMMU_MMIO in dma->prot for MMIO-backed addresses

[Tom Lendacky <thomas.lendacky@amd.com>]

On 11/7/25 10:36, Jason Gunthorpe wrote:
> On Fri, Nov 07, 2025 at 04:19:35PM +0000, Wei Wang wrote:
>> On Friday, November 7, 2025 11:57 PM, Jason Gunthorpe wrote:
>> On Fri, Nov 07, 2025 at 03:49:17PM +0000, Wei Wang wrote:
>>>    > (are you aware of any real examples in use?)
>>>    > VM_IO should indicate MMIO, yes, but we don't actually check that in
>>>    > this type 1 path..
>>
>>>    Is it because VFIO type1 didn’t need to check for MMIO before?
>>>    (not sure how this impacts this patch adding the VM_IO check for MMIO
>>>    :) )
>>
>>> Okay, but it still doesn't mean it has to be decrypted..
>>
>> I think "decrypted or not" is the job of the 1st patch. For now,
>> MMIO cannot be encrypted, particularly not via sme_set(). If MMIO
>> encryption is ever introduced in the future, a new flag (probably
>> different from sme_me_mask) would need to be added.
> 
> The kernel is using "decrypted" as some weirdo code-word to mean the
> memory is shared with the hypervisor. Only on AMD does it even have
> anything to do with actual memory encryption.
> 
> However when I look at swiotlb and dma coherent mmap I see it calls
> set_memory_decrypted(), uses pgprot_decrypted(), but still uses
> __sme_set() when forming the iommu page table??
> 
> So why is that OK, but MMIO needs to avoid the sme_set() in the iommu
> page table?
> 
> IOW I would like to hear from AMD some clear rules when sme_set needs
> to be called and when it isn't.

When you are on bare-metal, or in the hypervisor, System Memory Encryption
(SME) deals with the encryption bit set in the page table entries
(including the nested page table entries for guests). If the encryption
bit is not set (decrypted), data does not get encrypted when written to
memory and does not get decrypted when read from memory. If the encryption
bit is set (encrypted), data gets encrypted when written to memory and
decrypted when read from memory. MMIO, since it does not go through the
memory controller, does not support encryption capabilities and so should
not have the encryption bit set as it isn't recognized as system memory.

On the hypervisor, when using the IOMMU, SWIOTLB is not used and I/O to
and from system memory (DMA) will be encrypted and/or decrypted if the
encryption bit is set in the I/O page table leaf entry. If the IOMMU is
not enabled, then SWIOTLB is only used if the device does not support DMA
addressing at or above the encryption bit location.

In the guest (prior to Trusted I/O / TDISP), decrypted (or shared) memory
is used because a device cannot DMA to or from guest memory using the
guest encryption key. So all DMA must go to "decrypted" memory or be
bounce-buffered through "decrypted" memory (SWIOTLB) - basically memory
that does not get encrypted/decrypted using the guest encryption key.

It is not until we get to Trusted I/O / TDISP where devices will be able
to DMA directly to guest encrypted memory and guests will require secure
MMIO addresses which will need the encryption bit set (Alexey can correct
me on the TIO statements if they aren't correct, as he is closer to it all).

I hope I've explained it in a way that makes sense.

Thanks,
Tom

> 
> Then we can decide if VM_IO is sufficient and so on.
> 
> Jason