Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak <akrowiak@linux.ibm.com>]

On 3/30/26 12:27 PM, Alex Williamson wrote:
> On Wed, 25 Mar 2026 17:00:47 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> This patch series implements live guest migration of a guest to which AP
>> devices have been passed through. To better comprehend this design, one has
>> to understand that VFIO AP mediated device is not used to provide userspace
>> with direct access to a device as is the case with other devices that use
>> the VFIO framework to pass them through to a guest. The sole purpose of the
>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>> configuration is comprised of the AP adapter IDs (APID), AP queue
>> indexes (APQI) and domain numbers of the control domains to which a guest
>> will be granted access. Once the VFIO AP mediated device is attached to a
>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>> all access to the AP devices is handled by the s390 Interpretive Execution
>> facility; in other words, the vfio_ap device driver plays no role in
>> providing direct access to the AP devices in the guest's AP configuration.
>>
>> The only role that the vfio_ap device driver plays in the migration
>> process is to verify that the AP configuration for the source guest is
>> compatible with the AP configuration of the destination guest.
>> Incompatibility will result in a live guest migration failure.
>> In order to be compatible, the following requirements must be met:
>>
>> 1. The destination guest will be started with the same QEMU command line
>>     as the source guest, so the mediated device supplying the AP
>>     configuration on both guests must have the same name (UUID).
> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> supporting migration.  You're really concerned more with the
> configuration/composition of the mdev device, so requiring the same
> UUID seems a bit arbitrary.

As stated above, the destination guest will be started with the same
QEMU command line as the source guest. Within that command line
will be a '-device' parameter like the following:

-device 
'{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}

Note that sysfsdev is the path to the mdev named 
62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or
the migration will fail with the following error:

error: device not found: mediated device 
'62177883-f1bb-47f0-914d-32a22e3a8804' not found

>
>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>>     guests must be compatible. As such, each AP configuration must meet
>>     the following requirements:
>>
>>     * Both guests must have the same number of APQNs
>>
>>     * Each APQN assigned to the source guest must also be assigned to the
>>       destination guest
>>
>>     * Each APQN assigned to both guests must reference an AP queue with the
>>       same hardware capabilities
> Why isn't this sufficient vs also requiring the same UUID?
>        
>> Note: There is a forthcoming consumer of this series which will be a QEMU
>>        patch series is entitled:
>>        'hw/vfio/ap: Implement live guest migration of guests using AP
>>        devices'
>>
>> This design also adds a use case for enabling and disabling
>> migration of guests to which AP devices have been passed through. To
>> facilitate this, a new read/write sysfs 'migratable' attribute is added to
>> the mediated device. This attribute specifies whether the vfio device is
>> migratable (1) or not (0). When the value of this attribute is changed, the
>> vfio_ap device driver will signal an eventfd to userspace. It is up to
>> userspace to respond to the change by enabling or disabling migration of
>> the guest to which the mediated device is attached. The operation will be
>> rejected with a 'Device or resource busy' message if a migration is in
>> progress.
> This seems inherently racy.  What happens if the device becomes
> unmigratable while it's being migrated?
>
> If userspace is deciding that the device is now unmigratable, why does
> it need to signal this through the kernel driver rather than with the
> userspace orchestration agent?  The entire path seems unnecessary.
>
>> Userspace must also have a means for retrieving the value of the sysfs
>> 'migratable' attribute when the guest is started to initialize whether it
>> can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The
>> struct vfio_device_info object passed to the ioctl will be extended with a
>> capability specifying the vfio device attributes. One of the attributes
>> will contain the value of the mediated device's 'migratable' attribute.
> This is just broken, it's redundant to our current device feature
> mechanism for exposing migration support.  If you want the capability
> to create unmigratable devices statically, can't that be encompassed
> within the definition of the mdev type?  Dynamic migration support just
> seems like it's involving the kernel in orchestration it shouldn't be a
> part of.  Thanks,
>
> Alex
>   
>> Anthony Krowiak (23):
>>    s390/vfio-ap: Store queue hardware info when probed
>>    s390/vfio-ap: Provide access to queue objects and related info
>>    s390/vfio-ap: Add header file for xfer of vfio device caps to
>>      userspace
>>    MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
>>      maintainers
>>    s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
>>      migration of guest
>>    s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
>>    s390/vfio-ap: Signal event to enable/disable live guest migration
>>    s390/vfio-ap: Return value of sysfs migratable attribute from
>>      VFIO_DEVICE_GET_INFO ioctl
>>    s390/vfio-ap: Data structures for facilitating vfio device migration
>>    s390/vfio-ap: Initialize/release vfio device migration data
>>    s390-vfio-ap: Callback to set vfio device mig state during guest
>>      migration
>>    s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
>>    s390/vfio-ap: File ops called to save the vfio device migration state
>>    s390/vfio-ap: Transition device migration state from STOP to RESUMING
>>    s390/vfio-ap: File ops called to resume the vfio device migration
>>    s390/vfio-ap: Transition device migration state from RESUMING to STOP
>>    s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
>>    s390/vfio-ap: Transition device migration state from STOP to RUNNING
>>      and vice versa
>>    s390-vfio-ap: Callback to get the current vfio device migration state
>>    s390/vfio-ap: Callback to get the size of data to be migrated during
>>      guest migration
>>    s390/vfio-ap: Provide API to query whether migration is in progress
>>    s390/vfio-ap: Disallow blocking migration in progress
>>    s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
>>
>>   Documentation/arch/s390/vfio-ap.rst     |  339 +++++--
>>   MAINTAINERS                             |    1 +
>>   drivers/s390/crypto/Makefile            |    2 +-
>>   drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
>>   drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
>>   drivers/s390/crypto/vfio_ap_ops.c       |  263 +++++-
>>   drivers/s390/crypto/vfio_ap_private.h   |   20 +
>>   include/uapi/linux/vfio.h               |    2 +
>>   include/uapi/linux/vfio_ap.h            |   34 +
>>   9 files changed, 1685 insertions(+), 111 deletions(-)
>>   create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
>>   create mode 100644 include/uapi/linux/vfio_ap.h
>>