Re: [PATCH v9 2/2] hisi_acc_vfio_pci: adapt to new migration configuration

[liulongfang <liulongfang@huawei.com>]

On 2025/9/18 15:06, Shameer Kolothum wrote:
> On Thu, 18 Sept 2025 at 02:29, liulongfang <liulongfang@huawei.com> wrote:
>>
>> On 2025/9/11 19:30, liulongfang wrote:
>>> On 2025/9/11 16:05, Shameer Kolothum wrote:
>>>> On Tue, 2 Sept 2025 at 03:26, Longfang Liu <liulongfang@huawei.com> wrote:
>>>>>
>>>>> On new platforms greater than QM_HW_V3, the migration region has been
>>>>> relocated from the VF to the PF. The VF's own configuration space is
>>>>> restored to the complete 64KB, and there is no need to divide the
>>>>> size of the BAR configuration space equally. The driver should be
>>>>> modified accordingly to adapt to the new hardware device.
>>>>>
>>>>> On the older hardware platform QM_HW_V3, the live migration configuration
>>>>> region is placed in the latter 32K portion of the VF's BAR2 configuration
>>>>> space. On the new hardware platform QM_HW_V4, the live migration
>>>>> configuration region also exists in the same 32K area immediately following
>>>>> the VF's BAR2, just like on QM_HW_V3.
>>>>>
>>>>> However, access to this region is now controlled by hardware. Additionally,
>>>>> a copy of the live migration configuration region is present in the PF's
>>>>> BAR2 configuration space. On the new hardware platform QM_HW_V4, when an
>>>>> older version of the driver is loaded, it behaves like QM_HW_V3 and uses
>>>>> the configuration region in the VF, ensuring that the live migration
>>>>> function continues to work normally. When the new version of the driver is
>>>>> loaded, it directly uses the configuration region in the PF. Meanwhile,
>>>>> hardware configuration disables the live migration configuration region
>>>>> in the VF's BAR2: reads return all 0xF values, and writes are silently
>>>>> ignored.
>>>>>
>>>>> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
>>>>> Reviewed-by: Shameer Kolothum <shameerkolothum@gmail.com>
>>>>> ---
>>>>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 205 ++++++++++++------
>>>>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  13 ++
>>>>>  2 files changed, 157 insertions(+), 61 deletions(-)
>>>>>
>>>>> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>>>>> index 397f5e445136..fcf692a7bd4c 100644
>>>>> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>>>>> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>>>>> @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 *addr)
>>>>>         return 0;
>>>>>  }
>>>>>
>>>>> +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>>>>> +                          struct acc_vf_data *vf_data)
>>>>> +{
>>>>> +       struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
>>>>> +       struct device *dev = &qm->pdev->dev;
>>>>> +       u32 eqc_addr, aeqc_addr;
>>>>> +       int ret;
>>>>> +
>>>>> +       if (hisi_acc_vdev->drv_mode == HW_ACC_V3) {
>>>>> +               eqc_addr = QM_EQC_DW0;
>>>>> +               aeqc_addr = QM_AEQC_DW0;
>>>>> +       } else {
>>>>> +               eqc_addr = QM_EQC_PF_DW0;
>>>>> +               aeqc_addr = QM_AEQC_PF_DW0;
>>>>> +       }
>>>>> +
>>>>> +       /* QM_EQC_DW has 7 regs */
>>>>> +       ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
>>>>> +       if (ret) {
>>>>> +               dev_err(dev, "failed to read QM_EQC_DW\n");
>>>>> +               return ret;
>>>>> +       }
>>>>> +
>>>>> +       /* QM_AEQC_DW has 7 regs */
>>>>> +       ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
>>>>> +       if (ret) {
>>>>> +               dev_err(dev, "failed to read QM_AEQC_DW\n");
>>>>> +               return ret;
>>>>> +       }
>>>>> +
>>>>> +       return 0;
>>>>> +}
>>>>> +
>>>>> +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>>>>> +                          struct acc_vf_data *vf_data)
>>>>> +{
>>>>> +       struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
>>>>> +       struct device *dev = &qm->pdev->dev;
>>>>> +       u32 eqc_addr, aeqc_addr;
>>>>> +       int ret;
>>>>> +
>>>>> +       if (hisi_acc_vdev->drv_mode == HW_ACC_V3) {
>>>>> +               eqc_addr = QM_EQC_DW0;
>>>>> +               aeqc_addr = QM_AEQC_DW0;
>>>>> +       } else {
>>>>> +               eqc_addr = QM_EQC_PF_DW0;
>>>>> +               aeqc_addr = QM_AEQC_PF_DW0;
>>>>> +       }
>>>>> +
>>>>> +       /* QM_EQC_DW has 7 regs */
>>>>> +       ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
>>>>> +       if (ret) {
>>>>> +               dev_err(dev, "failed to write QM_EQC_DW\n");
>>>>> +               return ret;
>>>>> +       }
>>>>> +
>>>>> +       /* QM_AEQC_DW has 7 regs */
>>>>> +       ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
>>>>> +       if (ret) {
>>>>> +               dev_err(dev, "failed to write QM_AEQC_DW\n");
>>>>> +               return ret;
>>>>> +       }
>>>>> +
>>>>> +       return 0;
>>>>> +}
>>>>> +
>>>>>  static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
>>>>>  {
>>>>>         struct device *dev = &qm->pdev->dev;
>>>>> @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
>>>>>                 return ret;
>>>>>         }
>>>>>
>>>>> -       /* QM_EQC_DW has 7 regs */
>>>>> -       ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
>>>>> -       if (ret) {
>>>>> -               dev_err(dev, "failed to read QM_EQC_DW\n");
>>>>> -               return ret;
>>>>> -       }
>>>>> -
>>>>> -       /* QM_AEQC_DW has 7 regs */
>>>>> -       ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
>>>>> -       if (ret) {
>>>>> -               dev_err(dev, "failed to read QM_AEQC_DW\n");
>>>>> -               return ret;
>>>>> -       }
>>>>> -
>>>>>         return 0;
>>>>>  }
>>>>>
>>>>> @@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
>>>>>                 return ret;
>>>>>         }
>>>>>
>>>>> -       /* QM_EQC_DW has 7 regs */
>>>>> -       ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
>>>>> -       if (ret) {
>>>>> -               dev_err(dev, "failed to write QM_EQC_DW\n");
>>>>> -               return ret;
>>>>> -       }
>>>>> -
>>>>> -       /* QM_AEQC_DW has 7 regs */
>>>>> -       ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
>>>>> -       if (ret) {
>>>>> -               dev_err(dev, "failed to write QM_AEQC_DW\n");
>>>>> -               return ret;
>>>>> -       }
>>>>> -
>>>>>         return 0;
>>>>>  }
>>>>>
>>>>> @@ -522,6 +560,10 @@ static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>>>>>                 return ret;
>>>>>         }
>>>>>
>>>>> +       ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data);
>>>>> +       if (ret)
>>>>> +               return ret;
>>>>> +
>>>>>         ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
>>>>>         if (ret) {
>>>>>                 dev_err(dev, "set sqc failed\n");
>>>>> @@ -589,6 +631,10 @@ static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>>>>>         vf_data->vf_qm_state = QM_READY;
>>>>>         hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
>>>>>
>>>>> +       ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data);
>>>>> +       if (ret)
>>>>> +               return ret;
>>>>> +
>>>>>         ret = vf_qm_read_data(vf_qm, vf_data);
>>>>>         if (ret)
>>>>>                 return ret;
>>>>> @@ -1186,34 +1232,52 @@ static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)
>>>>>  {
>>>>>         struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
>>>>>         struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
>>>>> +       struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
>>>>>         struct pci_dev *vf_dev = vdev->pdev;
>>>>> +       u32 val;
>>>>>
>>>>> -       /*
>>>>> -        * ACC VF dev BAR2 region consists of both functional register space
>>>>> -        * and migration control register space. For migration to work, we
>>>>> -        * need access to both. Hence, we map the entire BAR2 region here.
>>>>> -        * But unnecessarily exposing the migration BAR region to the Guest
>>>>> -        * has the potential to prevent/corrupt the Guest migration. Hence,
>>>>> -        * we restrict access to the migration control space from
>>>>> -        * Guest(Please see mmap/ioctl/read/write override functions).
>>>>> -        *
>>>>> -        * Please note that it is OK to expose the entire VF BAR if migration
>>>>> -        * is not supported or required as this cannot affect the ACC PF
>>>>> -        * configurations.
>>>>> -        *
>>>>> -        * Also the HiSilicon ACC VF devices supported by this driver on
>>>>> -        * HiSilicon hardware platforms are integrated end point devices
>>>>> -        * and the platform lacks the capability to perform any PCIe P2P
>>>>> -        * between these devices.
>>>>> -        */
>>>>> +       val = readl(pf_qm->io_base + QM_MIG_REGION_SEL);
>>>>> +       if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN))
>>>>> +               hisi_acc_vdev->drv_mode = HW_ACC_V4;
>>>>> +       else
>>>>> +               hisi_acc_vdev->drv_mode = HW_ACC_V3;
>>>>
>>>> The check is for > QM_HW_V3 and drv_mode is set to HW_ACC_V4. From our
>>>> previous discussions I think the expectation is that future hardware will follow
>>>> this same behaviour. If that is the case, it is better to rename HW_ACC_  to
>>>> something more specific to this change than use the V3/V4 name.
>>>>
>>>
>>> If the goal is merely to reflect the context of this change, it would be better to add
>>> a comment describing this change directly at the variable's declaration.
>>>
>>
>> Hi, Shameer:
>> If there are no further issues, I will continue using the current naming convention and
>> add comments at the declaration sites.
> 
> I would still prefer naming it differently than just adding comments.
> Maybe something like
> below + comments in the header.
> 
> if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN))
>      hisi_acc_vdev->mig_ctrl_mode = HW_ACC_MIG_PF_CTRL;
> else
>     hisi_acc_vdev->mig_ctrl_mode = HW_ACC_MIG_VF_CTRL;
> 
> Also please don't respin just for this yet. Please wait for Alex to take a look.
>

Hi,Alex:
Could you please take a look at this review comment? What's your opinion on it?

Thanks.
Longfang.

> Thanks,
> Shameer
> .
>