[PATCH] drm/amdkfd: register HMM dev memory to DMA-able range first

[PATCH] drm/amdkfd: register HMM dev memory to DMA-able range first

[francisco_flynn]

HMM device memory is allocated at the top of
iomem_resource, when iomem_resource is larger than
GPU device's dma mask, after devm_memremap_pages,
max_pfn will also be update and exceed device's
dma mask, when there are multiple card on system
need to be init, ttm_device_init would be called
with use_dma32=true, and this is not necessary at
all. let's request dev memory region at DMA-able
range first.

Signed-off-by: francisco_flynn <francisco_flynn@foxmail.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index 79251f22b702..3856b9fd2a70 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -1020,6 +1020,7 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev)
 	struct amdgpu_kfd_dev *kfddev = &adev->kfd;
 	struct dev_pagemap *pgmap;
 	struct resource *res = NULL;
+	struct resource temp_res = iomem_resource;
 	unsigned long size;
 	void *r;
 
@@ -1042,7 +1043,10 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev)
 		pgmap->range.end = adev->gmc.aper_base + adev->gmc.aper_size - 1;
 		pgmap->type = MEMORY_DEVICE_COHERENT;
 	} else {
-		res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
+		temp_res.end = dma_get_mask(adev->dev);
+		res = devm_request_free_mem_region(adev->dev, &temp_res, size);
+		if (IS_ERR(res))
+			res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
 		if (IS_ERR(res))
 			return PTR_ERR(res);
 		pgmap->range.start = res->start;
-- 
2.25.1

Re: [PATCH] drm/amdkfd: register HMM dev memory to DMA-able range first

[Felix Kuehling]

On 2025-06-09 5:36, francisco_flynn wrote:
> HMM device memory is allocated at the top of
> iomem_resource, when iomem_resource is larger than
> GPU device's dma mask, after devm_memremap_pages,
> max_pfn will also be update and exceed device's
> dma mask, when there are multiple card on system
> need to be init, ttm_device_init would be called
> with use_dma32=true, and this is not necessary at
> all. let's request dev memory region at DMA-able
> range first.

That doesn't make sense to me. The addresses allocated here are not DMA addresses. They cannot be accessed by the GPU via DMA. They are purely fictional addresses for the purposes of creating struct pages for device-private memory. There should be no need to limit them by the GPU's DMA mask.

Regards,
  Felix


>
> Signed-off-by: francisco_flynn <francisco_flynn@foxmail.com>
> ---
>  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> index 79251f22b702..3856b9fd2a70 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> @@ -1020,6 +1020,7 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev)
>  	struct amdgpu_kfd_dev *kfddev = &adev->kfd;
>  	struct dev_pagemap *pgmap;
>  	struct resource *res = NULL;
> +	struct resource temp_res = iomem_resource;
>  	unsigned long size;
>  	void *r;
>  
> @@ -1042,7 +1043,10 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev)
>  		pgmap->range.end = adev->gmc.aper_base + adev->gmc.aper_size - 1;
>  		pgmap->type = MEMORY_DEVICE_COHERENT;
>  	} else {
> -		res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
> +		temp_res.end = dma_get_mask(adev->dev);
> +		res = devm_request_free_mem_region(adev->dev, &temp_res, size);
> +		if (IS_ERR(res))
> +			res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
>  		if (IS_ERR(res))
>  			return PTR_ERR(res);
>  		pgmap->range.start = res->start;

Re: [PATCH] drm/amdkfd: register HMM dev memory to DMA-able range first

[francisco_flynn]

On 6/9/25 20:46, Felix Kuehling wrote:
> On 2025-06-09 5:36, francisco_flynn wrote:
>> HMM device memory is allocated at the top of
>> iomem_resource, when iomem_resource is larger than
>> GPU device's dma mask, after devm_memremap_pages,
>> max_pfn will also be update and exceed device's
>> dma mask, when there are multiple card on system
>> need to be init, ttm_device_init would be called
>> with use_dma32=true, and this is not necessary at
>> all. let's request dev memory region at DMA-able
>> range first.
> That doesn't make sense to me. The addresses allocated here are not DMA addresses. They cannot be accessed by the GPU via DMA. They are purely fictional addresses for the purposes of creating struct pages for device-private memory. There should be no need to limit them by the GPU's DMA mask.


yes, this address is used by CPU to access VRAM. The patch is intended 
to cope with a special case, after checking the latest kernel code, i 
found this problem has been solved by this commit,

https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=7170130e4c72ce0caa0cb42a1627c635cc262821

thanks for you reply.

Best regards,
flynn


> Regards,
>    Felix
>
>
>> Signed-off-by: francisco_flynn <francisco_flynn@foxmail.com>
>> ---
>>   drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 6 +++++-
>>   1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>> index 79251f22b702..3856b9fd2a70 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>> @@ -1020,6 +1020,7 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev)
>>   	struct amdgpu_kfd_dev *kfddev = &adev->kfd;
>>   	struct dev_pagemap *pgmap;
>>   	struct resource *res = NULL;
>> +	struct resource temp_res = iomem_resource;
>>   	unsigned long size;
>>   	void *r;
>>   
>> @@ -1042,7 +1043,10 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev)
>>   		pgmap->range.end = adev->gmc.aper_base + adev->gmc.aper_size - 1;
>>   		pgmap->type = MEMORY_DEVICE_COHERENT;
>>   	} else {
>> -		res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
>> +		temp_res.end = dma_get_mask(adev->dev);
>> +		res = devm_request_free_mem_region(adev->dev, &temp_res, size);
>> +		if (IS_ERR(res))
>> +			res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
>>   		if (IS_ERR(res))
>>   			return PTR_ERR(res);
>>   		pgmap->range.start = res->start;