Re: [PATCH] arch/x86: memory_hotplug, do not bump up max_pfn for device private pages

[Balbir Singh <balbirs@nvidia.com>]

On 4/1/25 19:57, Ingo Molnar wrote:
> 
> * Balbir Singh <balbirs@nvidia.com> wrote:
> 
>>  arch/x86/mm/init_64.c | 15 ++++++++++++---
>>  1 file changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
>> index dce60767124f..cc60b57473a4 100644
>> --- a/arch/x86/mm/init_64.c
>> +++ b/arch/x86/mm/init_64.c
>> @@ -970,9 +970,18 @@ int add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
>>  	ret = __add_pages(nid, start_pfn, nr_pages, params);
>>  	WARN_ON_ONCE(ret);
>>  
>> -	/* update max_pfn, max_low_pfn and high_memory */
>> -	update_end_of_memory_vars(start_pfn << PAGE_SHIFT,
>> -				  nr_pages << PAGE_SHIFT);
>> +	/*
>> +	 * add_pages() is called by memremap_pages() for adding device private
>> +	 * pages. Do not bump up max_pfn in the device private path. max_pfn
>> +	 * changes affect dma_addressing_limited. dma_addressing_limited
>> +	 * returning true when max_pfn is the device's addressable memory,
>> +	 * can force device drivers to use bounce buffers and impact their
>> +	 * performance
>> +	 */
>> +	if (!params->pgmap)
>> +		/* update max_pfn, max_low_pfn and high_memory */
>> +		update_end_of_memory_vars(start_pfn << PAGE_SHIFT,
>> +					  nr_pages << PAGE_SHIFT);
> 
> So given that device private pages are not supposed to be mapped 
> directly, not including these PFNs in max_pfn absolutely sounds like 
> the correct fix to me.
> 
> But wouldn't the abnormally high max_pfn also cause us to create a too 
> large direct mapping to cover it, or does something save us there? Such 
> an overly large mapping would increase kernel page table size rather 
> substantially on non-gbpages systems, AFAICS.
> 
> Say we create a 16TB mapping on a 16GB system - 1024x larger: to map 16 
> TB with largepages requires 8,388,608 largepage mappings (!), which 
> with 8-byte page table entries takes up ~64MB of unswappable RAM. (!!)
> 
> Is my math off, or am I misunderstanding something here?
> 

That is a valid point, but that is only if we cover all of the max_pfn
with direct mapping (I can't seem to remember if we do so with sparsemem)

> Anyway, I've applied your fix to tip:x86/urgent with a few edits to the 
> comments and the changelog, but I've also expanded the Cc: list of the 
> commit liberally, in hope of getting more reviews for this fix. :-)
> 


Thanks and I'd like to get broader testing as well. I am also inclined to
send an RFC to add a WARN_ON_ONCE() if dma_addressing_limited returns
true for 64 bit systems, not sure if the DMA folks would be inclined and
how often it really happens on existing systems.

Balbir Singh