Re: [PATCH] ummunotify: Userspace support for MMU notifications

["Håkon Bugge" <Haakon.Bugge@Sun.COM>]

On Apr 13, 2010, at 20:02 , Peter Zijlstra wrote:

> 
> Yeah, virtual<->physical maps can change through swapping, page
> migration, memory compaction, huge-page aggregation (the latter two not
> yet being upstream).

Assuming this holds true, RDMA will not work. And with no RDMA, we do not need ummunotify. Is that your argument?

Seriously, RDMA requires the virtual to physical mapping to remain constant for a period of time. And that time period is from memory registration to de-registration. If the virtual to physical mapping changes in that period for the registered memory area, I can't see how an HCA can handle that.

For MPI applications, the MPI API defines that the buffers used in communication cannot be changed or freed while a non-blocking transfer is in progress. So far, we are good. But memory registration (and in particular de-registration) is a costly process. Since MPI is about performance, the MPI library would like to  re-use earlier memory registrations for other transfers. The MPI library records earlier memory mappings (VA+bound) and checks if the VA+bound of a new transfer is already contained in memory revisions registered earlier.

The problem with this approach is that _normal_ activity _between_ the transfers changes the virtual to physical mapping and ruins previous memory registrations. The problem is to catch these. I simply argue that they should be caught the simplest possible way; a call-back from the system calls affecting the virtual to physical mapping.

> 
> Even mlock() doesn't pin virtual<->physical maps.

Can you elaborate on what you mean here?


Thanks, Håkon