Re: Proposal of improvement for DMA - direct passing of hugepages to the SG list

[Tom Rix <trix@redhat.com>]

On 4/14/21 4:58 AM, wzab@ise.pw.edu.pl wrote:
> Hi,
>
> I'm working both on DMA engines implementations in FPGA and their 
> Linux drivers.
> Now I need to create an engine that takes the hugepages-backed buffer 
> allocated
> by the user-space application and passes it to the device.
> My current solution:
>
> https://forums.xilinx.com/t5/Embedded-Linux/How-to-pass-efficiently-the-hugepages-backed-buffer-to-the-BM/m-p/1229340/highlight/true#M49777 
>
> or https://stackoverflow.com/a/67065962/1735409
>
> uses the get_user_pages_fast function, to create the kernel mapping,
> and then uses sg_alloc_table_from_pages to build sg_table for it.
>
> I have verified that the created sg_table has one entry for each 
> individual
> hugepage (so I can efficiently map it for my SG-capable DMA device).
>
> The disadvantage of that solution is that I need to create and keep a 
> huge set
> of standard-size pages. Because the "struct page" occupies between 56 
> and 80
> bytes, I get the overhead up to 80/4096 which is approx. 2%.
>
> The open question is if I can free those pages as soon as the sg_table
> is created? (As far as I know, the hugepages are locked automatically).
> Of course it is unclear what happens if the application crashes and 
> its mmaped
> hugepage-based buffer gets freed. Will the driver be notified about 
> closing the
> file so that it can disable DMA before that memory can be taken for other
> purposes?
>
> To be sure that it doesn't happen maybe it is good to keep those pages 
> locked
> in the kernel as well.
> The big improvement would be if we could have the get_user_hugepages_fast
> function. The user should be allowed to create a smaller number of 
> page structs.
> The function should check if the requested region really consists of 
> hugepages
> and return an error if it doesn't.
>
> Another important fact is that we don't need a kernel mapping for 
> those pages
> at all. So it would be good to have yet another function:
> sg_alloc_table_from_user_pages
> which should take an additional "flag" argument enabling the user to 
> decide
> if the area used consists of normal pages or of hugepages.
>
> The function should return an error in  case if the flag does not 
> match the
> properties of the region. Of course the function should also lock the 
> pages,
> and sg_free_table should unlock them (protecting against the danger
> of application crash, that I described above).
>
> As a temporary workaround, is it possible to "manually" walk the pages
> creating the application-delivered buffer, verify that they are 
> hugepages,
> lock them and create the sg_table?
> What functions/macros should be used for that to ensure that the 
> implemented
> solution be portable and keeps working in a reasonable number of 
> future versions
> of the Linux kernel?
>
> I'll appreciate comments, suggestions and considering of the above 
> proposal.
> With best regards,
> Wojtek
>
Do you have trial patch you could RFC ?

Are you aware of the xilinx qdma patchset ?

https://www.xilinx.com/support/answers/71453.html

https://github.com/Xilinx/dma_ip_drivers

Maybe its kernel or dpdk driver has what you need.

Tom