Re: RE: [RFC] PAGE_RW Should be added to PAGE_COPY ?

Re: RE: [RFC] PAGE_RW Should be added to PAGE_COPY ?

[Yingchao Zhou]

I think it's OK to lock the page for the NIC driver. But I think it is an OS flaw to 
clear the _PAGE_RW bit for ptes in the mprotected space. If the _PAGE_RW is set, it 
should not get to do_wp_page.

Best regards
>Your analysis is correct. But locking a page to avoid swap has different
>meaning with page locked (TestSetPageLocked()). Looks you missed this.
>
>Thanks,
>Shaohua
>-----Original Message-----
>From: linux-kernel-owner@vger.kernel.org
>[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Yingchao Zhou
>Sent: Wednesday, September 13, 2006 10:03 PM
>To: linux-kernel
>Cc: akpm; alan; zxc
>Subject: [RFC] PAGE_RW Should be added to PAGE_COPY ?
>
>
>     The current kernel set PAGE_COPY without write bit. This will cause
>intermittent non-cosistent data for user-level network drivers such as
>Infiniband, Quadrics and Myrinet. Which has also be mentioned by Costin
>Iancu in the paper "HUNTing the Overlap " (PACT'05).
>    An example of such phenomena is the following sequences: 
>	register a memory space BUFF for receive message, 
>	receive message,
>	call mprotect(...PROT_NONE...) and
>mprotect(...PROT_READ|PROT_WRITE) one by one, 	
>	write into BUFF, 
>	then receive again.      
>    The second time received data will perhaps not be the data sent by
>the peer machine but the data written by itself in the 4th step.
>
>     The reson is that :
>     1) User-level network driver locks phy pages when memory space is
>registered;
>........

___________________________________________________
_      Yingchao Zhou                              _
_      ICT, CAS                                   _
_      (86)010-62601009                           _
___________________________________________________

Re: Re: [RFC] PAGE_RW Should be added to PAGE_COPY ?

[Hugh Dickins]

On Fri, 15 Sep 2006, Yingchao Zhou wrote:
> >
> >You want to mmap MAP_SHARED, which will use PAGE_SHARED instead,
> >including the write bit, both before and after the mprotects.
> >There should be no problem then: do you actually see a problem
> >when MAP_SHARED is used?
> It's ok to mmap MAP_SHARED. But is it not a normal way to malloc() a space and
> then registered to NIC ?

Not that I know of.  How would one register malloc()ed space to a NIC?
Sorry, I may well be misunderstanding you.

> >>      Adding PAGE_RW to PAGE_COPY will resolve this problem.  
> >
> >No!  That would give every user write access to shared files they
> >should have no write access to.
> I guess you refer to mmap a file MAP_READ|MAP_WRITE in MAP_PRIVATE way.
> I think it is probably more logical to read the file data into an anoymous page and filled the pte with _PAGE_RW in the first time page-fault. It will probably reduce numbers of page fault interrupt.

do_no_page() does just that when its fault demands write access; but
doesn't waste memory and time on copying when it's only a read access.

Hugh

Re: Re: [RFC] PAGE_RW Should be added to PAGE_COPY ?

[Yingchao Zhou]

>
>PAGE_COPY (without the write bit) is used when the area was mmap'ed
>MAP_PRIVATE: which indeed is asking for private copies of pages to
>be made - which will be left containing the data written there by the
>application, rather than shared data received later by the driver.
>
>You want to mmap MAP_SHARED, which will use PAGE_SHARED instead,
>including the write bit, both before and after the mprotects.
>There should be no problem then: do you actually see a problem
>when MAP_SHARED is used?
It's ok to mmap MAP_SHARED. But is it not a normal way to malloc() a space and
then registered to NIC ?
>
>(You don't mention which release you're describing, and some of
>the details may vary: the not-yet-started 2.6.19 is likely to use
>PAGE_COPY even when MAP_SHARED, to help it keep track of the number
>of dirty pages; but in that case, do_wp_page() won't make a copy.)
>
>> 
>>      The reson is that :
>>      1) User-level network driver locks phy pages when memory space is registered;
>>      2) 2 calls to mprotect change ptes in the space to PAGE_COPY, so write any page in the space will cause a page fault;
>
>Not if PROT_WRITE, MAP_SHARED I think.
Yeah, of course.
>
>>      3) In the page fault handler, it goes to do_wp_page, and in it if Page Is Locked, a new page is generated and filled into the pte. So the physical page seen by the host is not the same one by the NIC.
>
>When MAP_PRIVATE, it's not the page being locked that causes the copy
>(it's not normally locked there, is it?), it's that it's not PageAnon;
>or if you're looking at 2.6.12 or older, that page_count is raised.
>
>> 
>>      Adding PAGE_RW to PAGE_COPY will resolve this problem.  
>
>No!  That would give every user write access to shared files they
>should have no write access to.
I guess you refer to mmap a file MAP_READ|MAP_WRITE in MAP_PRIVATE way.
I think it is probably more logical to read the file data into an anoymous page and filled the pte with _PAGE_RW in the first time page-fault. It will probably reduce numbers of page fault interrupt.
>
>>      In my option, the reason for absense of RW is to save memory by mapping all those only read pages into ZERO_PAGE. But is there really programs which make many read-ops in memory space without even initialize them?
>
>Not just the ZERO_PAGE: initial program data is another common example.
Ok, we can deal with initial program data using the above flow.
>
>Hugh
>
Best regards
Yingchao Zhou