Re: Efficient IPC mechanism on Linux

[Alan Cox <alan@lxorguk.ukuu.org.uk>]

On Mer, 2003-09-10 at 10:40, Luca Veraldi wrote:
> To set the accessed or dirty bit you use
> 
> 38         __asm__ __volatile__( LOCK_PREFIX
> 39                 "btsl %1,%0"
> 40                 :"=m" (ADDR)
> 41                 :"Ir" (nr));
> 
> which is a ***SINGLE CLOCK CYCLE*** of cpu.

Its a _lot_ more than that in the normal case. Upwards of 60 clocks on 
a PIV. You then need to reload cr3 if you touched permissions which
means every cached TLB in the system is lost, and you may need to do
a cross CPU IPI on SMP (which takes a long time)

> You say "tlb's and internal cpu state will need to be flushed".
> The other cpus in an SMP environment can continue to work, indipendently.
> TLBs and cpu state registers are ***PER-CPU*** resorces.

Think of a threaded app passing a message to another app. You have to do
the cross CPU flush in order to prevent races where another thread can
scribble on data it doesnt own. Assuming a reasonable TLB reuse rate
thats 120 plus TLB reloads. The newer CPU's cache TLB's in L1/L2 so
thats not too bad but it all adds up. On SMP its a real pain

> Probably, it is worse the case of copying a memory page,
> because you have to hold some global lock all the time.
> This is deadly in an SMP environment, 

You don't need a global lock to copy memory. 

One thing I do agree with you on is the API aspect - and that is
problematic. The current API leaves data also owned by the source.
If I write "fred" down a pipe I still own the "fred" bits. The 
method you propose was added long ago go to Solaris (see "doors") and
its not exactly the most used interface even there.