Re: Exporting a lot of data to other processes?

[Nix <nix@esperi.org.uk>]

On 25 Oct 2007, Ph. Marek told this:
> -) use some ramfs/shmfs or similar, and overwrite the data occasionally
>    - not current data
>    - runtime overhead (processor load)

This is roughly what the nscd implementation in glibc does: the client
can work over a socket, but prefers to ask the daemon for a file
descriptor to an mmap()ed copy of the database. Then it works from that.

Properly done this is as efficient as working in the local process, with
only context switch overhead involved, and even that only when the
database is being updated. You *do* have to think about proper locking
of some kind, perhaps by designing the data structure in the shared
mmap()ed region appropriately.

(nscd has an ill-deserved reputation for sloth and memory-hungriness.
This is probably caused by bad experiences with old Solaris nscds, which
were quite appallingly inefficient. glibc nscd manages to speed up
passwd lookups even if your passwd database is only about ten lines
long, simply by amortizing the parsing overhead... and the nscd process
itself incurs *no* CPU overhead except as needed to reparse and update
the database, which is overhead that would otherwise be duplicated in
every process that did service parsing. As for memory hungriness, well,
yes, it looks like nscd is huge, but it's a huge *sparse* mmap()ed
block, so doesn't actually consume more than a few hundred Kb, not the
dozens of Mb it looks like in ps(1).)

-- 
`Some people don't think performance issues are "real bugs", and I think 
such people shouldn't be allowed to program.' --- Linus Torvalds