-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In the last weeks I have been working on designing a new API for a NUMA
support library.  I am aware of the code in libnuma by ak but this code
has many shortcomings:

~ inadequate topology discovery
~ fixed cpu set size
~ no clear separation of memory nodes
~ no inclusion of SMT/multicore in the cpu hierarchy
~ awkward (at best) memory allocation interface
~ etc etc

and last but not least

~ a completely unacceptable library interface (e.g., global variables as
part of the API, WTF?)

At the end of the attached document is a comparison of the two APIs.


I'm only posting now about this since I wanted to get some sanity checks
of the API first.  Some of our (i.e., Red Hat's) partners provided this.
 They might identify themselves, or not.  This is not because other
parties are meant to be excluded.


The API described here is meant to be a minimal which can be wrapped for
use in any kind of higher-level language (or even in another C library
using the interface).  For this reason the CPU and memory node sets are
not handled by an abstract data type but instead as bitmap.  Using an
abstract data types (in C) means restricting the way wrapper libraries
can be designed.  In a C++ wrapper, for instance, the bit sets certainly
should be abstract.  A later version of the attached document might try
provide higher-level interfaces.


The text of the API proposal is not yet polished.  In fact, most
descriptions are fairly short.  I want o get some more assurance that
the API is received well before spending significantly more time on it.

As specified, the implementation of the interface is designed with only
the requirements of a program on NUMA hardware in mind.  I have paid no
attention to the currently proposed kernel extensions.  If the latter do
not really allow implementing the functionality programmers need then it
is wasted efforts.

For instance, I think the way memory allocated in interleaved fashion is
not "ideal".  Interleaved allocation is a property of a specific
allocation.  Global states for processes (or threads) are a terrible way
to handle this and other properties since it requires the programmer to
constantly switch the mode back and forth since any part of the runtime
might be NUMA aware and reset the mode.

Also, the concept of hard/soft sets for CPUs is useful.  Likewise
"spilling" over to other memory nodes.  Usually using NUMA means hinting
the desired configuration to the system.  It'll be used whenever
possible.  If it is not possible (for instance, if a given processor is
not available) it is mostly no good idea to completely fail the
execution.  Instead a less optimal resource should be used.  For memory
it is hard to know how much memory on which node is in use etc.

Another missing feature in libnuma and the current kernel design is
support for changes in the configuration.  CPUs might be added or
removed, likewise memory.  Additional interconnects between NUMA blocks
might be added etc.


Overall I think the proposed API provides a architecture-independent,
future-safe NUMA API.  If no program uses the kernel functionality
directly (which is possible with the API) the kernel interface can be
changed and adopted for each architecture or even specific machine
without the program noticing it.


The selection of names for the functions is by no means fixed.  These
are proposals.  I'm open for constructive criticism.  In case you find
interfaces to be missing or wrong or not optimal, please let me know as
well.  Once the API is regarded useful we can start thinking about the
kernel interface so keep these two things separated.


Please direct comments to me.  In case there is interest I can set up a
separate mailing list since lkml is probably not the best venue.

- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFAkgG92ijCOnn/RHQRAqgUAJ9bJ83LxSZ43TW5+5I1VhXV+zRPNACgnjmQ
SnFjDhA7v+5CGaZO5/jOxhw=
=93mp
-----END PGP SIGNATURE-----