Hi,

Attached is a patch to 2.6.8-rc3 that adds NUMA mem policy
support for file mappings. With the patch, mbind() will "do the right thing"
for file mapped regions, ie. page cache pages for the region will be
allocated using the given policy, and all other vma's mapping the same
region of the file will use the same policy. It does this by adding a 
shared_policy
Red-Black tree to address_space. In more detail, the patch is doing the
following:

1. add a shared_policy tree to the address_space object in fs.h.

2. modify page_cache_alloc() in pagemap.h to take a page index in
    addition to a mapping object, and use those to allocate a page for
    the page cache using the appopriate policy found in the mapping->policy
    tree.

3. modify filemap.c to pass the additional page offset to 
page_cache_alloc().

4. Also in filemap.c, implement generic file {set|get}_policy() methods and
    add those to generic_file_vm_ops.

5. In filemap_nopage(), verify that any existing page located in the cache
    is located in a node that satisfies the file's policy. If it's not 
in a node that
    satisfies the policy, it must be because the page was allocated 
before the
    file had any policies. If it's unused, free it and goto retry_find 
(replace with
    a new page using the file's policy). In addition to the page being 
unused, the
    replace is qualified with a new page flag, signifying that this page was
    allocated without the guidance of a shared policy. A similar 
replacement is
    done in exec.c:setup_arg_pages() for arg pages that were allocated 
before any
    policy was known for the stack region.

6. Init the file's shared policy in alloc_inode(), and free the shared 
policy in
   destroy_inode().

This patch also paves the way for executable images to contain policy info,
so that their text and initialized data regions can be bound to specific
nodes. I'm working on a utility ("elfmembind") that tags ELF
executables and shared libs with policy info for text and data. A kernel
patch to fs/binfmt_elf.c parses a new ELF section added by the tool, and
generates a policy for the file's text and initialized data regions. The 
policies
are then inserted into the file mapping's shared_policy tree via 
sys_mbind().

But I'm not sure a post-link time tool is the right approach for 
applying policies
to executable loadable segments. ELF sections are really not supposed to 
be used
at load time, only the PT_LOAD segments are applicable for program loading.
Maybe a good long-term solution is to add policy support to the linker, 
ld. That
way the policy info can be folded into the loadable segments (a NOTE segment
might be the right approach). It's very difficult to modify loadable 
segments post-link
time, unless there are some free bits or fields that can be hijacked. 
Then again,
a post-link utility is more convenient than having to rebuild an app. 
Comments?
Are there already plans afoot to add NUMA mem policies to ELF executables,
and if so how?

Steve