Very rarely, we experience a setuid program not properly getting
the euid of its owner.  This happens with (at least) both Linux 2.6.24.7
and Linux 2.6.28.4, on multiple machines of at least two configurations
(Dell 860 and Dell 2950 - cpuinfo attached). 

The attached suid.c program does nothing except check that it got
the correct UID, and prints an error and exits with status 1 upon failure,
or status 0 on correct setuid.  A number of times it has taken over 7 hours
to fail, though it sometimes fails within minutes - failures are not
frequent.

The python script just counts the number of times, and elapsed walltime,
it is executed and includes some sanity checks to ensure the test is
being run properly.

localhost:~# chown root.root suid
localhost:~# chmod u+s suid
localhost:~# sudo -u metacarta python suid.py
euid:100 at Tue Feb 24 20:56:14 2009
Failed after 16640723 tries! Ran 3:41:40

Thus far, we have only seen failures for the program being setuid
root, being run by a non-root user, on a multi-core machine.  Trying to
setuid to a user from root, *or* booting with maxcpus=1 and trying to
setuid from a non-root user to root, both fail.  We don't have any
evidence that anything could possibly be changing the owner of the file
or permissions at failure, and immediately after a failure (if the test
harness is modified to not exit) setuid will be successful again.

The Dell 2950s are running the September 2008 Xeon 5000 microcode.
Note that despite Xen paravirtualization being enabled in the 2.6.28.4
config, the tests were not run virtualized, but on raw hardware.

The compiler used to compile the kernel:
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --program-suffix=-4.1 --enable-__cxa_atexit --enable-clocale=gnu --enable-libstdcxx-debug --enable-mpfr --with-tune=i686 --enable-checking=release i486-linux-gnu
Thread model: posix
gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)

The 2.6.28.4 kernel is vanilla except for three patches that aren't remotely
related:
  1) Modification to the mptbase driver for older VMWare compatibility.  We
     aren't using the mptbase driver in tests.
  2) Modifications to add an option to the megaraid_sas to expose individual
    disks to the SCSI midlayer (rather than returning ENXIO).  While the
    megaraid_sas driver is being used in the tests, the option is turned off.
  3) A bugfix to xen-blkfront not yet in 2.6.28.4, but we are not using Xen
     paravirtualization or the xen-blkfront driver in the tests.

The 2.6.24.7 kernel includes a number of other patches for security issues,
in addition to #1 and #2 above.  The kernels are x86-64, and do not use
SELinux or the new file capabilities feature.

We will try newer, and unmodified kernels, as well as we continue testing,
and other compilers as well, but this bug is completely perplexing.

Does anyone have any possible insights as to what this could possibly be,
or other things to try?  Is there any more information we can give that
will help?

Inspection of gdb disassembly of several functions from
security/commoncap.c where the setuid-to-root case is special look okay at
initial glance.

Thanks!
Joe Malicki