race leading to held mutexes, inode_cache corruption

["Sapan Bhatia" <sapan.bhatia@gmail.com>]

Hi,

We've been trying to investigate a file-system corruption issue in our
kernel (http://svn.planet-lab.org/browser/linux-2.6/trunk) that
manifests itself both with ext3 and ext2. It appears to be happening
to due a contamination of the inode cache (we spent some time
monitoring our systems to arrive at this hypothesis), and can be
reproduced on a vanilla kernel as well.

The race that leads to this issue involves a process being terminated
when it is waiting for a mutex in __mutex_lock_common. eg. when it is
sent a SIGKILL, and the mutex is unlocked, causing the process to be
woken up and sent to exit while now holding the lock.

The way it contaminates the inode_cache slab is that inode->i_mutex is
only initialized once, and assumes that inodes coming back into the
cache are initialized. It seems that in our case such poisoned inodes
were leaking out of pipe.c.

This (www.cs.princeton.edu/~sapanb/mut.c) is the module we used to
test the condition, as follows. Writing to the char device locks a
mutex and reading from it unlocks it.
# echo 1 > /dev/mut
# cat /etc/passwd > /dev/mut &
[2] 6232
# kill -9 6232
# cat /dev/mut
[2]-  Killed                  cat /etc/passwd > /dev/mut
# echo 1 > /dev/mut
(goes to sleep)

I suppose that one could also construct an attack to proactively
corrupt inode_cache, but I haven't tried that as yet.

Our base kernel is 2.6.22.19.

Thank you,
Sapan