Some CPUs have cache aliasing problem, where the same physical
memory could appear in more than one places in data cache
(such as those found in MIPS and Sparc).  For those CPUs, kiobuf
is not flushing cache properly.

The symptom can be easily found if you open files with O_DIRECT
flag and do file copies on those CPUs.

This patch fixes the problem.

Basically if it is a WRITE (from user to disk), we need to flush
cache before the IO.  If it is a READ, we need to flush cache after 
the IO.

Please apply.

Jun