RE: cleancache can lead to serious performance degradation

[Dan Magenheimer <dan.magenheimer@oracle.com>]

> From: Seth Jennings [mailto:sjenning@linux.vnet.ibm.com]
> 
> On 08/25/2011 11:56 AM, Dan Magenheimer wrote:
> 
> > Third, zcache is relatively new and can certainly benefit from
> > the input of other developers.  The lzo1x compression in the kernel
> > is fairly slow; Seth Jennings (cc'ed) is looking into alternate
> > compression technologies.  Perhaps there is a better compression
> > choice more suitable for older-slower processors, probably with a
> > poorer compression ratio.  Further, zcache currently does compression
> > and decompression with interrupts disabled, which may be a
> > significant factor in the slowdowns you've observed.  This should
> > be fixable.
> 
> This was something I've meaning to ask about.  Why are compression
> and decompression done with interrupts disabled?

The "hb->lock" is held during most tmem operations.  If a tmem
operation (or the callback to zcache) is interrupted and the
current cpu is scheduled to run another task, and the new task
calls into tmem, deadlock could occur.

In some cases, I think disabling preemption or bottom-halves instead
of disabling interrupts may be sufficient, but I ran into problems
when I tried that and never got back to it.  Note though that
interrupts are already disabled in some cases when cleancache is
called, so that would only solve part of the problem.

> What would need to change so that we don't have to disable interrupts?

Not easy, but not terribly hard I think:

1) On put, copy the page into tmem uncompressed, and keep a list
of not-yet-compressed pages.  (The copy would still need to be
done with interrupts disabled but copying a page is presumable
one-to-three orders of magnitude faster than compress/decompress.)
2) An independent lower-priority thread would be launched periodically
to compress one or more pages on the uncompressed list and
atomically replace the uncompressed page with the compressed version,
and change all pointers.  Then the uncompressed page could be freed.
There are likely some ugly race conditions in here.
3) On get, ensure not-yet-compressed pages are properly handled.

Dan

P.S. On vacation today, no more email until next week.