Linux Page Cache performance

Linux Page Cache performance

[Ravi Wijayaratne]

Hi All.

We are running dbench on a machine with Dual Xeon (Hyper threading 
turned off), 1GB RAM
and 4 Drive software RAID5. The kernel is 2.4.29-xfs 1.2. We are using 
LVM. However
similar test done using ext2 on a disk partiotion (no md or LVM) shows 

The throughput is find till the number of clients are Around 16. At 
that point the throughput
plummets about 40%. We are trying to avoid that and see how we could 
have a consistent throughput
perhaps sacrificing some peak performance.

One could argue that at the point the performance drops we are actualy 
beggining to see 
I/O requests missing page cache and going to disk. That is our current 
guess. We try to tune
the buffer age and the interval page buffer daemon runs, but had no 
effect on the curve.
So transactional meta data operations seems not be causing the bottle 
neck.

Are there any VM patches that smooths out the Page Cache dirty page 
swapping process so that we
wont see this sudden drop of through put, but could have a smoother 
transition. I ran a 
kernel profiler on the test and I dont see any dentry cache flushes or 
inode cache flushes.

Similar test done using ext2 on a disk partiotion (no md or LVM) shows 
us similar performances.
However for ext2 when we tweak the bdflush parameters in /proc 
(specifically the max age of meta
data buffers) we can push the point where the data throughput falls to 
around 40 clients. 

But we are more interested in finding out ways to solve the XFS case.

Any insights on this ?

Thanx
Ravi



=====
------------------------------
Ravi Wijayaratne

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus

Re: Linux Page Cache performance

[Mike Fedyk]

On Fri, Jan 16, 2004 at 11:11:02AM -0800, Ravi Wijayaratne wrote:
> Hi All.
> 
> We are running dbench on a machine with Dual Xeon (Hyper threading 

First off, dbench is a bad benchmark to tune for, it shows better numbers
for worse fairness.

Is yourworkload like dbench?  What will your workload be?

> turned off), 1GB RAM
> and 4 Drive software RAID5. The kernel is 2.4.29-xfs 1.2. We are using 
> LVM. However
> similar test done using ext2 on a disk partiotion (no md or LVM) shows 
> 
> The throughput is find till the number of clients are Around 16. At 
> that point the throughput
> plummets about 40%. We are trying to avoid that and see how we could 
> have a consistent throughput
> perhaps sacrificing some peak performance.
> 
> One could argue that at the point the performance drops we are actualy 
> beggining to see 
> I/O requests missing page cache and going to disk. That is our current 
> guess. We try to tune
> the buffer age and the interval page buffer daemon runs, but had no 
> effect on the curve.
> So transactional meta data operations seems not be causing the bottle 
> neck.
> 
> Are there any VM patches that smooths out the Page Cache dirty page 
> swapping process so that we

Of course, there's 2.4.24 which has XFS merged for you, and be sure to give
2.4.25-pre6 a try since it integrates part of the -aa VM for inodes in
highmem.

> wont see this sudden drop of through put, but could have a smoother 
> transition. I ran a 
> kernel profiler on the test and I dont see any dentry cache flushes or 
> inode cache flushes.
> 
> Similar test done using ext2 on a disk partiotion (no md or LVM) shows 
> us similar performances.
> However for ext2 when we tweak the bdflush parameters in /proc 
> (specifically the max age of meta
> data buffers) we can push the point where the data throughput falls to 
> around 40 clients. 
> 
> But we are more interested in finding out ways to solve the XFS case.
> 

There are options in XFS that change the layout of the filesystem that might
help (can't say more since I'm not too familiar with XFS).

Try without highmem, and see if that changes your results, and try with a
benchmark that behaves similair to how you expect from your workload.

Also, there's 2.6 which will give different characteristics, and especially
with the disk IO schedulers available there.

Re: Linux Page Cache performance

[Andrew Morton]

Ravi Wijayaratne <ravi_wija@yahoo.com> wrote:
>
> Hi All.
> 
> We are running dbench on a machine with Dual Xeon (Hyper threading 
> turned off), 1GB RAM
> and 4 Drive software RAID5. The kernel is 2.4.29-xfs 1.2. We are using 
> LVM. However
> similar test done using ext2 on a disk partiotion (no md or LVM) shows 
> 
> The throughput is find till the number of clients are Around 16. At 
> that point the throughput
> plummets about 40%. We are trying to avoid that and see how we could 
> have a consistent throughput
> perhaps sacrificing some peak performance.

Once the amount of dirty memory in the machine hits 40% of the total, the
VM starts to initiate writeout.  At 60% dirty the VM starts to deliberately
throttle the processes which are dirtying memory.

So you would expect to see extremely sudden transitions in overall
throughput at that point.  Note that if the test were to run for a longer
period of time, or if it were to leave the files behind rather than
suddenly removing them you would not notice this effect.


The probability that dbench's access patterns have any similarity to the
access patterns of the application which you care about is vanishingly
small.  The same can most probably be said of all the other off-the-shelf
benchmarks.  They're not very impressive.

You will need to analyse your application's access patterns and design a
benchmark which reasonably closely and repeatably models them.  Or, if
possible, use the live application itself.