Re: [PATCH 02/12] mm: supporting variables and functions for balanced zone aging

[Andrew Morton <akpm@osdl.org>]

Marcelo Tosatti <marcelo.tosatti@cyclades.com> wrote:
>
> 
> It all makes sense to me (Wu's description of the problem and your patch), 
> but still no good with reference to fair scanning.

Not so.  On a 4G x86 box doing a simple 8GB write this patch took the
highmem/normal scanning ratio from 0.7 to 3.5.  On that setup the highmem
zone has 3.6x as many pages as the normal zone, so it's bang-on-target.

There's not a lot of point in jumping straight into the complex stresstests
without having first tested the simple stuff.

> Moreover the patch hurts 
> interactivity _badly_, not sure why (ssh into the box with FFSB testcase 
> takes more than one minute to login, while vanilla takes few dozens of seconds). 

Well, we know that the revert reintroduces an overscanning problem.

How are you invoking FFSB?  Exactly?  On what sort of machine, with how
much memory?

> Follows an interesting part of "diff -u 2614-vanilla.vmstat 2614-akpm.vmstat"
> (they were not retrieve at the exact same point in the benchmark run, but 
> that should not matter much):
> 
> -slabs_scanned 37632
> -kswapd_steal 731859
> -kswapd_inodesteal 1363
> -pageoutrun 26573
> -allocstall 636
> -pgrotated 1898
> +slabs_scanned 2688
> +kswapd_steal 502946
> +kswapd_inodesteal 1
> +pageoutrun 10612
> +allocstall 90
> +pgrotated 68
>
> Note how direct reclaim (and slabs_scanned) are hugely affected. 

hm.  allocstall is much lower and pgrotated has improved and direct reclaim
has improved.  All of which would indicate that kswapd is doing more work. 
Yet kswapd reclaimed less pages.  It's hard to say what's going on as these
numbers came from different stages of the test.

> 
> Normal: 114688kB
> DMA: 16384kB
> 
> Normal/DMA ratio = 114688 / 16384 = 7.000
>
> pgscan_kswapd Normal/DMA = (450483 / 88869) = 5.069
> pgscan_direct Normal/DMA = (23826 / 4224) = 5.640
> pgscan Normal/DMA = (474309 / 88869) = 5.337
> pgscan_kswapd Normal/DMA = (441936 / 80520) = 5.488
> pgscan_direct Normal/DMA = (7392/1188) = 6.222
> pgscan Normal/DMA = (449328 / 81708) = 5.499
> pgalloc_normal_dma_ratio = (559994 / 8488) = 6.597
> pgscan_kswapd Normal/DMA (664883/82845) = 8.025
> pgscan_direct Normal/DMA = (13485/1745) = 7.727
> pgscan Normal/DMA = (678368 / 84590) = 8.019
> pgalloc_normal_dma_ratio = (699927/66313) = 10.554

All of these look close enough to me.  10-20% over- or under-scanning of
the teeny DMA zone doesn't seem very important.  Getting normal-vs-highmem
right is more important.

It's hard to say what effect the watermark thingies have on all of this. 
I'd sugget that you start out with much less complex tests and see if `echo
10000 10000 10000 > /proc/sys/vm/lowmem_reserve_ratio' changes anything. 
(I have that in my rc.local - the thing is a daft waste of memory).

I'd be more concerned about the interactivity thing, although it sounds
like the machine is so overloaded with this test that it'd be fairly
pointless to try to tune that workload first.  It's more important to tune
the system for more typical heavy loads.

Also, the choice of IO scheduler matters.  Which are you using?