[PATCH][2.6.8-rc1-mm1] Autotune swappiness01

[PATCH][2.6.8-rc1-mm1] Autotune swappiness01

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

Attached is a patch designed to improve the behaviour of the swappiness knob 
in 2.6.8-rc1-mm1. 

The current mechanism decides to reclaim mapped pages based on the 
combination of mapped_ratio/2 and the manual setting of swappiness currently 
tuned to 60. Biasing this mechanism to be proportional to the square root of 
mapped_ratio gives good overall performance improvement for desktop 
workloads without any noticable detriment to other loads. It has the effect 
of being fairly aggressive at avoiding loss of applications to swap under 
conditions of heavy or sustained file stress while allowing applications to 
swap out under what would be considered "application" memory stresses on a 
desktop. It has no measurable effect on any known benchmarks.

The swappiness knob is kept intact and ironically is set to the same value 
of 60, and overall behaves the same as previous patches posted for 
autoregulating swappiness. The idea of this patch is to ultimately deprecate 
the need for a swappiness knob if this achieves good performance in most 
workloads.

Signed-off-by: Con Kolivas <kernel@kolivas.org>


[-- Attachment #2: Autotune Swappiness --]
[-- Type: text/plain, Size: 1229 bytes --]

Index: linux-2.6.8-rc1/mm/vmscan.c
===================================================================
--- linux-2.6.8-rc1.orig/mm/vmscan.c	2004-07-15 09:59:34.820962044 +1000
+++ linux-2.6.8-rc1/mm/vmscan.c	2004-07-15 10:03:06.243433612 +1000
@@ -119,6 +119,7 @@
  * From 0 .. 100.  Higher means more swappy.
  */
 int vm_swappiness = 60;
+static int mapped_bias;
 static long total_memory;
 
 static LIST_HEAD(shrinker_list);
@@ -646,6 +647,7 @@
 	struct page *page;
 	struct pagevec pvec;
 	int reclaim_mapped = 0;
+	int swappiness_bias;
 	long mapped_ratio;
 	long distress;
 	long swap_tendency;
@@ -690,6 +692,7 @@
 	 * is mapped.
 	 */
 	mapped_ratio = (sc->nr_mapped * 100) / total_memory;
+	mapped_bias = mapped_ratio * mapped_ratio;
 
 	/*
 	 * Now decide how much we really want to unmap some pages.  The mapped
@@ -700,7 +703,9 @@
 	 *
 	 * A 100% value of vm_swappiness overrides this algorithm altogether.
 	 */
-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	swappiness_bias = vm_swappiness * vm_swappiness / 100;
+	swappiness_bias = 101 - swappiness_bias;
+	swap_tendency = distress + mapped_bias / swappiness_bias;
 
 	/*
 	 * Now use this metric to decide whether to start moving mapped memory

Re: [PATCH][2.6.8-rc1-mm1] Autotune swappiness01

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
> Attached is a patch designed to improve the behaviour of the swappiness knob 
> in 2.6.8-rc1-mm1. 
> 
> The current mechanism decides to reclaim mapped pages based on the 
> combination of mapped_ratio/2 and the manual setting of swappiness currently 
> tuned to 60. Biasing this mechanism to be proportional to the square root of 
> mapped_ratio gives good overall performance improvement for desktop 
> workloads without any noticable detriment to other loads.

OK...

> It has the effect 
> of being fairly aggressive at avoiding loss of applications to swap under 
> conditions of heavy or sustained file stress while allowing applications to 
> swap out under what would be considered "application" memory stresses on a 
> desktop.

But decreasing /proc/sys/vm/swappiness does that too?

> It has no measurable effect on any known benchmarks.

So how are we to evaluate the desirability of the patch???

> The swappiness knob is kept intact and ironically is set to the same value 
> of 60, and overall behaves the same as previous patches posted for 
> autoregulating swappiness. The idea of this patch is to ultimately deprecate 
> the need for a swappiness knob if this achieves good performance in most 
> workloads.

Don't think so.  If you have a machine with a lot of memory which is doing
mainly pagecache-intensive work and you also want it to aggressively swap
out anonymous pages (ie: your initials are akpm) then you'll be setting
swappiness to 100.


Shouldn't mapped_bias be local to refill_inactive_zone()?

Why is `swappiness' getting squared?  AFAICT this will simply make the
swappiness control behave nonlinearly, which seems undesirable?

Re: Autotune swappiness01

[Con Kolivas]

Andrew Morton writes:

> Con Kolivas <kernel@kolivas.org> wrote:
>>
>> Attached is a patch designed to improve the behaviour of the swappiness knob 
>> in 2.6.8-rc1-mm1. 
>> 
>> The current mechanism decides to reclaim mapped pages based on the 
>> combination of mapped_ratio/2 and the manual setting of swappiness currently 
>> tuned to 60. Biasing this mechanism to be proportional to the square root of 
>> mapped_ratio gives good overall performance improvement for desktop 
>> workloads without any noticable detriment to other loads.
> 
> OK...
> 
>> It has the effect 
>> of being fairly aggressive at avoiding loss of applications to swap under 
>> conditions of heavy or sustained file stress while allowing applications to 
>> swap out under what would be considered "application" memory stresses on a 
>> desktop.
> 
> But decreasing /proc/sys/vm/swappiness does that too?

Low memory boxes and ones that are heavily laden with applications find that 
ends up making things slow down trying to keep all applications in physical 
ram.

> 
>> It has no measurable effect on any known benchmarks.
> 
> So how are we to evaluate the desirability of the patch???

Get desktop users to report back their experiences which is what I have 
currently. Sorry we're in the realm of subjectivity again.

> Shouldn't mapped_bias be local to refill_inactive_zone()?

That is so a followup patch can use it elsewhere...

> Why is `swappiness' getting squared?  AFAICT this will simply make the
> swappiness control behave nonlinearly, which seems undesirable?

To parallel the nonlinear nature of the mapped bias effect. 

Con

Re: Autotune swappiness01

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
> >> It has the effect 
> >> of being fairly aggressive at avoiding loss of applications to swap under 
> >> conditions of heavy or sustained file stress while allowing applications to 
> >> swap out under what would be considered "application" memory stresses on a 
> >> desktop.
> > 
> > But decreasing /proc/sys/vm/swappiness does that too?
> 
> Low memory boxes and ones that are heavily laden with applications find that 
> ends up making things slow down trying to keep all applications in physical 
> ram.

Doesn't that mean that swappiness was decreased by too much?

> > 
> >> It has no measurable effect on any known benchmarks.
> > 
> > So how are we to evaluate the desirability of the patch???
> 
> Get desktop users to report back their experiences which is what I have 
> currently. Sorry we're in the realm of subjectivity again.

Seriously, we've seen placebo effects before...

> > Shouldn't mapped_bias be local to refill_inactive_zone()?
> 
> That is so a followup patch can use it elsewhere...

erk.  I guess it's OK because the thing is derived from global state which
changes slowly over time.

> > Why is `swappiness' getting squared?  AFAICT this will simply make the
> > swappiness control behave nonlinearly, which seems undesirable?
> 
> To parallel the nonlinear nature of the mapped bias effect. 

That doesn't really answer my question?  What goes wrong if swappiness is
not squared?

Re: Autotune swappiness01

[Con Kolivas]

Andrew Morton writes:

>> > But decreasing /proc/sys/vm/swappiness does that too?
>> 
>> Low memory boxes and ones that are heavily laden with applications find that 
>> ends up making things slow down trying to keep all applications in physical 
>> ram.
> 
> Doesn't that mean that swappiness was decreased by too much?

Sure does. But the desired effect when it's not applications is that of 
swappiness being excruciatingly low so people end up doing that. Then they 
find themselves dropping it at nighttime and increasing it during the day.

>> >> It has no measurable effect on any known benchmarks.
>> > 
>> > So how are we to evaluate the desirability of the patch???
>> 
>> Get desktop users to report back their experiences which is what I have 
>> currently. Sorry we're in the realm of subjectivity again.
> 
> Seriously, we've seen placebo effects before...

I am in full agreement there... It's easy to see that applications do not 
swap out overnight; but i'm having difficulty trying to find a way to 
demonstrate the other part. I guess timing the "linking the kernel with full 
debug" on a low memory box is measurable.

>> > Shouldn't mapped_bias be local to refill_inactive_zone()?
>> 
>> That is so a followup patch can use it elsewhere...
> 
> erk.  I guess it's OK because the thing is derived from global state which
> changes slowly over time.
> 
>> > Why is `swappiness' getting squared?  AFAICT this will simply make the
>> > swappiness control behave nonlinearly, which seems undesirable?
>> 
>> To parallel the nonlinear nature of the mapped bias effect. 
> 
> That doesn't really answer my question?  What goes wrong if swappiness is
> not squared?

Oh sorry, perhaps I should have said - that keeps people's current settings 
meaningful, but that can happily be broken. That would make the default 
setting something like 34 instead of 60.

Cheers,
Con

Re: Autotune swappiness01

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

Con Kolivas writes:

> Andrew Morton writes:

>> Seriously, we've seen placebo effects before...
> 
> I am in full agreement there... It's easy to see that applications do not 
> swap out overnight; but i'm having difficulty trying to find a way to 
> demonstrate the other part. I guess timing the "linking the kernel with full 
> debug" on a low memory box is measurable.

I should have said - finding a swappiness that ensures not swapping out 
applications with updatedb, then using that same swappiness value to do the 
linking test.

Con


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autotune swappiness01

[R. J. Wysocki]

On Monday 26 of July 2004 03:09, Con Kolivas wrote:
> Con Kolivas writes:
> > Andrew Morton writes:
> >> Seriously, we've seen placebo effects before...
> >
> > I am in full agreement there... It's easy to see that applications do not
> > swap out overnight; but i'm having difficulty trying to find a way to
> > demonstrate the other part. I guess timing the "linking the kernel with
> > full debug" on a low memory box is measurable.
>
> I should have said - finding a swappiness that ensures not swapping out
> applications with updatedb, then using that same swappiness value to do the
> linking test.

Please excuse me, but is that viable at all?  IMHO, it's just like trying to 
tune a radio including volume with only one knob.  I don't say it won't work, 
but the probability that it will is rather small, it seems ...

rjw

-- 
Rafael J. Wysocki
[tel. (+48) 605 053 693]
----------------------------
For a successful technology, reality must take precedence over public 
relations, for nature cannot be fooled.
					-- Richard P. Feynman

Re: Autotune swappiness01

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

R. J. Wysocki wrote:
> On Monday 26 of July 2004 03:09, Con Kolivas wrote:
> 
>>Con Kolivas writes:
>>
>>>Andrew Morton writes:
>>>
>>>>Seriously, we've seen placebo effects before...
>>>
>>>I am in full agreement there... It's easy to see that applications do not
>>>swap out overnight; but i'm having difficulty trying to find a way to
>>>demonstrate the other part. I guess timing the "linking the kernel with
>>>full debug" on a low memory box is measurable.
>>
>>I should have said - finding a swappiness that ensures not swapping out
>>applications with updatedb, then using that same swappiness value to do the
>>linking test.
> 
> 
> Please excuse me, but is that viable at all?  IMHO, it's just like trying to 
> tune a radio including volume with only one knob.  I don't say it won't work, 
> but the probability that it will is rather small, it seems ...

Well that's what we want. I cant remember other desktop operating 
systems setting a root only control between night and day, or between 
copying ISOs and running applications or...

Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Autotune swappiness01

[R. J. Wysocki]

On Monday 26 of July 2004 11:31, Con Kolivas wrote:
> R. J. Wysocki wrote:
> > On Monday 26 of July 2004 03:09, Con Kolivas wrote:
> >>Con Kolivas writes:
> >>>Andrew Morton writes:
> >>>>Seriously, we've seen placebo effects before...
> >>>
> >>>I am in full agreement there... It's easy to see that applications do
> >>> not swap out overnight; but i'm having difficulty trying to find a way
> >>> to demonstrate the other part. I guess timing the "linking the kernel
> >>> with full debug" on a low memory box is measurable.
> >>
> >>I should have said - finding a swappiness that ensures not swapping out
> >>applications with updatedb, then using that same swappiness value to do
> >> the linking test.
> >
> > Please excuse me, but is that viable at all?  IMHO, it's just like trying
> > to tune a radio including volume with only one knob.  I don't say it
> > won't work, but the probability that it will is rather small, it seems
> > ...
>
> Well that's what we want. I cant remember other desktop operating
> systems setting a root only control between night and day, or between
> copying ISOs and running applications or...

I agree, but isn't it related to the fact that other desktop OSes usually 
don't run anything like updatedb nightly?

Perhaps we need a bit more sophisticated swap algorithm than other OSes do.  
For example, couldn't we add an additional parameter to control the swapping 
"behavior", apart from the swappiness?  Something like adding the second knob 
in my radio example?  Just thinking,

rjw

-- 
Rafael J. Wysocki
[tel. (+48) 605 053 693]
----------------------------
For a successful technology, reality must take precedence over public 
relations, for nature cannot be fooled.
					-- Richard P. Feynman

Re: Autotune swappiness01

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1833 bytes --]

R. J. Wysocki wrote:
> On Monday 26 of July 2004 11:31, Con Kolivas wrote:
> 
>>R. J. Wysocki wrote:
>>
>>>On Monday 26 of July 2004 03:09, Con Kolivas wrote:
>>>
>>>>Con Kolivas writes:
>>>>
>>>>>Andrew Morton writes:
>>>>>
>>>>>>Seriously, we've seen placebo effects before...
>>>>>
>>>>>I am in full agreement there... It's easy to see that applications do
>>>>>not swap out overnight; but i'm having difficulty trying to find a way
>>>>>to demonstrate the other part. I guess timing the "linking the kernel
>>>>>with full debug" on a low memory box is measurable.
>>>>
>>>>I should have said - finding a swappiness that ensures not swapping out
>>>>applications with updatedb, then using that same swappiness value to do
>>>>the linking test.
>>>
>>>Please excuse me, but is that viable at all?  IMHO, it's just like trying
>>>to tune a radio including volume with only one knob.  I don't say it
>>>won't work, but the probability that it will is rather small, it seems
>>>...
>>
>>Well that's what we want. I cant remember other desktop operating
>>systems setting a root only control between night and day, or between
>>copying ISOs and running applications or...
> 
> 
> I agree, but isn't it related to the fact that other desktop OSes usually 
> don't run anything like updatedb nightly?
> 
> Perhaps we need a bit more sophisticated swap algorithm than other OSes do.  
> For example, couldn't we add an additional parameter to control the swapping 
> "behavior", apart from the swappiness?  Something like adding the second knob 
> in my radio example?  Just thinking,

I think one knob is one knob too many already. However as Andrew has 
pointed out there are server workloads that want swappiness of 100, 
hence I leave the knob in place.

Proof is in the pudding. Try my patch and/or post an alternative.

Cheers,
Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Autotune swappiness01

[R. J. Wysocki]

On Monday 26 of July 2004 12:29, Con Kolivas wrote:
> R. J. Wysocki wrote:
> > On Monday 26 of July 2004 11:31, Con Kolivas wrote:
> >>R. J. Wysocki wrote:
> >>>On Monday 26 of July 2004 03:09, Con Kolivas wrote:
> >>>>Con Kolivas writes:
> >>>>>Andrew Morton writes:
> >>>>>>Seriously, we've seen placebo effects before...
> >>>>>
> >>>>>I am in full agreement there... It's easy to see that applications do
> >>>>>not swap out overnight; but i'm having difficulty trying to find a way
> >>>>>to demonstrate the other part. I guess timing the "linking the kernel
> >>>>>with full debug" on a low memory box is measurable.
> >>>>
> >>>>I should have said - finding a swappiness that ensures not swapping out
> >>>>applications with updatedb, then using that same swappiness value to do
> >>>>the linking test.
> >>>
> >>>Please excuse me, but is that viable at all?  IMHO, it's just like
> >>> trying to tune a radio including volume with only one knob.  I don't
> >>> say it won't work, but the probability that it will is rather small, it
> >>> seems ...
> >>
> >>Well that's what we want. I cant remember other desktop operating
> >>systems setting a root only control between night and day, or between
> >>copying ISOs and running applications or...
> >
> > I agree, but isn't it related to the fact that other desktop OSes usually
> > don't run anything like updatedb nightly?
> >
> > Perhaps we need a bit more sophisticated swap algorithm than other OSes
> > do. For example, couldn't we add an additional parameter to control the
> > swapping "behavior", apart from the swappiness?  Something like adding
> > the second knob in my radio example?  Just thinking,
>
> I think one knob is one knob too many already.

Can you please tell me why do you think so?

rjw

-- 
Rafael J. Wysocki
[tel. (+48) 605 053 693]
----------------------------
For a successful technology, reality must take precedence over public 
relations, for nature cannot be fooled.
					-- Richard P. Feynman

Re: Autotune swappiness01

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1990 bytes --]

R. J. Wysocki wrote:
> On Monday 26 of July 2004 12:29, Con Kolivas wrote:
> 
>>R. J. Wysocki wrote:
>>
>>>On Monday 26 of July 2004 11:31, Con Kolivas wrote:
>>>
>>>>R. J. Wysocki wrote:
>>>>
>>>>>On Monday 26 of July 2004 03:09, Con Kolivas wrote:
>>>>>
>>>>>>Con Kolivas writes:
>>>>>>
>>>>>>>Andrew Morton writes:
>>>>>>>
>>>>>>>>Seriously, we've seen placebo effects before...
>>>>>>>
>>>>>>>I am in full agreement there... It's easy to see that applications do
>>>>>>>not swap out overnight; but i'm having difficulty trying to find a way
>>>>>>>to demonstrate the other part. I guess timing the "linking the kernel
>>>>>>>with full debug" on a low memory box is measurable.
>>>>>>
>>>>>>I should have said - finding a swappiness that ensures not swapping out
>>>>>>applications with updatedb, then using that same swappiness value to do
>>>>>>the linking test.
>>>>>
>>>>>Please excuse me, but is that viable at all?  IMHO, it's just like
>>>>>trying to tune a radio including volume with only one knob.  I don't
>>>>>say it won't work, but the probability that it will is rather small, it
>>>>>seems ...
>>>>
>>>>Well that's what we want. I cant remember other desktop operating
>>>>systems setting a root only control between night and day, or between
>>>>copying ISOs and running applications or...
>>>
>>>I agree, but isn't it related to the fact that other desktop OSes usually
>>>don't run anything like updatedb nightly?
>>>
>>>Perhaps we need a bit more sophisticated swap algorithm than other OSes
>>>do. For example, couldn't we add an additional parameter to control the
>>>swapping "behavior", apart from the swappiness?  Something like adding
>>>the second knob in my radio example?  Just thinking,
>>
>>I think one knob is one knob too many already.
> 
> 
> Can you please tell me why do you think so?

If you wanna discuss pedantics...

In my ideal, nonsensical, impossible to obtain world we have an 
autoregulating operating system that doesn't need any knobs.

Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Autotune swappiness01

[Nick Piggin]

Con Kolivas wrote:
> R. J. Wysocki wrote:
> 
>> On Monday 26 of July 2004 12:29, Con Kolivas wrote:
>>

>>> I think one knob is one knob too many already.
>>
>>
>>
>> Can you please tell me why do you think so?
> 
> 
> If you wanna discuss pedantics...
> 
> In my ideal, nonsensical, impossible to obtain world we have an 
> autoregulating operating system that doesn't need any knobs.
> 

Some thinks are fundamental tradeoffs that can't be autotuned.

Latency vs throughput comes up in a lot of places, eg. timeslices.

Maximum throughput via effective use of swap, versus swapping as
a last resort may be another.

Re: Autotune swappiness01

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

Nick Piggin wrote:
> Con Kolivas wrote:
> 
>> R. J. Wysocki wrote:
>>
>>> On Monday 26 of July 2004 12:29, Con Kolivas wrote:
>>>
> 
>>>> I think one knob is one knob too many already.
>>>
>>>
>>>
>>>
>>> Can you please tell me why do you think so?
>>
>>
>>
>> If you wanna discuss pedantics...
>>
>> In my ideal, nonsensical, impossible to obtain world we have an 
>> autoregulating operating system that doesn't need any knobs.
>>
> 
> Some thinks are fundamental tradeoffs that can't be autotuned.
> 
> Latency vs throughput comes up in a lot of places, eg. timeslices.
> 
> Maximum throughput via effective use of swap, versus swapping as
> a last resort may be another.

As I said... it was ideal, nonsensical, and impossible. Doesn't sound 
like you're arguing with me.

Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Autotune swappiness01

[Nick Piggin]

Con Kolivas wrote:
> Nick Piggin wrote:
> 
>> Con Kolivas wrote:

>>> In my ideal, nonsensical, impossible to obtain world we have an 
>>> autoregulating operating system that doesn't need any knobs.
>>>
>>
>> Some thinks are fundamental tradeoffs that can't be autotuned.
>>
>> Latency vs throughput comes up in a lot of places, eg. timeslices.
>>
>> Maximum throughput via effective use of swap, versus swapping as
>> a last resort may be another.
> 
> 
> As I said... it was ideal, nonsensical, and impossible. Doesn't sound 
> like you're arguing with me.

No, you're right. My ideal operating system knows what the user
wants too ;)

Most of the time though, you are right. The quality/desirability of an
implementation will be inversely proportional to the number of knobs
sticking out of it (with bonus points for those that are meaningful to
2 people on the planet).

And yes, I think one knob should be enough for swapping behaviour too.

Re: Autotune swappiness01

[R. J. Wysocki]

On Monday 26 of July 2004 13:47, Nick Piggin wrote:
> Con Kolivas wrote:
> > Nick Piggin wrote:
> >> Con Kolivas wrote:
> >>> In my ideal, nonsensical, impossible to obtain world we have an
> >>> autoregulating operating system that doesn't need any knobs.
> >>
> >> Some thinks are fundamental tradeoffs that can't be autotuned.
> >>
> >> Latency vs throughput comes up in a lot of places, eg. timeslices.
> >>
> >> Maximum throughput via effective use of swap, versus swapping as
> >> a last resort may be another.
> >
> > As I said... it was ideal, nonsensical, and impossible. Doesn't sound
> > like you're arguing with me.
>
> No, you're right. My ideal operating system knows what the user
> wants too ;)

Well, what I hate about various computer programs is that they seem to assume 
to know what I (the USER) want and they don't let me do anything else that 
they "know" what I should/would do. ;-)

> Most of the time though, you are right. The quality/desirability of an
> implementation will be inversely proportional to the number of knobs
> sticking out of it (with bonus points for those that are meaningful to
> 2 people on the planet).

Can you please tell me why you think that the least tunable implementation 
should be the best/most desirable one?  I always prefer the most tunable 
implementations which is quite opposite to what you have said, but this is my 
personal opinion, of course.

Yours,
rjw

-- 
Rafael J. Wysocki
[tel. (+48) 605 053 693]
----------------------------
For a successful technology, reality must take precedence over public 
relations, for nature cannot be fooled.
					-- Richard P. Feynman

Re: Autotune swappiness01

[Adam Kropelin]

On Mon, Jul 26, 2004 at 03:53:09PM +0200, R. J. Wysocki wrote:
> On Monday 26 of July 2004 13:47, Nick Piggin wrote:
> > Con Kolivas wrote:
> > > Nick Piggin wrote:
> > >> Con Kolivas wrote:
> > >>> In my ideal, nonsensical, impossible to obtain world we have an
> > >>> autoregulating operating system that doesn't need any knobs.
> > >>
> > >> Some thinks are fundamental tradeoffs that can't be autotuned.
> > >>
> > >> Latency vs throughput comes up in a lot of places, eg. timeslices.
> > >>
> > >> Maximum throughput via effective use of swap, versus swapping as
> > >> a last resort may be another.
> > >
> > > As I said... it was ideal, nonsensical, and impossible. Doesn't sound
> > > like you're arguing with me.
> >
> > No, you're right. My ideal operating system knows what the user
> > wants too ;)
> 
> Well, what I hate about various computer programs is that they seem to assume 
> to know what I (the USER) want and they don't let me do anything else that 
> they "know" what I should/would do. ;-)
> 
> > Most of the time though, you are right. The quality/desirability of an
> > implementation will be inversely proportional to the number of knobs
> > sticking out of it (with bonus points for those that are meaningful to
> > 2 people on the planet).
> 
> Can you please tell me why you think that the least tunable implementation 
> should be the best/most desirable one?  I always prefer the most tunable 
> implementations which is quite opposite to what you have said, but this is my 
> personal opinion, of course.

The implementation with the least *need* for tuning is the most
desirable. I, for one, don't care if there are a dozen knobs as long as
99% of users don't have to touch them. But if common usage scenarios
require turning knobs to get reasonable performance, the algorithm is
lacking.

Thanks to fuel injection and engine management I can drive from LA to
Denver and not need to tweak my carburator half way up the Rockies.
I've given up some chances for tuning, but overall I'm better off. If 
you want to stick a trimpot or ten out the side of the engine management 
computer so true gearheads can tweak another couple HP or MPG out of the
engine, great. But don't expect me to fiddle with it every time driving
conditions change; it's not an excuse to make the management algorithms
inadequate for common driving patterns.

--Adam

Re: Autotune swappiness01

[R. J. Wysocki]

On Monday 26 of July 2004 20:45, Adam Kropelin wrote:
> On Mon, Jul 26, 2004 at 03:53:09PM +0200, R. J. Wysocki wrote:
> > On Monday 26 of July 2004 13:47, Nick Piggin wrote:
> > > Con Kolivas wrote:
> > > > Nick Piggin wrote:
> > > >> Con Kolivas wrote:
> > > >>> In my ideal, nonsensical, impossible to obtain world we have an
> > > >>> autoregulating operating system that doesn't need any knobs.
> > > >>
> > > >> Some thinks are fundamental tradeoffs that can't be autotuned.
> > > >>
> > > >> Latency vs throughput comes up in a lot of places, eg. timeslices.
> > > >>
> > > >> Maximum throughput via effective use of swap, versus swapping as
> > > >> a last resort may be another.
> > > >
> > > > As I said... it was ideal, nonsensical, and impossible. Doesn't sound
> > > > like you're arguing with me.
> > >
> > > No, you're right. My ideal operating system knows what the user
> > > wants too ;)
> >
> > Well, what I hate about various computer programs is that they seem to
> > assume to know what I (the USER) want and they don't let me do anything
> > else that they "know" what I should/would do. ;-)
> >
> > > Most of the time though, you are right. The quality/desirability of an
> > > implementation will be inversely proportional to the number of knobs
> > > sticking out of it (with bonus points for those that are meaningful to
> > > 2 people on the planet).
> >
> > Can you please tell me why you think that the least tunable
> > implementation should be the best/most desirable one?  I always prefer
> > the most tunable implementations which is quite opposite to what you have
> > said, but this is my personal opinion, of course.
>
> The implementation with the least *need* for tuning is the most
> desirable. I, for one, don't care if there are a dozen knobs as long as
> 99% of users don't have to touch them. But if common usage scenarios
> require turning knobs to get reasonable performance, the algorithm is
> lacking.

I agree in 100%.

> Thanks to fuel injection and engine management I can drive from LA to
> Denver and not need to tweak my carburator half way up the Rockies.
> I've given up some chances for tuning, but overall I'm better off. If
> you want to stick a trimpot or ten out the side of the engine management
> computer so true gearheads can tweak another couple HP or MPG out of the
> engine, great. But don't expect me to fiddle with it every time driving
> conditions change; it's not an excuse to make the management algorithms
> inadequate for common driving patterns.

I didn't mean that.  Actually, I was trying to say that an additional "knob" 
(or "knobs") might be useful in determining the "common settings" acceptable 
for the 99% of users.  Then, it could be "hidden" (which I wouldn't do, but 
well ...).

Yours,
rjw

-- 
Rafael J. Wysocki
[tel. (+48) 605 053 693]
----------------------------
For a successful technology, reality must take precedence over public 
relations, for nature cannot be fooled.
					-- Richard P. Feynman

Re: Autotune swappiness01

[Gerrit Huizenga]

On Mon, 26 Jul 2004 12:54:01 +0200, "R. J. Wysocki" wrote:
> On Monday 26 of July 2004 12:29, Con Kolivas wrote:
> > R. J. Wysocki wrote:
> > > Perhaps we need a bit more sophisticated swap algorithm than other OSes
> > > do. For example, couldn't we add an additional parameter to control the
> > > swapping "behavior", apart from the swappiness?  Something like adding
> > > the second knob in my radio example?  Just thinking,
> >
> > I think one knob is one knob too many already.
> 
> Can you please tell me why do you think so?

The more knobs there are, the higher the chances that most of them
are incorrectly set, no matter the workload.

gerrit

Re: Autotune swappiness01

[Joel Becker]

On Mon, Jul 26, 2004 at 10:43:01AM +1000, Con Kolivas wrote:
> Low memory boxes and ones that are heavily laden with applications find 
> that ends up making things slow down trying to keep all applications in 
> physical ram.

	Lowish memory boxes with plain desktop loads find that the default
of '60' is a terrible one (I'm speaking of 1GHz-ish machines with 256MB
(like mine) or 512MB (like a guy next to me)).  Every person I know who
installs 2.6 complains about how it feels slow and choppy.  I tell them
"The first thing I do after installing 2.6 is set swappiness to '20'."
Sure enough, they set swappiness to 20 and their box starts behaving
like a properly tuned one.
	I don't know what workload the default of '60' is for, but for
the (128MB < x < 1GB) of RAM case, it sucks (and I've seen the same
behavior on a 300MHz 196MB box).

Joel

-- 

"Maybe the time has drawn the faces I recall.
 But things in this life change very slowly,
 If they ever change at all."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: Autotune swappiness01

[Andrew Morton]

Joel Becker <Joel.Becker@oracle.com> wrote:
>
> On Mon, Jul 26, 2004 at 10:43:01AM +1000, Con Kolivas wrote:
> > Low memory boxes and ones that are heavily laden with applications find 
> > that ends up making things slow down trying to keep all applications in 
> > physical ram.
> 
> 	Lowish memory boxes with plain desktop loads find that the default
> of '60' is a terrible one (I'm speaking of 1GHz-ish machines with 256MB
> (like mine) or 512MB (like a guy next to me)).  Every person I know who
> installs 2.6 complains about how it feels slow and choppy.  I tell them
> "The first thing I do after installing 2.6 is set swappiness to '20'."
> Sure enough, they set swappiness to 20 and their box starts behaving
> like a properly tuned one.
> 	I don't know what workload the default of '60' is for, but for
> the (128MB < x < 1GB) of RAM case, it sucks (and I've seen the same
> behavior on a 300MHz 196MB box).
> 

Yes, I think 60% is about right for a 512-768M box.  Too high for the
smaller machines, too low for the larger ones.

More intelligent selection of the initial value is needed.

Re: Autotune swappiness01

[Con Kolivas]

Andrew Morton writes:

> Joel Becker <Joel.Becker@oracle.com> wrote:
>>
>> On Mon, Jul 26, 2004 at 10:43:01AM +1000, Con Kolivas wrote:
>> > Low memory boxes and ones that are heavily laden with applications find 
>> > that ends up making things slow down trying to keep all applications in 
>> > physical ram.
>> 
>> 	Lowish memory boxes with plain desktop loads find that the default
>> of '60' is a terrible one (I'm speaking of 1GHz-ish machines with 256MB
>> (like mine) or 512MB (like a guy next to me)).  Every person I know who
>> installs 2.6 complains about how it feels slow and choppy.  I tell them
>> "The first thing I do after installing 2.6 is set swappiness to '20'."
>> Sure enough, they set swappiness to 20 and their box starts behaving
>> like a properly tuned one.
>> 	I don't know what workload the default of '60' is for, but for
>> the (128MB < x < 1GB) of RAM case, it sucks (and I've seen the same
>> behavior on a 300MHz 196MB box).
>> 
> 
> Yes, I think 60% is about right for a 512-768M box.  Too high for the
> smaller machines, too low for the larger ones.

Sigh.. 

I have a 1Gb desktop machine that refuses to keep my applications in ram 
overnight if I have a swappiness higher than the default so I think lots of 
desktop users with more ram will be unhappy with higher settings.

> More intelligent selection of the initial value is needed.

Perhaps, but I really doubt desktop users running mainline would be happy 
about it going significantly higher. 

Cheers,
Con

Re: Autotune swappiness01

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Con Kolivas wrote:
| Andrew Morton writes:

|> Yes, I think 60% is about right for a 512-768M box.  Too high for the
|> smaller machines, too low for the larger ones.
|
|
| Sigh..
| I have a 1Gb desktop machine that refuses to keep my applications in ram
| overnight if I have a swappiness higher than the default so I think lots
| of desktop users with more ram will be unhappy with higher settings.

I have 1 GB and I had a setting of 51 (seemed to be perhaps gentoo
default or so) and I especially after a weekend (2 days off) it is
always the "monday-morning-swap-hell" where I have to wait 5min until he
swapped in the apps he swapped out during weekend.

I changed that to 20 now, but I don't know if this will make things
worse or better.

|> More intelligent selection of the initial value is needed.
|
| Perhaps, but I really doubt desktop users running mainline would be
| happy about it going significantly higher.

no, but an already diskussed feature that would know, not to swap out
much used apps (eg mozilla which is used all the time, or openoffice)
during night or long idle times, if not _really_ needed.

- --
Clemens Schwaighofer - IT Engineer & System Administration
==========================================================
TEQUILA\Japan, 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN
Tel: +81-(0)3-3545-7703            Fax: +81-(0)3-3545-7343
http://www.tequila.co.jp
==========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBBadgjBz/yQjBxz8RAkj5AJ9527iKCfaJyWI4S8cDvKCIcyC7ZACgyqel
txTVMqqVJUuargYPnX8Fvyw=
=gmPf
-----END PGP SIGNATURE-----

Re: Autotune swappiness01

[Andrew Morton]

Clemens Schwaighofer <cs@tequila.co.jp> wrote:
>
> Con Kolivas wrote:
> | Andrew Morton writes:
> 
> |> Yes, I think 60% is about right for a 512-768M box.  Too high for the
> |> smaller machines, too low for the larger ones.
> |
> |
> | Sigh..
> | I have a 1Gb desktop machine that refuses to keep my applications in ram
> | overnight if I have a swappiness higher than the default so I think lots
> | of desktop users with more ram will be unhappy with higher settings.
> 
> I have 1 GB and I had a setting of 51 (seemed to be perhaps gentoo
> default or so) and I especially after a weekend (2 days off) it is
> always the "monday-morning-swap-hell" where I have to wait 5min until he
> swapped in the apps he swapped out during weekend.
> 
> I changed that to 20 now, but I don't know if this will make things
> worse or better.
> 

It may appear to be better, but you now have 100, maybe 200 megabytes less
pagecache available across the entire working day.

Re: Autotune swappiness01

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Morton wrote:
| Clemens Schwaighofer <cs@tequila.co.jp> wrote:

|>I have 1 GB and I had a setting of 51 (seemed to be perhaps gentoo
|>default or so) and I especially after a weekend (2 days off) it is
|>always the "monday-morning-swap-hell" where I have to wait 5min until he
|>swapped in the apps he swapped out during weekend.
|>
|>I changed that to 20 now, but I don't know if this will make things
|>worse or better.
|
| It may appear to be better, but you now have 100, maybe 200 megabytes less
| pagecache available across the entire working day.

which might slow down overall working speed? or responsness of programs?

- --
Clemens Schwaighofer - IT Engineer & System Administration
==========================================================
TEQUILA\Japan, 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN
Tel: +81-(0)3-3545-7703            Fax: +81-(0)3-3545-7343
http://www.tequila.co.jp
==========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBBa0bjBz/yQjBxz8RAmcQAJ4ieXmCIBpmeWQ1+kdVEPI0te7ezQCgppxG
AFA1t88+WofCN2WbjmpkZ7A=
=JVwT
-----END PGP SIGNATURE-----

Re: Autotune swappiness01

[Tim Connors]

Clemens Schwaighofer <cs@tequila.co.jp> said on Tue, 27 Jul 2004 10:17:16 +0900:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Andrew Morton wrote:
> | Clemens Schwaighofer <cs@tequila.co.jp> wrote:
> 
> |>I have 1 GB and I had a setting of 51 (seemed to be perhaps gentoo
> |>default or so) and I especially after a weekend (2 days off) it is
> |>always the "monday-morning-swap-hell" where I have to wait 5min until he
> |>swapped in the apps he swapped out during weekend.
> |>
> |>I changed that to 20 now, but I don't know if this will make things
> |>worse or better.
> |
> | It may appear to be better, but you now have 100, maybe 200 megabytes less
> | pagecache available across the entire working day.
> 
> which might slow down overall working speed? or responsness of programs?

Depends on what you do. Do you compile kernels regularly? In
particular, do you have to wait for them, or do you just let them sit
in the background, and come back to them when you rememeber, since
you've been busy doing real work for the past 5 hours? If you wait,
then I guess you want high swapiness.

For the rest of us who don't have to regularly read in hundreds of
megs of disk, and don't need to use that hundreds of megs of disk over
and over and over again (As far as I can see, this is just about
everyone who's not a kernel developer or some big app developer[1]), I
guess we get by just fine having smaller swapiness.

[1] Maybe kde is bloated enough that you if want to start the equiv of
an xterm all the time, maybe caching helps a lot there, but I make a
point of using lean apps[2].

[2] Sad when you consider xemacs lean, isn't it? ;)

-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GS d- s:- a-- C+++(++++) UL(SOBI)+++(++++) P+++ L+++ E++(----)
W++(--) N+++ o K+++ w---(++) O- M--(+) V PS++ PE-(--) Y PGP t->+
!5 X R? tv- b- DI+ D--- G e++>++++ h* r(--) !y+
------END GEEK CODE BLOCK------

Re: Autotune swappiness01

[Con Kolivas]

Tim Connors writes:

> Clemens Schwaighofer <cs@tequila.co.jp> said on Tue, 27 Jul 2004 10:17:16 +0900:
>> 
>> Andrew Morton wrote:
>> | Clemens Schwaighofer <cs@tequila.co.jp> wrote:
>> 
>> |>
>> |>I changed that to 20 now, but I don't know if this will make things
>> |>worse or better.
>> |
>> | It may appear to be better, but you now have 100, maybe 200 megabytes less
>> | pagecache available across the entire working day.
>> 
>> which might slow down overall working speed? or responsness of programs?
> 
> Depends on what you do. Do you compile kernels regularly? In
> particular, do you have to wait for them, or do you just let them sit
> in the background, and come back to them when you rememeber, since
> you've been busy doing real work for the past 5 hours? If you wait,
> then I guess you want high swapiness.

Well I'm tired of this discussion which comes up every month or so and I 
brought it up! Clearly my patch is not considered adequate so I promise 
never to bring it up again.

Cheers,
Con

Re: Autotune swappiness01

[Tim Connors]

On Tue, 27 Jul 2004, Con Kolivas wrote:

> Tim Connors writes:
>
> > Clemens Schwaighofer <cs@tequila.co.jp> said on Tue, 27 Jul 2004 10:17:16 +0900:
> >>
> >> Andrew Morton wrote:
> >> | It may appear to be better, but you now have 100, maybe 200 megabytes less
> >> | pagecache available across the entire working day.
> >>
> >> which might slow down overall working speed? or responsness of programs?
> >
> > Depends on what you do. Do you compile kernels regularly? In
> > particular, do you have to wait for them, or do you just let them sit
> > in the background, and come back to them when you rememeber, since
> > you've been busy doing real work for the past 5 hours? If you wait,
> > then I guess you want high swapiness.
>
> Well I'm tired of this discussion which comes up every month or so and I
> brought it up! Clearly my patch is not considered adequate so I promise
> never to bring it up again.

I am not trying to say anything bad about your work - I am trying to
caution Andrew that not everyone cares so much that they lose 200 megs of
pagecache. It doesn't affect everyone equally - I'm just trying to put the
voice of those of us who care more about responsiveness than throughput,
if I may borrow the argument from upthread.

I often fear that a lot of benchmarking is done one-sided, and does not
reflect the usage patterns of a typical Linux user.

Then there's the relative importance of subjective "feel", vs hard
numbers. Maybe some people prefer that their system feels nicer, even if
it doesn't compile kernels so fast. And I think this decision is best left
to at least one of those knobs (hey, all you know, people might even
adjust the knob in a crontab).

-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Usage: fortune -P [-f] -a [xsz] Q: file [rKe9] -v6[+] file1 ...

Re: Autotune swappiness01

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim Connors wrote:
| On Tue, 27 Jul 2004, Con Kolivas wrote:

| Then there's the relative importance of subjective "feel", vs hard
| numbers. Maybe some people prefer that their system feels nicer, even if
| it doesn't compile kernels so fast. And I think this decision is best left
| to at least one of those knobs (hey, all you know, people might even
| adjust the knob in a crontab).

I think, the more linux users are out there, the less is the percentage
who compiles kernels often. And often means once or twice a day. Most
people use stock kernels. I think a "knob" that says "compile a lot" or
"normal" or "server" or so would be nice.

Especially because a server has very different need of swap, etc than a
workstation.

- --
Clemens Schwaighofer - IT Engineer & System Administration
==========================================================
TEQUILA\Japan, 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN
Tel: +81-(0)3-3545-7703            Fax: +81-(0)3-3545-7343
http://www.tequila.co.jp
==========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBBc9qjBz/yQjBxz8RAqrJAKDkBo7ZnYzWOqdXABRUPYUoO1ooSACffSnk
uTERz/dQaEV1lSUsCEdsRhk=
=EBxd
-----END PGP SIGNATURE-----

Re: Autotune swappiness01

[Joel Becker]

On Tue, Jul 27, 2004 at 01:02:32PM +1000, Tim Connors wrote:
> I am not trying to say anything bad about your work - I am trying to
> caution Andrew that not everyone cares so much that they lose 200 megs of
> pagecache. It doesn't affect everyone equally - I'm just trying to put the
> voice of those of us who care more about responsiveness than throughput,
> if I may borrow the argument from upthread.

	I happen to be a person who rolls his eyes at everyone's mention
of micro-optimized "feel".  I've found that any system faster than
300MHz is pretty decent for normal desktop work (that is, moz + lots of
terminals in gnome/kde).  Yes, I'm a luddite, I used to wait 45 seconds
for moz to start in the morning on the 300Mhz.  I survived.
	In general, I can't notice the difference between 2.6.anything
on my 1GHz.  Maybe everyone else can, but I can't.
	HOWEVER, the swappiness of '60' puts my system into
fits-and-starts mode.  Not "It feels slower", but "It pauses for seconds
at a time."  So I chimed in on this.
	And yes, I'd give up oodles of pagecache to avoid fits and
starts.  But there's got to be a way to use the pagecache and not hang
for seconds at a time.

Joel

-- 

"I don't want to achieve immortality through my work; I want to
 achieve immortality through not dying."
        - Woody Allen

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: Autotune swappiness01

[William Lee Irwin III]

On Mon, Jul 26, 2004 at 08:47:39PM -0700, Joel Becker wrote:
> 	I happen to be a person who rolls his eyes at everyone's mention
> of micro-optimized "feel".  I've found that any system faster than
> 300MHz is pretty decent for normal desktop work (that is, moz + lots of
> terminals in gnome/kde).  Yes, I'm a luddite, I used to wait 45 seconds
> for moz to start in the morning on the 300Mhz.  I survived.
> 	In general, I can't notice the difference between 2.6.anything
> on my 1GHz.  Maybe everyone else can, but I can't.
> 	HOWEVER, the swappiness of '60' puts my system into
> fits-and-starts mode.  Not "It feels slower", but "It pauses for seconds
> at a time."  So I chimed in on this.
> 	And yes, I'd give up oodles of pagecache to avoid fits and
> starts.  But there's got to be a way to use the pagecache and not hang
> for seconds at a time.

I've had similar experiences except for the pauses. Could you identify
this as idle time, iowait, or cpu time (user/kernel)?

Also, does this behavior change at all as the IO scheduler varies? And
could you describe the system, e.g. IO devices/etc.?


-- wli

Re: Autotune swappiness01

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim Connors wrote:
| Clemens Schwaighofer <cs@tequila.co.jp> said on Tue, 27 Jul 2004
10:17:16 +0900:

| Depends on what you do. Do you compile kernels regularly? In
| particular, do you have to wait for them, or do you just let them sit
| in the background, and come back to them when you rememeber, since
| you've been busy doing real work for the past 5 hours? If you wait,
| then I guess you want high swapiness.

well this is a work box. I do coding, yes, but this is perl, php, etc,
mostly (99%) on remote machines. The big apps running here is mozilla,
openoffice, korganzier (yes I do the KDE bloat) and _a_ _lot_ of konsole
~ (kde terminals).

I want responsivness. I don't want to wait 10s to switch between virtual
desktops. I compile a kernel once in a while. Very rare actually,
because this is a work box and I don't reboot all the time (can't &
don't want). The last kernel change was 64d ago.

| For the rest of us who don't have to regularly read in hundreds of
| megs of disk, and don't need to use that hundreds of megs of disk over
| and over and over again (As far as I can see, this is just about
| everyone who's not a kernel developer or some big app developer[1]), I
| guess we get by just fine having smaller swapiness.

well the only thing that needs disk over and over again is this _crappy_
gentoo because it needs to compile everything. I really hate myself for
using it, because _this_ probably screws up _all_ the page cache, etc.

| [1] Maybe kde is bloated enough that you if want to start the equiv of
| an xterm all the time, maybe caching helps a lot there, but I make a
| point of using lean apps[2].

well, to be honest, I doubt it is so bloated anymore. I think gnome is
the better bloat of those two.

| [2] Sad when you consider xemacs lean, isn't it? ;)

well, if you call xemacs (the editor without an OS) lean ... *hmm* but I
am a vim man after all ;)

- --
Clemens Schwaighofer - IT Engineer & System Administration
==========================================================
TEQUILA\Japan, 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN
Tel: +81-(0)3-3545-7703            Fax: +81-(0)3-3545-7343
http://www.tequila.co.jp
==========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBBc7ejBz/yQjBxz8RAqYwAJoDCBWOxMRXAHHVelV+M9ObQjpsYQCg6a8n
GIRF1+LjDZ+U4TCovnk+RcA=
=K1Xc
-----END PGP SIGNATURE-----