RE: vm philosophising

RE: vm philosophising

[David Luyer]

I wrote:
> Alan Cox wrote:
> > > There is another VM that has a property that people would like:
> > > deterministically handling memory exhaustion. 
> > > Unfortunately, that VM
> > > probably can't co-exist with over-commit and the 
> > > performance gains that
> > > affords.
> > 
> > It can definitely co-exist. Overcommit control is just a 
> > book keeping
> > exercise on address space commits.

[...]

and the comment I somehow missed putting on the end:

If you want to philosophise about VM strategies, think of
overcommit as "ethernet" and precommit as "token ring".

David.

Re: vm philosophising

[David Weinehall]

On Sat, Jan 19, 2002 at 03:49:02PM +1100, David Luyer wrote:
> I wrote:
> > Alan Cox wrote:
> > > > There is another VM that has a property that people would like:
> > > > deterministically handling memory exhaustion. 
> > > > Unfortunately, that VM
> > > > probably can't co-exist with over-commit and the 
> > > > performance gains that
> > > > affords.
> > > 
> > > It can definitely co-exist. Overcommit control is just a 
> > > book keeping
> > > exercise on address space commits.
> 
> [...]
> 
> and the comment I somehow missed putting on the end:
> 
> If you want to philosophise about VM strategies, think of
> overcommit as "ethernet" and precommit as "token ring".

You mean, that while technically superior, precommit suffers from
a topological problem and the fact that a very expensive concentrator
is needed?! ;-)

Token Ring still lives in the spirit, though it's called FDDI
nowadays...


/David
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

Re: vm philosophising

[Bill Davidsen]

On Thu, 17 Jan 2002, Matthew Johnson wrote:

> How does one test the VM precisley? Sorry for the ignorance on this subject.

Having been active doing just that, I have been triggering my favorite bad
behaviour and trying to evaluate which (if either) makes the system run
better, as defined by both measurements and "feel."

The two problems I have been seeing are (1) load with low to moderate
memory, and (2) sudden i/o bursts freezing the system when doing large
writes (CD image creation).

My test for #1 is simple, I compile the 2.4.16 stock kernel after booting
with mem=64m or mem=128m options. I have a batch of files to hack into
something I can post, and I ran on an Athlon 1400 and dual Celeron 500
system, so I have moderate UP and SMP machines of similar performance when
full memory is used. I compile with:
  make clean; make dep
  make bzImage modules MAKE='make -j7'

I took all these numbers with the intent of posting, but the runs finished
at 0630 this morning and I haven't the time yet. Gut feeling is that
17-rmap-11c works better under small memory, 18pre2aa2 was better when
creating CD images on the Athlon, ran out of time for the SMP.

Neither crashed, hung, or caused the OOM to commit procedural genocide,
which plain 2.4.17 does. the -aa kernel was also tested with my own patch
for intermediate disk loads, I will post when I'm sure it's actually
better by enough to matter. I believe the extra tuning in -aa allows
better large i/o performance if you match bdflush to your load.

Chech large i/o by sync() followed by a fast CD build from wav files, on
fast disk. When the disk light come on hard, grab a window and try to wave
it around, or change X virtual desktops. Chances are that you will get bad
to nil response if you have fast disk and CPU. I get a whole 600MB in
memory before the light comes on :-(

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: vm philosophising

[Mike Harrold]

> 
> * Tommy Faasen (faasen@xs4all.nl) [020118 08:47]:
> > 2-DBMS: 1 or 2 big programs which sometimes even do their own
> > memory management.Fragmentation and latency isn't issue here I
> > think however moving ltos of data to and from swap is.
> 
> A lot of times a DBMS is bulit that way because they assume they
> know better than the OS designer how memory should be managed.  Same
> reason they usually use raw writting the the drive instead of using
> the OS calls.

Actually this isn't true. DBMS usually handle their own memory because
everything is done in blocks of the same size. Since this is configured
as part of the DBMS' parameters, it is much better at handling this
than the OS ever could be, once it has garnered the original memory
from the OS. Remember, this space is normally shared memory as well.

As for the FS, DBMS' prefer raw devices for consistency issues (as
well as speed). Raw devices prevent the use of the kernel's internal
buffers for files (thus reducing the number of memory copies involved).

/Mike

Re: vm philosophising

[The Doctor What]

* Tommy Faasen (faasen@xs4all.nl) [020118 08:47]:
> 2-DBMS: 1 or 2 big programs which sometimes even do their own
> memory management.Fragmentation and latency isn't issue here I
> think however moving ltos of data to and from swap is.

A lot of times a DBMS is bulit that way because they assume they
know better than the OS designer how memory should be managed.  Same
reason they usually use raw writting the the drive instead of using
the OS calls.

Is this right or fair?  I don't know.   But it does imply that if a
VM or FS layer for an OS performs well enough, that a DBM system
might be built that would be built to take advantage of the OS's VM
and FS layer.

Ciao!

-- 
"When you have to shoot, shoot! Don't talk."
		--Tuco (The Good, The Bad, and The Ugly)

The Doctor What: Kaboom!                         http://docwhat.gerf.org/
docwhat@gerf.org                                                   KF6VNC

Re: vm philosophising

[Stevie O]

Why don't we all follow the MSCommit method of VM? We simply allocate 99% 
of physical RAM for cache and other non-userspace purposes, and whenever an 
application needs memory, pop up a message:
         printk("Your system is out of virtual memory. Linux is increasing 
your virtual memory size. During this time, memory allocation requests may 
fail.\n");

Then spend a few minutes doing hard disk I/O, while exposing bugs in 
programs that don't check to make sure that malloc succeeded.


;)



--
Stevie-O

Real programmers use COPY CON PROGRAM.EXE

Re: vm philosophising

[Rob Landley]

On Friday 18 January 2002 11:42 pm, David Luyer wrote:

> And while precommit may be something people ask for, I'd have to say
> many
> of them would, having experienced the difference on identical hardware,
> then realise what a bad idea it was and go back to the current mode.
> That is, it sounds like a big waste of time to implement the
> 'traditional'
> behaviour which Linux is already so much better than.
>
> David.

Precommit basically just asks the application to die when it first allocates 
memory if it's even possible for it to die in the most pathlogical usage case 
of that memory.

I.E. "die up front" instead of "die while running".  (There's no possible way 
this can improve performance.  If you want to never swap, just don't mount a 
swap partition.)  You don't even have to change the VM's behavior, it can 
still copy on write and such.  You just add a test to cause allocations to 
fail unnecessarily at times.

Throwing in a little extra code on mmaps and allocations to kill a process 
wouldn't be too hard.  It would be stupid outside of something like a 
financial transaction system (and probably even in there), but it technically 
shouldn't be all that hard to do.

Unless I missed something...?

Rob

RE: vm philosophising

[David Luyer]

Alan Cox wrote:
> > There is another VM that has a property that people would like:
> > deterministically handling memory exhaustion. Unfortunately, that VM
> > probably can't co-exist with over-commit and the 
> performance gains that
> > affords.
> 
> It can definitely co-exist. Overcommit control is just a book keeping
> exercise on address space commits.

And that's a _definitely_; other OS's have done it.  Digital Unix, for
one,
on the basis of a file called 'swapdefault', swapped between overcommit
and precommit modes.  I was rather disappointed when I first tried to
enable overcommit mode on Solaris 2.x (where 'x' was probably somewhere
around 4) and searched for quite some time before giving up and deciding
it wasn't a tunable option...

Although I've never actually deliberately run a system in precommit
mode,
it always used to be the first thing to "fix" on a Digital Unix box, and
when I discovered Solaris had the same "flaw" my suggestion was to move
the affected applications (large applications which fork before exec'ing
or fork short lived-children, at around 1/2Gb+ each, which should just
be short-lived COW shared mappings and not exhaust memory) to Linux.

And while precommit may be something people ask for, I'd have to say
many
of them would, having experienced the difference on identical hardware,
then realise what a bad idea it was and go back to the current mode.
That is, it sounds like a big waste of time to implement the
'traditional'
behaviour which Linux is already so much better than.

David.
--
David Luyer                                     Phone:   +61 3 9674 7525
Network Manager                P A C I F I C    Fax:     +61 3 9699 8693
Pacific Internet (Australia)  I N T E R N E T   Mobile:  +61 4 1111 BYTE
http://www.pacific.net.au/                      NASDAQ:  PCNTF

Re: vm philosophising

[Alan Cox]

> On Fri, 18 Jan 2002 19:23:47 +0000 (GMT), Alan Cox wrote:
> 
> >Overcommit control is just a book keeping
> >exercise on address space commits.
> 
> 	A bookkeeping technique developed by Arthur Anderson.

Hardly, and for many workloads its actually a very good thing to do. Of
course there is always a small mostly theoretical risk of doing an Enron

Re: vm philosophising

[David Schwartz]

On Fri, 18 Jan 2002 19:23:47 +0000 (GMT), Alan Cox wrote:

>Overcommit control is just a book keeping
>exercise on address space commits.

	A bookkeeping technique developed by Arthur Anderson.

	DS

Re: vm philosophising

[Alan Cox]

> There is another VM that has a property that people would like:
> deterministically handling memory exhaustion. Unfortunately, that VM
> probably can't co-exist with over-commit and the performance gains that
> affords.

It can definitely co-exist. Overcommit control is just a book keeping
exercise on address space commits.

Re: vm philosophising

[Oliver Xymoron]

On Fri, 18 Jan 2002, Rik van Riel wrote:

> On Fri, 18 Jan 2002, Bosko Radivojevic wrote:
>
> > There is no way to make one good VM for all possible situations. But,
> > you can tune/make one VM to work great on large DBMS (e.g.) and
> > tune/make another one to work great on ordinary desktop systems
>
> This is an interesting assertion ... but up to date nobody has
> been able to tell me what exactly should be different between
> these two mythical VMs ;)

There is another VM that has a property that people would like:
deterministically handling memory exhaustion. Unfortunately, that VM
probably can't co-exist with over-commit and the performance gains that
affords.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: vm philosophising

[Mr. Shannon Aldinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 18 Jan 2002, Rik van Riel wrote:

> On Fri, 18 Jan 2002, Bosko Radivojevic wrote:
>
> > There is no way to make one good VM for all possible situations. But,
> > you can tune/make one VM to work great on large DBMS (e.g.) and
> > tune/make another one to work great on ordinary desktop systems
>
> This is an interesting assertion ... but up to date nobody has
> been able to tell me what exactly should be different between
> these two mythical VMs ;)
>
I can see two different "VMs". I say "VMs" because it could be the same
code with different magic numbers to control its behavior.

>From a file & database point of view throughput is the most crictical
aspect. Both disk and network throughput. Interactive response on such
systems isn't as critical as most of the time it will sit there processing
queries or sending files.

>From a desktop point of view interactive response is critical, however
disk and network throughput also have to have a fine balance. Maybe the
balance is three way here between interactive response, disk throughput
and network throughput.

Perhaps having a VM system that you select your main focus server vs
desktop would be the way to go. Also the end-user should be able to adjust
this balance. Say a person selected desktop, and is a graphic artist, they
may not care as much about network thoroughput and rather push up
interactive response and disk throughput at the expense of the network
thoroughput.

Regards.
PS: IANAVMP (I Am Not A VM Programmer)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjxIRZAACgkQwtU6L/A4vVDUTQCdG4Pg4hYGPvRXN9kBVfDyWBbD
bnsAnigMlPA21izLJUhKjZcTeeaaK9IC
=EKri
-----END PGP SIGNATURE-----

Re: vm philosophising

[listmail]

On Fri, 18 Jan 2002, Tommy Faasen wrote:


> I have no clue about VM's but I can imagine that for example the following situations have different requirements:
> 1-Desktop: many "small" apps, I believe exe's remain in memory and data is written to disk? Anyway I can imagine fragmentation and latency is an issue here.
> 2-DBMS: 1 or 2 big programs which sometimes even do their own memory management.Fragmentation and latency isn't issue here I think however moving ltos of data to and from swap is.
> 3-Webserver: for example apache with many childs being created under high load and killed under low load. The data is always small (in case of static pages). So a lot of small swaps? Latency is not as much as un issue but I can imagine that fragmentation can be an issue?

There is another situation to consider which I think more typical in the
"Real World"

4 - The Typical Web server Environment, in many many companies that I have
seen.
  The machine runs both the DBMS(mysql, Oracle, etc) and the web server
Apache.  Therefore a balence of memory use needs to be struck between the
two applications.  Then ususally the system also has a development
environment at the ready incase changes need to be made quickly from the
console.  While this may not be true as things scale up for a business and
functions get separated to different machines.  Many startups and small
businesses that I have worked with that have turned to LINUX for the OS
becuase the TCO, it offers over the other guys, tend to have just that
single does everything box.

-Bill

Re: vm philosophising

[Tommy Faasen]

On Fri, Jan 18, 2002 at 02:36:02AM -0200, Rik van Riel wrote:
> On Fri, 18 Jan 2002, Bosko Radivojevic wrote:
> 
> > There is no way to make one good VM for all possible situations. But,
> > you can tune/make one VM to work great on large DBMS (e.g.) and
> > tune/make another one to work great on ordinary desktop systems
> 
> This is an interesting assertion ... but up to date nobody has
> been able to tell me what exactly should be different between
> these two mythical VMs ;)
> 
I have no clue about VM's but I can imagine that for example the following situations have different requirements:
1-Desktop: many "small" apps, I believe exe's remain in memory and data is written to disk? Anyway I can imagine fragmentation and latency is an issue here.
2-DBMS: 1 or 2 big programs which sometimes even do their own memory management.Fragmentation and latency isn't issue here I think however moving ltos of data to and from swap is.
3-Webserver: for example apache with many childs being created under high load and killed under low load. The data is always small (in case of static pages). So a lot of small swaps? Latency is not as much as un issue but I can imagine that fragmentation can be an issue?

I think these 3 situations behave very differently, but then again it's just what I think. I can also imagine that more situations are possible but not many.
I also indictated that we have a few parameters we can optimise for like latency, fragmentation and moving a lot small chunks, or occasionally 1 big chunk.

I know from an AI perspective that optimize for 3 different parameters is difficult.
> regards,
> 
> Rik
> -- 
> "Linux holds advantages over the single-vendor commercial OS"
>     -- Microsoft's "Competing with Linux" document
> 
> http://www.surriel.com/		http://distro.conectiva.com/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: vm philosophising

[Matthew Johnson]

On Thursday 17 January 2002 09:18 pm, Ryan Cumming wrote:
> On January 17, 2002 20:58, Matthew Johnson wrote:
> > Well what is the different requirements for desktop use versus, the
> > server in terms of virtual memory? What I am doing on my system with
> > MP3's playing, running Xfree86 on a SuSE 7.3 system will be different
> > from a server running DMBS's.

Slight typo In last line, should read "What am I" not "What I am" perhaps 
this lead to some confusion. Shouldn't write to this list after driving for 
almost 12 hours :).

Matt

Re: vm philosophising

[Matthew Johnson]

> So, your argument basically boils down to "the differences will be that
> it's, well, different." Please do us a favour and never join a debating
> team of any sort.
>

Did you get out of the wrong side of the bed or something? Geez, try and help 
then get flamed by a troll for your efforts...Plus the fact I was not even 
arguing! I was asking questions...

Further, I was postulating to see what the differences (or indeed 
similarities) are between the VM's to cope with desktop use vs server use. 
No-one seems to know right now.

How does one test the VM precisley? Sorry for the ignorance on this subject.
 
Kind regards,

Matt
PS If you're going to reply with a childish, unproffessional remark do us all 
a favor and be quiet. If I say something stupid and nobody replies I get the 
hint usually.

Re: vm philosophising

[Ryan Cumming]

On January 17, 2002 20:58, Matthew Johnson wrote:
> Well what is the different requirements for desktop use versus, the server
> in terms of virtual memory? What I am doing on my system with MP3's
> playing, running Xfree86 on a SuSE 7.3 system will be different from a
> server running DMBS's.

So, your argument basically boils down to "the differences will be that it's, 
well, different." Please do us a favour and never join a debating team of any 
sort.

-Ryan

Re: vm philosophising

[Rik van Riel]

On Thu, 17 Jan 2002, Matthew Johnson wrote:

> > This is an interesting assertion ... but up to date nobody has
> > been able to tell me what exactly should be different between
> > these two mythical VMs ;)
>
> Well what is the different requirements for desktop use versus, the
> server in terms of virtual memory?

That's a good question ....

> What I am doing on my system with MP3's playing, running Xfree86 on a
> SuSE 7.3 system will be different from a server running DMBS's.

.... especially because I haven't seen any suggestion on
how these different workloads would get translated into
different VM requirements.

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: vm philosophising

[Matthew Johnson]

> This is an interesting assertion ... but up to date nobody has
> been able to tell me what exactly should be different between
> these two mythical VMs ;)
>

Well what is the different requirements for desktop use versus, the server in 
terms of virtual memory? What I am doing on my system with MP3's playing, 
running Xfree86 on a SuSE 7.3 system will be different from a server running 
DMBS's.

Another issue, would it be possible to select one or the other VM's like you 
do for example the CPU type? 

Kind regards,

Mayy

vm philosophising

[Rik van Riel]

On Fri, 18 Jan 2002, Bosko Radivojevic wrote:

> There is no way to make one good VM for all possible situations. But,
> you can tune/make one VM to work great on large DBMS (e.g.) and
> tune/make another one to work great on ordinary desktop systems

This is an interesting assertion ... but up to date nobody has
been able to tell me what exactly should be different between
these two mythical VMs ;)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/