Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[shane]

Hi,

I too can recommend the -aa kernels. I have been using the -aa kernels
since 2.4.10, and IMHO they are just as stable if not more stable than
the vanilla kernels. The difference is in the performance, especially
the swapping as Andrea noted. Specifically they are better at I/O, both
throughput and interactive responsiveness during heavy I/O. I have also
seen the best utilization on my little 100/switched network using these
kernels. I also use -aa on my Thinkpad and it works great there too. I
haven't tried the XFS but I did try tux and it just smokes!

Anyways just my 2 cents,

Shane

BTW: Big thanks to Andrew Morton for stepping up and splitting the VM
pieces for merging.  

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

On Thu, Apr 11, 2002 at 04:50:15PM -0700, Aviv Shavit wrote:
> This goes back to my previous post - 
> Applying only the vm patches didn't get me far.
> 
> I'm still trying to pin point what it is thats helping
> me out in -aa

For the level of cache during your workload (you mentioned that variable
in the previous email) what matters mostly is the vm-33 patch. Do you
get significantly different levels of cache with only the vm-33 patch
compared to the whole latest -aa? (there are other variables too that
could influence the level of cache, the readahead boost for example, but
they're much less likely to influence the cache levels than the vm-33
patch)

It maybe the benefit you see is not only in the VM part, but it could
came also the dozen of other fixes and improvements. For example
starting from the lowlatency fixes from Andrew (note: _fixes_) to highio
(from Jens) if you've highmem, to the dyn-sched (from Davide) if you've
tons of sleeping tasks or interactive processes etc...

The reason I maintain the main patches like the vm one also against
mainline, is exactly to address Ken's concern about being able to apply
just one patch if he's not confortable with the whole patchkit, and
secondly to be able to test it separately without the pollution. I feel
the vm patch is one of the most important and that's why I mentioned it
in particular, but the lowlatency fixes and lots of other stuff is
important too. But I've also the feeling the other stuff [modulo the
major things like pte-highmem and highio that at least affects only the
x86 high-end and not that much desktops or little server] is much much
easier to get integrated and that's why I worry much less about it.

Thanks for all the feedback and testing!

Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Aviv Shavit]

This goes back to my previous post - 
Applying only the vm patches didn't get me far.

I'm still trying to pin point what it is thats helping
me out in -aa

--- Ken Brownfield <ken@irridia.com> wrote: 
:
:
> 
> How much of the improved behavior that you're seeing
> is due to the vm-33
> tweaks and not pte-highmem, block-highmem, or any of
> the 100 or so other
> 2.4.19-pre6aa1 patches?
:
:
> 
> What I'd like to hear (and what I suspect many
> admins trying to get
> higher-end hardware working optimally in a
> production environment would
> like to hear) is what specific patches applied to
> mainline are needed to
> correct the current VM and I/O issues in the 2.4
> tree?
> 
> If it's vm, pte-highmem, and block-highmem, that's
> fine -- and separable
> from -aa.  Otherwise it's difficult to get people to
> test, use, and
> provide feedback that isn't polluted by unnecessary
> variables.
> 
> Thanks,
> -- 
> Ken.
> ken@irridia.com
> 



__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Ken Brownfield]

This sounds great, but I still have concerns with using -aa, or subsets
of same.

How much of the improved behavior that you're seeing is due to the vm-33
tweaks and not pte-highmem, block-highmem, or any of the 100 or so other
2.4.19-pre6aa1 patches?

For production use, I prefer to divert from mainline only with my
specific needs (or trivial fixes).  Using -aa would introduce a large
array of (to me) unknowns.  How many of the -aa patches are "ready" for
mainline and production?  vm is currently being debated on the floor --
but what about pte-highmem and block-highmem?  How many of the other
patches are as widely tested as the vm patch?  For some reason, applying
a patch called "00_readahead-got-broken-somewhere-1" doesn't give me the
utmost confidence in production.  Call it a failed bag check.

While 2.4.x is a stable kernel, it needs to be a working* kernel until
2.5 can sort out these and the vast array of other issues.  IMHO.
*Admittedly, "working" in this case only applies to larger servers, but
it would be quite tragic to delay the spread of Linux to hardware that's
been available and used in production for _years_.  Maybe 5% of the
installed base has relevant hardware, but the benefit to Linux _far_
outstrips that seemingly anemic number.  I've probably rehashed that
point too much as it is, but...

What I'd like to hear (and what I suspect many admins trying to get
higher-end hardware working optimally in a production environment would
like to hear) is what specific patches applied to mainline are needed to
correct the current VM and I/O issues in the 2.4 tree?

If it's vm, pte-highmem, and block-highmem, that's fine -- and separable
from -aa.  Otherwise it's difficult to get people to test, use, and
provide feedback that isn't polluted by unnecessary variables.

Thanks,
-- 
Ken.
ken@irridia.com


On Wed, Apr 10, 2002 at 01:36:09AM +0200, Andrea Arcangeli wrote:
| I recommend everybody to never use a 2.4 kernel without first applying
| this vm patch:
| 
| 	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/vm-33.gz
| 
| It applies cleanly to both 2.4.19pre5 and 2.4.19pre6. Andrew splitted it
| into orthogonal pieces for easy merging from Marcelo's side (modulo
| -rest that is important too but that it's still quite monolithic, but
| it's pointless to invest further effort at this time until we are
| certain Marcelo will do its job and eventually merge it in mainline):
| 
| 	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/
| 
| So far a first part of those patches is been merged into mainline into
| pre5 (not any previous kernel, if you've some problem reproducible with
| pre4 pre3 pre2 and pre1 or any previous kernel that's not related to the
| async flushing changes, I seen a bogus report floating around to Marcelo
| about pre1 pointing to the vm changes, it can't be the vm changes if
| it's pre[1234]).
| 
| This VM is under heavy stressing for weeks on my SMP highmem machine
| with a real life DBMS workload in a real life setup with huge VM
| pressure with mem=1024m and 1.2G of shm pushed in swap constantly by the
| kernel, performance of the workload is now very good and exactly
| reproducible and constant, so I recommend it for all production systems
| (both lowmem desktops and highend servers).
| 
| Alternatively you can use the whole -aa patchkit, to get all the other
| critical highend features like pte-highmem, highio etc...
| 
| I haven't bugreports pending on the vm patch.
| 
| Thanks,
| 
| Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

On Wed, Apr 10, 2002 at 04:40:22PM -0600, Richard Gooch wrote:
> Just because you want people to apply your patches, doesn't mean you
> should resort to alarmist-sounding messages. Let's at least have truth
> in advertising in one small corner of the world :-)

It's not a 99.99% that doesn't need it, I'd say a 5% would strictly need
it. There are various known problems, and they're all fixed by such
patch AFIK so unless you want to risk you _should_ apply it, that's the
whole point of the previous email.

Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Richard Gooch]

Andrea Arcangeli writes:
> On Tue, Apr 09, 2002 at 06:07:50PM -0600, Richard Gooch wrote:
> > Andrea Arcangeli writes:
> > > I recommend everybody to never use a 2.4 kernel without first applying
> > > this vm patch:
> > [...]
> > 
> > The way you write this makes it sound that the unpatched kernel is
> > very dangerous. Is this actually true? Or do you really just mean "the
> > patched kernel has better handling under extreme loads"?
> 
> The unpatched kernel isn't dangerous in the sense it won't destroy
> data, it won't corrupt memory and finally it won't deadlock on smp
> locks, but it can theoretically deadlock with oom and it has various
> other runtime issues starting from highmem balancing, too much
> swapping, lru list balancing, related-bhs in highmem, numa broken
> with += min etc... so IMHO it is better to _always_ use the patched
> kernel that takes care of all problems that I know of at the moment,
> plus it has further optimizations. OTOH for lots of workloads
> mainline is just fine, the deadlocks never trigger and the runtime
> behaviour is ok, but unless you are certain you don't need the
> vm-33.gz patch, I recommend to apply it.

So, in other words, 99.99% of users don't need to apply the patch.
They should, in order to have a better system, but 99.99% of them
won't notice the difference. That seems to be a more honest
recommendation than the "panic stations" alert that you posted.

Just because you want people to apply your patches, doesn't mean you
should resort to alarmist-sounding messages. Let's at least have truth
in advertising in one small corner of the world :-)

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Christoph Hellwig]

On Wed, Apr 10, 2002 at 01:36:09AM +0200, Andrea Arcangeli wrote:
> I recommend everybody to never use a 2.4 kernel without first applying
> this vm patch:
> 
> 	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/vm-33.gz
> 
> It applies cleanly to both 2.4.19pre5 and 2.4.19pre6. Andrew splitted it
> into orthogonal pieces for easy merging from Marcelo's side (modulo
> -rest that is important too but that it's still quite monolithic, but
> it's pointless to invest further effort at this time until we are
> certain Marcelo will do its job and eventually merge it in mainline):

Maybe you could start by explaining the questions asked by akpm ontop of
many of the split pages that you even merged into -aa?  Or by explaining why
you reverse all functional changes akpm did in the -rest patch?

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

On Tue, Apr 09, 2002 at 06:07:50PM -0600, Richard Gooch wrote:
> Andrea Arcangeli writes:
> > I recommend everybody to never use a 2.4 kernel without first applying
> > this vm patch:
> [...]
> 
> The way you write this makes it sound that the unpatched kernel is
> very dangerous. Is this actually true? Or do you really just mean "the
> patched kernel has better handling under extreme loads"?

The unpatched kernel isn't dangerous in the sense it won't destroy data,
it won't corrupt memory and finally it won't deadlock on smp locks, but
it can theoretically deadlock with oom and it has various other runtime
issues starting from highmem balancing, too much swapping, lru list
balancing, related-bhs in highmem, numa broken with += min etc... so
IMHO it is better to _always_ use the patched kernel that takes care of
all problems that I know of at the moment, plus it has further
optimizations. OTOH for lots of workloads mainline is just fine, the
deadlocks never trigger and the runtime behaviour is ok, but unless you
are certain you don't need the vm-33.gz patch, I recommend to apply it.

Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Richard Gooch]

Andrea Arcangeli writes:
> I recommend everybody to never use a 2.4 kernel without first applying
> this vm patch:
[...]

The way you write this makes it sound that the unpatched kernel is
very dangerous. Is this actually true? Or do you really just mean "the
patched kernel has better handling under extreme loads"?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

I recommend everybody to never use a 2.4 kernel without first applying
this vm patch:

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/vm-33.gz

It applies cleanly to both 2.4.19pre5 and 2.4.19pre6. Andrew splitted it
into orthogonal pieces for easy merging from Marcelo's side (modulo
-rest that is important too but that it's still quite monolithic, but
it's pointless to invest further effort at this time until we are
certain Marcelo will do its job and eventually merge it in mainline):

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/

So far a first part of those patches is been merged into mainline into
pre5 (not any previous kernel, if you've some problem reproducible with
pre4 pre3 pre2 and pre1 or any previous kernel that's not related to the
async flushing changes, I seen a bogus report floating around to Marcelo
about pre1 pointing to the vm changes, it can't be the vm changes if
it's pre[1234]).

This VM is under heavy stressing for weeks on my SMP highmem machine
with a real life DBMS workload in a real life setup with huge VM
pressure with mem=1024m and 1.2G of shm pushed in swap constantly by the
kernel, performance of the workload is now very good and exactly
reproducible and constant, so I recommend it for all production systems
(both lowmem desktops and highend servers).

Alternatively you can use the whole -aa patchkit, to get all the other
critical highend features like pte-highmem, highio etc...

I haven't bugreports pending on the vm patch.

Thanks,

Andrea