From: Marcelo Tosatti <marcelo.tosatti@cyclades.com>
To: Andrew Morton <akpm@osdl.org>
Cc: "Sergey S. Kostyliov" <rathamahata@php4.ru>,
linux-kernel@vger.kernel.org, gluk@php4.ru, anton@megashop.ru
Subject: Re: 2.6.1 IO lockup on SMP systems
Date: Tue, 24 Feb 2004 04:23:46 -0300 (BRT) [thread overview]
Message-ID: <Pine.LNX.4.58L.0402240357350.6209@logos.cnet> (raw)
In-Reply-To: <20040223142626.48938d7c.akpm@osdl.org>
On Mon, 23 Feb 2004, Andrew Morton wrote:
> "Sergey S. Kostyliov" <rathamahata@php4.ru> wrote:
> >
> > > > OK, so everything is stuck trying to allocate memory. Perhaps you ran out
> > > > of swapspace, or some process has gone berzerk allocating memory.
> >
> > The memory exhaustion is indeed possible for this box. I'll double check
> > ulimit and /etc/security/limits.conf stuff. The only thing which worries
> > me that this box had been running for months without any problems with
> > 2.4.23aa1.
>
> It is conceivable that you have some application which runs OK on 2.4.x but
> has some subtle bug which causes the app to go crazy on a 2.6 kernel
> consuming lots of memory. Or there's a bug in the 2.6 kernel ;)
>
> > I have added another 2Gb to swap space (hope this give enough time
> > to find the memory hungry process(es)).
> >
> > > >
> > > > How much memory does the machine have, and how much swap space?
> > > >
> > > # free
> > > total used free shared buffers cached
> > > Mem: 2073868 2067508 6360 0 232708 897828
> > > -/+ buffers/cache: 936972 1136896
> > > Swap: 1535976 5228 1530748
> > >
> > > > I suggest that you run a `vmstat 30' trace on a terminal somewhere, see
> > > > what it says prior to the hangs.
> > > Ok.We'll try to get it next time.
> >
> > Here it is:
> > procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
> > r b swpd free buff cache si so bi bo in cs us sy id wa
> > 1 0 551920 8108 203744 933532 0 0 4 68 1214 426 5 1 92 2
> > 0 0 551928 7140 203756 930316 0 0 17 61 1240 529 8 1 89 2
> > 0 0 551976 5788 203772 928224 1 6 360 139 1297 317 7 2 83 8
> > 0 0 551968 7588 203812 923504 0 0 19 125 1303 308 8 2 87 4
> > 0 1 551976 10444 203892 914100 0 0 25 127 1433 438 10 3 85 3
> > 0 0 551976 9220 204004 914804 0 0 123 126 1278 325 6 1 88 5
> > 0 0 551976 8108 204044 912248 0 0 38 69 1279 291 6 1 91 2
> > 0 1 551976 11828 204144 912320 1 0 135 94 1249 296 6 1 89 3
> > 0 5 562204 3280 203952 157084 1 566 305 674 1281 313 6 4 73 17
> > 0 18 598224 4276 1888 33356 91 2734 233 2761 1090 199 0 2 0 97
> > 1 38 662520 2760 2104 30520 110 3721 261 3738 1161 831 1 2 0 97
> > 10 41 699936 2772 1920 28716 123 2924 249 2946 1103 1273 0 3 0 97
> > 0 39 748588 2956 1956 22668 160 3313 245 3331 1056 1047 0 2 0 98
> > 0 38 796100 3108 1888 21348 321 3191 430 3206 1045 1002 0 2 0 97
> > 4 43 844532 3308 1956 17644 518 3719 670 3733 1357 999 0 2 0 98
> > 0 51 882596 2940 2052 13960 520 2796 705 2810 1048 1182 0 2 0 98
> > 3 59 913392 2456 2048 10900 1013 2524 1308 2542 1144 601 0 2 0 98
> > 5 71 937816 2760 2072 8584 1534 2681 1860 2702 1234 607 0 2 0 97
>
> OK, so it's doing a lot of swapping and your swap utilisation is
> continuously increasing. I would suspect an application or kernel memory
> leak.
>
> I suggest you keep that `vmstat 30' running all the time. When the machine
> dies, take a look at the final 20 lines.
>
> Also, run
>
> while true
> do
> cat /proc/meminfo
> sleep 10
> done
>
> and record the info which that leaves behind when the machine locks up.
> This should tell us whether it is an application or kernel memory leak. If
> it is indeed a leak.
Hi Andrew,
Care to explain me why should the kernel hang if due to an application
leak ?
The hang looks wrong even if the leak is in userspace app, yes?
next prev parent reply other threads:[~2004-02-24 6:32 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2004-01-31 16:40 Sergey S. Kostyliov
2004-02-01 0:17 ` Andrew Morton
2004-02-21 16:45 ` Sergey S. Kostyliov
2004-02-21 19:30 ` Andrew Morton
2004-02-22 17:39 ` Alexander Y. Fomichev
2004-02-23 17:27 ` Sergey S. Kostyliov
2004-02-23 21:30 ` Mike Fedyk
2004-02-24 11:56 ` Sergey S. Kostyliov
2004-02-23 22:26 ` Andrew Morton
2004-02-24 7:23 ` Marcelo Tosatti [this message]
2004-02-24 6:53 ` Andrew Morton
2004-02-24 11:54 ` Sergey S. Kostyliov
2004-02-26 12:19 ` Sergey S. Kostyliov
2004-02-26 12:53 ` Andrew Morton
2004-02-26 13:11 ` Andrew Morton
2004-02-26 14:37 ` Dave Jones
2004-02-26 15:37 ` Arjan van de Ven
2004-02-26 14:30 ` Sergey S. Kostyliov
2004-02-26 20:03 ` Andrew Morton
2004-02-28 14:56 ` Sergey S. Kostyliov
2004-04-08 9:08 ` 2.6.X kernel memory leak? (was: Re: 2.6.1 IO lockup on SMP systems) Sergey S. Kostyliov
2004-04-09 7:17 ` 2.6.X kernel memory leak? Sergey S. Kostyliov
2004-04-09 9:09 ` Andrew Morton
2004-04-09 12:15 ` Sergey S. Kostyliov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Pine.LNX.4.58L.0402240357350.6209@logos.cnet \
--to=marcelo.tosatti@cyclades.com \
--cc=akpm@osdl.org \
--cc=anton@megashop.ru \
--cc=gluk@php4.ru \
--cc=linux-kernel@vger.kernel.org \
--cc=rathamahata@php4.ru \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®