From: Mel Gorman <mel@csn.ul.ie>
To: Pekka Enberg <penberg@cs.helsinki.fi>
Cc: Rik van Riel <riel@redhat.com>,
Larry Finger <Larry.Finger@lwfinger.net>,
"Rafael J. Wysocki" <rjw@sisk.pl>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Kernel Testers List <kernel-testers@vger.kernel.org>,
Johannes Berg <johannes@sipsolutions.net>,
Andrew Morton <akpm@linux-foundation.org>,
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>,
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Subject: Re: [Bug #13319] Page allocation failures with b43 and p54usb
Date: Mon, 8 Jun 2009 11:17:40 +0100 [thread overview]
Message-ID: <20090608101739.GA15377@csn.ul.ie> (raw)
In-Reply-To: <84144f020906070732l31786156r5d9753a0cabfde79@mail.gmail.com>
On Sun, Jun 07, 2009 at 05:32:52PM +0300, Pekka Enberg wrote:
> On Sun, Jun 7, 2009 at 5:19 PM, Rik van Riel <riel@redhat.com> wrote:
> > Pekka Enberg wrote:
> >>
> >> Hi Larry,
> >>
> >> On Sun, Jun 7, 2009 at 4:10 PM, Larry Finger <Larry.Finger@lwfinger.net>
> >> wrote:
> >>>
> >>> Rafael J. Wysocki wrote:
> >>>>
> >>>> This message has been generated automatically as a part of a report
> >>>> of recent regressions.
> >>>>
> >>>> The following bug entry is on the current list of known regressions
> >>>> from 2.6.29. Please verify if it still should be listed and let me know
> >>>> (either way).
> >>>>
> >>>>
> >>>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=13319
> >>>> Subject : Page allocation failures with b43 and p54usb
> >>>> Submitter : Larry Finger <Larry.Finger@lwfinger.net>
> >>>> Date : 2009-04-29 21:01 (40 days old)
> >>>> References : http://marc.info/?l=linux-kernel&m=124103897101088&w=4
> >>>> Handled-By : Johannes Berg <johannes@sipsolutions.net>
> >>>
> >>> This bug is extremely difficult to pin down. I cannot reproduce it at
> >>> will. The system has to be up for a long time, which is difficult with
> >>> testing the late RC's of 2.6.30 and the code in wireless-testing so
> >>> that new bugs don't end up in 2.6.31-RCX. That said, it still was in
> >>> 2.6.30-RC6 and I'm not aware of any changes since that would fix it.
> >>>
> >>> My operating kernel is patched with additional diagnostics to help me
> >>> understand why a kmalloc request for a buffer of 1390 bytes suddenly
> >>> ends up as an O(1) request. Unfortunately, I don't have any answers.
> >>
> >> Looking at the out-of-memory trace, there's still memory available but
> >> the pskb_expand_head() allocation is GFP_ATOMIC so there's not much
> >> the page allocator can do here. The amount of memory consumed by
> >> inactive_file is pretty high so maybe the problem is related to the
> >> recent mm/vmscan.c changes. Lets copy some more mm developers and see
> >> if they can help out.
> >
> > That is a very strange trace. The Mem-Info indicates
> > that the system has more than enough memory free, and
> > also enough memory in higher-order free blocks.
> >
> > This would indicate a bug somewhere in the page
> > allocator - this memory should have been given to this
> > allocation request.
>
> Aha, I always have difficulties deciphering the traces. But lets
> invite Mel to the party then!
>
Nothing like a party on Monday morning to get the week started!
What we appear to have is
o Allocation failure is high-order, high-priority, compound and atomic.
o swap is mostly unused, but we cannot enter direct reclaim.
o ZONE_DMA32 can be used
o The allocation path is in the slub allocator
o Are way above the order-0 watermarks so kswapd is probably not awake
o The minimum watermark for an order-0 page was about 647 pages in ZONE_DMA32
o The minimum watermark for an order-1 page was about 323 pages in ZONE_DMA32
o There are 10244 pages free at the time of the failure
o With the order-0 pages taken out for watermark calculation, there are
182 free pages which is below the watermark of 323 pages for an
order-1 allocation
While there is enough free memory overall, the zone watermark calculation
takes into account the order of the request. As this is an order-1 allocation,
the free order-0 pages are taken out of consideration and so the allocation
fails.
We've encountered this before and the conclusion was that the current
adjustments for watermark calculations of high-order allocations is right,
or at least there is no better alternative. In other words, the page
allocator in this instance is behaving as expected. Do we want to
revisit that discussion as to whether the watermark calculations for
high-order allocation should change? I think we'll reach the same
conclusion or at least decide that allowing the order-1 atomic
allocation to succeed here would just postpone the problem.
So the question is why are we doing a high-order atomic allocation in
this path? According to an earlier discussion on this problemn
> I think something happened to change the allocation as I never saw these
> O(1) failures before with these particular drivers. I put in a few test
> printk's and the buffers were 700-800 bytes long, and I would not expect
> them to require more than an O(0) allocation.
So, SLUB is deciding to use order-1 pages for the slab allocation.
Ordinarily, it'll get away with that because order-1 pages will be
allocated from a path that can direct reclaim. However, if a slab is
being used for atomic allocations, there is a chance that it's the
atomic request that allocates a new page for the slab.
Larry, can you post the contents of /proc/slabinfo so we can see
what size pages are being used for the kmalloc() buckets please?
Larry, you say the buffer is 700-800 bytes. Can you confirm that 800 bytes
is roughly the request size being made by ieee80211_skb_resize()?
Pekka, assuming the request size is 800 bytes, and SLUB is using order-1
pages for allocations of that size, what happened order-1 allocations
falling back to order-0 allocations as necessary. That logic exists,
right? If so, could it be broken?
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
next prev parent reply other threads:[~2009-06-08 10:17 UTC|newest]
Thread overview: 170+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-06-07 9:47 2.6.30-rc8-git4: Reported regressions from 2.6.29 Rafael J. Wysocki
2009-06-07 9:47 ` [Bug #13109] High latency on /sys/class/thermal Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13116] Can't boot with nosmp Rafael J. Wysocki
2009-06-08 16:15 ` Stephen Hemminger
2009-06-08 16:29 ` Dan Williams
2009-06-09 0:04 ` Stephen Hemminger
2009-06-09 17:20 ` Dan Williams
2009-06-09 18:30 ` Avi Kivity
2009-06-09 18:36 ` Stephen Hemminger
2009-06-09 18:42 ` Avi Kivity
2009-06-09 20:58 ` Stephen Hemminger
2009-06-09 23:19 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13119] Trouble with make-install from a NFS mount Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13179] CD-R: wodim intermittent failures Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13277] 2.6.30 regression - unreliable resume - bisected - Thinkpad X40 Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13219] Since kernel 2.6.30-rc1, computers hangs randomly Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13180] 2.6.30-rc2: WARNING at i915_gem.c for i915_gem_idle Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13313] vm86old oops Rafael J. Wysocki
2009-06-11 13:02 ` Sergey Senozhatsky
2009-06-07 9:52 ` [Bug #13306] hibernate slow on _second_ run Rafael J. Wysocki
2009-06-08 6:36 ` Johannes Berg
2009-06-08 11:14 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-06-07 13:10 ` Larry Finger
2009-06-07 13:40 ` Pekka Enberg
2009-06-07 14:19 ` Rik van Riel
2009-06-07 14:32 ` Pekka Enberg
2009-06-07 16:35 ` Larry Finger
2009-06-08 8:32 ` KAMEZAWA Hiroyuki
2009-06-08 17:20 ` Larry Finger
2009-06-08 10:17 ` Mel Gorman [this message]
2009-06-08 10:52 ` Pekka Enberg
2009-06-08 11:03 ` Mel Gorman
2009-06-08 13:58 ` Pekka J Enberg
2009-06-08 14:12 ` Mel Gorman
2009-06-08 14:42 ` Christoph Lameter
2009-06-09 7:06 ` Pekka Enberg
2009-06-09 7:54 ` David Rientjes
2009-06-09 7:58 ` Pekka Enberg
2009-06-09 8:14 ` David Rientjes
2009-06-09 8:28 ` Pekka Enberg
2009-06-10 14:41 ` Larry Finger
2009-06-10 15:44 ` Pekka Enberg
2009-06-10 15:49 ` Pekka Enberg
2009-06-10 15:52 ` Johannes Berg
2009-06-10 16:06 ` Pekka Enberg
2009-06-10 16:16 ` Pekka Enberg
2009-06-10 16:10 ` Larry Finger
2009-06-11 14:41 ` Christoph Lameter
2009-06-11 15:09 ` Pekka Enberg
2009-06-11 18:41 ` Johannes Berg
2009-06-10 15:56 ` Mel Gorman
2009-06-10 18:03 ` Pekka Enberg
2009-06-09 7:50 ` Pekka Enberg
2009-06-08 13:20 ` Rik van Riel
2009-06-08 13:35 ` Mel Gorman
2009-06-08 13:34 ` Larry Finger
2009-06-07 9:52 ` [Bug #13318] AGP doesn't work anymore on nforce2 Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13337] [post 2.6.29 regression] hang during suspend of b44/b43 modules Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13328] b44: eth0: BUG! Timeout waiting for bit 00000002 of register 42c to clear Rafael J. Wysocki
2009-06-08 7:29 ` Francis Moreau
2009-06-12 13:27 ` Francis Moreau
2009-06-12 19:14 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13330] nfs4 NULL pointer dereference in _nfs4_do_setlk Rafael J. Wysocki
2009-06-07 19:28 ` Trond Myklebust
2009-06-07 21:04 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13341] Random Oops at boot at loading ip6tables rules Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13366] About 80% of shutdowns fail (blocking) Rafael J. Wysocki
2009-06-07 16:02 ` Martin Bammer
2009-06-07 21:09 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13351] 2.6.30 corrupts my system after suspend resume with readonly mounted hard disk Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13362] rt2x00: slow wifi with correct basic rate bitmap Rafael J. Wysocki
2009-06-07 12:58 ` Alejandro Riveira Fernández
2009-06-07 21:05 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13373] fbcon, intelfb, i915: INFO: possible circular locking dependency detected Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13374] reiserfs blocked for more than 120secs Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13391] Kernel boot hangs at about every second start when kms is activated Rafael J. Wysocki
2009-06-07 16:04 ` Martin Bammer
2009-06-07 21:11 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13389] Warning 'Invalid throttling state, reset' gets displayed when it should not be Rafael J. Wysocki
2009-06-08 11:31 ` Frans Pop
2009-06-07 9:52 ` [Bug #13401] pktcdvd writing is really slow with CFQ scheduler (bisected) Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13408] Performance regression in 2.6.30-rc7 Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13424] possible deadlock when doing governor switching Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13423] JMicron SATA controller not available Rafael J. Wysocki
2009-06-07 15:23 ` Marc Dionne
2009-06-07 21:13 ` Rafael J. Wysocki
2009-06-08 2:12 ` Marc Dionne
2009-06-07 9:52 ` [Bug #13407] adb trackpad disappears after suspend to ram Rafael J. Wysocki
2009-06-25 15:07 ` Jan Scholz
2009-06-07 9:52 ` [Bug #13446] resume after suspend-to-ram broken on Toshiba Satellite A100 with 2.6.30-rc8 (works in 2.6.28) Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13470] Machine doesn't boot due to mmconfig detection problem Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13462] Unused bands in intefb console and smaller 180x56 -> 128x48 Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13475] suspend/hibernate lockdep warning Rafael J. Wysocki
2009-06-07 13:21 ` Pekka Enberg
2009-06-08 7:35 ` Dave Young
2009-06-08 7:49 ` Pekka Enberg
2009-06-08 12:48 ` Mathieu Desnoyers
2009-06-08 14:32 ` Dave Jones
2009-06-08 15:23 ` [PATCH] remove rwsem lock from CPUFREQ_GOV_STOP call (second call site) Mathieu Desnoyers
2009-06-08 16:57 ` Pallipadi, Venkatesh
2009-06-08 17:17 ` Mathieu Desnoyers
2009-06-09 1:15 ` Dave Young
2009-06-09 15:23 ` Mathieu Desnoyers
2009-06-11 4:46 ` Dave Young
2009-06-11 13:39 ` [Bug #13475] suspend/hibernate lockdep warning Simon Holm Thøgersen
2009-06-11 15:23 ` Mathieu Desnoyers
2009-06-17 0:39 ` Pallipadi, Venkatesh
2009-06-17 1:05 ` Mathieu Desnoyers
2009-06-17 15:29 ` Thomas Renninger
2009-06-17 17:03 ` Pallipadi, Venkatesh
2009-06-18 5:46 ` Dave Young
2009-06-07 9:52 ` [Bug #13472] Oops with minicom and USB serial Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13473] Bug while trying to launch a KVM guest Rafael J. Wysocki
2009-06-08 4:26 ` Sachin Sant
2009-06-08 11:16 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13471] Loading parport_pc kills the keyboard if ACPI is enabled Rafael J. Wysocki
2009-06-07 13:25 ` Ozan Çağlayan
2009-06-07 21:14 ` Rafael J. Wysocki
2009-06-07 9:52 ` [Bug #13474] Oops whilst booting Rafael J. Wysocki
-- strict thread matches above, loose matches on Subject: below --
2009-08-25 20:37 2.6.31-rc7-git2: Reported regressions 2.6.29 -> 2.6.30 Rafael J. Wysocki
2009-08-25 21:05 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-08-26 6:25 ` Pekka Enberg
2009-08-26 20:53 ` Rafael J. Wysocki
2009-08-19 20:36 2.6.31-rc6-git5: Reported regressions 2.6.29 -> 2.6.30 Rafael J. Wysocki
2009-08-19 20:40 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-08-09 21:07 2.6.31-rc5-git5: Reported regressions 2.6.29 -> 2.6.30 Rafael J. Wysocki
2009-08-09 21:10 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-08-02 19:06 2.6.31-rc5: Reported regressions 2.6.29 -> 2.6.30 Rafael J. Wysocki
2009-08-02 19:09 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-07-26 20:41 2.6.31-rc4: Reported regressions 2.6.29 -> 2.6.30 Rafael J. Wysocki
2009-07-26 20:45 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-07-27 0:17 ` Larry Finger
2009-07-27 0:24 ` David Rientjes
2009-07-27 7:08 ` Pekka Enberg
2009-07-27 9:37 ` David Rientjes
2009-07-27 17:20 ` Christoph Lameter
2009-07-27 18:16 ` David Rientjes
2009-07-27 21:43 ` Christoph Lameter
2009-07-27 22:38 ` David Rientjes
2009-07-06 23:57 2.6.31-rc2: Reported regressions 2.6.29 -> 2.6.30 Rafael J. Wysocki
2009-07-07 0:00 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-07-07 1:05 ` Larry Finger
2009-07-07 6:29 ` David Rientjes
2009-07-07 6:57 ` Pekka Enberg
2009-07-08 13:18 ` Larry Finger
2009-06-29 0:26 2.6.31-rc1-git3: Reported regressions 2.6.29 -> 2.6.30 Rafael J. Wysocki
2009-06-29 0:30 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-06-29 16:51 ` Larry Finger
2009-06-29 23:15 ` Rafael J. Wysocki
2009-06-29 23:47 ` David Rientjes
2009-06-30 2:06 ` Larry Finger
2009-06-30 5:47 ` David Rientjes
2009-06-30 6:55 ` Pekka Enberg
2009-06-30 7:47 ` David Rientjes
2009-06-30 8:24 ` Pekka Enberg
2009-06-30 14:38 ` Larry Finger
2009-06-30 20:25 ` David Rientjes
2009-06-30 14:32 ` Christoph Lameter
2009-06-30 15:01 ` Pekka Enberg
2009-06-30 15:14 ` Christoph Lameter
2009-06-30 20:04 ` David Rientjes
2009-06-30 21:05 ` Christoph Lameter
2009-06-30 21:15 ` David Rientjes
2009-06-30 21:23 ` Christoph Lameter
2009-06-30 21:52 ` David Rientjes
2009-06-30 22:18 ` Christoph Lameter
2009-07-01 5:53 ` Pekka Enberg
2009-07-02 17:18 ` David Rientjes
2009-07-03 7:23 ` Pekka Enberg
2009-05-30 19:29 2.6.30-rc7-git4: Reported regressions from 2.6.29 Rafael J. Wysocki
2009-05-30 19:37 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-05-24 19:06 2.6.30-rc7: Reported regressions from 2.6.29 Rafael J. Wysocki
2009-05-24 19:11 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-05-16 19:14 2.6.30-rc6: Reported regressions from 2.6.29 Rafael J. Wysocki
2009-05-16 19:20 ` [Bug #13319] Page allocation failures with b43 and p54usb Rafael J. Wysocki
2009-05-16 23:36 ` Andrew Morton
2009-05-17 23:16 ` Larry Finger
2009-05-18 6:31 ` Pekka Enberg
2009-05-21 13:21 ` Larry Finger
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20090608101739.GA15377@csn.ul.ie \
--to=mel@csn.ul.ie \
--cc=Larry.Finger@lwfinger.net \
--cc=akpm@linux-foundation.org \
--cc=johannes@sipsolutions.net \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=kernel-testers@vger.kernel.org \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=penberg@cs.helsinki.fi \
--cc=riel@redhat.com \
--cc=rjw@sisk.pl \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®