From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751748AbdASJtb (ORCPT ); Thu, 19 Jan 2017 04:49:31 -0500 Received: from smtp-out-no.shaw.ca ([64.59.134.13]:55468 "EHLO smtp-out-no.shaw.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751420AbdASJtD (ORCPT ); Thu, 19 Jan 2017 04:49:03 -0500 X-Authority-Analysis: v=2.2 cv=BNTDlBYG c=1 sm=1 tr=0 a=DXb8iGicsy0C2lNtaEhhGg==:117 a=DXb8iGicsy0C2lNtaEhhGg==:17 a=kj9zAlcOel0A:10 a=IgFoBzBjUZAA:10 a=VwQbUJbxAAAA:8 a=PrrE3-KTAAAA:8 a=C872j_0ZgeNrdo-YOPcA:9 a=CjuIK1q_8ugA:10 a=k424ElkXfG8A:10 a=_yr3VdnE7YcA:10 a=AjGcO6oz07-iQ99wixmX:22 a=rB81Y03d9IYvG-749Y1b:22 Date: Thu, 19 Jan 2017 03:48:50 -0600 From: Trevor Cordes To: Michal Hocko Cc: Mel Gorman , linux-kernel@vger.kernel.org, Joonsoo Kim , Minchan Kim , Rik van Riel , Srikar Dronamraju Subject: Re: mm, vmscan: commit makes PAE kernel crash nightly (bisected) Message-ID: <20170119034850.0b7d504c@pog.tecnopolis.ca> In-Reply-To: <20170117145450.GQ19699@dhcp22.suse.cz> References: <20170111103243.GA27795@pog.tecnopolis.ca> <20170111121146.i53jyerxont7b74i@techsingularity.net> <20170111121428.4sswbkzyqz7vf6so@techsingularity.net> <20170111165232.0b7abe18@pog.tecnopolis.ca> <20170112093613.GF2264@dhcp22.suse.cz> <20170115002752.4559f668@pog.tecnopolis.ca> <20170116110934.7zopy3ecg2lfadkd@techsingularity.net> <20170117135228.GN19699@dhcp22.suse.cz> <20170117142114.r7abr3x2bbik47sd@techsingularity.net> <20170117145450.GQ19699@dhcp22.suse.cz> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.31; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-CMAE-Envelope: MS4wfJRG/+/Or3t5e1VQIpZDGHhivfx4dTxIF2peBN9w5ayUTJaIKBx9new661T8ysXf2cviQjv0cINGzu8kIZ732nuf3qZbY3rDhYftCBAz4JubNYvicjme 14XmRhHIH09lVsFepI6bdtzhpgu05QhfnhXWUnfwRaK6VuGt1sxyCs+iTKPl0iA66Yrpx50GQJLi0UEmvLvjX74tt98y+xtVURRsCFpaR1i/GbZNEANoT7ag NeMvgGlbXoyOz/U5mPchTt+gky1CAyhFCdZCdQFrGF4dj21S/GkryBhx91M1P+xRUSq7w3VeWRhxfsDLMEu28aPMKlm5TPxu2NEr4P+OUCffKzRktJ5B1+Pd RsAF/SJTYCGXUI53CWAaPOFDkfrfag== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2017-01-17 Michal Hocko wrote: > On Tue 17-01-17 14:21:14, Mel Gorman wrote: > > On Tue, Jan 17, 2017 at 02:52:28PM +0100, Michal Hocko wrote: > > > On Mon 16-01-17 11:09:34, Mel Gorman wrote: > > > [...] > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > > > index 532a2a750952..46aac487b89a 100644 > > > > --- a/mm/vmscan.c > > > > +++ b/mm/vmscan.c > > > > @@ -2684,6 +2684,7 @@ static void shrink_zones(struct zonelist > > > > *zonelist, struct scan_control *sc) continue; > > > > > > > > if (sc->priority != DEF_PRIORITY && > > > > + !buffer_heads_over_limit && > > > > !pgdat_reclaimable(zone->zone_pgdat)) > > > > continue; /* Let kswapd > > > > poll it */ > > > > > > I think we should rather remove pgdat_reclaimable here. This > > > sounds like a wrong layer to decide whether we want to reclaim > > > and how much. > > > > I had considered that but it'd also be important to add the other > > 32-bit patches you have posted to see the impact. Because of the > > ratio of LRU pages to slab pages, it may not have an impact but > > it'd need to be eliminated. > > OK, Trevor you can pull from > git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git tree > fixes/highmem-node-fixes branch. This contains the current mmotm tree > + the latest highmem fixes. I also do not expect this would help much > in your case but as Mel've said we should rule that out at least. Hi! The git tree above version oom'd after < 24 hours (3:02am) so it doesn't solve the bug. If you need a oom messages dump let me know. Let me know what to try next, guys, and I'll test it out. > > Before prototyping such a thing, I'd like to hear the outcome of > > this heavy hack and then add your 32-bit patches onto the list. If > > the problem is still there then I'd next look at taking slab pages > > into account in pgdat_reclaimable() instead of an outright removal > > that has a much wider impact. If that doesn't work then I'll > > prototype a heavy-handed forced slab reclaim when lower zones are > > almost all slab pages. I don't think I've tried the "heavy hack" patch yet? It's not in the mhocko tree I just tried? Should I try the heavy hack on top of mhocko git or on vanilla or what? I also want to mention that these PAE boxes suffer from another problem/bug that I've worked around for almost a year now. For some reason it keeps gnawing at me that it might be related. The disk I/O goes to pot on this/these PAE boxes after a certain amount of disk writes (like some unknown number of GB, around 10-ish maybe). Like writes go from 500MB/s to 10MB/s!! Reboot and it's magically 500MB/s again. I detail this here: https://muug.ca/pipermail/roundtable/2016-June/004669.html My fix was to mem=XG where X is <8 (like 4 or 6) to force the PAE kernel to be more sane about highmem choices. I never filed a bug because I read a ton of stuff saying Linus hates PAE, don't use over 4G, blah blah. But the other fix is to: set /proc/sys/vm/highmem_is_dirtyable to 1 I'm not bringing this up to get attention to a new bug, I bring this up because it smells like it might be related. If something slowly eats away at the box's vm to the point that I/O gets horribly slow, perhaps it's related to the slab and high/lomem issue we have here? And if related, it may help to solve the oom bug. If I'm way off base here, just ignore my tangent! The funny thing is I thought mem=XG where X<8 solved the problem, but it doesn't! It greatly mitigates it, but I still get subtle slowdown that gets worse over time (like weeks instead of days). I now use the highmem_is_dirtyable on most boxes and that seems to solve it for good in combo with mem=XG. Let me note, however, that I have NOT set highmem_is_dirtyable=1 on the test box I am using for all of this building/testing, as I wanted the config to stay static while I work through this oom bug. (I'm real curious to see if highmem_is_dirtyable=1 would have any impact on the oom though!) Thanks!