From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1763410AbdAKKdB (ORCPT ); Wed, 11 Jan 2017 05:33:01 -0500 Received: from smtp-out-no.shaw.ca ([64.59.134.13]:50139 "EHLO smtp-out-no.shaw.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758321AbdAKKc7 (ORCPT ); Wed, 11 Jan 2017 05:32:59 -0500 X-Authority-Analysis: v=2.2 cv=be0baKHB c=1 sm=1 tr=0 a=DXb8iGicsy0C2lNtaEhhGg==:117 a=DXb8iGicsy0C2lNtaEhhGg==:17 a=kj9zAlcOel0A:10 a=IgFoBzBjUZAA:10 a=R_Myd5XaAAAA:8 a=D19gQVrFAAAA:8 a=O6qgKI_S8FO_i2hvlPcA:9 a=CjuIK1q_8ugA:10 a=L2g4Dz8VuBQ37YGmWQah:22 a=W4TVW4IDbPiebHqcZpNg:22 Date: Wed, 11 Jan 2017 04:32:43 -0600 From: Trevor Cordes To: linux-kernel@vger.kernel.org Cc: Mel Gorman , Joonsoo Kim , Michal Hocko , Minchan Kim , Rik van Riel , Srikar Dronamraju Subject: mm, vmscan: commit makes PAE kernel crash nightly (bisected) Message-ID: <20170111103243.GA27795@pog.tecnopolis.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.7.1 (2016-10-04) X-CMAE-Envelope: MS4wfEcGI5EeUrZAoM4VqLZikS4zuw0S2Xw+nV6AOjwIvkQaMViy/e5cn2Vt99ZWO74gSgtcH+/N4D0Mch8QZlY+H917iFj1R99clWILr8kxXM6lOamkH8Bo AMh7c7BHYEz6vB2yZFZycGRsMZRio/Mysuwnldwanwvurl52MCITcje44PhezPiWtZ75KCtH++9VO+/XogiSZ1+lCSMI6LuGzV6uUX/fd7nzEBhDdcrfmMYY 61Fs4Lbie7RlGOXm1+9pwACqb6LNgLVCYX3HUXGlxV1vj1qICnkSVCbPU8+CBuew1LEiSkUfOPl+jkKe4yTe42lQaTbXjh+aiBxqouVCh4M4J/lBW5Wzjhto t72qCn9wM0kdLLl7wNAb365PrXBaw6+HSdTEU6w0WJssCeq3JQs= Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi! I have biected a nightly oom-killer flood and crash/hang on one of the boxes I admin. It doesn't crash on Fedora 23/24 4.7.10 kernel but does on any 4.8 Fedora kernel. I did a vanilla bisect and the bug is here: commit b2e18757f2c9d1cdd746a882e9878852fdec9501 Author: Mel Gorman Date: Thu Jul 28 15:45:37 2016 -0700 mm, vmscan: begin reclaiming pages on a per-node basis I bisected between: # bad: [69973b830859bc6529a7a0468ba0d80ee5117826] Linux 4.9 # good: [523d939ef98fd712632d93a5a2b588e477a7565e] Linux 4.7 I have not tried newer than 4.8.13 Fedora kernel, but if someone thinks this bug is already fixed in HEAD I could try that next. It took 3 weeks to bisect because the crash only seems to happen in the middle of the night, and not every, but most, nights. It does not occur on most of my other boxes, just this one. The box is a bit unique in that it's running 32-bit PAE on a 64-bit capable CPU, and I have the memory tuned down to mem=6G in the kernel command line (I think it has 16GB actual). I tuned the RAM down because around 8GB the PAE kernel has massive IO speed issues. It is a relatively new Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz on an Intel S1200BTL board. I will eventually change it to 64-bit Fedora which I'm sure will solve this bug, but since there's no easy upgrade path, that's on the backburner on this production box. I'm sure this will be another "PAE sucks, don't use it" issue, but like I said, I'm currently stuck with it, and in theory the kernel shouldn't crash like this (I'm guessing/hoping). I think I pinned the trigger down to either (or both) big dir scans (like "find /bigdir-foo") running at around 3am. It's either a remote box doing indexing via smbd and/or rsync or rdiff-backup also doing big dir scans. But when I do "find /" manually I can't trigger the bug. Very weird. The commit notes make it sound like the author thought perhaps there could be a problem in some scenarios? I guess I found the scenario. The only discussion I found on the net regarding this commit is https://lkml.org/lkml/2016/8/29/154 And perhaps it's somewhat relevant, it's a bit over my head. I'm available for testing, etc, and can usually rule out a bad kernel within 24-hours by just waiting for 3am to roll around. I also have copious logs I can provide and screenshots of the crashes. The box is extremely lightly loaded, and RAM use is almost always under 1GB, and swap is 0-20k used most of the time with GB's free. Everything looks great until all of a sudden oom-killer starts running and goes through 10-260 iterations before the system just dies. I wrote a script to watch for oom-killer and issue "reboot" immediately, but 80% of the time the box will hang before the reboot actually manages to shutdown. Any information/help I can provide, please just holler. Thanks!