From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2CBEC7618F for ; Wed, 17 Jul 2019 00:12:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D63D821871 for ; Wed, 17 Jul 2019 00:12:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388953AbfGQAMy (ORCPT ); Tue, 16 Jul 2019 20:12:54 -0400 Received: from out30-45.freemail.mail.aliyun.com ([115.124.30.45]:44118 "EHLO out30-45.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728848AbfGQAMx (ORCPT ); Tue, 16 Jul 2019 20:12:53 -0400 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01422;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0TX5-mrd_1563322365; Received: from US-143344MP.local(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0TX5-mrd_1563322365) by smtp.aliyun-inc.com(127.0.0.1); Wed, 17 Jul 2019 08:12:49 +0800 Subject: Re: list corruption in deferred_split_scan() To: Shakeel Butt , Kirill Tkhai , Vladimir Davydov , Hugh Dickins , Michal Hocko , Johannes Weiner , Roman Gushchin Cc: Qian Cai , "Kirill A. Shutemov" , Andrew Morton , Linux MM , LKML References: <1562795006.8510.19.camel@lca.pw> <1562879229.8510.24.camel@lca.pw> <1563225798.4610.5.camel@lca.pw> <5c853e6e-6367-d83c-bb97-97cd67320126@linux.alibaba.com> <8A64D551-FF5B-4068-853E-9E31AF323517@lca.pw> From: Yang Shi Message-ID: <50f57bf8-a71a-c61f-74f7-31fb7bfe3253@linux.alibaba.com> Date: Tue, 16 Jul 2019 17:12:45 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 7/16/19 4:36 PM, Shakeel Butt wrote: > Adding related people. > > The thread starts at: > http://lkml.kernel.org/r/1562795006.8510.19.camel@lca.pw > > On Mon, Jul 15, 2019 at 8:01 PM Yang Shi wrote: >> >> >> On 7/15/19 6:36 PM, Qian Cai wrote: >>>> On Jul 15, 2019, at 8:22 PM, Yang Shi wrote: >>>> >>>> >>>> >>>> On 7/15/19 2:23 PM, Qian Cai wrote: >>>>> On Fri, 2019-07-12 at 12:12 -0700, Yang Shi wrote: >>>>>>> Another possible lead is that without reverting the those commits below, >>>>>>> kdump >>>>>>> kernel would always also crash in shrink_slab_memcg() at this line, >>>>>>> >>>>>>> map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, true); >>>>>> This looks a little bit weird. It seems nodeinfo[nid] is NULL? I didn't >>>>>> think of where nodeinfo was freed but memcg was still online. Maybe a >>>>>> check is needed: >>>>> Actually, "memcg" is NULL. >>>> It sounds weird. shrink_slab() is called in mem_cgroup_iter which does pin the memcg. So, the memcg should not go away. >>> Well, the commit “mm: shrinker: make shrinker not depend on memcg kmem” changed this line in shrink_slab_memcg(), >>> >>> - if (!memcg_kmem_enabled() || !mem_cgroup_online(memcg)) >>> + if (!mem_cgroup_online(memcg)) >>> return 0; >>> >>> Since the kdump kernel has the parameter “cgroup_disable=memory”, shrink_slab_memcg() will no longer be able to handle NULL memcg from mem_cgroup_iter() as, >>> >>> if (mem_cgroup_disabled()) >>> return NULL; >> Aha, yes. memcg_kmem_enabled() implicitly checks !mem_cgroup_disabled(). >> Thanks for figuring this out. I think we need add mem_cgroup_dsiabled() >> check before calling shrink_slab_memcg() as below: >> >> diff --git a/mm/vmscan.c b/mm/vmscan.c >> index a0301ed..2f03c61 100644 >> --- a/mm/vmscan.c >> +++ b/mm/vmscan.c >> @@ -701,7 +701,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int >> nid, >> unsigned long ret, freed = 0; >> struct shrinker *shrinker; >> >> - if (!mem_cgroup_is_root(memcg)) >> + if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) >> return shrink_slab_memcg(gfp_mask, nid, memcg, priority); >> >> if (!down_read_trylock(&shrinker_rwsem)) >> > We were seeing unneeded oom-kills on kernels with > "cgroup_disabled=memory" and Yang's patch series basically expose the > bug to crash. I think the commit aeed1d325d42 ("mm/vmscan.c: > generalize shrink_slab() calls in shrink_node()") missed the case for > "cgroup_disabled=memory". However I am surprised that root_mem_cgroup > is allocated even for "cgroup_disabled=memory" and it seems like > css_alloc() is called even before checking if the corresponding > controller is disabled. I'm surprised too. A quick test with drgn shows root memcg is definitely allocated: >>> prog['root_mem_cgroup'] *(struct mem_cgroup *)0xffff8902cf058000 = { [snip] But, isn't this a bug? Thanks, Yang > > Yang, can you please send the above change with signed-off and CC to > stable as well? > > thanks, > Shakeel