From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=MAILING_LIST_MULTI,SPF_PASS, USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 843A8C6778F for ; Wed, 25 Jul 2018 12:32:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3CA9D20893 for ; Wed, 25 Jul 2018 12:32:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3CA9D20893 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729171AbeGYNoI (ORCPT ); Wed, 25 Jul 2018 09:44:08 -0400 Received: from mx2.suse.de ([195.135.220.15]:52232 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728725AbeGYNoI (ORCPT ); Wed, 25 Jul 2018 09:44:08 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id C688BAFA6; Wed, 25 Jul 2018 12:32:37 +0000 (UTC) Date: Wed, 25 Jul 2018 14:32:37 +0200 From: Michal Hocko To: Bruce Merry Cc: Shakeel Butt , Andrew Morton , LKML , Linux MM , Johannes Weiner , Vladimir Davydov Subject: Re: Showing /sys/fs/cgroup/memory/memory.stat very slow on some machines Message-ID: <20180725123237.GI28386@dhcp22.suse.cz> References: <20180717212307.d6803a3b0bbfeb32479c1e26@linux-foundation.org> <20180718104230.GC1431@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue 24-07-18 12:05:35, Bruce Merry wrote: [...] > I've also added some code to memcg_stat_show to report the number of > cgroups in the hierarchy (iterations in for_each_mem_cgroup_tree). > Running the script increases it from ~700 to ~41000. The script > iterates 250,000 times, so only some fraction of the cgroups become > zombies. So this is definitely "too many zombies" to delay the collecting of cumulative stats. Maybe we need to limit the number of zombies and reclaim them more actively. I have seen Shakeel has posted something but it looked more on the accounting side from a quick glance. I can see you are using cgroup v1 so your workaround would be to memory.force_empty before you remove the group. -- Michal Hocko SUSE Labs