From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FA0EC282C0 for ; Fri, 25 Jan 2019 07:52:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id ED849217D7 for ; Fri, 25 Jan 2019 07:52:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="vMFBqrK6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729007AbfAYHwQ (ORCPT ); Fri, 25 Jan 2019 02:52:16 -0500 Received: from mail-lf1-f67.google.com ([209.85.167.67]:40377 "EHLO mail-lf1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726669AbfAYHwP (ORCPT ); Fri, 25 Jan 2019 02:52:15 -0500 Received: by mail-lf1-f67.google.com with SMTP id v5so6244160lfe.7; Thu, 24 Jan 2019 23:52:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:cc:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=S/PecIcQc1kFgYlonoFxz1E4IJONkcf4aGIUu1zvd08=; b=vMFBqrK63eVtVNZj0Q/vwZC4cCnruJWwIClI0a5m7ZShYCSrcLdZl8CXhFmEz27nv1 qbt0icT+fv9/IPCBXvVL4SVgAmP/248fNBpn2LfTAVzFLKT4mIHLmD1qeV+l6AKhOFAC xojX7hMo3rfgBcVNeQRHsTDLHCBwiLXo6uPpcepX9K16FYASuc9hc3rAkmU0RdZpo8gZ kQhRWZ+fOLbQ/5ynd9aZ+9Q7r4l1ZYtHXw205ivFQoJNr1jLqRmRbSu/2keIBIeJYIo5 htWEeZG4BOQLVfEp9nDPBODwYcO2MD6pYE+Lv+gFrtOIilXkaXnheS+xDnqgecs7QCK8 WPtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:cc:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=S/PecIcQc1kFgYlonoFxz1E4IJONkcf4aGIUu1zvd08=; b=cUVeWul6qLP/KBx4BGeQTR5FHMJgkJnhXaBDEpFmvr2PpGbLzxOJaMpLj9sTgNdp1Q rLNzoQLYN8J3gxoxueSPdo2vXNkL0xLaLxXQa0V69ZGfBt2fNIfczrp1VVkzADAlALY2 EvR1/7QXBzC4u+nwJ85TfCNyfLDaYWnu8CM4DOzKuJ47wNqwoAqnvNqRk8KGFmBc6rat GYhGIhI5PybMNELp2aeJKlC0f3jBn+uYX6KEyVi9s0nSYVWfBK/4tcMQj17BUnQhRmBV pIEQTshKbLS45P6IVV+ZdJ3tjVXLjTjqYfS1yPdduaq+DGzkGuxWSNNpALEbjG6zpitw woSA== X-Gm-Message-State: AJcUuke7c2jb23aiBCIasen/wpoCft3agHiUbUK8Za6mT0p+EOuAzPYE thZV/81OrGFL0nkUqLEiuj+TdNsThkM= X-Google-Smtp-Source: ALg8bN7otxJKrI5WLSDESVMwICheRQoDxph9wU7Fe09s2gHoQ34UigkHR5gO+uVy94eh48TJ7dhHaQ== X-Received: by 2002:a19:c502:: with SMTP id w2mr7505557lfe.168.1548402732699; Thu, 24 Jan 2019 23:52:12 -0800 (PST) Received: from amb.local (out244.support.agnat.pl. [91.234.176.244]) by smtp.gmail.com with ESMTPSA id m1sm1330557lfb.56.2019.01.24.23.52.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jan 2019 23:52:12 -0800 (PST) Subject: Re: pids.current with invalid value for hours [5.0.0 rc3 git] From: =?UTF-8?Q?Arkadiusz_Mi=c5=9bkiewicz?= To: cgroups@vger.kernel.org References: <20190117122535.njcbqhlmzozdkncw@mikami> <1d36b181-cbaf-6694-1a31-2f7f55d15675@gmail.com> <96ef6615-a5df-30af-b4dc-417a18ca63f1@gmail.com> Cc: Aleksa Sarai , Jay Kamat , Roman Gushchin , Michal Hocko , Johannes Weiner , linux-kernel@vger.kernel.org Message-ID: <1cdbef13-564d-61a6-95f4-579d2cad243d@gmail.com> Date: Fri, 25 Jan 2019 08:52:11 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <96ef6615-a5df-30af-b4dc-417a18ca63f1@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: pl Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 24/01/2019 12:21, Arkadiusz Miśkiewicz wrote: > On 17/01/2019 14:17, Arkadiusz Miśkiewicz wrote: >> On 17/01/2019 13:25, Aleksa Sarai wrote: >>> On 2019-01-17, Arkadiusz Miśkiewicz wrote: >>>> Using kernel 4.19.13. >>>> >>>> For one cgroup I noticed weird behaviour: >>>> >>>> # cat pids.current >>>> 60 >>>> # cat cgroup.procs >>>> # >>> >>> Are there any zombies in the cgroup? pids.current is linked up directly >>> to __put_task_struct (so exit(2) won't decrease it, only the task_struct >>> actually being freed will decrease it). >>> >> >> There are no zombie processes. >> >> In mean time the problem shows on multiple servers and so far saw it >> only in cgroups that were OOMed. >> >> What has changed on these servers (yesterday) is turning on >> memory.oom.group=1 for all cgroups and changing memory.high from 1G to >> "max" (leaving memory.max=2G limit only). >> >> Previously there was no such problem. >> > > I'm attaching reproducer. This time tried on different distribution > kernel (arch linux). > > After 60s pids.current still shows 37 processes even if there are no > processes running (according to ps aux). The same test on 5.0.0-rc3-00104-gc04e2a780caf and it's easy to reproduce bug. No processes in cgroup but pids.current reports 91. memory.oom.group=0 - everything works fine, pids are counted properly memory.oom.group=1 - bug becomes visible [root@xps test]# python3 cg.py Created cgroup: /sys/fs/cgroup/test_5277 Start: pids.current: 0 Start: cgroup.procs: 0: pids.current: 103 0: cgroup.procs: 1: pids.current: 91 1: cgroup.procs: 2: pids.current: 91 2: cgroup.procs: 3: pids.current: 91 3: cgroup.procs: 4: pids.current: 91 4: cgroup.procs: 5: pids.current: 91 5: cgroup.procs: 6: pids.current: 91 6: cgroup.procs: 7: pids.current: 91 7: cgroup.procs: 8: pids.current: 91 8: cgroup.procs: 9: pids.current: 91 9: cgroup.procs: 10: pids.current: 91 10: cgroup.procs: 11: pids.current: 91 11: cgroup.procs: [root@xps test]# uname -a Linux xps 5.0.0-rc3-00104-gc04e2a780caf #288 SMP PREEMPT Thu Jan 24 19:00:32 CET 2019 x86_64 Intel(R)_Core(TM)_i9-8950HK_CPU_@_2.90GHz PLD Linux cc relevant people script is here: https://www.spinics.net/lists/cgroups/msg21330.html > > [root@warm ~]# uname -a > Linux warm 4.20.3-arch1-1-ARCH #1 SMP PREEMPT Wed Jan 16 22:38:58 UTC > 2019 x86_64 GNU/Linux > [root@warm ~]# python3 cg.py > Created cgroup: /sys/fs/cgroup/test_26207 > Start: pids.current: 0 > Start: cgroup.procs: > 0: pids.current: 62 > 0: cgroup.procs: > 1: pids.current: 37 > 1: cgroup.procs: > 2: pids.current: 37 > 2: cgroup.procs: > 3: pids.current: 37 > 3: cgroup.procs: > 4: pids.current: 37 > 4: cgroup.procs: > 5: pids.current: 37 > 5: cgroup.procs: > 6: pids.current: 37 > 6: cgroup.procs: > 7: pids.current: 37 > 7: cgroup.procs: > 8: pids.current: 37 > 8: cgroup.procs: > 9: pids.current: 37 > 9: cgroup.procs: > 10: pids.current: 37 > 10: cgroup.procs: > 11: pids.current: 37 > 11: cgroup.procs: > -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )