From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F14A1C43334 for ; Mon, 27 Jun 2022 06:50:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232506AbiF0Gub (ORCPT ); Mon, 27 Jun 2022 02:50:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232320AbiF0Gua (ORCPT ); Mon, 27 Jun 2022 02:50:30 -0400 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78F3726C5; Sun, 26 Jun 2022 23:50:28 -0700 (PDT) Received: from dggpeml500022.china.huawei.com (unknown [172.30.72.55]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4LWdYs5GQtzShYT; Mon, 27 Jun 2022 14:46:57 +0800 (CST) Received: from dggpeml500018.china.huawei.com (7.185.36.186) by dggpeml500022.china.huawei.com (7.185.36.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 27 Jun 2022 14:50:26 +0800 Received: from [10.67.111.186] (10.67.111.186) by dggpeml500018.china.huawei.com (7.185.36.186) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 27 Jun 2022 14:50:25 +0800 Message-ID: <5987be34-b527-4ff5-a17d-5f6f0dc94d6d@huawei.com> Date: Mon, 27 Jun 2022 14:50:25 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.1.1 From: Zhang Qiao Subject: [Question] The system may be stuck if there is a cpu cgroup cpu.cfs_quato_us is very low To: Tejun Heo , , , Juri Lelli , Vincent Guittot CC: , , , lkml , , , , , Steven Rostedt , Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.67.111.186] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpeml500018.china.huawei.com (7.185.36.186) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi all, I'm working on debuging a problem. The testcase does follew operations: 1) create a test task cgroup, set cpu.cfs_quota_us=2000,cpu.cfs_period_us=100000. 2) run 20 test_fork[1] test process in the test task cgroup. 3) create 100 new containers: for i in {1..100}; do docker run -itd --health-cmd="ls" --health-interval=1s ubuntu:latest bash; done These operations are expected to succeed and 100 containers create success. however, when creating containers, the system will get stuck and create container failed. After debug this, I found the test_fork process frequently sleep in freezer_fork()->mutex_lock()->might_sleep() with taking the cgroup_threadgroup_rw_sem lock, as follow: copy_process(): cgroup_can_fork() ---> lock cgroup_threadgroup_rw_sem sched_cgroup_fork(); ->task_fork_fair(){ ->update_curr(){ ->__account_cfs_rq_runtime() { resched_curr(); ---> the quota is used up, and set flag TIF_NEED_RESCHED to current } cgroup_post_fork(); ->feezer_fork() ->mutex_lock() { ->might_sleep() ---> schedule() and the current task will be throttled long time. ->cgroup_css_set_put_fork() ---> unlock cgroup_threadgroup_rw_sem Becuase the task cgroup's cpu.cfs_quota_us is very small and test_fork's load is very heavy, the test_fork may be throttled long time, therefore, the cgroup_threadgroup_rw_sem read lock is held for a long time, other processes will get stuck waiting for the lock: 1) a task fork child, will wait at copy_process()->cgroup_can_fork(); 2) a task exiting will wait at exit_signals(); 3) a task write cgroup.procs file will wait at cgroup_file_write()->__cgroup1_procs_write(); ... even the whole system will get stuck. Anyone know how to slove this? Except for changing the cpu.cfs_quota_us. [1] test_fork.c #include #include #include #include int main(int argc, char **argv) { pid_t pid; int count = 20; while(1) { for (int i = 0; i < count; i++) { if ((pid = fork()) <0) { printf("fork error"); return 1; } else if (pid ==0) { exit(0); } } for (int i = 0; i < count; i++) { wait(NULL); } sleep(1); } return 0; } Thanks a lot. -Qiao -