From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98753CDB482 for ; Fri, 13 Oct 2023 16:04:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232507AbjJMQEQ (ORCPT ); Fri, 13 Oct 2023 12:04:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50858 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232371AbjJMQEN (ORCPT ); Fri, 13 Oct 2023 12:04:13 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63FC7BF for ; Fri, 13 Oct 2023 09:03:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1697213003; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nqf6Wos6j/dOB8iX9uZ+sliE7IdSLRs7IhOwsFPasV8=; b=VODI7twvozOQjvrYoVl3L/C37UefnbihSBANHao4JKIwX7V0sFsOrXMHNMvpy7cNDpoGOE BzqayvxV7Nkk714OaXlwXcD4CoCvcN2xt3HzKUJla/HeLOeT8vpohf2qBRVnmMxtnt8vYy S7bp0GjdWzdXZ7lkDn6BLijzoJ4259M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-694-KMMaU5QkNK6QYcars7KklA-1; Fri, 13 Oct 2023 12:03:20 -0400 X-MC-Unique: KMMaU5QkNK6QYcars7KklA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3D804887E6B; Fri, 13 Oct 2023 16:03:19 +0000 (UTC) Received: from [10.22.17.138] (unknown [10.22.17.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 78D7D1C06535; Fri, 13 Oct 2023 16:03:18 +0000 (UTC) Message-ID: Date: Fri, 13 Oct 2023 12:03:18 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: [PATCH v8 0/7] cgroup/cpuset: Support remote partitions Content-Language: en-US To: =?UTF-8?Q?Michal_Koutn=c3=bd?= Cc: Tejun Heo , Zefan Li , Johannes Weiner , Christian Brauner , Jonathan Corbet , Shuah Khan , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Juri Lelli , Dietmar Eggemann , Giuseppe Scrivano References: <20230905133243.91107-1-longman@redhat.com> From: Waiman Long In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/13/23 11:50, Michal Koutný wrote: > Hello. > > (I know this is heading for 6.7. Still I wanted to have a look at this > after it stabilized somehow to understand the new concept better but I > still have some questions below.) > > On Tue, Sep 05, 2023 at 09:32:36AM -0400, Waiman Long wrote: >> Both scheduling and isolated partitions can be formed as a remote >> partition. A local partition can be created under a remote partition. >> A remote partition, however, cannot be formed under a local partition >> for now. >> >> >> With this patch series, we allow the creation of remote partition >> far from the root. The container management tool can manage the >> "cpuset.cpus.exclusive" file without impacting the other cpuset >> files that are managed by other middlewares. Of course, invalid >> "cpuset.cpus.exclusive" values will be rejected. > I take the example with a nested cgroup `cont` to which I want to > dedicate two CPUs (0 and 1). > IIUC, I can do this both with a chain of local root partitions or as a > single remote partion. > > > [chain] > root > | \ > mid1a mid1b > cpuset.cpus=0-1 cpuset.cpus=2-15 > cpuset.cpus.partition=root > | > mid2 > cpuset.cpus=0-1 > cpuset.cpus.partition=root > | > cont > cpuset.cpus=0-1 > cpuset.cpus.partition=root In this case, the effective CPUs of both mid1a and mid2 will be empty. IOW, you can't have any task in these 2 cpusets. > > [remote] > root > | \ > mid1a mid1b > cpuset.cpus.exclusive=0-1 cpuset.cpus=2-15 > | > mid2 > cpuset.cpus.exclusive=0-1 > | > cont > cpuset.cpus.exclusive=0-1 > cpuset.cpus.partition=root > > In the former case I must configure cpuset.cpus and > cpuset.cpus.partition along the whole path and in the second case > cpuset.cpus.exclusive still along the whole path and root at the bottom > only. > > What is the difference between the two configs above? > (Or can you please give an example where the remote partitions are > better illustrated?) For the remote case, you can have intermediate tasks in both mid1a and mid2 as long as cpuset.cpus contains more CPUs than cpuset.cpus.exclusive. > >> Modern container orchestration tools like Kubernetes use the cgroup >> hierarchy to manage different containers. And it is relying on other >> middleware like systemd to help managing it. If a container needs to >> use isolated CPUs, it is hard to get those with the local partitions >> as it will require the administrative parent cgroup to be a partition >> root too which tool like systemd may not be ready to manage. > Such tools ready aren't ready to manage cpuset.cpus.exclusive, are they? > IOW tools need to distinguish exclusive and "shared" CPUs which is equal > to distinguishing root and member partitions. They will be ready eventually. This requirement of remote partition actually came from our OpenShift team as the use of just local partition did not meet their need. They don't need access to exclusive CPUs in the parent cgroup layer for their management daemons. They do need to activate isolated partition in selected child cgroups to support our Telco customers to run workloads like DPDK. So they will add the support to upstream Kubernetes. Cheers, Longman