From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7702536493A for ; Sun, 22 Mar 2026 16:04:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774195491; cv=none; b=aQ05n2ieDjYNiS2lZor4bDh+zU3lKOIf0P6Gkap51f9LegfzFhuY4tuApJaImYudgzseQZkM5kDcFukrsRlLYWcfKPxlS4mmIAcn93OnMsxvLYbm6gFAgXKwWuetdYoL+vlDbWFTysV38+aY9XDsAtTkF83Avp+J+ERvrh318mw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774195491; c=relaxed/simple; bh=msvWOkN8xeAp8rUUOCDtIk18g+kiAsS+eYq4uYInZpk=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=lWFoSgR+dP15d0Ap1gRM+EmS+bgTi0dZXXELweWqV8J/MKBgSKLTjo6H1kuUGBq4B1NJBhqnHcO+n/LFiRzr/N6aaoSmoDVbk+nLHJIMYn1lb7SdjPQXSt8mUK/9nLw/fnvLTCFCwKv6WZUUFkK6x2q8fUAaWZ4QgOh9MYQq/EQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=h7YmQNKP; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="h7YmQNKP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1774195489; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XydlhdtzVdpdIVOeFW9cZPWTu8yF8byS5ERN2p7mlIo=; b=h7YmQNKPSrCeUyoj8dO8DALiN5bae865JHC1SL63z9kF0ZqopSlu5rRFgQNFD1+PjAkRIm porizy6HlW5GXw6JX/OCws+KR7zRu2BFOV8fmaqtFUu6B5tPkUbDb+m+Brxe7dKxksDv3w qhpPB1DUCAVnjAyiLi2/1aOJJm1vuMQ= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-633-Rr-j12j6OsaSbwPIEuRydA-1; Sun, 22 Mar 2026 12:04:45 -0400 X-MC-Unique: Rr-j12j6OsaSbwPIEuRydA-1 X-Mimecast-MFC-AGG-ID: Rr-j12j6OsaSbwPIEuRydA_1774195484 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 789D91800464; Sun, 22 Mar 2026 16:04:44 +0000 (UTC) Received: from [10.22.88.38] (unknown [10.22.88.38]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DEDAB1955F21; Sun, 22 Mar 2026 16:04:43 +0000 (UTC) Message-ID: <98b598f7-cbc6-42bd-bd01-d077549628fd@redhat.com> Date: Sun, 22 Mar 2026 12:04:43 -0400 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a To: sheviks , frederic@kernel.org Cc: linux-kernel@vger.kernel.org References: Content-Language: en-US From: Waiman Long In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 On 3/21/26 12:44 AM, sheviks wrote: > Hi Frederic and maintainers, > > I’m reaching out to discuss a change in kthread affinity behavior that > seems to be a regression for users relying on dynamic CPU isolation. > This started appearing after commit 041ee6f3727a ("kthread: Rely on > HK_TYPE_DOMAIN for preferred affinity management"). > > The Problem: > In my setup, I use nohz_full but intentionally avoid the deprecated > isolcpus= boot parameter. Instead, I use cgroup v2 > (cpuset.cpus.partition=isolated) to dynamically isolate CPUs after the > system has booted. > > The commit 041ee6f3727a changed kthreads to rely on HK_TYPE_DOMAIN. > However, since isolcpus= is not used, HK_TYPE_DOMAIN defaults to all > CPUs at boot time. Even after I later configure cgroups to isolate > CPUs 1-7, unbound kthreads (including kthreadd) remain on those > nohz_full CPUs. Frederic's patch series is supposed to make HK_TYPE_DOMAIN cpumask to dynamically exclude cpuset isolated CPUs. Then those unbound kthreads are supposed to be modified to remove these CPUs from their cpumasks. If that is not happening, it will be a problem we need to look at. Cheers, Longman > > It seems the assumption that "nohz_full implies domain isolation" only > holds true if isolation is statically defined at boot via isolcpus=. > For dynamic isolation via cgroups, HK_TYPE_KTHREAD and HK_TYPE_DOMAIN > no longer cover the same set of CPUs. > > System Log: > Here is the state of my system after setting up the cgroup isolation: > > $ uname -r > 7.0.0-rc4-1-rt > > # 1. Boot parameters (No isolcpus) > $ grep -oe "nohz_full=[^ ]*" -e "rcu_nocbs=[^ ]*" /proc/cmdline > nohz_full=1,2,3,4,5,6,7 > rcu_nocbs=1,2,3,4,5,6,7 > > # 2. Cgroup v2 Isolation is active > $ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.exclusive > 1-7 > $ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.partition > isolated > $ cat /sys/fs/cgroup/cpuset.cpus.effective > 0 > $ cat /sys/fs/cgroup/cpuset.cpus.isolated > 1-7 > > # 3. Unbound kthreads are still "trapped" on isolated/nohz_full CPUs > $ ps -eLo cpuid,comm | grep -e COMM -e "^ *[1-7] " | grep -ve > "/[1-7]$" -e "kworker/[1-7]:" | head > CPUID COMMAND > 4 pool_workqueue_release > 1 pr/legacy > 4 rcu_exp_gp_kthread_worker > 1 kdevtmpfs > 5 oom_reaper > 1 ksmd > 7 watchdogd > 7 kswapd0 > 6 scsi_eh_0 > > Questions: > 1. Is this an intended change that mandates the use of isolcpus= for > kthread exclusion? > > 2. If we prefer dynamic isolation via cgroup v2, is there a > recommended way to "refresh" or move these unbound kthreads once the > housekeeping mask changes at runtime? > > 3. Or should HK_TYPE_KTHREAD still be considered separately from > HK_TYPE_DOMAIN to account for nohz_full users without isolcpus=? > > I would appreciate any insights or suggestions you might have. > > Best regards, > Sheviks > > > 乾淨無病毒。www.avast.com > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >