From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CAC124887E for ; Fri, 30 Jan 2026 03:53:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769745200; cv=none; b=GJYXIg4iKtJGuhdw53JifMYu3EDi4ChFpUAC6d+aUKjYDUALIufR4DdjRtVcYELAc/CyUWtcXcIZOYLP/PtUrugBSSzi8Mse0pDPJWF+dhpsjcb3NED/UXv9Vf1DqHYfQ/LEdErdw8Mp8F9z0XErtSZRT5fb2kIxZfX+NxLyy24= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769745200; c=relaxed/simple; bh=YUglP/1o0VFlP49CUtC5odsiRK0KtdguVL0uUKW80B0=; h=From:Message-ID:Date:MIME-Version:Subject:To:Cc:References: In-Reply-To:Content-Type; b=p7fR0cR5tCVTeiUVnQEidox76WxOK89r46VUqCO3ARj5uSUFjPS4+C+77wVkwz7t7qWFs2ZLICnxJdxQTVIiPPL8ky7LtDRkUOGTzApzZE5JwG29p7LM7g5H+qIoZ6o70sy7RYcRuff0hlaXa8cRNsLfeg6CzdPEp3kicugYC7E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=KEnM5kcE; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=MwozaLmu; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KEnM5kcE"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="MwozaLmu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1769745197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/VgU+9mjJcQ6t3VpK1I61TTGyW0HQRG6T8xqgtO3yoQ=; b=KEnM5kcE8UZ01JEN+8wl0Z5gZDPcrBxKWmSuYH0eXsZ0D8Q79yLHqoJNQTqFxbituiqvhd huPGpddyLl23T6J25lizJpAhSJwieZItLG1VM0IbCmuKBYaSxZIiKEh2owzw/D6h4fIwts M74R8nBow0Gsgf3hinGgcezSzxU+GbI= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-402-DvwHkbpDN3eFNNg243AoeA-1; Thu, 29 Jan 2026 22:53:14 -0500 X-MC-Unique: DvwHkbpDN3eFNNg243AoeA-1 X-Mimecast-MFC-AGG-ID: DvwHkbpDN3eFNNg243AoeA_1769745194 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-5014b22d678so24181621cf.0 for ; Thu, 29 Jan 2026 19:53:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1769745193; x=1770349993; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from:from:to :cc:subject:date:message-id:reply-to; bh=/VgU+9mjJcQ6t3VpK1I61TTGyW0HQRG6T8xqgtO3yoQ=; b=MwozaLmuSUIeBWHes+EVkydppBp9mBXyy4UVSCj/2sJ3qHvw/LYZtkBLMbEPa67/4B wOgCoenocYDjbRI5Y5pfS4seXA7f52otShnOVTv8bCKK3A/QSUVUBk0+biJpHhn67W0x I861x1lTGRZ5Ntv+EDTWi3eK3V7TNAlcKT/k1UojS61I8DgO8D/h1SZpNtuG8yYoUFjn 13IH/BRqaVcoDaxyd/DMDcoHDELyCFhik7FzfjaonZPvF0t9Kh8Q5HHFXOCruYr3Rk1L knZ0CFF1CDfKQzTFCMCmL4Q8bTwTmmFQPwnFZwpMAHjx2Hny6HNSOqJPL5LnojkgyQu6 U0Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769745193; x=1770349993; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/VgU+9mjJcQ6t3VpK1I61TTGyW0HQRG6T8xqgtO3yoQ=; b=sx5SdPGS1Vxwcfc9UA/ipzpLGJzT1PjJ1FAHV33JZNEq94bgb5kS5auWA4WIIPqqhH H874OKigJGIGSQmxEhYE/9qIdZvhGqW40f9775pCnokunlznVmNSjoD3MYd+4ZhYCnlJ 2FciGyPHe4WrT+dukW/9nJuoozrKDCT23bvGhfzQIAB7fKgMs8uJD9h+28G+iZafvzyQ 9mERsEk03xubor9ueLnSDvWAu6a7/Ss5JTFRxsqeR0pliFB4Hj062QXs3gU7DXxXhmDv OjESo37vVbxmZI6x4HHc3+CUZQWMimPZ+JZSOEwj89rJz8EAvNYQXGmzJtKvLqi9XB9c dv/Q== X-Forwarded-Encrypted: i=1; AJvYcCXSga7MbCaWdvwayKdKrolNo4YEipfkSwDtP1vgfKARAue/zlU0o6fCtiibcK9257W+0ROP/4dpKYoPCeY=@vger.kernel.org X-Gm-Message-State: AOJu0YzealHj8RfZa3/tDAbWEftToYNTFI4vPgRJ3n/82sUriYAcrv77 h8SseHfdPkmGmEUxYl5f+vhhZUU8x7sAMLWQnBLaw24NhgHoVY3sST9OvOgyK+8YtTGpfHEPLVD nr84QB6i1m4iPKO3A3OHTnWTei9GAtESniSF+v3bAPQnEDS0nhn7fG7HHa2Uv3Ypvlw== X-Gm-Gg: AZuq6aIta9p3Zf3pe8Om+Gq7fPqoS7zlACuoQ9uAXIunABTp9azV4RCg4rDx9b53TBz NIVNxhsqa36iQdVbbNWar5EdIZOmS0yZlvnGB85/V0wLynxTbhWkFJGU4nmFhpEkWzED/FvlJmV 0l/5ooNIIcvE4Me2TMm/mJtP7TXdFL9Eve+ayv68m0zUa6KIFIY3E3UFw/tpZma8St/zACOIdv2 OFtrT95ZE3BJs3KTxBWrEi3bwL1KDEnBhe2DIqYp7zOvj0ksedhAggGUPkHWKzGDuca9kzJ0RtP SygZif9T8qVINfodQJ/5ZjQS7ohR22S+kquveUeRm6CRkzF4ILb1FzZWk4p/0WyAxEcqU0VIoTP pZvaTOOEnpAOb5EKM0OR0zVfRVY6ncc6Q5r5kFwW6cNpcFeobb99dtR3b X-Received: by 2002:ac8:7dc4:0:b0:4ec:f477:60e9 with SMTP id d75a77b69052e-505d22c8c79mr22427531cf.76.1769745193524; Thu, 29 Jan 2026 19:53:13 -0800 (PST) X-Received: by 2002:ac8:7dc4:0:b0:4ec:f477:60e9 with SMTP id d75a77b69052e-505d22c8c79mr22427251cf.76.1769745193054; Thu, 29 Jan 2026 19:53:13 -0800 (PST) Received: from ?IPV6:2601:188:c102:b180:1f8b:71d0:77b1:1f6e? ([2601:188:c102:b180:1f8b:71d0:77b1:1f6e]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-894d375debasm51663666d6.42.2026.01.29.19.53.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 29 Jan 2026 19:53:12 -0800 (PST) From: Waiman Long X-Google-Original-From: Waiman Long Message-ID: Date: Thu, 29 Jan 2026 22:53:10 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH/for-next 2/2] cgroup/cpuset: Introduce a new top level isolcpus_update_mutex To: Chen Ridong , Waiman Long , Tejun Heo , Johannes Weiner , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , Shuah Khan Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org References: <20260128044251.1229702-1-longman@redhat.com> <20260128044251.1229702-3-longman@redhat.com> <08c3fad6-b881-4089-b081-bde6efbafbd2@huaweicloud.com> <11d8ff97-74e5-440e-b56a-af590da5a3f6@huaweicloud.com> <80842353-e054-4c70-a560-f67401c5b4a2@redhat.com> <7a67b419-1f94-441d-9d15-66dce03f9268@huaweicloud.com> Content-Language: en-US In-Reply-To: <7a67b419-1f94-441d-9d15-66dce03f9268@huaweicloud.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 1/29/26 8:42 PM, Chen Ridong wrote: > > On 2026/1/30 9:35, Waiman Long wrote: >> On 1/29/26 7:56 PM, Chen Ridong wrote: >>> On 2026/1/30 5:16, Waiman Long wrote: >>>> On 1/29/26 3:01 AM, Chen Ridong wrote: >>>>> On 2026/1/28 12:42, Waiman Long wrote: >>>>>> The current cpuset partition code is able to dynamically update >>>>>> the sched domains of a running system and the corresponding >>>>>> HK_TYPE_DOMAIN housekeeping cpumask to perform what is essentally the >>>>>> "isolcpus=domain,..." boot command line feature at run time. >>>>>> >>>>>> The housekeeping cpumask update requires flushing a number of different >>>>>> workqueues which may not be safe with cpus_read_lock() held as the >>>>>> workqueue flushing code may acquire cpus_read_lock() or acquiring locks >>>>>> which have locking dependency with cpus_read_lock() down the chain. Below >>>>>> is an example of such circular locking problem. >>>>>> >>>>>>     ====================================================== >>>>>>     WARNING: possible circular locking dependency detected >>>>>>     6.18.0-test+ #2 Tainted: G S >>>>>>     ------------------------------------------------------ >>>>>>     test_cpuset_prs/10971 is trying to acquire lock: >>>>>>     ffff888112ba4958 ((wq_completion)sync_wq){+.+.}-{0:0}, at: >>>>>> touch_wq_lockdep_map+0x7a/0x180 >>>>>> >>>>>>     but task is already holding lock: >>>>>>     ffffffffae47f450 (cpuset_mutex){+.+.}-{4:4}, at: >>>>>> cpuset_partition_write+0x85/0x130 >>>>>> >>>>>>     which lock already depends on the new lock. >>>>>> >>>>>>     the existing dependency chain (in reverse order) is: >>>>>>     -> #4 (cpuset_mutex){+.+.}-{4:4}: >>>>>>     -> #3 (cpu_hotplug_lock){++++}-{0:0}: >>>>>>     -> #2 (rtnl_mutex){+.+.}-{4:4}: >>>>>>     -> #1 ((work_completion)(&arg.work)){+.+.}-{0:0}: >>>>>>     -> #0 ((wq_completion)sync_wq){+.+.}-{0:0}: >>>>>> >>>>>>     Chain exists of: >>>>>>       (wq_completion)sync_wq --> cpu_hotplug_lock --> cpuset_mutex >>>>>> >>>>>>     5 locks held by test_cpuset_prs/10971: >>>>>>      #0: ffff88816810e440 (sb_writers#7){.+.+}-{0:0}, at: >>>>>> ksys_write+0xf9/0x1d0 >>>>>>      #1: ffff8891ab620890 (&of->mutex#2){+.+.}-{4:4}, at: >>>>>> kernfs_fop_write_iter+0x260/0x5f0 >>>>>>      #2: ffff8890a78b83e8 (kn->active#187){.+.+}-{0:0}, at: >>>>>> kernfs_fop_write_iter+0x2b6/0x5f0 >>>>>>      #3: ffffffffadf32900 (cpu_hotplug_lock){++++}-{0:0}, at: >>>>>> cpuset_partition_write+0x77/0x130 >>>>>>      #4: ffffffffae47f450 (cpuset_mutex){+.+.}-{4:4}, at: >>>>>> cpuset_partition_write+0x85/0x130 >>>>>> >>>>>>     Call Trace: >>>>>>      >>>>>>        : >>>>>>      touch_wq_lockdep_map+0x93/0x180 >>>>>>      __flush_workqueue+0x111/0x10b0 >>>>>>      housekeeping_update+0x12d/0x2d0 >>>>>>      update_parent_effective_cpumask+0x595/0x2440 >>>>>>      update_prstate+0x89d/0xce0 >>>>>>      cpuset_partition_write+0xc5/0x130 >>>>>>      cgroup_file_write+0x1a5/0x680 >>>>>>      kernfs_fop_write_iter+0x3df/0x5f0 >>>>>>      vfs_write+0x525/0xfd0 >>>>>>      ksys_write+0xf9/0x1d0 >>>>>>      do_syscall_64+0x95/0x520 >>>>>>      entry_SYSCALL_64_after_hwframe+0x76/0x7e >>>>>> >>>>>> To avoid such a circular locking dependency problem, we have to >>>>>> call housekeeping_update() without holding the cpus_read_lock() >>>>>> and cpuset_mutex. One way to do that is to introduce a new top level >>>>>> isolcpus_update_mutex which will be acquired first if the set of isolated >>>>>> CPUs may have to be updated. This new isolcpus_update_mutex will provide >>>>>> the need mutual exclusion without the need to hold cpus_read_lock(). >>>>>> >>>>>> As cpus_read_lock() is now no longer held when >>>>>> tmigr_isolated_exclude_cpumask() is called, it needs to acquire it >>>>>> directly. >>>>>> >>>>>> The lockdep_is_cpuset_held() is also updated to check the new >>>>>> isolcpus_update_mutex. >>>>>> >>>>> I worry about the issue: >>>>> >>>>> CPU1                CPU2 >>>>> rmdir >>>>> css->ss->css_killed(css); >>>>> cpuset_css_killed >>>>>                  __update_isolation_cpumasks >>>>>                  cpuset_full_unlock >>>>> css->flags |= CSS_DYING; >>>>> css_clear_dir(css); >>>>> ... >>>>> // offline and free do not >>>>> // get isolcpus_update_mutex >>>>> cpuset_css_offline >>>>> cpuset_css_free >>>>>                  cpuset_full_lock >>>>>                  ... >>>>>                  // UAF? >>>>> >>> Hi, Longman, >>> >>> In this patch, I noticed that cpuset_css_offline and cpuset_css_free do not >>> acquire the isolcpus_update_mutex. This could potentially lead to a UAF issue. >>> >>>> That is the reason why I add a new top-level isolcpus_update_mutex. >>>> cpuset_css_killed() and the update_isolation_cpumasks()'s unlock/lock sequence >>>> will have to acquire this isolcpus_update_mutex first. >>>> >>> However, simply adding isolcpus_update_mutex to cpuset_css_killed and >>> update_isolation_cpumasks may not be sufficient. >>> >>> As I mentioned, the path that calls __update_isolation_cpumasks may first >>> acquire isolcpus_update_mutex and cpuset_full_lock, but once cpuset_css_killed >>> is completed, it will release the “full” lock and then attempt to reacquire it >>> later. During this intermediate period, the cpuset may have already been freed, >>> because cpuset_css_offline and cpuset_css_free do not currently acquire the >>> isolcpus_update_mutex. >> You are right that acquisition of the new isolcpus_update_mutex should be in all >> the places where cpuset_full_lock() is acquired. Will update the patch to do >> that. That should eliminate the risk. >> > I suggest that putting isolcpus_update_mutex into cpuset_full_lock, since this > function means that all the locks needed have been acquired. > > void cpuset_full_lock(void) > { > mutex_lock(&isolcpus_update_mutex); > cpus_read_lock(); > mutex_lock(&cpuset_mutex); > } > > void cpuset_full_unlock(void) > { > mutex_unlock(&cpuset_mutex); > cpus_read_unlock(); > mutex_unlock(&isolcpus_update_mutex); > } That is what I had done. Cheers, Longman > > In the __update_isolation_cpumasks function, we can pair: > > ``` > ... > mutex_unlock(&cpuset_mutex); > cpus_read_unlock(); > ... Actions > cpus_read_lock(); > mutex_lock(&cpuset_mutex); > ... > ``` >