From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from crocodile.elm.relay.mailchannels.net (crocodile.elm.relay.mailchannels.net [23.83.212.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8236314B90 for ; Mon, 17 Nov 2025 18:59:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=23.83.212.45 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405997; cv=pass; b=Tc93+g6sttxOFbGb6EZJtJJ4NPlc7sbM0iFOfTqXaboOEagk7ssqkpNloySkAxWnIn/1NIlaxeROo/lFLXXgo+11qLo2Wjl//X/5DcDPPXt8PwY4z7HsXKjn3/pfSKWWmKs6xGiRli3c7F622d1zqY5w0T6Yy70dYNWm7cqn+vo= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405997; c=relaxed/simple; bh=xcq71eQ31zpkCarpPxBw4HtpxF7qs+P9eGV2XoGTxQo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=sSzJ7+cDJgpS9gHqUqBZglrU8p1vi3a+g+GN8wMGKYF2gs9uCr3uSMZuEuLzJzH3hIJYTG4GOwuyf1OTNEKJmqxmzidYvWiU0wVEmoz1ZwIFde7JG8Xr6M9eR9DRMFk5sXemZG3iIWxRM72iEVrAhlef5uyVcSf/sn9UbkL5nPA= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=templeofstupid.com; spf=pass smtp.mailfrom=templeofstupid.com; dkim=pass (2048-bit key) header.d=templeofstupid.com header.i=@templeofstupid.com header.b=PNBIN+e3; arc=pass smtp.client-ip=23.83.212.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=templeofstupid.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=templeofstupid.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=templeofstupid.com header.i=@templeofstupid.com header.b="PNBIN+e3" X-Sender-Id: dreamhost|x-authsender|kjlx@templeofstupid.com Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id ABDD35612A6 for ; Mon, 17 Nov 2025 18:59:48 +0000 (UTC) Received: from pdx1-sub0-mail-a235.dreamhost.com (trex-green-6.trex.outbound.svc.cluster.local [100.98.23.58]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 4587E56132A for ; Mon, 17 Nov 2025 18:59:48 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; d=mailchannels.net; s=arc-2022; cv=none; t=1763405988; b=JegE/8T6dQ6aaoP5hLikqHaLvaMzthwcmIu3nUF4441f9uJiPPT1hJnPFDRLcp1YRjPtxT wyL8ageGZ1Dl22+PpPigD7g6wx6sLaN19XUwNyMxG08sy5PyXmcgEIrYHNH/HyhQgkV6L+ 47Lq/MAdAmW5N8RUmWJaJfYGw91llvWmlIFuLxjr0aa0oNaq3cSJcJTs9z+DU0uPKjljap i8r6DJECvIWw6lU0Lbd2pkJlTRUyFo9vdaua7ZFwJrCMeyuALAk3Zzw8TKUCPkZHeNZWii 3lIVsGgY8misTsqHdf1vAUtIfctnoeaSIzT+DZHcw+I5j28OoVUEp1Kh3KUPLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1763405988; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9ojL4yqdUIrcGLfcuY0kbGMHQib0KvDTiqeQJOpH6BI=; b=tXmrlnRdG3v5wXzzFaieidIgzsSDCAVXggqSdS6BmjdEOMjF+CjpyNblYDPNACyvIzSze2 dt1DoSozr2wxdT6glSV3BL3kdHp+vfZqeu6EwNBqPPH+FPyLEQ8PIZsoSlUoeAKG7iEkoD jwCPSr/ZshMxnbW3GeSFTSyVbdYJVohecX2HKG+TQLMN+RQM6Jkx6GP4iUwfC+B2pJqlPi vM7vS6xWFLmiR+WVtcEuuA50sjcWXgQuh/ardQQ3O2YO6Q9L9KFuATy8RqDhKW5JrnUxvR YTCL9c0cR2ASbgYEZqFkxJ8qhSB7cE8DgCJYt5BgnouQhuXDqKc8CwaYO4psDA== ARC-Authentication-Results: i=1; rspamd-5f77df855-thpzq; auth=pass smtp.auth=dreamhost smtp.mailfrom=kjlx@templeofstupid.com X-Sender-Id: dreamhost|x-authsender|kjlx@templeofstupid.com X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|kjlx@templeofstupid.com X-MailChannels-Auth-Id: dreamhost X-Squirrel-Relation: 4019d5fb3556800e_1763405988497_1396359076 X-MC-Loop-Signature: 1763405988497:2112906658 X-MC-Ingress-Time: 1763405988497 Received: from pdx1-sub0-mail-a235.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.98.23.58 (trex/7.1.3); Mon, 17 Nov 2025 18:59:48 +0000 Received: from kmjvbox.templeofstupid.com (c-73-231-224-43.hsd1.ca.comcast.net [73.231.224.43]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: kjlx@templeofstupid.com) by pdx1-sub0-mail-a235.dreamhost.com (Postfix) with ESMTPSA id 4d9H9b6qFKz107t for ; Mon, 17 Nov 2025 10:59:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=templeofstupid.com; s=dreamhost; t=1763405988; bh=9ojL4yqdUIrcGLfcuY0kbGMHQib0KvDTiqeQJOpH6BI=; h=Date:From:To:Cc:Subject:Content-Type:Content-Transfer-Encoding; b=PNBIN+e3tNAZ4S6kMJLBt4mY39RqooBrHt7DHUWYwlSEJlJCnyk+6P1BnEQZveUHE O7zvFG8TSnp9CLfyJL4WJXR9s97xzXefb/JTMkTO9h58G4hOg3tUqEbUpdFC105Hg+ 5jlKHcNi3zEQyVEH4+Yajt/FLj3C8RCmgnzdYH1tYyZH0hcH9EbJTIk1cuTPSr+9Qv 7pjKkKICgcC7qA+cOSlASOKum0/aEcXSRekbWtBXzC0FmFHVDZREZXBss6iIUzKTqm 4fxg7mAS/lsZMUZZRet7IKFGpSo3pxfcdmvfBgmFHU3FkEhcN5V+FKsk7Zvls+sYav GRi1LDPfCq3hw== Received: from johansen (uid 1000) (envelope-from kjlx@templeofstupid.com) id e0108 by kmjvbox.templeofstupid.com (DragonFly Mail Agent v0.13); Mon, 17 Nov 2025 10:59:47 -0800 Date: Mon, 17 Nov 2025 10:59:47 -0800 From: Krister Johansen To: Peng Wang Cc: Peter Zijlstra , mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, vdavydov.dev@gmail.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] sched/fair: Clear ->h_load_next after hierarchical load Message-ID: References: <20251015124422.GD3419281@noisy.programming.kicks-ass.net> <20251016030617.GA46570@j38d01266.eu95sqa> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20251016030617.GA46570@j38d01266.eu95sqa> On Thu, Oct 16, 2025 at 11:06:17AM +0800, Peng Wang wrote: > On Wed, Oct 15, 2025 at 02:44:22PM +0200, Peter Zijlstra wrote: > > On Wed, Oct 15, 2025 at 08:19:50PM +0800, Peng Wang wrote: > > > > > We found that the task_group corresponding to the problematic se > > > is not in the parent task_group’s children list, indicating that > > > h_load_next points to an invalid address. Consider the following > > > cgroup and task hierarchy: > > > > > > A > > > / \ > > > / \ > > > B E > > > / \ | > > > / \ t2 > > > C D > > > | | > > > t0 t1 > > > > > > Here follows a timing sequence that may be responsible for triggering > > > the problem: > > > > > > CPU X CPU Y CPU Z > > > wakeup t0 > > > set list A->B->C > > > traverse A->B->C > > > t0 exits > > > destroy C > > > wakeup t2 > > > set list A->E wakeup t1 > > > set list A->B->D > > > traverse A->B->C > > > panic > > > > > > CPU Z sets ->h_load_next list to A->B->D, but due to arm64 weaker memory > > > ordering, Y may observe A->B before it sees B->D, then in this time window, > > > it can traverse A->B->C and reach an invalid se. > > > > Hmm, I rather think we should ensure update_cfs_rq_h_load() is > > serialized against unregister_fair_sched_group(). > > I might be mistaken, but it seems that, even with RCU protection around > update_cfs_rq_h_load(), there remains a risk of reading stale values. > > > CPU X CPU Y CPU Z > > wakeup t0 > rcu_read_lock() > set list A->B->C > traverse A->B->C > rcu_read_unlock() > t0 exits > destroy C > > After the prior RCU grace period has elapsed, C has already been reclaimed, > yet the stale A->B->C remains. > > > wakeup t2 > rcu_read_lock() > set list A->E wakeup t1 > rcu_read_lock() > set list A->B->D > ... > traverse A->B->C > panic > > A subsequent rcu_read_lock() only guarantees that A/B/D/E will not be > reclaimed while the list is being traversed; C had already been freed > before the next grace period even began. FWIW, I've caught arm64 machines running into this problem recently on 6.x kernels. These particular systems are small enough that they have just a single memory node and no NUMA balancing enabled. Would the scheduling experts be willing to consider picking up Peng's fix while the 6.18 release is still open for bug fixes? -K