From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from PH0PR06CU001.outbound.protection.outlook.com (mail-westus3azon11011003.outbound.protection.outlook.com [40.107.208.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE5E13612DB for ; Wed, 21 Jan 2026 05:24:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.208.3 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768973065; cv=fail; b=V8Uxh13CZYn4SPbKwN3KV3oQTOfvqODoUN+FA54XQc0TGVRMDP2mu52ecBDbQ1PacK2bd0Ut+GwkRS5+Ltm2AB+h3E9goNn4V0Ue94WwD6dpOqnpyE6VbNCSEqkYC12Ry+UMOZ6Pt+ORAJVSxi0h9akUWZspcwQvTv2A4DbWPJk= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768973065; c=relaxed/simple; bh=DYGM96+nMEcEFm5OG4Cod2d0BFfzDToC4y3eSfGW7z0=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=Hs3RDUITgvkFJxhyo86Qo0nGAXKL2f80dF2iyq+bTPqh/UFRRs3wD1vz/3Tj/f0wD9G0lKBUHTB8Sonp83x2sN07S4Xi8syXMfJLCMwrMMlukf7lSM/QGr75UZ73eVT4bO4Gt8kaZmJn3XJoCzZVt7Q72Kh8rBOKEs4cbQr/5kg= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=sooZOD6C; arc=fail smtp.client-ip=40.107.208.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="sooZOD6C" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UtsF5JGq7WFF3ZaJEOBxqiZBSgMgwwHvCEf7i+GZrxa7IAdqXDEUF0/dVWUYtTUXz06BXOXYjtDrbjhE2ky2sj+7TITqgcnjD7nmPgJSK0e3tOMsenBakzbIafWjk/WfCxNELCbrWj2Siab1XLCf6Dp2lqT8g4WiLSDAnO7W2O7RqnZBaL+bz4JE7m77yKnV9ov2yiMOyL9TRw3+An563xcPvunHnKlu/XA766HnbLFnT/6BR7CYik1VKZBI3kUVxjM3Pdi+fcaGFEsCwGjRiDzccPI7vDElLNBci/wnG8Nj/IvhrAN3EGX27z5ImgPQceEzcSbvjgffPU6LotXftQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bldUi9JYzmIsqNTUmLJAgaW1MQLnYcLkxSBW7dHbXnE=; b=yOQW4wF9y5vlp6k1eSmr5pP5fQlbZ5f2ey+NpE/lEXh1hLX+UYg805PA98+gIuFRq9jihKM/D9t0voDXA8EZwoXh8f2p1DOB1ja1Wgt2lm24hnXUnphOo9NpIXutQwbQoTP9ZtLyGxu/Gk+/805lyupv9oPU1VsHRZFKpqxn+K/WkI0O9v8kTtN1Agb3DBBSCOJ1WYiNTzNDgewcz0S3ZSnSR+bFWROgBf5waj33LUJC9x/yyHtBFmq/CsJw9MK0yiNdEFtJrGb4i4sJDELGHmmZr368i7Hb0LVeEZg0q37jmvRgUDcYoHN65/fTSr/3DcnUNiSao+FwYMI47kPsPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=bytedance.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bldUi9JYzmIsqNTUmLJAgaW1MQLnYcLkxSBW7dHbXnE=; b=sooZOD6C9/k7/1Sf4k3iWm4yRlj5O+/7fZXEGIehwF6Z/BqEylw8wZhjVbN0AiyQvdPE5+LZMYquf9msgvyu8C2ubNb7r0Eo2Boowig0NA69xZcFMSNsLvn+uJ03C/VXkhY5oRQ/O92kozTaoYNUqMOFkCBxsAbyCqi5hS229vE= Received: from DS7PR03CA0057.namprd03.prod.outlook.com (2603:10b6:5:3b5::32) by LV3PR12MB9166.namprd12.prod.outlook.com (2603:10b6:408:19c::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9520.6; Wed, 21 Jan 2026 05:24:18 +0000 Received: from DS1PEPF00017090.namprd03.prod.outlook.com (2603:10b6:5:3b5:cafe::24) by DS7PR03CA0057.outlook.office365.com (2603:10b6:5:3b5::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9542.8 via Frontend Transport; Wed, 21 Jan 2026 05:24:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by DS1PEPF00017090.mail.protection.outlook.com (10.167.17.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.4 via Frontend Transport; Wed, 21 Jan 2026 05:24:17 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.2562.17; Tue, 20 Jan 2026 23:24:17 -0600 Received: from satlexmb08.amd.com (10.181.42.217) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 20 Jan 2026 23:24:17 -0600 Received: from [10.136.37.139] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Tue, 20 Jan 2026 23:24:12 -0600 Message-ID: Date: Wed, 21 Jan 2026 10:54:11 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] sched: Re-evaluate scheduling when migrating queued tasks out of throttled cgroups To: Aaron Lu , Zicheng Qu CC: , , , , , , , , , , , , , References: <20260120032549.186733-1-quzicheng@huawei.com> <20260121034918.GA1303836@bytedance.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260121034918.GA1303836@bytedance.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Received-SPF: None (SATLEXMB04.amd.com: kprateek.nayak@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF00017090:EE_|LV3PR12MB9166:EE_ X-MS-Office365-Filtering-Correlation-Id: 96b2e74a-f669-4002-0fff-08de58ad52d9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?utf-8?B?b25sLzhTbmZhTHFsMlN5R1hMNm9lcEFyMkJyVHpTRENEMFE4MlpKWWxTMGtH?= =?utf-8?B?TGNsbmZPYkxkaHVPTTdqaks5a3NJV2UwTUJqTVRnSllyWDJPSUpkdkZuOTBW?= =?utf-8?B?Rys5UHM3QlByTmJqUTlpYnJ5WUpCMVg5dVpBdzhuemFRTHloOUkvK3E5UVpX?= =?utf-8?B?TW5TMko5SGxQclJVRFNnNjcvUk1Kd0RMK1l0VStCVFkyd1Q4UE9XRUdIcSt2?= =?utf-8?B?ZmlSS3ZPT1RLRkY5bWxVWWhoZEMvRHpadHFGZ1gvcVpnTUhOUGgxZVVrRHJI?= =?utf-8?B?dUlnWnhES2xOSEs5MmhIdlIzU084MGg2eTlHQzBUc3JNU3pCa0pqa1l2QjBj?= =?utf-8?B?dWVjcjhPMkFFTGQ0bzN2Q2dhUm5HOHFzSnNrajlVYnBsb0VDalBCOXJUWkNU?= =?utf-8?B?VmI3RTlrZFdOOHlMMm01RUQ0V2dMZ2hVYitCS3pyaXAvNWpqb2lwMEltZVpJ?= =?utf-8?B?WTZmV0Q0c0lCWGFLVFRyVnI2U3VzbTZsY0o4MVZSTDRVNjdnTEtPcm1KVTFq?= =?utf-8?B?cThaUThLSDJvcGg4UUZ0QnNZYlpwMkluRlFWVEJlMVVoNVVmWkJNVVNmY3lV?= =?utf-8?B?bmlhYThkSFNCN3JlSkNTL2dvei9heTVuVEpwV1hseUh5dkR6bW05OGlTRGk3?= =?utf-8?B?OWtudnBEUzI5MmZYOTVST3F4VXFZa0I4YVM4QWZhTnRSSmVqckZCRFFGZ2N1?= =?utf-8?B?eWpyaGhqank0UmNSZktnVE4xMmJqR21uKzFCL3VZQzV3MDRBWVcxbzdFR1RH?= =?utf-8?B?Qm1kWkQxRFpBTCswVDBJWnRSVURFTkdQZTRXdnEvMHd6ZkxNUTRkb21EeGY2?= =?utf-8?B?MTlGT2t1Q0FLWWNlZWtKVXpxRi9KdFdFQUFmNy9CaythcTE1cFFDdXFsS2Z0?= =?utf-8?B?ZHpXWTZxbnI1VmxOOCtUUDUwc2h6OHJBYUFrM3BTRDZXeHRITytQWXBjZjNY?= =?utf-8?B?ZUJNdmVyTEhTV1BxVkxwMUZnRGVMeGJIYjZlZnBpOStVTlNWWXpuR003NmNX?= =?utf-8?B?UkFUdzRZdFRLaEFITFVOOGxNdGw1TER0NE8rTUJSVDRaZ0FWMVRMdXFySjE4?= =?utf-8?B?ZFYvcE9saVBNeWJqSHRpaXZkQ25jbTRjc1dLalBaM1FjR0NXMWN3UGZEUEpU?= =?utf-8?B?T1lNRllhZlNNYzZOMWI5NTRESDMrQURFS3djb2d3SnVUbTNOOHRNcGZZQ0F0?= =?utf-8?B?VEthanhIaFZWazhyclpJaXBsYm1wNE95QlpwYkNVV3pMdFlkNHdIZTRRZEht?= =?utf-8?B?T0ZsZXJIMWJDMVowejh4VE12eVRuMGFFNGJWbDBhWHI0dXAyRUpXSGxMWVND?= =?utf-8?B?MHJkTTFvdmtIZm9BRnFvcGZsNW1rdWVvWStRcS80ZmNpWFplMXRGT2wwVCtT?= =?utf-8?B?NGxkamtsQ1hJejN2VVJ2amtvRTU1V3hkaG8ra3lqVEdDVmlkMklsU2FMQkFv?= =?utf-8?B?bXFqRC9zM05EL2IxR2FlOFdPZXFadnN5dDQ3RmFPWEdsMmg0dVRjQ2dtVmpi?= =?utf-8?B?K1IySkxPV3N1Sit6WlFGbHRJd3ExSzBPVG0rSDNVMkppVlM5T0UrYjBKT1gx?= =?utf-8?B?K2lQL21TeUJ3VmhOUERDMVRaTGNzQzR5R253NzlDZzJQcWd4T3VLMVQzMDZt?= =?utf-8?B?OW5sOFBVYnlBTjBoRy8wRStkeGJTRVlhY05wZzNpWlZOaENkRVpRTlV6UDVI?= =?utf-8?B?MTB6aDQvajQrKzhsV3NLQmRZZHRtbzFHanZycUFkRFBXS2RGZTlOYk9SSmF5?= =?utf-8?B?QkJwb3lUVEpqTW5TZjRmQ2VxUzVDT2srN25vM3FYTHdBZEpZMTV6MSt6Qzlp?= =?utf-8?B?OU12OHdxd0J4eDF2WXN2ZGtBSUFWZkdpOUVpZUNjUlJza1Vsb09zOTBmWTBK?= =?utf-8?B?M3NlMEs3S0VhM3YvZWc1UzZ2bUloRkNCWHZZNVNkWUpkalRUWTJWOE9IdXJm?= =?utf-8?B?RnliQXh6cGVJYWdTMHZOSytRck1kcDB2ZUszL0RUc2h0Mm9Eb05meWIyNmVV?= =?utf-8?B?SHRkTXd0V2RhL2tRZEZvRlBHelloYjExY3AwbVFHYWMySUYvVkUyMlpnUCtn?= =?utf-8?B?dEdOMG5oaTBLWEx1TFppRU4rbCt0UWo1ckU0cGtxVGcyNjVrcTFyYnlieGNm?= =?utf-8?B?dFRxVGJDcnlYTE5yMUFzN1hiZDNHUkhtK2Rwc2IvNkg5d09Od25jRk1HS2VO?= =?utf-8?B?bFhSZ2dpOVRWTUdvWURHRVBHNnRaUWhPMUF3VERRcmdxUkpRbzVmWERvNU9P?= =?utf-8?B?NnBoVE4rd09OWUkzMHpQTE5PNlVRPT0=?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jan 2026 05:24:17.5226 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 96b2e74a-f669-4002-0fff-08de58ad52d9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF00017090.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9166 Hello Aaron, On 1/21/2026 9:19 AM, Aaron Lu wrote: > On Tue, Jan 20, 2026 at 03:25:49AM +0000, Zicheng Qu wrote: >> Consider the following sequence on a CPU configured with nohz_full: >> >> 1) A task P runs in cgroup A, and cgroup A becomes throttled due to CFS >> bandwidth control. The gse (cgroup A) where the task P attached is >> dequeued and the CPU switches to idle. >> >> 2) Before cgroup A is unthrottled, task P is migrated from cgroup A to >> another cgroup B (not throttled). >> >> During sched_move_task(), the task P is observed as queued but not >> running, and therefore no resched_curr() is triggered. >> >> 3) Since the CPU is nohz_full, it remains in do_idle() waiting for an >> explicit scheduling event, i.e., resched_curr(). >> >> 4) Later, cgroup A is unthrottled. However, the task P has already been >> migrated out of cgroup A, so unthrottle_cfs_rq() may observe >> load_weight == 0 and return early without resched_curr() called. > > I suppose this is only possible when the unthrottled cfs_rq has been > fully decayed, i.e. !cfs_rq->on_list is true? Ack! Since we detach the task from cfs_rq during task_change_group_fair(), the cfs_rq_is_decayed() during tg_unthrottle_up can return true and we skip putting the cfs_rq on the leaf_cfs_rq_list and unthrottle_cfs_rq() will skip the resched. > Because only in that case, > it will skip the resched_curr() in the bottom of unthrottle_cfs_rq() for > the scenario you have described. Indeed. Happy coincidence that we checked for "rq->cfs.nr_queued" and an unrelated unthrottle could still force a resched for a missed one :-) > > Looking at this logic, I feel the early return due to > (!cfs_rq->load.weight) && (!cfs_rq->on_list) is strange, because the > resched in bottom: > > /* Determine whether we need to wake up potentially idle CPU: */ > if (rq->curr == rq->idle && rq->cfs.nr_queued) > resched_curr(rq); > > should not depend on whether cfs_rq is fully decayed or not... But if it is off list, then it doesn't have any tasks to resched anyways and in Zicheng scenario too, the cfs_rq won't have any tasks either at the time of unthrottle. > > I think it should be something like this: > - complete the branch if no task enqueued but still on_list; > - only resched_curr() if task gets enqueued > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index e71302282671c..e09da54a5d117 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -6009,9 +6009,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) > /* update hierarchical throttle state */ > walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_up, (void *)rq); > > - if (!cfs_rq->load.weight) { > - if (!cfs_rq->on_list) > - return; > + if (!cfs_rq->load.weight && cfs_rq->on_list) { > /* > * Nothing to run but something to decay (on_list)? > * Complete the branch. > @@ -6025,7 +6023,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) > assert_list_leaf_cfs_rq(rq); > > /* Determine whether we need to wake up potentially idle CPU: */ > - if (rq->curr == rq->idle && rq->cfs.nr_queued) > + if (rq->curr == rq->idle && cfs_rq->nr_queued) > resched_curr(rq); > } > > > Thoughts? Yes, checking for cfs_rq->nr_queued should indicate if new tasks were woken on this unthrottled hierarchy. Should make it easier to spot the scenarios like the one that Zicheng experienced if there are any more of those lurking around. -- Thanks and Regards, Prateek