From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010041.outbound.protection.outlook.com [52.101.85.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04DDB3A7589 for ; Mon, 30 Mar 2026 07:56:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.85.41 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774857371; cv=fail; b=buAjYa2DVUVGN9e2Ze5mJ0nLLxRSNZ7zXO60tEMMV6O91RErruISzDoLeoV1kePRDjAQ8lgY1sFu4T9qpnlPzCwY2aThsdo9mh+453eW1fhDujrpX8DunrkCas66agnDeR/1RRZQPaf1aHCeqx5gu2+fSczDj0LeoLZWLVohDhM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774857371; c=relaxed/simple; bh=XJ1yXsW1y78dwo+VVWcyck87NKipc9mt8qdeE8Qqcnk=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=XoZ3JHhGLs1HFn85g/R/U7AUflB1zInM/xSvBiO7y154itwkukSbJfbhlziIAadRYfhkW8m3WVQuVTI8i25X2vrJDjsEHVnmQm7+x7Lsnmh8uoxx7jzfI7fD1kF4fQzj2rI7ePAqUD0RPYYzy7e6KNPfsH956SAIlUFfqGNjhwk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=1vCRKtHV; arc=fail smtp.client-ip=52.101.85.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="1vCRKtHV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nxM8GZ6zlh367EwFowS9Vrt/bsKvRUHEeJqWmrs4jsIzl5CLcoS7p14Y4pwabIncJa5lb5ifYbFVp+5yyIbkoUPXFrm/S3D/xtlTzhY0kstuFk2H+I8hK6f6uNyFu+i7myLBrfMm9Qhc7YR5iKM+pCS34iOAdRKHYvOuXnQBCPKaaG1UClv+biFchtEKH7w3qe9r375o8wtSxbWAn5TsNg1SOzxJmjYhTTdORmfIGzrr90BrXqIOlQZE7mAdifeqAI8glSWtssgsrFoFbsC3jggnf8x1UDbaPswNbPjpwrRoZVsWFbZPZpZVUvuyMRqDL7o9Cok3PZA2UP6ppoGb6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2Er90HFzjc/7qls+19bEMSwmYTW8+c1bp0Y/Kd34QCY=; b=QnK36hnWWMdgYZN0KJi2qWPu7IUo9UJS9JcuYOEqZxFdsB6EEI9wZafZCuzz4553FtWXdmSkzV4BTz1VjnTXgV+oS7BqdTfM6r0EEJFlRrZq1RApn2wkUC9lt+xeJ+PGa75XzeNWkT92AAMZA1qdGvNEOoVx21gSapOkodykLD/I/wQ+HJLB+aV+nUvsJBK+fhU6YslAQdyW5+Bi/xCJywCu5t2j4PMHuzdfTYwl7T5yklBXBGDxFiyEIke2qLEmmKJ26aAyJvZm8qu1zjztNVDOuJto1dFlzYi/uhBGv15nejmVylz5QxTHsFozGk6uRrEPFimjUZBjF10Lzj7D9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2Er90HFzjc/7qls+19bEMSwmYTW8+c1bp0Y/Kd34QCY=; b=1vCRKtHVlNj0PynmGoVexBCPzkO3PcLdONjnHwi6recjZ/X+XuZD3VyMtvnEiiPraPtPacZLFyItumpe2FXxJfYaMjftgvspH4LIUquKWsaR2GQbPzsJjZI1OEu4fWR4KKODIEK66zKaY2gKJFqe59P0npyU7M1dJBQ0L+C4w0o= Received: from BYAPR05CA0036.namprd05.prod.outlook.com (2603:10b6:a03:c0::49) by IA1PR12MB7685.namprd12.prod.outlook.com (2603:10b6:208:423::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.8; Mon, 30 Mar 2026 07:56:05 +0000 Received: from SJ5PEPF00000204.namprd05.prod.outlook.com (2603:10b6:a03:c0:cafe::33) by BYAPR05CA0036.outlook.office365.com (2603:10b6:a03:c0::49) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9745.23 via Frontend Transport; Mon, 30 Mar 2026 07:56:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by SJ5PEPF00000204.mail.protection.outlook.com (10.167.244.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9745.21 via Frontend Transport; Mon, 30 Mar 2026 07:56:05 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 30 Mar 2026 02:56:04 -0500 Received: from satlexmb08.amd.com (10.181.42.217) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 30 Mar 2026 00:56:04 -0700 Received: from [172.31.184.125] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Mon, 30 Mar 2026 02:56:00 -0500 Message-ID: <0d3680c3-3e17-47b8-8fdb-0cc1f97ffce0@amd.com> Date: Mon, 30 Mar 2026 13:25:59 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 5/7] sched/fair: Increase weight bits for avg_vruntime To: Peter Zijlstra , Vincent Guittot CC: , , , , , , , , , , , References: <20260219075840.162631716@infradead.org> <20260219080624.942813440@infradead.org> <20260223115100.GI2995752@noisy.programming.kicks-ass.net> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260223115100.GI2995752@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000204:EE_|IA1PR12MB7685:EE_ X-MS-Office365-Filtering-Correlation-Id: f21f1bc9-4021-443a-ef90-08de8e31cb78 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700016|376014|7416014|18002099003|56012099003|22082099003; X-Microsoft-Antispam-Message-Info: udh53UxpsSs5djdzYdh258FiOj5Q2lMiPBSnqXB+1ZgGtu1pXVNy4hQNjqqR/y6jODmw4JS865LnRVyLkWzU3U9In+mghHBZ1w5bJOodrJDdsKgcrGAuyNeHCqnS0TsWvXWBz0lx89X0P5aIp+Ws1S8oWhJSHC7eX555u7S5ACXhVh6X1rRv/WBMIk1Ijcce+StOC+O2J+kw23PxVfuk4cnSm80JZuqHCtcqHARtFyf8OMflRzHE5fe7ZnJMmXf143uFmi2p8j8gdpoOnyl5GYNxaEm6UrYQDkAa10QrbOZRIuEOs1SbPj3ATgE2AvYrPmBv905UYHkKWtCOd/lClFGvg+JUoNKs+6+Gwt4nL/mhLYvXaIIyV7y+v0QkCbZICo6d+/0n+mOat4uksVAZHgR+2k/oxfs5T9KVSBLSNHtyQhU+SuHCyb1lmwOQfa7WPyVO4VBnNCCj3yQpBfsRksLWHozjyKc1HodR5BdozBUTs3h0WZJxExf4HA6Lvgg4TApSNNgCOq6m+rvdkjycPYanAAHliORSEUCEkVKko5PLYYAiQrX5IxjNZY+JPDud4TaXD5sV3UANA/t7xodfo5BACji8pxk55vU8hLn4UG7MCWez1SlMa0go9+kXN2d/Z5j/MLytbH+Q9YGvQ0UT63uHAyt4w38pN4tTTVNe8YaiaCSXiSiix88vvoTrK5mZC/ZTbTZP8ftZ/whiT81wqrJJPng48EFms8pgED6bVhQZ91cI2LP5DZ3Kgn47VE4gYNGI0VQ9ecp1tV+aKTGkPA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700016)(376014)(7416014)(18002099003)(56012099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: JJaiNMOi3XHjZiKTCrbSnyxYhfSMxcmg4fBpgDoRK+FKQYjPGnlX/flrjx1NGzpnqW5TWgH9maKYImCKjflLzhjYi9h1jPhYRYOBKsjhTpwVvJS2QPN7NwLn07bx4XOzaXSTY8zQubGGNBX9XafArEX0xGwm8RVkQem9xq8lj8hCoydNI5cR+7jdpf2aSAhBCxgqfh+QPwhZP4uEY+UsJMReLg+Xx6zn+gEAy4e93debv9w7d2GOZJXughpI5z6b++tJzLYbng/sxULIsUbsBaehJB79OyQHbi+3hPoBFTz0bp5w8uhcKRJ8QLFOZ88zEWVw+emHF/08LigRQYdLchjmWWFcyjNoMelvEV6Nfotk+aXVghZY5G0E5wBWfNsP7Sb7dn/dMnJYiX/fvpBiU394tCu+5V1zuMxXOFeSAtr14nxZUJrOWy2dMosw+Nm9 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Mar 2026 07:56:05.0088 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f21f1bc9-4021-443a-ef90-08de8e31cb78 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000204.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7685 On 2/23/2026 5:21 PM, Peter Zijlstra wrote: >> We should always have >> key < 110ms (max slice+max tick) * nice_0 (2^20) / weight (2) >> key < 2^46 >> >> We can use 50 bits to get margin >> >> Weight is always less than 27bits and key*weight gives us 110ms (max >> slice+max tick) * nice_0 (2^20) so we should never add more than 2^47 >> to ->sum_weight >> >> so a WARN_ONCE (cfs_rq->sum_weight > 2^63) should be enough > > Ha, I was >< close to pushing out these patches when I saw this. > > The thing is signed, so bit 63 is the sign bit, but I suppose we can > test bit 62 like so: > > Let me go build and boot that. > > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -679,9 +679,13 @@ static inline void > __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se) > { > unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight); > - s64 key = entity_key(cfs_rq, se); > + s64 w_vruntime, key = entity_key(cfs_rq, se); > > - cfs_rq->sum_w_vruntime += key * weight; > + w_vruntime = key * weight; > + > + WARN_ON_ONCE((w_vruntime >> 63) != (w_vruntime >> 62)); I was trying to reproduce the crash that John mentioned on the Patch 1 and although I couldn't reproduce that crash (yet), I tripped this when running stress-ng yield test (32 copies x 256 children + sched messaging 16 groups) on my dual socket system (2 x 64C/128T): ------------[ cut here ]------------ (w_vruntime >> 63) != (w_vruntime >> 62) WARNING: kernel/sched/fair.c:692 at __enqueue_entity+0x382/0x3a0, CPU#5: stress-ng/5062 Modules linked in: ... CPU: 5 UID: 1000 PID: 5062 Comm: stress-ng Not tainted 7.0.0-rc5-topo-test+ #40 PREEMPT(full) Hardware name: Dell Inc. PowerEdge R6525/024PW1, BIOS 2.7.3 03/30/2022 RIP: 0010:__enqueue_entity+0x382/0x3a0 Code: 4c 89 4b 48 4c 89 4b 50 e9 61 fe ff ff 83 f9 3f 0f 87 b8 27 e5 ff 49 d3 ec b8 02 00 00 00 49 39 c4 4c 0f 42 e0 e9 16 ff ff ff <0f> 0b e9 d8 fc ff ff 0f 0b e9 e1 fe ff ff 0f 0b 66 66 2e 0f 1f 84 RSP: 0018:ffffcf6b8ea88c18 EFLAGS: 00010002 RAX: bf38ba3b09dc2400 RBX: ffff8d546f832680 RCX: ffffffffffffffff RDX: fffffffffffffffe RSI: ffff8d1587818080 RDI: ffff8d546f832680 RBP: ffff8d1587818080 R08: 000000000000002d R09: 00000000ffffffff R10: 0000000000000001 R11: ffffcf6b8ea88ff8 R12: 00000000056ae400 R13: ffff8d1587818080 R14: 0000000000000001 R15: ffff8d546f832680 FS: 00007f8c742c9740(0000) GS:ffff8d54b0c82000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2e09381358 CR3: 00000040db039002 CR4: 0000000000f70ef0 PKRU: 55555554 Call Trace: enqueue_task_fair+0x1a3/0xe50 ? srso_alias_return_thunk+0x5/0xfbef5 ? place_entity+0x21/0x160 enqueue_task+0x88/0x1b0 ttwu_do_activate+0x74/0x1c0 try_to_wake_up+0x277/0x840 ... asm_sysvec_call_function_single+0x1a/0x20 RIP: 0010:do_sched_yield+0x73/0xa0 Code: 89 df 48 8b 80 e8 02 00 00 48 8b 40 18 e8 75 a9 fd 00 65 ff 05 9e 94 fc 02 66 90 48 8d 7b 48 e8 d3 96 fd 00 fb 0f 1f 44 00 00 <65> ff 0d 86 94 fc 02 5b e9 10 12 fd 00 83 83 70 0d 00 00 01 eb bb RSP: 0018:ffffcf6b9b7bbd78 EFLAGS: 00000282 RAX: ffffffffbbbb4560 RBX: ffff8d546f832580 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8d546f8325c8 RBP: ffffcf6b9b7bbf38 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d15d1070000 R13: 0000000000000018 R14: 0000000000000000 R15: 0000000000000018 ? __pfx_yield_task_fair+0x10/0x10 ? do_sched_yield+0x6d/0xa0 __do_sys_sched_yield+0xe/0x20 ... Since this wasn't suppose to trip, I'm assuming we are somehow in the wrap around territory again :-( I don't see anything particularly interesting in the sched/debug entry after the fact: cfs_rq[5]:/user.slice .left_deadline : 26249498461.397509 .left_vruntime : 26249498270.250843 .zero_vruntime : 26249456859.395628 .sum_w_vruntime : 2547538312135680 (51 bits) .sum_weight : 61440 .sum_shift : 0 .avg_vruntime : 26249498338.158417 .right_vruntime : 26249498381.633124 .spread : 111.382281 .nr_queued : 5 .h_nr_runnable : 5 .h_nr_queued : 5 .h_nr_idle : 0 ... I still haven't figured out how this happens I'll start running with some debug prints next. On a tangential note, now that we only yield on eligibility, would something like below make sense? diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 226509231e67..55ab1f58d703 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9265,9 +9265,10 @@ static void yield_task_fair(struct rq *rq) struct sched_entity *se = &curr->se; /* - * Are we the only task in the tree? + * Single task is always eligible on the cfs_rq. + * Don't pull the vruntime needlessly. */ - if (unlikely(rq->nr_running == 1)) + if (unlikely(cfs_rq->nr_queued == 1)) return; clear_buddies(cfs_rq, se); -- Thanks and Regards, Prateek