From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011070.outbound.protection.outlook.com [52.101.57.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01D473DD52C for ; Wed, 22 Jul 2026 08:53:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.70 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784710413; cv=fail; b=tlwHVoquyvLBEbNs0E+8an0cC+Wdu9ANdJHiyhzxHh2l6MVPwa9r/IptiiiM6WLGXButfZ9vNQ2uA8Z3vVdkOyCftvlZEHmu5Vi0hknQ6O0HFN8jSPlCtYKXsEitwnTC60BiDY00nfFow+iPVSad3eZwDBdHiUY0FjBEDW9xpNA= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784710413; c=relaxed/simple; bh=N67UTlvMiqAvDpU9xGyadFPp6m1HN0jkwTBQr5Q/N8c=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=OhCA0mHOi6XXmbvyaNbUREGJSDxGm72qX25N4mKvSPMmyhZVg4QygqtmOoMMJ0I9y0IFuSZ36dbfuU3j2pJ45+LAZzkp6lp2gRDdi/XAshUgXwgeSAM3yPKWC+E18oBpIXEZTw8C4mBIk7SscKoNI//7AiIEB5SiK6rpqynkSOY= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=LxZpsfx0; arc=fail smtp.client-ip=52.101.57.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="LxZpsfx0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xD14zd7N3IdldJonGvsfeBuq8XjNgOFjB8Qi+ROyolw2tjNZ8oEgs+MuCtgJL9/V+bKaEsc9K8T8ioubIVSVU73v7o4My6XWBw4ilGozpWta9qVUC7YqeyVECed7Jadm9LX85acEGxVZFoPlXYBXdcY3y4KLCMjYtJQ89UQsOv51muYTlkt7tcqm28OTCNzZw3szcO28cHUjv7etPkUybmSKRSyw51iY9W5u8CVltjtOvv8U5DGzbpfonbKSkWBFlyE38yKvaeA8ACgNR/Hd4nAwKaYsfnZumtfY4a1jpOi9qrzMvhTEas8SGPx0SIekO2kngX5LFq1ial6ru8eR/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3spJ5Gv9/ybcQEw734zVgNl9h+8/CWZ7m3hYBaGTlsM=; b=arln/JM3fTmCE1b8EtGyfmxdaxBAAvIm9HdnlG3/jTDpLYwOa0xjPuhxEkivSypn0qV5khC1Frhxbl7tVJc6L8wmOpR/fl7d4RvKPCfyds73tDt5KraeYX399IkxENWkXx0cWUAa0yqUUfow3epAy0YhW64o7xKTG9fTjfMnUNP611UaMXjPqnqOY96BRxb/QMM4G+Sbsc4alH08GLFPhGE9bQKai43+NYh44RbX2PLKycg1sr1LM3b4mv39rUdv5BEMx2zug5ba5PSkAitrB/EVy7lairyQ5pGWuSIWulwNzBzUCI+GEDPk40Mcv93A8sCInx26mwqXIW4TBEVXYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=gentwo.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3spJ5Gv9/ybcQEw734zVgNl9h+8/CWZ7m3hYBaGTlsM=; b=LxZpsfx0+g1R9vQjCrdc+hdwuNEariR/tn2Q7+nJA6/5dstkr2r5eRkSzbA8nkocoyZpFO5FLjwVS/QN76GEVjtVMt6avBzn68013bt3yqnqyzYYW9G+8weIsdmG3VValHeNypnl/rFkq9zDJ9nTVbEMbPkicLr+y1yhrSmLK0k= Received: from SJ0PR03CA0372.namprd03.prod.outlook.com (2603:10b6:a03:3a1::17) by DS7PR12MB6336.namprd12.prod.outlook.com (2603:10b6:8:93::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.10; Wed, 22 Jul 2026 08:53:25 +0000 Received: from SJ1PEPF000026C4.namprd04.prod.outlook.com (2603:10b6:a03:3a1:cafe::56) by SJ0PR03CA0372.outlook.office365.com (2603:10b6:a03:3a1::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.245.10 via Frontend Transport; Wed, 22 Jul 2026 08:53:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by SJ1PEPF000026C4.mail.protection.outlook.com (10.167.244.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.5 via Frontend Transport; Wed, 22 Jul 2026 08:53:24 +0000 Received: from satlexmb08.amd.com (10.181.42.217) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Wed, 22 Jul 2026 03:53:24 -0500 Received: from [172.31.184.125] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.41 via Frontend Transport; Wed, 22 Jul 2026 03:53:20 -0500 Message-ID: <1bebcf2d-2325-4e11-bb5f-347a0ac10fcc@amd.com> Date: Wed, 22 Jul 2026 14:23:18 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3] sched/core: Preserve wake flags across queued wakeups To: "Shubhang Kaushik (Ampere)" , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider CC: Christoph Lameter , Shubhang Kaushik , References: <20260721-b4-sched-ttwu-wake-flags-v3-1-07b92122bf28@gentwo.org> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260721-b4-sched-ttwu-wake-flags-v3-1-07b92122bf28@gentwo.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF000026C4:EE_|DS7PR12MB6336:EE_ X-MS-Office365-Filtering-Correlation-Id: cd404a6a-00fd-4d05-e8e5-08dee7ceb0e4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|82310400026|1800799024|7416014|376014|36860700016|13003099007|921020|6133799003|56012099006|11063799006|10067099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: QAtpz0LsMTbf451sp8AUkY7XxOARtC/015nim0LobRlTmJlkud7qSfpTe9TTChwZW4yPn8uxFhayb6I/JAk223vjS1ti+whtIHG/3W88wv5XNUxAmB3HGlY0EP5x0/M7F5osCSEIhYQYdb8p5onn/ReB6mDfE/ysY5yokRma2RNrzaSQuguez/DThc0F2a2kEKMp+3gOx1Y6FVGHcWbaiAsPH2meT3hbCrGOLmgZDbPvcuHNb6rof6XQWMw8iy0D0FrhBE5U0yqakpTAAQT4OX1bWQjBgQ1yCVhPwn2epUHqbRYLboeCBcBqAJx8gpszxoIlzOjx92OyQlqdPIDYrJrg4hVaFJO2ZY1p34WLyO+84hiwEE7Na1Bz57BuWYWQQWL722M/L0yLJd512/U82L8Z2vUA2O+LwInUVmCWWXiV6hcoTqtZgREPJ0vnHRCkCztY5AhunyyCML8uxKPAY1sBelXNd1ZieNgp+LAwv7e/+/+V8FBxPtBFaJZAVlBG2L/BepR9eHk375PvgoiBO3A68AgZrqBRhRj+AwBijBTUROKymYm4xqTWsc3a7PI4w+kdmyVDipQDjyLhMMw4ZEWTMUu1EmduEmDaMapXBO/UXDyQgNc0GMnx0jrgCcYg4ub5N8bxuSje8xcyxjOTbOAMltsvhXLEqotObRjKOdywNazzR3qwqm0fwFn8pSWeYU7v831rzkwClUWSXEf0I76NFX+7oRtGfNRDaLqlIEEw9BNiKFXNVVjastX07WB2 X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(23010399003)(82310400026)(1800799024)(7416014)(376014)(36860700016)(13003099007)(921020)(6133799003)(56012099006)(11063799006)(10067099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xUDx+D/WhKISQQKZt0sH3jHSwPyGbIlmxgYKZqKO/eUZ6lZmiu3jouYau80Mr7CNMZOsC7Lw5rHwVSJEpRVjjeFrZdFwa2sTECKTKCzEIQ33vNcuzuyMpot41fY7Wxk8co1eA4106zdxdY4u3FVbPZy4JEEjX6u8dUjtUzvvXbhXdQUDeKYxBWCHTiqcZ0q7mc5t95/GoSvVmg1c9vfQn+TYbjcRHyMUNW4lk7qIL+0jthx3DkKKIIzQGbKIeYE2Gy6bmKfKtSZvjmTVsBtPWh+Gh1SdC/9W7Hj3ll+T2hciMfK4J8myREsdws4eTeLn3E0911iYzCJ68z295Jd5CiIHu/ymq4ABDyUGflpHoiIrn5dHbp4+CCaHDeWrEtRh9dYDdlWVnwp4HLyhvOsCt9ifq4Plk/jS94wmt2s/haxEgW4TmNan4GgT90bC8HzN X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2026 08:53:24.8992 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cd404a6a-00fd-4d05-e8e5-08dee7ceb0e4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF000026C4.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6336 Hello Shubhang, On 7/22/2026 12:24 PM, Shubhang Kaushik (Ampere) wrote: > Queued wakeups currently save only whether the wakee migrated. When the > target CPU drains the wakelist, sched_ttwu_pending() therefore passes only > WF_MIGRATED or 0 to ttwu_do_activate(). > > That loses flags which are still used after CPU selection. For example, > WF_RQ_SELECTED is set after select_task_rq() chooses a runqueue, then used > by ttwu_do_activate() for ENQUEUE_RQ_SELECTED and by wakeup_preempt_fair() > for the preemption threshold: > > direct wakeup: > select_task_rq() -> WF_RQ_SELECTED > ttwu_do_activate(WF_RQ_SELECTED) > wakeup_preempt_fair(WF_RQ_SELECTED) > > queued wakeup: > select_task_rq() -> WF_RQ_SELECTED > ttwu_queue_wakelist() -> save WF_MIGRATED only > sched_ttwu_pending() -> ttwu_do_activate(WF_MIGRATED or 0) > > Preserve the wake flags that still matter after queueing: WF_TTWU, > WF_SYNC, WF_MIGRATED and WF_RQ_SELECTED. Do not save WF_CURRENT_CPU, > which is only a CPU-selection hint. > > Signed-off-by: Shubhang Kaushik (Ampere) > --- > Tested on an 80 CPU Ampere Altra system with perf bench sched messaging, > perf bench sched pipe, hackbench and SPECjBB. No material regression was > observed. > > Baseline: mainline origin/master at v7.2-rc4 (248951ddc14d) > --- > Changes in v3: > - Use READ_ONCE()/WRITE_ONCE() for sched_remote_wakeup_flags. Why? Adding to the llist will have release semantics and I don't think there is any way for the receiver to see an object on the llist and it not having "sched_remote_wakeup_flags" set properly. Otherwise, a bunch of csd initialization bits would also need READ_ONCE() / WRITE_ONCE() but I don't see any. > - Anchor the changelog on the WF_RQ_SELECTED preemption behavior. > - Keep the cross-CPU ordering comment for the task field. > > Link to v2: https://lore.kernel.org/r/20260713-b4-sched-ttwu-wake-flags-v2-1-76e23a8cc313@gentwo.org > > Changes in v2: > - Move sched_remote_wakeup_flags to a standalone u8 outside the scheduler > bitfields. > - Drop the unnecessary reset in sched_ttwu_pending(). > - Add WF_TTWU_QUEUE_MASK for the wake flags preserved across the wakelist. > - Tighten the changelog around direct-vs-queued wakeup consistency. > > Link to v1: https://lore.kernel.org/r/20260710-b4-sched-ttwu-wake-flags-v1-1-23182a4c5367@gentwo.org > --- > include/linux/sched.h | 26 ++++++++++---------------- > kernel/sched/core.c | 10 ++++++++-- > 2 files changed, 18 insertions(+), 18 deletions(-) > > diff --git a/include/linux/sched.h b/include/linux/sched.h > index 373bcc0598d10b4256a11f8c8373ece78fa5e0e5..c0de2ea02eeeb3ad9ca1a090260b00e1b35cbc39 100644 > --- a/include/linux/sched.h > +++ b/include/linux/sched.h > @@ -983,6 +983,16 @@ struct task_struct { > > /* Used for emulating ABI behavior of previous Linux versions: */ > unsigned int personality; > + /* > + * Must not share the scheduler bitfield word because wakelist > + * queueing is not serialized by p->on_cpu. > + * > + * smp_load_acquire(&p->on_cpu) in ttwu_queue_wakelist() pairs This should be "before" and not "in" here ^^ > + * with smp_mb__after_spinlock() in schedule(), making current's What? That load acquire pairs with smp_store_release() in finish_task() before interrupts are enabled after context-switch on the destination CPU. > + * stores visible before the target CPU uses these flags from > + * sched_ttwu_pending(). > + */ > + u8 sched_remote_wakeup_flags; > > /* Scheduler bits, serialized by scheduler locks: */ > unsigned sched_reset_on_fork:1; > @@ -993,22 +1003,6 @@ struct task_struct { > /* Force alignment to the next boundary: */ > unsigned :0; > > - /* Unserialized, strictly 'current' */ > - > - /* > - * This field must not be in the scheduler word above due to wakelist > - * queueing no longer being serialized by p->on_cpu. However: > - * > - * p->XXX = X; ttwu() > - * schedule() if (p->on_rq && ..) // false > - * smp_mb__after_spinlock(); if (smp_load_acquire(&p->on_cpu) && //true > - * deactivate_task() ttwu_queue_wakelist()) > - * p->on_rq = 0; p->sched_remote_wakeup = Y; > - * > - * guarantees all stores of 'current' are visible before > - * ->sched_remote_wakeup gets used, so it can be in this word. > - */ > - unsigned sched_remote_wakeup:1; > #ifdef CONFIG_RT_MUTEXES > unsigned sched_rt_mutex:1; > #endif > diff --git a/kernel/sched/core.c b/kernel/sched/core.c > index 96226707c2f6135341aa779b8262f113e103d8ad..7ddcc6a7f1c06675ce8084cd304475d61f869eb7 100644 > --- a/kernel/sched/core.c > +++ b/kernel/sched/core.c > @@ -3902,13 +3902,16 @@ void sched_ttwu_pending(void *arg) > update_rq_clock(rq); > > llist_for_each_entry_safe(p, t, llist, wake_entry.llist) { > + int wake_flags; Do we really need that local variable? > + > if (WARN_ON_ONCE(p->on_cpu)) > smp_cond_load_acquire(&p->on_cpu, !VAL); > > if (WARN_ON_ONCE(task_cpu(p) != cpu_of(rq))) > set_task_cpu(p, cpu_of(rq)); > > - ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); > + wake_flags = READ_ONCE(p->sched_remote_wakeup_flags); > + ttwu_do_activate(rq, p, wake_flags, &rf); > } > > /* > @@ -3947,11 +3950,14 @@ bool call_function_single_prep_ipi(int cpu) > * via sched_ttwu_wakeup() for activation so the wakee incurs the cost > * of the wakeup instead of the waker. > */ > +#define WF_TTWU_QUEUE_MASK (WF_TTWU | WF_SYNC | WF_MIGRATED | \ > + WF_RQ_SELECTED) I much rather prefer it in single line even though it crosses 80 characters. #defines like SCHED_DL_FLAGS run even longer. > static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags) > { > struct rq *rq = cpu_rq(cpu); > > - p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED); > + WRITE_ONCE(p->sched_remote_wakeup_flags, > + wake_flags & WF_TTWU_QUEUE_MASK); > > WRITE_ONCE(rq->ttwu_pending, 1); > #ifdef CONFIG_SMP FWIW, feel free to include: Tested-by: K Prateek Nayak -- Thanks and Regards, Prateek