From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012006.outbound.protection.outlook.com [52.101.53.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD96E17B418 for ; Wed, 5 Aug 2026 03:09:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.6 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785899375; cv=fail; b=H6M3UAalUMGu1CLxWJszxfq7hxzFd0ztWOkb52hXk4qTaXvP/H96x/mM/ytXjX0qEv1xmB91Z7Dit08OhyrHLEZe5xtcXO4/3rg6tMOQXlqesM2jX+XY4WCjDdOuJye9E8kAEUz0N+FDL9IRAa9ZiyNpC0CKluZR7Siek37kdqc= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785899375; c=relaxed/simple; bh=0wjGCyL8FilMBEUXHw0Key85h0VRYGf1Dco13FpNEOo=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=rOXCH69po0hzhjth9Gn63mFhPfh4gKw2xduBopCKZIz8LIEGoMOo0e08DmI3OaHsvpavRk1bfLNqY6WVxjxkgpFHoUUWc8Cms+JCIMKXF1lAsCm1C4kY0tQAMao/i4tiC6IgDiFRQmU3spWiqWtjCuJpqpVS5heDmpawhxrlMj8= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=LDte2nT7; arc=fail smtp.client-ip=52.101.53.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="LDte2nT7" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ae2xh/KdV145A7bWOdc/PknkTEDLAHOvjPiVWo8T4S8yO8Pf2bZabSI0MjuKzNk6cSEbtlKfRQEAkgmdViYp6MaAELQazMk1fx7Flcz7BJ3OKzDr7W9j1Bp50S9S2OH7p3gA3428vdyG6LaULA1lsZYpOQmDQwZ5Tker1GuanANuMVnnHxcoBCPNsKcJEo3+zeQHYT14GbxN+M/4JCqbHy0I57DCgXgv8KTvHiuRArf5VcjCOb3yHPgEEma7/m2l6DRYSMRjHTh7whk2MWqMKO2lgOtGpPdyhh7UyGrhtlE7OlWHm4yzeK+bNN5bgep4B6WcABYVxBj63OqzjssSbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+mVhyFVRtAHF8i30OvFLcEynft1bk0Lo8KYW8fIs6NA=; b=GbmHn69bmXNWM2EoU4qlsH6pIGNNhg+8hxTnyiEvdOwBN7g3yZzblucn6mdmr1c/vhltcn3twjgLpnqBmnwgTklt+9xtfGFuzQyrNkUdaaoGRZIRePbPpwM7izERjgayjOuEdQvQ7twYJRBUIfRX82Noe94+vhgfOvecYVJveicRLkVn/9BzUD8hVCHCiDJV7SWe1OWTozaEeR9yP6/6pln/juZcChVQlh6ZhkalAq+9yNujYo/mBZkwNYjrcRDGx+01ci3RqNm6Zu0ltGj/dQUMVPcogjDhXYykkx7Z7epNLHKDS5bhpypdDsBRUYy5cozKcmMH2iEWbvu7kFSI8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux.ibm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+mVhyFVRtAHF8i30OvFLcEynft1bk0Lo8KYW8fIs6NA=; b=LDte2nT7Jc7uUWVKTUe+VyHD/FqP28ehuvca6hlQEpiKGLLkPe9F92R//OpXh2tIJlBgDVVcbA2nvDM8QepZMJHkgA7ormbZCl2PawTFSBoQrLGEi/HzSgDuAiV373UP/h7QlsBOGy+azsH6uNb2fERSYWHRDxs1+8iztnYCbVk= Received: from DS1PR07CA0028.namprd07.prod.outlook.com (2603:10b6:8:44d::17) by IA0PR12MB8894.namprd12.prod.outlook.com (2603:10b6:208:483::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.270.17; Wed, 5 Aug 2026 03:09:27 +0000 Received: from DS3PEPF000099DC.namprd04.prod.outlook.com (2603:10b6:8:44d:cafe::25) by DS1PR07CA0028.outlook.office365.com (2603:10b6:8:44d::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.292.16 via Frontend Transport; Wed, 5 Aug 2026 03:09:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by DS3PEPF000099DC.mail.protection.outlook.com (10.167.17.198) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.292.8 via Frontend Transport; Wed, 5 Aug 2026 03:09:25 +0000 Received: from satlexmb10.amd.com (10.181.42.219) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Tue, 4 Aug 2026 22:09:24 -0500 Received: from satlexmb07.amd.com (10.181.42.216) by satlexmb10.amd.com (10.181.42.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Tue, 4 Aug 2026 22:09:24 -0500 Received: from [10.136.47.95] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.41 via Frontend Transport; Tue, 4 Aug 2026 22:09:18 -0500 Message-ID: Date: Wed, 5 Aug 2026 08:39:17 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4] sched/fair: Preserve wake-affine CPU for non-SMT reciprocal sync wakeups To: Shrikanth Hegde , "Shubhang Kaushik (Ampere)" , Peter Zijlstra , Vincent Guittot , Ingo Molnar , Mel Gorman CC: "Christoph Lameter (Ampere)" , Shubhang Kaushik , , Juri Lelli , Dietmar Eggemann , "Steven Rostedt" , Ben Segall , "Valentin Schneider" , Christian Loehle , Madadi Vineeth Reddy References: <20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org> <98bbe2d7-2401-4713-b49e-12f5dfc36471@linux.ibm.com> <97bc48b8-f00d-4c75-97ac-6feadf7d3372@amd.com> <4d531a51-4ec9-4df0-98e4-ec1f314891ba@linux.ibm.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <4d531a51-4ec9-4df0-98e4-ec1f314891ba@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DC:EE_|IA0PR12MB8894:EE_ X-MS-Office365-Filtering-Correlation-Id: 4641d99c-09d5-4f82-2656-08def29ef470 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|1800799024|7416014|376014|23010399003|82310400026|22082099003|18002099003|13003099007|3023799007|6133799003|10067099003|56012099006|11063799006|4143699003; X-Microsoft-Antispam-Message-Info: TceZFwdlDdfMc6RYe75uq7puVxGcDmukX43zEoW8jVnKpufCdRErqP4Z4hLMzeg6CtSfx6Q8iDLagDagi4QfC42HP2UjyZ/THtAOXQ6LE80EVTiuK9ZKlTYBJ5TWmJechxnRCZjfuRITT5X5kwhcp+V9vHPuII/V9dNTGvs99i9oHfKJNp4a/0udP/cvqDIoE25GKnonTmqiY6pcQatyrWOHqnLNA0C/S69+clJVHykbkj829R9nM0CO2Z6Hd50Xq/9RbRKytagOuiWJORzuW8d3Deqd807Heslh9uk3qOCRKfyiwKp6WCEwgz61YuZcrKuyTrqzJDbo4pDZC4tSmBMzvR7xTAprxybYF4pmh1yqtP7gLB3OwjdDwwEva6Rz1NVXi8VV9k93VqBJCZBGGq+pJLtAFcUYRtqNjUxHFLQ8sL+vtwrieljaNJLHRejNVm5U458+aqF2i+qNTpIRa8dT6IANnCCbw6WY4rhoM7V2bPRJK3rqz3pNJBtQ0T8gmqDzBUy01HvAgUXKKJLTmU7b0r86qW1I+N+a2HRqQK5SH8I9S9KUhioqYJKbhrBpHoOdgrXbitt16uHzr3xebOhxC83aL7XIN40t0XO+eQ5qiqaOLGpsQo61+WTtIJX/iKZW2iMFkDenFuQB551bDSh+hjlALJPVmCTHSmGt57FnaSSUOmRb4rZefz7t1w2qYjtPk0cDPepX2FX3EmPW9g== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(1800799024)(7416014)(376014)(23010399003)(82310400026)(22082099003)(18002099003)(13003099007)(3023799007)(6133799003)(10067099003)(56012099006)(11063799006)(4143699003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1XqW+79W/IDo1a7jClAR+SrC1raWhb/9RqzTl9QUYDd5Lm0FhKyeFPnOXc2adN4n86/CFesMmj4yFyWQaDAc2Lzshli+TJwQufKOYwuFEFzTP/LLLDC9/7GQZCrCqo+ukFXqySbPZPpU08tqPD7SunuNBW6in3Y3ziLKoYU0T0FznAcLOfPrP9VRlIKUIQuc2rQC8gOciYGI3n61bjN/AJJS/TC7u7zVQGdWqhIfClKVW5uj5rysfx5iEg16rmWAx+4VFXgcYP8rr5mo4miHvZU7SLvEckOMgatkTgcc3QvxWvgGXv+MnU7NAAG37x+MeetX4AjWwReE/ANRp0eDkcXM/HSxf8TnDhYUeqibO/0HtdjSlCef9jTtgTXp6q9u1IvAMdZLlLlR3ZAbn4PT86xNYzFgSuMUf9xFqOJq5YtshZBC6bhw0kc6q7ZyPGEM X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Aug 2026 03:09:25.2199 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4641d99c-09d5-4f82-2656-08def29ef470 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DC.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8894 Hello Shrikanth, On 8/4/2026 4:10 PM, Shrikanth Hegde wrote: > Hi Prateek. > > On 8/4/26 2:12 PM, K Prateek Nayak wrote: >> Hello Shrikanth, >> >> On 8/4/2026 10:10 AM, Shrikanth Hegde wrote: >>> As I said in v3, before we add bells/whistles to sync path, i want >>> to know what is expected of sync behavior today. >>> And that should be documented in Documentation/scheduler/ >>> >>> Be it, >>> - current way of hint only and scheduler can still choose an idle core/idle cpu etc. >>> - Should it be enforcing it to waker cpu if waker cpu has only one task. >>> - Whatever the policy maybe. >>> >>> Current api usage is tricky to use and effect is visible in real life workloads. >>> The case I mentioned in v3 of networking code using sync api leads to strange >>> results due to sync mechanism. >>> - It depends whether waker/wakee are running on same node. >>> - Result of wake_wide. >>> In other end, user sees inconsistent latency/throughput. >>> >>> We can keep on adding minor changes to sync api path, >>> but one benchmark will benefit and one will suffer. >>> Having the behavior documented is a good start. >>> >>> Peter, Ingo, Vincent, Mel, Prateek, >>> What do you guys think? >> >> Currently it is very arbitrary and WF_SYNC may, or may not, indicate a >> true voluntary blocking behavior. For example, anon_pipe_read() uses a >> wake_up_interruptible_sync_poll() to wake up writers once reader has >> drained the pipe but if you think about it, why would the reader block >> soon after just having the data it needed? > > Doesn't "perf bench sched pipe" also use anon_pipe_read/write? > > -   27.49%     0.32%  sched-pipe       [kernel.kallsyms]                 [k] ksys_read >    - 27.17% ksys_read >       - 26.57% vfs_read >          - 22.33% anon_pipe_read Exactly! Highly depends on the workload - if you are using pipe for a signal, great, but if you are piping gigabytes of data, and there is a continuous consumption, then co-locating the readers and writers makes sense. This is probably why wake_wide doesn't even care about the sync hint and makes a call purely on waker_flips. > >> >> Here are the results on my Zen4 system from running perf bench >> sched messaging (threads + pipes) at varying worker counts with >> all wake_up_interruptible_sync_poll converted to >> wake_up_interruptible_poll: >> >> Test:                   tip                     no_sync >>   1-groups:         3.79 (0.00 pct)         3.35 (11.60 pct) >>   2-groups:         3.85 (0.00 pct)         3.41 (11.42 pct) >>   4-groups:         4.02 (0.00 pct)         3.32 (17.41 pct) >>   8-groups:         4.33 (0.00 pct)         4.38 (-1.15 pct) >> 16-groups:         6.09 (0.00 pct)         6.12 (-0.49 pct) >> --- >> >> So seems like WF_SYNC hint on this machine with perf bench sched >> messaging (thread + pipes) pattern is actually holding it back. >> Lemme check processes ... >> >> Test:                   tip                     no_sync >>   1-groups:         3.48 (0.00 pct)         3.08 (11.49 pct) >>   2-groups:         3.80 (0.00 pct)         3.07 (19.21 pct) >>   4-groups:         3.91 (0.00 pct)         3.09 (20.97 pct) >>   8-groups:         4.13 (0.00 pct)         4.10 (0.72 pct) >> 16-groups:         5.81 (0.00 pct)         5.74 (1.20 pct) >> >> Similar stuff. At some point it was pretty bad for Zen3 but >> situation might have changed since ¯\_(ツ)_/¯ I'll let you >> know once I have a machine. >> >> But ... If I have true 1:1 waiting on pipe as in the case of >> "perf bench sched pipe -l 1000000" I go from ~2.5usecs/op on >> average to ~4.2usecs/op which is close to a 50% increase in the >> benchmark time so that WF_SYNC hint can also help if all we have >> is looping over a read waiting for one page worth of write. >> >> The way I look at WF_SYNC nowadays is that it indicates a local LLC >> wakeup is beneficial. wake_wide() doesn't even consider WF_SYNc and >> simply uses wake-wakee flips and then only at want_affine() do we >> actually check the sync hint. >> > > Yes, it is difficult to say when would sync actually kick in. > Also, if we say hint, onus now falls on scheduler to optimize all > call sites. > > Clearly comments around __wake_up_sync_key are outdated. At some point (1da177e4c3f4), sync + wakeup on waker's CPU also inhibited resched_curr() and allowed the waker to naturally yield the CPU (which is the reason for the UP comment) but today, we do a resched_curr() unconditionally. Signs of different times. > >> Most benefit come from wake_affine_idle() for the 1 task case where >> target is set to current and select_idle_sibling() uses that as the >> target from there on. >> >> It could purely be a coincidence that it benefits at all - most of >> these microbenchmark we have always hit the same two syscall (mostly >> read() and write() in turns) and a most of benefit for those comes >> from kernel instructions being primed in cache. >> >> I remember a while back, removing the effect of WF_SYNC on the >> networking side hampered a lot of performance - especially for >> localhost communications. This one specifically >> https://lore.kernel.org/lkml/20220711224704.1672831-1-libo.chen@oracle.com/ >> >> Let me see if things have miraculously changed there too but for >> TCP sockets in real world, with blocking for ACKs, I think >> WF_SYNC still makes sense there but I feel most of the benefits >> from sync are a second order effect. >> > > But today it calls sync even for non-blocking. Ack. I think it just got carried over and for most real world scenarios, it probably didn't made a difference until the processor topologies diverged and we have many machines with many small LLCs and many more machines with many large LLCs on the same socket. -- Thanks and Regards, Prateek