From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from YT3PR01CU008.outbound.protection.outlook.com (mail-canadacentralazon11020127.outbound.protection.outlook.com [52.101.189.127]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF6A8379991 for ; Tue, 7 Jul 2026 19:40:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.189.127 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783453249; cv=fail; b=fmcUfNlQAimG2Gzrz2yGFjhrTc9QefLxmXm4yR7zVux3Ga8nc/R5PGgX8WSdcb+Wula3/8dYVE1t6DT/hN3h+xN6hxZrGVOux7yGPLb/r3NgIg9UG+k6xt0uIhX0+SZSp6f7HmBiXsq5+9V9pep56tFu9RjUbUAIB1jU9TnGYH0= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783453249; c=relaxed/simple; bh=hMTdoe6oJJ20zPW5KvnUgTpTjl8lOAcAuP8tk4hkC5E=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=nNrtiGU5Vo5qoSEF2CLorrnZHczehJyYi1cjBwdx4ONbY7GVx4zBQmQK/8OuAyukBRpIMxCVWP+x6EeNNTXAwH0LfLO5IPVlmFeA5Nm42T4+uLjh/K8GkGSVVvcH8QhCsnlTg+PzezhGp82oYX0XUXfUcQr9+8WpsdbAxAnJyko= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=W8otZuXE; arc=fail smtp.client-ip=52.101.189.127 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="W8otZuXE" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=SCaONKQsRZAOIPJx5pdBYwtyiMncNqN31EEaqyWcMSJA9df8ZL+OgXcntpA7y4uRhkcaIS2b90OQrRUjRjvPWLZ9GCTt+/53aV0aiV5DFo5XePDx29gZokHgtOaIdxiPsonah2Y1+ish6Fsa+V7KivGcjXfYhlLnDIY7WCrwt2u115AZ7RXr1DWknIEtZmx26LkU0CT7IWXBwPctvlzvfOHtIAGTFsPZTwPdk1E4aWJ0fGKgoPERBCEDkZp08NKVowYL0SRYl0vqhB9RJOBp639Ww1veyJozow09UZ1E681LTy3r2sOqsIQe2/b//+va5SM3zvWNZUUJBwnlRZx81w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Yr8YTNHnkWbWwoEz6Mmd9hBEHHdnUxnIGL19C6G2BrI=; b=Dx9trdzrfgrw6u3+rvfSPp5K3CLZCZRY03a2BBmNdb5S2WUnKys/KEPKsXQwOnREuCYtwPqozZkURM1rynP5mNZboPdZ+AaXtzeJBqWSUmP/18AZVo9l8VDXYtjIDvaeqy4KtRL+EpxcKdK2QIFtWsfDnNLgWp3OxoGykB7/fSYPr61UVp1U5NaVYlJ75pFEraxCpgVMphKiB4BprdCxp2X7NJuQLTJhz64B2JRhRYXCjmjg9Db7xMp11L0b0iAfBWJOdDpAw2p08TR4ZXGnp/91kyK/EsInO+7lILXluk7yK+grBnqKaWaTc7VEsNk/FloIFbO6QuvmWTtDV5dWeA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=efficios.com; dmarc=pass action=none header.from=efficios.com; dkim=pass header.d=efficios.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yr8YTNHnkWbWwoEz6Mmd9hBEHHdnUxnIGL19C6G2BrI=; b=W8otZuXEGjpbCgfqdcLzT5IDTE9bOeAknpYXVXHgiYVo/78qjbxb1NybAFFBCPN/T3B6XC/NGG80pmen+q3YRm6M+fSOBqg0J+Znu9lwYvwmlBVakPH2rMicOJWMkQm4g9SZi6l4L/nO1Y0uoy/V7+fijPJNM+aZquV3upKkf/cAcF43eXvLNm+rXH0FcNl+1gaN6JQsnFoAalWt05EkZ1GGdLnahXWOkSmJ7gcQmqGLBx1dyhBJbh7N+CR01i/mN6A0KWtaVCLN8pj5ZY/84r6vz5vNUBi3HfKqmMisXyfFssyORWMXQuRwLeLlYgE4QElmtd8dXZPg1JBBeadB0g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=efficios.com; Received: from YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:be::5) by QB1PPF49BB1DED8.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c08::236) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.14; Tue, 7 Jul 2026 19:40:43 +0000 Received: from YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM ([fe80::6004:a862:d45d:90c1]) by YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM ([fe80::6004:a862:d45d:90c1%4]) with mapi id 15.21.0181.009; Tue, 7 Jul 2026 19:40:43 +0000 Message-ID: Date: Tue, 7 Jul 2026 15:40:41 -0400 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH v5 1/2] hazptr: Implement Hazard Pointers To: paulmck@kernel.org Cc: Boqun Feng , linux-kernel@vger.kernel.org References: <20260223204418.1429025-1-mathieu.desnoyers@efficios.com> <20260223204418.1429025-2-mathieu.desnoyers@efficios.com> <0d9f61c1-4a93-4f5a-a703-12ef4f325aa9@efficios.com> <8c12a0bf-68b8-4677-9745-c39b79d1f1c7@paulmck-laptop> <6623407f-8219-4a9a-8c6d-f17e0e73f828@efficios.com> <7741b89e-e36e-4288-ad85-83a413529526@efficios.com> From: Mathieu Desnoyers Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: YQBPR0101CA0212.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:67::18) To YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:be::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: YT2PR01MB9175:EE_|QB1PPF49BB1DED8:EE_ X-MS-Office365-Filtering-Correlation-Id: 72542d8c-2aee-42f8-b6c8-08dedc5fa1bd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|23010399003|366016|1800799024|22082099003|18002099003|4143699003|56012099006|6133799003; X-Microsoft-Antispam-Message-Info: qwo/MLXYTDLYG75OcCNSYvXWiZvaKZex+SpYAVjhPGtCuFnrgNbuwgOZB4fbp/ioBlKaubnGhNhH1l8iLLx4ch4UOfhjmXpxx9axZTUvwF13ESaW1sWVmbui7aPio0LW9O+FgfwXQTeLJnqInt4bI4MdWT4mGyDg7XUDEOP7jjmTZtliqnT/rqfFyfTFTSmZyhk12IuCeRVIgaONoU44AtRIZfXxcWtH3nxZbmy9sLfBf1FDDiiXf+gOTXYfb0cQSr9VjIOe63wqcJ01ow/4vC+6BCOxIewBBpl+CZjvvaoKSG3fvKfS+5f1i8p03fcY5h2odwGdge0OLZpBVELaheG/Io4m08TkupSyzJapiJdCWy7T6HvqqGjX1lz7RwR/kzADxAl4upVYEehTkgRaXzdh9MU0kiMobQL3aJpsSgYUZ/a6KXiCXELJOXEmQ5Mmbq3p5XHdkKATWvhIHJtVMw1anyPY6yXGwgPvuCY3cM+5rVG4v1edh3UUGRx0t3rFMkWP5+XIGsWbB5U6wxbxhC9C4LvEfHFAKnczpwedlonx8kmry4X/0sASV/aJiyrPqUKBPv1+LyIVmKelnUQMATZ2GlR58dLHFk7nhth71fI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(23010399003)(366016)(1800799024)(22082099003)(18002099003)(4143699003)(56012099006)(6133799003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RSszSzFNYlZMNjRTb1g3Zkc3dk1nME1pM29Na0pUdzlsTzNGMnpUZVJxdlQv?= =?utf-8?B?Ky9yR0J5QWxvajNiZ0Q3UHNKWUpZSmZYRmk1K0dndmNKMTdRN2hnZHR2Z3pu?= =?utf-8?B?Ym0wMUtkQThBUnRjT3pSblBxdUhrdmNYMkpWVEFYVG5uZnd2TkJFWGpvMzRa?= =?utf-8?B?SDJOK0l1WklBcGFSaFY5Mi93MVVHS09aaEd4VVZ4eC9JQ0pUMDl0NVRiYmxi?= =?utf-8?B?aU9ZbDVVdDBIWTBCMGpLQnppOWZHT3o4dFQvQnR2VDkwR2xadjFaaExoZlNF?= =?utf-8?B?bUV3eW5qZWtEM20zdkhVV1pYckhIZGc0elhnbW9lYW1FUzFjRDgyUlpEUzZQ?= =?utf-8?B?UUtiMm84S0pzTUFna29XTEw1OHNxcWxKNzJYeERVWVZCeHhqWGtkZ21KVms1?= =?utf-8?B?ajE2ckhIQUdQVzllTTNibmJCSFdvRlZvUEZ0SGx4OWozaHZHTzZaTmp2djdx?= =?utf-8?B?aTE1ZDc3cHFWTjJVZGJqa2pTUkd0MEVEZzBtbFNNTUlQOXNIdDBJNnpvRnAx?= =?utf-8?B?WWVmZ3oxdjJLSEtjWlBiK2xSRWptWEMrUWJOUllJYTF5Nmt6N3dkTDBNMDZa?= =?utf-8?B?TytidGw0QmZNNmQzcktKQllNR25nYkVoMGo0OE5vTHE5NWV4Ull4SHpoNjdk?= =?utf-8?B?YjRBRGRzbUo3YVF5RnBUbEltUnZJYy9TTzJrOHRwbnlRcVdsNEN2cmRMV09i?= =?utf-8?B?cHpiNXdGc1BSbnNRZ1l2SmlLSi82VUVsT3BFS2JUT1UraFAvend3QVhxRmF5?= =?utf-8?B?bzl5WTZ5MTlSQ3FmYUg3ZWE4dVRKanVBd21Pd1VSVDdXS2l6cTkyNnRFSU12?= =?utf-8?B?UDdENE1OV1pvbmVOMjFVeWFhMTBpRzIza2hQYTBzSEVlLzJjNUNvbVZHMURW?= =?utf-8?B?VU15dzFMNGNIbWJHdjhzZFFzNmVOZHlVZXNNZ2VjN2tzbzdza0ExRExwejJy?= =?utf-8?B?MGRBakN5MEZPdW1KK0ZkMUhKRnFJTHpEZGF4NDh1OHZvZWVObTQrOEF5cVZi?= =?utf-8?B?RzNzRFg1eGhSeStaOGV1SElQSkYxZ2hlSVRWNzQ2Zk5Fb3dlcXRET1ZBVGNr?= =?utf-8?B?bmF2RThERnVSS01MdjdrNjc3T2piQ1Y1Q3FqMWRQMi9EbklyaWRZMDNZcVEz?= =?utf-8?B?UUtGU0tldTk0Q1pjTlFEcVVsOHUwSXFpR0xUL1BULzRSU2s3SmlJY2k0S3hN?= =?utf-8?B?MXo1d2lEM3RuT1kwbjk0d1ZtS1dSRnErQk1JVEpMSDJiSXd3WGUvbm9VZGJn?= =?utf-8?B?a3ZwQ3M2MExpTDdNQUpSaGIrcllIMk0wWEJ1QW03YXd3QTBzWk5jOTVxWTNY?= =?utf-8?B?WkVDVTZ5bWs3QlpFNER3VnpQdFJGRmFsZ3B4a2dXMzZZeEg5aXowVHd0S1Za?= =?utf-8?B?eU4wVGFVZk5McElBQU8ydFc5cUx2S1U0czRMRUsrYWhyZ0VXL0dnclNEU3Ev?= =?utf-8?B?UE5TcXhtQ2hDS21ZZjE0SmVoWUhKU08wcFlLcURMNWtzSENUNFZaZVNDL21K?= =?utf-8?B?RERUbEo2RjlNd2tYa2UxdXZuL0RBU0pxOGN1S085ck02SkRCVkI4MUFDY09l?= =?utf-8?B?aFNyWW8yN3hFbE9oc0F6YUhoSzRZZUMzVmM2TnhDd29Yd29ZWDNJRjdkL2VT?= =?utf-8?B?Ry9VcFE3ZGJpUTNDMnVTUTIrQWN0aVFvUjFRWDNPZEY3d3dZdjVKcnpqMnVC?= =?utf-8?B?Zk8wbnpWSjlOc1orM2I5TTdPTHkvaHBrSnZVbWJucHd0Nk95UUt6NU1kbEFo?= =?utf-8?B?Tldrc3NVeVEzZmE4MWRSTUNIQ21GL2QybXhWanRpU3NTS0xoS0JOSG42NFps?= =?utf-8?B?QVhWUkV3WjZxRjdmUzJGVk1LNkV3ay9sMG84WSszaTdpVFA1ODdYLzdIZ2Mz?= =?utf-8?B?YzlTU2RVKy9LdVhWeXZpR0ZZekVoR3ZhZk1hMzRrK2NnTmUzTGYyUkFaQWZo?= =?utf-8?B?cHUyd3lXRTNQMDdTK2dVenp6S3B5dFVvZDUxUW5RTFZYNkFJbDBiOWpuL3A4?= =?utf-8?B?ZS9IRjBUa2hJSGUxcGQyc08wTFd5OUFnT1cwWlJEdkhDYXdXaFNjTUduL1p3?= =?utf-8?B?THJaZHRzZ1N4K1V2c2szaGRFKy9qK3crczNxdFNIMFhKU1hVUnZwRVdzUUc4?= =?utf-8?B?UmNyNFdsazR5MWhmc3RjKzM2TFYvaFlGSyt3ZnVsUEtlcEw5RlhtVlU2NzZy?= =?utf-8?B?MS81V0xPTlpJNnhCc3BpZGcrVEtaMjhjRjlpY28vTXFFY291U3BuSEx5VU8y?= =?utf-8?B?M3RUcjVpSmpRRXJpVmlKRmNGZjhVYVVLNjNTYVhTN3pvVFBhdERpbVV2TXV4?= =?utf-8?B?bWJlM0RJY09McnB6ZU0zT0k4ZFNFeXFMY0xyaXplc0xFUUNiWmczNlc5b0tI?= =?utf-8?Q?9i9n+TayD74NpAspItxRtplOHQK1K/SXHGfVJ?= X-OriginatorOrg: efficios.com X-MS-Exchange-CrossTenant-Network-Message-Id: 72542d8c-2aee-42f8-b6c8-08dedc5fa1bd X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jul 2026 19:40:43.1067 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4f278736-4ab6-415c-957e-1f55336bd31e X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XLrRDSAJekXRr5XEhzHcZSsfM16Gf5rqZwcTF4hisqwLspP3/FIpLARAROzdRj0WZMP7Rpjb5OVMzZjcGGFvC726aTYdc85sGOKTsak//rU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PPF49BB1DED8 On 2026-06-27 13:36, Paul E. McKenney wrote: > On Sat, Jun 27, 2026 at 11:21:28AM -0400, Mathieu Desnoyers wrote: >> On 2026-06-27 08:56, Mathieu Desnoyers wrote: >>> On 2026-06-27 00:14, Paul E. McKenney wrote: >>>> On Fri, Jun 26, 2026 at 11:56:13PM -0400, Mathieu Desnoyers wrote: >>>>> On 2026-06-26 19:06, Paul E. McKenney wrote: >>> [...] >>>>>> Initial runs suggest that hazard pointers cannot be acquired by one >>>>>> task and released by another.  Is that expected behavior, or is my test >>>>>> improperly passing the hazard pointers?  If the latter, we should of >>>>>> course document the proper hazard-pointer-passing mechanism. >>>>> >>>>> This is unexpected. I've looked at your test code quickly and could >>>>> not figure out what's wrong though. But it's late. How do you observe >>>>> that it fails ? >>>> >>>> Not an emergency, so your schedule is my schedule.  Especially given >>>> that I am mostly taking this coming week off.  And that it took me such >>>> a long time to get this torture test done.  ;-) >>>> >>>> I run either scenario (PREEMPT or NOPREEMPT) with: >>>> >>>> --bootarg "hazptrtorture.onoff_interval=3333 >>>> hazptrtorture.stat_interval=15" >>>> >>>> I get the following on the console, and things go downhill from there. >>>> I get the same thing when I turn off CPU hotplug, for whatever that is >>>> worth. >>> That OOPS is very instructive. I think I found the root cause. >>> >>> I need to add an API to hazptr so users wishing to explicitly transfer >>> ownership of the hazptr ctx to another thread can do so. >>> >>> Currently the scheduler hook is responsible for moving the hazptr >>> ctx from per-cpu to the backup slots, which makes it entirely movable >>> afterwards. >>> >>> But this assumes the ctx is thread-local. Once you explicitly move >>> ownership of that ctx to another thread, you race with the scheduler of >>> the original thread. >>> >>> Let me see what I can do to fix that. >>> >>> Thanks a lot for the test-case ! > > Glad it helped? ;-) > >> Can you try with the attached POC diff ? > > This increases the kernel's life significantly, but still fails as > shown below. This is the NOPREEMPT scenario as follows: > > tools/testing/selftests/rcutorture/bin/kvm.sh --torture hazptr --allcpus --duration 5m --bootargs "hazptrtorture.onoff_interval=3333 hazptrtorture.nwriters=1 hazptrtorture.stat_interval=15" --trust-make > > The PREEMPT failure gets similar failures, and the above command runs > both of these scenarios. > Re-reviewing kernel/rcu/hazptrtorture.c, I found the following issues: In my patch provided earlier in this thread, I was missing a "detach from task" call within hazptr_torture_reader_tail, because it calls an IPI to do the hazptr release on a remote CPU. e.g.: diff --git a/kernel/rcu/hazptrtorture.c b/kernel/rcu/hazptrtorture.c index 311b387d6ae7..ae9ccd89ff87 100644 --- a/kernel/rcu/hazptrtorture.c +++ b/kernel/rcu/hazptrtorture.c @@ -411,6 +411,7 @@ hazptr_torture_reader_tail(struct hazptr_pending *hppp, struct torture_random_st preempt_enable(); if (irq_release && !(torture_random(trsp) % irq_release)) { guard(preempt)(); + hazptr_detach_from_task(&hppp->hpp_hc); cpu = cpumask_next_wrap(smp_processor_id(), cpu_online_mask); smp_call_function_single(cpu, hazptr_torture_release, hppp, 1); } else { @@ -428,6 +429,7 @@ static void hazptr_torture_defer(struct hazptr_pending *hppp, struct torture_ran struct llist_head *llhp; guard(preempt)(); + hazptr_detach_from_task(&hppp->hpp_hc); cpu = cpumask_next_wrap(cpu, cpu_online_mask); llhp = per_cpu_ptr(&hazptr_pending, cpu); llist_add(&hppp->hpp_node, llhp); Moreover, I think the whole ownership model of hazptr_torture_do_pending is broken wrt hazptr requirements. We would need to "detach from task" each of the hazptr before handing them off to such a remote drain, otherwise the remote drain iteration on all cpus would basically have to hold the runqueue locks for each of those cpus to prevent the scheduler from racing against either hazptr_detach_from_task or hazptr release. Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com