From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from YQZPR01CU011.outbound.protection.outlook.com (mail-canadaeastazon11020133.outbound.protection.outlook.com [52.101.191.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A420D211474 for ; Fri, 10 Jan 2025 17:02:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.191.133 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736528554; cv=fail; b=a0AJaFV9REKi8qpeWom8O2TVFQUQNtzAhSD6bPFp/eLeY4X/xTXiSeqvmnpinzQMMvaZxlhv7dXGsxiSgNLtpsHWyZGxRljoy7buy4OYZGD3WpOB63NA7yfXS4dA/XnkMigyVmoWS3/4pqBdTERrFN6TqWzxkeRMepCCgG3e0Zo= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736528554; c=relaxed/simple; bh=CZB2R2wBbhnw9pgqEnMY5LQhl4N14EKhaLN+W3sQZUY=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=i9e2oEzopW/hFt1QXUjpkejmZ2nVLactpDg+p4pzaEMGMzTwCjk7Y2IH1OoLErMBOZzC9hL5NYM6Cn9uo4+sxUUq4UJ7/ZQGrMQU/M7Fl1fj6d8oadWVVApOK47z5ltxkNG+GmfzzX0dJsxs/jKzpsTv+7ZP88igyAHNql1408A= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=JHpIm2pb; arc=fail smtp.client-ip=52.101.191.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="JHpIm2pb" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vXcuZ2gOQLF2Q5eECZZgHmpmhRBAaHAz3OMn8iq4UwHd5Hr+w1bQGdYha0TTQysviLr6sB4MpWaUaLC+jfce9k0Fcq9aPbX+18Go06D73uNdYbcDoFZzZ+i3QEfO+kv3s+RZq3KyE4c7AYwnWqIuMF5PGYxnmbzD3OTttjITYMk64tmvo/AVhwGgFQSqKNqkAyJAhGbyCN0TLnyPzH2i+g4hwV9AIwsSAYD8z3Qz/4kBh3olhHzjh+2HKMHf4EnnjZ3m98cGCnX+9ScybuDod9qHsc0rgrGXpeRh35iQH/yEHjp6pxiCxRjFEWcxJ7ZEZ2CJNO9nkwtYAIbG4KiYiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uP8Jq7nyPIuKeRdVWfAV2pOX+JQ/uQRA69XhYMO4qSM=; b=iHEEN6O4o6R9VBA+hQODJ4L6ZxTKmed5unUZN12UZV0r7dlEKnYEmPjq1rB7t5uMxW4TUWG6IXuQko2aZcSQW8x0wAxkZ7X0n/YxukqJLvFIoN7fEPhB63xWFGkm0/jWB6RU33SpU2uqJ3oyP5zZaycWx5R/r/UC90VNUAMGv+1xLyVeV7jk0ZoqYbN0sDZ0bJAl9qwUW5eBZL/zOlmOyYy/P9C0CekFb9cjaARVMUtvkSaeqd7g416JIbcNFBWxwZLaFqmVNRvoEtzJJffgYXEurakq/67riFjnu8PLD+CsU6TnUAgRew/tqsmhAndNW4JbqpCdyEiQxgxM4uUPQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=efficios.com; dmarc=pass action=none header.from=efficios.com; dkim=pass header.d=efficios.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uP8Jq7nyPIuKeRdVWfAV2pOX+JQ/uQRA69XhYMO4qSM=; b=JHpIm2pb4sYI2W/q/rdIwbT0ElSm07IZ9rT09TbpJl9VwMaitYPmVWVOC6/yQADEyLnd+6bgSnlGF/SbmxXzEMF9PvPF1SSg9IalBitRg8yAb4azObv6lh5shX2Z4951mPy0AHU03MalpB5n95fgcfqFk/zzymfNNwtvBb5UfuzyhZ8OefF4rf6GVW/R29A/zmRM0H8UzJVZgkWWt9IrUhvOAA2x+TqxMvjZefxkpNNLmGaiJghUw4ustwJ/sFWPNw4eESls2w1cg6y68dO1Vr9hIKZQd18OnB0zWbxd+J6ej7arFr1+z1Q9ClRmJfPrVG+FC/A74LtbPagw0d4EGQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=efficios.com; Received: from YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:be::5) by YT3PR01MB10329.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:89::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8335.13; Fri, 10 Jan 2025 17:02:30 +0000 Received: from YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM ([fe80::50f1:2e3f:a5dd:5b4]) by YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM ([fe80::50f1:2e3f:a5dd:5b4%4]) with mapi id 15.20.8335.012; Fri, 10 Jan 2025 17:02:29 +0000 Message-ID: <8c1ad304-61bb-4bdf-aa75-8633f3d0196c@efficios.com> Date: Fri, 10 Jan 2025 12:02:27 -0500 User-Agent: Mozilla Thunderbird Subject: Re: Prevent inconsistent CPU state after sequence of dlclose/dlopen To: Peter Zijlstra Cc: libc-alpha , Florian Weimer , "carlos@redhat.com" , Mark Rutland , linux-kernel , x86@kernel.org, paulmck , Michael Jeanson References: <20250110165412.GC4213@noisy.programming.kicks-ass.net> From: Mathieu Desnoyers Content-Language: en-US In-Reply-To: <20250110165412.GC4213@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: YQBPR01CA0104.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:3::40) To YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:be::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: YT2PR01MB9175:EE_|YT3PR01MB10329:EE_ X-MS-Office365-Filtering-Correlation-Id: 8a647d00-291f-4926-aa74-08dd319890ee X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?WlZSV21qVENXOWZHNmM5TWdWY3NyeDgvdmJjVDBaY3pYUnMvYkxnaXVpUUsy?= =?utf-8?B?Rm9mcU1nVFU5bU9GM2hsQTN1UTJWalk3R2dxWVRkaVArUlpEVGFndWlldEpV?= =?utf-8?B?NEZaZEI3ZFg2VHRDQWZTMnc1SHBwS2hwaklJMGIrWGR1eXY0TkpIRFZaQU9p?= =?utf-8?B?NUt2cW9UT241RlZBcHVZQjNwRmFFbU9qenRXTDVuUHY4TFlKU01nZTFDY3M1?= =?utf-8?B?SUczL09SRFJtMEI3K2xwTmpVL1hoYlVwTlpVUnJMd3FzYlozRURxS2Nhbno2?= =?utf-8?B?MWQrbVNqMWJreHZiTjJkSG1UWDZIT2x5VXJQc2NvajR5bDU3NUFBM0V5ZWR6?= =?utf-8?B?c2tBUGhnSWVsMWlwWjQ5ajladm5UUDV4V3NlZHlrNXhnbk1lMTR6RnZxVUtm?= =?utf-8?B?em9sZXdXTTZNVjlLVU1MNnJMbG80WnNHYjdJTXZld01QL0R1Q2gyaU43V3or?= =?utf-8?B?VFUvVUErd0dGck1BaFVWeG5JVGlGWDhZRWZPWkoyQVcwc1FuVlUyRHYxcTdC?= =?utf-8?B?ZG9NQnJ0WVEwWDNpZGVuZUZlTTZHNUtocVpLejBtaytZN2tveTdSZENjdldh?= =?utf-8?B?b3JIYXA4MDUvMDF1dzF3S2UrbW5SQWdiMU5PaUt3M3VNS1E4Qys3Y2IvdnAv?= =?utf-8?B?RmJsVWZxa0lNRDRXWnpzaEVzQTVWVWFwWENIT0cySHgycjJkcHVQcmcvWG82?= =?utf-8?B?S0hjdTM0UW1mYmZmU1JEcEhuTmJiTElKZHJGMUoxNkp0MzJRUHlsMElFekpW?= =?utf-8?B?bEtuazlUSVlqVVJDN3JkbWJDRmRzZkFHTkw3NWNwT3hPSTUwaGpnK1ZiZG8r?= =?utf-8?B?dTU5KzR3MUY5RThUWkRzQ0NCN3lLNlpwSWR6ZFhjNXJhOCtLY1JGbWFMNWNt?= =?utf-8?B?WDhoZUxDYVNraFZlWnFrM0F2RkNseEpZeFhyYlY0Y3loeWppb3RZVlJQU1VL?= =?utf-8?B?dWdJMWpsSHdTOTZ2MVdhVVVzRlJ4Rkx5azArTjNVWjZTWlFJVjlwQnhUWDZn?= =?utf-8?B?M2ptT2x0WC80YzRvYUxIcEFhMlRUancyNjJSZS9Hc0l5ckNBTFdKZk9WT3Ny?= =?utf-8?B?bGZ0UGJ4bEExYmxSQ0xVOWZYZDJXOERSaGZ2NHdnNm9rT2NyaVVQOHN2d21n?= =?utf-8?B?Wm02dTVtU2l4MlAyR0ZFRVF6WmV2WThEMWoxbUpIcFJDQk44emphUEtxRkZU?= =?utf-8?B?eGptSkdnUWRENVZHbmJvZnlJaHFPREowdk1sNXJ3QmJXOWZrOG93dERFUk9k?= =?utf-8?B?eE5lV0VxRnpQRjUzNjdTYkZKR1A2dm9MWmszWWJFeHFYdTVjQUNnUk9RZzdB?= =?utf-8?B?bTM4Ykc5eE9yK0xVTTkwakV2VEpGTFZrSkRhb1lObDVMUjRQVWNiY3FLQXg0?= =?utf-8?B?SnByZDd4Q3Q4ckRlUzBjYkNrOVVwTzUxZExMbE1SZWh0eUF3REFZM0lTSHZU?= =?utf-8?B?Z05nOVpYODNBaXFCZC9jWkZmTjQxWFA5WEJBbmFVY2krR0NaeFdTMnROZXI5?= =?utf-8?B?cldheXpQTDB6OFZZMFA2T0UxMThYeXpsbUZQVnlpOXdpTFdsci9xak5xYWx1?= =?utf-8?B?Tk4rU2dGbWVTMzdkdWh2OGQ0Qkdpb2FCZTR5L0JMSHhiR2xLN0FrUEdXeWVX?= =?utf-8?B?cjV3em94TkFJeUFDSklqbS8wRWhCanloajJreUxxU2J0Z1E5cUExZFI5bHlZ?= =?utf-8?B?cVUxQmpQdXJlOWxXRFdvaDZrcERZVUw4UkErU09GeEhHbTBpRXZERXVqZlFp?= =?utf-8?B?Q0xQa2tNZUxWb000TmNnT0tLS0Z5QjZ5T1N2Ylg2TFJzQmZLV3VZd1BCVFJq?= =?utf-8?B?N09odndPc3BqUEtEUUJTdz09?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bHQwZ0EzcTQ3c0pmdXpFZDJ0bk52cmZLVFBxRjBPQ1dWakFMcUl6MzVNVUNh?= =?utf-8?B?VENKcDIvb2N3MTdWb1gvUDJXZ0xuejVtN2wrYkhZNldBSkVXMjN4R3U1eTNa?= =?utf-8?B?SjY5NHhNNDVwZkt6eGhNRWF2NnFjT01vN0xkQVo1MHdPVHAzNkZ5T1QyRDZV?= =?utf-8?B?bnoyVUMrbzJ6UnZ6UUNnZlJQby9SREp1eVF3MVJWekl0UGE3czNQUGlmUWRq?= =?utf-8?B?Y05xRHZkMjZOa01tTEVlSHVLbXRML3AyMjRRZnV1ZmxGUU1RcjJnQlJuSllO?= =?utf-8?B?SXowajdxZ211d0tsSzVTc2Z3L1lJSlNHS1UwNHRWRGZ0QnVaRmxqNkNoWHVo?= =?utf-8?B?UGh3UnlzOU9KUmU3VlZ2TDQ0L0JUMDk5N2liT0FieFVMdmRnV2NoVFJoWU91?= =?utf-8?B?bmhqcFI4TUN0WGRlTHh3VTd3M2Q1UHpSc0MvL0NieUsybkJwOS9pa1NzMENn?= =?utf-8?B?UmkyT0s1KzAyTlgrK1dPTjlYYi9Ra1ROYk0zMWF0T3pTdDA2VTM2MzgwekhG?= =?utf-8?B?N0I5U0Y0OGdtaXdmWGRzUFVISmxNVlVnYVg5M1V6QVdvaW9BSlhjZHJKVEVH?= =?utf-8?B?bm1RRW5GdnJrRFRvVW1Sb0J2QjR6N1Nnb2JBUFd0aFFFTXErRFpDQkYyMDFp?= =?utf-8?B?K2xZQ0FzM2FIUGR1SzZvM2cvV1JCKzBwRE9Kb2MwVFhXdU1Jbm0rV2JlTFhY?= =?utf-8?B?VVVReDR0ZzdGc2g2VmVvalNKVVlGNitDbngwRm05K0ppNWJNbStJaTlFZUpW?= =?utf-8?B?UE1ZMVJ4M1VUSXQxZmJIMjAydlNuaUVUdjRqaFNUYTdibXpEYllyT295SFpN?= =?utf-8?B?UzZuN3FZYm0ySzd2cU9YTm13a3dOdG50ZVV0b0NCV2ttVks4aGVzaERETCs1?= =?utf-8?B?SDN2R0V3d24wbXZRUTdoSHZsWE9lc0hyNWhCY3dJTFB4S3ExNkpyZVRUMmwx?= =?utf-8?B?dFJnb2E0RWpDbCtjblRnNmsza0ZtOHp3aUpUdHN4eDNEWHpyNG1odjc0U1o5?= =?utf-8?B?ZWROeEpwam8vUUNNVkFDRzlpVnQvUkdGSFZ6Y3FMUTMyVW5WY0c4Q0hDalB0?= =?utf-8?B?aUprRTZtTWM2ZXByc3VBdTFZcUlVVnpGc25KbFhtSVBMdGhqOEZXQnZ4SzN0?= =?utf-8?B?OTNHYkpxRjZwT1FwQ25ZdWtOZjJJd1c4NVNlckhVcjRaWXlObkVIdmJQc09J?= =?utf-8?B?TnJGTzBCSVAxWWhQZE5uK0tSWE8zekowRnpvWUFoQWRodFlib3VNN0ZYMzM5?= =?utf-8?B?YitCbGwvRVp1VXp0VUozVDJlT2dJaWt4MFNOQ0dKY0VwQUtFVjhEQW1DUzhO?= =?utf-8?B?ZCs0VnhIZ0xKejRkM1hOTThielY0bFRXckk4c2x1UDNDbk1ZMXdLWE9UUHNO?= =?utf-8?B?QUhSRGNYeUF6ZkttcU9LdkU1TDdjUEk1NEJEMThMWk9GOG5JdkhzVHhGWnRz?= =?utf-8?B?NFc4QXpUdVhVSEVZbTBIUEV5MTArZWNzbVNMcFBTYjhFRyt4SDhEZHAvU05B?= =?utf-8?B?NDdhVk5uTEhVSHRqU3FZa3pHK3dNY1JPT3hQV1FNYnhmMDRDaDVDaTgwSTFp?= =?utf-8?B?N0k4TXpMNUJpM1Fjd1I3bnRoV2Q0N3VnNjl3dUlYK0hmNVpFZTNhRGxYb3ZS?= =?utf-8?B?eC95YWdaeW90MHk0aUxaL3hXVkRTR0p2R2drVmJ2RFh2NFNlUFZKbVNNVWhX?= =?utf-8?B?QlBreHZZYVhqdmgxZW45elBTandoUXYvY0haWDFQYm5mcGV0Y2p5KzBnNkta?= =?utf-8?B?SHNtZVZkNXYvR2UzUE4zYnowdmRRSzB4UW5KcFVQd1B4OWZSVURiZXI1Y2dC?= =?utf-8?B?L0hWS1lDZ0tBNzFOZDZDaWVjdmFveGlkQ0NEdnlLbVo0NUl4V0E4a3ZjZVpP?= =?utf-8?B?TU5HRXk4Z0tHS0NOZGFtbTFRYnlqcy95M0EvT2lTYy9hMzg5ZWdzRTB0TjZO?= =?utf-8?B?ckJqOG1laEVGZVQwNThrUFFBZGxjY1dCMHRPM1BIYUt1WVVSbkFaRjhZMllW?= =?utf-8?B?RXVOalgrSnhub2FhUWlFYllzT3BBRmtJRnhuTUdWbXBMUG9pZlRhazV1dmlx?= =?utf-8?B?d3JkRHU4eGhmK1BoSEIxVUZqRFAzc01ORnVUc0JNZEliQXRYMm1vYU1rZThO?= =?utf-8?B?RWlOZCtRMUZhVGhnUUxMQWpkQVdjazBGdXZBTzhxajMwcFN4MDl0a0V6SDN2?= =?utf-8?Q?F7/sBVYaDs9zkpcidpCwNaY=3D?= X-OriginatorOrg: efficios.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8a647d00-291f-4926-aa74-08dd319890ee X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2025 17:02:29.4756 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4f278736-4ab6-415c-957e-1f55336bd31e X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cB+4Of/8zk4+VznDyYantT2nRVADhDCUPzpckupTCWID9nSQH67fZPm1HtEW5P5XIqzKM8EuAUhJkBKgJMdD09LnVT87Qfo0eIH+Nal2Lpc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT3PR01MB10329 On 2025-01-10 11:54, Peter Zijlstra wrote: > On Fri, Jan 10, 2025 at 10:55:36AM -0500, Mathieu Desnoyers wrote: >> Hi, >> >> I was discussing with Mark Rutland recently, and he pointed out that a >> sequence of dlclose/dlopen mapping new code at the same addresses in >> multithreaded environments is an issue on ARM, and possibly on Intel/AMD >> with the newer TLB broadcast maintenance. > > What is the exact race? Should not munmap() invalidate the TLBs before > it allows overlapping mmap() to complete? The race Mark mentioned (on ARM) is AFAIU the following scenario: CPU 0 CPU 1 - dlopen() - mmap PROT_EXEC @addr - fetch insn @addr, CPU state expects unchanged insn. - execute unrelated code - dlclose(addr) - munmap @addr - dlopen() - mmap PROT_EXEC @addr - fetch new insn @addr. Incoherent CPU state. > > Any concurrent access after munmap() / before mmap() completes is UB > anyway, no? The problematic access happens after the second mmap. The issue is stale CPU state. > >> I maintain the membarrier(2) system call, which provides a >> MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE command for this >> purpose. It's been there since Linux 4.16. It can be configured >> out (CONFIG_MEMBARRIER=n), but it's enabled by default. >> >> Calling this after dlclose() in glibc would prevent this issue. >> >> Is it handled in some other way, or should we open a bugzilla >> entry to track this ? > > The problem is that the membarrier() call has significant cost, and is > only really needed if dlopen() is called right after (in the same > location). Or if it has any overlapping executable range. > > Unconditionally adding that barrier, just in case, might regress things, > no? Or perhaps we could add this barrier within mprotect(2) and munmap(2) in the following cases: - mprotect removes PROT_EXEC from a mapping, - munmap unmaps a PROT_EXEC mapping. Else userspace has to explicitly invoke membarrier sync-core from dlclose. Thoughts ? Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com