From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-vs1-f49.google.com (mail-vs1-f49.google.com [209.85.217.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A276A2135B9 for ; Fri, 10 Jan 2025 17:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.217.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736529168; cv=none; b=Wz0lK5IVJbAER9G3bAka3TJBvFrmkOC2h2pA7t8QtPSpjwrO/IeObUNfMYQ9Y5RIurLqcd/Kila8hs+Rv5Y1eS+u+5GqRSJUuRGdxxeF3q9U13OEjU56OeYN9BFqa9xxzJCgM6w1uJv9XPQ8M5016cnqwomrUKE48DdvQqro6Xc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736529168; c=relaxed/simple; bh=6sUMiQH4SPgKD35xv/kOAoXtnA2cY3MO+jtzoe0WRdU=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=bxUp0Ifi66BpKsmyFE4HAmyUGivpr/6aKtqtCh1j5axp3bih66LX6kGMqQn5q//WmPql4qRw65c9slnJdGoNs6YagvUoZqMmQFD0w7FY8jARufGzXhFXYfmV+DKNSeDAVlfXET3FNI6uVH95RIj9cirFF73OruauCHwVhe8RpOs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=Q0aB2aqe; arc=none smtp.client-ip=209.85.217.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Q0aB2aqe" Received: by mail-vs1-f49.google.com with SMTP id ada2fe7eead31-4afdf300d07so1316609137.3 for ; Fri, 10 Jan 2025 09:12:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1736529165; x=1737133965; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:organization:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:from:to:cc:subject:date:message-id:reply-to; bh=BL+YU/3kI+titGXrArU1QWxrcn2C5B4gKn6BsyEVRK0=; b=Q0aB2aqeuM51EBytuW/kcMek5xcxyACjhK/qkK+J5AK6odfhRd4mclyFVktHUt6C9z 39EeNYDcgBxD3HC32J8M7j8UGzaq1LoI5KN7zBFkQE5nd1/VUdl4sE9VND1yA/T0Y71k luLFMuCcRFArSat8mmkfDKXCEn4G+fC3u7QVCa5pDHIAFcc5FZHtRWdBpGWG/l9tTbB3 XtVzsmApY1t7IRwi8PUy7KM9SfHR00pRZgxNzNIQPCe1tsXqjl/x/AmGELjoWqhBeD0x fd5rfmuUqpTql3t/UvcyH6zfQFMMfQdgcN1lYYrkC/sL3O62wpKvSniCWrGmWwI5don6 ifhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736529165; x=1737133965; h=content-transfer-encoding:in-reply-to:organization:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BL+YU/3kI+titGXrArU1QWxrcn2C5B4gKn6BsyEVRK0=; b=ITUQO1lk1N+9FvBWU/NPDhWOyO9XqHcZ11pOVKyJZbMIYSMSf+quLk1L2wUZ7DIURQ FbXmFb0NbB83NlvzU4NDIT96+ycEZHaUCfXjmS3ZMnsHY/U8M3BpC6Ve8i5nghsbVT/G fcwJuQdNf60vkumMMVdb+0aWCxRGMjPvZ9xyPFQb8vWn5X7fdVL7vXDuU4nOUuIci80d SpIgGBu1Jj2IBNaP3yeVMYlvFP20wmqfbkojAFzf6siTkDPItSxua1wv7Lgd/9MA7yfI RlxlTfoqZbFstxKrumSvjnUayEtz2uoILRYhxpn3bKgOlm+g3Bs6HvSmN5eLgo92P5pP Z1uQ== X-Forwarded-Encrypted: i=1; AJvYcCVXTSUvkbFpdhoXwVXcaJjdXFvEUsWFO5c9hFknLs1IncgjhazLzWvaleh6Zj753UWHw1/jHVXzWGA4wlU=@vger.kernel.org X-Gm-Message-State: AOJu0YxX0LCKvHU6cKMAMIMcCZn5XyBl3751gFIpxg3Oeh4EKokXvL65 1Owpek51LQQ2D9IJAIRFf509ccHUvGQ9r05yJu+hmTLO2dp4qVVf1R9r/JtN/Q0= X-Gm-Gg: ASbGnctCN0QVBfy0Lh61T8H5YM230WCCREox80TFbaNqKGcbLBspgj3Fej2Heykmcdz aKNSNacvVChR6ylC1UYKlw1/8AumZFd1QD0IBSQP17wISqJPrrvJFbmbMv+hGIf16Lu3SCSyrjc tXA+tmc/aScaDgPwqByJziTWlj/83bIqun0WMOE1I6raryoectzRFFuwHYtzu0VFEu8LeGbhpbH CYYUvtVtQ4xomJNCuYnXuaah/68s8qmPbXjfU5+2G8z+NK3SFQRfiBWcicpreZ6W3bT2YEM/BEh 6AbMycSHYeq2L42eW6LTBtxzhabWnjx/RncF3OavsM6C0JlE/BZDAeoQ0fY= X-Google-Smtp-Source: AGHT+IFAtk6snK4szIRfhSomgW11KU8P9SReHGmWsnIk9HwEhwf2rdV/AbkD2rQ/+G0/2tIRF+zvJg== X-Received: by 2002:a05:6102:5491:b0:4af:b94a:3c3e with SMTP id ada2fe7eead31-4b3d0d78d32mr11724406137.5.1736529164216; Fri, 10 Jan 2025 09:12:44 -0800 (PST) Received: from ?IPV6:2804:1b3:a7c0:f41c:e104:7f6c:f2c3:2134? ([2804:1b3:a7c0:f41c:e104:7f6c:f2c3:2134]) by smtp.gmail.com with ESMTPSA id ada2fe7eead31-4b6091da783sm2691498137.29.2025.01.10.09.12.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 10 Jan 2025 09:12:43 -0800 (PST) Message-ID: Date: Fri, 10 Jan 2025 14:12:40 -0300 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Prevent inconsistent CPU state after sequence of dlclose/dlopen To: Mathieu Desnoyers , Peter Zijlstra Cc: libc-alpha , Florian Weimer , "carlos@redhat.com" , Mark Rutland , linux-kernel , x86@kernel.org, paulmck , Michael Jeanson References: <20250110165412.GC4213@noisy.programming.kicks-ass.net> <8c1ad304-61bb-4bdf-aa75-8633f3d0196c@efficios.com> Content-Language: en-US From: Adhemerval Zanella Netto Organization: Linaro In-Reply-To: <8c1ad304-61bb-4bdf-aa75-8633f3d0196c@efficios.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 10/01/25 14:02, Mathieu Desnoyers wrote: > On 2025-01-10 11:54, Peter Zijlstra wrote: >> On Fri, Jan 10, 2025 at 10:55:36AM -0500, Mathieu Desnoyers wrote: >>> Hi, >>> >>> I was discussing with Mark Rutland recently, and he pointed out that a >>> sequence of dlclose/dlopen mapping new code at the same addresses in >>> multithreaded environments is an issue on ARM, and possibly on Intel/AMD >>> with the newer TLB broadcast maintenance. >> >> What is the exact race? Should not munmap() invalidate the TLBs before >> it allows overlapping mmap() to complete? > > The race Mark mentioned (on ARM) is AFAIU the following scenario: > > CPU 0                     CPU 1 > > - dlopen() >   - mmap PROT_EXEC @addr >                           - fetch insn @addr, CPU state expects unchanged insn. >                           - execute unrelated code It is not clear to me from userland/libc perspective how this would happen, since to dlopen get the same address you will need to either dlclose or call munmap. Either you have UB where some thread dclose a library while is being used by a different thread, or the thread will ended up executing a potentially different code it is intended to do. Do you have a realworld case where current glibc code show this issue? > - dlclose(addr) >   - munmap @addr > - dlopen() >   - mmap PROT_EXEC @addr >                           - fetch new insn @addr. Incoherent CPU state. > >> >> Any concurrent access after munmap() / before mmap() completes is UB >> anyway, no? > > The problematic access happens after the second mmap. The issue is > stale CPU state. > >> >>> I maintain the membarrier(2) system call, which provides a >>> MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE command for this >>> purpose. It's been there since Linux 4.16. It can be configured >>> out (CONFIG_MEMBARRIER=n), but it's enabled by default. >>> >>> Calling this after dlclose() in glibc would prevent this issue. >>> >>> Is it handled in some other way, or should we open a bugzilla >>> entry to track this ? >> >> The problem is that the membarrier() call has significant cost, and is >> only really needed if dlopen() is called right after (in the same >> location). > > Or if it has any overlapping executable range. > >> >> Unconditionally adding that barrier, just in case, might regress things, >> no? > > Or perhaps we could add this barrier within mprotect(2) and munmap(2) in the > following cases: > > - mprotect removes PROT_EXEC from a mapping, > - munmap unmaps a PROT_EXEC mapping. > > Else userspace has to explicitly invoke membarrier sync-core from dlclose. > > Thoughts ? > > Thanks, > > Mathieu > >