From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC3A53D9035 for ; Wed, 11 Mar 2026 11:53:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773229990; cv=none; b=Raklfc0hTyYCDCLzoUPVbT7mm50Rs/xvnhpoU6e3l5oLxIh4AFg9MxEGpuawbhyWYJNq2Bn1MNhbozoxiu53czIWLy+xGbybL8yQ9vNNP+PJAhYlG5A6lVrmR1U/Z8QnzBHCAJTdstbP7BRBClVe+rnJyyqrDwBleCrw12qSQbA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773229990; c=relaxed/simple; bh=SgzzhxCWuPKU87hF7zwmiXoVGYhYQd/RZUgCpsFS37A=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=oPfS2zmvWkcjCDUvzJixhXroNMWX/u2JrnuCY8ZKWCwqlk3uGgWUAQqN1btqbxVWJC/iz9/nHjmIvujqLkuP9LuWahq0FpHk7rDu1rIpiVoxLuUn0QS8P/4fQN1YEBVn4ydu/vAHeyBIVreFFQEyzElXq8i9ZG6g7Pllyjw8pUU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bmZniczg; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bmZniczg" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-82976220e97so4033085b3a.3 for ; Wed, 11 Mar 2026 04:53:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773229988; x=1773834788; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=+Tcuv/FKdpmW54ZWbVfc0fc5h5dY5q1jjZuFCsgdu2Q=; b=bmZniczgU2Ju5l7eI+lyUeMEjSHBDh0rEMeqGGZjn8SdE5QSELzM6F1qyvuHRlS6Cf rpfdULXtrxCzpuO5J8S+eehgInc7NrWju9anRqk9ofGDOmEYuS0I9MnSit4alhLFfHR/ cFSfTdpWDOZ1nj0r9veeXP23UFMKJzCrkUyB79WlRt270EOEuyn8UGwxJhWm2IqKrYep HGI8b35ZBc4rqINSSwpbx0A7gpmJ/pBozjEjU2t6RKKaFfuUG9VH4FnTOq97w+/lA+2O xQZg3WA0hld/jFA9a6O07sV5Q2VX+c2K32sybuF+E67+4XCveGmXKmLA8HQrxNus81gA G/Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773229988; x=1773834788; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=+Tcuv/FKdpmW54ZWbVfc0fc5h5dY5q1jjZuFCsgdu2Q=; b=P9fD0nmkrfsaCdOfJW4ARZIK+U8Q1I+JXgaTbLtdGvvRu1iv6lSVllRJzi/i/wBfHk nOu0YPOH9Ti44HHHGDckZJst+9cNfyIhzRCz7Arg5qvO22fRplnPQ/ABJSRKVgjNPSk0 fEB1/65TQeToUF7YzBdriySzfBeLdMgM/JPsom0Z2Cg4uKM8PZW8TQOLYBFdu33w2t23 1MDGcyBt6Tfc97ae+3eGHAeIJsyXREZcgKXHYGn5cVql5ve4P5eTqDHszSgBBUwBJnmh 3ISHGCFos5e39KHIEqpc53o+Pjii4oZQI2efVNTfA8nre0HimXDVZPsnxoUZghdEFq2J ciuA== X-Gm-Message-State: AOJu0YzsuGQ2O1sOjkAbhn5Yy7Z5DNRo9sjNUQoCbFSERJVnB1U1uavH 1UMA3Z/MLsg5bqZ248rxGl7W54vfNQZDrZtSgEqWraY29ZkFAeYeFg6j X-Gm-Gg: ATEYQzwVM1jl1MyRbY7Wx+pQxXW0U8WzrAASHZgs7UtMdpTDAK+d6LR12l9RPU0xbQf DI5/ObkiL1an1FRWy/HCO4JqAm0SE9V9ogtxU+gUceKEv72hiow/zcpBVt8TIMg9SICOqUcGaQ9 AbOpN00ta/9Bhb+rmlYvdj16uEQ2vfiYTO1n5cROn5o3HZBffQppq/e+iHXx/ik9K4ZkAWDIHXC rvNxtVFh7G08AI0BTKp1OZPbsSJTLJiaseCIn333RZoJ+PYnn1iBwVkF0bjJgDcvKjYzBeh5rvq cFcumurkUQTU2m21y5npUvWMgZq+PTUXyL73GpvFOOFO7pAgmVZpjlSzDkCeZKtjCYgTYwiIRHl PT8pC3WEjB9bWBF8IPm83aerkfofIvWzhAjdaNIZfTio1R7Qs/fhM3ihH6AslhmbDl6LfWuckdc S/8Bl5TQ4ygd2bEQYH99LDQ7TS9m/Qh+J2ZlHxVQlMjUcg5OuhTSxjQCd7ksvCA88= X-Received: by 2002:a05:6a00:1c9e:b0:829:88e7:c89e with SMTP id d2e1a72fcca58-829f6ee65abmr2514048b3a.9.1773229988180; Wed, 11 Mar 2026 04:53:08 -0700 (PDT) Received: from localhost.localdomain ([2409:891f:1b46:28af:1445:b4e2:b8bf:2d47]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-829f6df3aedsm2500588b3a.18.2026.03.11.04.53.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 11 Mar 2026 04:53:07 -0700 (PDT) From: Yafang Shao To: peterz@infradead.org, mingo@redhat.com, will@kernel.org, boqun@kernel.org, longman@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, david.laight.linux@gmail.com Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Yafang Shao Subject: [RFC PATCH v2 0/3] disable optimistic spinning for ftrace_lock Date: Wed, 11 Mar 2026 19:52:47 +0800 Message-ID: <20260311115250.78488-1-laoar.shao@gmail.com> X-Mailer: git-send-email 2.50.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Recently, we resolved a latency spike issue caused by concurrently running bpftrace processes. The root cause was high contention on the ftrace_lock due to optimistic spinning. We can optimize this by disabling optimistic spinning for ftrace_lock. While semaphores may present similar challenges, I'm not currently aware of specific instances that exhibit this exact issue. Should we encounter problematic semaphores in production workloads, we can address them at that time. PATCH #1: introduce slow_mutex_[un]lock to disable optimistic spinning PATCH #2: add variant for rtmutex PATCH #3: disable optimistic spinning for ftrace_lock v1->v2: - add slow_mutex_[un]lock (Steven) - add variant for rtmutex (Waiman) - revise commit log for clarity and accuracy (Waiman, Peter) - note that semaphores may present similar challenges (David) RFC v1: https://lore.kernel.org/bpf/20260304074650.58165-1-laoar.shao@gmail.com/ Yafang Shao (3): locking/mutex: Add slow path variants for lock/unlock locking/rtmutex: Add slow path variants for lock/unlock ftrace: Disable optimistic spinning for ftrace_lock include/linux/mutex.h | 4 ++ include/linux/rtmutex.h | 3 + kernel/locking/mutex.c | 41 +++++++++++--- kernel/locking/rtmutex.c | 37 +++++++----- kernel/locking/rtmutex_api.c | 47 +++++++++++++--- kernel/trace/ftrace.c | 106 +++++++++++++++++------------------ 6 files changed, 157 insertions(+), 81 deletions(-) -- 2.47.3