From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f73.google.com (mail-dl1-f73.google.com [74.125.82.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B9A037CD20 for ; Sun, 3 May 2026 21:23:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.73 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777843386; cv=none; b=TwvoK0y7yXSO381vm8faOCMYqysCeGvL7QWF0aCBdNeCYOoZZwhSGpV1EsXI7ZrO3+FBahPu80G5FkIpn2GyLsYLigGxf3gaAU2ngJltxlTpRyv+rNiS1QVb5pG7/SPBTF9A0j+vsiR6PhIW4csjnqLtKTUyT0XV+mbDJHdpOxQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777843386; c=relaxed/simple; bh=WXrx9c+nMNDPeUEQKmBb/sq2LfOjqtKHDbSbS2TtogI=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=Td8gifzDjicDKeWk3hQrR4DCpV6PTAJxggNMe0RqsvZHJpLlDzaMlJSLmVtYHt3bYuFNfZZowCDwpjWOmekaljpWlc37xsGsCfi0T9zMakZsdUETgI/UfWLL/1/0L0/Kn+ufydwjDZ61EqQZd4fjDNomPncejD5/RRpf7mB5rs4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BS8g/htP; arc=none smtp.client-ip=74.125.82.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BS8g/htP" Received: by mail-dl1-f73.google.com with SMTP id a92af1059eb24-12dfe06b670so7173494c88.0 for ; Sun, 03 May 2026 14:23:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777843383; x=1778448183; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=Gq2Gng3Hvmwz5Vm+i5jnIMG3GH6+vilWdigdGYt3TFs=; b=BS8g/htPfQRbNSVCgHeDGETDQtqBFiWiWIngdm4lUd4Fgbanj5FnMVGOPf/D6f5Od9 WCKCWndWAuoFwcOqwto5Nhnaf3vLnVgORfM8SPtToyhqCuWSNUuiwLfChM7ndcHMVdq4 nnHuukxq/gXzYA0z5eCn5i075cZSRkGpYJCyvT+BVQqSbblcpU0RltwJFlp5MXAjA8Mo 9ATW6VwMn+Uciq5fHNi7+3/9yVWLjj4jr+Nd/737f5FDTFFw3jJZqi79IuMLtmnmUz/c KcNVZ+xZ0Bf+3THuYrr1tr84JNNTDSifDHrjdC7dJZIPv+uXF1aUX7tVey/LaVo09kkR RpAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777843383; x=1778448183; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Gq2Gng3Hvmwz5Vm+i5jnIMG3GH6+vilWdigdGYt3TFs=; b=BJHWPPtp/IQRBbTPSUPJctWoA0eW8+Zfz0qon78XQ08M0EFuK6rc25c173q/a9hHfp /vTrJXjhe0WswRDMK3isM3DHVmgy21U7QjpFWb+Bkm2DsIp2RXjaQJgTw6bxui7KLqEj vpTniLQOvp656vkabGJOkgi8ZvJQyt6MQ8Hp6BbNh5fCGoro3+z5qJZlKoa5cv0GWmF3 25xVET/ng//LPy5hPotoxMpca/X96tXCmWN4br+3Mck8hnTuywLt6y/KOrD7ygcNvOjK QiYjI0Wvjtmov7LPRHctUhDta+B6MhIgiCPzcFVAf4ntb1Q1sdCiJgAGACfyxxwvyDQZ l2xg== X-Forwarded-Encrypted: i=1; AFNElJ80QAVlRQFAzBzGTt2OTlJr3Ukl7RZ7J97pzhUx0iJ/tKVhTQ0HBnbMUxcheTO7PAazDtKx7zlkLPhmets=@vger.kernel.org X-Gm-Message-State: AOJu0YyZeR4qcWpiTatQvaisFEX0rRcVpUKH0gKA/8G1oB7ScUd1PPgx D64NRb6CsKEDlPCVQa34OLup8MTJTIuIeiGd/mcewmbySjvImLnQtDeLwNSQMnnZadPgdSakmgj x291JFaAzp3Vay5BwXqIjrIIeO1uvsKctVA== X-Received: from dll26.prod.google.com ([2002:a05:7022:21a:b0:12d:b48e:8597]) (user=aniketgattani job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7022:699d:b0:12d:ba50:4baf with SMTP id a92af1059eb24-12dfd85ec6cmr3267112c88.33.1777843383182; Sun, 03 May 2026 14:23:03 -0700 (PDT) Date: Sun, 3 May 2026 21:22:02 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260503212205.3714217-1-aniketgattani@google.com> Subject: [PATCH v3 0/3] sched/membarrier: Use per-CPU mutexes for targeted commands From: Aniket Gattani To: Mathieu Desnoyers , "Paul E . McKenney" Cc: Peter Zijlstra , Ingo Molnar , Ben Segall , Josh Don , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Aniket Gattani Content-Type: text/plain; charset="UTF-8" This series mitigates a severe contention issue in the membarrier system call by replacing the global membarrier_ipi_mutex with per-CPU mutexes for targeted expedited commands. Problem: Currently, MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ relies on a single global mutex to serialize IPIs. On large systems with heavy concurrent usage, this creates significant contention. The issue cascades into hard lockups when combined with CFS bandwidth throttling, during which target CPUs may have interrupts disabled for extended periods. If membarrier is waiting on such a CPU, it holds the global mutex, stalling all other membarrier callers system-wide. Solution: Patch 1 introduces membarrier_cpu_mutexes to serialize expedited commands specifically when a target CPU is provided, isolating the lock contention to the targeted CPU. Broadcast commands continue to use the global mutex. Patch 2 modernizes `membarrier_global_expedited` by using scoped cleanup guards (`guard` and `__free`) to simplify error paths and resource management. Patch 3 introduces a dedicated kselftest reproducer (membarrier_rseq_stress) to permanently test this interaction. It aggressively hammers targeted membarrier commands within a deep, aggressively throttled cgroup hierarchy to prove the lockup scenario and validate the fix. Results: As measured by the stress test introduced in Patch 3, testing on an AMD Turin machine with 384 CPUs (2 NUMA nodes, SMT=2) shows: Throughput: ~200x increase in successful membarrier calls. --- Changes in v3: v2: https://lore.kernel.org/lkml/20260415232106.2803644-1-aniketgattani@google.com/ - Fixed the code path when `cpu_id < 0` in membarrier_private_expedited as reported by Peter. - Added Patch 2 to use cleanup guards in `membarrier_global_expedited`. Aniket Gattani (3): sched/membarrier: Use per-CPU mutexes for targeted commands sched/membarrier: Modernize membarrier_global_expedited with cleanup guards selftests/membarrier: Add rseq stress test for CFS throttle interactions kernel/sched/membarrier.c | 98 +- tools/testing/selftests/membarrier/Makefile | 5 +- .../membarrier/membarrier_rseq_stress.c | 951 ++++++++++++++++++ 3 files changed, 1002 insertions(+), 52 deletions(-) create mode 100644 tools/testing/selftests/membarrier/membarrier_rseq_stress.c -- 2.54.0.545.g6539524ca2-goog