From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f202.google.com (mail-dy1-f202.google.com [74.125.82.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64AF037CD20 for ; Sun, 3 May 2026 21:23:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777843391; cv=none; b=kfSaDNF+7sNq5/Gk22XVd1mnXx5IUDc/1Uc1s4Ry05M7KfDfaiNjRLA3Y2IKSeURPdZBy/ONVFw+Tje2Wip9ZiJN+F6Ye++2lz4sMUv+zG/SOb3pqGbka5c1eSbWANBozOc5cn010ADPJjfRRNstC5IcvgqVZXQHN0WANh2Qas8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777843391; c=relaxed/simple; bh=tcfwiHfBmdPlDswQOFCg9Ur4ZjaLagCYDx1ehhgDEQk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OFy7UwSR/0vZAlZvgQGHl1UuvGC0N4OIAxnLpW+hYZKATIU5A9iYkEVN8XxzZ/rEB3leWD9AnwWV0UMHW9pZ6Dp1hHR/wGgShXIdI44MvDiZ2fQb1Grs+RKitil6Q79xt9TWVRscmPKdULviM6ub7Wo/5B2p6fMnY5oZ8XCweGs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=p7i0qwuy; arc=none smtp.client-ip=74.125.82.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="p7i0qwuy" Received: by mail-dy1-f202.google.com with SMTP id 5a478bee46e88-2ba8013a9e3so5651090eec.0 for ; Sun, 03 May 2026 14:23:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777843389; x=1778448189; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dHo3S2sC3hPKYGBbJhzMJdbsQdshtIhZZObsDHnBi1U=; b=p7i0qwuy3JnZnNzgmJksfIiMNm/QjtbdUSpH0iDwjg2t4BriTrRXeQKV4js0MkOIRB FNGJaS0pb/JvBs2FSp+t1bQPymGRIQ8NEzi1GSDBhO5tGvfYZFrJT8tnf4KQoFAKtP7S EwOnRG1Nwv27o3hzb180zRmNFqt+Qx+h9qXzDHkvaz8hxvi1BQ5E+pedtySAMTC+K4nu Z+WmqBzhDr0xi9R68zMf95GX6TGBA8eCO+lwKCI9vEhx1M7OiYGesjXPIprauNFE+KYF rNdT817Udg9dmpZBhQ3jbyJqSb9WRYqvPDsQbRphrWmyf4PbZZFojEo9C0PBnKmzZKgs clDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777843389; x=1778448189; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dHo3S2sC3hPKYGBbJhzMJdbsQdshtIhZZObsDHnBi1U=; b=NntbpmEjPi10iYK2Kscr8M+narri9+VzK92QXvS2mliOhHln+r27Gzff/FLzWWyyMw L48WP4NUAEl2vpFn6LUcLcVMpoXumqJiZrQ/O84vbXffvLGr7BLNH92OzANkUFDReFjz nhjdiLziMPSQbdhVvs6xYBQtcdRSiJOHAntHa2XZRvaEjrR0I5n5agb6gVGdijRPYJMp NmVR4Rz9bfLN3TLHrXXd24oCt/xPtJL2MgkLFBBKQ7hSORkQX5E6S+npUZ2H5ijXvLZo aVUHvxoGH4SS3yQgR6J2UNiJxb5cYRdknJkVXfgoK0f4bbhqDtTO0lc/Y2gk8zs5EEnX ZVVA== X-Forwarded-Encrypted: i=1; AFNElJ9QUD52nqgUaqYMEgr6A7JkdL8eJE0m4sXXpFQ+iiR3C36fDXQMAwmk8Kbhijm3qzldi0QZ1PPR1aOnhSY=@vger.kernel.org X-Gm-Message-State: AOJu0YwGywPUJwnYcwWywaCNXj0WTGzQB1QCot/IlmAOHIsOpOHTAMIw 7ZIEHUe7KKkBqxTnsSPk7jfLnWo4bS210wX7o8rc6nSr9a2PZR4y0sx9pH6bBzgaNSkDnWG0k39 SSfdwfUkWDj8QD/dAbKdz0yaR96j0phjgPQ== X-Received: from dybcr21.prod.google.com ([2002:a05:7300:ac95:b0:2e2:2088:dca7]) (user=aniketgattani job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7300:cb17:b0:2ed:e14:7f54 with SMTP id 5a478bee46e88-2efba7a6c9dmr2712951eec.30.1777843389183; Sun, 03 May 2026 14:23:09 -0700 (PDT) Date: Sun, 3 May 2026 21:22:03 +0000 In-Reply-To: <20260503212205.3714217-1-aniketgattani@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260503212205.3714217-1-aniketgattani@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260503212205.3714217-2-aniketgattani@google.com> Subject: [PATCH v3 1/3] sched/membarrier: Use per-CPU mutexes for targeted commands From: Aniket Gattani To: Mathieu Desnoyers , "Paul E . McKenney" Cc: Peter Zijlstra , Ingo Molnar , Ben Segall , Josh Don , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Aniket Gattani Content-Type: text/plain; charset="UTF-8" Currently, the membarrier system call uses a single global mutex (`membarrier_ipi_mutex`) to serialize expedited commands. This causes significant contention on large systems when multiple threads invoke membarrier concurrently, even if they target different CPUs. This contention becomes critical when combined with CFS bandwidth throttling/unthrottling, during which interrupts can be disabled for relatively long periods on target CPUs. If membarrier is waiting for a response from such a CPU, it holds the global mutex, blocking all other membarrier calls on the system. This cascade effect can lead to hard lockups when thousands of threads stall waiting for the mutex. Optimize `MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ` when a specific CPU is targeted by introducing per-CPU mutexes. Broadcast commands and commands without a specific CPU target continue to use the global mutex. This prevents the cascade lockup scenario. As measured by the stress test introduced in the subsequent patch, on an AMD Turin machine with 384 CPUs (2 NUMA nodes with SMT=2), this optimization yields 200x more throughput. Changes in v3: - Fixed the code path when `cpu_id < 0` in membarrier_private_expedited as reported by Peter. Link: https://lore.kernel.org/lkml/20260428124823.GY3102624@noisy.programming.kicks-ass.net/ Signed-off-by: Aniket Gattani --- kernel/sched/membarrier.c | 77 ++++++++++++++++++++++----------------- 1 file changed, 44 insertions(+), 33 deletions(-) diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c index 623445603725..3d88e900a17f 100644 --- a/kernel/sched/membarrier.c +++ b/kernel/sched/membarrier.c @@ -164,8 +164,26 @@ | MEMBARRIER_PRIVATE_EXPEDITED_RSEQ_BITMASK \ | MEMBARRIER_CMD_GET_REGISTRATIONS) +/* + * Scoped guard for memory barriers on entry and exit. + * Matches memory barriers before & after rq->curr modification in scheduler. + */ +DEFINE_LOCK_GUARD_0(mb, smp_mb(), smp_mb()) static DEFINE_MUTEX(membarrier_ipi_mutex); +static DEFINE_PER_CPU(struct mutex, membarrier_cpu_mutexes); + #define SERIALIZE_IPI() guard(mutex)(&membarrier_ipi_mutex) +#define SERIALIZE_IPI_CPU(cpu_id) guard(mutex)(&per_cpu(membarrier_cpu_mutexes, cpu_id)) + +static int __init membarrier_init(void) +{ + int i; + + for_each_possible_cpu(i) + mutex_init(&per_cpu(membarrier_cpu_mutexes, i)); + return 0; +} +core_initcall(membarrier_init); static void ipi_mb(void *info) { @@ -315,7 +333,6 @@ static int membarrier_global_expedited(void) static int membarrier_private_expedited(int flags, int cpu_id) { - cpumask_var_t tmpmask; struct mm_struct *mm = current->mm; smp_call_func_t ipi_func = ipi_mb; @@ -352,30 +369,45 @@ static int membarrier_private_expedited(int flags, int cpu_id) * On RISC-V, this barrier pairing is also needed for the * SYNC_CORE command when switching between processes, cf. * the inline comments in membarrier_arch_switch_mm(). + * + * Memory barrier on the caller thread _after_ we finished + * waiting for the last IPI. Matches memory barriers before + * rq->curr modification in scheduler. */ - smp_mb(); /* system call entry is not a mb. */ - - if (cpu_id < 0 && !zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) - return -ENOMEM; - - SERIALIZE_IPI(); - cpus_read_lock(); - + guard(mb)(); if (cpu_id >= 0) { + if (cpu_id >= nr_cpu_ids || !cpu_possible(cpu_id)) + return 0; + + SERIALIZE_IPI_CPU(cpu_id); + guard(cpus_read_lock)(); struct task_struct *p; - if (cpu_id >= nr_cpu_ids || !cpu_online(cpu_id)) - goto out; + if (!cpu_online(cpu_id)) + return 0; + rcu_read_lock(); p = rcu_dereference(cpu_rq(cpu_id)->curr); if (!p || p->mm != mm) { rcu_read_unlock(); - goto out; + return 0; } rcu_read_unlock(); + /* + * smp_call_function_single() will call ipi_func() if cpu_id + * is the calling CPU. + */ + smp_call_function_single(cpu_id, ipi_func, NULL, 1); } else { + cpumask_var_t __free(free_cpumask_var) tmpmask = CPUMASK_VAR_NULL; int cpu; + if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) + return -ENOMEM; + + SERIALIZE_IPI(); + guard(cpus_read_lock)(); + rcu_read_lock(); for_each_online_cpu(cpu) { struct task_struct *p; @@ -385,15 +417,6 @@ static int membarrier_private_expedited(int flags, int cpu_id) __cpumask_set_cpu(cpu, tmpmask); } rcu_read_unlock(); - } - - if (cpu_id >= 0) { - /* - * smp_call_function_single() will call ipi_func() if cpu_id - * is the calling CPU. - */ - smp_call_function_single(cpu_id, ipi_func, NULL, 1); - } else { /* * For regular membarrier, we can save a few cycles by * skipping the current cpu -- we're about to do smp_mb() @@ -420,18 +443,6 @@ static int membarrier_private_expedited(int flags, int cpu_id) } } -out: - if (cpu_id < 0) - free_cpumask_var(tmpmask); - cpus_read_unlock(); - - /* - * Memory barrier on the caller thread _after_ we finished - * waiting for the last IPI. Matches memory barriers before - * rq->curr modification in scheduler. - */ - smp_mb(); /* exit from system call is not a mb */ - return 0; } -- 2.54.0.545.g6539524ca2-goog