From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EE634FD791 for ; Fri, 2 Oct 2026 17:09:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790960948; cv=none; b=tWr9WmXa/qSOSMRssuXCKlTfXC9pRWyIaCaOFDlC4unEtOGllQKJZ2aXaH5dQCNhpdmQDxGVRAFapyTp/eVGA/bTjxW4/0Op1TL0e94K0R9h2EByV/i9+c6Ms5OpgvwMI+vjMmL8pyz0RjAtVfLSsN0epTWdYmntOvhsT9QkO/I= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790960948; c=relaxed/simple; bh=P6kLUfqJKeds6mBcoX+09pwoLlGMX2WhGD61+udcnHM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=sDsiFOFkSdKAdit1W8P3DO9StCCrA+tGPLaxcKN6/vJB/CnlzX0NWCcorDRpR2b+VCcDTdP16dfc580cV19iExx61aSNjSTYQ7RVNhSxXCr3g2M3PftbKS9TWKqmpGVemtl+oatqAVyo4+1idqoo7bUIr8ImJnGBfVA91eQ5ccM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=arLAY77L; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="arLAY77L" Received: by mail-pj2-f13.google.com with SMTP id 98e67ed59e1d1-396ccc02279so128288a91.1 for ; Fri, 02 Oct 2026 10:09:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790960943; x=1791565743; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=Fl0M9kkOu81fD6UmnHPgP6gAcI7k1L5BHVF53b6ArLM=; b=arLAY77Lf5hP9ARIas2zgFclVUwfWR96quc5vpfBfKQ3ipa+ra9xPuTk+LJSPq1Szs gHVX+/ipK3emNX1fGPme1m2O39rxYIpZ4cI0IysNlfzvUZWlR59z+MTl9IVxIrLlnVIB OYv/ZETIGDsvFCY+w+8IleuES5EwIv87nn4q5Jm1rRa54TskejcnKAL9tIxaEsjPc6Tp 9RRwcnw85aDqVddaCPdnAvRA7tY/2yWDEx5CUibU6GjuWF/UPHkM5KCGFNxF8/rloVlA MJ9EslIJr2irn8RhrUf+30MLGE8pHxm02kmvvD2s4kswVPiOnE4N/yAEk5Ad3iYV0307 zs1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790960943; x=1791565743; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Fl0M9kkOu81fD6UmnHPgP6gAcI7k1L5BHVF53b6ArLM=; b=mbrM6VCk8H1OhDtRYOzTqu6D6F/SkJcitIZyJLx61mCyQzqbZTt+YmN0PShOEFbt8Q 7urEdHvBuwTq392kEkj992/z2P4GMIweon8eVNGapT7OKKzfFgjx9MwDpKqNVHntwDCk mpAtlGzHN73zD4R10D+vzNucDY4QrSzK3uYV8jVFHXkMQpmXqDXvHBzwOamnZuziKcFd eCNywDKbhUqN9cxPmKinh2dENwgw1XPyM+mw1AogAzUJB8DC+0DccUxaCejEGaEmJ9M9 kzVBleyM/A9qSw0XuZmLSewcCd3I/RIhaKgXa6OEzvSaSDfKRlyt4wzXkXSjb72cTf5H DR/A== X-Forwarded-Encrypted: i=1; AKwUvBwZOWPLKNM1MFyuAt+IyqgM2Hi59M/Qc/JWZRbs6WXXD8iJTj0mMRs7b7YogLSztgjci4LdR9zUoVhYj08=@vger.kernel.org X-Gm-Message-State: AFq9FYJNawao1Yli9O12J7hJP4IHeFq5Ffx8R6mUMgSqfIiESyyoOu67 ZJW4Dxw9QQJfKOnu3FNlsl+w1MMnewRrgNRHX3v2rfiPW8SjalisTbse X-Gm-Gg: AYBFou06BuMlo64ArnmRM5PnX77z/GooH6RtFy2H5fVI6x6ahIJRhcpCDKfoRg02Lvx Hl5LqWZPGU5J3qOcSKSZo0gd/PLLnlXiogXVrmsYwhQKSYJp5QBe0o6MiX2ehPkXRX1rYG4z6dY XoU90lS5uiN2s8HFZ11fR8YycsD8alfm1VNbmSMrEu2HBE+3PIxtg5i8ADFe96v/F+Rnbh5Nths v+s60NdgsU0M9RGVgw42n6ywhry0A//S4aMOPoToAXq0DshxfuImBW9R5UFfurwnidAORCtLYpB koJyPQKtAKlh6fcThA7X75OMIURQx4PY8UaT/oaC9HGchQtxY1JvW6CYQ0bUpx/790vcBDyFzaC i+rvtIr5bl0+CGH7MkiMbe1UYbnMd4TfmxIHVpOTpFhzQiDQa076d0gxphBX2qGsDbAOTNYO87z TLvvNH12XW1hF+amvx7cFBs5Aac1oqhmKltaYr3PcR1w2IcXe5uHKzyPMlLlZrVcN4UQqtJ2O1d RYJkJuMUvvMAckwIg== X-Received: by 2002:a17:90b:1c86:b0:3a4:d5c0:f826 with SMTP id 98e67ed59e1d1-3a6ce9b0fe4mr3264396a91.65.1790960942759; Fri, 02 Oct 2026 10:09:02 -0700 (PDT) Received: from kernel.tail6741c6.ts.net ([185.220.238.43]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3a7024d0f5fsm1597719a91.0.2026.10.02.10.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Oct 2026 10:09:02 -0700 (PDT) From: Kunwu Chan To: paulmck@kernel.org, corbet@lwn.net, mingo@redhat.com, frederic@kernel.org, neeraj.upadhyay@kernel.org, josh@joshtriplett.org, urezki@gmail.com, dave@stgolabs.net, lianux.mm@gmail.com Cc: stern@rowland.harvard.edu, parri.andrea@gmail.com, will@kernel.org, peterz@infradead.org, boqun@kernel.org, npiggin@gmail.com, dhowells@redhat.com, j.alglave@ucl.ac.uk, luc.maranget@inria.fr, akiyks@gmail.com, dlustig@nvidia.com, joelagnelf@nvidia.com, skhan@linuxfoundation.org, rdunlap@infradead.org, longman@redhat.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang@linux.dev, kunwu.chan@gmail.com, brads@mainlining.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, lkmm@lists.linux.dev, linux-doc@vger.kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH RFC v2 00/15] hazptr: batch synchronize operations through a shared scan Date: Sat, 3 Oct 2026 01:08:32 +0800 Message-ID: <20261002170847.3653663-1-kunwu.chan@gmail.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This series extends Paul McKenney's v3 hazptr implementation [1] with a shared scan path for concurrent hazptr_synchronize() callers, and adapts the lockdep dynamic-key hashlist use case from Boqun Feng's 2025 shazptr series [2] to the current hazptr API. The series also adds rcuscale support and torture coverage for the hazptr implementation. The lockdep conversion replaces the expedited RCU wait in lockdep_unregister_key() with hazptr_synchronize(). This limits the wait to hazard pointers protecting the target hash bucket instead of waiting for a system-wide expedited RCU grace period. [1] https://lore.kernel.org/all/20260919000056.3132131-26-paulmck@kernel.org/ [2] https://lore.kernel.org/lkml/20250625031101.12555-1-boqun.feng@gmail.com/ Performance data ================ All measurements are on an ARM64 KVM guest with HAZPTR_SCALE_NR_OBJS=8 and round-robin updater selection; unless noted otherwise, nreaders=0. Scale tests run with PROVE_LOCKING=n. rcuscale synchronize latency (96 CPUs, nw=1): nreaders=0 nreaders=1 nreaders=4 nreaders=96 scale avg avg avg avg -------------------------------------------------------------------- hazptr 46 us 47 us 102 us 106 ms RCU 8.3 ms 8.5 ms 16.6 ms 54.4 ms SRCU 8.2 ms 8.0 ms 11.7 ms 11.8 ms Hazptr single-writer latency remains below 110 us with up to four readers on this 96-CPU guest, but rises to 106 ms when all 96 CPUs hold hazard pointers. Under 16 concurrent synchronize callers (nw=16), per-writer latency at four CPU counts: hazptr nw=16 RCU nw=16 SRCU nw=16 CPUs avg avg avg ----------------------------------------------------------- 24 8.0 ms 13.6 ms 9.2 ms 96 8.0 ms 13.3 ms 8.6 ms 128 7.9 ms 13.0 ms 11.7 ms 256 8.0 ms 20.8 ms 9.6 ms Hazptr remains around 8.0 ms across these CPU counts, with the scan-kthread retry interval contributing to the latency. RCU rises to ~21 ms at 256 CPUs while SRCU stays around 9-12 ms. Reader-side overhead (refscale, 96 readers on the 96-CPU guest, 3 runs each): hazptr 25.7 ns/op RCU 96.7 ns/op SRCU 134.6 ns/op lockdep workload -- tc qdisc mq x100, ARM64 KVM, PROVE_LOCKING=y, 96 background hazptr readers, function-call IPIs per 100 ops: CPUs hazptr exp RCU reduction ------------------------------------------------------------ 24 196 207 5% 96 189 265 29% 128 193 266 27% 256 222 384 42% Hazptr issues fewer function-call IPIs at each CPU count, with the difference reaching 42% at 256 CPUs. The hazptr.sh test suite passes, including the lockdep scenarios and the 8-to-256 CPU sweep, with no lockdep warnings, deadlocks, or crashes. Additional x86 server testing with Lian Wang is planned(maybe after LPC). Changes since RFC/WIP ===================== - RFC/WIP: https://lore.kernel.org/all/20260922070950.4173245-1-kunwu.chan@gmail.com/ - Split the original 4-patch RFC/WIP into smaller commits covering shared scanning, correctness, API support, lockdep, scaling, and torture testing. - Incorporated Boqun Feng's review feedback: use a Bloom filter to avoid per-waiter allocation, add scoped_guard() support, and add a debug option to force the hazptr acquire slow path. - Fixed scan ordering around backup-slot promotion by scanning all per-CPU slots before the overflow lists, with a separate overflow-list phase. - Simplified the scan cycle to flip first and drain only the old wildcard generation, with herd7-verified LKMM tests for both the in-flight and resolved publication cases. - Extended rcuscale and hazptrtorture coverage, added a selftest script for the torture configurations, and fixed the hazptr_release() kernel-doc. Kunwu Chan (15): hazptr: add shared scan kthread hazptr: use Bloom filter for shared scan waiters hazptr: scan all per-CPU slots before overflow lists hazptr: add scoped_guard() support hazptr: add debug option to force the acquire slow path hazptr: elide redundant first drain pass Documentation/litmus-tests: add hazptr wildcard-flip escape test locking/lockdep: use hazptr to wait for dynamic key lookups rcuscale: add hazptr scale type hazptr: fix kernel-doc of hazptr_release() Documentation/litmus-tests: add hazptr acquire-before-scan test hazptrtorture: add slowpath and lockdep scenarios hazptrtorture: add READERS4 and READERS0 torture configs hazptrtorture: add 128- and 256-CPU configs selftests/rcutorture: add hazptr torture test script Documentation/litmus-tests/README | 13 + .../hazptr/hazptr-acquire-before-scan.litmus | 45 +++ .../hazptr/hazptr-wildcard-flip-escape.litmus | 46 +++ include/linux/hazptr.h | 56 ++- kernel/hazptr.c | 336 +++++++++++++++++- kernel/locking/lockdep.c | 25 +- kernel/rcu/Kconfig.debug | 10 + kernel/rcu/hazptrtorture.c | 57 ++- kernel/rcu/rcuscale.c | 70 +++- .../selftests/rcutorture/bin/hazptr.sh | 146 ++++++++ .../rcutorture/configs/hazptr/CFLIST | 6 + .../rcutorture/configs/hazptr/CPU128 | 16 + .../rcutorture/configs/hazptr/CPU128.boot | 1 + .../rcutorture/configs/hazptr/CPU256 | 16 + .../rcutorture/configs/hazptr/CPU256.boot | 1 + .../rcutorture/configs/hazptr/LOCKDEP | 17 + .../rcutorture/configs/hazptr/LOCKDEP.boot | 1 + .../rcutorture/configs/hazptr/READERS0 | 16 + .../rcutorture/configs/hazptr/READERS0.boot | 2 + .../rcutorture/configs/hazptr/READERS4 | 16 + .../rcutorture/configs/hazptr/READERS4.boot | 2 + .../rcutorture/configs/hazptr/SLOWPATH | 16 + 22 files changed, 881 insertions(+), 33 deletions(-) create mode 100644 Documentation/litmus-tests/hazptr/hazptr-acquire-before-scan.litmus create mode 100644 Documentation/litmus-tests/hazptr/hazptr-wildcard-flip-escape.litmus create mode 100755 tools/testing/selftests/rcutorture/bin/hazptr.sh create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU128 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU128.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU256 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU256.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/LOCKDEP create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/LOCKDEP.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS0 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS0.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS4 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS4.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/SLOWPATH -- 2.43.0