From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.169 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793811331; b=rsfPDZ6uikFChK4PUZp6gsV10LruWkcn00APjUIQIj0OMBWEG3OvHYQCviuoC+4F5UZE FYLxSo2on6j6zvPjB+bXPPiSvoUqJyz6QE1RVrxCIGfGGpProoX8lymp4axsPA6/bAXfe IXgzAJymdZsD4FF9MKlSuenTRgHSyTzumElyKkj1CYD8P2yMb8usD8bQJtoY2KIHP7cye z8cbAfrzMCqkWBZcK4JLWrW38guhkrhpLKMLk/I6UqLYvvG48oZaMqvqZ4L9dVRX6Gr+t k4C2KWPS57QmRA4+fD1ZgPnjlBNTRwBUjddW2ETkMOaoiZ4pEdXdVW9AXUY6GLssa0g== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793811331; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; bh=svAmYvSZ5GutHDmHcCUE5c3ghjL8LeEt2/DysvEiXC8=; b=Iyl0krk01PKW7Ms9iZCOrAyT3zugYLlktj/gXnmaGgbDiTc03gaee18vnCB6UBdrnL3R npmNGrDp2B0681T1BAN5imujlU0/XWKEzLG65uJGLgzgJzxjX9UUWxtpmLqCJG8VYdUSw aHvhc1oS69YMMfVXa6AiCcx0Fq8MPTHbY4fJUhAeJ5kqgRgROFPz04QhN5pqCIKV8EW1j A8L+dtJ6y+v77pkBe4jakAV9ff73a+mr/wGoG9mWd3+lqArhFaCnZ50MUKq74tdb7S/nU MI1UGdYvpg9f++9WDRN10Mz3TuGn6KslInge2omeP1MALv5Y9WNhfyz6sAjRYKHdQmw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=q0uqz35L; arc=none smtp.remote-ip=74.125.82.169 Received: from mail-dy1-f169.google.com (mail-dy1-f169.google.com [74.125.82.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9DD7606A32E for ; Wed, 4 Nov 2026 16:55:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org D9DD7606A32E Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=q0uqz35L Received: by mail-dy1-f169.google.com with SMTP id 5a478bee46e88-3514b90bb89so1742067eec.1 for ; Wed, 04 Nov 2026 08:55:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1793811330; x=1794416130; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=svAmYvSZ5GutHDmHcCUE5c3ghjL8LeEt2/DysvEiXC8=; b=q0uqz35LO15DM4psgbOfG0V+FsAvkp5rl/pScZ0E/Zq8+SfrQji7h6wIFyIc7EcHTU Y+lSGlLv/WcQ80KJXV6uMP1oQUKWLoRAUEQvnRpkkrEVTV0eIWFLJMCj+COsZSOKAhV9 Ph2AUvPaNZdKZwK7RhJvSBeA8edXi4plGh+ULoiBC58GpI+FrF7gPPNSwswEhmvMcjjs mP07hEwy3jZ/X/RXk2MggMcaps8ZB/l9D5TJclAtn5NFRsuYSh3mP8fOuT+JQ4UWeBvJ rKLlVf8objJ9AdWsPqUmuJSpI+5wM5qVRy+/FCa3HnqVQgdRSPMcwxd70SXcAbdcwATT B4Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793811330; x=1794416130; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=svAmYvSZ5GutHDmHcCUE5c3ghjL8LeEt2/DysvEiXC8=; b=j6zuNUrW64j8Xzk3JE3/iDvTitJmW7PdNYAKf5h7fiKXkbCzl2w9diSkEqdFFODUpb sEt59CP6SU2e/secWCyJ/tdEWDhlrUzqgo3uuADYf95pUqV7QnplbJ3Cyv+9MiSzX/Yx GCy83MqU9FR6i4U6lFP58KCWjVMVc/lZiNMe+c7Tqa46kdtSCW5v0vtJqB0cNFYfAZLR VFnMY0EFsI+7YWp+nma2JnNdEbpjmyfkaBDC7oKfBDNgvzv9edyStDog5b01F86+4VBB 1vIwmPmB1e161N0eBJddq2lz5ITMXB0b3ZSWveJZSuSVQohvAOtCAcCCmy28t70hsmAS sShQ== X-Forwarded-Encrypted: i=1; AKwUvBxVdoidas5DhJL+SxkbwS37v2yuKWMG4k9YklhZ0nUdMTcX4CeSZmRDtepv3N+Ql3GBAN0bK1cfUJe1Yw8=@vger.kernel.org X-Gm-Message-State: APuU4Z8eT4+Wym5oDkeR9EQ01YMx0jkerqcBrHLj9A5CSAywV/IkdM+q z6CrxOYE37a3zkytWQdPSOMv6tkQafYfGrZUIh+UJfIBWlFiaO7Fdc8n X-Gm-Gg: AYBFou2xvYWqtyldFuaPFtch9vaBWaW5yph0a56xDMUNRtdT/ELaiSi8pRIma6Z+A7v Pdjo/Zubpzzb+C1JchpbT/VY5P5GXRryQp4UbzGB7lpASunpHLiCOFoT0lciad5WYOw7ikWs7uL iqm3aQ+wJJoLFBfiaU+1PWSZixU8xPXRfnmMV69D4Jmcd9rRqBm+H4ELGbniN8GKeKAmMpGE4CG szEKXSxuYmeSZhgB2uOM1e0rSTf5Z9CaUpJPXBpQ8j4XI2cFTRUGN2Uo7+/BSkNgfpE7FIKO+rT RxcyYUTpuYbX3EKq8Tv3Tq8FW3RatE4k4GS9Q8GXsCDRMDhVHRXEYZ1PjEYiGRRXOT/A42tCeej C/QXjvHdr9wG3wxNO0dbuJe/9uf82yRLMNEi+sL/LzlQs4L0zW3okrR1yDGvDfPOmhd1s3tclZS +61btvR3FtOzxdVY+PmbDv9wZW+hgjavN2ivtd9yJcoSkDC5irX+7/YI5FtgisDBFiFYYyPAzSY CGXh39dt96qRJjVo5pNir6hbJQkN735oYjfGOCZrebU2C0WKku5nzUI4ni4tyDrU3ieFqlM2Eeq baYyenAXZMnsP+LZVs7J6nJHyLacWgfcGhyMQIk= X-Received: by 2002:a05:7301:4242:b0:358:dfed:d0fe with SMTP id 5a478bee46e88-36a42219d4fmr23626146eec.5.1793811329298; Wed, 04 Nov 2026 08:55:29 -0800 (PST) Received: from kernel.. ([216.195.201.23]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-36ca38d46fcsm9019567eec.13.2026.11.04.08.55.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Nov 2026 08:55:28 -0800 (PST) From: Kunwu Chan To: paulmck@kernel.org, boqun@kernel.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, corbet@lwn.net, mingo@redhat.com, frederic@kernel.org, neeraj.upadhyay@kernel.org, josh@joshtriplett.org, urezki@gmail.com, dave@stgolabs.net Cc: stern@rowland.harvard.edu, parri.andrea@gmail.com, will@kernel.org, dhowells@redhat.com, j.alglave@ucl.ac.uk, luc.maranget@inria.fr, akiyks@gmail.com, dlustig@nvidia.com, joelagnelf@nvidia.com, skhan@linuxfoundation.org, rdunlap@infradead.org, longman@redhat.com, rostedt@goodmis.org, qiang.zhang@linux.dev, kunwu.chan@gmail.com, brads@mainlining.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, lkmm@lists.linux.dev, linux-doc@vger.kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH 01/28] hazptr: Add shared scan kthread Date: Thu, 5 Nov 2026 00:54:19 +0800 Message-ID: <20261104165504.2430480-2-kunwu.chan@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261104165504.2430480-1-kunwu.chan@gmail.com> References: <20261104165504.2430480-1-kunwu.chan@gmail.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Batch concurrent hazptr_synchronize() callers into a single kthread scan. Waiters sleep on completions instead of busy-waiting. The kthread processes queued waiters sequentially: for each waiter, it scans per-CPU slots and both overflow-list phases until the address is released. This is the same scan the direct fallback performs, delegated to a kthread. Waiters are embedded in the caller's stack frame, so no dynamic allocation is needed in the synchronize path. Fall back to the direct scan if the kthread is unavailable. Signed-off-by: Kunwu Chan Acked-by: Bradley Morgan --- kernel/hazptr.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/kernel/hazptr.c b/kernel/hazptr.c index b06164aac3a7..013c24021c43 100644 --- a/kernel/hazptr.c +++ b/kernel/hazptr.c @@ -12,6 +12,10 @@ #include #include #include +#include +#include +#include +#include #define HAZPTR_WILDCARD ((void *) 1UL) @@ -234,6 +238,145 @@ void hazptr_scan_overflow_list_period(void *addr, unsigned int scan_idx) } } +/* Helper: scan both overflow-list phases. */ +static void hazptr_scan_overflow_list_all(void *addr) +{ + unsigned int scan_idx = READ_ONCE(hazptr_overflow_list_phase); + int cpu; + + for_each_possible_cpu(cpu) { + struct hazptr_overflow_list_flip *overflow_list_flip = per_cpu_ptr(&percpu_overflow_list_flip, cpu); + + hazptr_synchronize_overflow_list(&overflow_list_flip->array[scan_idx], addr); + hazptr_synchronize_overflow_list(&overflow_list_flip->array[flip_list_phase(scan_idx)], addr); + } +} + +/* + * Shared scan: batch concurrent hazptr_synchronize() callers into a + * single kthread-driven scan, avoiding redundant per-CPU and + * overflow-list walks per caller. Waiters sleep on completions + * instead of busy-waiting. + */ + +struct hazptr_waiter { + struct list_head node; + void *addr; + struct completion done; +}; + +struct hazptr_scan_state { + struct task_struct *kthread; + struct swait_queue_head wq; + bool wakeup; + struct mutex lock; + struct list_head pending; + struct list_head scanning; /* kthread only */ +}; +static struct hazptr_scan_state hazptr_scan; + +/* + * Queue @addr for the shared scan. The waiter lives on the caller's + * stack, so no allocation is needed. + */ +static void hazptr_synchronize_queued(void *addr) +{ + struct hazptr_waiter waiter = { + .addr = addr, + }; + + init_completion(&waiter.done); + INIT_LIST_HEAD(&waiter.node); + + mutex_lock(&hazptr_scan.lock); + list_add_tail(&waiter.node, &hazptr_scan.pending); + if (!READ_ONCE(hazptr_scan.wakeup)) { + WRITE_ONCE(hazptr_scan.wakeup, true); + swake_up_one(&hazptr_scan.wq); + } + mutex_unlock(&hazptr_scan.lock); + + wait_for_completion(&waiter.done); +} + +static void hazptr_scan_do_cycle(void) +{ + struct hazptr_waiter *w, *n; + LIST_HEAD(done); + + mutex_lock(&hazptr_phase_lock); + mutex_lock(&hazptr_scan.lock); + list_splice_tail_init(&hazptr_scan.pending, &hazptr_scan.scanning); + mutex_unlock(&hazptr_scan.lock); + + if (list_empty(&hazptr_scan.scanning)) { + mutex_unlock(&hazptr_phase_lock); + return; + } + + /* + * Process each waiter: scan per-CPU slots and overflow lists + * until the waiter's address is released. This is the same + * scan the direct fallback performs, delegated to a kthread so + * callers sleep instead of busy-waiting. + */ + list_for_each_entry_safe(w, n, &hazptr_scan.scanning, node) { + hazptr_scan_cpu_slots(w->addr); + hazptr_scan_overflow_list_all(w->addr); + list_move(&w->node, &done); + } + + mutex_unlock(&hazptr_phase_lock); + + list_for_each_entry_safe(w, n, &done, node) { + list_del_init(&w->node); + complete(&w->done); + } +} + +static int hazptr_scan_kthread(void *unused) +{ + for (;;) { + bool idle; + + swait_event_idle_exclusive(hazptr_scan.wq, + READ_ONCE(hazptr_scan.wakeup)); + hazptr_scan_do_cycle(); + mutex_lock(&hazptr_scan.lock); + idle = list_empty(&hazptr_scan.pending) && + list_empty(&hazptr_scan.scanning); + if (idle) + WRITE_ONCE(hazptr_scan.wakeup, false); + mutex_unlock(&hazptr_scan.lock); + if (idle) + continue; + schedule_timeout_idle(1); + } + return 0; +} + +/* + * Initialize the scan kthread. Failure falls back to the + * direct scan. + */ +static int __init hazptr_scan_init(void) +{ + struct task_struct *t; + + init_swait_queue_head(&hazptr_scan.wq); + mutex_init(&hazptr_scan.lock); + INIT_LIST_HEAD(&hazptr_scan.pending); + INIT_LIST_HEAD(&hazptr_scan.scanning); + + t = kthread_run(hazptr_scan_kthread, NULL, "hazptr_scan"); + if (!IS_ERR(t)) + smp_store_release(&hazptr_scan.kthread, t); + else + pr_warn("hazptr: scan kthread failed, using direct scan\n"); + return 0; +} +core_initcall(hazptr_scan_init); + /* * hazptr_synchronize: Wait until @addr is released from all slots. * @@ -261,6 +404,12 @@ void hazptr_synchronize(void *addr) /* Memory ordering: Store A before Load B. */ smp_mb(); + /* Pairs with smp_store_release in hazptr_scan_init(). */ + if (smp_load_acquire(&hazptr_scan.kthread)) { + hazptr_synchronize_queued(addr); + return; + } + guard(mutex)(&hazptr_phase_lock); /* Scan per-CPU slots. */ -- 2.43.0