From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53F44315793 for ; Fri, 3 Apr 2026 14:24:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775226250; cv=none; b=RJjXAli54eAYcK8UUwYvH1vEpTBKtwXr89yKi6V/xSg9N0RXpBXRaZxARb1567seSBm4zJKxHdfbMNXWvVaKtaOaVK3dQrkP5c7vdIrgA+UHP5F+X8WgYo6ZwABxEnzKRpKkYY8fhslbeCa+ALH6fn95MHQ/MiFgwFAEqo03qZY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775226250; c=relaxed/simple; bh=pJfJdjXsFxp+vHlMX8+PqPqmT5yKVyGTHvTGSx9lyfM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=cOY7xG4FZrylhmDFFZNwfLoLfW3mdcjiR7iVkeSGTiOlfuydF4hhYmORU4nDm9pp1t76QKbgKh4rsgc6qdJ1ooH7/jpvfoR/lm1c88uAXCB2ejdm95NXJZrT9UbgcA8KiEtrnZaiijE9oX2nuQE/tq/7w/QXkC95Dd4n0Bre3pU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OxIXoEx9; arc=none smtp.client-ip=209.85.210.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OxIXoEx9" Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-7d9bba96f7dso985853a34.1 for ; Fri, 03 Apr 2026 07:24:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775226247; x=1775831047; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=G30dQsKqYIosacZzs/J3gVKCWVZXMr1CrYV8Y1uNaFI=; b=OxIXoEx9bh9snhAfFdswROawn1tt5VLFUXkWjouhnE2ggPYy8XFA4EiToNkyflOjtO hpQxVX8SQd1+pcoM9KWgn458ESLZCZRLPLyyZQgRWuDr95TieSe6di8zo1PMTa65wTX9 9MRtccla9AYScZJh/WU5ilq9eVztMZ85mzXlBVdw+x4wAIm4ERmRY6I2Gf4XkkJJL7KJ HB8JAtJf63TblOBpmhL7gcJsS1jOnizhsJ9TSrU9feYRDA6XDHnR9Ht7BW32F3dT2s+/ GyYTMA87BK6AVkpt1+4E8kPF0eXsUlsBWQh4kchtXBorqhqMyeMsf4+5edtL5yjaFOF1 TEFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775226247; x=1775831047; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=G30dQsKqYIosacZzs/J3gVKCWVZXMr1CrYV8Y1uNaFI=; b=qm3L+D0UnVtKVNPFV6zYFWe26gAjcE1LfB4TPtwIer1g6EI6uHwIqmBalECUtS49VZ OL/9mB9M9tLtIWo1cp3CcoJepqW6L18hSxkz0Yv2nqjHeS2alHplvxS9EA90eCaXuoYs TZdFCx64PMYtQ1uNiBj4wYvPBsF7V42aWM+/mqMtTaBD85qKl0tAK7yHUO0WVICGfLjv CRnmhtJU2nRZoraKfkAzX8PHwRjZhO8peGY/oohzF4U/nRmrzFBxurGUcT5pHOc1Lkea NJhzrYjk825qCcjHx6oBiSzhwEYSCi0tOsrrrJXJIrWbZWtMFsZbFv9kccuIXkLWNITS 2jug== X-Forwarded-Encrypted: i=1; AJvYcCUoJf7eXkG0X4gpjtaI8yy34h2ZTAqpnJKE2kxZe7Wk+39+mqhlXJzyEIeZqq11bolOhM5DN22cTiYfrDc=@vger.kernel.org X-Gm-Message-State: AOJu0YxI7ExEQfiFraVTD3Z8pNFR2qIpVp36A+WVhKgbu8nJ/K9LXshs 69I+vtd1J7Ci5Nw4jzX5BBIx7jeCY9lKFXMvl/zYoR32X+bxTPsiw5vK X-Gm-Gg: ATEYQzwJgQeqYoJW2NIsGEtdFZv6du0msQRasPsGO/iAU6LA1yO1UsGQTHcEWF9xEq+ FUt+q2CYx1fyBPkTme15kvPCCriYU4SYE6xTE1xhzsc47xzyR2rrZFxd2YYv0jBxgGiHeubnv7t dYFexhVnym6/4f/KD2nbigLAxnHQhbXkBfsAAOqYBnO0PZcE9PKgxNwFFywYirWQZPsz/VFaD05 JyL7oR+4y5TtkODD5c6a11gmp5dNt3FE9HAZa18/GLNehf4aefwA1sb1RqAg+k0OgEsIB1bklWT l+UImoPSdGyDNIAQgdKuvEJtW7XnGp8gExn8iMXxs05D+H+vUF2VA3gxjdEMqcn2JDo+p9VO1Ko xy6i2m1zRhaiYsCSqjYzqua7zF6FKxJGIN8Z4U8sykYo2x2aGBSul26zaVt8rLpd+ohJsGWflo2 TY3zItEriu8YK42UKieTWUi/eHSvnn2xwzufVl26Js9v1liZSW4ocUrngFkuMFXxkDdGf6alkpU gs= X-Received: by 2002:a05:6830:448e:b0:7d7:cc0f:4680 with SMTP id 46e09a7af769-7dbb6f6dc7cmr1751457a34.9.1775226247091; Fri, 03 Apr 2026 07:24:07 -0700 (PDT) Received: from frodo.raven-morpho.ts.net (c-98-38-17-99.hsd1.co.comcast.net. [98.38.17.99]) by smtp.googlemail.com with ESMTPSA id 46e09a7af769-7dbb7c3a0c8sm1806671a34.12.2026.04.03.07.24.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Apr 2026 07:24:06 -0700 (PDT) From: Jim Cromie To: peterz@infradead.org, gregkh@linuxfoundation.org Cc: jpoimboe@kernel.org, jbaron@akamai.com, aliceryhl@google.com, rostedt@goodmis.org, ardb@kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, Jim Cromie Subject: [PATCH 0/5] jump_label: mitigate dynamic-debug induced IPI storms Date: Fri, 3 Apr 2026 08:23:56 -0600 Message-ID: <20260403142401.1387033-1-jim.cromie@gmail.com> X-Mailer: git-send-email 2.53.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This series adds batched toggling of jump_labels to address IPI storms triggered by bulk toggles of dynamic-debug pr_debug()s. patches 1-3 are for submission. 4-5 are for demonstration only. The Scaling Problem: In testing on 7.1-rc6 guests (virtme-ng) hosted on a stock Fedora kernel, I measured the cost of toggling ~1,900 dyndbg callsites. The current one-at-a-time patching approach creates a massive synchronization bottleneck that scales linearly with both key count and core count: Baseline IPIs (Total CAL Interrupts) Successive query pairs (+p ; -p ;) toggling ~3,792 keys per pair. Pairs | -p 2 | -p 3 | -p 4 | -p 5 | Notes ----------------------------------------------------------- 1 | 10,738 | 21,468 | 32,148 | 42,837 | ~11.3 IPI/key/CPU 2 | 21,269 | 42,976 | 64,416 | 85,880 | Linear scaling (K*M) 3 | 32,125 | 64,434 | 96,604 | 128,793 | 4 | 42,831 | 85,886 | 128,776 | 171,697 | 5 | 53,540 | 107,353 | 160,985 | 214,636 | 6 | 64,174 | 128,800 | 193,159 | 257,562 | > 250k IPI storm On a 5-VCPU system, a sequence of 6 query pairs triggers over a quarter-million interrupts. This overhead is a direct result of invoking the full patching machinery for every individual key toggle. The Solution: While x86 already has a 256-entry queue for jump_label, 2 issues currently block its use: 1. dyndbg uses the synchronous API; an IPI is issued for every static-key. patch-1 provides a fix for this with a _queued api extension. patch-3 uses the new api in dyndbg. 2. The queue flushes if current_patch_addr < previous_patch_addr. dyndbg has consecutive descriptors, but not consecutive patch_addrs. Using the API with just patch-1 yields ~1/2 the IPIs. Patch 2 sorts the patch_addrs, allowing the queue to fill to its 256-entry capacity before flushing, for a bigger win. Results Summary: Jump-Batch IPIs (Total CAL Interrupts) Pairs | -p 2 | -p 3 | -p 4 | -p 5 | --------------------------------------- 1 | 135 | 202 | 239 | 290 | 2 | 145 | 256 | 347 | 445 | 3 | 198 | 352 | 479 | 640 | 4 | 246 | 445 | 630 | 840 | 5 | 293 | 545 | 774 | 1,033 | 6 | 341 | 642 | 915 | 1,222 | Marginal Cost Comparison (IPIs added per '+p;-p;' query pair) VCPUs | Baseline (m) | Jump-Batch (m) | Efficiency Gain ------------------------------------------------------- 2 | 10,696 | 42.4 | 252x 3 | 21,466 | 89.3 | 240x 4 | 32,207 | 138.1 | 233x 5 | 42,945 | 190.5 | 225x Why not static_key_deferred? static_key_deferred is a rate-limiter for single, high-frequency keys. For a bulk update of 1,900 distinct keys, a deferred timer would still fire 1,900 times, resulting in the same IPI storm spread across a longer window. This will make it harder to use, particularly for tools which are fast enough to perceive the asynchrony. In contrast, jump-batch just amortizes the IPI cost, eliminating the need for rate-throttling. Why wasnt this done previously ? The presence of the queue for x86 suggests it was done, minimally, for the cases where the patch-addresses were ordered, and where performance mattered more (ftrace etc). In contrast, pr_debug()s are pretty basic, for direct use by humans, who dont toggle them on and off at high rates, and who are far more selective than "+p" the 2nd time they use dyndbg. Model Summary: The baseline cost is driven by (Keys * CPUs), resulting in 43k IPIs per transaction on a 5-VCPU VM. The jump-batch solution reduces this to ~190 IPIs, moving the bottleneck from key density (K) to architectural rendezvous cost (M). Detailed Data: NB: most tests below used the " !module virtio* +p ; -p ; " query-pair, relying on patch-5 for the "!module virtio*" parsing to avoid flooding the guest/VM logs with messages. This is less important when both on;off are sent together, as console flooding impedes submitting a separate off command. Baseline: jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 0 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 0 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10387 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 10111 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 161 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 28 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 10162 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10418 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10390 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10379 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10467 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 9812 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20856 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 20277 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 163 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 97 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 20259 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20856 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20861 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20818 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20848 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 20275 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31237 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 30349 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 171 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 101 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 30367 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31224 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31233 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31223 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31219 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 30346 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41593 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 40447 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 174 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 91 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 40451 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41546 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41595 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41622 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41607 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 40442 Patched: jimc@frodo:~/projects/lx/wk-A$ . ../farmit.sh jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 0 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 0 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 131 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 127 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 130 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 78 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 124 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 116 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 134 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 141 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 153 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 124 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 359 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 200 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 175 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 100 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 191 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 186 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 186 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 171 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 181 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 182 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 235 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 225 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 173 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 93 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 227 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 294 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 250 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 273 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 307 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 266 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 277 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 265 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 189 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 96 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 267 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 278 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 272 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 285 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 280 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 265 LINEAR PROGRESSION both progressions are linear, but patched has much lower slope and intercept (y=mx+b) Baseline: jimc@frodo:~/projects/lx/linux.git$ cd b0-dd jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 10738 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 21269 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 32125 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 42831 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 53540 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 64174 jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 21468 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 42976 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 64434 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 85886 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 107353 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 128800 jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 32148 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 64416 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 96604 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 128776 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 160985 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 193159 jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 42837 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 85880 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 128793 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 171697 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 214636 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 257562 jimc@frodo:~/projects/lx/linux.git/b0-dd$ PATCHED: jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 135 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 145 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 198 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 246 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 293 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 341 jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 202 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 256 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 352 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 445 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 545 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 642 jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 239 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 347 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 479 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 630 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 774 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 915 jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 290 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 445 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 640 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 840 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 1033 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 1222 jimc@frodo:~/projects/lx/wk-A/b0-dd$ Acknowldegement: I used google-gemini extensively to discuss and review this work, and to read the detailed data, and prepare the summary tables and prose above. It is patient with my pedantry. Signed-off-by: Jim Cromie Jim Cromie (5): jump_label: add queueing API for batched static key updates x86/alternative.c: sort text-pokes before flushing the queue dyndbg: use static-key queueing API in dynamic-debug dyndbg: factor ddebug_match_desc out from ddebug_change (for demonstration only) lib/dynamic_debug: add negation support to queries arch/x86/kernel/alternative.c | 42 +++++------ include/linux/jump_label.h | 18 +++++ kernel/jump_label.c | 104 +++++++++++++++++++------ lib/dynamic_debug.c | 138 ++++++++++++++++++++++------------ 4 files changed, 211 insertions(+), 91 deletions(-) base-commit: 7aaa8047eafd0bd628065b15757d9b48c5f9c07d -- 2.53.0