[PATCH 0/5] jump_label: mitigate dynamic-debug induced IPI storms

[PATCH 0/5] jump_label: mitigate dynamic-debug induced IPI storms

[Jim Cromie]

This series adds batched toggling of jump_labels to address IPI storms
triggered by bulk toggles of dynamic-debug pr_debug()s.

patches 1-3 are for submission. 4-5 are for demonstration only.

The Scaling Problem:

In testing on 7.1-rc6 guests (virtme-ng) hosted on a stock Fedora
kernel, I measured the cost of toggling ~1,900 dyndbg callsites. The
current one-at-a-time patching approach creates a massive
synchronization bottleneck that scales linearly with both key count
and core count:

Baseline IPIs (Total CAL Interrupts)
Successive query pairs (+p ; -p ;) toggling ~3,792 keys per pair.
Pairs | -p 2 | -p 3 | -p 4 | -p 5 | Notes
-----------------------------------------------------------
  1   | 10,738 |  21,468 |  32,148 |  42,837 | ~11.3 IPI/key/CPU
  2   | 21,269 |  42,976 |  64,416 |  85,880 | Linear scaling (K*M)
  3   | 32,125 |  64,434 |  96,604 | 128,793 | 
  4   | 42,831 |  85,886 | 128,776 | 171,697 | 
  5   | 53,540 | 107,353 | 160,985 | 214,636 | 
  6   | 64,174 | 128,800 | 193,159 | 257,562 | > 250k IPI storm

On a 5-VCPU system, a sequence of 6 query pairs triggers over a 
quarter-million interrupts. This overhead is a direct result of 
invoking the full patching machinery for every individual key toggle.

The Solution:

While x86 already has a 256-entry queue for jump_label, 2 issues
currently block its use:

1. dyndbg uses the synchronous API; an IPI is issued for every static-key.
   patch-1 provides a fix for this with a _queued api extension.
   patch-3 uses the new api in dyndbg.

2. The queue flushes if current_patch_addr < previous_patch_addr.
   dyndbg has consecutive descriptors, but not consecutive patch_addrs.
   Using the API with just patch-1 yields ~1/2 the IPIs.
   Patch 2 sorts the patch_addrs, allowing the queue to fill to its 
   256-entry capacity before flushing, for a bigger win.

Results Summary:

Jump-Batch IPIs (Total CAL Interrupts)
Pairs | -p 2  | -p 3  | -p 4  | -p 5  |
---------------------------------------
  1   |  135  |  202  |  239  |  290  |
  2   |  145  |  256  |  347  |  445  |
  3   |  198  |  352  |  479  |  640  |
  4   |  246  |  445  |  630  |  840  |
  5   |  293  |  545  |  774  | 1,033 |
  6   |  341  |  642  |  915  | 1,222 |

Marginal Cost Comparison (IPIs added per '+p;-p;' query pair)
VCPUs | Baseline (m) | Jump-Batch (m) | Efficiency Gain
-------------------------------------------------------
  2   |    10,696    |      42.4      |      252x
  3   |    21,466    |      89.3      |      240x
  4   |    32,207    |     138.1      |      233x
  5   |    42,945    |     190.5      |      225x

Why not static_key_deferred?

static_key_deferred is a rate-limiter for single, high-frequency keys.
For a bulk update of 1,900 distinct keys, a deferred timer would still
fire 1,900 times, resulting in the same IPI storm spread across a
longer window.  This will make it harder to use, particularly for
tools which are fast enough to perceive the asynchrony.  In contrast,
jump-batch just amortizes the IPI cost, eliminating the need for
rate-throttling.

Why wasnt this done previously ?

The presence of the queue for x86 suggests it was done, minimally, for
the cases where the patch-addresses were ordered, and where
performance mattered more (ftrace etc).  In contrast, pr_debug()s are
pretty basic, for direct use by humans, who dont toggle them on and
off at high rates, and who are far more selective than "+p" the 2nd
time they use dyndbg.

Model Summary:

The baseline cost is driven by (Keys * CPUs), resulting in 43k IPIs 
per transaction on a 5-VCPU VM. The jump-batch solution reduces 
this to ~190 IPIs, moving the bottleneck from key density (K) 
to architectural rendezvous cost (M).

Detailed Data:

NB: most tests below used the " !module virtio* +p ; -p ; "
query-pair, relying on patch-5 for the "!module virtio*" parsing to
avoid flooding the guest/VM logs with messages.  This is less
important when both on;off are sent together, as console flooding
impedes submitting a separate off command.

Baseline:

jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 0
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 0
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10387
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 10111
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 161
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 28
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 10162
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10418
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10390
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10379
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10467
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 9812
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20856
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20277
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 163
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 97
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20259
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20856
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20861
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20818
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20848
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20275
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31237
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30349
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 171
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 101
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30367
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31224
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31233
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31223
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31219
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30346
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41593
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40447
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 174
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 91
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40451
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41546
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41595
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41622
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41607
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40442


Patched:

jimc@frodo:~/projects/lx/wk-A$ . ../farmit.sh 
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 0
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 0
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 131
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 127
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 130
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 78
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 124
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 116
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 134
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 141
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 153
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 124
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 359
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 200
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 175
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 100
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 191
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 186
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 186
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 171
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 181
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 182
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 235
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 225
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 173
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 93
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 227
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 294
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 250
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 273
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 307
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 266
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 277
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 265
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 189
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 96
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 267
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 278
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 272
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 285
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 280
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 265

LINEAR PROGRESSION

both progressions are linear, but patched has much lower slope and
intercept (y=mx+b)

Baseline:

jimc@frodo:~/projects/lx/linux.git$ cd b0-dd
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 10738
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 21269
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 32125
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42831
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 53540
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64174
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 21468
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42976
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64434
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 85886
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 107353
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128800
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 32148
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64416
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 96604
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128776
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 160985
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 193159
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42837
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 85880
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128793
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 171697
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 214636
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 257562
jimc@frodo:~/projects/lx/linux.git/b0-dd$ 

PATCHED:

jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 135
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 145
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 198
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 246
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 293
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 341
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 202
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 256
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 352
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 445
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 545
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 642
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 239
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 347
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 479
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 630
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 774
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 915
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 290
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 445
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 640
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 840
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 1033
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 1222
jimc@frodo:~/projects/lx/wk-A/b0-dd$ 

Acknowldegement:

I used google-gemini extensively to discuss and review this work, and
to read the detailed data, and prepare the summary tables and prose
above. It is patient with my pedantry.

Signed-off-by: Jim Cromie <jim.cromie@gmail.com>


Jim Cromie (5):
  jump_label: add queueing API for batched static key updates
  x86/alternative.c: sort text-pokes before flushing the queue
  dyndbg: use static-key queueing API in dynamic-debug
  dyndbg: factor ddebug_match_desc out from ddebug_change (for
    demonstration only)
  lib/dynamic_debug: add negation support to queries

 arch/x86/kernel/alternative.c |  42 +++++------
 include/linux/jump_label.h    |  18 +++++
 kernel/jump_label.c           | 104 +++++++++++++++++++------
 lib/dynamic_debug.c           | 138 ++++++++++++++++++++++------------
 4 files changed, 211 insertions(+), 91 deletions(-)


base-commit: 7aaa8047eafd0bd628065b15757d9b48c5f9c07d
-- 
2.53.0

[PATCH 1/5] jump_label: add queueing API for batched static key updates

[Jim Cromie]

Introduce a new queueing API for static keys to allow batching of
instruction updates.  This is provided for subsystems like
dynamic-debug, which may toggle thousands of static keys in a single
operation.  Currently, this suffers from (N)$ overhead due to repeated
machine-wide `stop_machine` or `text_poke_sync` synchronizations.

The new API includes:
- static_key_enable_queued(key)
- static_key_disable_queued(key)
- static_key_apply_queued() (the global synchronization barrier)
- Corresponding static_branch_*_queued(x) macros.

The _queued functionality is achieved by adding a 'bool apply'
parameter to 3 existing midlayer static functions, and using it to
decide whether or not to call arch_jump_label_transform_apply().  By
not explicitly calling it, we allow the transform queue to fill 1st.
This amortizes the cost of the IPI sync to 1/N (256 on x86).

1. __jump_label_update(struct static_key *key, bool apply)
   this implements the if (apply) arch_jump_label_transform_apply();

2. __jump_label_mod_update(struct static_key *key, bool apply)
   extends _queued feature to modules' static-keys

3. rename jump_label_update(struct static_key *key)
   to   __jump_label_update_key(struct static_key *key, bool apply)
   and wrap it with 1-liners:
   jump_label_update() & jump_label_update_queued()

On architectures supporting HAVE_JUMP_LABEL_BATCH, the _queued variants
lazily fill the architecture's patch queue.

It is safe to mix queued and immediate updates; any non-queued
static_key_enable/disable call will effectively "flush" any previously
queued changes as a side effect.  Or just call static_key_apply_queued().

On architectures that do not need a patch queue (e.g., those with atomic
patching) or that do not support batching, the API transparently falls back
to immediate patching.

Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
---
 include/linux/jump_label.h |  18 +++++++
 kernel/jump_label.c        | 104 +++++++++++++++++++++++++++++--------
 2 files changed, 100 insertions(+), 22 deletions(-)

diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
index fdb79dd1ebd8..283ee9360026 100644
--- a/include/linux/jump_label.h
+++ b/include/linux/jump_label.h
@@ -234,6 +234,9 @@ extern void static_key_slow_dec_cpuslocked(struct static_key *key);
 extern int static_key_count(struct static_key *key);
 extern void static_key_enable(struct static_key *key);
 extern void static_key_disable(struct static_key *key);
+extern void static_key_enable_queued(struct static_key *key);
+extern void static_key_disable_queued(struct static_key *key);
+extern void static_key_apply_queued(void);
 extern void static_key_enable_cpuslocked(struct static_key *key);
 extern void static_key_disable_cpuslocked(struct static_key *key);
 extern enum jump_label_type jump_label_init_type(struct jump_entry *entry);
@@ -340,6 +343,18 @@ static inline void static_key_disable(struct static_key *key)
 	atomic_set(&key->enabled, 0);
 }
 
+static inline void static_key_enable_queued(struct static_key *key)
+{
+	static_key_enable(key);
+}
+
+static inline void static_key_disable_queued(struct static_key *key)
+{
+	static_key_disable(key);
+}
+
+static inline void static_key_apply_queued(void) {}
+
 #define static_key_enable_cpuslocked(k)		static_key_enable((k))
 #define static_key_disable_cpuslocked(k)	static_key_disable((k))
 
@@ -535,6 +550,9 @@ extern bool ____wrong_branch_error(void);
 
 #define static_branch_enable(x)			static_key_enable(&(x)->key)
 #define static_branch_disable(x)		static_key_disable(&(x)->key)
+#define static_branch_enable_queued(x)		static_key_enable_queued(&(x)->key)
+#define static_branch_disable_queued(x)		static_key_disable_queued(&(x)->key)
+#define static_branch_apply_queued()		static_key_apply_queued()
 #define static_branch_enable_cpuslocked(x)	static_key_enable_cpuslocked(&(x)->key)
 #define static_branch_disable_cpuslocked(x)	static_key_disable_cpuslocked(&(x)->key)
 
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
index 7cb19e601426..9826aefd77e5 100644
--- a/kernel/jump_label.c
+++ b/kernel/jump_label.c
@@ -91,6 +91,7 @@ jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop)
 }
 
 static void jump_label_update(struct static_key *key);
+static void jump_label_update_queued(struct static_key *key);
 
 /*
  * There are similar definitions for the !CONFIG_JUMP_LABEL case in jump_label.h.
@@ -250,10 +251,57 @@ void static_key_disable(struct static_key *key)
 }
 EXPORT_SYMBOL_GPL(static_key_disable);
 
+void static_key_enable_queued(struct static_key *key)
+{
+	STATIC_KEY_CHECK_USE(key);
+
+	if (atomic_read(&key->enabled) > 0) {
+		/*
+		 * already enabled, don't act.
+		 * warn if static-key-inc was used.
+		 */
+		WARN_ON_ONCE(atomic_read(&key->enabled) != 1);
+		return;
+	}
+
+	scoped_guard(jump_label_lock) {
+		if (atomic_read(&key->enabled) == 0) {
+			/*
+			 * set transitional value, update,
+			 * then set stable enabled state
+			 */
+			atomic_set(&key->enabled, -1);
+			jump_label_update_queued(key);
+			atomic_set_release(&key->enabled, 1);
+		}
+	}
+}
+
+void static_key_disable_queued(struct static_key *key)
+{
+	STATIC_KEY_CHECK_USE(key);
+
+	if (atomic_read(&key->enabled) != 1) {
+		/*
+		 * not simply enabled; is disabled, inc-enabled, or
+		 * in transition. don't act. warn if inc-enabled.
+		 */
+		WARN_ON_ONCE(atomic_read(&key->enabled) != 0);
+		return;
+	}
+
+	scoped_guard(jump_label_lock) {
+		/*
+		 * update if simply enabled. dont act if in transition.
+		 */
+		if (atomic_cmpxchg(&key->enabled, 1, 0) == 1)
+			jump_label_update_queued(key);
+	}
+}
+
 static bool static_key_dec_not_one(struct static_key *key)
 {
 	int v;
-
 	/*
 	 * Go into the slow path if key::enabled is less than or equal than
 	 * one. One is valid to shut down the key, anything less than one
@@ -488,28 +536,16 @@ static bool jump_label_can_update(struct jump_entry *entry, bool init)
 	return true;
 }
 
-#ifndef HAVE_JUMP_LABEL_BATCH
 static void __jump_label_update(struct static_key *key,
 				struct jump_entry *entry,
 				struct jump_entry *stop,
-				bool init)
+				bool init, bool apply)
 {
 	for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) {
-		if (jump_label_can_update(entry, init))
-			arch_jump_label_transform(entry, jump_label_type(entry));
-	}
-}
-#else
-static void __jump_label_update(struct static_key *key,
-				struct jump_entry *entry,
-				struct jump_entry *stop,
-				bool init)
-{
-	for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) {
-
 		if (!jump_label_can_update(entry, init))
 			continue;
 
+#ifdef HAVE_JUMP_LABEL_BATCH
 		if (!arch_jump_label_transform_queue(entry, jump_label_type(entry))) {
 			/*
 			 * Queue is full: Apply the current queue and try again.
@@ -517,10 +553,24 @@ static void __jump_label_update(struct static_key *key,
 			arch_jump_label_transform_apply();
 			BUG_ON(!arch_jump_label_transform_queue(entry, jump_label_type(entry)));
 		}
+#else
+		arch_jump_label_transform(entry, jump_label_type(entry));
+#endif
 	}
-	arch_jump_label_transform_apply();
+#ifdef HAVE_JUMP_LABEL_BATCH
+	if (apply)
+		arch_jump_label_transform_apply();
+#endif
 }
+
+void static_key_apply_queued(void)
+{
+#ifdef HAVE_JUMP_LABEL_BATCH
+	jump_label_lock();
+	arch_jump_label_transform_apply();
+	jump_label_unlock();
 #endif
+}
 
 void __init jump_label_init(void)
 {
@@ -671,7 +721,7 @@ static int __jump_label_mod_text_reserved(void *start, void *end)
 	return ret;
 }
 
-static void __jump_label_mod_update(struct static_key *key)
+static void __jump_label_mod_update(struct static_key *key, bool apply)
 {
 	struct static_key_mod *mod;
 
@@ -692,7 +742,7 @@ static void __jump_label_mod_update(struct static_key *key)
 		else
 			stop = m->jump_entries + m->num_jump_entries;
 		__jump_label_update(key, mod->entries, stop,
-				    m && m->state == MODULE_STATE_COMING);
+				    m && m->state == MODULE_STATE_COMING, apply);
 	}
 }
 
@@ -762,7 +812,7 @@ static int jump_label_add_module(struct module *mod)
 		/* Only update if we've changed from our initial state */
 do_poke:
 		if (jump_label_type(iter) != jump_label_init_type(iter))
-			__jump_label_update(key, iter, iter_stop, true);
+			__jump_label_update(key, iter, iter_stop, true, true);
 	}
 
 	return 0;
@@ -892,7 +942,7 @@ int jump_label_text_reserved(void *start, void *end)
 	return ret;
 }
 
-static void jump_label_update(struct static_key *key)
+static void jump_label_update_key(struct static_key *key, bool apply)
 {
 	struct jump_entry *stop = __stop___jump_table;
 	bool init = system_state < SYSTEM_RUNNING;
@@ -901,7 +951,7 @@ static void jump_label_update(struct static_key *key)
 	struct module *mod;
 
 	if (static_key_linked(key)) {
-		__jump_label_mod_update(key);
+		__jump_label_mod_update(key, apply);
 		return;
 	}
 
@@ -916,7 +966,17 @@ static void jump_label_update(struct static_key *key)
 	entry = static_key_entries(key);
 	/* if there are no users, entry can be NULL */
 	if (entry)
-		__jump_label_update(key, entry, stop, init);
+		__jump_label_update(key, entry, stop, init, apply);
+}
+
+static void jump_label_update(struct static_key *key)
+{
+	jump_label_update_key(key, true);
+}
+
+static void jump_label_update_queued(struct static_key *key)
+{
+	jump_label_update_key(key, false);
 }
 
 #ifdef CONFIG_STATIC_KEYS_SELFTEST
-- 
2.53.0

[PATCH 2/5] x86/alternative.c: sort text-pokes before flushing the queue

[Jim Cromie]

Until now, x86 can queue jump_label text-pokes as long as the
poke-addr is monotonically increasing, but flushes the queue when a
new poke-addr is less than the previous.

Dynamic-debug now uses the queued static-key API, but the advantage is
limited; we see a ~2x reduction in IPIs.  Although the pr_debug
descriptors are ordered, the patch-address in them are not; about 1/2
violate the ordering constraint.

So this patch drops that requirement, and sorts the text-pokes by
their address before applying them.  Doing so lets us fill the queue
before sorting then flushing, giving a dramatic ~125x reduction in
IPIs over the traditional single IPI per pr_debug.

Other arches don't need a queue, and so have nothing to sort.

   #> dd_ipis
   [   23.100381] dyndbg: query 0: "module !virtio* +p " mod:*
   [   23.103432] dyndbg: query 1: "-p" mod:*
   Delta-CAL (IPI): 242

Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
---
 arch/x86/kernel/alternative.c | 42 ++++++++++++++++-------------------
 lib/dynamic_debug.c           |  1 +
 2 files changed, 20 insertions(+), 23 deletions(-)

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index e87da25d1236..92987954b8aa 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -2,6 +2,7 @@
 #define pr_fmt(fmt) "SMP alternatives: " fmt
 
 #include <linux/mmu_context.h>
+#include <linux/sort.h>
 #include <linux/perf_event.h>
 #include <linux/vmalloc.h>
 #include <linux/memory.h>
@@ -2823,6 +2824,18 @@ static __always_inline int patch_cmp(const void *tpl_a, const void *tpl_b)
 	return 0;
 }
 
+static int text_poke_loc_cmp(const void *a, const void *b)
+{
+	const struct smp_text_poke_loc *tpl_a = a;
+	const struct smp_text_poke_loc *tpl_b = b;
+
+	if (tpl_a->rel_addr < tpl_b->rel_addr)
+		return -1;
+	if (tpl_a->rel_addr > tpl_b->rel_addr)
+		return 1;
+	return 0;
+}
+
 noinstr int smp_text_poke_int3_handler(struct pt_regs *regs)
 {
 	struct smp_text_poke_loc *tpl;
@@ -2935,6 +2948,10 @@ void smp_text_poke_batch_finish(void)
 	if (!text_poke_array.nr_entries)
 		return;
 
+	if (text_poke_array.nr_entries > 1)
+		sort(text_poke_array.vec, text_poke_array.nr_entries,
+		     sizeof(struct smp_text_poke_loc), text_poke_loc_cmp, NULL);
+
 	lockdep_assert_held(&text_mutex);
 
 	/*
@@ -3151,28 +3168,6 @@ static void __smp_text_poke_batch_add(void *addr, const void *opcode, size_t len
 	}
 }
 
-/*
- * We hard rely on the text_poke_array.vec being ordered; ensure this is so by flushing
- * early if needed.
- */
-static bool text_poke_addr_ordered(void *addr)
-{
-	WARN_ON_ONCE(!addr);
-
-	if (!text_poke_array.nr_entries)
-		return true;
-
-	/*
-	 * If the last current entry's address is higher than the
-	 * new entry's address we'd like to add, then ordering
-	 * is violated and we must first flush all pending patching
-	 * requests:
-	 */
-	if (text_poke_addr(text_poke_array.vec + text_poke_array.nr_entries-1) > addr)
-		return false;
-
-	return true;
-}
 
 /**
  * smp_text_poke_batch_add() -- update instruction on live kernel on SMP, batched
@@ -3189,8 +3184,9 @@ static bool text_poke_addr_ordered(void *addr)
  */
 void __ref smp_text_poke_batch_add(void *addr, const void *opcode, size_t len, const void *emulate)
 {
-	if (text_poke_array.nr_entries == TEXT_POKE_ARRAY_MAX || !text_poke_addr_ordered(addr))
+	if (text_poke_array.nr_entries == TEXT_POKE_ARRAY_MAX)
 		smp_text_poke_batch_finish();
+
 	__smp_text_poke_batch_add(addr, opcode, len, emulate);
 }
 
diff --git a/lib/dynamic_debug.c b/lib/dynamic_debug.c
index 18a71a9108d3..b5060749464e 100644
--- a/lib/dynamic_debug.c
+++ b/lib/dynamic_debug.c
@@ -264,6 +264,7 @@ static int ddebug_change(const struct ddebug_query *query,
 		}
 	}
 	mutex_unlock(&ddebug_lock);
+	v2pr_info("applied %d queued updates to sites in total\n", nfound);
 
 	if (!nfound && verbose)
 		pr_info("no matches for query\n");
-- 
2.53.0

[PATCH 3/5] dyndbg: use static-key queueing API in dynamic-debug

[Jim Cromie]

Use the new static-key queueing API in dynamic-debug to reduce IPIs.

this replaces the 2 calls to static_branch_{enable.disable}() in
ddebug_change() with their _queued() counterparts, and adds a call to
static_branch_apply_queued() at the bottom.

This gives a theoretical reduction of IPIs up to 256x, and a practical
reduction by 120-180x.

Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
---
 lib/dynamic_debug.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/dynamic_debug.c b/lib/dynamic_debug.c
index b5060749464e..9bee7dde5148 100644
--- a/lib/dynamic_debug.c
+++ b/lib/dynamic_debug.c
@@ -250,9 +250,9 @@ static int ddebug_change(const struct ddebug_query *query,
 #ifdef CONFIG_JUMP_LABEL
 			if (dp->flags & _DPRINTK_FLAGS_PRINT) {
 				if (!(newflags & _DPRINTK_FLAGS_PRINT))
-					static_branch_disable(&dp->key.dd_key_true);
+					static_branch_disable_queued(&dp->key.dd_key_true);
 			} else if (newflags & _DPRINTK_FLAGS_PRINT) {
-				static_branch_enable(&dp->key.dd_key_true);
+				static_branch_enable_queued(&dp->key.dd_key_true);
 			}
 #endif
 			v4pr_info("changed %s:%d [%s]%s %s => %s\n",
@@ -263,6 +263,7 @@ static int ddebug_change(const struct ddebug_query *query,
 			dp->flags = newflags;
 		}
 	}
+	static_branch_apply_queued();
 	mutex_unlock(&ddebug_lock);
 	v2pr_info("applied %d queued updates to sites in total\n", nfound);
 
-- 
2.53.0

[PATCH 4/5] dyndbg: factor ddebug_match_desc out from ddebug_change (for demonstration only)

[Jim Cromie]

ddebug_change() is a big (~100 lines) function with a nested for loop.

The outer loop walks the per-module ddebug_tables list, and does
module stuff: it filters on a query's "module FOO*" and "class BAR",
failures here skip the entire inner loop.

The inner loop (60 lines) scans a module's descriptors.  It starts
with a long block of filters on function, line, format, and the
validated "BAR" class (or the legacy/_DPRINTK_CLASS_DFLT).

These filters "continue" past pr_debugs that don't match the query
criteria, before it falls through the code below that counts matches,
then adjusts the flags and static-keys.  This is unnecessarily hard to
think about.

So move the per-descriptor filter-block into a boolean function:
ddebug_match_desc(desc), and change each "continue" to "return false".
This puts a clear interface in place, so any future changes are either
inside, outside, or across this interface.

also fix checkpatch complaints about spaces and braces.

Reviewed-by: Louis Chauvet <louis.chauvet@bootlin.com>
Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
---
 lib/dynamic_debug.c | 83 +++++++++++++++++++++++++--------------------
 1 file changed, 47 insertions(+), 36 deletions(-)

diff --git a/lib/dynamic_debug.c b/lib/dynamic_debug.c
index 9bee7dde5148..0b13810b03db 100644
--- a/lib/dynamic_debug.c
+++ b/lib/dynamic_debug.c
@@ -172,6 +172,52 @@ static struct ddebug_class_map *ddebug_find_valid_class(struct ddebug_table cons
  * callsites, normally the same as number of changes.  If verbose,
  * logs the changes.  Takes ddebug_lock.
  */
+static bool ddebug_match_desc(const struct ddebug_query *query,
+			      struct _ddebug *dp,
+			      int valid_class)
+{
+	/* match site against query-class */
+	if (dp->class_id != valid_class)
+		return false;
+
+	/* match against the source filename */
+	if (query->filename &&
+	    !match_wildcard(query->filename, dp->filename) &&
+	    !match_wildcard(query->filename,
+			    kbasename(dp->filename)) &&
+	    !match_wildcard(query->filename,
+			    trim_prefix(dp->filename)))
+		return false;
+
+	/* match against the function */
+	if (query->function &&
+	    !match_wildcard(query->function, dp->function))
+		return false;
+
+	/* match against the format */
+	if (query->format) {
+		if (*query->format == '^') {
+			char *p;
+			/* anchored search. match must be at beginning */
+			p = strstr(dp->format, query->format + 1);
+			if (p != dp->format)
+				return false;
+		} else if (!strstr(dp->format, query->format)) {
+			return false;
+		}
+	}
+
+	/* match against the line number range */
+	if (query->first_lineno &&
+	    dp->lineno < query->first_lineno)
+		return false;
+	if (query->last_lineno &&
+	    dp->lineno > query->last_lineno)
+		return false;
+
+	return true;
+}
+
 static int ddebug_change(const struct ddebug_query *query,
 			 struct flag_settings *modifiers)
 {
@@ -204,42 +250,7 @@ static int ddebug_change(const struct ddebug_query *query,
 		for (i = 0; i < dt->num_ddebugs; i++) {
 			struct _ddebug *dp = &dt->ddebugs[i];
 
-			/* match site against query-class */
-			if (dp->class_id != valid_class)
-				continue;
-
-			/* match against the source filename */
-			if (query->filename &&
-			    !match_wildcard(query->filename, dp->filename) &&
-			    !match_wildcard(query->filename,
-					   kbasename(dp->filename)) &&
-			    !match_wildcard(query->filename,
-					   trim_prefix(dp->filename)))
-				continue;
-
-			/* match against the function */
-			if (query->function &&
-			    !match_wildcard(query->function, dp->function))
-				continue;
-
-			/* match against the format */
-			if (query->format) {
-				if (*query->format == '^') {
-					char *p;
-					/* anchored search. match must be at beginning */
-					p = strstr(dp->format, query->format+1);
-					if (p != dp->format)
-						continue;
-				} else if (!strstr(dp->format, query->format))
-					continue;
-			}
-
-			/* match against the line number range */
-			if (query->first_lineno &&
-			    dp->lineno < query->first_lineno)
-				continue;
-			if (query->last_lineno &&
-			    dp->lineno > query->last_lineno)
+			if (!ddebug_match_desc(query, dp, valid_class))
 				continue;
 
 			nfound++;
-- 
2.53.0

[PATCH 5/5] lib/dynamic_debug: add negation support to queries

[Jim Cromie]

This allow users to invert the selection of most keywords, excluding
lineno and class.  For example:

  echo "!module virtio* +p" > /proc/dynamic_debug/control

When testing a kernel running in virtme-ng, this cmd prevents flooding
the logs with virtio activity, and spoiling the the test scenario.

NOTE:

This is convenient, but not perfect; it cannot also avoid flooding
from pr_debugs in serial_core or other potential sources.

A more robust command is:

  echo "!module virtio* +p % module serial -p" > /proc/dynamic_debug/control

It also "interferes" with evolution of the grammar; particularly
around and/or combinations.  For example this could become meaningful,
though perhaps inadvisable:

  echo " !module virtio* !module serial +p "

lineno is currently excluded, pending review of its usefulness vs
ambiguities, for example " !lineno 1-20 " has straightforward meaning.

!class FOO is unlikely to be justifiable; it defeats the "protection
thru specificity" that classes are designed for.

Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
---
 lib/dynamic_debug.c | 81 +++++++++++++++++++++++++++++++--------------
 1 file changed, 57 insertions(+), 24 deletions(-)

diff --git a/lib/dynamic_debug.c b/lib/dynamic_debug.c
index 0b13810b03db..3addfa06e54d 100644
--- a/lib/dynamic_debug.c
+++ b/lib/dynamic_debug.c
@@ -58,6 +58,10 @@ struct ddebug_query {
 	const char *format;
 	const char *class_string;
 	unsigned int first_lineno, last_lineno;
+	unsigned int filename_neg:1;
+	unsigned int module_neg:1;
+	unsigned int function_neg:1;
+	unsigned int format_neg:1;
 };
 
 struct ddebug_iter {
@@ -139,11 +143,12 @@ static void vpr_info_dq(const struct ddebug_query *query, const char *msg)
 			fmtlen--;
 	}
 
-	v3pr_info("%s: func=\"%s\" file=\"%s\" module=\"%s\" format=\"%.*s\" lineno=%u-%u class=%s\n",
+	v3pr_info("%s: func%s=\"%s\" file%s=\"%s\" module%s=\"%s\" format%s=\"%.*s\" lineno=%u-%u class=%s\n",
 		  msg,
-		  query->function ?: "",
-		  query->filename ?: "",
-		  query->module ?: "",
+		  query->function_neg ? "!" : "", query->function ?: "",
+		  query->filename_neg ? "!" : "", query->filename ?: "",
+		  query->module_neg ? "!" : "", query->module ?: "",
+		  query->format_neg ? "!" : "",
 		  fmtlen, query->format ?: "",
 		  query->first_lineno, query->last_lineno, query->class_string);
 }
@@ -176,35 +181,38 @@ static bool ddebug_match_desc(const struct ddebug_query *query,
 			      struct _ddebug *dp,
 			      int valid_class)
 {
+	bool match;
+
 	/* match site against query-class */
 	if (dp->class_id != valid_class)
 		return false;
 
 	/* match against the source filename */
-	if (query->filename &&
-	    !match_wildcard(query->filename, dp->filename) &&
-	    !match_wildcard(query->filename,
-			    kbasename(dp->filename)) &&
-	    !match_wildcard(query->filename,
-			    trim_prefix(dp->filename)))
-		return false;
+	if (query->filename) {
+		match = match_wildcard(query->filename, dp->filename) ||
+			match_wildcard(query->filename, kbasename(dp->filename)) ||
+			match_wildcard(query->filename, trim_prefix(dp->filename));
+		if (match == query->filename_neg)
+			return false;
+	}
 
 	/* match against the function */
-	if (query->function &&
-	    !match_wildcard(query->function, dp->function))
-		return false;
+	if (query->function) {
+		match = match_wildcard(query->function, dp->function);
+		if (match == query->function_neg)
+			return false;
+	}
 
 	/* match against the format */
 	if (query->format) {
 		if (*query->format == '^') {
-			char *p;
 			/* anchored search. match must be at beginning */
-			p = strstr(dp->format, query->format + 1);
-			if (p != dp->format)
-				return false;
-		} else if (!strstr(dp->format, query->format)) {
-			return false;
+			match = (strstr(dp->format, query->format + 1) == dp->format);
+		} else {
+			match = !!strstr(dp->format, query->format);
 		}
+		if (match == query->format_neg)
+			return false;
 	}
 
 	/* match against the line number range */
@@ -234,9 +242,11 @@ static int ddebug_change(const struct ddebug_query *query,
 	list_for_each_entry(dt, &ddebug_tables, link) {
 
 		/* match against the module name */
-		if (query->module &&
-		    !match_wildcard(query->module, dt->mod_name))
-			continue;
+		if (query->module) {
+			bool match = match_wildcard(query->module, dt->mod_name);
+			if (match == query->module_neg)
+				continue;
+		}
 
 		if (query->class_string) {
 			map = ddebug_find_valid_class(dt, query->class_string, &valid_class);
@@ -399,6 +409,16 @@ static int parse_linerange(struct ddebug_query *query, const char *first)
 	return 0;
 }
 
+static char *check_neg(char *src, unsigned int *neg)
+{
+	if (*src == '!') {
+		*neg = 1;
+		return src + 1;
+	}
+	*neg = 0;
+	return src;
+}
+
 static int check_set(const char **dest, char *src, char *name)
 {
 	int rc = 0;
@@ -441,12 +461,15 @@ static int ddebug_parse_query(char *words[], int nwords,
 	}
 
 	for (i = 0; i < nwords; i += 2) {
-		char *keyword = words[i];
+		unsigned int neg;
+		char *keyword = check_neg(words[i], &neg);
 		char *arg = words[i+1];
 
 		if (!strcmp(keyword, "func")) {
+			query->function_neg = neg;
 			rc = check_set(&query->function, arg, "func");
 		} else if (!strcmp(keyword, "file")) {
+			query->filename_neg = neg;
 			if (check_set(&query->filename, arg, "file"))
 				return -EINVAL;
 
@@ -464,16 +487,26 @@ static int ddebug_parse_query(char *words[], int nwords,
 					return -EINVAL;
 			}
 		} else if (!strcmp(keyword, "module")) {
+			query->module_neg = neg;
 			rc = check_set(&query->module, arg, "module");
 		} else if (!strcmp(keyword, "format")) {
 			string_unescape_inplace(arg, UNESCAPE_SPACE |
 							    UNESCAPE_OCTAL |
 							    UNESCAPE_SPECIAL);
+			query->format_neg = neg;
 			rc = check_set(&query->format, arg, "format");
 		} else if (!strcmp(keyword, "line")) {
+			if (neg) {
+				pr_err("negation not supported for \"line\"\n");
+				return -EINVAL;
+			}
 			if (parse_linerange(query, arg))
 				return -EINVAL;
 		} else if (!strcmp(keyword, "class")) {
+			if (neg) {
+				pr_err("negation not supported for \"class\"\n");
+				return -EINVAL;
+			}
 			rc = check_set(&query->class_string, arg, "class");
 		} else {
 			pr_err("unknown keyword \"%s\"\n", keyword);
-- 
2.53.0