[RFC PATCH 0/2] x86/mm/tlb: Skip stale and lazy CPUs in batched TLB flushes

[Bing Jiao <bingjiao@google.com>]

Since commit 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") and
commit 6db2526c1d69 ("x86/mm/tlb: Only trim the mm_cpumask once a second"),
context switches leave outgoing CPUs set in mm_cpumask(mm). Per-mm flushes
(flush_tlb_mm_range()) filter and periodically trim those stale CPUs using
info->mm, but batched TLB flushes (arch_tlbbatch_add_pending() and
arch_tlbbatch_flush()) pass info.mm = NULL and end up sending IPIs to
every non-lazy stale CPU in mm_cpumask(mm) on every reclaim, migration, or
page demotion batch without ever trimming them.

Patch 1 skips updating batch->cpumask in arch_tlbbatch_add_pending() when
X86_FEATURE_INVLPGB is enabled, where arch_tlbbatch_flush() uses broadcast
invalidation instead of batch->cpumask.

Patch 2 tracks up to ARCH_TLBBATCH_MAX_MMS (8) unique mm_struct pointers
per batch and filters stale and lazy CPUs out of batch->cpumask in
arch_tlbbatch_flush() before issuing remote IPIs.

Tested on 112-CPU Intel Skylake, 384-CPU AMD EPYC Genoa, and 512-CPU AMD
EPYC Turin with CXL memory, all using IPI-based TLB shootdowns (5-run
mean). Following the setup in commit ed15468787dd ("x86/mm: Move
flush_tlb_info back to the stack") and commit 7e12beb8ca2a
("migrate_pages: batch flushing TLB"), a task with all CPUs set in its
mm_cpumask runs on 1 CPU (2 CPUs in test 4) while the other CPUs run
unrelated userspace tasks:

1. Pageout reclaim (81,920 4K pages reclaimed to zram):

   Platform          Metric        Baseline       Patched        Change
   --------------------------------------------------------------------
   112-CPU Skylake   Remote IPIs   9,092,156          191       -99.99%
                     Time (ms)       2,974.6        763.0  -74.3% (3.9x)
   384-CPU Genoa     Remote IPIs  31,377,184          157       -99.99%
                     Time (ms)       1,652.6        483.1  -70.8% (3.4x)
   512-CPU Turin     Remote IPIs  41,859,670          119       -99.99%
                     Time (ms)       1,695.7        476.3  -71.9% (3.6x)

2. Page demotion (81,876 4K pages demoted from top-tier to CXL):

   Platform          Metric        Baseline       Patched        Change
   --------------------------------------------------------------------
   512-CPU Turin     Remote IPIs     958,699          526        -99.9%
                     Time (ms)         213.0        168.2        -21.0%

3. Single-threaded move_pages() (331,600 4K pages migrated):

   Platform          Metric        Baseline       Patched        Change
   --------------------------------------------------------------------
   112-CPU Skylake   Remote IPIs      73,370        4,964        -93.2%
                     Time (ms)         702.4        699.7         -0.4%
   384-CPU Genoa     Remote IPIs     252,899       13,899        -94.5%
                     Time (ms)         392.4        347.1        -11.6%
   512-CPU Turin     Remote IPIs     337,426       18,193        -94.6%
                     Time (ms)         370.1        368.1         -0.5%

4. Multi-threaded migrate_pages() + active memory access (656,940 4K
   pages migrated, 2 active threads):

   Platform          Metric        Baseline       Patched        Change
   --------------------------------------------------------------------
   112-CPU Skylake   Remote IPIs     144,610        3,362        -97.7%
                     Pages/sec       386,385      391,050         +1.2%
   384-CPU Genoa     Remote IPIs     493,035        7,730        -98.4%
                     Pages/sec       673,166      725,680         +7.8%
   512-CPU Turin     Remote IPIs     658,193        9,685        -98.5%
                     Pages/sec       670,288      733,020         +9.4%

Bing Jiao (2):
  x86/mm/tlb: Skip batch->cpumask updates when X86_FEATURE_INVLPGB is
    enabled
  x86/mm/tlb: Skip stale and lazy CPUs in arch_tlbbatch_flush()

 arch/x86/include/asm/tlbbatch.h | 14 +++++++++++++
 arch/x86/include/asm/tlbflush.h | 17 +++++++++++++++-
 arch/x86/mm/tlb.c               | 36 +++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+), 1 deletion(-)

-- 
2.56.0.385.gac671c819a-goog