[PATCH 00/25] mm/fbatch: drain lru_add_drain() and _all()

[Hugh Dickins <hughd@google.com>]

This series was prompted by lru_add_drain_all() appearing in watchdog
backtraces: not to blame, but blocked on an unresponsive CPU to run
its workqueue. lru_add_drain_all() is a heavyweight operation, which
we often hope to avoid by a much lighter lru_add_drain(); but even
those local drains can aggravate the lruvec lock contention which
per-cpu fbatches are intended to ease.

Now let the folios on the lru_add and other per-cpu fbatches remain
there, isolatable, with PG_lru set and refcount unraised, just like
when on an actual LRU. Then many calls to lru_add_drain() and _all()
can be removed.

It's something I've wanted to do for years, tried several times, but
only hit on a good way to do it a few weeks ago: now it's uncomfortable
to watch others wrestling with the drainage, while I'm sitting on this.

The key idea came from the "if (!folio_test_clear_lru(folio)) continue;".
If we don't mind missing to take an action on rare occasions, then maybe
we won't mind taking an action on the wrong folio on rare occasions, so
long as it is a folio consenting to PG_lru rules.

No new locking, but relies on folio_try_get() and  folio_test_clear_lru()
even on the lru_add fbatch; with use of bits not set in aligned pointers,
and some try_cmpxchg()ing.  Speculative references to folios are already
accepted: this adds another source of them.

Performance? You (and the bots) tell me. I'm considering this as a
cleanup, to make life easier for developers. I expect that some loads
will show improvement, but also expect some disappointments (perhaps
I go too far against lru_cache_disable()? or not far enough).

The timing is not so good: middle of a merge window is not a great
time to present new work; but I hope to be taking three weeks off in
two weeks time, so best to get this out in the open early, while I can
respond. If it's welcome in principle for 7.4, but too many changes
are demanded, maybe someone else can step up to shepherd it through.

Rebased and retested on Linus's tree of Sunday afternoon,
base-commit: 4352b8aee98005853aa63f57d6377282de17a33f
which includes the mm/swap.c to mm/folio.c renaming,
but not yet David's mods to mm/gup.c which conflict with my 13/25:
I'll reply to that one with an alternate patch to use once David's
two have gone in (reverting both of them and what was there before).

01/25 mm/fbatch: remove !CONFIG_SMP special case of folio_activate()
02/25 mm/fbatch: allow folios_put_refs() to skip xa_is_value() entries
03/25 mm/fbatch: temporarily disable lazyfree and mlock+munlock batching
04/25 mm/fbatch: lru bit set, no extra ref, while folio on per-cpu fbatch
05/25 mm/fbatch: lru_add_del_folio()+folio_add_lru() after clear_lru()
06/25 mm/fbatch: fbatch_drain_lazyfree(onstack fbatch) before ptl unlock
07/25 mm/fbatch: LRU_NEXT_ACTIVATE bit to optimize folio_activate()
08/25 mm/fbatch: replace mlock_new_folio() by __folio_add_lru(,mlockit)
09/25 mm/fbatch: restore mlock+munlock batching, without extra ref
10/25 mm/fbatch: remove several uses of mlock_drain_local()
11/25 mm/fbatch: remove migration's PAGE_WAS_MLOCKED lru_add_drain()
12/25 mm/fbatch: remove percpu_pvec_drained and folios_put()
13/25 mm/fbatch: no lru_add drain to collect_longterm_unpinnable_folios()
14/25 mm/fbatch: no lru_add_drain() nor _all() for memfd_wait_for_pins()
15/25 mm/fbatch: remove shake_folio() shake_page() from memory-failure
16/25 mm/fbatch: remove lru_cache_disable() from NUMA folio migration
17/25 mm/fbatch: no lru_cache_disable() in __alloc_contig_migrate_range()
18/25 mm/fbatch: remove lru_add_drain() and _all() calls from various
19/25 mm/fbatch: vm/stat_refresh include lru_add_drain() on each cpu
20/25 s390/fbatch: no lru_add_drain_all() in s390_wiggle_split_folio()
21/25 block/fbatch: no lru_add_drain_all() in invalidate_bdev()
22/25 fs/fbatch: drop_caches invalidate_bh_lrus() not lru_add_drain_all()
23/25 fs,mm/fbatch: use invalidate_bh_lrus() not invalidate_bh_lrus_cpu()
24/25 fs,mm/fbatch: lru_cache_disable() keep off buffer_head lrus only
25/25 mm/fbatch: move lru_add_drain_all() declaration to mm/internal.h

 Documentation/mm/unevictable-lru.rst |   2 +-
 arch/s390/kernel/uv.c                |   1 -
 block/bdev.c                         |   1 -
 fs/buffer.c                          |  40 +--
 fs/drop_caches.c                     |   4 +-
 include/linux/buffer_head.h          |   8 +-
 include/linux/folio_batch.h          |   7 +-
 include/linux/huge_mm.h              |   6 +-
 include/linux/mm.h                   |  18 --
 include/linux/mm_inline.h            |  22 ++
 include/linux/mm_types.h             |  12 +-
 include/linux/swap.h                 |  14 +-
 mm/compaction.c                      |  25 +-
 mm/fadvise.c                         |  17 +-
 mm/folio.c                           | 375 +++++++++------------------
 mm/gup.c                             |  14 -
 mm/huge_memory.c                     |  16 +-
 mm/hwpoison-inject.c                 |   1 -
 mm/internal.h                        |  15 +-
 mm/khugepaged.c                      |  11 -
 mm/ksm.c                             |  12 -
 mm/madvise.c                         |   9 +-
 mm/memfd.c                           |   6 +-
 mm/memory-failure.c                  |  38 +--
 mm/memory.c                          |  14 +-
 mm/memory_hotplug.c                  |   4 +
 mm/mempolicy.c                       |   7 -
 mm/migrate.c                         |  13 +-
 mm/migrate_device.c                  |   9 -
 mm/mlock.c                           | 192 ++++++--------
 mm/page_alloc.c                      |   3 -
 mm/rmap.c                            |   4 -
 mm/shmem.c                           |   2 -
 mm/truncate.c                        |  17 +-
 mm/vmscan.c                          |  16 +-
 mm/vmstat.c                          |   1 +
 36 files changed, 344 insertions(+), 612 deletions(-)

Hugh