Re: [PATCH] squashfs: avoid thundering-herd cache wakeups

[Phillip Lougher <phillip@squashfs.org.uk>]

On 24/07/2026 20:05, Usama Arif wrote:
> squashfs_cache_get() puts a task to sleep when its block is not cached and
> every cache entry is busy.  Those sleeps are non-exclusive, so a single
> squashfs_cache_put() makes every waiter runnable when one entry is freed.
> A wakee returns to squashfs_cache_get() only if it observes cache->unused
> before the entry is reclaimed.  Such tasks can rescan and share a block
> published meanwhile; later wakees see zero and queue again inside
> wait_event() without rescanning.
> 
> The waste is dramatic under load.  On a Meta production host serving a
> Python web application from a packaged squashfs image, a 30-second trace
> caught 1,045,132 cache-release wake calls and 19,511,556 remote wakeups:
> 18.7 per release, although each release added only one reusable cache
> entry. This was causing significant spikes in CPU usage.
> 
> Simply making the waits exclusive (i.e. using
> wait_event_state_exclusive()) is not enough.  A waiter can make
> progress two ways, when an entry frees (capacity), or when another task
> publishes the block it wants and it shares that entry.  The only wait
> condition available is cache->unused, which captures capacity but not
> publication, and wake-one can release just a single sharer at a time.  A
> waiter woken for capacity may also find its block already published, share
> it, and leave the freed entry unclaimed, so that wake must be handed on.
> A block-targeted wake and that handoff need a custom wake callback with
> per-waiter state; wait_event_state_exclusive() provides neither.
> 
> Wake selectively instead.  Waiters are exclusive and keyed by requested
> block: freeing an entry wakes one waiter (capacity), publishing a block
> wakes every waiter for that block (sharing), and a capacity waiter that
> ends up sharing hands its wake to the next waiter.  Waiters enqueue while
> holding cache->lock so lookup and publication are ordered against sleeping.
> 
> The result is an ~2x increase in throughput on stat and read.
> Measured on a 32-CPU VM against a read-only squashfs (gzip, per-cpu
> decompressor, default 8 metadata / 3 fragment cache entries) staged in tmpfs
> with caches dropped each iteration to force cold decompression:
> 
>    elbencho, 64 threads
>      metadata stat      700 ->  1320 files/s    1.9x
>      small-file read     40 ->    60 MiB/s      1.6x
> 
>    filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
>      throughput       11,314 -> 25,186 ops/s    2.2x
>      latency           11.30 ->   5.05 ms/op    2.2x lower
>      sched:sched_wakeup 9.18M ->  3.44M         2.7x fewer
>      context switches  12.62M ->  5.77M         2.2x fewer
> 
> The 2.7x cut in scheduler wakeups explains the results: it roughly doubles
> throughput and halves latency under contention, and is a no-op on
> workloads that never queue for a cache entry.
> 
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>   fs/squashfs/cache.c | 66 ++++++++++++++++++++++++++++++++++++++++-----

>   1 file changed, 60 insertions(+), 6 deletions(-)
> 

Hi Usama,

Sashiko (the new patch reviewing AI) failed to apply and review your patch.

See https://sashiko.dev/#/patchset/20260724190556.1950693-1-usama.arif%40linux.dev

I have been dealing with this issue all week because Sashiko also rejected my
patch sent this week.

https://sashiko.dev/#/patchset/20260805175900.600140-1-phillip%40squashfs.org.uk

After quite a bit of investigation, it appears the problem is Sashiko is applying
the patches against the squashfs-next-git URL listed in the MAINTAINERS file.

Unfortunately that squashfs-next.git tree is unused and twelve years out of date
(last updated 2014).

See

https://lore.kernel.org/all/267938f6-8212-4ddc-9055-452f9d166010@squashfs.org.uk/

and

https://lore.kernel.org/all/anUi3bkwfM8teo_R@debarbos-thinkpadt14gen5.rmtusma.csb/


But as it stands your patch and mine are going nowhere because Andrew won't
pull any patch that has not passed Sashiko.

See

https://lwn.net/Articles/1064830/


I can see two ways of getting past this issue, and get Sashiko to approve the patches.

1.  Wait until my patch to the MAINTAINERS file goes through and then send a V2
     patch.  Unfortunately, that may take months.

2. The URL mentioned in the last email details how Sashiko chooses the baseline.
    See https://github.com/sashiko-dev/sashiko/pull/398/files.  Apparently the
    first thing it uses is any base-commit: trailer in the patch body.  So if
    you put a base-commit: in your patch it should force Sashiko to use the Linus
    Torvald's development kernel.

For info about base-commit: tags see

https://people.kernel.org/monsieuricon/all-patches-must-include-base-commit-info

I'm going to resend my patch with a base-commit: tag to see if it makes
a difference to Sashiko.

Phillip