Re: [BUG] NULL deref in mempool_free_bulk via crypt_endio on dm-crypt teardown (6.8 -> 7.0 regression)

[Mikulas Patocka <mpatocka@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 3543 bytes --]



On Wed, 16 Sep 2026, Léo Terziman wrote:

> NULL pointer dereference in mempool_free_bulk via crypt_endio during
> dm-crypt device teardown
> 
> SUMMARY
> =======
> Since upgrading from Ubuntu 24.04 (Linux 6.8) to Ubuntu 26.04 (Linux 7.0),
> this machine panics roughly once a week. The panic is always a NULL pointer
> dereference in mempool_free_bulk(), reached from crypt_endio() in dm_crypt
> via bio_put()/bio_free(), running in softirq context from a SCSI completion.
> 
> It reproduces during teardown of a dm-crypt device: a late bio completion
> arrives after the dm-crypt target's bio_set has already been destroyed, so
> bio_free() returns the bio to a mempool whose ->elements is NULL.
> 
> The system was stable on 6.8 for approximately two years with the identical
> storage configuration and the identical backup script. Nothing in userspace
> changed at the time the crashes began.
> 
> 
> ENVIRONMENT
> ===========
> Distribution:  Ubuntu 26.04 LTS
> Kernel:        7.0.0-31-generic #31-Ubuntu PREEMPT(lazy) x86_64
> Hardware:      Supermicro X13SAE-F, BIOS 5.3 06/04/2026
> Memory:        64 GB ECC (ie31200_edac; 0 CE, 0 UE reported)
> DKMS modules:  none
> Taint:         G W  -- the W is an unrelated boot-time WARN in i915
>                (print_ddi_port, intel_bios.c VBT parsing, headless system).
>                No out-of-tree modules are loaded.
> 
> 
> STORAGE STACK
> =============
> Four backup volumes, each:
> 
>   iSCSI (iscsi_tcp, Synology target over TCP/IP)
>     -> sd (SCSI disk)
>       -> dm-crypt
>         -> btrfs (zstd:3, blake2b csums, async discard enabled)
> 
> Mounted together nightly by a backup script, then unmounted and torn down
> (umount -> cryptsetup close -> iscsiadm logout) when the backup finishes.
> 
> Separate from this, the system also uses hardware RAID -> bcache -> btrfs
> for primary storage and md RAID1 for SSD storage. Neither appears in any
> crash trace.
> 
> 
> TRIGGER
> =======
> Every crash occurs during teardown of the backup volumes, within seconds of
> unmount. The preceding log is consistent across occurrences:
> 
>   [256109.45] BTRFS info (device dm-15): last unmount of filesystem ...
>   [256109.53] sd 9:0:0:1:  [sdb] Synchronizing SCSI cache
>   [256109.75] BTRFS info (device dm-16): last unmount of filesystem ...
>   [256109.96] sd 10:0:0:1: [sdc] Synchronizing SCSI cache
>   [256110.06] BTRFS info (device dm-17): last unmount of filesystem ...
>   [256110.17] sd 11:0:0:1: [sdd] Synchronizing SCSI cache
>   [256111.50] BTRFS info (device dm-18): last unmount of filesystem ...
>   [256115.15] BTRFS warning (device dm-18): folio private not zero on folio
>               906100736
>               [~100 more identical warnings for consecutive folios]
>   [256115.18] BUG: kernel NULL pointer dereference, address: 0000000000000000
> 
> The "folio private not zero" flood immediately before the oops indicates
> btrfs is releasing folios that still carry I/O state, i.e. the unmount is
> completing while work is still outstanding beneath it.

I suspect that this is btrfs bug. It seems that btrfs closes a block 
device without waiting for bios to finish.

The mempool_free function was reworked between 6.8 and 7.0. In 6.8 it 
didn't crash if we attempted to free an entry into a free mempool. In 7.0 
it crashes in this case. So, the btrfs bug could have been there forever, 
it was just latent and didn't result in a crash on old kernels.

You can try to apply the patch 83f7e52b7ed1c3e03b79123e20b6f6adf8d886bb - 
maybe it helps.

Mikulas