[PATCH v4 0/4] scsi: target: rd: Fix oversized ramdisk allocations

[Sanan Hasanov <sanan.hasanou@gmail.com>]

From: Sanan Hasanov <sanan.hasanov@ucf.edu>

A ramdisk page count set through configfs (rd_pages=) has no upper bound.
syzbot found that a large value makes rd_build_device_space() attempt a
single sg table array allocation above the page allocator's maximum
order, which triggers a WARNING. Patch 1 rejects page counts that
exceed system RAM and allocates the array with kvzalloc so that valid
large ramdisks also work.

A page count close to system RAM still passes that check, and the
backing pages are allocated with GFP_KERNEL, which invokes the OOM
killer instead of failing. On a 1 GiB VM, rd_pages=250000 panics the
system with "System is deadlocked on memory". Patch 3 allocates the
backing pages with __GFP_RETRY_MAYFAIL so that enabling such a device
fails with -ENOMEM instead.

That makes the failure path in rd_build_prot_space() reachable, and it
leaks the partially allocated protection space when pi_prot_type is
written again. Patch 2 fixes that first, so that patch 3 does not
introduce a leak.

Patch 4 fixes an older bug in the same function: the protection space
size is computed in 32 bits and wraps for ramdisks of 2 TiB and larger,
which leaves the protection sg tables too small.

Tested on an arm64 QEMU VM with 1 GiB RAM, with the syzbot reproducer,
normal and NULLIO ramdisks, DIF protection space, device removal, and
rd_pages=250000. The leak was checked with kmemleak by making
pi_prot_type=1 fail under memory pressure and then writing it again:
two unreferenced objects from rd_init_prot() are reported without
patch 2, none with it. I could not test the 2 TiB case of patch 4.

pi_prot_type_store() does not serialize concurrent writes, so two
writers can still race in rd_build_prot_space() and
rd_release_prot_space(). That was already the case before this series
and is not addressed here.

Changes in v4:
- Patch 2: build the protection space in local variables and store it
  in rd_dev only on success, so that the error path cannot free an
  array installed by a concurrent pi_prot_type write (Sashiko AI
  review).
- New patch 4 to compute the protection space size in 64 bits
  (Sashiko AI review).

Changes in v3:
- New patch 2 to release the protection space when its allocation
  fails (Sashiko AI review).

Changes in v2:
- Allocate the sg table arrays with kvzalloc_objs() so that ramdisks
  larger than 512 GiB do not hit the same WARNING (Sashiko AI review).
- New patch to avoid the OOM killer when the backing pages cannot be
  allocated (Sashiko AI review).

v3: https://lore.kernel.org/all/20261008235411.56641-1-sanan.hasanou@gmail.com/
v2: https://lore.kernel.org/all/20261008223712.49827-1-sanan.hasanou@gmail.com/
v1: https://lore.kernel.org/all/20261008215709.46014-1-sanan.hasanou@gmail.com/

Sanan Hasanov (4):
  scsi: target: rd: Fix oversized sg table array allocation
  scsi: target: rd: Release protection space on allocation failure
  scsi: target: rd: Don't invoke the OOM killer for ramdisk pages
  scsi: target: rd: Avoid 32-bit overflow in protection space size

 drivers/target/target_core_rd.c | 29 ++++++++++++++++++++---------
 1 file changed, 20 insertions(+), 9 deletions(-)


base-commit: 6c377d19d4a5116d9bec5203aa3c6c11523e7898
-- 
2.48.1