[PATCH net 0/2] net/mlx5: command mailbox use-after-free, and the teardown hang hiding it

[Danielle Costantino <dcostantino@meta.com>]

Two defects in the mlx5 command interface, both reached through the same
door: a firmware command that times out.

1/2 bounds the drain in mlx5_cmd_allowed_opcode() and
mlx5_cmd_change_mod().  Both take every unit of cmd->vars.sem, and since
commit 8e715cd613a1 ("net/mlx5: Set command entry semaphore up once got
index free") a unit is only returned when the entry's refcount reaches
zero, which for a timed-out entry never happens.  One stalled slot blocks
them forever, and destroy_async_eqs() calls both from mlx5_unload(), so a
function with any stalled command cannot be removed: the task stays in D
state holding the devlink instance lock, and reboot hits the same path.

2/2 stops the driver handing a firmware-owned DMA mailbox back to
dev->cmd.pool.  When mlx5_cmd_comp_handler(forced) keeps an entry and its
queue slot because firmware may still complete the command, cmd_exec() and
the callback path free the mailboxes anyway.  dmapool keeps its free list
node in the first 16 bytes of the block, which for mlx5 is the start of the
command payload, so a late firmware write corrupts the allocator and a
subsequent dma_pool_alloc() follows a poisoned pointer.  That is the crash.

1/2 comes first because it is what makes 2/2's teardown cleanup reachable:
without it teardown deadlocks long before mlx5_cmd_disable(), so nothing in
this series would ever run there.  They are otherwise independent - 1/2
does not touch anything 2/2 changes - and each builds on its own.

Testing
=======

A 32-slot aarch64 device with 22 mlx5 functions, debug kernel (KASAN,
DEBUG_OBJECTS, DEBUG_LIST, DMA_API_DEBUG, DEBUG_SPINLOCK), driving real
-ETIMEDOUT by swallowing firmware completions with a kprobe, then
unbinding the function.

  control, unbind/rebind with no stalled slots
      unbind 10s, rebind 3s, no warnings, no splats - the drain refactor
      leaves the normal path alone

  1/2, with 13 stalled slots
      before  unbind never returns; task in D state in
              mlx5_cmd_allowed_opcode() with cmd->vars.sem at 0
      after   the drain gives up three times, once per quiesce call in
              destroy_async_eqs(), at 61.4s intervals - the command
              timeout - then the unbind completes in 195s and the
              function rebinds in 2s
      the partial drain declines to narrow the allowed opcode, once,
      and still widens it back afterwards

  2/2, same 13 stalled slots
      mailbox ownership taken on 13 of 13 timed-out entries
      slots sharing one lay->out_ptr ............. 0   (7 and 6 before)
      firmware-owned mailbox on pool free list ... 0   (8 and 7 before)
      pool->next_block .......................... valid, was garbage
      entries retired ........................... 13, matching the slots
      "dma_pool_destroy mlx5_cmd busy" .......... 0   (1 without the
                                                      reclaim)
      kmemleak reports of mlx5_cmd_work_ent ..... 0   (14 without it)

  across every run
      no KASAN, refcount_t, list corruption or dma_pool warnings

Danielle Costantino (2):
  net/mlx5: Bound the command interface drain so teardown cannot hang
  net/mlx5: Don't return firmware-owned command mailboxes to the DMA
    pool

 drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 231 +++++++++++++++---
 .../net/ethernet/mellanox/mlx5/core/main.c    |  21 +-
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |   2 +-
 include/linux/mlx5/driver.h                   |   6 +
 4 files changed, 228 insertions(+), 32 deletions(-)


base-commit: 83a945a529d6e002dd7339c532288a931f463dba