Re: [PATCH] nvme: reject completions for requests that are not in flight

[Mauricio Faria de Oliveira <mfo@igalia.com>]

On 2026-11-03 16:21, K wrote:
> Thanks, pls let me take a look, it looks like all AI generated. May I

AI has not been used for analysis or the commit message; just the
reproducer, as mentioned.
Please check git log for previous contribution history.

If you're not comfortable with the commit message, I can certainly
change the authorship.
Please just let me know your preference (e.g., Originally-by,
Co-developed-by, name in the commit message, or none).

> know what kind of test you run or LLM run?

Sure. It's a synthetic reproducer that runs the completion handler
again.

Code: https://termbin.com/40tl

In a VM with 1 vCPU, (so nvme_handle_cqe() -> nvme_try_complete_req() ->
blk_mq_complete_request_remote() returns false),
echo mq-deadline >/sys/block/nvme0n1/queue/scheduler # (so
nvme_handle_cqe() -> blk_mq_add_to_batch() also returns false),
  # with both returning false, nvme_handle_cqe() ->
nvme_pci_complete_rq() is taken, reaching __blk_mq_free_request(), which
does rq->mq_hctx = NULL
echo 1 >/sys/module/nvme/parameters/repro_arm # prepare reproducer
dd if=/dev/nvme0n1 of=/dev/null bs=4k count=1 iflag=direct # oops

cheers,

> 
> Regards,
> Chao
> 
> On Tue, Nov 3, 2026 at 2:17 PM Mauricio Faria de Oliveira
> <mfo@igalia.com> wrote:
>>
>> From: Chao Shi <coshi036@gmail.com>
>>
>> There are a few kernel crash reports from the Steam Deck where incorrect
>> completions received by the nvme driver have a command id/tag associated
>> with a freed request (rq->mq_hctx = NULL), dereferencing a NULL pointer.
>>
>>   nvme_handle_cqe()
>>   -> nvme_try_complete_req(req, ...)
>>      -> blk_mq_complete_request_remote(req)
>>         -> if ((rq->mq_hctx->nr_ctx == 1 &&   /* Oops! */
>>
>> Although the usual suspect is the nvme device firmware, one report shows
>> incorrect completions also received by the ath11k wireless driver at the
>> same time, which suggests the underlying issue might be something else.
>>
>>   [ 2585.223155] ath11k_pci 0000:03:00.0: tx completion for unknown msdu_id 0
>>   [ 2585.223163] ath11k_pci 0000:03:00.0: tx completion for unknown msdu_id 0
>>   ...
>>   [ 2585.228077] nvme nvme0: request 0xec genctr mismatch (got 0x2 expected 0xa)
>>   [ 2585.228079] nvme nvme0: invalid id 8428 completed on queue 3
>>   [ 2585.228205] nvme nvme0: request 0xe2 genctr mismatch (got 0xe expected 0x2)
>>   [ 2585.228208] nvme nvme0: invalid id 57570 completed on queue 3
>>   ...
>>   [ 2585.228726] BUG: kernel NULL pointer dereference, address: 00000000000000fe
>>   ...
>>   [ 2585.228752] RIP: 0010:blk_mq_complete_request_remote+0x1c/0x160
>>   ...
>>   [ 2585.228787] Call Trace:
>>   [ 2585.228790]  <IRQ>
>>   [ 2585.228828]  nvme_poll_cq+0x14b/0x300
>>   [ 2585.228838]  nvme_irq+0x41/0x80
>>   [ 2585.228845]  __handle_irq_event_percpu+0x4a/0x180
>>   [ 2585.228850]  handle_irq_event+0x38/0x90
>>   [ 2585.228853]  handle_edge_irq+0x8b/0x230
>>   [ 2585.228858]  __common_interrupt+0x3e/0x90
>>   [ 2585.228862]  common_interrupt+0x80/0xa0
>>   [ 2585.228866]  </IRQ>
>>   [ 2585.228868]  <TASK>
>>   ...
>>
>> However, even when the nvme device firmware is at fault and need an update,
>> it is interesting for the kernel not to crash, so that it is even possible
>> to run the firmware update where the issue happens frequently or early on,
>> and while waiting for the update to be made available by the device vendor.
>>
>> The fix is to check that a request is actually in-flight before completing
>> it. This has been previously reported and proposed by Chao Shi, found with
>> fuzzing, but without further feedback since July. I happened to learn of it
>> only after root-cause analysis and reaching a similar conclusion, checking
>> for prior work/feedback, thus keeping them as Author/Signed-off-by, adding
>> myself as Co-developed-by for the commit message and root cause analysis.
>>
>> In Chao Shi's patch, there's been feedback/question of whether this should
>> be done in blk-mq, as this is not specific to nvme. I'd say no, at first:
>> - Some blk-mq calls for completion have no return value (e.g., completion
>>   on other CPUs), which suggests maybe an API change and updates to many
>>   callers, all for an edge case.
>> - An alternative available currently might be blk_mq_ops.complete(), but
>>   that is handled in the driver anyway, with risk of hitting an error in
>>   blk-mq (like this) as the code changes, and seems semantically wrong,
>>   i.e., completing a request that should not have been completed.
>> - Keeping in mind a small fix for stable, for now, should this be merged.
>> - But happy to take suggestions if blk-mq is indeed the way to go.
>>
>> This has been tested with a synthetic reproducer produced by an LLM.
>>
>> Originally-by: Chao Shi <coshi036@gmail.com>
>> Signed-off-by: Chao Shi <coshi036@gmail.com>
>> Link: https://lore.kernel.org/linux-nvme/20260522153034.2168862-1-coshi036@gmail.com/
>> Link: https://lore.kernel.org/linux-nvme/20260715005105.1115958-1-coshi036@gmail.com/
>> Co-developed-by: Mauricio Faria de Oliveira <mfo@igalia.com>
>> Signed-off-by: Mauricio Faria de Oliveira <mfo@igalia.com>
>> ---
>>  drivers/nvme/host/nvme.h | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
>> index 2d8f681079e85601a92624d196e77f1ba3ace34a..3ba0666da7be6a3de2e5f0fd50155af115b4e1f6 100644
>> --- a/drivers/nvme/host/nvme.h
>> +++ b/drivers/nvme/host/nvme.h
>> @@ -708,6 +708,11 @@ static inline struct request *nvme_find_rq(struct blk_mq_tags *tags,
>>                         tag, genctr, nvme_genctr_mask(nvme_req(rq)->genctr));
>>                 return NULL;
>>         }
>> +       if (unlikely(blk_mq_rq_state(rq) != MQ_RQ_IN_FLIGHT)) {
>> +               dev_err(nvme_req(rq)->ctrl->device,
>> +                       "completion for request %#x not in flight\n", tag);
>> +               return NULL;
>> +       }
>>         return rq;
>>  }
>>
>>
>> ---
>> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
>> change-id: 20261102-nvme-double-cmpl-uaf-blk-mq-e3d28f74cdaf
>>
>> Best regards,
>> --
>> Mauricio Faria de Oliveira <mfo@igalia.com>
>>

-- 
Mauricio