From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26892368D6D for ; Sun, 21 Jun 2026 13:59:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782050388; cv=none; b=IgD3bjjHNSaEmmSCKhY/80GSGuXK8AU+Qh/rtvZOrgZT5g1QFgSJ8/CBI2rsNEiSSTAOVEs17MNQys1kDdA3XLnCA9jhAk1bBiRFioatFzWo04HYjExFTUTc/62MDJL/sCwmNkOx1QbGSEcAGECn0gNblW5VTdYS97mVTleLATo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782050388; c=relaxed/simple; bh=ZYSDKTxUHodZ61wAYmAN91JQ9Nfbdpg5rfMp1d8fV8Y=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=uyLVUgYLaE09+CM/87nH9WgNPpNFNuawggsHzOX4TM8ziPFOSD8eSLvVNuutqjEywi+1MqyWf0rYpT/tm+408WM96blwqWfmL1zqnkCWV5KbR11sV57w8fdqe3u0uuBR5Ga+cjRKi/sy44ZpsnOf9n1+hBEOVfgKOFaCht8ysS0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VTb7DO/h; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VTb7DO/h" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2c0b9328c4aso24416495ad.0 for ; Sun, 21 Jun 2026 06:59:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782050386; x=1782655186; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Z5ZY8L9TFmkr2jMI9G4ucnP4Z/ebhyAzAin+hRxCCFc=; b=VTb7DO/hImYay3fuPjorRcH79E9Qf3iX/VRp/yQeyF9cUAmK+hju8PzKW7xIdbxgZh MdZNZM8S3XBxK9LrodEN7sfBWHHNG7sAoyDQz15CLltpEMBfRSyKZEHQvHPhtkdmV0z4 mHjMRaf3FNGOkiezLchmQ3uIVqyrEYgW5rYbZl6phDz320kULc5Qv8PZWH6jOFSt9yAf AAPQnEcNtd36Zg+xSuWFMXzDWCkCnMV1pvBqexR///rGL4bUPSD6tCdD4rRJJfRraz5f R4dMvvYV6xb8PQ8LM8MLraqWuDFBeZpcufdvbQOLmV0bpwdiiw2/Se8FUlLAgI1qaOZW SGuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782050386; x=1782655186; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Z5ZY8L9TFmkr2jMI9G4ucnP4Z/ebhyAzAin+hRxCCFc=; b=raVErVulygyO9kXG+/b8kGoSjRVuWyPM61Gf2AipREHcz4MLj9juXIctJ7HZIqcxy7 nd2x16VG45OpixVxAAqGxAQ2hyBIaya+7jOBw82w0JzY2sa0C4avC9JWB8IBB3zKWot3 u/IlpYbVWz8a1H91Pc4SnLkZII9cLssDSjWgi4RnPDJP7CokxwBTMSeeSugW8b3sitWR BQbVmF3eOBp2NFhsfaGaIPbkCzUiLqKoTIkoze7J7xcNaVTPTC2rj6iM/V+nf/XbDmGo xiw52lzAnwGqBlF56hmGjmkdobdDX2wG+lx/fWQk+yrmCW/MVWJ+jVT/muvmOSMIu7wj 9iPQ== X-Forwarded-Encrypted: i=1; AHgh+Rr4ueZFNGif8ZXh0sfwwOEc/Nxo7oM3ZecvkqR36g9cgke1blAeDGn4cgFkMC18m0KkdGui0134Dzomw1c=@vger.kernel.org X-Gm-Message-State: AOJu0YyjPtr8C9z2lJkABy4i7ne1FXblnqXL/+ZD/pnqFQMKmPp8bxCH qLz/LOkbcb+Vcq8aKgFgHepeTlYtIskPwcnkjyiBNbPZDsRaK9TSk7Ec X-Gm-Gg: AfdE7ckjjlJ8xHXKjWSCei2CDpbS1HdreGdFpBXmvnbPB1jU8Nn4kaCP+qy21sLH/O/ VPqONEiJySE+Z00wMdB4LH7rNtOAjS/i56tmvRTmKv0zS/v1clGnUDfQ0Nt2eORnK8sx7ZmxDqX Hvhov5tYFP2844k/n7JJFVon7LLFmzpycHy1TiQG96BRUng6GEXfPnLEEu31EaIMOIxgZMRfkuA UxGLqX69ReaQkfy/4kA+gO7FC+JS+p1JTztKJpU5NKQ1VlHzjNaHJHoDgVA/vo26ECUDBjDTCIE U+JpsEbNqu/vkcbY2n+mlTaeEIvy18KNOL9iKtmfKWpIjIMLDqaShnSvQwl0Edq4HS07noiYAEV BN/Du/k7MkvS1FlNQc7mvKPMQZ27Yo+Vsq9jgYsc3FclI6ZqVWUjv8yopG9OngFGVsIvFGQmOUO bSK0ICaq8= X-Received: by 2002:a17:902:d4ca:b0:2c2:21e9:5a7d with SMTP id d9443c01a7336-2c742b68fb0mr74488095ad.34.1782050386382; Sun, 21 Jun 2026 06:59:46 -0700 (PDT) Received: from localhost ([111.228.63.84]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c7439f9026sm48518445ad.53.2026.06.21.06.59.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Jun 2026 06:59:45 -0700 (PDT) From: Cen Zhang To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Steve Wise Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, baijiaju1990@gmail.com, zzzccc427@gmail.com Subject: [PATCH] nvme-rdma: avoid stale device wrapper in remove_one Date: Sun, 21 Jun 2026 21:59:39 +0800 Message-Id: <20260621135939.2658295-1-zzzccc427@gmail.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit nvme_rdma_remove_one() walks nvme_rdma_ctrl_list under nvme_rdma_ctrl_mutex, but it identifies matching controllers by reading ctrl->device->dev. The mutex only protects controller list membership. ctrl->device is a cached copy of queue 0's nvme_rdma_device, and that wrapper is refcounted by queue lifetime. The buggy scenario involves two paths, with each column showing the order within that path: RDMA remove callback: Controller error recovery: 1. enter nvme_rdma_remove_one() 1. run nvme_rdma_error_recovery_work() 2. walk nvme_rdma_ctrl_list 2. tear down the admin queue 3. read ctrl->device->dev 3. drop the final queue device ref 4. free the nvme_rdma_device wrapper Fix this by caching the ib_device identity in the controller when the admin queue is configured. The remove callback can then compare the cached pointer value against the ib_device being removed without dereferencing the queue-owned nvme_rdma_device wrapper. Keep the delete workqueue flush conditional on actually matching a controller. Validation reproduced this kernel report: BUG: KASAN: slab-use-after-free in nvme_rdma_remove_one+0x281/0x2c0 [nvme_rdma] Call Trace: dump_stack_lvl+0x66/0xa0 print_report+0xce/0x630 ? nvme_rdma_remove_one+0x281/0x2c0 [nvme_rdma] ? srso_alias_return_thunk+0x5/0xfbef5 ? __virt_addr_valid+0x20d/0x410 ? nvme_rdma_remove_one+0x281/0x2c0 [nvme_rdma] kasan_report+0xe0/0x110 ? nvme_rdma_remove_one+0x281/0x2c0 [nvme_rdma] nvme_rdma_remove_one+0x281/0x2c0 [nvme_rdma] remove_client_context+0xa9/0xf0 [ib_core] disable_device+0x12d/0x240 [ib_core] ? __pfx_disable_device+0x10/0x10 [ib_core] ? srso_alias_return_thunk+0x5/0xfbef5 ? __mutex_unlock_slowpath+0x147/0x900 __ib_unregister_device+0x26f/0x460 [ib_core] ib_unregister_device_and_put+0x55/0x70 [ib_core] nldev_dellink+0x29e/0x3c0 [ib_core] ? unwind_next_frame+0x6e3/0x2190 ? __pfx_nldev_dellink+0x10/0x10 [ib_core] ? lock_acquire+0x2b8/0x2f0 ? srso_alias_return_thunk+0x5/0xfbef5 ? cap_capable+0x196/0x330 ? __pfx_down_read+0x10/0x10 rdma_nl_rcv_msg+0x2db/0x5f0 [ib_core] ? __pfx_rdma_nl_rcv_msg+0x10/0x10 [ib_core] rdma_nl_rcv_skb.constprop.0.isra.0+0x222/0x380 [ib_core] ? __pfx_rdma_nl_rcv_skb.constprop.0.isra.0+0x10/0x10 [ib_core] ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? netlink_deliver_tap+0x150/0xac0 netlink_unicast+0x47c/0x790 ? __pfx_netlink_unicast+0x10/0x10 netlink_sendmsg+0x767/0xc30 ? __pfx_netlink_sendmsg+0x10/0x10 ? lock_release+0x1e0/0x280 __sys_sendto+0x339/0x390 ? __pfx___sys_sendto+0x10/0x10 ? srso_alias_return_thunk+0x5/0xfbef5 __x64_sys_sendto+0xe0/0x1c0 ? do_syscall_64+0x81/0x6a0 ? srso_alias_return_thunk+0x5/0xfbef5 ? trace_hardirqs_on+0x18/0x160 do_syscall_64+0x115/0x6a0 entry_SYSCALL_64_after_hwframe+0x77/0x7f Allocated by task 436: kasan_save_stack+0x33/0x60 kasan_save_track+0x14/0x30 __kasan_kmalloc+0xaa/0xb0 nvme_rdma_cm_handler+0xcbc/0x2914 [nvme_rdma] cma_cm_event_handler+0xb2/0x390 [rdma_cm] addr_handler+0x199/0x2b0 [rdma_cm] process_one_req+0x113/0x650 [ib_core] process_one_work+0x8d0/0x1870 worker_thread+0x575/0xf80 kthread+0x2e7/0x3c0 ret_from_fork+0x576/0x810 ret_from_fork_asm+0x1a/0x30 Freed by task 436: kasan_save_stack+0x33/0x60 kasan_save_track+0x14/0x30 kasan_save_free_info+0x3b/0x60 __kasan_slab_free+0x5f/0x80 kfree+0x307/0x580 nvme_rdma_free_dev+0x16d/0x260 [nvme_rdma] nvme_rdma_free_queue+0x6d/0x90 [nvme_rdma] nvme_rdma_error_recovery_work+0x7f/0x110 [nvme_rdma] process_one_work+0x8d0/0x1870 worker_thread+0x575/0xf80 kthread+0x2e7/0x3c0 ret_from_fork+0x576/0x810 ret_from_fork_asm+0x1a/0x30 Fixes: e87a911fed07 ("nvme-rdma: use ib_client API to detect device removal") Assisted-by: Codex:gpt-5.5 Signed-off-by: Cen Zhang --- drivers/nvme/host/rdma.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 6909e3542794..7c4a1e5f4991 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -116,6 +116,7 @@ struct nvme_rdma_ctrl { struct blk_mq_tag_set admin_tag_set; struct nvme_rdma_device *device; + struct ib_device *ib_device; u32 max_fr_pages; @@ -788,6 +789,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, return error; ctrl->device = ctrl->queues[0].device; + WRITE_ONCE(ctrl->ib_device, ctrl->device->dev); ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev); /* T10-PI support */ @@ -2372,31 +2374,20 @@ static struct nvmf_transport_ops nvme_rdma_transport = { static void nvme_rdma_remove_one(struct ib_device *ib_device, void *client_data) { struct nvme_rdma_ctrl *ctrl; - struct nvme_rdma_device *ndev; bool found = false; - mutex_lock(&device_list_mutex); - list_for_each_entry(ndev, &device_list, entry) { - if (ndev->dev == ib_device) { - found = true; - break; - } - } - mutex_unlock(&device_list_mutex); - - if (!found) - return; - /* Delete all controllers using this device */ mutex_lock(&nvme_rdma_ctrl_mutex); list_for_each_entry(ctrl, &nvme_rdma_ctrl_list, list) { - if (ctrl->device->dev != ib_device) + if (READ_ONCE(ctrl->ib_device) != ib_device) continue; + found = true; nvme_delete_ctrl(&ctrl->ctrl); } mutex_unlock(&nvme_rdma_ctrl_mutex); - flush_workqueue(nvme_delete_wq); + if (found) + flush_workqueue(nvme_delete_wq); } static struct ib_client nvme_rdma_ib_client = { -- 2.43.0