From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.174 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793735512; b=aO8bEKKLfyTeyBHKW6ONuHES+uztTCF9nuqtNMEcuO4cKJrVpxvp1KfHQ7+CVlj8Fxzu YzV5TDQPK5NlYEQdD1l+2wfGjM3BvH+4GzC5DIc04QSCDf7MbSxNToKdps/BGdDvzT7PZ 60hK+KheRIA8W9ZBsMztak1BZHEvJf3hcqqulVMKnAIEuosaXnMBXEKYori/8oWX4LgFV Wb8Rcg5IJI7yaqQgi1uue94EcY6Gc1oHQlO5bv4+QHPOXcFekG+Fh5KIuv+ONoae0oAL+ CZDaxg6CLEc9StB1EgRH7LanlVMQIgpaKBkWFN/x097ezGCAfrycf5PkQ7wTyIPbFTQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793735512; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; bh=WxrrLBpL+5w882waq3Llcz01wK2qP4mY4UJBFpkoOwI=; b=On/gVdVodLjvYbnpOgb1kqHwEsfy6FcAeNhny8SBahf1ni1q99qg1xTW3wGU8BzpOFp1 09Fp646WXQAoOiwrnC27Lk6qqEB7Qzxs2zU8xse4tzX1fPIp6N+0EJmWs52DVB8Jh80YA K6vtl61I2eNvTy9Z7aXYr3MFscCR6+g0Ihqc5EeKK98kNc6IU8BOKP0ywSjdyu01mqjAx GgcIcjEoWOYUzqWnxrcVhSiYFbxyW9p8H+/URyitN78y2Ymof+uOAvRJhv2nY/MsQ4tNZ h58liemk8WZRM8gKWIvP4DRRfloxoDOpfB0bLgJn/wBczn4vQ2U/BAs6bxlKjzAELHw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=purestorage.com; spf=pass smtp.mailfrom=purestorage.com; dkim=pass header.d=purestorage.com header.i=@purestorage.com header.a=rsa-sha256 header.s=google2022 header.b=WwSYwHPi; arc=none smtp.remote-ip=74.125.82.174 Received: from mail-dy1-f174.google.com (mail-dy1-f174.google.com [74.125.82.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1171D606A324 for ; Tue, 3 Nov 2026 19:51:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 1171D606A324 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=purestorage.com header.i=@purestorage.com header.a=rsa-sha256 header.s=google2022 header.b=WwSYwHPi Received: by mail-dy1-f174.google.com with SMTP id 5a478bee46e88-35cd4e8bdbdso5649041eec.1 for ; Tue, 03 Nov 2026 11:51:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1793735510; x=1794340310; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=WxrrLBpL+5w882waq3Llcz01wK2qP4mY4UJBFpkoOwI=; b=WwSYwHPitFHEdjhsZkjJSH7aWMdpA2YA//1JDRcF1RwEGXr/nMaefmAdOd6dOQ/eiX Bi7QfWeOW3DsCYxr2DGzEpqrrcTsjf6leFK3LNZ1J0NbB4jAi0qwokTKx0sXCZ6TC+BK iUO9xNgvq21yma8wfdqVogCOc82bHMBBblWnwsgvOOvJWC9OGGQNHiIEo0/VSpCjdyV8 o+FXQ5kUCmjbZ9kqx5DzijIRPUndCy54O0gm8w6yGHaGaKNUjmWXiKmaN0No6IRLw1ZB MUxkEvLyfuySLLaaQfvEFblT+krcv31ckRI+DkWYKsLD4IsQ4eey+jvUC1fiEoh9j7s2 bLAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793735510; x=1794340310; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=WxrrLBpL+5w882waq3Llcz01wK2qP4mY4UJBFpkoOwI=; b=LvjlWqDq7koVITji0dkcXGktxnYcbpwVry8TMpR+FNvVAZ6FKEldy87TsVfW3g+eh4 UhOwNPDjLmvjJ9hKnL10CQ5olWTAcuAK04OzPjsJ3jC35cckcL9eYtS7/AfsYaYgFbIX WBBQQ4WNWv8qwLM2kmLstvz6xeRPsLlOxeL3R54g9dlKFowtOXCVQUR4UfWpNjEouLfL gNKpKkjIIe0jx/DNNzu9T8k9RDKNVyP6AvrjFoLTTriLsRjm5x1qdZq6Sz90FV2AYsY2 Udnrn0ROT1GfEFck8ZabQLtgVU04+0siyV+qgYNwOEQn/jw7NLMGRNWU7dG4dnQwXd0o MPFw== X-Forwarded-Encrypted: i=1; AKwUvBxbAyNMP9g6dMVZFUscaef+j0f1xXEOZoi2e86f5Bw2DwIfj/uMRkuKXtJpQBObe8oh2s18QpDmE0KHevM=@vger.kernel.org X-Gm-Message-State: APuU4Z+KUpF9iA7ZEn+yCLpWqSQFPkfMW2cTY/P9dqvEHiqdwttAwkAW 5aCZJOePZloNCXowAOaI4oO7iOHnh3giEOYt9V80mdBDp1jJdQ5X6z8/IQGVSZCxWrw= X-Gm-Gg: AYBFou1tGj+i7Ibr3l7bV5bISrgTZmC1VjRiwGlXGuW8bP7pLk1LyND6o57lbeQtdS1 m/IUV8W9Y6j7YDdcMu9d/mhXIZISSGNNV5z9lBYcQ6PvWbv0h539UkttgAnCjyvs5aLlFFWts0A anEXygvkeuK3MEkXN45Qa3hLmUT2sHfzRKiwv/zptv+m5V2vcTuJ/22sQNPRnRRqSXLCAdqU/4Y JJpcVDk2Cin15t9EESleZQ/JPqzWl7i9kh1T46alXuUscCR2ywrKgTMAw/XMzX/wHm+5BQ5lKAz gRz0AL3SxqdAbj8sXLrxaFvUjX1/XU3Xq5C86tURkYooms+bB1MX96BIIs6zbxBFyCyQIPh83Us vDiYTxbnR5Cj+LXb73qMgKciCoXbEX9j4b843sy3zNvhg+9nxPPt25idQFb9FUsbZQd6zyXPUeR zBZHwoOaDB1gkFiWGC1PZPxBSgAXsRxKDWpRc0D/MbaJ7xSQRhAIda3DfBPZIGl9XbEL3VhIGho CzV3CRWZ6VA9aNBhP2FZvyINd1vmH7XfWMC4OxEe8XA5NMlWeF9bkf9PmQIG854j2cfo3jlIXmR DszZY8YRzv8PJtsuIseEdJAePkWZIZXsboHiWG4OKTdrzPvh X-Received: by 2002:a05:7301:438e:b0:36a:3767:ede0 with SMTP id 5a478bee46e88-36a4240176emr20856013eec.26.1793735509795; Tue, 03 Nov 2026 11:51:49 -0800 (PST) Received: from medusa.lab.kspace.sh ([208.88.152.253]) by smtp.googlemail.com with ESMTPSA id 5a478bee46e88-36ca38d36a1sm361141eec.12.2026.11.03.11.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Nov 2026 11:51:49 -0800 (PST) Date: Tue, 3 Nov 2026 11:51:48 -0800 From: Mohamed Khalfella To: mt4022@columbia.edu Cc: Jens Axboe , Keith Busch , Ming Lei , Bart Van Assche , Christoph Hellwig , Nilay Shroff , Josef Bacik , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, tz2294@columbia.edu, stable@vger.kernel.org Subject: Re: [PATCH] blk-mq: take quiesce_lock when removing a queue from the tag set Message-ID: <20261103195148.GB5019-mkhalfella@purestorage.com> References: <20261103-blk-mq-del-quiesce-lock-v1-1-04aa29a481f6@columbia.edu> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20261103-blk-mq-del-quiesce-lock-v1-1-04aa29a481f6@columbia.edu> On Tue 2026-11-03 14:04:10 -0500, Mani Tofigh via B4 Relay wrote: > From: Mani Tofigh > > Unquiescing a tag set can run the hw queues of a queue whose hctxs were > already freed. blk_mq_unquiesce_tagset() walks the tag set's queues > under quiesce_lock and rcu_read_lock(). Commit 71ec3c7d6db8 ("blk-mq: > quiesce a queue that joins a quiesced tag set") added quiesce_lock to > the walks and to adding a queue, but blk_mq_del_queue_tag_set() still > removes a queue without it. The hctxs are then freed right away since > RCU only keeps the request_queue itself alive: > > Task A (controller reset) Task B (namespace removal) > blk_mq_unquiesce_tagset() > /* finds q */ > blk_mq_del_queue_tag_set(q) > blk_mq_release(q) > /* frees the hctxs */ > blk_mq_run_hw_queues(q) > /* reads freed hctxs */ > > The window is short. On PREEMPT_RT quiesce_lock is a sleeping lock and > the walk can be preempted inside it, which is how it's hit here: > > [ 399.851277] BUG: KASAN: slab-use-after-free in blk_mq_run_hw_queues+0x4bd/0x550 > [ 399.851328] Read of size 8 at addr ff11000003014148 by task kworker/u9:2/116 > [ 399.851352] CPU: 1 UID: 0 PID: 116 Comm: kworker/u9:2 Not tainted 7.3.0-16299-g81cb9aa43246 #1 PREEMPT_RT > [ 399.851425] Workqueue: nvme-reset-wq nvme_loop_reset_ctrl_work > [ 399.851441] Call Trace: > [ 399.851457] > [ 399.851551] blk_mq_run_hw_queues+0x4bd/0x550 > [ 399.851564] blk_mq_unquiesce_tagset+0x128/0x210 > [ 399.851576] nvme_loop_reset_ctrl_work+0x78/0x4c0 > [ 399.851600] process_one_work+0x92b/0x16d0 > [ 399.851660] worker_thread+0x4e6/0xd60 > [ 399.851707] kthread+0x300/0x3e0 > [ 399.851727] ret_from_fork+0x4f0/0x6f0 > [ 399.851776] ret_from_fork_asm+0x1a/0x30 > [ 399.851793] > ... > [ 399.851988] Freed by task 111: > [ 399.852040] kfree+0x1b3/0x4b0 > [ 399.852050] blk_mq_release+0x400/0x550 > [ 399.852060] blk_put_queue+0x94/0x120 > [ 399.852068] disk_release+0x1bd/0x3f0 > [ 399.852102] nvme_free_ns+0x39/0x270 > [ 399.852113] nvme_remove_nsid_range+0x2d1/0x4a0 > [ 399.852123] nvme_scan_ns_list+0x2cc/0x390 > [ 399.852134] nvme_scan_work+0x421/0x9f0 > ... > > With BLK_CGROUP the disk's root blkg holds the queue until a grace > period later, which hides this for NVMe namespaces. It's still open > without BLK_CGROUP, and from reading the code also for queues freed > without a disk (e.g., SCSI device queues on a UFS host). > > Take quiesce_lock around the list_del_rcu() as well so that the walk > either finishes with a queue before it leaves the list or never sees > it. With the fix, KASAN doesn't report the use-after-free anymore. Do we need to update the documentation of @quiesce_lock to mention that it is now taken when a queue is deleted also? > > This bug was identified and confirmed with the help of a deterministic > hypervisor currently under development. The C reproducer below however > hits it about once every 20 minutes on a PREEMPT_RT kernel. > > Fixes: 59e25ef2b413 ("block: Use RCU in blk_mq_[un]quiesce_tagset() instead of set->tag_list_lock") Yep, my commit introduced the race by not taking set->tag_list_lock which provided the needed serialization previously. Thank you for fixing it. > Cc: stable@vger.kernel.org > Signed-off-by: Mani Tofigh Reviewed-by: Mohamed Khalfella