From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 238852FD667 for ; Fri, 28 Nov 2025 09:44:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764323078; cv=none; b=fpq9bbOGoVuCvWEWjILCtdk8GYSeD9WpIHldttW74/zH03+EbumEtfLsQ1HEW/nqpHvfQhPglc5Q93wndi4ZKcmOCstZZQqQNkb4MdVnNaxFzSo04P//sYUI32tDIz2/B+cIEFazg6wNemA+vnqrhblI0kyWOU/V8inEKIjBJdE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764323078; c=relaxed/simple; bh=bjf33Eee+B+ByGgqZ49aTSVnUyhn9Q5javy2MOEEd4c=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=ftjvjZ8uMF2KX7JBUZF8V1UxZ+wd2RF2/5c9sRjF7Y7TdLkT+GQBcxqOf8J9XqN3t7aOliTA67u02T1Si7YANjirvh+0DLk/oElqwhNgcMk+KuUg+fLScgTFUe29gqvL9gjSoh3JCNhilFxzheTFsDS6sA4b/3IU7yXfgXKO7rU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.235]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4dHpJw3qMTzKHMK0 for ; Fri, 28 Nov 2025 17:43:44 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id 24CC01A0FCD for ; Fri, 28 Nov 2025 17:44:26 +0800 (CST) Received: from [10.174.178.72] (unknown [10.174.178.72]) by APP2 (Coremail) with SMTP id Syh0CgD3VHv2bilpsKYxCQ--.41006S3; Fri, 28 Nov 2025 17:44:23 +0800 (CST) Message-ID: Date: Fri, 28 Nov 2025 17:44:22 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown To: Nilay Shroff Cc: cve@kernel.org, linux-kernel@vger.kernel.org, yukuai@fnnas.com, ming.lei@redhat.com, "zhangyi (F)" , Hou Tao , yangerkun , Greg KH , zhengqixing@huawei.com References: <2025111256-CVE-2025-40146-b919@gregkh> <15fb6eed-e107-4ef7-a569-3328a02e2c39@huaweicloud.com> <2025112730-semicolon-verse-164a@gregkh> <82df4ae1-2940-4214-b46e-a2f8242b3582@huaweicloud.com> <813a7d60-6211-4048-9267-ec4bd4a75f73@linux.ibm.com> <9ef3f966-c9bc-2da0-3fbf-a926eee740c9@huawei.com> <8ad23abf-b5d2-4a16-a8bd-28df4ab6823f@linux.ibm.com> From: Zheng Qixing In-Reply-To: <8ad23abf-b5d2-4a16-a8bd-28df4ab6823f@linux.ibm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-CM-TRANSID:Syh0CgD3VHv2bilpsKYxCQ--.41006S3 X-Coremail-Antispam: 1UD129KBjvJXoWxXr4fWr4UJw1rCry7tryrJFb_yoW5Zr17pa 93K3yYywn5Crn7uw1Iyw1DWry8Cw10gF4UGrWSqFyUZ39xAan7KFs2kF4jgry8Ars2yayD tr98WrZ2yFyDAw7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUv0b4IE77IF4wAFF20E14v26r4j6ryUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x 0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG 6I80ewAv7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFV Cjc4AY6r1j6r4UM4x0Y48IcVAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7MxkF7I0En4kS 14v26r1q6r43MxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I 8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVWUtVW8 ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x 0267AKxVWUJVW8JwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_ Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IU1 7KsUUUUUU== X-CM-SenderInfo: x2kh0wptl0x03j6k3tpzhluzxrxghudrp/ 在 2025/11/28 15:15, Nilay Shroff 写道: >> commit b86433721f46d934940528f28d49c1dedb690df1 (HEAD -> master) >> Author: Yu Kuai >> Date:   Wed Sep 10 16:04:43 2025 +0800 >> >>     blk-mq: fix potential deadlock while nr_requests grown >> >>     Allocate and free sched_tags while queue is freezed can deadlock[1], >>     this is a long term problem, hence allocate memory before freezing >>     queue and free memory after queue is unfreezed. >> >>     [1] https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/ >>     Fixes: e3a2b3f931f5 ("blk-mq: allow changing of queue depth through sysfs") >> >>     Signed-off-by: Yu Kuai >>     Reviewed-by: Nilay Shroff >>     Signed-off-by: Jens Axboe >> >> We are assume that what's the problem Yu describe is when we update >> nr_request, we may need some memory allocation(nr_requests grows). And >> the memory allocation may trigger some memory reclaim, and fall into >> another I/O process, and since the request_queue has been freezen, there >> exist deadlock. >> >> But after checking the source code, there exist >> queue_requests_store->blk_mq_freeze_queue->memalloc_noio_save, the >> whole process which may trigger memory allocation won't trigger I/O >> process. So deadlock can not happened... And if that's true, this patch >> does not fix any problem. >> > Yes, memalloc_noio_save() is invoked before we freeze the queue (e.g., in > elv_iosched_store()), but that does not prevent the deadlock scenario described > in the lockdep splat. > > If you look closely at the splat, the problematic lock is not fs_reclaim (which > may be the first impression), but rather ->pcpu_alloc_mutex. From the splat, the > chain of dependencies looks like this: > > thread #0: blocked on q->elevator_lock > thread #1: blocked on ->pcpu_alloc_mutex > thread #2: blocked on fs-reclaim > > Here is the key detail: > > Thread #0 is running under GFP_NOIO scope (due to memalloc_noio_save()). > However, it is not blocked on fs_reclaim. Instead, it is blocked > on ->elevator_lock. > > Thread #1 is also running with GFP_NOIO and holds ->elevator_lock > while the queue is frozen. It is blocked on ->pcpu_alloc_mutex, > which is already held by Thread #2 (the thread that is stuck in > fs_reclaim). Thread #2 is running without GFP_NOIO scope. > > In other words: > - GFP_NOIO prevents a thread from entering fs_reclaim, but it does > not prevent triggering per-CPU memory allocations, which require > taking ->pcpu_alloc_mutex. > - This ->pcpu_alloc_mutex is the actual source of contention in the > splat, and it sits outside the protections offered by GFP_NOIO. > > That means: > - Even though memalloc_noio_save() avoids fs reclaim recursion, > it does not prevent per-CPU allocations from blocking, and thus > it cannot prevent the deadlock involving ->pcpu_alloc_mutex. > Thank you for the detailed explanation. Now I understand that there could indeed be a deadlock issue here :) Thanks, Qixing