From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bsdbackstore.eu (128-116-240-228.dyn.eolo.it [128.116.240.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1201220B210; Wed, 12 Feb 2025 15:33:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=128.116.240.228 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739374429; cv=none; b=Ul3MdBRAuoUlgPcN7wORe1yJzzI7NhD5PBOju7PvOxM9WLuroPo+0CgCvkzL1cYctMgsSLlN9qEkNUB6hWxoKJCZrV0Qv15Tla21QwUpd33cUQfnsA+G6Ma9V/UqgxgVyFD9oy6g/It12idqMnSyUVlt5vS/3gmPQoEeEvwX4Fw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739374429; c=relaxed/simple; bh=KtmEQw2SuitsUN1ob3U9iVTKc6sddhkgw1X5cOM+cEM=; h=Mime-Version:Content-Type:Date:Message-Id:From:To:Cc:Subject: References:In-Reply-To; b=aJrJbCbvMwxN7yG3sM8hitGMo6agTMQIjvwtmRfPCpHNonYcQU3Pttoa/1aNeJBAtMatLw0/A9SwKI6WFn6CLaE2huewP6hUOrMfDPRkukwfuF/U2m4/55nEBQmPq4KFsxwW/q9FtfbzKKNnP23ZFPO/ZSqMnYDzWoAsVCXfxtE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=bsdbackstore.eu; spf=pass smtp.mailfrom=bsdbackstore.eu; arc=none smtp.client-ip=128.116.240.228 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=bsdbackstore.eu Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bsdbackstore.eu Received: from localhost (25.205.forpsi.net [80.211.205.25]) by bsdbackstore.eu (OpenSMTPD) with ESMTPSA id 394be0f3 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Feb 2025 16:33:43 +0100 (CET) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Wed, 12 Feb 2025 16:33:41 +0100 Message-Id: From: "Maurizio Lombardi" To: "zhang.guanghui@cestc.cn" , "sagi" , "mgurtovoy" , "kbusch" , "sashal" , "chunguang.xu" Cc: "linux-kernel" , "linux-nvme" , "linux-block" Subject: Re: nvme-tcp: fix a possible UAF when failing to send request X-Mailer: aerc References: <2025021015413817916143@cestc.cn> In-Reply-To: <2025021015413817916143@cestc.cn> On Mon Feb 10, 2025 at 8:41 AM CET, zhang.guanghui@cestc.cn wrote: > Hello=20 > > =C2=A0 =C2=A0=C2=A0When using the nvme-tcp driver in a storage cluster, t= he driver may trigger a null pointer causing the host to crash several time= s. > By analyzing the vmcore, we know the direct cause is that=C2=A0 the reque= st->mq_hctx was used after free.=20 > > CPU1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0CPU2 > > nvme_tcp_poll=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 nvme_= tcp_try_send=C2=A0 --failed to send reqrest 13=C2=A0 > > =C2=A0 =C2=A0=C2=A0nvme_tcp_try_recv=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 nvme_tcp_fail_request > > =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0nvme_tcp_recv_skb=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 nvme_tcp_end_request > > =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0nvme_tcp_recv_pd= u=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 nvme_complete_rq=C2=A0 > > =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0nvme_tcp_handle_comp=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0nvme_retry_req --=C2=A0request-= >mq_hctx have been freed, is NULL. =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0 > =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0nvme_tcp_process_nvme_cqe=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2= =A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0 > > =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0 nvme_complete_rq > > =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0 nvme_end_req > > =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 blk_mq_end_request Taking a step back. Let's take a different approach and try to avoid the double completion. The problem here is that apparently we received a nvme_tcp_rsp capsule from the target, meaning that the command has been processed (I guess the capsule has an error status?) So maybe only part of the command has been sent? Why we receive the rsp capsule at all? Shouldn't this be treated as a fatal error by the controller? Maurizio