From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7136C433ED for ; Wed, 31 Mar 2021 22:38:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6277361008 for ; Wed, 31 Mar 2021 22:38:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229787AbhCaWhe (ORCPT ); Wed, 31 Mar 2021 18:37:34 -0400 Received: from mail-qk1-f171.google.com ([209.85.222.171]:42579 "EHLO mail-qk1-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230380AbhCaWhZ (ORCPT ); Wed, 31 Mar 2021 18:37:25 -0400 Received: by mail-qk1-f171.google.com with SMTP id y5so399107qkl.9 for ; Wed, 31 Mar 2021 15:37:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=+C5LPs0kaCkAOGtyUr9wjTwxTgqJsfzOVQYj3EvBqCI=; b=Ih70V0q5qnh233mEVhOTZnqYNsDR5H1ncJgMAkZGEgDMl4ploRp+74tgO4jmEpLyGS f5XoDaeW/agc5q6/M0kuhoVcd4GuEK+c8iks5KwR+PiVFJK54M7KgQcR5v0xdeGyg6iV Ui0tn0sWhcRHJ+FuYQTbXYPV6I9d4SkHFfdT9PpYwU8bDcAF0II6pO6wyKZAREtnjH13 tn4j3roDrNwRfcy3NpWAUWa2g348mC1kc+14CWBLIaFtcyfRwELs4EQvUI5V4DqEXLcH HLY6uPd5krkB0AshVRXYONjJz5WnpBj1VuS42KlK1c4y9mMyaYuWDw33GjOb0dbVbXMP 9Nng== X-Gm-Message-State: AOAM531gB3Kf8PS8TrDyaOd4GoXN2oh1c1U7NL3Mibl1KzoAz1cSAT9g 3d8CIJFI6JSwB6y8sv5cKFE= X-Google-Smtp-Source: ABdhPJwnyEfjA2R/2V/L5FusNudKlixHn/eMgkOOFBgpwD1Qrw1jiioFMoTF3opfrajqk0ksYE1MFg== X-Received: by 2002:a37:bc45:: with SMTP id m66mr5692073qkf.82.1617230244711; Wed, 31 Mar 2021 15:37:24 -0700 (PDT) Received: from ?IPv6:2600:1700:65a0:78e0:6302:5415:8f3:c3fc? ([2600:1700:65a0:78e0:6302:5415:8f3:c3fc]) by smtp.gmail.com with ESMTPSA id j18sm2387930qtl.83.2021.03.31.15.37.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 31 Mar 2021 15:37:24 -0700 (PDT) Subject: Re: [PATCH v2] nvme-tcp: Check if request has started before processing it To: Keith Busch Cc: "Ewan D. Milne" , Daniel Wagner , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Jens Axboe , Hannes Reinecke , Christoph Hellwig References: <20210301175601.116405-1-dwagner@suse.de> <6b51a989-5551-e243-abda-5872411ec3ff@grimberg.me> <20210311094345.ogm2lxqfuszktuhp@beryllium.lan> <70af5b02-10c1-ab0b-1dfc-5906216871b4@grimberg.me> <2fc7a320c86f75507584453dd2fbd744de5c170d.camel@redhat.com> <20210330232813.GA1935968@dhcp-10-100-145-180.wdc.com> From: Sagi Grimberg Message-ID: <756aef10-e693-276f-82ac-514a2832b07f@grimberg.me> Date: Wed, 31 Mar 2021 15:37:22 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <20210330232813.GA1935968@dhcp-10-100-145-180.wdc.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org >>>> It is, but in this situation, the controller is sending a second >>>> completion that results in a use-after-free, which makes the >>>> transport irrelevant. Unless there is some other flow (which is >>>> unclear >>>> to me) that causes this which is a bug that needs to be fixed rather >>>> than hidden with a safeguard. >>>> >>> >>> The kernel should not crash regardless of any network traffic that is >>> sent to the system. It should not be possible to either intentionally >>> of mistakenly contruct packets that will deny service in this way. >> >> This is not specific to nvme-tcp. I can build an rdma or pci controller >> that can trigger the same crash... I saw a similar patch from Hannes >> implemented in the scsi level, and not the individual scsi transports.. > > If scsi wants this too, this could be made generic at the blk-mq level. > We just need to make something like blk_mq_tag_to_rq(), but return NULL > if the request isn't started. Makes sense... >> I would also mention, that a crash is not even the scariest issue that >> we can see here, because if the request happened to be reused we are >> in the silent data corruption realm... > > If this does happen, I think we have to come up with some way to > mitigate it. We're not utilizing the full 16 bits of the command_id, so > maybe we can append something like a generation sequence number that can > be checked for validity. That's actually a great idea. scsi needs unique tags so it encodes the hwq in the upper 16 bits giving the actual tag the lower 16 bits which is more than enough for a single queue. We can do the same with a gencnt that will increment in both submission and completion and we can validate against it. This will be useful for all transports, so maintaining it in nvme_req(rq)->genctr and introducing a helper like: rq = nvme_find_tag(tagset, cqe->command_id) That will filter genctr, locate the request. Also: nvme_validate_request_gen(rq, cqe->command_id) that would compare against it. And then a helper to set the command_id like: cmd->common.command_id = nvme_request_command_id(rq) that will both increment the genctr and build a command_id from it. Thoughts?