From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 258D0C3DA62 for ; Thu, 17 Aug 2023 15:53:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353365AbjHQPx0 (ORCPT ); Thu, 17 Aug 2023 11:53:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346695AbjHQPwx (ORCPT ); Thu, 17 Aug 2023 11:52:53 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3563530D8 for ; Thu, 17 Aug 2023 08:52:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692287528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cb3vOuQ+kH0/pSZM4Qtlmw5+LiUYQstAOby+SW+Bxfk=; b=ToyGIX+gFplI16cnC1/4acUyFWmSX0RjPNyWHqxHCNU0GyDNz/GdzV4v6KvvKNTGK/Mald vo+h1nmLzefrlSAYQ5iCCzWXl7pfHzdwFDtdcdv1m3GNmB7ySFgpWDb+Z5hr5Fl9ALyrvr N8hlLCca+1zh3iOR3yOvygqSkEN94oQ= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-581-AUaE4jQ9OvCDZ0u-Al1L_Q-1; Thu, 17 Aug 2023 11:52:07 -0400 X-MC-Unique: AUaE4jQ9OvCDZ0u-Al1L_Q-1 Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-978a991c3f5so119664066b.0 for ; Thu, 17 Aug 2023 08:52:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692287524; x=1692892324; h=content-transfer-encoding:mime-version:user-agent:date:cc:to:from :subject:message-id:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=cb3vOuQ+kH0/pSZM4Qtlmw5+LiUYQstAOby+SW+Bxfk=; b=QLnMvuvp3n7v7r3Yhmm+wyr3EsiSQ3rf3aOSPkCN48OHiJfwYh+bCzp6f6RWL7UBtH tUsE7nkmDBBnDLe+3ckACAxbj4tHl44lKdgylvuQ3NflqKoIoGlPhH8ol68BuVpdRUh/ zRlSx1fpL/YYfqOyClUmjavikbMgyjL8G57gtTZoRwPe/CcZkskTNGy5xhBLn891AV7u tnk0NUI3uJj8CSk49w44idsWpcYZeHsdnwTB0juy1emMOSe/nX9HsGQXmF4mepSorFCN l5OWZBhiXsyLuPaDHdRGldPlBLlWou+Rc24l+hQ2ZXbkYUkkvUg21pBkUqirnBumc1jk P1Gg== X-Gm-Message-State: AOJu0YzJBqePHE9AJNGPASle8hvQFZmMo3R3TeMq3hW0t4wJmHJg+Gq2 X7m2QtBgmtPjbqztY30CABXQXmHOV6etYlnlwj8OpZi++NR2mAqobhcHWov8kjyYmXOA5MNgFvL HhFmUE62aDwyPHbMT2jGAvzV3Lay7zRhl X-Received: by 2002:a17:907:78d6:b0:994:580c:5049 with SMTP id kv22-20020a17090778d600b00994580c5049mr3921259ejc.5.1692287524483; Thu, 17 Aug 2023 08:52:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGEYSuhHshTbCW1nXq3/UGD+eEan0fWSN3Vav9nH3UaSvKfRR7o325QMlMkQgHjS9vknG90yw== X-Received: by 2002:a17:907:78d6:b0:994:580c:5049 with SMTP id kv22-20020a17090778d600b00994580c5049mr3921251ejc.5.1692287524165; Thu, 17 Aug 2023 08:52:04 -0700 (PDT) Received: from starship ([77.137.131.138]) by smtp.gmail.com with ESMTPSA id l18-20020a1709065a9200b00991e2b5a27dsm10312068ejq.37.2023.08.17.08.52.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Aug 2023 08:52:03 -0700 (PDT) Message-ID: <2d47431decaaf4bba0023c91ef0d7fd51b84333b.camel@redhat.com> Subject: Commit 'sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage' broke O_DIRECT over NFS From: Maxim Levitsky To: linux-nfs@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Date: Thu, 17 Aug 2023 18:52:01 +0300 Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi! I just updated my developement systems to 6.5-rc6 (from 6.4) and now I can't start a VM with a disk which is mounted over the NFS. The VM has two qcow2 files, one depends on another and qemu opens both. This is the command line of qemu: -drive if=none,id=os_image,file=./disk_s1.qcow2,aio=native,discard=unmap,cache=none The disk_s1.qcow2 depends on disk_s0.qcow2 However this is what I get: qemu-system-x86_64: -drive if=none,id=os_image,file=./disk_s1.qcow2,aio=native,discard=unmap,cache=none: Could not open backing file: Could not open './QFI?': No such file or directory 'QFI?' is qcow2 file signature, which signals that there might be some nasty corruption happening. The program was supposed to read a field inside the disk_s1.qcow2 file which should read 'disk_s0.qcow2' but instead it seems to read the first 4 bytes of the file. Bisect leads to the above commit. Reverting it was not possible due to many changes. Both the client and the server were tested with the 6.5-rc6 kernel, but once rebooting the server into the 6.4, the bug disappeared, thus I did a bisect on the server. When I tested a version before the offending commit on the server, the 6.5-rc6 client was able to work with it, which increases the chances that the bug is in nfsd. Switching qemu to use write back paging also helps (aio=threads,discard=unmap,cache=writeback) The client and the server (both 6.5-rc6) work with this configuration. Running the VM on the same machine (also 6.5-rc6) where the VM disk is located (thus avoiding NFS) works as well. I tested several VMs that I have, all are affected in the same way. I run somewhat outdated qemu, but running the latest qemu doesn't make a difference. I use nfs4. I can test patches and provide more info if needed. Best regards, Maxim Levitsky