From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38775E728E0 for ; Fri, 29 Sep 2023 19:53:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232977AbjI2TxA (ORCPT ); Fri, 29 Sep 2023 15:53:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229508AbjI2Tw6 (ORCPT ); Fri, 29 Sep 2023 15:52:58 -0400 Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 999EBB4 for ; Fri, 29 Sep 2023 12:52:56 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 5B7C0320024A; Fri, 29 Sep 2023 15:52:54 -0400 (EDT) Received: from imap51 ([10.202.2.101]) by compute6.internal (MEProxy); Fri, 29 Sep 2023 15:52:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arndb.de; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm1; t= 1696017173; x=1696103573; bh=Xnud7i5mgK2eh+mvhdn41o9zzX9PrdtiIrm +V6iN9Ac=; b=LETLOekAVF6F970qHbuvW17gVSn4E4NspDqbmHAPksHJ5BNT2Xb LhpfDpWBwiR+Ajjp0qexs2NtpACv2ACWbpdoIsOGRtZC1Y/lDrAkil/fiPT+ZHGf nYGVVV/a80owmQKh0lWZnZLHM5ctBNa3XZdccmY9YpW3xYOi/gM6GZyuK1TRBVDm 8uwL2y2L1YKDFWozuyNWvXHijQYlNhYVDkhWiZIlIK9+CVdJRGzggGqMRS0BgfK6 OaNXBxv08HH9nzVt7cmszTD4rh6AQaoZOtJxmOsBCArTWI7M6gVsufPCOq3wS+fn BCU+R8NqmNncsWhf9gE7Suis4WiAbFFkKcA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1696017173; x=1696103573; bh=Xnud7i5mgK2eh+mvhdn41o9zzX9PrdtiIrm +V6iN9Ac=; b=Mq5thkXuxCuBCKOFim/mtAJSIub9suLEpKOAW5FnU6rofegxxfA XbhKFW/7alwyo3OoaVWtCMbYslPoLrGDQAWHlkXt1wWP6kZuLuyJaJalX9fbxuYX WLI3C33Wny7TJYjWRJ0KX3oQCfep/X4PjWkKJfvTUyMo3qaXqMfx5lspwjZsLTj+ +Z83LoETzZv+3iM3ovD1bDfFFXM/ua8Jkn7LwtMSztBy57iEy+rH+s7hSU3dBkYN WktIuivEM3aLqEkh2Uki+iF3Vy36t3eHvU4DCJdCTDb8S7ASoA9CNEbMAiTAJvjv 4KsRjtSskVgnmnbYnS0OX1BUoUC/4QNzCMA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrtdefucetufdoteggodetrfdotffvucfrrh hofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgenuceurghi lhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurh epofgfggfkjghffffhvfevufgtgfesthhqredtreerjeenucfhrhhomhepfdetrhhnugcu uegvrhhgmhgrnhhnfdcuoegrrhhnugesrghrnhgusgdruggvqeenucggtffrrghtthgvrh hnpefgkeeuleegieeghfduudeltdekfeffjeeuleehleefudettddtgfevueeffeeigeen ucffohhmrghinhepkhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenuc frrghrrghmpehmrghilhhfrhhomheprghrnhgusegrrhhnuggsrdguvg X-ME-Proxy: Feedback-ID: i56a14606:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 4952BB60089; Fri, 29 Sep 2023 15:52:52 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-958-g1b1b911df8-fm-20230927.002-g1b1b911d MIME-Version: 1.0 Message-Id: In-Reply-To: References: <20230926175208.9298-1-james.quinlan@broadcom.com> <20230926175208.9298-2-james.quinlan@broadcom.com> <1f08bd12-0ac4-43ea-b058-7836521eec12@app.fastmail.com> Date: Fri, 29 Sep 2023 15:52:17 -0400 From: "Arnd Bergmann" To: "Jim Quinlan" Cc: "Linus Walleij" , "Christoph Hellwig" , bcm-kernel-feedback-list@broadcom.com, jim2101024@gmail.com, "Russell King" , "Geert Uytterhoeven" , "Russell King" , "Andrew Morton" , "Jonathan Corbet" , "Thomas Gleixner" , "Sebastian Reichel" , "Mike Rapoport" , "Eric DeVolder" , "Nathan Chancellor" , "Kirill A. Shutemov" , "Christophe Leroy" , "moderated list:ARM PORT" , "open list" , "Claire Chang" , "Robin Murphy" Subject: Re: [PATCH v1 1/1] ARM: Select DMA_DIRECT_REMAP to fix restricted DMA Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Sep 29, 2023, at 15:24, Jim Quinlan wrote: > On Thu, Sep 28, 2023 at 11:17=E2=80=AFAM Arnd Bergmann = wrote: >> On Thu, Sep 28, 2023, at 10:00, Jim Quinlan wrote: > > Our RC is definitely not coherent with the ARM/ARM64 caches. Ok, thanks for the confirmation. >> It's unlikely but not impossible, as the driver has some >> unusual constructs, using a lot of coherent mappings that >> might otherwise be streaming mappings, and relying on >> dma_sync_single_for_device(..., DMA_BIDIRECTIONAL) for other >> data, but without the corresponding dma_sync_single_for_cpu(). >> If all the testing happens on x86, this might easily lead >> to a bug that only shows up on non-coherent systems but >> is never seen during testing. >> >> If the problem is not the "dma-coherent" property, can you >> double-check if using a different PCIe device works, or narrow >> down which specific buffer you saw get corrupted? > > I've done some testing, below are the results. The new two devices, a > USB controller > and an M2 NVMe stick, behave the same as iwlwifi. > > Note that I'm not advocating that "select DMA_DIRECT_REMAP" is the > anser, I'm just showing that it fixes my examples. Ok, so I think we can stop looking at the device drivers for bugs then. > VER PCI-DEV <--------- RESTRICTED DMA -----= ----> > ARM64 ARM ARM64 ARM ARM+DMA_DIRECT_= REMAP > 5.15 iwlwifi P P P F P > 5.15 nvme P P P F P > 5.15 usb P P P F P > > 6.1 iwlwifi P P P F P > 6.1 nvme P P P F P > 6.1 usb P P P F P > > Upstrm iwlwifi P P F F F > Upstrm nvme P P F F F > Upstrm usb P P F F F > ARM64 ARM ARM64 ARM ARM+DMA_DIRECT_= REMAP > VER PCI-DEV <--------- RESTRICTED DMA -----= ----> > > LEGEND: > P :=3D pass, driver probe and some functional test performed > F :=3D fail, usually when probe is called; impossible to do > functional test > Upstrm :=3D 633b47cb009d "Merge tag 'scsi-fixes' of > git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi" > > iwlwifi :=3D 7260 Wifi 8086:08b1 > nvme :=3D 1e95:1007 > usb :=3D Supahub, 1912:0014 Thanks for the thorough testing, that looks very useful, even though I don't have an answer immediately. Maybe Robin can see something here that I don't. It's particularly interesting how arm64 only started failing on fairly recent kernels, so if nothing else helps you could always try bisecting the history between 6.1 and 633b47cb009d, hoping that the commit that broke it points us to the arm32 problem. The only change I see in that time frame that seems related is 7bd6680b47fa ("Revert "Revert "arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()"""), so you could start by reverting that. However, it's probably something else since this is for the coherent mappings, not the streaming ones. Arnd