From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS, USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADB42C04EB8 for ; Thu, 6 Dec 2018 06:45:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 71226214C1 for ; Thu, 6 Dec 2018 06:45:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1544078717; bh=g6Czx7nLEy93+bDguAtYiatB9x3PPOqHGI/j5b57/Ak=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=QkjCinuOoS0PFPjBgr1VCZR6ibLU+lGX2taoISUSgkQ67ww0AlUQlo0XYSh72HS4i FTkhmTluMfkVlGooS/To5Ys/hK3wIYLLpHr7l4zq3jljlF/1A2JFrZMQAQTlQeCNFD J7U96U+c/EKXKkSnKASAr4wl4jWipclOSQN5ZYbM= DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 71226214C1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729123AbeLFGpP (ORCPT ); Thu, 6 Dec 2018 01:45:15 -0500 Received: from mail.kernel.org ([198.145.29.99]:57192 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728294AbeLFGpP (ORCPT ); Thu, 6 Dec 2018 01:45:15 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3EB3220989; Thu, 6 Dec 2018 06:45:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1544078714; bh=g6Czx7nLEy93+bDguAtYiatB9x3PPOqHGI/j5b57/Ak=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=OJLScHgLgDQ1JZdxw2PmXeliVG9qLciE4iGyalPGKT+MNhI93eh5pXXbcuP2axMx2 ccmKIP3uIzsXH0/L7Q5FdKTNj8uMco4eTGSAqj5/S+b7z7HeKBOTvLQae1yJ13vM4w Jv1XIReSKLcBBTG4dxsiL7CsUOxuhY/+G4wjWdyw= Date: Thu, 6 Dec 2018 08:45:09 +0200 From: Leon Romanovsky To: David Gibson Cc: davem@davemloft.net, saeedm@mellanox.com, ogerlitz@mellanox.com, tariqt@mellanox.com, bhelgaas@google.com, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, netdev@vger.kernel.org, alex.williamson@redhat.com, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, sbest@redhat.com, paulus@samba.org, benh@kernel.crashing.org Subject: Re: [PATCH] PCI: Add no-D3 quirk for Mellanox ConnectX-[45] Message-ID: <20181206064509.GM15544@mtr-leonro.mtl.com> References: <20181206041951.22413-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Re2uCLPLNzqOLVJA" Content-Disposition: inline In-Reply-To: <20181206041951.22413-1-david@gibson.dropbear.id.au> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --Re2uCLPLNzqOLVJA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Dec 06, 2018 at 03:19:51PM +1100, David Gibson wrote: > Mellanox ConnectX-5 IB cards (MT27800) seem to cause a call trace when > unbound from their regular driver and attached to vfio-pci in order to pass > them through to a guest. > > This goes away if the disable_idle_d3 option is used, so it looks like a > problem with the hardware handling D3 state. To fix that more permanently, > use a device quirk to disable D3 state for these devices. > > We do this by renaming the existing quirk_no_ata_d3() more generally and > attaching it to the ConnectX-[45] devices (0x15b3:0x1013). > > Signed-off-by: David Gibson > --- > drivers/pci/quirks.c | 17 +++++++++++------ > 1 file changed, 11 insertions(+), 6 deletions(-) > Hi David, Thank for your patch, I would like to reproduce the calltrace before moving forward, but have trouble to reproduce the original issue. I'm working with vfio-pci and CX-4/5 cards on daily basis, tried manually enter into D3 state now, and it worked for me. Can you please post your full calltrace, and "lspci -s PCI_ID -vv" output? Thanks --Re2uCLPLNzqOLVJA Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJcCMV1AAoJEORje4g2clinjAQP/1QohDmpz3Nx1/0h9KEad/Gv xb/XKSpynLFHroGdEEAHUaDgLOfsp5WoJxVs5sj8/EI7ZB7+jT89DAByeXkFQI5J 8LBm+eEZboNTLa2lOcl5yBSVNWb3POGFmsuTAf2q4F+fijds+lyATk3kKYpDc7wC 4zJ22LiEr4rryTOuxZyFi5YTaNso3WhHs0XB8bAFoJar9ypY+aSPq48C+wU+08da AbD3yThzPT++z61FgK5tmyaG2xN8uZmHzeRpSaX7vEDMgPznJk4qnIaA0FtCnjRu SYOg8XLPksqmrdU/N/vV5FM2uk77s8yoNtOEkdCg19rkG04c2pzqdX/aZtW4epKX h1mKWK2x21U2+l/gd5atRuFuJfP3k7P6LEsfzWprmFsyC6fwrt2H8ojN+10fuEiP 1KURSsGdMYCrKrDklpsAeu5JHRc5UEOKyQsWPQZ+wmZtpI4zJTW9Av9YfYd/RZ6d PefEN2j6gmDpC7ffZqV0oenEuEgGAPGFiBRRTeOqgggfLMT1Rn9FZHX4zAGLZMpR jciOwJNwN9rPTkLyEyCQItjhqQjty3gvDMlYpuMpNsYIXcyl/gkjMWuke4TaHHZP M6g+P6qfUvDEDTlMO+wU6hcKRmqLFr+LsjQhtfHza9j1gGIcYO6FFTvPN0o0tGcq mItAd9DzgosnlmUT2DHQ =Hvye -----END PGP SIGNATURE----- --Re2uCLPLNzqOLVJA--