From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752970AbdKWONR (ORCPT ); Thu, 23 Nov 2017 09:13:17 -0500 Received: from aserp1040.oracle.com ([141.146.126.69]:23392 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752036AbdKWONP (ORCPT ); Thu, 23 Nov 2017 09:13:15 -0500 Subject: Re: [PATCH v9 4/5] x86/PCI: Enable a 64bit BAR on AMD Family 15h (Models 30h-3fh) Processors v5 To: =?UTF-8?Q?Christian_K=c3=b6nig?= , helgaas@kernel.org, linux-pci@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, amd-gfx@lists.freedesktop.org, xen-devel References: <20171018135821.3248-1-deathsimple@vodafone.de> <20171018135821.3248-5-deathsimple@vodafone.de> <26df0a78-8028-e42c-ce50-4cefe612a7e1@oracle.com> <3443aad0-8c3b-b97e-685a-96b0866827be@amd.com> <7936fdd3-1615-ac1f-7b75-330ccb6a1a3f@amd.com> <0bb468d9-bc1d-e3c5-e313-1cf9408380f0@oracle.com> <275e859a-0ddd-ea7f-a681-67e42a5233fe@amd.com> From: Boris Ostrovsky Message-ID: <38bc76d9-2ed5-61f3-24b3-085031825181@oracle.com> Date: Thu, 23 Nov 2017 09:12:57 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <275e859a-0ddd-ea7f-a681-67e42a5233fe@amd.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Source-IP: aserv0022.oracle.com [141.146.126.234] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/23/2017 03:11 AM, Christian König wrote: > Am 22.11.2017 um 18:27 schrieb Boris Ostrovsky: >> On 11/22/2017 11:54 AM, Christian König wrote: >>> Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky: >>>> On 11/22/2017 05:09 AM, Christian König wrote: >>>>> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >>>>>> On 11/21/2017 08:34 AM, Christian König wrote: >>>>>>> Hi Boris, >>>>>>> >>>>>>> attached are two patches. >>>>>>> >>>>>>> The first one is a trivial fix for the infinite loop issue, it now >>>>>>> correctly aborts the fixup when it can't find address space for the >>>>>>> root window. >>>>>>> >>>>>>> The second is a workaround for your board. It simply checks if there >>>>>>> is exactly one Processor Function to apply this fix on. >>>>>>> >>>>>>> Both are based on linus current master branch. Please test if they >>>>>>> fix >>>>>>> your issue. >>>>>> Yes, they do fix it but that's because the feature is disabled. >>>>>> >>>>>> Do you know what the actual problem was (on Xen)? >>>>> I still haven't understood what you actually did with Xen. >>>>> >>>>> When you used PCI pass through with those devices then you have made a >>>>> major configuration error. >>>>> >>>>> When the problem happened on dom0 then the explanation is most likely >>>>> that some PCI device ended up in the configured space, but the routing >>>>> was only setup correctly on one CPU socket. >>>> The problem is that dom0 can be (and was in my case() booted with less >>>> than full physical memory and so the "rest" of the host memory is not >>>> necessarily reflected in iomem. Your patch then tried to configure that >>>> memory for MMIO and the system hang. >>>> >>>> And so my guess is that this patch will break dom0 on a single-socket >>>> system as well. >>> Oh, thanks! >>> >>> I've thought about that possibility before, but wasn't able to find a >>> system which actually does that. >>> >>> May I ask why the rest of the memory isn't reported to the OS? >> That memory doesn't belong to the OS (dom0), it is owned by the >> hypervisor. >> >>> Sounds like I can't trust Linux resource management and probably need >>> to read the DRAM config to figure things out after all. >> >> My question is whether what you are trying to do should ever be done for >> a guest at all (any guest, not necessarily Xen). > > The issue is probably that I don't know enough about Xen: What exactly > is dom0? My understanding was that dom0 is the hypervisor, but that > seems to be incorrect. > > The issue is that under no circumstances *EVER* a virtualized guest > should have access to the PCI devices marked as "Processor Function" on > AMD platforms. Otherwise it is trivial to break out of the virtualization. > > When dom0 is something like the system domain with all hardware access > then the approach seems legitimate, but then the hypervisor should > report the stolen memory to the OS using the e820 table. > > When the hypervisor doesn't do that and the Linux kernel isn't aware > that there is memory at a given location mapping PCI space there will > obviously crash the hypervisor. > > Possible solutions as far as I can see are either disabling this feature > when we detect that we are a Xen dom0, scanning the DRAM settings to > update Linux resource handling or fixing Xen to report stolen memory to > the dom0 OS as reserved. > > Opinions? You are right, these functions are not exposed to a regular guest. I think for dom0 (which is a special Xen guest, with additional privileges) we may be able to add a reserved e820 region for host memory that is not assigned to dom0. Let me try it on Monday (I am out until then). -boris