From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751836AbdKVQyw (ORCPT ); Wed, 22 Nov 2017 11:54:52 -0500 Received: from mail-by2nam03on0068.outbound.protection.outlook.com ([104.47.42.68]:64235 "EHLO NAM03-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751544AbdKVQyu (ORCPT ); Wed, 22 Nov 2017 11:54:50 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Christian.Koenig@amd.com; Subject: Re: [PATCH v9 4/5] x86/PCI: Enable a 64bit BAR on AMD Family 15h (Models 30h-3fh) Processors v5 To: Boris Ostrovsky , helgaas@kernel.org, linux-pci@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, amd-gfx@lists.freedesktop.org, xen-devel References: <20171018135821.3248-1-deathsimple@vodafone.de> <20171018135821.3248-5-deathsimple@vodafone.de> <26df0a78-8028-e42c-ce50-4cefe612a7e1@oracle.com> <3443aad0-8c3b-b97e-685a-96b0866827be@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: <7936fdd3-1615-ac1f-7b75-330ccb6a1a3f@amd.com> Date: Wed, 22 Nov 2017 17:54:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Originating-IP: [2a02:908:1251:7981:5d0:8ffd:957f:10e7] X-ClientProxiedBy: DB3PR0202CA0020.eurprd02.prod.outlook.com (52.134.64.161) To DM5PR12MB1305.namprd12.prod.outlook.com (10.168.237.148) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: af206c55-511f-4a86-8bbe-08d531c9bdbb X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(48565401081)(2017052603199);SRVR:DM5PR12MB1305; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1305;3:yjUUpFUOGim1Dzm1zQYO8OGFr0AHPOFDcE3bJwrEtsV6+Bv2DRC47jzxRUkQcjBzjWzfjfeE8i8jKAcSbkx1gTZCdib+VA3xhBAGUpdFEOtIFaHzQYNnjgq0XegV/oWN1EUVRWq+FkBZzfqoSoYKyNaJUxSU4FaQGhM/jU1HPG/dZGWqsWfb3kYIDvF5YLVLaiGfXaugZEHeD0Jt/UoKnkmbxoohFTC/ijNmY8mN2KDX+CxCKiIha4NcgadbGLm0;25:k/xrpymO6l5uc1M35KwaiBuY1PERvgbBUY7t/3hMrF+5zTK2c0oGEF+bJkFUvnx1kg9lrV0jLmUNACPcFanJPm2GnZkvutbt7/VA64i0dqVptYzguLiOtSpW11B5q0wGWFMTVYK9v9u0umeagj35JBu26PSMj5HoKIN5A2ZcehqP5ft2DWOGIAAjbYk8XPff4BTJdsNIagRxSdhLEOKqHrBEc9ss3Nqr6EjXSmJsbgFsoxIpnhQnhr49g09YOPr9CS4/pEDtjZkKixFuq+gMvp620IMyUJV+P5Mvb3jRLIh5GBpZqK/EwFJBWYC55gGewspgfJ9bRpTs4zMIEOZWog==;31:GK91zslu9/BLSvBiH/eGHGmrIyPu4Sm2xl18khjg04S0qRva25ys8nTIBBgpq5hcrnej6RifaLj0Hiz5R2UJHWTey/pii31iIfZLyP96Xo8ztmhLksg6uezNo3cREqPhCj2NdKHtJiU870NuQXHBNdF5AlJ08PdcGDrihz8VkoZTtxSYbB16c8cXfndoeGcB0phT/577AZq9sYLGjgh/vYLwakDaC8mgMnTBz4dNAEk= X-MS-TrafficTypeDiagnostic: DM5PR12MB1305: X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1305;20:r4u2rTdiffT8wunZeiKQTq7lzPRCOcirZL+DgEM2ofqHav4sgfHm/OIlNIz0iH5I6qmh4mswzlGEyk//Z6wajZsDNXfMJW1rpNfPZZstFtSD5ezss4jP1ewh7UmNXOZifvmqqzyXF0sLPDAZovaketuETQw96Qu5cuo6UuoLmRPmzpUQuLNWI8njnAv/dhZAGLQHO31Lb7z1Nki0lZf3LOBjMF5zWknYDbPlHR1OgzYvA2aQAphnINDg6xLTbltYzSGPl7RIts/yBj774RxC4Vjt7YzXAbofy/YYC2JXHl9AL97cyIHCaON1DiSKD4YrX2mrBeTC0LJ5jRiMlAHKmOtTgMy1XLq19aYA90TqA7WHjJdj3hOUn94eUuNF2i6X7DrQEUJClRUbgSQVpP6s97opQ3Fhvay/Ej3MBA3MTCx4NnvOSSI7gE5lsX49EBTzbZTxxDPMtybfVZno3BcsJ6rBPhE+Ev+nw9FPk8ar8FyeM9tl+V+GVUmnxv5/Su2l;4:U727Nj3XG24XPUG2GSx5GZUzcJ2zueUvhFWvC0Skc3S1OQpAe6/W/uNcTvtqrMzJLe7UhMH+6YJCAIFI4sFncUTNPWc0aLUhdXdZGojwRb+HfHZVnLB/CANO4SB60I9sqdT36e/TtOL1G7zDPL2FyqKavFGpeaW1u6VlYDLyPuvMiVGGNVB0OzJwkbjWpwX6Lwkzebsiy3fmDjPMv+iChQbUM3izy7EBrnz1oM7/6Frt9WItAGYbERKDLRPvKlUr+38mE1qmrs/RjcHHi8QYOQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(3231022)(3002001)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(6055026)(6041248)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123555025)(20161123564025)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:DM5PR12MB1305;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:DM5PR12MB1305; X-Forefront-PRVS: 0499DAF22A X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6009001)(39860400002)(376002)(346002)(51914003)(24454002)(199003)(189002)(25786009)(6116002)(72206003)(97736004)(31686004)(50466002)(5660300001)(305945005)(65826007)(6666003)(65956001)(52146003)(93886005)(52396003)(65806001)(68736007)(53546010)(36756003)(110136005)(478600001)(47776003)(31696002)(2950100002)(86362001)(8936002)(23676004)(1706002)(316002)(58126008)(6246003)(54356999)(105586002)(106356001)(2870700001)(67846002)(101416001)(6486002)(53936002)(33646002)(50986999)(83506002)(229853002)(7736002)(81166006)(81156014)(2906002)(189998001)(2486003)(5890100001)(8676002)(76176999)(64126003);DIR:OUT;SFP:1101;SCL:1;SRVR:DM5PR12MB1305;H:[IPv6:2a02:908:1251:7981:5d0:8ffd:957f:10e7];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjEyTUIxMzA1OzIzOkZMaGFiK0owMFJCZXNMMms3SGFBUVBFMC9z?= =?utf-8?B?MHdYeVhWRzlyd2kwQjVVdTREQk9RZjZ2WFllT2hUS3E4VUl5SzRTWkFLMDV6?= =?utf-8?B?Z2w3NnBiNEtmbHNWMDhrM1lrNXdwNXFzNHNTVUNUd3Qrdk5tc21uM3JRdnZM?= =?utf-8?B?OXFiSytiZ2NqdU5xQi9aeWlub24vU3B0NkduYXFmZGk5K2xjS2JpMC8vSFhj?= =?utf-8?B?T3I4Tm1zK0gvbmxXN2lmT2o4VUNya2pTMmtmbzRJeFhKSFlPUm0yQ1dYMUlF?= =?utf-8?B?eE1DckZ3MGtuYmVLWVYwODdzNVJ1TWFRV2ozblcyREtxRm03V0k5NC91NEFY?= =?utf-8?B?VDZ3RTR5RkdnS0FtOEVONXpyYzZCaERoQTNPRUIzOXEzTXVzc04rV1MxTSsr?= =?utf-8?B?MXRXSkQrVTgwdCtPMVlnQmNhTERhakJWZmw4UWJOWFNLeEJGbWt2dERGUG0z?= =?utf-8?B?cjk1dFRDUXBHZlJQdjZxNWVDRXlZb1dHY2x3RUoxbm1GMHIxSnZQSjFtYysv?= =?utf-8?B?SFdma3dLQWc3NGNwU1UrWXgzUFRqemtHSmVMMGZ6UlVVSHU1YW1qK0JnMm9a?= =?utf-8?B?SzRkNkxxMjhSY1BvMXhyTHhKTGlyU0J5VmowZHRyWWRUSXZTeVJ5UlZXR3Ja?= =?utf-8?B?ZmhOdy9NS081V3Z5R0FuOVhxMGFpVWU3RnAyVk9HaDNFMnZMWEllbXBWdGNP?= =?utf-8?B?ek5sdGdBUmNUektNd3dxVHUvVWxhTjJwOFp2ZTJKelhsWHZ6OHpxM2pPWHhp?= =?utf-8?B?aTBmTTRPN0t3bGZOZzRRVlhiWG04bllTL1VCai92UWcxb0tZYW1zVnU0OHkw?= =?utf-8?B?ODRLQ3pvSzlzQ21iK2ZsbVFZZVFzZzlDYjFDSm5VemRXbmJrZyt3UGJMZnlZ?= =?utf-8?B?TGJzK01sNkI0bm90anNuMlZPR1JmWWdPM2lpdUZydjBzNVlHYjBzZUREaXg3?= =?utf-8?B?d094cWdWUkZDajBZcVFEc3F4bGZLMGJueHRmR1ZMS0IrbzZzSGM3VkpEYTNu?= =?utf-8?B?cndEc05HVU92MXgrV1NYenNtdjk0RE9nTVdQRzdlS2RMSmMzT3BjTlg0TmNE?= =?utf-8?B?cGs3and1R2kwMXRKRzZRUXU4SnY4TEJxQ0x1RGdNSXVaTTdtcFQxaE96NlRE?= =?utf-8?B?cS9TSUNxTGx6d3dSMlNyWVpWalBXMy82ZFFiYURMV29ZWGtkN1R1L2QyR1lB?= =?utf-8?B?NUFvS2FNNjZ5dHdLVlhlYUl2UndsWGt2NmVJclFiNy9hZkg0eTQ2MFFvQ2gy?= =?utf-8?B?SnI3VkdZcUxyRFR5TUIzZG1yWFAzWFZSRHFlU2JkYXFzTG9jR2lzb3p5VXQ0?= =?utf-8?B?b3cyektqc3ZMUEVOd2FPb3FtaUJ3ZmhwZ00xU0ZpVHV5Vkx1SjVyMjJXK3pr?= =?utf-8?B?MTlNMllKQ2hnK256b2FzbXBFM2JCa2FPaXpIdUc5bHI1QkpjS1ZaaUNJeTNP?= =?utf-8?B?MG9HTVFaaUFCWmIzQkliaWJEUDFpWE5EQ3hBMUk0eE9keGJ0NC9NTzl5cGJk?= =?utf-8?B?MUtncFk4bzdvWDgvS3RTQXlxUzFQam83OGV0Rk9SbEIvc2l2WkVSL3NQSWhH?= =?utf-8?B?QzFReXdmS25KM1lMSEJ6c1laNTlkblFCSDdtc2FIK3NGMXltT2Y1R0s4Rm15?= =?utf-8?B?d3FBd0FsYVRzZWNueWZaWFcrSi9RMG1RQ0xUNGY3YWczZm9kNEZHV2VTb3Za?= =?utf-8?B?eFlrTXl2UDVJVGFyUDdwZW9UTnl2QitzSG1kMisxaGhBZmV6U0J2SzR1MVEz?= =?utf-8?B?b2lZRUNpZDZZUEZpNG9LcTZxZWdCMWgvdVZEK0Q2SVlmTzV6VnpudFNGZVFL?= =?utf-8?B?VmgrUFVzNktxZUMycXpDektYalcwcnVwWnFyRjNRMWZ4bFlGVFZJM2hNWFE1?= =?utf-8?Q?IKZQmfGACHsATVNbRM1UP6zKnAqACR/2?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1305;6:sinIiESjkhIQRffo3o6MofzARo9ZaYy4u/7Od6NAt+eNgxQeOdOHfocI967rZtdOS0/2JC26FbP+i6tfIn3BkaW0zP1395QSfYyZnrkGrnildqgYbABGDaCR8QSG4HUJWCN2HxZvCrYMdQ1APm3oK/FtAg5q2XCbReHneYKQdE5FoVQSXM7MtID9ja9gsGdeOAtSNFSh7h94GE/liGt+HC8QeFLi+BnRSDN/YodnEVv6HK48W2krsilSEZGUq6BtB9ScmVU4jUmQXs75mzojjA6kejJyv3Cb6mlUtod0ChQXu/OT089w8JWVBz+Hhhm25sV69g3N3vHheuLQAWDPSPh1RDqC5N1eGVoXDDN55xI=;5:TQ1uUYc9VnCOsbEH57tFLL9AyVQnvotXLZujcJtSeqIdmuDkuHSia33MBtj6lW9pQk7ztt93D4lmPopjHTH4oVtAYbZNEN6cX4skHq6x0zf4PwL+/WcWTnOK9E6tVpFVBQMetyL8H4muLjjMVCRRrAwyu3jFCWdGBDGfYH23kyo=;24:RRCwLfwqF+Xj0DG0zxwYwLFkLOR2KmOxnuFa+9jin2/Qq+MkrF6SBUTyOkXLvs3WF4MbrNHZPbjUQ6qAM7AVQizPAKeESB3HG7AKRcjnpZ4=;7:iPPR7YIW0X5UKp3n6vbi50AJCA2MKWF/aUl33GUVIdNYXe+pPu71Ijh0ypISwvDBTHn/n8tppmhp7bFKAYa6hYj+6EkVGWgBGPAh8t2RzaubCpI4m8qI+wJwQ9mzWQsqvfCTfgGOjhkuv/Da2W+upzCFUQ2eWyZmAVs8J4fm3zJGTuBkdztgBRwBFdXyR0bv9ssQH6x78nIa4mq+CXCcUL2sAtlfdK+sk/Ov02aTidUJgdSaoVyMlg+atemCvP8m SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1305;20:89F9XlAlmerGLr/yYdxCCA4aseWPQBGgEM+FoEIn0B2o+RXXDeMNo1zpRJNIP98jKT8h3LrzQb3HCW16Pu1mnPBM84MW8Zky1iVw8Pzu22XLCwQJPC3vSnfgw722kDN+PrIo3pBCPCkNG49aUsUZ9U0f0lS1icd+tFusc8L5Z0QDCRTk8A8IVkN1mr8mRTzpXRP0Yk2Q3AgBb4XlVe+HlUr/phk4+ArRw2qH1sBDyqhbYIUKQpVsVg5SmAm0uWsE X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2017 16:54:47.0884 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: af206c55-511f-4a86-8bbe-08d531c9bdbb X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1305 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky: > On 11/22/2017 05:09 AM, Christian König wrote: >> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >>> On 11/21/2017 08:34 AM, Christian König wrote: >>>> Hi Boris, >>>> >>>> attached are two patches. >>>> >>>> The first one is a trivial fix for the infinite loop issue, it now >>>> correctly aborts the fixup when it can't find address space for the >>>> root window. >>>> >>>> The second is a workaround for your board. It simply checks if there >>>> is exactly one Processor Function to apply this fix on. >>>> >>>> Both are based on linus current master branch. Please test if they fix >>>> your issue. >>> Yes, they do fix it but that's because the feature is disabled. >>> >>> Do you know what the actual problem was (on Xen)? >> I still haven't understood what you actually did with Xen. >> >> When you used PCI pass through with those devices then you have made a >> major configuration error. >> >> When the problem happened on dom0 then the explanation is most likely >> that some PCI device ended up in the configured space, but the routing >> was only setup correctly on one CPU socket. > The problem is that dom0 can be (and was in my case() booted with less > than full physical memory and so the "rest" of the host memory is not > necessarily reflected in iomem. Your patch then tried to configure that > memory for MMIO and the system hang. > > And so my guess is that this patch will break dom0 on a single-socket > system as well. Oh, thanks! I've thought about that possibility before, but wasn't able to find a system which actually does that. May I ask why the rest of the memory isn't reported to the OS? Sounds like I can't trust Linux resource management and probably need to read the DRAM config to figure things out after all. Thanks a lot for this information, Christian. > > -boris > >> Regards, >> Christian. >> >>> Thanks. >>> -boris >>> >>>> Thanks for the help, >>>> Christian. >>>> >>>> Am 20.11.2017 um 17:33 schrieb Boris Ostrovsky: >>>>> On 11/20/2017 11:07 AM, Christian König wrote: >>>>>> Am 20.11.2017 um 16:51 schrieb Boris Ostrovsky: >>>>>>> (and then it breaks differently as a Xen guest --- we hung on the >>>>>>> last >>>>>>> pci_read_config_dword(), I haven't looked at this at all yet) >>>>>> Hui? How does this fix applies to a Xen guest in the first place? >>>>>> >>>>>> Please provide the output of "lspci -nn" and explain further what is >>>>>> your config with Xen. >>>>>> >>>>>> >>>>> This is dom0. >>>>> >>>>> -bash-4.1# lspci -nn >>>>> 00:00.0 Host bridge [0600]: ATI Technologies Inc RD890 Northbridge >>>>> only >>>>> dual slot (2x16) PCI-e GFX Hydra part [1002:5a10] (rev 02) >>>>> 00:00.2 Generic system peripheral [0806]: ATI Technologies Inc Device >>>>> [1002:5a23] >>>>> 00:0d.0 PCI bridge [0604]: ATI Technologies Inc RD890 PCI to PCI >>>>> bridge >>>>> (external gfx1 port B) [1002:5a1e] >>>>> 00:11.0 SATA controller [0106]: ATI Technologies Inc SB700/SB800 SATA >>>>> Controller [AHCI mode] [1002:4391] >>>>> 00:12.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> OHCI0 Controller [1002:4397] >>>>> 00:12.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>>>> Controller [1002:4398] >>>>> 00:12.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> EHCI >>>>> Controller [1002:4396] >>>>> 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> OHCI0 Controller [1002:4397] >>>>> 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>>>> Controller [1002:4398] >>>>> 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> EHCI >>>>> Controller [1002:4396] >>>>> 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller >>>>> [1002:4385] (rev 3d) >>>>> 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB700/SB800 LPC host >>>>> controller [1002:439d] >>>>> 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI >>>>> Bridge >>>>> [1002:4384] >>>>> 00:14.5 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> OHCI2 Controller [1002:4399] >>>>> 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1600] >>>>> 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1601] >>>>> 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1602] >>>>> 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1603] >>>>> 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1604] >>>>> 00:18.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1605] >>>>> 00:19.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1600] >>>>> 00:19.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1601] >>>>> 00:19.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1602] >>>>> 00:19.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1603] >>>>> 00:19.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1604] >>>>> 00:19.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1605] >>>>> 01:04.0 VGA compatible controller [0300]: Matrox Graphics, Inc. MGA >>>>> G200eW WPCM450 [102b:0532] (rev 0a) >>>>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>>>> Network Connection [8086:10c9] (rev 01) >>>>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>>>> Network Connection [8086:10c9] (rev 01) >>>>> -bash-4.1# >>>>> >>>>> >>>>> -boris