From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758844AbbIVQ5H (ORCPT ); Tue, 22 Sep 2015 12:57:07 -0400 Received: from mail-bn1on0069.outbound.protection.outlook.com ([157.56.110.69]:24379 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752991AbbIVQ5A (ORCPT ); Tue, 22 Sep 2015 12:57:00 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=David.Daney@caviumnetworks.com; Message-ID: <56018854.1030106@caviumnetworks.com> Date: Tue, 22 Sep 2015 09:56:52 -0700 From: David Daney User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130625 Thunderbird/17.0.7 MIME-Version: 1.0 To: Lorenzo Pieralisi CC: David Daney , "linux-kernel@vger.kernel.org" , Bjorn Helgaas , "linux-pci@vger.kernel.org" , Will Deacon , Rob Herring , Pawel Moll , Mark Rutland , Ian Campbell , Kumar Gala , "linux-arm-kernel@lists.infradead.org" , "devicetree@vger.kernel.org" , Marc Zyngier , David Daney Subject: Re: [PATCH 3/3] PCI: generic: Add support for Cavium ThunderX PCIe root complexes. References: <1442529694-1792-1-git-send-email-ddaney.cavm@gmail.com> <1442529694-1792-4-git-send-email-ddaney.cavm@gmail.com> <20150922160549.GE17444@red-moon> <56017E39.8060606@caviumnetworks.com> <20150922164012.GA3692@red-moon> In-Reply-To: <20150922164012.GA3692@red-moon> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [64.2.3.194] X-ClientProxiedBy: SN1PR07CA0022.namprd07.prod.outlook.com (25.162.170.160) To BLUPR0701MB1714.namprd07.prod.outlook.com (25.163.85.140) X-Microsoft-Exchange-Diagnostics: 1;BLUPR0701MB1714;2:1zQ0QpDYm1qXcc8bI8oIjQ494qHs73GxBjrs5j13hz+/VReYeiIXSbtDrX3tsBjz6tqVX0p9z3AXiM+wx1NsbiGuNKmA7beU7PKZ5tlljgIsnvtOxYNo+ekxACZzNukB3iHpleAGFUcJ+pDZzd6y5yn/+Ln4GAxycXSVEiQQq5M=;3:kYxNmSo87IRbw4gbXlfc4IFbsAs1WU50O5s3xtNvu0nHiTn/P6I5rFTcsWYtWQVl4qfmz66/M2F0nN1ekgkOPIOBUamHdH+X8KyJzI9ul6wmy6CAqy9giR/1ijLO7EiUN0JMwtu+9Crusa0IX2yH6sPrSFWuPyYlreyk0YLnApX2sDu/wuQ3NtHUAef6PgHA;25:gGLU3u7FbuItoBk59b+JCd/D4i1P752cyMvFTNR+uPmdWXquxsPNLz+cFNww/ql2MwmgLHOd0NpMYG57WT5OgI+j0ZSVHR+LQpycaWphVrjJMJksdNWfoyDLqerDBSzg5zmnSrJqi0GdV59ZOZKstNHTO4pl6sn9O29/4cLtvV3B64K4P6xCDVjjWMS+y+g2n1aMRTM3XYIJrRgW941kYpoJkEBcXsncUp5ysFIIXJ3wSJUavt7OnS6LVmSE/3MY X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(42134001)(42139001);SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1;BLUPR0701MB1714;20:gXnaECi4+79fE02uvj6hhdkuqenL4vwEG1IlG0aERmyaK72aaGCVLohV6E4TIqSHBvKWR1/REZcd9dy/3CS4afwR441tONJkh9CV6RfJb7Es4KSSawpHipMdEtXSHJIGoZYrZ8xBik8nFhkcFapH0fWX1XfVBxv+qruPVFdibALySD3+WASNuTd8N3SrNUdZtsfTN5FnRjVJ/2WSgzQXU/ISruluu8LP6tm2fb3SetKiM1NrKjTLT5WfnHkZ58BPNN1kx+BenOrvyAe/UAXO80Dnpxl4dcFvH7XzUq1II0t9aD07UCuSK0Z2RIsZc9erwf5SVitujBcvB0hRty7kIid9p0+2geM1MsjAF+Thk8hP+9Mmu8dSrOnfq6pynpS9C1aozGO1c02AFRyuiMUy4qWUHkEQdDEf5SrlGlKfsJIACG7Pv+/Pevc/JbpPqxO7Wx1m+x/ET3FJRWvr+a2N5EiT9drjj61r38feViKEJsV42cx8yEQpn2+ev8DCicoFnROsrHZRw7hsQM07y0JUoUvfD+3x4m86aOvXoTjf6UC2lhxKDOvlqjnLRL++ivCmfZ3ufbGn0o0s2Yw7S5bB34nbQHEeREKm+pwR+qOdGPY= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001);SRVR:BLUPR0701MB1714;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1;BLUPR0701MB1714;4:FIMNFmNUVP9PSnyU1kmxyJtI6MQ1nHdzkb62gjItmllhGMbdmqu3aXDgtGjBUd+d/VJ3Z99PwYK7Y0A9wfG2X2WllpwbdWkrlReQp4GEflIEUjSWVYknhTKD8ki63cXBPwiE8sbCiDSzn2XU5rDZx5gXPyOgkJBIxiFRcR5nhI1gC9LU9p2Oxnz6I4O0anWqXdHSeMXNvQSBY1OS7p/tyyiNloOPj9Osz/2iLiP+Ydrx9GA2qp6bjAJC/BFnt+9/v6pKHRxnscXvI+tnSIeiXTKYCbnITe43uEWFEgjSeyrkmKnF3T0rxuYrmxDr5wmUBM24sYcVU2XZeed5WYOgyXOSU5zmnTRwo6FSwpyqGow= X-Forefront-PRVS: 0707248B64 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6009001)(377454003)(24454002)(189002)(199003)(479174004)(164054003)(69596002)(4001540100001)(77156002)(81156007)(23756003)(64706001)(97736004)(101416001)(122386002)(62966003)(65816999)(105586002)(76176999)(68736005)(15975445007)(46102003)(117636001)(5001960100002)(106356001)(77096005)(50986999)(36756003)(87266999)(87976001)(65806001)(42186005)(5001860100001)(2950100001)(189998001)(5004730100002)(93886004)(64126003)(65956001)(5007970100001)(110136002)(53416004)(99136001)(83506001)(54356999)(66066001)(59896002)(4001350100001)(5001830100001)(19580405001)(92566002)(19580395003)(50466002)(47776003)(40100003)(41533002)(62816006);DIR:OUT;SFP:1101;SCL:1;SRVR:BLUPR0701MB1714;H:dl.caveonetworks.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;BLUPR0701MB1714;23:ioi9Uf3EqAEYUOV4q20HCuBj27bYu2abjtoQU?= =?iso-8859-1?Q?r2ITOZHVxsA94enSWymEdT9TqZNrXozuCrXGsCAeWJ5odOfZZ6PxPFhEtW?= =?iso-8859-1?Q?elWAaYM708CIFqpUwQl7CcPNWxxQnJNU0M7sA2gW8X+V5qpPPU/AYv/qvp?= =?iso-8859-1?Q?ZpyTbTS4HfqKJueZX5+YKjFJflno82x7Q+ejWblViEScXIBvV/ViCMX5xb?= =?iso-8859-1?Q?4HTmzKRqp+dqQNiFGGLPZHLEEz/L3S12NyoHGkfwjAjiR6hW07DtX1a95k?= =?iso-8859-1?Q?cDsvK3fYTSteijzTygfPsJWgdlT4CfcidNGstGOEMY3sF1Ns+NOjBqTXMa?= =?iso-8859-1?Q?7Eitufu/Vcku/YrwMRIo9RBIq5yjXJKbdokR7kRQTA6eEWTS+CJFJNz6bn?= =?iso-8859-1?Q?bLJ2nMUyvi6J1nCgGrOHfTV/NuDeoBegxRpySKAb139s1F/99dPHSeLd3I?= =?iso-8859-1?Q?nDpY8w0fPuMnfM5xHQhHr1NvFEkuX+wHFgjYNEz+BhoOCQ5LCQcaMEsItu?= =?iso-8859-1?Q?1LcnfU5ogbMPsuInFER6lgUWATqDLN1KNdGvnpwmwOBIYmlN+WGQoHKYXF?= =?iso-8859-1?Q?RhJ5ontD1E343NoWAoKEi/poambngeqheHGDld4Chx4NLEjZ+LQbiArrP/?= =?iso-8859-1?Q?NXwKRK2ehE5VUwLxHlten49HGA3zSYwNpO6bayIPcO+N5v80AlOoAmfKV6?= =?iso-8859-1?Q?/CNO//Y8o9r6uuIyAHe8rZrV1sGZQO9qkA+yTwCYyDdLe16Ef7i7HSjpyC?= =?iso-8859-1?Q?PGnZn9/x8QUgFYzJtyqIjksOyBxjhXoRi4vT/ZxwqlP8FbfFikYBCmijrw?= =?iso-8859-1?Q?Au7m9X/vVmgOcTqlc+fs9AUT/GNszBd+2xeUrsSkV+oSLNLtFluQEX1dW9?= =?iso-8859-1?Q?yDCHTdRCCMu35tHzedSum5Hgi0SjxLX6x72/fll3IS+g9D04wi8qAHLMFF?= =?iso-8859-1?Q?G56QKP8gdGT/IOQ9x5LqI9+XVB1QjHB2lD4AYPq5tWvSKxf8DXOnBTobbT?= =?iso-8859-1?Q?m6wVHTH9jQt0zhu01DELxc/xRRUJDrfFForZ4lM8mh3DyWdZZM/999gP9c?= =?iso-8859-1?Q?uw3mRtuXgyt0AlSs1KHXb4WJX+2O3+C00ZG6py64lr/dOiJrsLVENqFeoU?= =?iso-8859-1?Q?89uHPTDpQD2rBwAeGy+6WS6b3rTZ/73Mnv08fC+2ubfTBNbcv0Nx2VwDt3?= =?iso-8859-1?Q?GRlDdhmJ+QoJutI/z/SfH0KXX4/C463QZCXdKX8NGoju7GWqedk2+NeKfr?= =?iso-8859-1?Q?YodHanYFFR0FH2kTx1rOi+mkqodyOzJmsUZLaOyOKjf/H9ajvI7n6EWgVO?= =?iso-8859-1?Q?glhepDu+zcLy+cAr3ASyAznQTuAm96i6XCbuagmawMLttp+CXVYs7Z2pp1?= =?iso-8859-1?Q?viRHRUjpWqSPp06NyoGUIyBc7YgSeZ6RXogANppOB9X73sXA5K3FzYS9K4?= =?iso-8859-1?Q?gBi4lB+9BI8iOPNoE13anx08J2UzQHARl3PHyN4Yhj5wPTB8FHtBqiLztp?= =?iso-8859-1?Q?i346NKdEU9Dru9qNq5T+/+T58/1nKf509rIuxzAJpWraV32xza8wWtY3Jo?= =?iso-8859-1?Q?V5BpUQ4gHbH1xefOTV147WobAp5o=3D?= X-Microsoft-Exchange-Diagnostics: 1;BLUPR0701MB1714;5:pIHx7EFflqb9/K5ff6fbssyg6THUOeo7K6HvsbbTljzDSzpI43KdNS1K5TYNfDExCfmu5Xc3rZNj4b6J2yJXD+JrLtEei1zOTjHdpM3ZIyGywBf5SlVPb7W18HrzVNP3MCAKGA27YfWwIRWVE+dtzg==;24:O8+bC0KJ83Hs4ZX+h6JhYwRw8oFOmcSYEnXOzpAymnVvGkXCHNGHbMDoSBCSzmbYU4zTmjexVXCKJjU3u6mcOiAId/m4TA5G7OuvYa9kwl8=;20:AvzOKyyhIygop1SIzaTU7F56Z9Jzxg10rNVt4OGkWnl57ojpRr6doJA2bPsUqLBhzIif0a6oDqQ4m1gRSljVSg== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2015 16:56:55.5097 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0701MB1714 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 09/22/2015 09:40 AM, Lorenzo Pieralisi wrote: > On Tue, Sep 22, 2015 at 05:13:45PM +0100, David Daney wrote: >> On 09/22/2015 09:05 AM, Lorenzo Pieralisi wrote: >>> On Thu, Sep 17, 2015 at 11:41:34PM +0100, David Daney wrote: > > [...] > >>>> Properties of the host controller node: >>>> >>>> -- compatible : Must be "pci-host-cam-generic" or "pci-host-ecam-generic" >>>> - depending on the layout of configuration space (CAM vs >>>> - ECAM respectively). >>>> +- compatible : One of the following with bus:devfn:reg mapped to the >>>> + PCI config space address window in the bit positions shown: >>>> + "pci-host-cam-generic" -- 'CAM' bits 16:8:0 >>>> + "pci-host-ecam-generic" -- 'ECAM' bits 20:12:0 >>>> + "cavium,pci-host-thunder-pem" -- bits 24:16:0 >>> >>> To me that's ECAM left shifted by 4. Is not it just a matter of defining >>> config space base address (ie reg property) differently ? >> >> No. >> >> That's like saying ECAM is just CAM shifted by 4, can't we just specify >> the "reg" differently. > > Ok, sorry I was not clear, apologies. I meant to ask if it is something > that can be configured in your platform and apparently it is not, No, the hardware cannot be adjusted. > I worded it wrongly. OK, I guess I also misunderstood what you were suggesting. > >> The "reg" property describes the base of the config space access window. >> CAM/ECAM/cavium,pci-host-thunder-pem describe how to generate offsets >> from that base, The offset calculation requires information that is not >> present in the "reg" property, it is enumerated in the various >> "compatible" values listed above. > > Agreed, this also forces you to map in more virtual address space for > a given bus, right ? Yes. It takes a full 32-bits of address space to have access to all PCI buses on the root complex. > I am also a bit worried about how we can make this > work with ACPI, where MCFG accessors take the bus and devfn shifts for > granted (ie ECAM specs), so I asked. If ACPI is not flexible enough to describe actual hardware, it will either have to be enhanced, or not used. > > Thanks, > Lorenzo > >> >> David Daney >> >>> >>> Lorenzo >>> >>>> >>>> - device_type : Must be "pci". >>>> >>>> diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c >>>> index e364232..e1d8d5b 100644 >>>> --- a/drivers/pci/host/pci-host-generic.c >>>> +++ b/drivers/pci/host/pci-host-generic.c >>>> @@ -91,6 +91,32 @@ static struct gen_pci_cfg_bus_ops gen_pci_cfg_ecam_bus_ops = { >>>> } >>>> }; >>>> >>>> +static void __iomem *gen_pci_map_cfg_bus_thunder_pem(struct pci_bus *bus, >>>> + unsigned int devfn, >>>> + int where) >>>> +{ >>>> + struct gen_pci *pci = bus->sysdata; >>>> + resource_size_t idx = bus->number - pci->cfg.bus_range->start; >>>> + >>>> + /* >>>> + * Thunder PEM is a PCIe RC, but without a root bridge. On >>>> + * the primary bus, ignore accesses for devices other than >>>> + * the first device. >>>> + */ >>>> + if (idx == 0 && (devfn & ~7u)) >>>> + return NULL; >>>> + return pci->cfg.win[idx] + ((devfn << 16) | where); >>>> +} >>>> + >>>> +static struct gen_pci_cfg_bus_ops gen_pci_cfg_thunder_pem_bus_ops = { >>>> + .bus_shift = 24, >>>> + .ops = { >>>> + .map_bus = gen_pci_map_cfg_bus_thunder_pem, >>>> + .read = pci_generic_config_read, >>>> + .write = pci_generic_config_write, >>>> + } >>>> +}; >>>> + >>>> static const struct of_device_id gen_pci_of_match[] = { >>>> { .compatible = "pci-host-cam-generic", >>>> .data = &gen_pci_cfg_cam_bus_ops }, >>>> @@ -98,6 +124,9 @@ static const struct of_device_id gen_pci_of_match[] = { >>>> { .compatible = "pci-host-ecam-generic", >>>> .data = &gen_pci_cfg_ecam_bus_ops }, >>>> >>>> + { .compatible = "cavium,pci-host-thunder-pem", >>>> + .data = &gen_pci_cfg_thunder_pem_bus_ops }, >>>> + >>>> { }, >>>> }; >>>> MODULE_DEVICE_TABLE(of, gen_pci_of_match); >>>> -- >>>> 1.9.1 >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>