From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751440AbdBWRnv (ORCPT ); Thu, 23 Feb 2017 12:43:51 -0500 Received: from mail-cys01nam02on0062.outbound.protection.outlook.com ([104.47.37.62]:15091 "EHLO NAM02-CY1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751098AbdBWRnt (ORCPT ); Thu, 23 Feb 2017 12:43:49 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Suravee.Suthikulpanit@amd.com; Subject: Re: [PATCH v8 9/9] perf/amd/iommu: Enable support for multiple IOMMUs To: Peter Zijlstra References: <1484551416-5440-1-git-send-email-Suravee.Suthikulpanit@amd.com> <1484551416-5440-10-git-send-email-Suravee.Suthikulpanit@amd.com> <20170125094653.GO6515@twins.programming.kicks-ass.net> <20170214123149.GV6515@twins.programming.kicks-ass.net> CC: , , , , From: Suravee Suthikulpanit Message-ID: Date: Fri, 24 Feb 2017 00:43:19 +0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <20170214123149.GV6515@twins.programming.kicks-ass.net> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [110.169.70.255] X-ClientProxiedBy: KL1PR02CA0003.apcprd02.prod.outlook.com (10.165.15.13) To MWHPR12MB1455.namprd12.prod.outlook.com (10.172.55.136) X-MS-Office365-Filtering-Correlation-Id: 7adce594-91bd-4a60-a2cf-08d45c13849c X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081);SRVR:MWHPR12MB1455; X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1455;3:IToIqAK2kY29gR8lZa0ZEvxLSLZ3YS5tyZ8WGbtKm2IMyRIDJB71LPFVUWAF7PS8E5Un/T6jK5o04u4NBv+FfwP3ZFrNGH6yVTxTFZfrIEbVmHo9WXysoflc9KXDvfdSP69eCgC52ErU4g3UgVD7pb4l2TbGt4wI57ZWkrmhpdeokzN9bt9wVPhhqN8Fj9HK3ZISKTfazaQImInlPZU96RTHO6eeqnJ6M7UNhylrQlW1bqjZvsw7QSlMoNgBl5+RtYu5zjWskhIbPKAcelZp/4l6vdRMf9wgGQ0b4uIOvA0=;25:STQfPCOuXIM9UV3dTz0LES6m6TI0oPT4+DNg1mBXLHb5sbz6nNt+7A8AAGjX5XcLKXmenDsv91VZaUN7+B8EDtlW92HPWJeuG5/RXbubnwoSKBx7id1fOiW5GmmEUa+Apo5PWN5GC90tty9dREPQlVrY8bxXerBzbBobfuCW+y1zN9Vx7J1GNsusLjGPzTPVnH63VJzBydUhnPr1UHV2j6tzs+nr46UPUzIzGKKwl5V7Y61G2/wyMkArcRLMzeneErfMYVqe5AXAGc04wdzukla6uWNKBKQe6APnNiQiKK4VGskUMPmnBs4MdW/uRVNUCY0hNsfd+m2h9LmOTPIzWW3cgqKqW4uXFRjsYjHXiR4ckqD7JCOXtDTEwL6nXGLtIZRzyg/4aa/8FYNIhrDYCnLxmABRPvAc3rkFKLqobgsEHhEUNmKd62ZT0ILob3YvsKSbKBjubfhswompfwiN3Q== X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1455;31:nqM0aJ2Lv6w66w95iYJi9T52x9dfUCvkB1evIm4onkO5cWLyc2Brl8XQxTYHCw7eQHNT9UXtd/I1++7rTzoN/mzS0p3H8x+OWXotny9rJ4F1B9RSR9P7KGHaOb2J3WfknGMI3JIkCXtzB2oWhleiLDjxW/NyBkDIKmpB5Hf2Bl0TH21SNwHfI+tpcibWDJbxKY4I99dY9+tdem0UDK80eNqb15KB83gEmTkreL05X/LKuUF7friJZZaPmH+FlzmK+PBiSY2QrVt9/dTyCR7PTDrAlGf8KWobgJaUk0oybn8=;20:FyntZNYzxUVn8WaHTgH+Des+gJce7hsiEvzRr3Z1iaPqgay8RVsk6+koRAhGY99yAuvwfn6alGU4Sx0ZVMT+Xfjn45VXfJ/MFDMe1ULbxSyruYFjRtFG3ebUO4OkCp+V+ICwqul7QhmLfdyozOpwdQ8zy0x/7XTpK9RP9QF6P/6y+RfDGAh43TS6jLk/qSNrLcK2P1gxB59FeiZSqK9rMu6qLN92hes+Iy6zn0EZAbex1B/E0LP01R8IA1ausw1toiV8GDNUY4QOFVZWKT/yABydKYUek2zP73pTcdLMPv21T+tg13y5q1mYjaUav3HRZpqph89a8dr7fPnFfi1b10Ulq8ZbwfVdO2tFvt+oljESN4yC+Rsud2iyAqoK2/1/SlfdqKxTFusw0T45fkxwjDmsg0aOiHGjocOETrtrPeHu4x3717omlpnqikZKxiM7xggi08wEq+w+J2rhT5sJChJ1VAlrSoKDSEUhwR+DTVgLyl98pmOWp184aJOgN7Oi X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6055026)(6041248)(20161123558025)(20161123560025)(20161123564025)(20161123562025)(20161123555025)(6072148);SRVR:MWHPR12MB1455;BCL:0;PCL:0;RULEID:;SRVR:MWHPR12MB1455; X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1455;4:BjEi/V6jVAiIO/YRwibjZTOH8CMtnUNtpXWrrXNJIx9C35Lobeyopcgy6lCn0JDDFW6B6MPRLnFRrAik7yM1KKK2IDRGQPtJ2/9qrvoyMYgo0cD39atKMvnaeBzyfZowWBEbr+ariAOBGFDO5zFyQevsJ4gFcwGvyxTier6eV00zgtwK0GV587GUISBmOr8b1UgoumPN2I77OKVPIJCk9XPdYVuRmXv7bZ9acrx75vKOCjNXdHEXEITYqa5ceMGHjRrru3BrmGF5JepPNNe/CPx+6LMt7ylyjptChug3cqVcqjM+9CCaA36Dh4SU2FGyMeVKWJKwGz5KkNUVuZIGe6KTUbcqZAT9Hi2XlN64OGqwLCNdzTh3ofbiv42DPWTu+7ON90qgz/6WyaElwiuWhJjeUym2oZvqeDRjH35ftb+/RF2Zvo1b7H5TpECYRT9hv6gmLDSNst9RLK7MIKu2sdKNBVDZdzoNTt6FgNq8C6Kux1up8UaW0hMrriaKfbbhC15JQZEBfV5UyV1r7AiPDUUsOp/DRQkG13vP9C0ClW6d+fOA+uFSWhRqHZZdxmPM+BpkvPqzbe8aiPQMwg1lZkmLc4UXFoLha/1Uq4mxrGnmyxQROmBKmEn6M8QQseuK X-Forefront-PRVS: 02272225C5 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(7916002)(39840400002)(39860400002)(39850400002)(39410400002)(39450400003)(189002)(76104003)(199003)(24454002)(6916009)(33646002)(81166006)(47776003)(93886004)(4326007)(92566002)(81156014)(76176999)(65826007)(2950100002)(83506001)(6666003)(68736007)(2906002)(8676002)(86362001)(31696002)(5660300001)(42186005)(106356001)(23746002)(305945005)(7736002)(105586002)(54356999)(6506006)(31686004)(36756003)(101416001)(189998001)(97736004)(4001350100001)(50986999)(50466002)(6512007)(65956001)(64126003)(38730400002)(53546006)(230700001)(53936002)(229853002)(54906002)(110136004)(6116002)(66066001)(6246003)(6486002)(25786008)(65806001)(3846002)(7756004);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR12MB1455;H:Suravees-MacBook-Pro.local;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;MWHPR12MB1455;23:wL+mOgm//PuuUlYeAadhkbE+JWO1lx1Uqo0Gu?= =?Windows-1252?Q?uPuAnGlOdm6cwiTJgWTHRyHrGr7cZm/M1CdR6bfj936luTQIhz1qhUW0?= =?Windows-1252?Q?uUZGSK/7MhjZnCTdFt5MR57hdVk2hAVvvrBD5TXMu8nFzjuj1HOHbouy?= =?Windows-1252?Q?k6RPnF1hKHwN56PgqmVWJYH4uU1KY6ILcVwaXg0TQz2Z10DUmOeuDqwN?= =?Windows-1252?Q?6zkwlspnTuRBKA0ccblQBd3MAhcdBQQY3TnzPWei1I8ZnPqrzd1XcviO?= =?Windows-1252?Q?N4AY0px5G0qC7UpgkJVcMxrnBMRyT66MEWP+s5bGFuwxmjQhhXKdMcd3?= =?Windows-1252?Q?HFyuOUoMCcgBNEXQwmou4nT28ME2s2HH/XOq2L3IaDN1OIZSJWDwEhml?= =?Windows-1252?Q?C9jmQzK+Th1q6WtzxnO80QAQrlJgGbwcDF10jq0FBzVmIhdgVgutg823?= =?Windows-1252?Q?YowgavNezuFuOK9R4nm9+e78LvKXtlzM/BqvdSNt4wQ7Or2UVZOUUQqs?= =?Windows-1252?Q?Hy1KC82BRZmTdSIxtSQxmdF4Su/QqErhvTYmhoFEC7N4Pc/AmSnDQJzj?= =?Windows-1252?Q?h5q+j127AG3/PcH6w6mjVhzOZ6jQZsDFso08K5Dq5f7GkWq3erN0PrRz?= =?Windows-1252?Q?aPGFAvM3WwsTz/bzZKZCRDfgxnLoLkiQSUafYCXr1D0WTuWGtXsBTWnk?= =?Windows-1252?Q?lxkma4GOk4a5Rhv9YQk5XDnMCnwSuXKxVueFTOHt9f+NrLSVXNTBVyG2?= =?Windows-1252?Q?2cuAG+cXGTreaj1mJP1CvzhLcDAncyMa4NfBhuMJBMRfi/SM9+qP2sOK?= =?Windows-1252?Q?bDJd92N5THmFwK4tsuIzdd1tthpYtzQ5hM6W22+gXEuoTXD030gnyoVi?= =?Windows-1252?Q?peD4w8OyKj4CG4/q2V/e7PyglNCafsWgpAC7zYkpYeHaWLbd8VRVaiRz?= =?Windows-1252?Q?Kwr2phg6l/gSVeF4uIgFjfsHc5VkjkZakGrcIG76wGSPbCwfnCbjXjZ2?= =?Windows-1252?Q?hRIcqwWxduU/nf4jFSMmebt4iseF2g7glffm6b1ycrDF9GpW5YmwHpIS?= =?Windows-1252?Q?oEfPJKufkQnd9ky+suaTdN862EjwgkP+K8S0tpQIfEhNEoRp6qZQY0uu?= =?Windows-1252?Q?NymSGS2s7s2DYmcHxuj841l6ym+qub1QJ81vz8+f7IrT14xVGbPTN+Xh?= =?Windows-1252?Q?zpyxZ/P+PoauYCk15f4lWJBzwrpZoGJRTe3NQ+yCTDbGwY+T4weBPCnt?= =?Windows-1252?Q?n62TjlL2TvW7IlXE+XAjO05zpSBMlzRH1em4gU+AA0AHYaq/S4RqEBP2?= =?Windows-1252?Q?eHQxBOdEeZTRwKJuBlUOsZRC+mkd/gM2sugA2bqrjvNOP8y98E36ZL88?= =?Windows-1252?Q?cUKec9x8/IZlSgl6ZjztgmJhjJXOfvNScEOin9MJmIs1hWvIxofve4JF?= =?Windows-1252?Q?JHYK211wq+oorIRdx3OidLl6ExNPCUDfYasfY/0wvPSLFMGR++e2Ev95?= =?Windows-1252?Q?Nm1DYG9VaVKd3M5o40t8wL05NEt2Cj3q+I9MZ8SdpASjkSX4qxkNBaeo?= =?Windows-1252?Q?2Ptb8oQudE2ZJ0X6T6UMsviv7MkQM5cw2KhZvoaDVrxxK4HIIE/5Zx46?= =?Windows-1252?Q?QTtQgh2iK3f8D1s7mumeLHBaz5y9f+P2tXI2rNDB1J/sqOQiuJi2C3eD?= =?Windows-1252?Q?ZwppsxJoQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1455;6:7eHvOktl9cfx44W/mIpiFIYfAtue1HHOyjdZOfYDqXa3x5g2NRn2C8FHQwhbS389F4IxZofJFSBKifvMCJx35CwJuJxTKgLZOsORMfbQRHHuU4kHFkB1JlQ+C61Vpd9zR66ibH6LQ9P0zVWP/NrhtQ5CIA9Yhkj2vD0KTQHj4kSOIEGI2W0dHAHpExKmhrluJ5XLXUW52ennoggZYiBmuh/FHIyYwyrl3uLGef2mr5ljZRXuCNkds7eJ+y9mGTwOWFJm2gur5+E4v8QM4pgC2oWRk+jW3qBps8mRh6CKro5Fe53+k/rL9s8vN7RjCqOWHOLfbMbNtHKQsvSCCRDDaE/FRXvzeeOQcut8nH2LfT3jEL0/9lcPXCRwBaZ+/iagbA8BidfVDH3ceaFcVi0lN9eumkHkuhM+1INtIe4fu1A=;5:+6qX79rz2SDa9gWafTzL3dfuIQVLHSfZSXI0SelfWOBQ6lOOsVJi3blm+hewm4NObpI/c9D2I+8vIRN9ch72mHIjbWpcE3MAGoDxXlo/bjBEU3Dnt5DIhJtO2sIbaMx3WejHFclPWRm/P6RMnuFzEg==;24:O2hgcYph7RWL5TBSnqUbOfbS8dnSo6Gz5MTE4ECg8hlX+qtGrUxHxyl+Sl1Ger6TxwiuLMb6kOSERZ/Ny3wr227Sds5dZJv4QuottoIjRyI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1455;7:HDtRqesWPX13VZCtWXdPWQWxPvkt2/zUw9ilnyRTRcB3c5Hk3JxoL+iW9ZnPCgVF+32oqL0JgO+DERWoiyM+Q5WG+fj3rulqaheOpFueHrNDsR6BA7cWhvjyleW/f19VG8hv0CRfgUgSHysUsCrBifT/K5E71sHNiQ3snFsxuro4FmQM4pDKSh5cx5fdrXv6YuYvCGVpJ6LlszJHw5UUhe3EN4dowELSEAJ+Q8i4Lve00x2vBKlS4hBlLyxh7Nv93FRsx5K9zrdL01dTaxOO53unMwumMGwCnyzy4hksiqxq6UMo/eVPbNJfAjHrJWa5CxNSpKIxV96dST1rm4c34A==;20:5/z6WzDl1kYNypJHnmGyfsLd1TTvHbAfm3747FBSZ5NTWnG+bP+3sJUSQGYbqRrHyugXR2qbJSOdY+FS+vgc5HS4ZyYe9DGV6e3sNrcSRWRzkQK8gkTU8NtUUBxgvW3CigXB5PAZoEMyyQ5gyyKdFnjaKUCDH2lIkIKudXFJRI118W4HbEoBfPNwR1ndlNcOnWszXDm84Jt4U21W4Zh63UFMTyx+BvZM9cwHechP7YxS98MwMwsjkV7FEkAQ8XUD X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2017 17:43:44.7339 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1455 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Peter, On 2/14/17 19:31, Peter Zijlstra wrote: > On Tue, Feb 07, 2017 at 08:57:52AM +0700, Suravee Suthikulpanit wrote: >>> But instead it looks like you get the counter form: >>> >>> #define _GET_CNTR(ev) ((u8)(ev->hw.extra_reg.reg)) >>> >>> Which is absolutely insane. >>> >> >> So, the IOMMU counters are grouped into bank, and there could be >> many banks. I use the extra_reg.reg to hold the bank and counter >> indices. This will be used to program onto the counter configuration >> register. This is handled in get_next_avail_iommu_bnk_cntr() and >> clear_avail_iommu_bnk_cntr(). > > But this is crazy. That's not what extra_regs are for. Ok, I understand that we should not be using the extra_regs since it is intended for other purposes. Please see more detail below. > Also, who cares about the banks, why is this exposed? The bank and counter values are not exposed to the user-space. The amd_iommu PMU only expose, csource, devid, domid, pasid, devid_mask, domid_mask, and pasid_mask as event attributes. > That is, I would very much expect a linear range of counters. You can > always decompose this counter number if you really need to somewhere > down near the hardware accessors. > Actually, the counters are treated as linear range of counters. For example, the IOMMU hardware has 2 banks with 4 counters/bank. So, we have total of 8 counters. The driver then assigns an index to each events when an event is added. Here, the bank/counter are derived from the assigned index, and stored in the perf_event as bank and counter values. However, I have looked into reworking to not use the extra_regs, and I see that the union in struct hw_perf_event currently contains various PMU-specific structures (hardware, software, tracepoint, intel_cqm, itrace, amd_power, and breakpoint). For amd_iommu PMU, we need additional registers for holding amd_iommu-specific parameters. So, it seems that we can just introduce amd_iommu-specific struct instead of re-using the existing structure for hardware events. I'm planning to add the following structure in the same union: union { ...... struct { /* amd_iommu */ u8 iommu_csource; u8 iommu_bank; u8 iommu_cntr; u16 iommu_devid; u16 iommu_devid_msk; u16 iommu_domid; u16 iommu_domid_msk; u32 iommu_pasid; u32 iommu_pasid_msk; }; }; Please let me know what you think, of if I am still missing your points. Thanks, Suravee