From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756206AbdJLJli (ORCPT ); Thu, 12 Oct 2017 05:41:38 -0400 Received: from mail-sn1nam01on0079.outbound.protection.outlook.com ([104.47.32.79]:26432 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752483AbdJLJlg (ORCPT ); Thu, 12 Oct 2017 05:41:36 -0400 Subject: Re: [PATCH V2 0/1] Optimise IOVA allocations for PCI devices To: Tomasz Nowicki , joro@8bytes.org, robin.murphy@arm.com Cc: will.deacon@arm.com, Jayachandran.Nair@cavium.com, ard.biesheuvel@linaro.org, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, Ganapatrao.Kulkarni@cavium.com References: <1505897522-9237-1-git-send-email-tomasz.nowicki@caviumnetworks.com> From: Tomasz Nowicki Message-ID: <604a93b9-1af4-a961-6c78-cb755d0caa1e@caviumnetworks.com> Date: Thu, 12 Oct 2017 11:41:20 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <1505897522-9237-1-git-send-email-tomasz.nowicki@caviumnetworks.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Originating-IP: [31.172.191.173] X-ClientProxiedBy: AM5P190CA0021.EURP190.PROD.OUTLOOK.COM (2603:10a6:206:14::34) To MWHPR0701MB3658.namprd07.prod.outlook.com (2603:10b6:301:7d::39) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fce704a9-48f4-4f86-582d-08d511556c7d X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075);SRVR:MWHPR0701MB3658; X-Microsoft-Exchange-Diagnostics: 1;MWHPR0701MB3658;3:L04AOM+2XysBonu5hz1u2xeD3mJ9xve3t7Akk8KsK223HE8hY5gWr9m5guXzsLUU5+rSjj7CGtL+il2YatWnKHcPahc6/8pLPCgKA43k1jrzu19OmeYEDrwG+vtpGGKZKsk13OxErrCvwzbiPkxNMM2dSmC1XIeGT33W9dFS5DPxqUw4R6GM7mykjTGA+mURJHt1vnE8mpQGuYs3U5SByY2f9m3Qi7vRSTOuNi3OGEVSmiyPgZF0/96dBhJhbW/I;25:wJWUZVdR311IMiuPv2JdXKS13rzGrou/njOCj9vpZv7uqIE2G9vliq7azD1Eki5R44BuyqzPe2aQ3XQZiZIC6ld4UQ0ktoSLPpfNOoUY3rJlqI6Zrw21C2MKDdCo2umB6z1nSgxVt0FiCEOu4d2Py6UrcyAR4AIVb8a0aOyXRDSYFEaZ0HriyoryN0yyjf6kq1cDck9fuEwyWRSY7KWsIf7JJSbYJ5+Id2+xLI/BqwnopATRbzK6XKEWchclhny7HBjVnnZ8sKuAGh99aCKzvfqkEgydP+j2x3pwN9pHGKdcVEB4nYIdXhcTsr5wjJccaEub0Zk8ydTgy/Ep93vSMw==;31:5WqDfIZIZvFAt5dUtziBYVhAO9+/flb2dKSWhejJm4SeZqihtChdHZ+BT18OwP/T+Oah0qSQEfLMYAeU9YUtomhFFVgXK40ipXQgyjTGaq3K+tMyvrs6JUdrb/ADIZ4Ma90WShqOLWHLPu1FqgH69QwqVW3iUUq9KSW+wc3TciNmF31FXUfjx+Kt2wqyLXjefwAcJ27fEcKy7bhrmn3mh7gEWc/TCoGyI1M0AFVpUHU= X-MS-TrafficTypeDiagnostic: MWHPR0701MB3658: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Tomasz.Nowicki@cavium.com; X-Microsoft-Exchange-Diagnostics: 1;MWHPR0701MB3658;20:kQRQBajzjtBXL9kvgcv14mRmU+DQtxjqYaEBkQpjpdCJn9jyYcSjEUbGlCP5vSc0JHBtbbwk2Ws/yTur2Mcr1oYOrATn7e+5macMPfIrCNZ3qDejC92OgT2+RXmbnAg2gek1tfut6GYekxf+9wPOwklQFImozkK7NKcZZV9m+/24B7aNcgrpcnRmYvRdCsCKV5d75hhycQgnV73F/OJw9fYf0IreAU+W0Dh6ixFBGU1iYgNaQGLsL1228O2kpnbRZB37qG+7i/pArBx3fSKzg/I+lVGQIwTlD5HM5JHRjib70sOX0DaagKoT+te2zlujp/yUlRTkxi9o0xXiC+H9IsEDgymo70scI8huNICdMQlg+LtYEfFF1GXwGOQiu1WPsLOfG+SeTu9IgW8FguMiDVTi2sNmv3e4jW8X9ejkQQ9fItjT7rypPlbRbxPbLAggfSBQcYYTPA50t8FHWQ//e7T+7Fni7mnBqEvmB9UwTtfLd5g8Qh8xK2GQgfq/m2dI3QPuhAl8CW+2n84VUVzKoLtsscNHoI1eppCLcT0HRul87hgudm43NA/tE17Odh9ihMiqrn95Otw+0Tx/Lg7pBRNCLt0kjuJ4Ueenyx7T0i8=;4:ZzGVsgk5lJK7ddECpDC9sqODr/jEH/N7MkTEcLYtUwvIh4UsdOHD6neAy5vGC+uo3EQ8NT5XPDpOqHVeWGyJvQDHxR9qnQjo/CsE7xZuiW+DtS0f4fxelJux7JMIuN15T6SC1mo4M6WqrlMVDlahibwtdBLhAkDrPBn9S5h+lPur6PW3VWQrfRDtI7o1U0fA13Q5xzBq/AfuOJZJtP5NFyAzcRtn8h6hmNLqbQrOyJbEzpgCheqeRAOykRdmyGd2 X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(93006095)(10201501046)(3002001)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(20161123555025)(20161123564025)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR0701MB3658;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR0701MB3658; X-Forefront-PRVS: 04583CED1A X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6009001)(6049001)(376002)(346002)(51874003)(199003)(189002)(24454002)(54534003)(53936002)(72206003)(50466002)(2906002)(189998001)(31686004)(64126003)(7736002)(54356999)(50986999)(305945005)(105586002)(76176999)(101416001)(31696002)(5660300001)(25786009)(316002)(2950100002)(42882006)(4326008)(23676002)(16526018)(16576012)(6666003)(36756003)(6116002)(3846002)(68736007)(106356001)(97736004)(47776003)(58126008)(53546010)(66066001)(77096006)(33646002)(65956001)(6486002)(478600001)(230700001)(8676002)(81166006)(81156014)(8936002)(107886003)(6246003)(83506001)(229853002);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR0701MB3658;H:[10.0.0.85];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtNV0hQUjA3MDFNQjM2NTg7MjM6RFM0SWE0ckhiK256cHh6U1dKdTQ1UDBl?= =?utf-8?B?M29EV1dwM3ZtRTE0MHZxZ3hlb1hNNzVPV2JOU0pyUjZEZ2ZUSjhEZk5sWHY4?= =?utf-8?B?bGtuL2Z3L2RnZW1yVkdidHJPaG94UU1QdytxOGRGYmtqbmh6Rk90MEFKeEVC?= =?utf-8?B?b0pRbFZGeUsySVUrRGlCQUxheFdlTFpSYnllNmdXeGNFYmMyQUFJQ3JmRmd0?= =?utf-8?B?bHArTHA2Y1Z1WEpzdkdNcDhoWU5KdjZZMVBSNGMwSS9nODVzR1oyWmlGNXRh?= =?utf-8?B?bDVNaW5qcHhDOHNTeXl6WGJNVGJ4NzhrQm1leVh2T2xmc09VaUI3QTgvc282?= =?utf-8?B?d3VyUFlxT1JzUEEwR3Z0cFROT2RhVkpKbW1iK2tmcFYyajFsaC93MGczc1JQ?= =?utf-8?B?MHpwQXlEQVVIdGtoNXNwWElTVWVVdmVwbDVvcXl1aENPVlVna2NPeWNjT09M?= =?utf-8?B?emRxNjhlalhyNmJrSGxJUjB5eFl6U3FzclhYSXZ3RlRlVHlnQkhOZytKZ203?= =?utf-8?B?ZXlDMitCWTlHVmgwc0R6V1A3czVGaUVvVkthLzl3cDRXejQ4bndMSXppOWNr?= =?utf-8?B?bEhiMEhKK1BUVXJ1c1ovSVNrL0NKTVJWWFVSY3ZvS2M4cEI1NU5PdHRzZjF6?= =?utf-8?B?dHFVNEU4ZHk2elByeXpreFI0Slo4QzZOMjBsTkNPMzQ4cG90VCtMN0JPUmF3?= =?utf-8?B?eTYvY25qeHkySnR4YUJNa29IN1Y1TUs2cUFZOForZ002d3FkcW4rUEtqV2RR?= =?utf-8?B?dnRudDNiM0xuZ1grejJ2ZDZUcHQ0MmdyMHBuWUxhQXo4WVlwYk01UjBoUFVI?= =?utf-8?B?SXJ3d202YVNPK3FkMHNhTjBrYTBXTmVNWHJ4MEJkV1pIRi9wNGxNaE05R1Zq?= =?utf-8?B?MmhJb2lUeXRPUGRLY2d4L1pvWkJmSi8rUTdJeDF4aVZnSFBzQUVseTg3WDNB?= =?utf-8?B?RWw5a0tqdDBoanQzamNIVzBNNkt0WUpZbUlWZzQxVzhpUjE2OFQ1UGRLVytM?= =?utf-8?B?QjZUVHdoU1JscDV1MUlURTVhV29qcENrWmZZYU5LaXlIZzRLUGlaSFkrbWti?= =?utf-8?B?SGNSNWtuMTBaS2ZLajhsMXN4WUhxc1l0MU9vaEdGczNYQlFmVUhBVUsrcDh0?= =?utf-8?B?WXhUbWdPT3o3TUtKTUk5UDNTV2dZVHA5VlJoTG1GazlzclBKRmJxSGgwMDRn?= =?utf-8?B?Um5TeHlmd0VLank3WFNhN2dCREM0UHlVaEdjenVicTQ2dXRHN2o0WmtOUmxY?= =?utf-8?B?U1pkOFNMczJuTjRrYlVRRW9RdnVZaFBqRURReW9SQlo4L3FWYkRHOFZqOWJC?= =?utf-8?B?OElORnByYWQydkVyVUVIWURTd3o1OFJwZFFxclo3Y3FpM2xQTHVNcERyUXFt?= =?utf-8?B?eThQZExyVzMzUjRuR2tWMDZnV3B5NEZVcXRPQVFReHBDRlZ0dXpnNDRYZ095?= =?utf-8?B?emtDQ0FmakhMN1lydThmSXR1M2dtU28yRmUzWVRGLzhpSWh1TUlwazAzVFg4?= =?utf-8?B?K1hHZFpmV2kzYUxFU1pacGx6RVRwR2tTRTlmTC9BTStMOVFwNXlzU1gzMWtH?= =?utf-8?B?TXBUalpxb3pWVWxTK204OUpFbHBKWE1pRlE3VDhPeEV3OE5ORXN5cm4wQlpS?= =?utf-8?B?TlBzZm0zRHFZYzdLZCtRUDE0TWhETjZ1N1FVcjg4Tjl1QjR0akJMMCtGQ0Vp?= =?utf-8?B?OGh5alRhZjAwZEJDek9xOHFXSkJBTHRiRHhCUkJpS1BjYk51a2ZrS082K2J2?= =?utf-8?B?WVdMSEhaR0VScmdBdVBGNFlOUXNXOVRLT1hzY1VDVWFtbFZieFpVUEZYV1ZR?= =?utf-8?Q?0Va9KBWVDAP4VVB?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR0701MB3658;6:9OErtI41nLSWG4lHQkTxRXlF8B0iCDPkZ5isy/D+SR4EpAxB62g8H2yw3R8u8FoMuxOXYBa7eakFqOvDsw0BuDHkrcqkB/bouKLyEy6eHPmhvaIYiQ/9GE5ZncybL+wt5XR9jGOtBPIsVWIwr1PWlmzZLN1W7poua0um9S2ksFl3zHGk92LM789gHBOu0cTiv0kiCNMK0DPE6cX8lKub/Uzecp3qj1rJzU9q/m71+nh8sME5S+6+a3hbwA+DVC1C+sF+vsVQlbp63KNKS7OJF2gPsWJRCUwD2643PTiTbaY3TbPz5Yb0UcFBkfwFogtxJL/eYOZi+XMrhVu5Is5ifQ==;5:rMrJMAQX7Bf7N+VVuWA2UAnjt5Sw4xKICOd73XhHC0XyVbTDOdpUZOGDOPoALNcwonbVgD9wc9TIKypPSu1R2Fd82ZxcLKQAviR2CtdX4kLk3hhjYR0894miB4wfaYI4bfDr8kzxa/JzFr+B9ACLHw==;24:13qMWOpHpzznQp82YHvMSunjcuh7S33w7DMAyhYJ2pRw+uhIcBp23MglinhqBN5pvziVrDy+S4Mj8E/J/dVNmzs0LGGIQkHuPtOEUD0gl68=;7:5wSGjCd01wklfB58rYiSdH63J0dnOTZacZEPnq5e1IJ8fQGUHdcJTYuZd6WVT2YPmsqRfHfxCiAV9qTSnCnWzSeTDv2CNlQ+3k4EOPcomen8bJFsL+lXHXasgUgqPsbkkIUE9bWCJx9cIca2Z6pqp6Gc6mKyZgDWH5Xy4XPPEBla+qXo5j4fTNR5HidYjLmoW+tM29U5G19swUWDPb+mrfm7DVZ2UKjP1FNbC/TR/d4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2017 09:41:31.0651 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR0701MB3658 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Joerg, Can you please have a look and see if you are fine with this patch? Thanks in advance, Tomasz On 20.09.2017 10:52, Tomasz Nowicki wrote: > Here is my test setup where I have stareted performance measurements. > > ------------ PCIe ------------- TX ------------- PCIe ----- > | ThunderX2 |------| Intel XL710 | ---> | Intel XL710 |------| X86 | > | (128 cpus) | | 40GbE | | 40GbE | ----- > ------------ ------------- ------------- > > As the reference lets take v4.13 host, SMMUv3 off and 1-thread iperf > taskset to one CPU. The performance results I got: > > SMMU off -> 100% > SMMU on -> 0,02% > > I followed down the DMA mapping path and found out IOVA 32-bit space > full so that kernel was flushing rcaches for all CPUs in (1). > For 128 CPUs, this kills the performance. Furthermore, for my case, rcaches > contained PFNs > 32-bit mostly so the second round of IOVA allocation failed > as well. As the consequence IOVA had to be allocated outside of 32-bit (2) > from scratch since all rcaches have been flushed in (1). > > if (dma_limit > DMA_BIT_MASK(32) && dev_is_pci(dev)) > (1)--> iova = alloc_iova_fast(iovad, iova_len, DMA_BIT_MASK(32) >> shift); > > if (!iova) > (2)--> iova = alloc_iova_fast(iovad, iova_len, dma_limit >> shift); > > My fix simply introduces parameter for alloc_iova_fast() to decide whether > rcache flush has to be done or not. All users follow mentioned scenario > so they should let flush as the last chance to avoid time costly iteration > over all CPUs. > > This bring my iperf performance back to 100% with SMMU on. > > My bad feelings regarding this solution is that machines with relatively > small numbers of CPUs may get DAC addresses more frequently for PCI > devices. Please let me know your thoughts. > > Changelog: > > v1 --> v2 > - add missing documentation > - fix typo > > Tomasz Nowicki (1): > iommu/iova: Make rcache flush optional on IOVA allocation failure > > drivers/iommu/amd_iommu.c | 5 +++-- > drivers/iommu/dma-iommu.c | 6 ++++-- > drivers/iommu/intel-iommu.c | 5 +++-- > drivers/iommu/iova.c | 11 ++++++----- > include/linux/iova.h | 5 +++-- > 5 files changed, 19 insertions(+), 13 deletions(-) >