From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95905C001B0 for ; Wed, 16 Aug 2023 03:49:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241688AbjHPDsj (ORCPT ); Tue, 15 Aug 2023 23:48:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241768AbjHPDsE (ORCPT ); Tue, 15 Aug 2023 23:48:04 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24EEE1FC3 for ; Tue, 15 Aug 2023 20:40:40 -0700 (PDT) Received: from kwepemi500019.china.huawei.com (unknown [172.30.72.55]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4RQYlm5rptzrSD7; Wed, 16 Aug 2023 11:39:16 +0800 (CST) Received: from [10.174.176.82] (10.174.176.82) by kwepemi500019.china.huawei.com (7.221.188.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Wed, 16 Aug 2023 11:40:36 +0800 Message-ID: <80286146-578e-5814-5a2b-5535dc476782@huawei.com> Date: Wed, 16 Aug 2023 11:40:35 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.1.1 Subject: Re: [RESEND PATCH 2/2] iommu/iova: allocate iova_rcache->depot dynamicly To: John Garry CC: , , , , , , Robin Murphy , , References: <20230811130246.42719-1-zhangzekun11@huawei.com> <20230811130246.42719-3-zhangzekun11@huawei.com> <36924a4d-e62c-68d5-3cb0-375b7fe1d5c0@huawei.com> <99d905fb-0732-bb7b-f631-f08c897f1d8c@oracle.com> From: "zhangzekun (A)" In-Reply-To: <99d905fb-0732-bb7b-f631-f08c897f1d8c@oracle.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [10.174.176.82] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To kwepemi500019.china.huawei.com (7.221.188.117) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 在 2023/8/15 23:15, John Garry 写道: > On 12/08/2023 08:18, zhangzekun (A) wrote: >> >> >> 在 2023/8/11 22:14, John Garry 写道: >>> On 11/08/2023 14:02, Zhang Zekun wrote: >>>> In fio test with 4k,read,and allowed cpus to 0-255, we observe a >>>> performance >>>> decrease of IOPS. >> The normal IOPS >>> >>> What do you mean by normal IOPS? Describe this "normal" scenario. >>> >> Hi, John >> >> The reason why I think 1980K is normal is that I have test the >> iova_rache >> hit rate with all iova size, the average iova cache hit rate can >> reach up to > > Sorry to say, but I still don't know what you mean by the terms > "normal" and "abnormal" here. Is "normal" prior to the drop in IOPS > which you mention, above? If so, at what time do this occur? Hi, John The decrease is first observe in high concurrency fio test based on openeulr kernel-5.10, and the IOPS is about 300~400K , which is quite abnormal for some low fio concurrency test can have more than 1000K IOPS. The frame graph show that iovas alloc from slow path alloc_iova() takes a high percentage, and I find out that current struct of iova_rcache could  behave poor with machine have 256 cores and with device driver does not free and alloc iova evenly in a heavy work load. After optimizing the iova_rcache, the IOPS come to more than 1900K IOPS on openeuler kernel-5.10. The mainline of linux kernel have the same problem on openeuler kernel-5.10, but the IOPS does not improve such a large percentage. I use the term "normal" and "abnormal" here, maybe these pair of words should be replaced by "before optimization" and "after optimization". Thanks, Zekun