From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BACA920485B for ; Tue, 4 Mar 2025 13:56:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741096590; cv=none; b=ToAPFMrBEUx++vcjQZSHHr1B04D1XbCI8gBS+svOc/DXq2O1cDMGz+18NOtaS/7cRXWq+ccki3xnA5L6Jd0yLaVu6oxqxViNFb+U0pbvNQYltt8r3QPVkI2UNgZpeB6y6Lrm326TX0m0n+t2EzIBiW+kdWK1EFlr6BKsvhrA3D0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741096590; c=relaxed/simple; bh=B1HhjIwvey1c5ezYoPA1l5ZccU/01rKBuPoMwstDUJ8=; h=Message-ID:Date:MIME-Version:Subject:From:To:CC:References: In-Reply-To:Content-Type; b=mRaH3ZGIVC6qP6xIzMPHcQtqZw1r1pbSRC+GSxEnDwf3/wmHmHt3rhQUwQQEJZaQKPq2CtHmAGVEhD3rI2pcwNiN3juHFE0GEDu30KnXxQraPQ3lk2ofm0yKFYfyE5Uvxo+kJXXx3Li1thThLA/pOW8RYzZXyW5HOpMid/bakLc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.174]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Z6cXz4wvlzwXNs; Tue, 4 Mar 2025 21:51:31 +0800 (CST) Received: from kwepemj200003.china.huawei.com (unknown [7.202.194.15]) by mail.maildlp.com (Postfix) with ESMTPS id 0C7131400CD; Tue, 4 Mar 2025 21:56:25 +0800 (CST) Received: from [10.67.120.170] (10.67.120.170) by kwepemj200003.china.huawei.com (7.202.194.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 4 Mar 2025 21:56:24 +0800 Message-ID: Date: Tue, 4 Mar 2025 21:56:23 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 2/3] dma-mapping: benchmark: add support for dma_map_sg From: Qinxin Xia To: Barry Song <21cnbao@gmail.com> CC: yangyicong , "hch@lst.de" , "iommu@lists.linux.dev" , Jonathan Cameron , "Zengtao (B)" , "fanghao (A)" , "linux-kernel@vger.kernel.org" References: <20250212022718.1995504-1-xiaqinxin@huawei.com> <20250212022718.1995504-3-xiaqinxin@huawei.com> <43618c9167654c68945ec5e7d9bf69d5@huawei.com> <66470ea1-668a-418a-bcf6-c8702be799ce@huawei.com> In-Reply-To: <66470ea1-668a-418a-bcf6-c8702be799ce@huawei.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemj200003.china.huawei.com (7.202.194.15) 在 2025/3/4 21:49, Qinxin Xia 写道: > > 在 2025/2/22 14:36, Barry Song 写道: >> On Fri, Feb 21, 2025 at 4:16 PM xiaqinxin wrote: >>> >>> >>> -----邮件原件----- >>> 发件人: Barry Song <21cnbao@gmail.com> >>> 发送时间: 2025年2月18日 4:59 >>> 收件人: xiaqinxin >>> 抄送: chenxiang66@hisilicon.com; yangyicong ; >>> hch@lst.de; iommu@lists.linux.dev; Jonathan Cameron >>> ; Zengtao (B) >>> ; fanghao (A) ; >>> linux-kernel@vger.kernel.org >>> 主题: Re: [PATCH 2/3] dma-mapping: benchmark: add support for >>> dma_map_sg >>> >>> On Wed, Feb 12, 2025 at 3:27 PM Qinxin Xia >>> wrote: >>>> Support for dma scatter-gather mapping and is intended for testing >>>> mapping performance. It achieves by introducing the dma_sg_map_param >>>> structure and related functions, which enable the implementation of >>>> scatter-gather mapping preparation, mapping, and unmapping operations. >>>> Additionally, the dma_map_benchmark_ops array is updated to include >>>> operations for scatter-gather mapping. This commit aims to provide a >>>> wider range of mapping performance test  to cater to different >>>> scenarios. >>> This benchmark is mainly designed to debug contention issues, such >>> as IOMMU TLB flushes or IOMMU driver bottlenecks. I don't fully >>> understand how SG or single will impact the evaluation of the IOMMU >>> driver, making it unclear if the added complexity is justified. >>> >>> Can you add some explanation on why single mode is not sufficient >>> for profiling and improving IOMMU drivers? >>> >>> Hello Barry ! 😊 >>> Currently, the HiSilicon accelerator service uses the dma_map_sg >>> interface. We want to evaluate the performance of the entire DMA map >>> process. (including not only the iommu, but also the map framework). >>> In addition, for scatterlist, __iommu_map is executed for each nent. >>> This increases the complexity and time overhead of mapping. The >>> effect of this fragmentation is not obvious in dma_map_single, which >>> only handles a single contiguous block of memory. >>> >> Thanks! >> Please update your editor to ensure it doesn't respond with such long >> sentences >> without line breaks in the future :-) > > Hello Barry ! > > Thank you for your advice, I will I'll pay attention. Leon A little mistake, sorry for it... I will I'll pay attention. Leon  ==> I will pay attention. > >> Can you provide concrete examples or data showing how the newly added >> mode >> improves profiling of the entire DMA map process? For instance, what >> limitations >> exist without this mode? What performance issues cannot be identified >> without >> it? > > You can see this patch > :https://lore.kernel.org/all/cover.1738765879.git.leonro@nvidia.com/ > > Leon provides new interface for scatterlist scenarios to improve > performance and gives some > > application instance in rdma and vfio. Users can use dma_map_sg > benchmark to measure > > the performance improvement of the optimized interface compared with > the previous one. > > Thanks! > >> >>>> Signed-off-by: Qinxin Xia >>>> --- >>>>   include/linux/map_benchmark.h |   1 + >>>>   kernel/dma/map_benchmark.c    | 102 >>>> ++++++++++++++++++++++++++++++++++ >>>>   2 files changed, 103 insertions(+) >>>> >>>> diff --git a/include/linux/map_benchmark.h >>>> b/include/linux/map_benchmark.h index 054db02a03a7..a9c1a104ba4f >>>> 100644 >>>> --- a/include/linux/map_benchmark.h >>>> +++ b/include/linux/map_benchmark.h >>>> @@ -17,6 +17,7 @@ >>>> >>>>   enum { >>>>          DMA_MAP_SINGLE_MODE, >>>> +       DMA_MAP_SG_MODE, >>>>          DMA_MAP_MODE_MAX >>>>   }; >>>> >>>> diff --git a/kernel/dma/map_benchmark.c b/kernel/dma/map_benchmark.c >>>> index d8ec0ce058d8..b5828eeb3db7 100644 >>>> --- a/kernel/dma/map_benchmark.c >>>> +++ b/kernel/dma/map_benchmark.c >>>> @@ -17,6 +17,7 @@ >>>>   #include >>>>   #include >>>>   #include >>>> +#include >>>>   #include >>>>   #include >>>> >>>> @@ -111,8 +112,109 @@ static struct map_benchmark_ops >>>> dma_single_map_benchmark_ops = { >>>>          .do_unmap = dma_single_map_benchmark_do_unmap, >>>>   }; >>>> >>>> +struct dma_sg_map_param { >>>> +       struct sg_table sgt; >>>> +       struct device *dev; >>>> +       void **buf; >>>> +       u32 npages; >>>> +       u32 dma_dir; >>>> +}; >>>> + >>>> +static void *dma_sg_map_benchmark_prepare(struct map_benchmark_data >>>> +*map) { >>>> +       struct scatterlist *sg; >>>> +       int i = 0; >>>> + >>>> +       struct dma_sg_map_param *mparam __free(kfree) = >>>> kzalloc(sizeof(*mparam), GFP_KERNEL); >>>> +       if (!mparam) >>>> +               return NULL; >>>> + >>>> +       mparam->npages = map->bparam.granule; >>>> +       mparam->dma_dir = map->bparam.dma_dir; >>>> +       mparam->dev = map->dev; >>>> +       mparam->buf = kmalloc_array(mparam->npages, >>>> sizeof(*mparam->buf), >>>> +                                   GFP_KERNEL); >>>> +       if (!mparam->buf) >>>> +               goto err1; >>>> + >>>> +       if (sg_alloc_table(&mparam->sgt, mparam->npages, GFP_KERNEL)) >>>> +               goto err2; >>>> + >>>> +       for_each_sgtable_sg(&mparam->sgt, sg, i) { >>>> +               mparam->buf[i] = (void *)__get_free_page(GFP_KERNEL); >>>> +               if (!mparam->buf[i]) >>>> +                       goto err3; >>>> + >>>> +               if (mparam->dma_dir != DMA_FROM_DEVICE) >>>> +                       memset(mparam->buf[i], 0x66, PAGE_SIZE); >>>> + >>>> +               sg_set_buf(sg, mparam->buf[i], PAGE_SIZE); >>>> +       } >>>> + >>>> +       return_ptr(mparam); >>>> + >>>> +err3: >>>> +       while (i-- > 0) >>>> +               free_page((unsigned long)mparam->buf[i]); >>>> + >>>> +       pr_err("dma_map_sg failed get free page on %s\n", >>>> dev_name(mparam->dev)); >>>> +       sg_free_table(&mparam->sgt); >>>> +err2: >>>> +       pr_err("dma_map_sg failed alloc sg table on %s\n", >>>> dev_name(mparam->dev)); >>>> +       kfree(mparam->buf); >>>> +err1: >>>> +       pr_err("dma_map_sg failed alloc mparam buf on %s\n", >>>> dev_name(mparam->dev)); >>>> +       return NULL; >>>> +} >>>> + >>>> +static void dma_sg_map_benchmark_unprepare(void *arg) { >>>> +       struct dma_sg_map_param *mparam = arg; >>>> +       int i; >>>> + >>>> +       for (i = 0; i < mparam->npages; i++) >>>> +               free_page((unsigned long)mparam->buf[i]); >>>> + >>>> +       sg_free_table(&mparam->sgt); >>>> + >>>> +       kfree(mparam->buf); >>>> +       kfree(mparam); >>>> +} >>>> + >>>> +static int dma_sg_map_benchmark_do_map(void *arg) { >>>> +       struct dma_sg_map_param *mparam = arg; >>>> + >>>> +       int sg_mapped = dma_map_sg(mparam->dev, mparam->sgt.sgl, >>>> +                                  mparam->npages, mparam->dma_dir); >>>> +       if (!sg_mapped) { >>>> +               pr_err("dma_map_sg failed on %s\n", >>>> dev_name(mparam->dev)); >>>> +               return -ENOMEM; >>>> +       } >>>> + >>>> +       return 0; >>>> +} >>>> + >>>> +static int dma_sg_map_benchmark_do_unmap(void *arg) { >>>> +       struct dma_sg_map_param *mparam = arg; >>>> + >>>> +       dma_unmap_sg(mparam->dev, mparam->sgt.sgl, mparam->npages, >>>> +                    mparam->dma_dir); >>>> + >>>> +       return 0; >>>> +} >>>> + >>>> +static struct map_benchmark_ops dma_sg_map_benchmark_ops = { >>>> +       .prepare = dma_sg_map_benchmark_prepare, >>>> +       .unprepare = dma_sg_map_benchmark_unprepare, >>>> +       .do_map = dma_sg_map_benchmark_do_map, >>>> +       .do_unmap = dma_sg_map_benchmark_do_unmap, }; >>>> + >>>>   static struct map_benchmark_ops >>>> *dma_map_benchmark_ops[DMA_MAP_MODE_MAX] = { >>>>          [DMA_MAP_SINGLE_MODE] = &dma_single_map_benchmark_ops, >>>> +       [DMA_MAP_SG_MODE] = &dma_sg_map_benchmark_ops, >>>>   }; >>>> >>>>   static int map_benchmark_thread(void *data) >>>> -- >>>> 2.33.0 >>>> >> Thanks >> Barry