From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0A8FC4167B for ; Tue, 8 Dec 2020 22:52:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A79A123AC9 for ; Tue, 8 Dec 2020 22:52:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731510AbgLHWwU (ORCPT ); Tue, 8 Dec 2020 17:52:20 -0500 Received: from vps0.lunn.ch ([185.16.172.187]:45196 "EHLO vps0.lunn.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731438AbgLHWwQ (ORCPT ); Tue, 8 Dec 2020 17:52:16 -0500 Received: from andrew by vps0.lunn.ch with local (Exim 4.94) (envelope-from ) id 1kmlpN-00AvrE-AX; Tue, 08 Dec 2020 23:51:25 +0100 Date: Tue, 8 Dec 2020 23:51:25 +0100 From: Andrew Lunn To: Sven Van Asbroeck Cc: Jakub Kicinski , Bryan Whitehead , Microchip Linux Driver Support , David S Miller , netdev , Linux Kernel Mailing List Subject: Re: [PATCH net v1 2/2] lan743x: boost performance: limit PCIe bandwidth requirement Message-ID: <20201208225125.GA2602479@lunn.ch> References: <20201206034408.31492-1-TheSven73@gmail.com> <20201206034408.31492-2-TheSven73@gmail.com> <20201208114314.743ee6ec@kicinski-fedora-pc1c0hjn.DHCP.thefacebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > That's a good question. I used perf to create a flame graph of what > the cpu was doing when receiving data at high speed. It showed that > __dma_page_dev_to_cpu took up most of the cpu time. Which is triggered > by dma_unmap_single(9K, DMA_FROM_DEVICE). > > So I assumed that it's a PCIe dma bandwidth issue, but I could be wrong - > I didn't do any PCIe bandwidth measurements. Sometimes it is actually cache operations which take all the time. This needs to invalidate the cache, so that when the memory is then accessed, it get fetched from RAM. On SMP machines, cache invalidation can be expensive, due to all the cross CPU operations. I've actually got better performance by building a UP kernel on some low core count ARM CPUs. There are some tricks which can be played. Do you actually need all 9K? Does the descriptor tell you actually how much is used? You can get a nice speed up if you just unmap 64 bytes for a TCP ACK, rather than the full 9K. Andrew