From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751844AbdHXH3L (ORCPT ); Thu, 24 Aug 2017 03:29:11 -0400 Received: from mail-cys01nam02on0054.outbound.protection.outlook.com ([104.47.37.54]:23242 "EHLO NAM02-CY1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751549AbdHXH2m (ORCPT ); Thu, 24 Aug 2017 03:28:42 -0400 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=nxp.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=fail action=none header.from=nxp.com; From: Madalin Bucur To: , CC: , Subject: [PATCH v3 6/7] Documentation: networking: add RSS information Date: Thu, 24 Aug 2017 10:28:27 +0300 Message-ID: <1503559708-13405-7-git-send-email-madalin.bucur@nxp.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1503559708-13405-1-git-send-email-madalin.bucur@nxp.com> References: <1503559708-13405-1-git-send-email-madalin.bucur@nxp.com> Reply-To: X-EOPAttributedMessage: 0 X-Matching-Connectors: 131480333200577488;(91ab9b29-cfa4-454e-5278-08d120cd25b8);() X-Forefront-Antispam-Report: CIP:192.88.168.50;IPV:NLI;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(336005)(39380400002)(39860400002)(2980300002)(1109001)(1110001)(339900001)(189002)(199003)(50466002)(86362001)(48376002)(104016004)(2950100002)(6666003)(53936002)(43066003)(47776003)(106466001)(33646002)(50986999)(76176999)(105606002)(50226002)(498600001)(5660300001)(54906002)(97736004)(81156014)(8676002)(81166006)(77096006)(85426001)(8936002)(305945005)(2906002)(68736007)(626005)(4326008)(5003940100001)(3450700001)(189998001)(356003)(36756003);DIR:OUT;SFP:1101;SCL:1;SRVR:BN3PR03MB2260;H:tx30smr01.am.freescale.net;FPR:;SPF:Fail;PTR:InfoDomainNonexistent;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BY2FFO11FD007;1:Bkk9ujK1zrOQzLqsZteIvSJHbPhKKdzUAbax7g56Q3YXelIjtL1GORaYFldBXgPXNK2AkVdgHoyQGLQP5gCGBxxMtaUETJbLBJ9deSPRc7F9caeq48Pq37wEKVhya8y/ MIME-Version: 1.0 Content-Type: text/plain X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0fb34049-7e9a-4cf9-726f-08d4eac1bdd3 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(300000503095)(300135400095)(2017052603196)(201703131430075)(201703131517081)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:BN3PR03MB2260; X-Microsoft-Exchange-Diagnostics: 1;BN3PR03MB2260;3:mPsaczV0kWjjusmz7FQxtdjj3Yesvd47A/u/Flbk/MXztjhPTpiNhi276ZJaIEj4DUx2y9gDvDii43yDjmxfXtIq3BRN0kh7YCqewyi5AHQiwMrLzApIWWB7V6iGDZJUw8s/QfvBf6t0YiEAnfGF1DoJsifwY7QK0Lq1AswG1GFTKvxEmTrNUXj1F2FEfRNlVGdMR9IILEWXK5vlY9mHvkSOSHMaVJda2RE5VeCbxcA/seaxROk7HHgQ9TjpqxNhApwFaxYZl52gGRutGvYn4xEzh40v/o7LqROuokUwmRS7STrAPMLXMNVyfJWCWHsgHv2pw5ssmAFYSGva3DE4+Db1UfZQnCBAEFp3tCA8fnU=;25:lZGIJoHh+k6S7x5E1oBlpKQOmmp5evyncEyhvg2C8SHSAroOr6S8KyRZC84d6kp57BXhCZnrSN8cVu3atyfoUMHlgm7ZqF45c0SRtFLoNVnsUJzvPLfAvYtMWXkMkqhPtCW31x511sQBxnbS6476SmTa/2t1vOBjWcBKURUbu0GoKrGerM2zgiwaRMr5F9AYKd8/fQE9p0M8r8DAnaW1kNYinTcORIvTmnd2DpwmW22yfkck5be/pQctBOWzUoJoEU+0QVARUTd5gW2H6L4mPQBkPBhj+B06HeCN4rTj3OtikjA6h2T6oRwd2ppmevxLXDL1WtDnlDpYMeagsn8rVw== X-MS-TrafficTypeDiagnostic: BN3PR03MB2260: X-Microsoft-Exchange-Diagnostics: 1;BN3PR03MB2260;31:FAbV310pAtpzbbao7UYBqzrxuHRi8cPGi4lwOZFF2iKv3fX/5uqQy3uMj1APbvlQFp+2knLCPWAZhXcjhZcnvAuTxpUXtAeu/+4xMP1Bdc1Mrgvi95pINwfmWACDugnQmrn6LnXSu13oWySAephlDguDKhJrYjlYNocBi7R3wFZZua3CxQV3ec9DBrbPRNrNqoX/Y9Gs+PHxeEaqlSXHrl7Pj1PXmbwoHdMIf/5R+/g=;4:8WBQb7J3gshkJi4qa5rNOuxlIHBNYyq/BSsUOiu9tOGf62fKBwkdpIyVQJPhPV9CL0Bju2frExgylFOGCdDVvt9ok8LeXc3V4CYEYJtKVwVcIZmnondPR6jqHURhiIAZTp5eu5z3AwU2CJtbMLC+DzApoLNqfWIblg8npXX1Uy3q0eJW2CitU17jNy2JdnHnQnPSY3vZrUl6RnGJumszNX5+URrr9bRsye4cUPBRq+RnkcYXJDzDu+3uioOQimNwy/fMrRuyDHXNZ6wGJ+HREJi3Uy/mhMJA7vn6r44fjUBkNhTVnKGSN57TUP0x6LF50IfZsKMxQ3P0coayr3BK8id6IYE0pgZWybBp20DRtAMBr5ibBpdRYMZ7MZs9wcdW X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(275809806118684)(211171220733660); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6095135)(601004)(2401047)(5005006)(8121501046)(13016025)(13018025)(100000703101)(100105400095)(3002001)(10201501046)(93006095)(93001095)(6055026)(6096035)(20161123559100)(201703131430075)(201703131441075)(201703131448075)(201703131433075)(201703161259150)(20161123561025)(20161123565025)(20161123563025)(20161123556025)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:BN3PR03MB2260;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(400006)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:BN3PR03MB2260; X-Forefront-PRVS: 04097B7F7F X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;BN3PR03MB2260;23:U/OOK5Nreq9XCPfNHUd4J6P9ps6ma0UyU4Vfwq/9E?= =?us-ascii?Q?S+FQTshj+u9v7N85EP1e5zuc1szWlsJ+4ypVF6Ae2cjdVVWQbdN57eUh5JWp?= =?us-ascii?Q?v0fFZtDDKXOJ2jwsuSPSjAi4AO1J1kHDRA9DvajMsyuaGpMsPIYIhi1BXvAU?= =?us-ascii?Q?mP9BKHP3K+kkhohR1gsgEohRqquH8oLC4Lv0zpKtjGMi+BXRwRWOXEn3bOOK?= =?us-ascii?Q?sMEVxSirLUsSLRsDWQnbOA4o4Ui/G0vYGNBy5v0Sf+24/CQd7JvGqTTx3AF7?= =?us-ascii?Q?AcIIpjROABEmYpxNfzZOhlsRy71Nyv/XXuqcT1leP4Dz9ULNdNDQk7mNcB1W?= =?us-ascii?Q?3kMkusvpjheFJw3HT6CwHwlwrqX+KOs9k30WM6n+HdB/N7BvrIWJNPS/kIqh?= =?us-ascii?Q?ZvfyIxHnW5oSe6KCwlEro5Qg2E849GrmIMJ7OeAjymqiChe6MgS6ytbMNPmc?= =?us-ascii?Q?3Jpue1bQsWYtF2UyE0ZXFqnExGP8CVF2HJ47YnI7ocVHUebpEHNHT5wZ6ii0?= =?us-ascii?Q?cnWTcHTzLVwwDpjMMJOg91m0KgyNLVzlAqhj3mVbVTgKvoIilaiU486JkrIL?= =?us-ascii?Q?lyhBzPk0cOXWv+DBiYdtmjUq4a23beZt8IoVOfSzG0GEWyQYHyR25uu7tfr6?= =?us-ascii?Q?FtdkwYDTFtJvMJIr5+YKDpVlre7pLnH/ktJvmeAgninv0BpnXvHuZLkvsU/l?= =?us-ascii?Q?YaYETX77Gwc1XgaJNG7wT0LXBYQp8F1s+tmqDm/pFvmIKkYbdRhz+oAXQER5?= =?us-ascii?Q?Nc9TEkM2qW5OhagD+ddDYtC4ucU/UnPkHun9PagZ0Y2jE3BzKUxPnL2m0DjK?= =?us-ascii?Q?1si0wXBmhd0IGJrUf5E/cgGxCRJ/ySg6Mt5RRvgDVbaTsygRLJ+3zL1DWycc?= =?us-ascii?Q?/T255FTVwsvprAs038+Op+G0vRqQLfth1n+EjoWSzgMqHqQXJmbb4R8CGvqB?= =?us-ascii?Q?6y7YQOlxS9xzIkilU901IlJajr4pAnilf9rQQ/VSomYL5Ivpnbt2NaD0Z1lg?= =?us-ascii?Q?g6NtKT337rjwASIXYVXFPKEtkJx+3afjAbk7P/OZ0MZLB7XEGGcaLxvWDCIg?= =?us-ascii?Q?rq3LvHGXedqZXonJRmJD5qW2jSryOyxaagemor4ebH25qh4Jw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN3PR03MB2260;6:2RbawEHp6E9BI9u1XTtdLeCszeUyaboB2Br7kG9kN5Tv9BB5qQIUCipTNaRWPhUoBZMM8RUVhsmZj+EnAawwtXoUQ/hGhwHiidVH5hzq/vvGJYuQ7iELBWsXUiZXu7HuB4H+NTqOcYD7NUxzcjJu9NiVpqDrPBqlOPH4tPW91IVr77GIK4C9csYre8a3UJaO6369OtIpi9iao+SLcBhwW1lelB6FRpsrtD4Kq8U9nnyJWZ3HpUFD4lKsAPWm/BpXXfxcfc1MpX+clHMzj4vuessUeBhElIGbdGq7Lgue3hvAkr/kIxc4UrlvfeqOHPAz/8hCDTikIkP7qyzvQZz67Q==;5:am4TZQqubDErF22XofCl3Ml4f61yhvsWxHXtGmqPW3nBbOkWohvIUqu+AEJ4Sak59y3YKdv3G4E4I5bDBKKvtNsxEgsA8pKDtftZYrE/GuPSO54Jg9H10cYc8OwtjqF3OSqyXEwvDtDRDBN4sko+fA==;24:Hl3cZYYiWx/PFal60+bZLIaiCUWNTL9b5+gdJy8oiROnGEYDxAD/jbdRG/YsamLXmLzRfyOYOW5YHDGKylhr2SnaynNFt23S4o/1k7hG8LM=;7:JaWnWDSqjr2IaAtJgQq+QWl+yycE56uiE7yp+zqK7ax/+UHrZSn9nQTHbkYhH4Pha/QTDld0wnFz9gxZ6bGkalUwOvai23UPsJkpF7Y/jIGbqfaJK+Q6aaNrpWBPOUbfiGeh0vtGAlOkfWEE1UxdidDo4cs0UhJhBeyy1loXTS0E1jNr0FfYS783sbtqO+Ak/AZmS0FIgx2KtxbmxcVig/dpy3fqEVEJE39HGsJ1VSI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Aug 2017 07:28:39.8549 (UTC) X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e;Ip=[192.88.168.50];Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR03MB2260 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Signed-off-by: Madalin Bucur --- Documentation/networking/dpaa.txt | 68 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/Documentation/networking/dpaa.txt b/Documentation/networking/dpaa.txt index 76e016d..f88194f 100644 --- a/Documentation/networking/dpaa.txt +++ b/Documentation/networking/dpaa.txt @@ -13,6 +13,7 @@ Contents - Configuring DPAA Ethernet in your kernel - DPAA Ethernet Frame Processing - DPAA Ethernet Features + - DPAA IRQ Affinity and Receive Side Scaling - Debugging DPAA Ethernet Overview @@ -147,7 +148,10 @@ gradually. The driver has Rx and Tx checksum offloading for UDP and TCP. Currently the Rx checksum offload feature is enabled by default and cannot be controlled through -ethtool. +ethtool. Also, rx-flow-hash and rx-hashing was added. The addition of RSS +provides a big performance boost for the forwarding scenarios, allowing +different traffic flows received by one interface to be processed by different +CPUs in parallel. The driver has support for multiple prioritized Tx traffic classes. Priorities range from 0 (lowest) to 3 (highest). These are mapped to HW workqueues with @@ -166,6 +170,68 @@ classes as follows: tc qdisc add dev root handle 1: \ mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1 +DPAA IRQ Affinity and Receive Side Scaling +========================================== + +Traffic coming on the DPAA Rx queues or on the DPAA Tx confirmation +queues is seen by the CPU as ingress traffic on a certain portal. +The DPAA QMan portal interrupts are affined each to a certain CPU. +The same portal interrupt services all the QMan portal consumers. + +By default the DPAA Ethernet driver enables RSS, making use of the +DPAA FMan Parser and Keygen blocks to distribute traffic on 128 +hardware frame queues using a hash on IP v4/v6 source and destination +and L4 source and destination ports, in present in the received frame. +When RSS is disabled, all traffic received by a certain interface is +received on the default Rx frame queue. The default DPAA Rx frame +queues are configured to put the received traffic into a pool channel +that allows any available CPU portal to dequeue the ingress traffic. +The default frame queues have the HOLDACTIVE option set, ensuring that +traffic bursts from a certain queue are serviced by the same CPU. +This ensures a very low rate of frame reordering. A drawback of this +is that only one CPU at a time can service the traffic received by a +certain interface when RSS is not enabled. + +To implement RSS, the DPAA Ethernet driver allocates an extra set of +128 Rx frame queues that are configured to dedicated channels, in a +round-robin manner. The mapping of the frame queues to CPUs is now +hardcoded, there is no indirection table to move traffic for a certain +FQ (hash result) to another CPU. The ingress traffic arriving on one +of these frame queues will arrive at the same portal and will always +be processed by the same CPU. This ensures intra-flow order preservation +and workload distribution for multiple traffic flows. + +RSS can be turned off for a certain interface using ethtool, i.e. + + # ethtool -N fm1-mac9 rx-flow-hash tcp4 "" + +To turn it back on, one needs to set rx-flow-hash for tcp4/6 or udp4/6: + + # ethtool -N fm1-mac9 rx-flow-hash udp4 sfdn + +There is no independent control for individual protocols, any command +run for one of tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 is +going to control the rx-flow-hashing for all protocols on that interface. + +Besides using the FMan Keygen computed hash for spreading traffic on the +128 Rx FQs, the DPAA Ethernet driver also sets the skb hash value when +the NETIF_F_RXHASH feature is on (active by default). This can be turned +on or off through ethtool, i.e.: + + # ethtool -K fm1-mac9 rx-hashing off + # ethtool -k fm1-mac9 | grep hash + receive-hashing: off + # ethtool -K fm1-mac9 rx-hashing on + Actual changes: + receive-hashing: on + # ethtool -k fm1-mac9 | grep hash + receive-hashing: on + +Please note that Rx hashing depends upon the rx-flow-hashing being on +for that interface - turning off rx-flow-hashing will also disable the +rx-hashing (without ethtool reporting it as off as that depends on the +NETIF_F_RXHASH feature flag). + Debugging ========= -- 2.1.0