From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753197AbdHDRL4 (ORCPT ); Fri, 4 Aug 2017 13:11:56 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:33844 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752608AbdHDRLy (ORCPT ); Fri, 4 Aug 2017 13:11:54 -0400 Message-ID: <1501866711.25002.46.camel@edumazet-glaptop3.roam.corp.google.com> Subject: Re: [PATCH net-next] net: dsa: User per-cpu 64-bit statistics From: Eric Dumazet To: Florian Fainelli Cc: netdev@vger.kernel.org, davem@davemlot.net, Andrew Lunn , Vivien Didelot , "David S. Miller" , open list Date: Fri, 04 Aug 2017 10:11:51 -0700 In-Reply-To: <455c39d3-302e-68ab-3812-112c48ed48c2@gmail.com> References: <20170804043328.4730-1-f.fainelli@gmail.com> <1501824968.25002.35.camel@edumazet-glaptop3.roam.corp.google.com> <455c39d3-302e-68ab-3812-112c48ed48c2@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 2017-08-04 at 08:51 -0700, Florian Fainelli wrote: > On 08/03/2017 10:36 PM, Eric Dumazet wrote: > > On Thu, 2017-08-03 at 21:33 -0700, Florian Fainelli wrote: > >> During testing with a background iperf pushing 1Gbit/sec worth of > >> traffic and having both ifconfig and ethtool collect statistics, we > >> could see quite frequent deadlocks. Convert the often accessed DSA slave > >> network devices statistics to per-cpu 64-bit statistics to remove these > >> deadlocks and provide fast efficient statistics updates. > >> > > > > This seems to be a bug fix, it would be nice to get a proper tag like : > > > > Fixes: f613ed665bb3 ("net: dsa: Add support for 64-bit statistics") > > Right, should have been added, thanks! > > > > > Problem here is that if multiple cpus can call dsa_switch_rcv() at the > > same time, then u64_stats_update_begin() contract is not respected. > > This is really where I struggled understanding what is wrong in the > non-per CPU version, my understanding is that we have: > > - writers for xmit executes in process context > - writers for receive executes from NAPI (from the DSA's master network > device through it's own NAPI doing netif_receive_skb -> netdev_uses_dsa > -> netif_receive_skb) > > readers should all execute in process context. The test scenario that > led to a deadlock involved running iperf in the background, having a > while loop with both ifconfig and ethtool reading stats, and somehow > when iperf exited, either reader would just be locked. So I guess this > leaves us with the two writers not being mutually excluded then, right? You could add a debug version of u64_stats_update_begin() doing int ret = atomic_inc((atomic_t *)syncp); BUG_ON(ret & 1); And u64_stats_update_end() int ret = atomic_inc((atomic_t *)syncp); BUG_ON(!(ret & 1)); We probably could have a CONFIG_DEBUG_U64_STATS that could be used on 64bit kernels as well...