From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752875AbcAMVSM (ORCPT ); Wed, 13 Jan 2016 16:18:12 -0500 Received: from out03.mta.xmission.com ([166.70.13.233]:37396 "EHLO out03.mta.xmission.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751425AbcAMVSJ (ORCPT ); Wed, 13 Jan 2016 16:18:09 -0500 From: ebiederm@xmission.com (Eric W. Biederman) To: Hans Westgaard Ry Cc: Hannes Frederic Sowa , David Laight , "David S. Miller" , Alexey Kuznetsov , James Morris , Hideaki YOSHIFUJI , Patrick McHardy , Alexei Starovoitov , Jiri Pirko , Eric Dumazet , Daniel Borkmann , Nicolas Dichtel , Salam Noureddine , Jarod Wilson , Toshiaki Makita , Julian Anastasov , Ying Xue , Craig Gallek , Mel Gorman , Edward Jee , Julia Lawall , "netdev\@vger.kernel.org" , "linux-kernel\@vger.kernel.org" , Haakon Bugge , Knut Omang , Wei Lin Guay , Santosh Shilimkar , Yuval Shaia References: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com> <063D6719AE5E284EB5DD2968C1650D6D1CCBE5AA@AcuExch.aculab.com> <568F87AC.60405@oracle.com> <568FA1E5.7060204@stressinduktion.org> <569657D8.1020807@oracle.com> Date: Wed, 13 Jan 2016 15:07:11 -0600 In-Reply-To: <569657D8.1020807@oracle.com> (Hans Westgaard Ry's message of "Wed, 13 Jan 2016 14:57:44 +0100") Message-ID: <877fjd89z4.fsf@x220.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-AID: U2FsdGVkX19s5tw8JMyWRRwfaaZl4l1PBYRf/1u8TlQ= X-SA-Exim-Connect-IP: 97.121.81.63 X-SA-Exim-Mail-From: ebiederm@xmission.com X-Spam-Report: * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP * 0.0 TVD_RCVD_IP Message was received from an IP address * 0.0 T_TM2_M_HEADER_IN_MSG BODY: No description available. * 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60% * [score: 0.5000] * -0.0 DCC_CHECK_NEGATIVE Not listed in DCC * [sa05 1397; Body=1 Fuz1=1 Fuz2=1] X-Spam-DCC: XMission; sa05 1397; Body=1 Fuz1=1 Fuz2=1 X-Spam-Combo: ;Hans Westgaard Ry X-Spam-Relay-Country: X-Spam-Timing: total 356 ms - load_scoreonly_sql: 0.15 (0.0%), signal_user_changed: 10 (2.8%), b_tie_ro: 7 (1.9%), parse: 1.39 (0.4%), extract_message_metadata: 16 (4.6%), get_uri_detail_list: 3.3 (0.9%), tests_pri_-1000: 6 (1.8%), tests_pri_-950: 1.04 (0.3%), tests_pri_-900: 0.88 (0.2%), tests_pri_-400: 37 (10.3%), check_bayes: 35 (9.9%), b_tokenize: 10 (2.7%), b_tok_get_all: 11 (3.1%), b_comp_prob: 6 (1.5%), b_tok_touch_all: 5 (1.5%), b_finish: 0.99 (0.3%), tests_pri_0: 273 (76.7%), check_dkim_signature: 0.51 (0.1%), check_dkim_adsp: 4.0 (1.1%), tests_pri_500: 6 (1.8%), rewrite_mail: 0.00 (0.0%) Subject: Re: [PATCH] net: add per device sg_max_frags for skb X-Spam-Flag: No X-SA-Exim-Version: 4.2.1 (built Wed, 24 Sep 2014 11:00:52 -0600) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hans Westgaard Ry writes: > On 01/08/2016 12:47 PM, Hannes Frederic Sowa wrote: >> On 08.01.2016 10:55, Hans Westgaard Ry wrote: >>> >>> >>> On 01/06/2016 02:59 PM, David Laight wrote: >>>> From: Hans Westgaard Ry >>>>> Sent: 06 January 2016 13:16 >>>>> Devices may have limits on the number of fragments in an skb they >>>>> support. Current codebase uses a constant as maximum for number of >>>>> fragments (MAX_SKB_FRAGS) one skb can hold and use. >>>>> >>>>> When enabling scatter/gather and running traffic with many small >>>>> messages the codebase uses the maximum number of fragments and thereby >>>>> violates the max for certain devices. >>>>> >>>>> An example of such a violation is when running IPoIB on a HCA >>>>> supporting 16 SGE on an architecture with 4K pagesize. The >>>>> MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another >>>>> segment we end up with send_requests with 18 SGE resulting in >>>>> kernel-panic. >>>>> >>>>> The patch allows the device to limit the maximum number fragments used >>>>> in one skb. >>>> This doesn't seem to me to be the correct way to fix this. >>>> Anything that adds an extra fragment (in this case IPoIB) should allow >>>> for the skb already having the maximum number of fragments. >>>> Fully linearising the skb is overkill, but I think the first fragment >>>> can be added to the linear part of the skb. >>>> >>>> David >>>> >>>> >>> When IpoIB handles a skb-request it converts fragments to SGEs to >>> be handled by a HCA. >>> The problem arises when the HCA have a limited number of SGEs less than >>> MAX_SKB_FRAGS. >>> (it gets a little worse since IPoIB need to yet another segment) >>> I have not found any easy way of fixing this with currenct codebase. >> >> I think because of the complex forwarding nature, a global counter which >> driver's can reduce during initialization time is the only solution I see >> right now without changing the layout of the skb later on. >> >> Unfortunately this doesn't resolve the cases were virtual machines inject gso >> skbs, for those there still needs to be a slow path to do the reformatting of >> the skb. :/ >> >> Bye, >> Hannes >> >> > The use-case for this patch is an application which sends many small messages, > by write(2) on a TCP socket which has Nagle enabled. A scatter-gather capable > NIC (potentially also supporting tso) will then be asked to send an skb > containing up to MAX_SKB_FRAGS worth of fragments (17 considering a 4kb page > size, hypothetically 65 considering an arch supporting 1kb page size). > > Now, if the NIC hardware supports less _gather-fragments_, said hardware must > run with scatter-gather disabled - or - the NIC driver has to implement a > partial linearization of the skb to reduce #frags to what the hardware > supports. The latter is far from elegant, and must be implemented in all NIC > drivers which have this restriction. > > This patch provides the flexibility to choose the maximum number of fragments > that can be passed down to the NIC in order to > utilize the NIC SG hardware features. > > > In our view we are discussing two different issues: > > 1. Is it reasonable that a NIC can restrict #frags in an skb when > transmitting? > 2. If yes to the above, how is this implemented the best possible way. > > Thanks a lot for feedback on the implementation from David Laight, Eric Dumazet > and Hannes Fredreric Sowa. > > What do you think? *Scratches my head* Why doesn't someone fix the infiniband firmware so that it supports more scatter gather entries? Last I looked everything like this in infiniband was all implemented in firmware and there is only one vendor to pick on, so it should be comparatively easy to just fix the hardware so it does not have this limitation. Eric