From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965345AbcKODO6 (ORCPT ); Mon, 14 Nov 2016 22:14:58 -0500 Received: from mx1.redhat.com ([209.132.183.28]:40282 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S941015AbcKODOy (ORCPT ); Mon, 14 Nov 2016 22:14:54 -0500 Subject: Re: [PATCH 1/3] tuntap: rx batching To: "Michael S. Tsirkin" References: <1478677113-13126-1-git-send-email-jasowang@redhat.com> <20161109183259-mutt-send-email-mst@kernel.org> <20161111053048-mutt-send-email-mst@kernel.org> <58254654.4000501@gmail.com> <2b357b06-f749-5897-f882-aa3121e49e89@redhat.com> <20161111181754-mutt-send-email-mst@kernel.org> Cc: John Fastabend , netdev@vger.kernel.org, linux-kernel@vger.kernel.org From: Jason Wang Message-ID: <39c36d36-9029-5d1f-496f-6ff404c3b77a@redhat.com> Date: Tue, 15 Nov 2016 11:14:48 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161111181754-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 15 Nov 2016 03:14:53 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2016年11月12日 00:20, Michael S. Tsirkin wrote: > On Fri, Nov 11, 2016 at 12:28:38PM +0800, Jason Wang wrote: >> >> On 2016年11月11日 12:17, John Fastabend wrote: >>> On 16-11-10 07:31 PM, Michael S. Tsirkin wrote: >>>>> On Fri, Nov 11, 2016 at 10:07:44AM +0800, Jason Wang wrote: >>>>>>> >>>>>>> On 2016年11月10日 00:38, Michael S. Tsirkin wrote: >>>>>>>>> On Wed, Nov 09, 2016 at 03:38:31PM +0800, Jason Wang wrote: >>>>>>>>>>> Backlog were used for tuntap rx, but it can only process 1 packet at >>>>>>>>>>> one time since it was scheduled during sendmsg() synchronously in >>>>>>>>>>> process context. This lead bad cache utilization so this patch tries >>>>>>>>>>> to do some batching before call rx NAPI. This is done through: >>>>>>>>>>> >>>>>>>>>>> - accept MSG_MORE as a hint from sendmsg() caller, if it was set, >>>>>>>>>>> batch the packet temporarily in a linked list and submit them all >>>>>>>>>>> once MSG_MORE were cleared. >>>>>>>>>>> - implement a tuntap specific NAPI handler for processing this kind of >>>>>>>>>>> possible batching. (This could be done by extending backlog to >>>>>>>>>>> support skb like, but using a tun specific one looks cleaner and >>>>>>>>>>> easier for future extension). >>>>>>>>>>> >>>>>>>>>>> Signed-off-by: Jason Wang >>>>>>>>> So why do we need an extra queue? >>>>>>> The idea was borrowed from backlog to allow some kind of bulking and avoid >>>>>>> spinlock on each dequeuing. >>>>>>> >>>>>>>>> This is not what hardware devices do. >>>>>>>>> How about adding the packet to queue unconditionally, deferring >>>>>>>>> signalling until we get sendmsg without MSG_MORE? >>>>>>> Then you need touch spinlock when dequeuing each packet. >>> Random thought, I have a cmpxchg ring I am using for the qdisc work that >>> could possibly replace the spinlock implementation. I haven't figured >>> out the resizing API yet because I did not need it but I assume it could >>> help here and let you dequeue multiple skbs in one operation. >>> >>> I can post the latest version if useful or an older version is >>> somewhere on patchworks as well. >>> >>> .John >>> >>> >> Look useful here, and I can compare the performance if you post. >> >> A question is can we extend the skb_array to support that? >> >> Thanks > I'd like to start with simple patch adding napi with one queue, then add > optimization patches on top. The point is tun is using backlog who uses two queues (process_queue and input_pkt_queue). How about something like: 1) NAPI support with skb_array 2) MSG_MORE support 3) other optimizations on top ? > > One issue that comes to mind is that write queue limits > are byte based, they do not count packets unlike tun rx queue. I'm not sure I get the issue, write queue is not exported and only used for batching. We probably need an internal limit in tun to avoid OOM attacker from guest. Thanks