From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1751431AbbEJN0k (ORCPT <rfc822;w@1wt.eu>);
	Sun, 10 May 2015 09:26:40 -0400
Received: from mail-wg0-f45.google.com ([74.125.82.45]:33144 "EHLO
	mail-wg0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751171AbbEJN0j (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Sun, 10 May 2015 09:26:39 -0400
Date: Sun, 10 May 2015 16:26:34 +0300
From: Alexey Dobriyan <adobriyan@gmail.com>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Cc: akpm@linux.foundation.org, mmarek@suse.cz, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3] tags: much faster, parallel "make tags"
Message-ID: <20150510132633.GA1717@p183.telecom.by>
References: <20150508132606.GA16613@p183.telecom.by>
 <554D9606.7090908@draigBrady.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <554D9606.7090908@draigBrady.com>
User-Agent: Mutt/1.5.23 (2014-03-12)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sat, May 09, 2015 at 06:07:18AM +0100, Pádraig Brady wrote:
> On 08/05/15 14:26, Alexey Dobriyan wrote:

> >  exuberant()
> >  {
> > -	all_target_sources | xargs $1 -a                        \
> > +	rm -f .make-tags.*
> > +
> > +	all_target_sources >.make-tags.src
> > +	NR_CPUS=$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1)
> 
> `nproc` is simpler and available since coreutils 8.1 (2009-11-18)

nproc was discarded because getconf is standartized.

> > +	NR_LINES=$(wc -l <.make-tags.src)
> > +	NR_LINES=$((($NR_LINES + $NR_CPUS - 1) / $NR_CPUS))
> > +
> > +	split -a 6 -d -l $NR_LINES .make-tags.src .make-tags.src.
> 
> `split -d -nl/$(nproc)` is simpler and available since coreutils 8.8 (2010-12-22)

-nl/ can't count and always make first file somewhat bigger, which is
suspicious. What else it can't do right?

> > +	sort .make-tags.* >>$2
> > +	rm -f .make-tags.*
> 
> Using sort --merge would speed up significantly?

By ~1 second, yes.

> Even faster would be to get sort to skip the header lines, avoiding the need for sed.
> It's a bit awkward and was discussed at:
> http://lists.gnu.org/archive/html/coreutils/2013-01/msg00027.html
> Summarising that, is if not using merge you can:
> 
>   tlines=$(($(wc -l < "$2") + 1))
>   tail -q -n+$tlines .make-tags.* | LC_ALL=C sort >>$2
> 
> Or if merge is appropriate then:
> 
>   tlines=$(($(wc -l < "$2") + 1))
>   eval "eval LC_ALL=C sort -m '<(tail -n+$tlines .make-tags.'{1..$(nproc)}')'" >>$2

Might as well teach ctags to do real parallel processing.
LC_* are set by top level Makefile.

> p.p.s. You may want to `trap EXIT cleanup` to rm -f .make-tags.*

The real question is how to kill ctags reliably.
Naive

	trap 'kill $(jobs -p); rm -f .make-tags.*' TERM INT

doesn't work.

Files are removed, but processes aren't.