From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 943C6C0044C for ; Wed, 7 Nov 2018 06:13:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 609DA20827 for ; Wed, 7 Nov 2018 06:13:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 609DA20827 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=davemloft.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727397AbeKGPmq (ORCPT ); Wed, 7 Nov 2018 10:42:46 -0500 Received: from shards.monkeyblade.net ([23.128.96.9]:60026 "EHLO shards.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726516AbeKGPmq (ORCPT ); Wed, 7 Nov 2018 10:42:46 -0500 Received: from localhost (unknown [IPv6:2601:601:9f80:35cd::cf9]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: davem-davemloft) by shards.monkeyblade.net (Postfix) with ESMTPSA id 67D1513FD10BE; Tue, 6 Nov 2018 22:13:52 -0800 (PST) Date: Tue, 06 Nov 2018 22:13:49 -0800 (PST) Message-Id: <20181106.221349.1296959035550004994.davem@davemloft.net> To: jolsa@redhat.com Cc: acme@kernel.org, linux-kernel@vger.kernel.org, namhyung@kernel.org, jolsa@kernel.org Subject: Re: [PATCH RFC] hist lookups From: David Miller In-Reply-To: <20181106204255.GB25020@krava> References: <20181105.200321.1569039579935458312.davem@davemloft.net> <20181105.205342.1568518551892125916.davem@davemloft.net> <20181106204255.GB25020@krava> X-Mailer: Mew version 6.8 on Emacs 26.1 Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.5.12 (shards.monkeyblade.net [149.20.54.216]); Tue, 06 Nov 2018 22:13:52 -0800 (PST) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Jiri Olsa Date: Tue, 6 Nov 2018 21:42:55 +0100 > I pushed that fix in perf/fixes branch, but I'm still occasionaly > hitting the namespace crash.. working on it ;-) Jiri, how can this new scheme work without setting copy_on_queue for the queued_events we use here? I don't see copy_on_queue being set and that means the queued event structures reference the event memory directly in the mmaps, after the mmap thread has released them back to the queue. That means new events can come in to the mmap ring and overwrite what was there previously, maybe even while deliver_event() is in the middle of parsing the event. Setting copy_on_queue for data[0] and data[1] makes all of the crashes go away for me. I get a lot of "[unknown]" shared objects shortly after perf top starts up during a full workload. I've been wondering about one side effect of how the mmap queues are processed, consider the following: cpu 0 cpu 1 exec create new mmap2 events scheduled to cpu 0 for whatever reason sample 1 sample 2 And let's say that perf top is backlogged processing the mmap ring of events generated for cpu 0, and sees sample 1 and sample 2 before getting to any of cpu 1's events. This means the thread and map and symbol objects won't exist and we'll get those '[Unknown]' histogram entries, and they won't go away. When it finally stops looping over the mmap ring for cpu 0's events it gets to cpu 1's mmap ring and sees the exec and mmap2 events but at that point it's far too late. I surmise from what I see with perf top right now that this happens a lot.