From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E0763E51E2 for ; Mon, 18 May 2026 09:41:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779097289; cv=none; b=g4Cj+E0YFAmuojC2Ng+VtFAmTp74iF7cGRzR+BFnwKSV1F6gIgw1YdSHAKu170Xozxq6SxliVbqTg8FuiQ5rnHrSkjBJ51APce5kOm7xMKZfn7klr2SBgcKXIA9qZ1+QmQJ+B2G7toMRDLwGxeaNZeicubJeBieungGWeTym/lw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779097289; c=relaxed/simple; bh=b7kTseQV9X7ouOaMCcSJ2B/4RKvW5upBUS0xlwJWHfo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ZeI7lfzDY4+pGNTICdTJMUqTVgVk/nygSEngCqSS/hSWrBXbUCPhPtNg5k1iLGAXorsFLqvmCw056M2mmosigPHlFQ35UX/dMyAb4NvSYisoEmVL1MSQrzmw2VMFc6KoB0tIxDPZ6tbdgcyAKyrDACbR1LUD7Kxipz0JhOjnix0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=IBKoT+rD; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="IBKoT+rD" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Transfer-Encoding: Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date: Sender:Reply-To:Content-ID:Content-Description; bh=H1P4UlUMv3DejLniEmn9pL0PyLufHjW8/db7z6KlByY=; b=IBKoT+rDjh+oAYRcd4BpUz6hbU ONoCyHs1xgrMUVX8CJkUs5gr1h6utNdvEWtKUMCXBXyr9FZudyp3uDg2BNs1KIwaACHevRadYfdLV 1zqkHchpFmlj8zuDUbx/K7dE/tBvT/LpzVVV8YX4w5SrDtVkuFXYkbg2bTxXzAEH3RtS49qry62cI iTcCQB4Atf1lgJeMsz7/ACALSp09ci4WzSr995Ru8pbakH+z3+0YlfhVrmgW2CeyENHNzVh2GIu0V SYA7pY8PfyK4OLEmmiqj37qWnQvWuXwfxMmln8shyUZi2wP3VrEwkGPxTfxhUQfS8dKANIsuohKqc 21IPSPKg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.99.1 #2 (Red Hat Linux)) id 1wOuTT-00000004aRI-0VZN; Mon, 18 May 2026 09:41:23 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id BF2E03007A4; Mon, 18 May 2026 11:41:21 +0200 (CEST) Date: Mon, 18 May 2026 11:41:21 +0200 From: Peter Zijlstra To: =?utf-8?B?0JrQvtC90YHRgtCw0L3RgtC40L0g0JzQuNGF0LDQudC70L7Qsg==?= Cc: adrian.hunter@intel.com, Mark Rutland , linux-kernel@vger.kernel.org Subject: Re: perf AUX: race causes poll() hang Message-ID: <20260518094121.GT3102624@noisy.programming.kicks-ass.net> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Fri, May 15, 2026 at 01:35:48AM +0300, Константин Михайлов wrote: > Hello Peter and Adrian, > > I'd like to report a potential race condition in perf AUX buffer handling. > > AUX tracing is designed to allow the tracee continue running when the AUX > buffer fills. The PMU driver must disable tracing when AUX buffer is full. > Typically, it schedules IRQ work to disable the event later. Meanwhile, a > typical tracer's workflow looks like: poll() on perf FDs, consume the data, > re-enable the event via PERF_EVENT_IOC_ENABLE ioctl(), then poll() again. > > Given this, the following race is possible: > ------------------- > | CPU #0 | CPU #1 | > | tracee | tracer | > ------------------- > | ** | | tracee fills the AUX buffer completely with some data > ------------------- > | ** | | PMU driver updates aux_head accordingly and schedules IRQ works > | ** | | to disable the event and wake up the tracer (setting rb->poll in > | ** | | perf_output_wakeup() along the way) > ------------------- > | | ** | tracer consumes all the data from AUX buffer, > | | ** | thus clears rb->poll in perf_poll() > ------------------- > | | ** | tracer re-enables the tracing (the event is still active, > | | ** | so ioctl(...) returns immediately) > ------------------- > | | ** | tracer starts poll()'ing the AUX buffer again > ------------------- > | ** | | IRQ work handler finally disables the event and > | ** | | wakes up tracer > ------------------- > | | ** | tracer obtains zero rb->poll and continues polling > ------------------- > As a result, tracee runs without PMU tracing, and tracer's poll() will > never be woken up unless it has some timeout. > > I reproduced this on an x86 machine with intel_pt and kernel v6.17. > Reproducing this race on the vanilla kernel is timing-sensitive, so I added > 30 ms delay in the error path in intel_pt_interrupt() when > pt_buffer_reset_markers() returns an error - this delay widens the window > between aux_head update and actual event disable in IRQ work handler. I'm > not sure that pt_buffer_reset_markers()'s error means that buffer > overflowed, but this error branch is taken sometimes and all needed IRQ > works are scheduled during a call to perf_aux_output_end(). I also added 3 > ms delay in perf in __auxtrace_mmap__read() before itr->read_finish(), > ensuring the ioctl() falls into that window. With these changes, some perf > runs collected smaller traces than usual. I added traceprints to intel_pt > driver and enabled tracing for sys_poll and sys_ioctl, which confirmed the > exact sequence described above. The problem was sometimes mitigated by > tracee migration to another cpu (as perf creates an event for every cpu, > the event is re-enabled by kernel when it is set on a new cpu). Otherwise, > tracee stayed on the same cpu and tracer hung on poll() until tracee exited. > > Could you please confirm if this analysis is correct? Should we move > setting of rb->poll *after* the event is disabled in IRQ work handler? I *think* (its been a minute since I looked at this code), that you're right. Does something like the below cure things? --- kernel/events/core.c | 54 +++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 24 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 7935d5663944..490407618f36 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -2677,6 +2677,9 @@ static void __perf_event_disable(struct perf_event *event, struct perf_event_context *ctx, void *info) { + if (event->pending_disable) + event->pending_disable = 0; + if (event->state < PERF_EVENT_STATE_INACTIVE) return; @@ -3278,32 +3281,37 @@ static void _perf_event_enable(struct perf_event *event) { struct perf_event_context *ctx = event->ctx; - raw_spin_lock_irq(&ctx->lock); - if (event->state >= PERF_EVENT_STATE_INACTIVE || - event->state < PERF_EVENT_STATE_ERROR) { -out: - raw_spin_unlock_irq(&ctx->lock); - return; - } + scoped_guard (raw_spinlock_irq, &ctx->lock) { + if (event->state < PERF_EVENT_STATE_ERROR) + return; - /* - * If the event is in error state, clear that first. - * - * That way, if we see the event in error state below, we know that it - * has gone back into error state, as distinct from the task having - * been scheduled away before the cross-call arrived. - */ - if (event->state == PERF_EVENT_STATE_ERROR) { /* - * Detached SIBLING events cannot leave ERROR state. + * If the event is in error state, clear that first. + * + * That way, if we see the event in error state below, we know that it + * has gone back into error state, as distinct from the task having + * been scheduled away before the cross-call arrived. */ - if (event->event_caps & PERF_EV_CAP_SIBLING && - event->group_leader == event) - goto out; + if (event->state == PERF_EVENT_STATE_ERROR) { + /* + * Detached SIBLING events cannot leave ERROR state. + */ + if (event->event_caps & PERF_EV_CAP_SIBLING && + event->group_leader == event) + return; - event->state = PERF_EVENT_STATE_OFF; + event->state = PERF_EVENT_STATE_OFF; + } + + if (event->pending_disable) + event->pending_disable = 0; + + /* + * Already running, nothing to do. + */ + if (event->state >= PERF_EVENT_STATE_INACTIVE) + return; } - raw_spin_unlock_irq(&ctx->lock); event_function_call(event, __perf_event_enable, NULL); } @@ -7612,10 +7620,8 @@ static void __perf_pending_disable(struct perf_event *event) * Yay, we hit home and are in the context of the event. */ if (cpu == smp_processor_id()) { - if (event->pending_disable) { - event->pending_disable = 0; + if (event->pending_disable) perf_event_disable_local(event); - } return; }