From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B1E71E5724 for ; Fri, 12 Jun 2026 08:54:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781254446; cv=none; b=Sg9RXlFbN5XacuRSDVG9D7L7q7COYOZ1mg6veJjyxbbpCyeGD0c9R+oPMLjTwRlKqPI5JJvQUzHo5JN0Y9yzo1wSPJEkgPq2asGrIENsN1vcXtc7qPKskuRrCCfROjzJmLgR22J9JKA+ir6OsZWY2nSs3DuU3hhyUQSlax1mCA4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781254446; c=relaxed/simple; bh=n8K/b72C/9MBI+mxx/ZG41711yy+Vsm0lOk+p/lYzq8=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=ifZg4HP8y+N/Z9bdlnk3BF8vI7txACEApJVCKUqZfKK5cCDVcWB0L5jIZUR9Mh5KC9Flml//1DIP0ysjclOyecSPnBeUhszqcIMAwzDBaQ2tzqq+Igtx/FLsEpGysDQfE8VAUb+VISpQTa2l4De8bVD6tUZP0TcKd595kAXFNMI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=aSVmI3da; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=bFvR6pMh; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aSVmI3da"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="bFvR6pMh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1781254444; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3h3Z17YlU+JUM0dzmHKckOTf08CDPs3EE5yZwLDugEw=; b=aSVmI3dakwCE8iW9BPnALUA9QJ2HKdkOZBSgqGim7n9lJMK/rGF4j19D0xwe3j2+2tG08j dtOjSGsQxxpKhThtjmlT3uz/oAMxj6qngEl8/Qv5ptszYRBLeTXgvPO6PYmmdogud4ykl6 TKE9nLYrWv8SCY7XZpLEdO5A3Fy0Ayk= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-310-fKjMJUQPOTyEiMpVISRyXQ-1; Fri, 12 Jun 2026 04:54:03 -0400 X-MC-Unique: fKjMJUQPOTyEiMpVISRyXQ-1 X-Mimecast-MFC-AGG-ID: fKjMJUQPOTyEiMpVISRyXQ_1781254442 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-45eec2ba533so456223f8f.3 for ; Fri, 12 Jun 2026 01:54:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1781254442; x=1781859242; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=3h3Z17YlU+JUM0dzmHKckOTf08CDPs3EE5yZwLDugEw=; b=bFvR6pMhmY33DwdYZvklQg0nDKLs+0yL04KPPWO9pEAbDCclk5vj4VTW64Ql4V/5ot AJUmgydaVeOFhjZiMXpgO61HeVWB0pS7M0iKKufrZkb+yRc4IFWQg9RKOiGVcFv0aYgY cJ90ON7ODvCgUejXigGcOmruZ0Z3J/8IUr8bhergL9Eppu0vkK1MhvSm8NIwhCYruRyd stgoVCcs6aIzSPG2zEjmTiVqIu4cMTFgC33BauW+aUowSt2XFCPNvx72y1i/O/N5CkPI j+txgijZxgOAwahvDOLP66epF6bhjilfWBZt7dP34KkmnRq3Yf6cEt0wSgDivBmMeW5O Dzsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781254442; x=1781859242; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=3h3Z17YlU+JUM0dzmHKckOTf08CDPs3EE5yZwLDugEw=; b=CI/SJbxjMURVHi7GY7PylswvS/oQX7bCwlPUI3SK9p+hIBsTyXJHLve9NsYCXBtZT6 Xaqg4u7eETZCU/4AM1mow2cK2abfb1wYRJULyCzs5w/nj9Btf9q/YTSMEKZu1yCxoWge kZ2flNozQYS1aQoC/xY4SekdZHpMXFZM+Jmk5ZPiSk6+k64T028hVxxA9RGQgg+iynkJ sJi+24GdFEx/gK5y382hdD6A1FYVVPAfbMw2Z/VPivbfF0XwE5ZiYMEZdjf18rdtphAc mBxS0YqsPJUUK3VF/gRVn8OgSkQ0QksLd3r0fJAFrxSmTkhrv6SKhpdVv+skL7IE1Y2g 4+7g== X-Forwarded-Encrypted: i=1; AFNElJ8cMXkmSm+Ck8gmZdJabQVK8Jmvl08r+h2+42zBzsjLrnpUs9aWA5NFWoTlHK/ZQGhOgZlsP8RjacSEkuU=@vger.kernel.org X-Gm-Message-State: AOJu0YwsTvQ0MmqggKj2PkQ1j7Y45APknWlXTvyp8vBqxXEoOGpByY8q IyTTlnMd/W0C/k/sK0ynrFVHvBH21XlEzKWtF1HD41cux8HmB1uUmJ0cJt3/1KJCSiqGg6GyYI3 CGGiiHBw2NMonqNJR89JccHr7+gePRqYDWgZ3CPdtDZQ9L6w6hMzhJUtNFdQm96849p4YGxXzkQ == X-Gm-Gg: Acq92OF3Gbc4EtZvm42o13VS5lFL3629hE7NVdwabDd768zdXPLZ5xrLjvlsPm0Y0Z+ a8Qwera47r4pQ1wTMFw09nS/XcdvNqlwKJgIcxHuJHgP8ObdJE0WRzzLQNfnrued94OO9V1h+01 aghuuBJLnz78444OMDc4WfM0xVERAOJgg77a1H33A7MtjIe2FIKBSyKQn36z5Od2wTWysqPqPi1 Y9PIw/c7bKQga/4UAYHw7lUhyNA/JufdnvGrah/t4ZENQIXg3F78EkUdkCmn01iOdAp9eLCL+VG aoStAYpz87q3gzgxQbV6yNPDaOkSn4Gm47SFO65Fz+utxqYuuhLs/moAXPlkKSxBSff/r612GM6 UQazi3DmSZ4bLTY32cnxgMX8sGASAKpShp/y6SC2JcvSWROivCuunlwQ4yH3EbQ== X-Received: by 2002:a5d:5d0e:0:b0:460:3210:434a with SMTP id ffacd0b85a97d-4606dbc6b71mr2814009f8f.43.1781254441905; Fri, 12 Jun 2026 01:54:01 -0700 (PDT) X-Received: by 2002:a5d:5d0e:0:b0:460:3210:434a with SMTP id ffacd0b85a97d-4606dbc6b71mr2813966f8f.43.1781254441401; Fri, 12 Jun 2026 01:54:01 -0700 (PDT) Received: from vschneid-thinkpadt14sgen2i.remote.csb ([2a01:cb16:2069:d022:ee80:c854:b3a4:c6d1]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4606f2c3782sm3737386f8f.25.2026.06.12.01.53.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Jun 2026 01:54:00 -0700 (PDT) From: Valentin Schneider To: Tomas Glozar Cc: Crystal Wood , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Costa Shulyupin , Ivan Pravdin Subject: Re: [RFC PATCH 1/2] tracing/osnoise: Sample IPI counts In-Reply-To: References: <20260610130457.1304245-1-vschneid@redhat.com> <20260610130457.1304245-2-vschneid@redhat.com> Date: Fri, 12 Jun 2026 10:53:58 +0200 Message-ID: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 11/06/26 13:55, Tomas Glozar wrote: > =C4=8Dt 11. 6. 2026 v 12:31 odes=C3=ADlatel Valentin Schneider > napsal: >> > >> > Isn't that precisely what the ipi tracepoints used by this >> > implementation (ipi:ipi_send_cpu) are for? >> > >> >> Well, these catch the emission of the IPI, which is great for investigat= ion >> - slap a stacktrace trigger and you (most of the time) get the source of >> your interference. >> >> However Crystal's point is that on x86 (and I assume other archs) receiv= ing >> & handling these IPIs is "special" and doesn't go through the generic irq >> subsystem and thus has to be tracked separately, which is why osnoise has >> this fairly lengthy osnoise_arch_register() thing. >> > > Ah, right. This is not IPI specific, though, IIUC - Intel also has > other IRQs that have to be traced using Intel-specific trace points, > like irq_vectors:local_timer, which is also handled in > osnoise_arch_register(). On ARM from what I recall, most (all?) IRQs > are traced with irq:* tracepoints. > > So there are two parts to this: > > - Detecting interference from IPIs firing as osnoise:irq_noise (to be > analyzed by timerlat auto analysis, and also will appear by default in > trace output if enabled, regardless of the tool, as all osnoise:* > tracepoints are enabled there). This is done locally using the already > existing path (no race hazard), but requires arch-specific detection. > > - Counting IPIs when they are being sent. This is the new feature, and > the count is being recorded in osnoise_sample. > > I guess that means that if there were a generic IPI interface, it > would be easier to use that for IPI counting, as the event would be > CPU-local? As you say, for tracing of the IPI source, the sending > tracepoints are better, and that you can already dump the stack trace > of with --event/--trigger. timerlat auto-analysis could be extended to > connect the specific IPI to the IRQ noise and display its stack trace > automatically, instead of manually analyzing the trace output. > Right, at least for the smp_call stuff (which includes irq_work) we can leverage: csd_queue_cpu (on the sending CPU) csd_func_start (on the receiving CPU) by indexing on the @csd address; once upon a time [1] I had this: $ echo 'hist:keys=3Dcpu,csd.hex:ts=3Dcommon_timestamp.usecs:src=3Dcommon_= cpu' >\ /sys/kernel/tracing/events/csd/csd_queue_cpu/trigger $ echo 'csd_latency unsigned int src_cpu; '\ 'unsigned int dst_cpu; '\ 'unsigned long csd; u64 time' >\ /sys/kernel/tracing/synthetic_events $ echo 'hist:keys=3Dcommon_cpu,csd.hex: time=3Dcommon_timestamp.usecs-$ts: onmatch(csd.csd_queue_cpu).trace(csd_latency,$src,common_cpu,csd,$time)' = >\ /sys/kernel/tracing/events/csd/csd_function_entry/trigger $ trace-cmd record -e 'synthetic:csd_latency' hackbench $ trace-cmd report -0 [001] 115.236810: csd_latency: src_cpu=3D7, dst_c= pu=3D1, csd=3D18446612682588476192, time=3D134 -0 [000] 115.240676: csd_latency: src_cpu=3D7, dst_c= pu=3D0, csd=3D18446612682588214048, time=3D103 -0 [009] 115.241320: csd_latency: src_cpu=3D7, dst_c= pu=3D9, csd=3D18446612682143963384, time=3D83 -0 [007] 115.242817: csd_latency: src_cpu=3D8, dst_c= pu=3D7, csd=3D18446612682150759032, time=3D93 -0 [005] 115.247802: csd_latency: src_cpu=3D7, dst_c= pu=3D5, csd=3D18446612682144441144, time=3D114 -0 [005] 115.271775: csd_latency: src_cpu=3D7, dst_c= pu=3D5, csd=3D18446612682144441144, time=3D151 -0 [000] 115.279620: csd_latency: src_cpu=3D7, dst_c= pu=3D0, csd=3D18446612682588214048, time=3D87 -0 [000] 115.281727: csd_latency: src_cpu=3D7, dst_c= pu=3D0, csd=3D18446612682588214048, time=3D101 [1]: https://lore.kernel.org/lkml/xhsmh4jn8y8vt.mognet@vschneid.remote.csb/ I believe you're right that leveraging this would be useful for timerlat-aa; I'll add it to my todolist :-) >> >> Isn't this racy to do from a different CPU? Both in terms of the >> >> counter, and the timing of the increment relative to when the IPI is >> >> actually received. Not necessarily a huge deal if you only care about >> >> zero versus bignum, but still. At least worth a comment, if we go wi= th >> >> this approach. >> >> >> > >> > I also think it's a bit confusing, especially as the other accesses to >> > osn_var are cpu-local, but here, "cpu" is the *target* CPU, not the >> > current CPU. Not sure how expensive it would be to do atomic_add for >> > that, at least it's something to consider. >> > >> >> I suppose that could be an argument for doing that stat aggregation in >> userspace osnoise - event handlers are run after the fact via >> tracefs_iterate_raw_events(), it's all inherently slower since it's just >> increments of one (one per handled event) but it's also all done in >> userspace on a control thread and doesn't bog down the kernelspace. >> > > You can also do per-cpu counters in-kernel and sum them in the end, > but that would take cpus^2 space (indexed by [current_cpu, > target_cpu]). The question is whether there could be enough samples to > overload sample collection (like it happens for timerlat, which > collects data in-kernel using BPF instead). > > In-kernel counting can be tested with " --event ipi:ipi_send_cpu > --trigger hist:key=3Dcpu" - IIRC, tracefs histograms use atomic > operations (via tracing_map) to protect the entries from races in > multi thread access. Of course, that is inferior to what the patchset > implements, as it doesn't record which osnoise cycle the IPI was sent > in, nor can record cpumask IPIs. > I suppose I'll need to go do some benchmarking, but I'm starting to lean towards the side of atomic incs for IPI counts being okay considering the sort of latencies we track. > > Tomas