From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C1EACD6102 for ; Mon, 9 Oct 2023 16:23:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377174AbjJIQXs (ORCPT ); Mon, 9 Oct 2023 12:23:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376889AbjJIQXq (ORCPT ); Mon, 9 Oct 2023 12:23:46 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C02A9AF for ; Mon, 9 Oct 2023 09:23:43 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 66BDDC433C8; Mon, 9 Oct 2023 16:23:42 +0000 (UTC) Date: Mon, 9 Oct 2023 12:25:00 -0400 From: Steven Rostedt To: Jinyu Tang Cc: mhiramat@kernel.org, bristot@kernel.org, ttjjyystupid@163.com, falcon@tinylab.org, wangjiexun@tinylab.org, linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v1] Ftrace: make sched_wakeup can focus on the target process Message-ID: <20231009122500.69854f94@gandalf.local.home> In-Reply-To: <20231009153714.10743-1-tangjinyu@tinylab.org> References: <20231009153714.10743-1-tangjinyu@tinylab.org> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 9 Oct 2023 23:37:14 +0800 Jinyu Tang wrote: > When we want to know what happened in kernel when our app > has more latency than we hope, but the larger latency of > our app may be lower than other process in the syetem. > We feel sad after waiting a long time but only get other > process sched_wakeup trace. > > This Patch can let us only trace target process sched-wakeup > time, other process sched-wakeup will be dropped and won't > change tracing_max_latency. > > The patch is tested by the following commands: > > $ mount -t tracefs none /sys/kernel/tracing > $ echo wakeup_rt > /sys/kernel/tracing/current_tracer > # some other stress-ng options are also tested > $ stress-ng --cpu 4 & > $ cyclictest --mlockall --smp --priority=99 & > $ cyclictest_pid=$! > # child thread of cyclictest main process > $ thread_pid=$((cyclictest_pid + 1)) > > $ echo ${thread_pid} > /sys/kernel/tracing/set_wakeup_pid > > $ echo 1 > /sys/kernel/tracing/tracing_on > $ echo 0 > /sys/kernel/tracing/tracing_max_latency > $ wait ${cyclictest_pid} > $ echo 0 > /sys/kernel/tracing/tracing_on > $ cat /sys/kernel/tracing/trace > > The maximum latency and backtrace recorded in the trace file will be only > generated by the target process. > Then we can eliminate interference from other programs, making it easier > to identify the cause of latency. > > Tested-by: Jiexun Wang > Signed-off-by: Jinyu Tang > --- Honestly, the wakeup tracers are obsolete. I haven't used them in years. I use synthetic events instead: # cd /sys/kernel/tracing # echo 'wakeup_lat pid_t pid; u64 delay;' > synthetic_events # echo 'hist:keys=pid:ts=common_timestamp.usecs' if pid==$thread_pid > events/sched/sched_waking/trigger # echo 'hist:keys=next_pid:delta=common_timestamp.usecs-$ts:onmax($delta).trace(wakeup_lat,next_pid,$delta)' > events/sched/sched_switch/trigger # echo 1 > events/synthetic/wakeup_lat/enable # cat trace # tracer: nop # # entries-in-buffer/entries-written: 3/3 #P:8 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | -0 [000] d..4. 350799.423428: wakeup_lat: pid=59921 delay=1281 -0 [000] d..4. 350800.423441: wakeup_lat: pid=59921 delay=1317 -0 [000] d..4. 350801.423445: wakeup_lat: pid=59921 delay=1331 I could also make it record stack traces, disable tracing, and all sorts of other nifty things. -- Steve