From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lf1-f48.google.com (mail-lf1-f48.google.com [209.85.167.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A20C519D09C for ; Thu, 6 Feb 2025 18:41:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738867276; cv=none; b=k8CrAXnqrWF6VWOJH+m/x98CW4Cx0ar1yXmZogPLqp4baz71cWosh8rJhNxWfGMynbRtGUX6olywnpDzQ8A2fMAWopsFNWsAMgl08PjGb3+VakidktbCUbm9u/k/7inm32p3RHwyKYdJ6ieI+iDa4tAgRvrZjS3NMDK5RNUwivI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738867276; c=relaxed/simple; bh=nvffeYadc3fpJEHQH7BAtYU030TLHFW2jziAIX4uGTk=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=VycEndg035gst3DIFtw90uUQcAux/jfms2ZW5RRqofn9RASK5jyCGO+IhzO1lO/3DoHBhKXOIJA0/4OqsSuZ/v/qgfH3EL+5vI5U8QE8kLvGtAzc/fDymwVEkSMJezi3HFN46ho+rm1SidNjcnRo+d9ZMSEMFs7zp3FLm4+u8io= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HHTeqURp; arc=none smtp.client-ip=209.85.167.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HHTeqURp" Received: by mail-lf1-f48.google.com with SMTP id 2adb3069b0e04-5440f22aabdso1167061e87.0 for ; Thu, 06 Feb 2025 10:41:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867272; x=1739472072; darn=vger.kernel.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=qk45u5eL6UrjO7ujgKkxwsztPQtoNsoUKCE3ySDBx+o=; b=HHTeqURpL5ERyXOvt+kZWqGTgRxrNEP4yndbT4HAZIrVvZkOkyKElDzZvouy8KF5HP 3HsV04RJPqsRhNMjZDpudnTjg7pztxBLZI11WHWbUuvzfyEiMibqzH2TCauB9xeGdxuD MSAb364tIZkIf4ACn9+3G9d1Rjc2aWtPrmRd5PXhO4a906/ylsMzDy8eUEz6J4871csx S6/ZnR7hWtUZs/lcPW7ZEvaOfK+wtSy9JyR9YtRXq6wdjB/gcDDBLV6W71p5GzBHt8VX iUA9ICG6fwEL5aiJdlfQUZ4XQQ5mBMFHv8CF/U4kK2FFDsjwl8lfA7iWcjPFeko/VvhU AxIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867272; x=1739472072; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qk45u5eL6UrjO7ujgKkxwsztPQtoNsoUKCE3ySDBx+o=; b=Yi5Qu6KrAsDZJDtobtKCrJBbJp0mcj7n8HMNhHoVSvjPEXmRpplcm6gqgMEakWqlua eQmbIt3cMhmJd+JP3aMWmPKNmMNW21qTMTfb/5ZLKEXYlRCKFpuXtJ7Fx1dB8OboxjnM i7lfD6JySeFiOzsa0rppRS262tQ810vEcyM9gG5TkpSqQaEcnhCXMxiHJNNYLPOVr4qk 20MdzMc3As9dkxqcMuVUBR8hY5JwGYr0JEiUvhUET6PUIDu5xigNsJP8V+pa2nFY49DV eweedQ5u5rlEsQYLEaW46Ugt5klVX1YbfmHo3/w/GxGWc137dzCdExODlApCNci23NBc MsUw== X-Forwarded-Encrypted: i=1; AJvYcCU0KDXjzYQRwLt9fH1CbeczmuTVDOfxKElu0e9kYwXlWTy0OMM4bZl40bms0r+aWx6K7MDUSgeMJeb3/vU=@vger.kernel.org X-Gm-Message-State: AOJu0Yw7POQNGbiE2pvmsIyBe5faapSK2Jz26xd6kzIJ0LINA6mXa+tz G8WUUb/yzP0Gw+iuoIN6Y9d0bccXHRCWIyNh4hgy73OJ8jv0q1frs0Kd0glhqLcENWjrW9esNu3 xbEM/jrQB03VVKkUQ5tAzlZfIqzCcMr50LknG X-Gm-Gg: ASbGnctTCPZth6riE3WaMRnaSgYTyWObIeN+npWUTSUNTKXsDkullHAIDlom1cdu1R8 Z8wSf6kRBRLdRcAawylUdteTiqcaii4hvEdpGIPo8oNHM0LJsJkTY+RxXsf9bWOfbzXam+ujDQL wJbYjosEeTcEYDcdCfrCUC3B/ZonLFWQ== X-Google-Smtp-Source: AGHT+IFLD0AMOveUgIxfuaFXAqWtATyO7+L5W8zqlnflXwDNc6nRZ8Py06TbH/yLATDOr7fr48Td6f9hVXFBvs+WP+E= X-Received: by 2002:a05:6512:6d2:b0:544:123e:331a with SMTP id 2adb3069b0e04-544123e3555mr894484e87.14.1738867271542; Thu, 06 Feb 2025 10:41:11 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <87ldujkjsi.fsf@linux.intel.com> In-Reply-To: <87ldujkjsi.fsf@linux.intel.com> From: Dmitry Vyukov Date: Thu, 6 Feb 2025 19:41:00 +0100 X-Gm-Features: AWEUYZlYIYYrxtBDXZaUPnHA29Ll1mqyfEjy95EoBl8Vi8hkcfBXpCgUndmyCqA Message-ID: Subject: Re: [PATCH v5 0/8] perf report: Add latency and parallelism profiling To: Andi Kleen Cc: namhyung@kernel.org, irogers@google.com, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, Arnaldo Carvalho de Melo Content-Type: text/plain; charset="UTF-8" On Thu, 6 Feb 2025 at 19:30, Andi Kleen wrote: > > Dmitry Vyukov writes: > > > There are two notions of time: wall-clock time and CPU time. > > For a single-threaded program, or a program running on a single-core > > machine, these notions are the same. However, for a multi-threaded/ > > multi-process program running on a multi-core machine, these notions are > > significantly different. Each second of wall-clock time we have > > number-of-cores seconds of CPU time. > > I'm curious how does this interact with the time / --time-quantum sort key? > > I assume it just works, but might be worth checking. I will check later. But if you have some concrete commands to try, it will help. I never used --time-quantum before. > It was intended to address some of these issues too. > > > Optimizing CPU overhead is useful to improve 'throughput', while > > optimizing wall-clock overhead is useful to improve 'latency'. > > These profiles are complementary and are not interchangeable. > > Examples of where latency profile is needed: > > - optimzing build latency > > - optimizing server request latency > > - optimizing ML training/inference latency > > - optimizing running time of any command line program > > > > CPU profile is useless for these use cases at best (if a user understands > > the difference), or misleading at worst (if a user tries to use a wrong > > profile for a job). > > I would agree in the general case, but not if the time sort key > is chosen with a suitable quantum. You can see how the parallelism > changes over time then which is often a good enough proxy. Never used it. I will look at what capabilities it provides. > > We still default to the CPU profile, so it's up to users to learn > > about the second profiling mode and use it when appropriate. > > You should add it to tips.txt then It is done in the docs patch. > > .../callchain-overhead-calculation.txt | 5 +- > > .../cpu-and-latency-overheads.txt | 85 ++++++++++++++ > > tools/perf/Documentation/perf-record.txt | 4 + > > tools/perf/Documentation/perf-report.txt | 54 ++++++--- > > tools/perf/Documentation/tips.txt | 3 + > > tools/perf/builtin-record.c | 20 ++++ > > tools/perf/builtin-report.c | 39 +++++++ > > tools/perf/ui/browsers/hists.c | 27 +++-- > > tools/perf/ui/hist.c | 104 ++++++++++++------ > > tools/perf/util/addr_location.c | 1 + > > tools/perf/util/addr_location.h | 7 +- > > tools/perf/util/event.c | 11 ++ > > tools/perf/util/events_stats.h | 2 + > > tools/perf/util/hist.c | 90 ++++++++++++--- > > tools/perf/util/hist.h | 32 +++++- > > tools/perf/util/machine.c | 7 ++ > > tools/perf/util/machine.h | 6 + > > tools/perf/util/sample.h | 2 +- > > tools/perf/util/session.c | 12 ++ > > tools/perf/util/session.h | 1 + > > tools/perf/util/sort.c | 69 +++++++++++- > > tools/perf/util/sort.h | 3 +- > > tools/perf/util/symbol.c | 34 ++++++ > > tools/perf/util/symbol_conf.h | 8 +- > > We traditionally didn't do it, but in general test coverage > of perf report is too low, so I would recommend to add some simple > test case in the perf test scripts. What of this is testable within the current testing framework? Also how do I run tests? I failed to figure it out.