From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932438AbcGKQzc (ORCPT ); Mon, 11 Jul 2016 12:55:32 -0400 Received: from smtprelay0045.hostedemail.com ([216.40.44.45]:45913 "EHLO smtprelay.hostedemail.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1755763AbcGKQz0 (ORCPT ); Mon, 11 Jul 2016 12:55:26 -0400 X-Session-Marker: 726F737465647440676F6F646D69732E6F7267 X-Spam-Summary: 2,0,0,,d41d8cd98f00b204,rostedt@goodmis.org,:::::::,RULES_HIT:2:41:355:379:541:599:800:960:973:988:989:1260:1277:1311:1313:1314:1345:1359:1437:1515:1516:1518:1535:1593:1594:1605:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2907:3138:3139:3140:3141:3142:3622:3865:3866:3867:3868:3870:3871:3872:4049:4118:4321:4605:5007:6119:6261:7514:7875:7903:8660:10004:10848:10967:11026:11232:11658:11914:12043:12114:12295:12296:12438:12517:12519:12555:12740:13148:13161:13229:13230:13439:14096:14097:14659:21080:21433:30054:30090:30091,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:4,LUA_SUMMARY:none X-HE-Tag: eyes67_11faa1c3c3e18 X-Filterd-Recvd-Size: 7251 Date: Mon, 11 Jul 2016 12:55:22 -0400 From: Steven Rostedt To: Namhyung Kim Cc: LKML , Joonsoo Kim , Namhyung Kim Subject: Re: [PATCH 2/2] trace-cmd: Use tracecmd_peek_next_data() in fgraph_ent_handler Message-ID: <20160711125522.794dfccb@gandalf.local.home> In-Reply-To: <20160708055612.32221-2-namhyung@gmail.com> References: <20160708055612.32221-1-namhyung@gmail.com> <20160708055612.32221-2-namhyung@gmail.com> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.30; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 8 Jul 2016 14:56:12 +0900 Namhyung Kim wrote: > When a task was migrated to other cpu in the middle of a function, the > fgraph_exit record will be in a different cpu than the enter record. > But currently fgraph_ent_handler() only peeks at the same cpu so it > could read an incorrect record. > > For example, please see following raw records: > > bash-10478 [007] 73.454273: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [000] 73.454650: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x111a37483c rettime=0x111a3d0285 overrun=0x0 depth=0 > bash-10478 [000] 74.456383: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [000] 74.456655: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x1155f24337 rettime=0x1155f66559 overrun=0x0 depth=0 > bash-10478 [000] 75.458517: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [001] 75.458849: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x1191ad9de0 rettime=0x1191b2a6aa overrun=0x0 depth=0 > bash-10478 [001] 76.460482: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [000] 76.460679: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x11cd6662b4 rettime=0x11cd695e03 overrun=0x0 depth=0 > bash-10478 [000] 77.462334: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [004] 77.462564: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x12091d71c4 rettime=0x120920e977 overrun=0x0 depth=0 > bash-10478 [004] 78.464315: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [001] 78.464644: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x1244d674de rettime=0x1244db7329 overrun=0x0 depth=0 > bash-10478 [001] 79.466018: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [004] 79.466326: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x12808b3940 rettime=0x12808fe819 overrun=0x0 depth=0 > bash-10478 [004] 80.468005: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [002] 80.468291: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x12bc44551f rettime=0x12bc48ac9a overrun=0x0 depth=0 > bash-10478 [002] 81.469718: funcgraph_entry: func=0xffffffff8123bf90 depth=0 > bash-10478 [007] 81.470088: funcgraph_exit: func=0xffffffff8123bf90 calltime=0x12f7f945b8 rettime=0x12f7fee028 overrun=0x0 depth=0 > > The first entry was call to cma_alloc function, it was on cpu 7 but the > task was migrated to cpu 0 before returning from the function. > Currently trace-cmd shows like below: > > bash-10478 [007] 73.454273: funcgraph_entry: ! 367.216 us | cma_alloc(); > bash-10478 [000] 73.454650: funcgraph_exit: ! 375.369 us | } > bash-10478 [000] 74.456383: funcgraph_entry: ! 270.882 us | cma_alloc(); > bash-10478 [000] 75.458517: funcgraph_entry: ! 195.407 us | cma_alloc(); > bash-10478 [001] 75.458849: funcgraph_exit: ! 329.930 us | } > bash-10478 [001] 76.460482: funcgraph_entry: ! 327.243 us | cma_alloc(); > bash-10478 [000] 77.462334: funcgraph_entry: ! 293.465 us | cma_alloc(); > bash-10478 [004] 77.462564: funcgraph_exit: ! 227.251 us | } > bash-10478 [004] 78.464315: funcgraph_entry: ! 306.905 us | cma_alloc(); > bash-10478 [001] 79.466018: funcgraph_entry: ! 303.196 us | cma_alloc(); > bash-10478 [004] 80.468005: funcgraph_entry: | cma_alloc() { > bash-10478 [002] 80.468291: funcgraph_exit: ! 284.539 us | } > bash-10478 [002] 81.469718: funcgraph_entry: ! 323.215 us | cma_alloc(); > > This is because the first funcgraph_entry on cpu 7 matched to the last > funcgraph_exit on cpu 7. And second funcgraph_exit on cpu 0 was shown > alone. We need to match record from all cpu rather than the same cpu. > In this case, entry on cpu 7 should be paired with exit on cpu 0. > > With this patch, the output look like below: > > bash-10478 [007] 73.454273: funcgraph_entry: ! 375.369 us | cma_alloc(); > bash-10478 [000] 74.456383: funcgraph_entry: ! 270.882 us | cma_alloc(); > bash-10478 [000] 75.458517: funcgraph_entry: ! 329.930 us | cma_alloc(); > bash-10478 [001] 76.460482: funcgraph_entry: ! 195.407 us | cma_alloc(); > bash-10478 [000] 77.462334: funcgraph_entry: ! 227.251 us | cma_alloc(); > bash-10478 [004] 78.464315: funcgraph_entry: ! 327.243 us | cma_alloc(); > bash-10478 [001] 79.466018: funcgraph_entry: ! 306.905 us | cma_alloc(); > bash-10478 [004] 80.468005: funcgraph_entry: ! 284.539 us | cma_alloc(); > bash-10478 [002] 81.469718: funcgraph_entry: ! 367.216 us | cma_alloc(); > > Maybe we can separate enter and exit if they happened on different > cpu. Anyway the time duration has correct value now. > > Reported-by: Joonsoo Kim > Signed-off-by: Namhyung Kim > --- > trace-ftrace.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/trace-ftrace.c b/trace-ftrace.c > index 636b08b..edc9349 100644 > --- a/trace-ftrace.c > +++ b/trace-ftrace.c > @@ -297,7 +297,7 @@ fgraph_ent_handler(struct trace_seq *s, struct pevent_record *record, > struct tracecmd_ftrace *finfo = context; > struct pevent_record *rec; > unsigned long long val, pid; > - int cpu = record->cpu; > + int cpu; > > ret_event_check(finfo, event->pevent); > > @@ -307,7 +307,7 @@ fgraph_ent_handler(struct trace_seq *s, struct pevent_record *record, > if (pevent_get_field_val(s, event, "func", record, &val, 1)) > return trace_seq_putc(s, '!'); > > - rec = tracecmd_peek_data(tracecmd_curr_thread_handle, cpu); > + rec = tracecmd_peek_next_data(tracecmd_curr_thread_handle, &cpu); Hmm, but what happens if the next data is just some random event on another CPU. Do we want to break it up just because there's data on another cpu? I wonder if we should grab a record from the same cpu and if it isn't the return, then try another cpu? -- Steve > if (rec) > rec = get_return_for_leaf(s, cpu, pid, val, rec, finfo); >