Re: [PATCH] Enhance perf to collect KVM guest os statistics from host side

["Zhang, Yanmin" <yanmin_zhang@linux.intel.com>]

On Tue, 2010-03-23 at 10:15 -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Mar 23, 2010 at 11:14:41AM +0800, Zhang, Yanmin escreveu:
> > On Mon, 2010-03-22 at 13:44 -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Mon, Mar 22, 2010 at 03:24:47PM +0800, Zhang, Yanmin escreveu:
> > > > On Fri, 2010-03-19 at 09:21 +0100, Ingo Molnar wrote:
> > > > Then, perf could access all files. It's possible because guest os instance
> > > > happens to be multi-threading in a process. One of the defects is the accessing to
> > > > guest os becomes slow or impossible when guest os is very busy.
> > > 
> > > If the MMAP events on the guest included a cookie that could later be
> > > used to query for the symtab of that DSO, we wouldn't need to access the
> > > guest FS at all, right?
> 
> > It depends on specific sub commands. As for 'perf kvm top', developers
> > want to see the profiling immediately. Even with 'perf kvm record',
> > developers also want to
> 
> That is not a problem, if you have the relevant buildids in your cache
> (Look in your machine at ~/.debug/), it will be as fast as ever.
> 
> If you use a distro that has its userspace with build-ids, you probably
> use it always without noticing :-)
> 
> > see results quickly. At least I'm eager for the results when
> > investigating a performance issue.
> 
> Sure thing.
>  
> > > With build-ids and debuginfo-install like tools the symbol
> > > resolution could be performed by using the cookies (build-ids) as
> > > keys to get to the *-debuginfo packages with matching symtabs (and
> > > DWARF for source annotation, etc).
> 
> > We can't make sure guest os uses the same os images, or don't know
> > where we could find the original DVD images being used to install
> > guest os.
> 
> You don't have to have guest and host sharing the same OS image, you
> just have to somehow populate your buildid cache with what you need, be
> it using sshfs or what Ingo is suggesting once, or using what your
> vendor provides (debuginfo packages). And you just have to do it once,
> for the relevant apps, to have it in your buildid cache.
>  
> > Current perf does save build id, including both kernls's and other
> > application lib/executables.
> 
> Yeah, I know, I implemented it. :-)
>  
> > > We have that for the kernel as:
> 
> > > [acme@doppio linux-2.6-tip]$ l /sys/kernel/notes 
> > > -r--r--r-- 1 root root 36 2010-03-22 13:14 /sys/kernel/notes
> > > [acme@doppio linux-2.6-tip]$ l /sys/module/ipv6/sections/.note.gnu.build-id 
> > > -r--r--r-- 1 root root 4096 2010-03-22 13:38 /sys/module/ipv6/sections/.note.gnu.build-id
> > > [acme@doppio linux-2.6-tip]$
> 
> > > That way we would cover DSOs being reinstalled in long running 'perf
> > > record' sessions too.
> 
> > That's one of objectives of perf to support long running.
> 
> But it doesn't fully supports right now, as I explained, build-ids are
> collected at the end of the record session, because we have to open the
> DSOs that had hits to get the 20 bytes cookie we need, the build-id.
> 
> If we had it in the PERF_RECORD_MMAP record, we would close this race,
> and the added cost at load time should be minimal, to get the ELF
> section with it and put it somewhere in task struct.
Well, you are improving upon perfection.

> 
> If only we could coalesce it a bit to reclaim this:
> 
> [acme@doppio linux-2.6-tip]$ pahole -C task_struct ../build/v2.6.34-rc1-tip+/kernel/sched.o  | tail -5
> 	/* size: 5968, cachelines: 94, members: 150 */
> 	/* sum members: 5943, holes: 7, sum holes: 25 */
> 	/* bit holes: 1, sum bit holes: 28 bits */
> 	/* last cacheline: 16 bytes */
> };
> [acme@doppio linux-2.6-tip]$ 
That reminds me I listened to your presentation on 2007 OLS. :)

> 
> 8-)
> 
> Or at least get just one of those 4 bytes holes then we could stick it
> at the end to get our build-id there, accessing it would be done only
> at PERF_RECORD_MMAP injection time, i.e. close to the time when we
> actually are loading the executable mmap, i.e. close to the time when
> the loader is injecting the build-id, I guess the extra memory and
> processing costs would be in the noise.
> 
> > > This was discussed some time ago but would require help from the bits
> > > that load DSOs.
> 
> > > build-ids then would be first class citizens.
> 
> - Arnaldo