Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

[Steven Noonan <steven@uplinklabs.net>]

On Sun, Apr 15, 2012 at 02:14:57PM +0800, Lin Ming wrote:
> (Forgot mail subject, add it)
> 
> On Sun, Apr 15, 2012 at 2:09 PM, Lin Ming <mlin@ss.pku.edu.cn> wrote:
> > Hi all,
> >
> > These 2 patches try to fix the "perf top" soft lockups under Xen
> > reported by Steven at: https://lkml.org/lkml/2012/2/9/506
> >
> > I tested it with 3.4-rc2 and "perf top" works well now.
> >
> > Steven,
> > Could you please help to test it too?
> >
> > The soft lockup code path is:
> >
> > __irq_work_queue
> >  arch_irq_work_raise
> >    apic->send_IPI_self(IRQ_WORK_VECTOR);
> >      apic_send_IPI_self
> >        __default_send_IPI_shortcut
> >          __xapic_wait_icr_idle
> >
> > static inline void __xapic_wait_icr_idle(void)
> > {
> >        while (native_apic_mem_read(APIC_ICR) & APIC_ICR_BUSY)
> >                cpu_relax();
> > }
> >
> > The lockup happens at above while looop.
> >
> > The cause is that Xen has not implemented the APIC IPI interface yet.
> > Xen has IPI interface: xen_send_IPI_one, but it's only used in
> > xen_smp_send_reschedule, xen_smp_send_call_function_ipi and
> > xen_smp_send_call_function_single_ipi, etc.
> >
> > So we need to implement Xen's APIC IPI interface as Ben's patch does.
> > And implement Xen's IRQ_WORK_VECTOR handler.
> >
> > Ben Guthro (1):
> >      xen: implement apic ipi interface
> >
> > Lin Ming (1):
> >      xen: implement IRQ_WORK_VECTOR handler
> >
> >  arch/x86/include/asm/xen/events.h |    1 +
> >  arch/x86/xen/enlighten.c          |    7 ++
> >  arch/x86/xen/smp.c                |  111 +++++++++++++++++++++++++++++++++++-
> >  arch/x86/xen/smp.h                |   12 ++++
> >  4 files changed, 127 insertions(+), 4 deletions(-)
> >
> > Any comment is appreciated.
> > Lin Ming

OK, finally got around to testing this. Sorry about the delay, been
rather busy at work.

These two patches definitely fix the lockups, but 'perf test' doesn't
pass every test:

	 1: vmlinux symtab matches kallsyms:
	--- start ---
	Looking at the vmlinux_path (6 entries long)
	dso__load_sym: cannot get elf header.
	Using /lib/modules/3.3.2-00002-gf57130d/build/vmlinux for symbols
	Maps only in vmlinux:
	 ffffffff81ae61b1-ffffffff81b9d7b7 0 [kernel].init.text
	 ffffffff81b9d7b8-ffffffff9fffffff 0 [kernel].exit.text
	Maps in vmlinux with a different name in kallsyms:
	Maps only in kallsyms:
	---- end ----
	vmlinux symtab matches kallsyms: Ok
	 2: detect open syscall event:
	--- start ---
	---- end ----
	detect open syscall event: Ok
	 3: detect open syscall event on all cpus:
	--- start ---
	---- end ----
	detect open syscall event on all cpus: Ok
	 4: read samples using the mmap interface:
	--- start ---
	---- end ----
	read samples using the mmap interface: Ok
	 5: parse events tests:
	--- start ---
	---- end ----
	parse events tests: Ok
	 6: Validate PERF_RECORD_* events & perf_sample fields:
	--- start ---
	perf_evlist__open: Bad file descriptor
	---- end ----
	Validate PERF_RECORD_* events & perf_sample fields: FAILED!

I couldn't figure out what exactly was failing in that test with a
quick inspection... Someone else want to take a stab at it? If someone
wants to give me patches that add extra output, I could run those too.

Anyway:

Tested-by: Steven Noonan <steven@uplinklabs.net>

- Steven