[BUG] perf failed on multi-cpus system with multi-events specified

[BUG] perf failed on multi-cpus system with multi-events specified

[Han Pingtian]

Hi,

On a Nehalem-EX system, which has 24 CPUs, perf commands with
multi-events specified fail:

[root@intel-s3e36-01 perf]# ./perf record -e syscalls:sys_enter_sched_get_priority_min  -e kmem:mm_kernel_pagefault  -e workqueue:workqueue_execution  -e writeback:writeback_pages_written  -e kmem:mm_kernel_pagefault  -e writeback:writeback_bdi_unregister  -e syscalls:sys_exit_capset  -e syscalls:sys_enter_sched_get_priority_min  -e writeback:writeback_exec  -e syscalls:sys_enter_vmsplice  -e syscalls:sys_exit_mprotect  -e block:block_rq_complete  -e syscalls:sys_enter_readlinkat  -e signal:signal_overflow_fail  -e syscalls:sys_exit_sendfile64  -e block:block_bio_backmerge  -e syscalls:sys_exit_unlink  -e syscalls:sys_exit_keyctl  -e jbd2:jbd2_submit_inode_data  -e syscalls:sys_enter_tgkill  -e syscalls:sys_exit_lremovexattr  -e syscalls:sys_exit_munmap  -e syscalls:sys_enter_mmap  -e syscalls:sys_exit_sched_getparam  -e syscalls:sys_exit_getpgid  -e syscalls:sys_exit_clock_getres  -e block:block_split  -e ext4:ext4_mballoc_discard  -e task-clock  -e syscalls:sys_enter_sendfile64  -e syscalls:sys_enter_ptrace  -e syscalls:sys_enter_nanosleep  -e syscalls:sys_exit_setdomainname  -e writeback:writeback_nowork  -e syscalls:sys_exit_mbind  -e ext4:ext4_da_write_begin  -e syscalls:sys_enter_recvfrom  -e syscalls:sys_enter_write  -e syscalls:sys_exit_getresgid  -e syscalls:sys_exit_fsync  -e syscalls:sys_enter_rt_sigprocmask  -e syscalls:sys_enter_getdents  -e syscalls:sys_exit_semtimedop -f ls

  Error: sys_perf_event_open() syscall returned with -1 (Too many open files).  /bin/dmesg may provide additional information.

  Fatal: No CONFIG_PERF_EVENTS=y kernel support configured?

trace-cmd: Too many open files
  No memory to alloc tracepoints list
[root@intel-s3e36-01 perf]# ./perf --version
perf version 2.6.38.rc6.166.g493f33
[root@intel-s3e36-01 perf]#

On another latop which has only two CPUs, there is no this problem.
-- 
Han Pingtian
Quality Engineer
hpt @ #kernel-qe
Red Hat, Inc
Freedom ... courage ... Commitment ... ACCOUNTABILITY

Re: [BUG] perf failed on multi-cpus system with multi-events specified

[Eric Dumazet]

Le mercredi 02 mars 2011 à 11:58 +0800, Han Pingtian a écrit :
> Hi,
> 
> On a Nehalem-EX system, which has 24 CPUs, perf commands with
> multi-events specified fail:
> 
> [root@intel-s3e36-01 perf]# ./perf record -e
> syscalls:sys_enter_sched_get_priority_min  -e kmem:mm_kernel_pagefault
> -e workqueue:workqueue_execution  -e writeback:writeback_pages_written
> -e kmem:mm_kernel_pagefault  -e writeback:writeback_bdi_unregister  -e
> syscalls:sys_exit_capset  -e syscalls:sys_enter_sched_get_priority_min
> -e writeback:writeback_exec  -e syscalls:sys_enter_vmsplice  -e
> syscalls:sys_exit_mprotect  -e block:block_rq_complete  -e
> syscalls:sys_enter_readlinkat  -e signal:signal_overflow_fail  -e
> syscalls:sys_exit_sendfile64  -e block:block_bio_backmerge  -e
> syscalls:sys_exit_unlink  -e syscalls:sys_exit_keyctl  -e
> jbd2:jbd2_submit_inode_data  -e syscalls:sys_enter_tgkill  -e
> syscalls:sys_exit_lremovexattr  -e syscalls:sys_exit_munmap  -e
> syscalls:sys_enter_mmap  -e syscalls:sys_exit_sched_getparam  -e
> syscalls:sys_exit_getpgid  -e syscalls:sys_exit_clock_getres  -e
> block:block_split  -e ext4:ext4_mballoc_discard  -e task-clock  -e
> syscalls:sys_enter_sendfile64  -e syscalls:sys_enter_ptrace  -e
> syscalls:sys_enter_nanosleep  -e syscalls:sys_exit_setdomainname  -e
> writeback:writeback_nowork  -e syscalls:sys_exit_mbind  -e
> ext4:ext4_da_write_begin  -e syscalls:sys_enter_recvfrom  -e
> syscalls:sys_enter_write  -e syscalls:sys_exit_getresgid  -e
> syscalls:sys_exit_fsync  -e syscalls:sys_enter_rt_sigprocmask  -e
> syscalls:sys_enter_getdents  -e syscalls:sys_exit_semtimedop -f ls
> 
>   Error: sys_perf_event_open() syscall returned with -1 (Too many open
> files).  /bin/dmesg may provide additional information.
> 
>   Fatal: No CONFIG_PERF_EVENTS=y kernel support configured?
> 
> trace-cmd: Too many open files
>   No memory to alloc tracepoints list
> [root@intel-s3e36-01 perf]# ./perf --version
> perf version 2.6.38.rc6.166.g493f33
> [root@intel-s3e36-01 perf]#
> 
> On another latop which has only two CPUs, there is no this problem.

perf needs nr_cpus * nr_events file descriptors

Try before :

ulimit -n 1000000

Re: [BUG] perf failed on multi-cpus system with multi-events specified

[Han Pingtian]

On Wed, Mar 02, 2011 at 05:35:30AM +0100, Eric Dumazet wrote:
> Le mercredi 02 mars 2011 à 11:58 +0800, Han Pingtian a écrit :
> > Hi,
> > 
> > On a Nehalem-EX system, which has 24 CPUs, perf commands with
> > multi-events specified fail:
> > 
> > [root@intel-s3e36-01 perf]# ./perf record -e
> > syscalls:sys_enter_sched_get_priority_min  -e kmem:mm_kernel_pagefault
> > -e workqueue:workqueue_execution  -e writeback:writeback_pages_written
> > -e kmem:mm_kernel_pagefault  -e writeback:writeback_bdi_unregister  -e
> > syscalls:sys_exit_capset  -e syscalls:sys_enter_sched_get_priority_min
> > -e writeback:writeback_exec  -e syscalls:sys_enter_vmsplice  -e
> > syscalls:sys_exit_mprotect  -e block:block_rq_complete  -e
> > syscalls:sys_enter_readlinkat  -e signal:signal_overflow_fail  -e
> > syscalls:sys_exit_sendfile64  -e block:block_bio_backmerge  -e
> > syscalls:sys_exit_unlink  -e syscalls:sys_exit_keyctl  -e
> > jbd2:jbd2_submit_inode_data  -e syscalls:sys_enter_tgkill  -e
> > syscalls:sys_exit_lremovexattr  -e syscalls:sys_exit_munmap  -e
> > syscalls:sys_enter_mmap  -e syscalls:sys_exit_sched_getparam  -e
> > syscalls:sys_exit_getpgid  -e syscalls:sys_exit_clock_getres  -e
> > block:block_split  -e ext4:ext4_mballoc_discard  -e task-clock  -e
> > syscalls:sys_enter_sendfile64  -e syscalls:sys_enter_ptrace  -e
> > syscalls:sys_enter_nanosleep  -e syscalls:sys_exit_setdomainname  -e
> > writeback:writeback_nowork  -e syscalls:sys_exit_mbind  -e
> > ext4:ext4_da_write_begin  -e syscalls:sys_enter_recvfrom  -e
> > syscalls:sys_enter_write  -e syscalls:sys_exit_getresgid  -e
> > syscalls:sys_exit_fsync  -e syscalls:sys_enter_rt_sigprocmask  -e
> > syscalls:sys_enter_getdents  -e syscalls:sys_exit_semtimedop -f ls
> > 
> >   Error: sys_perf_event_open() syscall returned with -1 (Too many open
> > files).  /bin/dmesg may provide additional information.
> > 
> >   Fatal: No CONFIG_PERF_EVENTS=y kernel support configured?
> > 
> > trace-cmd: Too many open files
> >   No memory to alloc tracepoints list
> > [root@intel-s3e36-01 perf]# ./perf --version
> > perf version 2.6.38.rc6.166.g493f33
> > [root@intel-s3e36-01 perf]#
> > 
> > On another latop which has only two CPUs, there is no this problem.
> 
> perf needs nr_cpus * nr_events file descriptors
> 
> Try before :
> 
> ulimit -n 1000000
It works. Thanks!
> 
> 

-- 
Han Pingtian
Quality Engineer
hpt @ #kernel-qe
Red Hat, Inc
Freedom ... courage ... Commitment ... ACCOUNTABILITY

Re: [BUG] perf failed on multi-cpus system with multi-events specified

[Peter Zijlstra]

On Wed, 2011-03-02 at 11:58 +0800, Han Pingtian wrote:
>   Error: sys_perf_event_open() syscall returned with -1 (Too many open
> files).  /bin/dmesg may provide additional information.

Well, you tried to use 42 events (quick count), we need to open a fd for
each cpu for each event, which results in 1008 fds, which together with
some actual file handles might have tickled your fd limit.

Re: [BUG] perf failed on multi-cpus system with multi-events specified

[Han Pingtian]

On Wed, Mar 02, 2011 at 10:38:36AM +0100, Peter Zijlstra wrote:
> On Wed, 2011-03-02 at 11:58 +0800, Han Pingtian wrote:
> >   Error: sys_perf_event_open() syscall returned with -1 (Too many open
> > files).  /bin/dmesg may provide additional information.
> 
> Well, you tried to use 42 events (quick count), we need to open a fd for
> each cpu for each event, which results in 1008 fds, which together with
> some actual file handles might have tickled your fd limit.
Got it. Thanks.

-- 
Han Pingtian
Quality Engineer
hpt @ #kernel-qe
Red Hat, Inc
Freedom ... courage ... Commitment ... ACCOUNTABILITY