[BUG] perf: can not resolve symbols for forked threads

[BUG] perf: can not resolve symbols for forked threads

[Tony Lu]

Hi

I got the below output that shows perf can not resolve symbols for forked threads. I did a system-wide collection from all CPUs after the application hello run.

-bash-4.1$ ./perf --version
perf version 3.16.rc1.ge99cfa2
-bash-4.1$ ./hello & ./perf record -a sleep 5
-bash-4.1$ ./perf report
# Overhead      Command       Shared Object                         Symbol
# ........  ...........  ..................  .............................
#
    54.77%        hello  [unknown]           [.] 0x0000000000400610       
    28.59%        hello  [unknown]           [.] 0x0000000000400615       
    16.53%        hello  [unknown]           [.] 0x000000000040061f       
     0.01%        hello  [kernel.kallsyms]   [k] __rcu_process_callbacks  
     0.01%         perf  [kernel.kallsyms]   [k] copy_user_generic_string 
     0.01%        sleep  [kernel.kallsyms]   [k] filemap_fault
....

/***************************************************************************** FILE: hello.c
** DESCRIPTION:
** A "hello world" Pthreads program to trigger this bug.
***************************************************************************/
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define NUM_THREADS     5

void foo(void)
{
  long i = 1000000000;
  while (i--) {
    ;
  }
}

void *PrintHello(void *threadid)
{
   long tid;
   tid = (long)threadid;
   printf("Hello World! It's me, thread #%ld!\n", tid);
   foo();
   pthread_exit(NULL);
}

int main(int argc, char *argv[])
{
   pthread_t threads[NUM_THREADS];
   int rc;
   long t;
   for(t=0;t<NUM_THREADS;t++){
     printf("In main: creating thread %ld\n", t);
     rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t);
     if (rc){
       printf("ERROR; return code from pthread_create() is %d\n", rc);
       exit(-1);
       }
     }

   /* Last thing that main() should do */
   pthread_exit(NULL);
}

Thanks
-Tony

Re: [BUG] perf: can not resolve symbols for forked threads

[Peter Zijlstra]

On Thu, Jun 19, 2014 at 07:39:42AM +0000, Tony Lu wrote:
> Hi
> 
> I got the below output that shows perf can not resolve symbols for
> forked threads. I did a system-wide collection from all CPUs after the
> application hello run.

There's no fork() in... :-)

> #include <pthread.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> #define NUM_THREADS     5
> 
> void foo(void)
> {
>   long i = 1000000000;
>   while (i--) {
>     ;
>   }
> }
> 
> void *PrintHello(void *threadid)
> {
>    long tid;
>    tid = (long)threadid;
>    printf("Hello World! It's me, thread #%ld!\n", tid);
>    foo();
>    pthread_exit(NULL);
> }
> 
> int main(int argc, char *argv[])
> {
>    pthread_t threads[NUM_THREADS];
>    int rc;
>    long t;
>    for(t=0;t<NUM_THREADS;t++){
>      printf("In main: creating thread %ld\n", t);
>      rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t);
>      if (rc){
>        printf("ERROR; return code from pthread_create() is %d\n", rc);
>        exit(-1);
>        }
>      }
> 
>    /* Last thing that main() should do */
>    pthread_exit(NULL);
> }

That pthread_exit() is the problem; this results in:

29456 pts/23   Zl     0:00          |               \_ [hello] <defunct>

You want to wait for the threads to complete using pthread_join().

I suspect the defunct state hides the process.

RE: [BUG] perf: can not resolve symbols for forked threads

[Tony Lu]

Thanks for your reply.

>-----Original Message-----
>From: Peter Zijlstra [mailto:peterz@infradead.org]
>Sent: Monday, June 23, 2014 5:13 PM
>To: Tony Lu
>Cc: Paul Mackerras; Ingo Molnar; Arnaldo Carvalho de Melo;
>linux-kernel@vger.kernel.org; Chris Metcalf
>Subject: Re: [BUG] perf: can not resolve symbols for forked threads
>
>On Thu, Jun 19, 2014 at 07:39:42AM +0000, Tony Lu wrote:
>> Hi
>>
>> I got the below output that shows perf can not resolve symbols for
>> forked threads. I did a system-wide collection from all CPUs after the
>> application hello run.
>
>There's no fork() in... :-)

Yes, but perf regards pthread_created threads as forked threads, and a PERF_RECORD_FORK event will be delivered when a thread is forked or cloned.

>
>> #include <pthread.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <unistd.h>
>> #define NUM_THREADS     5
>>
>> void foo(void)
>> {
>>   long i = 1000000000;
>>   while (i--) {
>>     ;
>>   }
>> }
>>
>> void *PrintHello(void *threadid)
>> {
>>    long tid;
>>    tid = (long)threadid;
>>    printf("Hello World! It's me, thread #%ld!\n", tid);
>>    foo();
>>    pthread_exit(NULL);
>> }
>>
>> int main(int argc, char *argv[])
>> {
>>    pthread_t threads[NUM_THREADS];
>>    int rc;
>>    long t;
>>    for(t=0;t<NUM_THREADS;t++){
>>      printf("In main: creating thread %ld\n", t);
>>      rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t);
>>      if (rc){
>>        printf("ERROR; return code from pthread_create() is %d\n", rc);
>>        exit(-1);
>>        }
>>      }
>>
>>    /* Last thing that main() should do */
>>    pthread_exit(NULL);
>> }
>
>That pthread_exit() is the problem; this results in:
>
>29456 pts/23   Zl     0:00          |               \_ [hello] <defunct>
>
>You want to wait for the threads to complete using pthread_join().
>
>I suspect the defunct state hides the process.

Yes, using pthread_join() can workaround this problem.
Does that mean the parent thread can not exit before the child threads? It still seems like a perf bug.

-Thanks
-Tony

Re: [BUG] perf: can not resolve symbols for forked threads

[Peter Zijlstra]

On Tue, Jun 24, 2014 at 06:47:38AM +0000, Tony Lu wrote:
> >That pthread_exit() is the problem; this results in:
> >
> >29456 pts/23   Zl     0:00          |               \_ [hello] <defunct>
> >
> >You want to wait for the threads to complete using pthread_join().
> >
> >I suspect the defunct state hides the process.
> 
> Yes, using pthread_join() can workaround this problem.
> Does that mean the parent thread can not exit before the child threads? It still seems like a perf bug.

So we scan /proc at record start to acquire the state for the existing
tasks, it might be defunct tasks simply aren't there anymore, in which
case there's nothing we can do.

But I haven't checked..

That said, yes it would be nice if we could cure this.