core dump problem with a multi-threaded program

core dump problem with a multi-threaded program

[Yahui Lin]

[-- Attachment #1: Type: text/plain, Size: 1872 bytes --]

Hi,

We are facing the problem that a post-mortem investigation of a
multi-threaded program is not possible under RedHat Linux 7.1 (kernel
version 2.4.2-2), because loading the core into gdb 5.0 does not show
the correct crash location.

Attached is a test program, linux.c.
This is a producer-consumer program, with pairs of threads sending each
other messages. Its structure is described in comments at the very
beginning of source code. If 0 is typed as stdin, a divide-by-zero
exception happens.

If you use GDB to look into the core, you'll see it gets some errors
while loading, and doesn't show the correct line which causes the
arithmetic exception:

rock[19] gdb yahuitest/p15 core
GNU gdb 5.0rh-5 Red Hat Linux 7.1
Copyright 2001 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux"...
Core was generated by `yahuitest/p15 100000'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/i686/libpthread.so.0...done.

warning: Unable to set global thread event mask: generic error
[New Thread 1024 (LWP 5458)]
Error while reading shared library symbols:
Can't attach LWP 5458: No such process
...

Obviously, if one runs the program in the debugger, it shows the correct
location.

Another problem is that core is not always dumpable. I repeat running
this program with 0.csh script as attached. You'll find it stops with no
core dumped after few iterations.

Does anyone know if this is a problem that we can work-around with the
current versions of the kernel (eg. setting a signal handler, etc)? Do
we need a newer version of GDB?

Thanks.

Yahui

[-- Attachment #2: 0.csh --]
[-- Type: application/x-csh, Size: 261 bytes --]

[-- Attachment #3: linux.c --]
[-- Type: text/plain, Size: 4724 bytes --]

/* linux.c
 *
 * This is a N-paired program. Each consumer takes message from the buffer in which
 * the corresponding producer puts data. Each pair is independent of the others. 
 * A producer is randomly picked to read from stdin. If 0 is typed as stdin, an
 * arithmetic exception happens. Core file is expected to dump at this point.
 * Mutex is used to synchronize the communication between producer and consumer. 
 * To guarantee each message is consumed before producer puts in new one, condition 
 * variables are used.
 *
 * command line: linux 1000000
 * The second argument is the time (microseconds) the producer sleeps aftet each
 * iteration.
 *
 * Written by Yahui Lin
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <pthread.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>

#define  NUMPAIRS       15

int     buffer[NUMPAIRS];
int     seg_fault;
int     p_sleep;

pthread_mutex_t     prod_mutex[NUMPAIRS], cons_mutex[NUMPAIRS];
pthread_t           prod_id[NUMPAIRS], cons_id[NUMPAIRS];
pthread_cond_t      prod_cond[NUMPAIRS], cons_cond[NUMPAIRS];

void *producer(void *arg);
void *consumer(void *arg);

void  sighandler(int sig)
{
     if( sig != SIGFPE )
        fprintf(stderr, "unexpected signal in handler: %d\n", sig);
     else
        abort();
}


int main(int argc, char *argv[])
{
     int                i;
     struct sigaction   act;
     sigset_t		mask_sig;
     

     if( argc != 2 )
     {
        fprintf(stderr, "usage: %s prod_sleep_time\n", argv[0]);
        exit(1);
     }
     if( (p_sleep  = atoi(argv[1])) < 0 )
        p_sleep = 0;

     sigemptyset(&mask_sig);
     sigaddset(&mask_sig, SIGFPE);

     act.sa_handler = sighandler;
     act.sa_mask = mask_sig;
     act.sa_flags = 0;
     if(sigaction(SIGFPE, &act, NULL) == -1){
        perror("sigaction error");
        exit(1);
     }

     for(i=0; i<NUMPAIRS; i++){
         buffer[i] = 999;
         pthread_mutex_init(&prod_mutex[i], NULL);
         pthread_mutex_init(&cons_mutex[i], NULL);
         if(pthread_cond_init(&prod_cond[i], NULL) != 0 )
         {
            perror("producer condition init error");
            exit(1);
         }
         if(pthread_cond_init(&cons_cond[i], NULL) != 0 )
         {
            perror("consumer condition init error");
            exit(1);
         }
     }
     srand( (unsigned)time( NULL ) );
     seg_fault = rand() % NUMPAIRS;
     fprintf(stderr, "producer %d can signal consumer to generate an exception\n",
             seg_fault);

     for(i=0; i<NUMPAIRS; i++)
     {
         if(pthread_create(&prod_id[i], NULL, producer, (void *)i) != 0)
         {
            perror("producer thread create error");
            exit(2);
         }
         if(pthread_create(&cons_id[i], NULL, consumer, (void *)i) != 0)
         {
            perror("consumer thread create error");
            exit(3);
         }
     }

     for(i=0; i<NUMPAIRS; i++)
         if( pthread_join(cons_id[i], NULL) != 0 )
            perror("pthread join error");

     while(1)
         sleep(1);

     fprintf(stderr, "unexpected exit\n");  /* we don't want this happened */
     exit(0);
}


void *producer(void *arg)
{
    int    id;
    char   input[10];

    id = (int)arg;
    fprintf(stderr, "producer %2d starts\n", id);

    while(1)
    {
        pthread_mutex_lock(&prod_mutex[id]);
        while( buffer[id] != 999 )  /* number isn't consumed yet */
              pthread_cond_wait(&prod_cond[id], &prod_mutex[id]);
        pthread_mutex_unlock(&prod_mutex[id]);
        if( id == seg_fault )
        {
           fgets(input, sizeof(input), stdin);
           buffer[id] = strtol(input, NULL, 10);
        }
        else
           buffer[id] = 100;
        pthread_mutex_lock(&cons_mutex[id]);
        pthread_cond_signal(&cons_cond[id]);
        pthread_mutex_unlock(&cons_mutex[id]);

        if( p_sleep )
           usleep(p_sleep);
    }
}


void *consumer (void *arg)
{
    int   test, id;

    id = (int)arg;

    fprintf(stderr, "consumer %2d starts\n", id);

    while(1)
    {
        pthread_mutex_lock(&cons_mutex[id]);
        while( buffer[id] == 999 )  /* number isn't put in yet */
              pthread_cond_wait(&cons_cond[id], &cons_mutex[id]);
        pthread_mutex_unlock(&cons_mutex[id]);

        /* exception will happen here if buffer[id]=0 */
        if( buffer[id] == 0 )
           fprintf(stderr, "consumer %d is going to cause an exception\n", id);
        test = 10 / buffer[id];
        buffer[id] = 999;
        pthread_mutex_lock(&prod_mutex[id]);
        pthread_cond_signal(&prod_cond[id]);
        pthread_mutex_unlock(&prod_mutex[id]);
    }
}

Re: core dump problem with a multi-threaded program

[Alan Cox]

> multi-threaded program is not possible under RedHat Linux 7.1 (kernel
> version 2.4.2-2), because loading the core into gdb 5.0 does not show
> the correct crash location.

2.4.2 doesn't support multithreaded core dumps. 
The RH errata kernel will generate a dump for each thread as/if/when that thread
crashes

You can then inspect the relevant core.pid file, I've no idea how well the
gdb thread stuff works with it.