[Starpu-devel] Problem with pthread_mutex_lock

[Nathalie Furmento <nathalie.furmento@labri.fr>]

Hi starpu-devel,

The application ./tests/core/subgraph_repeat_regenerate fails every now 
and then on a pthread_mutex_lock. This also happens i think with other 
applications.

You can find below the backtrace of the threads.

Thanks,

Nathalie


$ libtool --mode=execute gdb ./tests/core/subgraph_repeat_regenerate
....
wait for worker 0
pthread_mutex_lock : Invalid argument
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff110d910 (LWP 27512)]
0x00007ffff2c05f45 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007ffff2c05f45 in raise () from /lib/libc.so.6
#1  0x00007ffff2c08d80 in abort () from /lib/libc.so.6
#2  0x00007ffff7b97182 in _starpu_not_all_task_deps_are_fulfilled 
(j=0x145aad0, job_is_already_locked=0) at ../../trunk/src/core/jobs.c:210
#3  0x00007ffff7b97243 in _starpu_enforce_deps_and_schedule 
(j=0x145aad0, job_is_already_locked=0) at ../../trunk/src/core/jobs.c:244
#4  0x00007ffff7b9798b in _starpu_submit_job (j=0x145aad0, 
do_not_increment_nsubmitted=1) at ../../trunk/src/core/task.c:158
#5  0x00007ffff7b97044 in _starpu_handle_job_termination (j=0x145aad0) 
at ../../trunk/src/core/jobs.c:160
#6  0x00007ffff7bb98e4 in _starpu_cpu_worker (arg=0x7ffff7dd5480) at 
../../trunk/src/drivers/cpu/driver_cpu.c:197
#7  0x00007ffff2f2e73a in start_thread () from /lib/libpthread.so.0
#8  0x00007ffff2c9f69d in clone () from /lib/libc.so.6
#9  0x0000000000000000 in ?? ()
(gdb) thread apply all bt

Thread 5 (Thread 0x7ffff090c910 (LWP 27513)):
#0  0x00007ffff7df6784 in ?? () from /lib64/ld-linux-x86-64.so.2
#1  0x00007ffff7de8acc in ?? () from /lib64/ld-linux-x86-64.so.2
#2  0x00007ffff7df263c in ?? () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7dee0c6 in ?? () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff7df1fd7 in ?? () from /lib64/ld-linux-x86-64.so.2
#5  0x00007ffff2cd59f0 in ?? () from /lib/libc.so.6
#6  0x00007ffff7dee0c6 in ?? () from /lib64/ld-linux-x86-64.so.2
#7  0x00007ffff2cd5a8f in ?? () from /lib/libc.so.6
#8  0x00007ffff2cd5b87 in __libc_dlopen_mode () from /lib/libc.so.6
#9  0x00007ffff2f3708c in pthread_cancel_init () from /lib/libpthread.so.0
#10 0x00007ffff2f371ec in _Unwind_ForcedUnwind () from /lib/libpthread.so.0
#11 0x00007ffff2f350f0 in __pthread_unwind () from /lib/libpthread.so.0
#12 0x00007ffff2f2f815 in pthread_exit () from /lib/libpthread.so.0
#13 0x00007ffff7bb99a6 in _starpu_cpu_worker (arg=0x7ffff7dd5588) at 
../../trunk/src/drivers/cpu/driver_cpu.c:223
#14 0x00007ffff2f2e73a in start_thread () from /lib/libpthread.so.0
#15 0x00007ffff2c9f69d in clone () from /lib/libc.so.6
#16 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7ffff110d910 (LWP 27512)):
#0  0x00007ffff2c05f45 in raise () from /lib/libc.so.6
#1  0x00007ffff2c08d80 in abort () from /lib/libc.so.6
#2  0x00007ffff7b97182 in _starpu_not_all_task_deps_are_fulfilled 
(j=0x145aad0, job_is_already_locked=0) at ../../trunk/src/core/jobs.c:210
#3  0x00007ffff7b97243 in _starpu_enforce_deps_and_schedule 
(j=0x145aad0, job_is_already_locked=0) at ../../trunk/src/core/jobs.c:244
#4  0x00007ffff7b9798b in _starpu_submit_job (j=0x145aad0, 
do_not_increment_nsubmitted=1) at ../../trunk/src/core/task.c:158
#5  0x00007ffff7b97044 in _starpu_handle_job_termination (j=0x145aad0) 
at ../../trunk/src/core/jobs.c:160
#6  0x00007ffff7bb98e4 in _starpu_cpu_worker (arg=0x7ffff7dd5480) at 
../../trunk/src/drivers/cpu/driver_cpu.c:197
#7  0x00007ffff2f2e73a in start_thread () from /lib/libpthread.so.0
#8  0x00007ffff2c9f69d in clone () from /lib/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7ffff190e910 (LWP 27511)):
#0  0x00007ffff2f355f5 in __lll_unlock_wake () from /lib/libpthread.so.0
#1  0x00007ffff2f322c4 in _L_unlock_524 () from /lib/libpthread.so.0
#2  0x00007ffff2f321f4 in __pthread_mutex_unlock_usercnt () from 
/lib/libpthread.so.0
#3  0x00007ffff7df2005 in ?? () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff2cd59f0 in ?? () from /lib/libc.so.6
#5  0x00007ffff7dee0c6 in ?? () from /lib64/ld-linux-x86-64.so.2
#6  0x00007ffff2cd5a8f in ?? () from /lib/libc.so.6
#7  0x00007ffff2cd5b87 in __libc_dlopen_mode () from /lib/libc.so.6
#8  0x00007ffff2f3708c in pthread_cancel_init () from /lib/libpthread.so.0
#9  0x00007ffff2f371ec in _Unwind_ForcedUnwind () from /lib/libpthread.so.0
#10 0x00007ffff2f350f0 in __pthread_unwind () from /lib/libpthread.so.0
#11 0x00007ffff2f2f815 in pthread_exit () from /lib/libpthread.so.0
#12 0x00007ffff7bb99a6 in _starpu_cpu_worker (arg=0x7ffff7dd5378) at 
../../trunk/src/drivers/cpu/driver_cpu.c:223
#13 0x00007ffff2f2e73a in start_thread () from /lib/libpthread.so.0
#14 0x00007ffff2c9f69d in clone () from /lib/libc.so.6
#15 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7ffff210f910 (LWP 27510)):
#0  0x00007ffff3454310 in ?? () from /usr/lib/libcuda.so.1
#1  0x00007ffff345d323 in ?? () from /usr/lib/libcuda.so.1
#2  0x00007ffff344ab55 in ?? () from /usr/lib/libcuda.so.1
#3  0x00007ffff34de65e in ?? () from /usr/lib/libcuda.so.1
#4  0x00007ffff3c2ecdc in ?? () from /usr/local/cuda/lib64/libcudart.so.3
#5  0x00007ffff3c26913 in ?? () from /usr/local/cuda/lib64/libcudart.so.3
#6  0x00007ffff3c09317 in ?? () from /usr/local/cuda/lib64/libcudart.so.3
#7  0x00007ffff3c31129 in ?? () from /usr/local/cuda/lib64/libcudart.so.3
#8  0x00007ffff3c090c1 in cudaThreadExit () from 
/usr/local/cuda/lib64/libcudart.so.3
#9  0x00007ffff7bb9adc in deinit_context (workerid=0) at 
../../trunk/src/drivers/cuda/driver_cuda.c:58
#10 0x00007ffff7bba36d in _starpu_cuda_worker (arg=0x7ffff7dd5270) at 
../../trunk/src/drivers/cuda/driver_cuda.c:291
---Type <return> to continue, or q <return> to quit---
#11 0x00007ffff2f2e73a in start_thread () from /lib/libpthread.so.0
#12 0x00007ffff2c9f69d in clone () from /lib/libc.so.6
#13 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7ed5710 (LWP 27507)):
#0  0x00007ffff2f2f99d in pthread_join () from /lib/libpthread.so.0
#1  0x00007ffff7b989dd in _starpu_terminate_workers 
(config=0x7ffff7dd50c0) at ../../trunk/src/core/workers.c:348
#2  0x00007ffff7b99013 in starpu_shutdown () at 
../../trunk/src/core/workers.c:510
#3  0x0000000000400c4a in main (argc=<value optimized out>, argv=<value 
optimized out>) at ../../trunk/tests/core/subgraph_repeat_regenerate.c:131
(gdb)