[Starpu-devel] Deadlock when calling starpu_shutdown

[Nathalie Furmento <nathalie.furmento@labri.fr>]

When running the pi/pi_redux example on the AC cluster (with the default 
StarPU configuration), the application sometimes deadlocks when calling 
starpu_shutdown().

Here the backtrace obtained from gdb:

(gdb) bt
#0  0x00007f49c7aa1d5d in pthread_join () from /lib64/libpthread.so.0
#1  0x00007f49cea359e9 in _starpu_terminate_workers 
(config=0x7f49cec80300) at ../../src/core/workers.c:397
#2  0x00007f49cea35bb4 in starpu_shutdown () at ../../src/core/workers.c:467
#3  0x0000000000401df3 in main (argc=1, argv=0x7fff059fdee8) at 
../../examples/pi/pi_redux.c:287

(gdb) info thread
  2 Thread 0x7f49c5307710 (LWP 29062)  0x00007f49c7aa5ec5 in ?? () from 
/lib64/libpthread.so.0
* 1 Thread 0x7f49c7310740 (LWP 29038)  0x00007f49c7aa1d5d in 
pthread_join () from /lib64/libpthread.so.0

(gdb) thread 2
[Switching to thread 2 (Thread 0x7f49c5307710 (LWP 29062))]#0  
0x00007f49c7aa5ec5 in ?? () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f49c7aa5ec5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f49cea32f23 in _starpu_spin_lock (lock=0x249d3e0) at 
../../src/common/starpu_spinlock.c:71
#2  0x00007f49cea3aee9 in _starpu_notify_data_dependencies 
(handle=0x249d3d0) at ../../src/core/dependencies/data_concurrency.c:335
#3  0x00007f49cea47269 in _starpu_release_data_on_node 
(handle=0x249d3d0, default_wt_mask=0, replicate=0x249d410) at 
../../src/datawizard/coherency.c:374
#4  0x00007f49cea4762c in _starpu_push_task_output (task=0x24a2ee0, 
mask=0) at ../../src/datawizard/coherency.c:513
#5  0x00007f49cea5f7d3 in execute_job_on_cpu (j=0x24a3030, 
cpu_args=0x7f49cec80b20, is_parallel_task=0, rank=0, 
perf_arch=STARPU_CPU_DEFAULT)
    at ../../src/drivers/cpu/driver_cpu.c:99
#6  0x00007f49cea5fda2 in _starpu_cpu_worker (arg=0x7f49cec80b20) at 
../../src/drivers/cpu/driver_cpu.c:217
#7  0x00007f49cec9cfb0 in start_routine_wrapper (_arg=0x249c1a0) at 
cuda_wrapper.c:1276
#8  0x00007f49c7aa08e0 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f49c77fdc9d in clone () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()

The output of the application is as follows:

[starpu][_starpu_opencl_init] Initialising OpenCL
[starpu][_starpu_opencl_init] Platforms detected: 1
[starpu][_starpu_opencl_init] Platform: NVIDIA CUDA - NVIDIA Corporation
[starpu][_starpu_opencl_init]   0 devices detected
[starpu][load_sched_policy] Use eager scheduler (greedy policy)
[starpu][_starpu_launch_drivers] initialising worker 0
[starpu][_starpu_launch_drivers] initialising worker 1
[starpu][_starpu_launch_drivers] initialising worker 2
[starpu][_starpu_launch_drivers] initialising worker 3
[starpu][_starpu_cpu_worker] cpu worker 1 is ready on logical cpu 2
[starpu][_starpu_cpu_worker] cpu worker 0 is ready on logical cpu 1
[starpu][_starpu_cuda_worker] cuda (Tesla T10 Processor) dev id 0 thread 
is ready to run on CPU 3 !
[starpu][_starpu_cpu_worker] cpu worker 2 is ready on logical cpu 0
Pi approximation : 3.141604 (843317769 / 1073741824)
Error 1.084979e-05
Total time : 891.155000 ms
Speed : 1.204888 GShot/s
[starpu][_starpu_terminate_workers] wait for worker 0
[starpu][_starpu_terminate_workers] wait for worker 1
[starpu][_starpu_terminate_workers] wait for worker 2
[starpu][_starpu_terminate_workers] wait for worker 3

Thanks,

Nathalie