[Starpu-devel] Deadlock in StarPU-MPI

[Martin Khannouz <martin.khannouz@inria.fr>]

Bonjour,

J'ai rencontré un deadlock (ou livelock) avec ScalFMM en utilisant StarPU-MPI avec les communications implicites.

J'ai utilisé la branche starpu/branches/starpu-1.2 à la révision r18437. Le compilateur utilisé est GCC 5.3 et la lib MPI est OpenMPI 2.0. L'éxécution utilise 5 process MPI exécutés sur 5 nœuds miriel.

le fichier all_bt contient la liste des backtrace de tous les threads.

Le fichier bt_starpumpi est la backtrace du thread de StarPU-MPI et sa transformation après plusieurs appels à finish dans gdb.

À priori, le thread de StarPU-MPI est bloqué en train de faire un rendez-vous dans une fonction qui est censé être non-bloquante.

Cordialement,

Martin.

Thread 41 (Thread 0x7f8662953700 (LWP 88272)):
#0  0x00007f8664a96863 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f866448e24b in epoll_dispatch (base=0x19ac5d0, tv=<optimized out>) 
at epoll.c:407
#2  0x00007f866449162e in opal_libevent2022_event_base_loop (base=0x19ac5d0, 
flags=1) at event.c:1630
#3  0x00007f8662978fbd in progress_engine () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_pmix_pmix112.so
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f8662152700 (LWP 88273)):
#0  0x00007f8664a8bc3d in poll () from /lib64/libc.so.6
#1  0x00007f866449abfb in poll_dispatch (base=0x19ad8f0, tv=<optimized out>) 
at poll.c:165
#2  0x00007f866449162e in opal_libevent2022_event_base_loop (base=0x19ad8f0, 
flags=1) at event.c:1630
#3  0x00007f86644566fe in progress_engine () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/libopen-pal.so.20
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f8652da7700 (LWP 88275)):
#0  0x00007f8664a8bc3d in poll () from /lib64/libc.so.6
#1  0x00007f8659abc57a in ips_ptl_pollintr (rcvthreadc=0x1b10bf8) at 
ptl_rcvthread.c:324
#2  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7f856854e700 (LWP 88276)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f8567d4d700 (LWP 88277)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f856754c700 (LWP 88278)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
---Type <return> to continue, or q <return> to quit---
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f8566d4b700 (LWP 88279)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f856654a700 (LWP 88280)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f8565d49700 (LWP 88281)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f8565548700 (LWP 88282)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f8564d47700 (LWP 88283)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
---Type <return> to continue, or q <return> to quit---
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f8564546700 (LWP 88284)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f8563d45700 (LWP 88285)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f8563544700 (LWP 88286)):
#0  futex_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/x86/futex.h:45
#1  do_wait (val=16, addr=0x1b25004) at 
../../../gcc-5.3.0/libgomp/config/linux/wait.h:67
#2  gomp_barrier_wait_end (bar=0x1b25000, state=16) at 
../../../gcc-5.3.0/libgomp/config/linux/bar.c:48
#3  0x00007f86651a3068 in gomp_thread_start (xdata=<optimized out>) at 
../../../gcc-5.3.0/libgomp/team.c:122
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f8533fff700 (LWP 88329)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f8533ffe9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f8533ffe9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666290940 <_starpu_config+1184>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666290940 <_starpu_config+1184>, 
workerid=workerid@entry=0, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666290940 <_starpu_config+1184>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666290940 
<_starpu_config+1184>) at drivers/cpu/driver_cpu.c:361
---Type <return> to continue, or q <return> to quit---
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f85337fe700 (LWP 88330)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f85337fd9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f85337fd9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666290d48 <_starpu_config+2216>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666290d48 <_starpu_config+2216>, 
workerid=workerid@entry=1, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666290d48 <_starpu_config+2216>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666290d48 
<_starpu_config+2216>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f8532ffd700 (LWP 88331)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f8532ffc9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f8532ffc9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666291150 <_starpu_config+3248>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666291150 <_starpu_config+3248>, 
workerid=workerid@entry=2, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666291150 <_starpu_config+3248>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666291150 
<_starpu_config+3248>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f85327fc700 (LWP 88332)):
#0  _starpu_fifo_empty (fifo=0x190584930) at sched_policies/fifo_queues.c:71
#1  0x00007f8665fdad49 in pop_task_eager_policy (sched_ctx_id=0) at 
sched_policies/eager_central_policy.c:166
#2  0x00007f8665fd2a96 in _starpu_pop_task 
(worker=worker@entry=0x7f8666291558 <_starpu_config+4280>) at 
core/sched_policy.c:863
#3  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666291558 <_starpu_config+4280>, 
workerid=workerid@entry=3, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#4  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666291558 <_starpu_config+4280>)
    at drivers/cpu/driver_cpu.c:247
---Type <return> to continue, or q <return> to quit---
#5  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666291558 
<_starpu_config+4280>) at drivers/cpu/driver_cpu.c:361
#6  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f8531ffb700 (LWP 88333)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f8531ffa9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f8531ffa9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666291960 <_starpu_config+5312>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666291960 <_starpu_config+5312>, 
workerid=workerid@entry=4, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666291960 <_starpu_config+5312>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666291960 
<_starpu_config+5312>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f85317fa700 (LWP 88334)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f85317f99c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f85317f99f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666291d68 <_starpu_config+6344>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666291d68 <_starpu_config+6344>, 
workerid=workerid@entry=5, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666291d68 <_starpu_config+6344>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666291d68 
<_starpu_config+6344>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f8530ff9700 (LWP 88335)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f8530ff89c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f8530ff89f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666292170 <_starpu_config+7376>) at 
core/sched_policy.c:908
---Type <return> to continue, or q <return> to quit---
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666292170 <_starpu_config+7376>, 
workerid=workerid@entry=6, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666292170 <_starpu_config+7376>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666292170 
<_starpu_config+7376>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f852bfff700 (LWP 88336)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f852bffe9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f852bffe9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666292578 <_starpu_config+8408>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666292578 <_starpu_config+8408>, 
workerid=workerid@entry=7, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666292578 <_starpu_config+8408>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666292578 
<_starpu_config+8408>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f852b7fe700 (LWP 88337)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f852b7fd9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f852b7fd9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666292980 <_starpu_config+9440>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666292980 <_starpu_config+9440>, 
workerid=workerid@entry=8, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666292980 <_starpu_config+9440>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666292980 
<_starpu_config+9440>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f852affd700 (LWP 88338)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f852affc9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f852affc9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666292d88 <_starpu_config+10472>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666292d88 <_starpu_config+10472>, 
workerid=workerid@entry=9, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666292d88 <_starpu_config+10472>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666292d88 
<_starpu_config+10472>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f852a7fc700 (LWP 88339)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f852a7fb9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f852a7fb9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666293190 <_starpu_config+11504>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666293190 <_starpu_config+11504>, 
workerid=workerid@entry=10, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666293190 <_starpu_config+11504>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666293190 
<_starpu_config+11504>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f8529ffb700 (LWP 88340)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f8529ffa9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f8529ffa9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666293598 <_starpu_config+12536>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666293598 <_starpu_config+12536>, 
workerid=workerid@entry=11, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666293598 <_starpu_config+12536>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666293598 
<_starpu_config+12536>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 15 (Thread 0x7f85297fa700 (LWP 88341)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f85297f99c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f85297f99f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f86662939a0 <_starpu_config+13568>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662939a0 <_starpu_config+13568>, 
workerid=workerid@entry=12, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662939a0 <_starpu_config+13568>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662939a0 
<_starpu_config+13568>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f8528ff9700 (LWP 88342)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f8528ff89c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f8528ff89f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f8666293da8 <_starpu_config+14600>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666293da8 <_starpu_config+14600>, 
workerid=workerid@entry=13, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666293da8 <_starpu_config+14600>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666293da8 
<_starpu_config+14600>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f850bfff700 (LWP 88343)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f850bffe9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f850bffe9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f86662941b0 <_starpu_config+15632>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662941b0 <_starpu_config+15632>, 
workerid=workerid@entry=14, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662941b0 <_starpu_config+15632>)
---Type <return> to continue, or q <return> to quit---
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662941b0 
<_starpu_config+15632>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f850b7fe700 (LWP 88344)):
#0  0x00007f8664d6d51f in pthread_getspecific () from /lib64/libpthread.so.0
#1  0x00007f8665fb89fb in _starpu_get_local_worker_key () at 
./core/workers.h:427
#2  _starpu_worker_get_id () at ./core/workers.h:542
#3  _starpu_worker_get_id_check (f=f@entry=0x7f8666023e58 
"sched_policies/eager_central_policy.c", l=l@entry=159) at core/workers.c:1702
#4  0x00007f8665fdad33 in pop_task_eager_policy (sched_ctx_id=0) at 
sched_policies/eager_central_policy.c:159
#5  0x00007f8665fd2a96 in _starpu_pop_task 
(worker=worker@entry=0x7f86662945b8 <_starpu_config+16664>) at 
core/sched_policy.c:863
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662945b8 <_starpu_config+16664>, 
workerid=workerid@entry=15, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662945b8 <_starpu_config+16664>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662945b8 
<_starpu_config+16664>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f850affd700 (LWP 88345)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f850affc9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f850affc9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f86662949c0 <_starpu_config+17696>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662949c0 <_starpu_config+17696>, 
workerid=workerid@entry=16, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662949c0 <_starpu_config+17696>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662949c0 
<_starpu_config+17696>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f850a7fc700 (LWP 88346)):
#0  0x00007f8664d6abd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x00007f8665fe4039 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666294dc8 <_starpu_config+18728>, 
workerid=workerid@entry=17, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:383
#2  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666294dc8 <_starpu_config+18728>)
    at drivers/cpu/driver_cpu.c:247
---Type <return> to continue, or q <return> to quit---
#3  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666294dc8 
<_starpu_config+18728>) at drivers/cpu/driver_cpu.c:361
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f8509ffb700 (LWP 88347)):
#0  0x00007f8664d6ac02 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x00007f8665fe4039 in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662951d0 <_starpu_config+19760>, 
workerid=workerid@entry=18, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:383
#2  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662951d0 <_starpu_config+19760>)
    at drivers/cpu/driver_cpu.c:247
#3  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662951d0 
<_starpu_config+19760>) at drivers/cpu/driver_cpu.c:361
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f85097fa700 (LWP 88348)):
#0  0x00007f8664d6bbdc in pthread_mutex_unlock () from /lib64/libpthread.so.0
#1  0x00007f8665fe40fe in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662955d8 <_starpu_config+20792>, 
workerid=workerid@entry=19, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:397
#2  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662955d8 <_starpu_config+20792>)
    at drivers/cpu/driver_cpu.c:247
#3  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662955d8 
<_starpu_config+20792>) at drivers/cpu/driver_cpu.c:361
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f8508ff9700 (LWP 88349)):
#0  0x00007f8664d6bbdc in pthread_mutex_unlock () from /lib64/libpthread.so.0
#1  0x00007f8665fe40fe in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662959e0 <_starpu_config+21824>, 
workerid=workerid@entry=20, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:397
#2  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662959e0 <_starpu_config+21824>)
    at drivers/cpu/driver_cpu.c:247
#3  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662959e0 
<_starpu_config+21824>) at drivers/cpu/driver_cpu.c:361
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f84ebfff700 (LWP 88350)):
#0  0x00007f8664d6ac02 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x00007f8665fe4039 in _starpu_get_worker_task 
(worker=worker@entry=0x7f8666295de8 <_starpu_config+22856>, 
workerid=workerid@entry=21, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:383
#2  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f8666295de8 <_starpu_config+22856>)
    at drivers/cpu/driver_cpu.c:247
#3  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f8666295de8 
<_starpu_config+22856>) at drivers/cpu/driver_cpu.c:361
---Type <return> to continue, or q <return> to quit---
#4  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f84eb7fe700 (LWP 88351)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f84eb7fd9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f84eb7fd9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f86662961f0 <_starpu_config+23888>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662961f0 <_starpu_config+23888>, 
workerid=workerid@entry=22, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662961f0 <_starpu_config+23888>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662961f0 
<_starpu_config+23888>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f84eaffd700 (LWP 88352)):
#0  0x00007ffef2d137d2 in clock_gettime ()
#1  0x00007f8664aa9edd in clock_gettime () from /lib64/libc.so.6
#2  0x00007f8665fad2c5 in _starpu_clock_readtime (ts=0x7f84eaffc9c0) at 
common/timing.c:73
#3  _starpu_clock_gettime (ts=ts@entry=0x7f84eaffc9f0) at common/timing.c:101
#4  0x00007f8665fad3ad in starpu_timing_now () at common/timing.c:245
#5  0x00007f8665fd2bc4 in _starpu_pop_task 
(worker=worker@entry=0x7f86662965f8 <_starpu_config+24920>) at 
core/sched_policy.c:908
#6  0x00007f8665fe4152 in _starpu_get_worker_task 
(worker=worker@entry=0x7f86662965f8 <_starpu_config+24920>, 
workerid=workerid@entry=23, 
    memnode=memnode@entry=0) at drivers/driver_common/driver_common.c:411
#7  0x00007f866601a15f in _starpu_cpu_driver_run_once 
(cpu_worker=cpu_worker@entry=0x7f86662965f8 <_starpu_config+24920>)
    at drivers/cpu/driver_cpu.c:247
#8  0x00007f866601a95d in _starpu_cpu_worker (arg=0x7f86662965f8 
<_starpu_config+24920>) at drivers/cpu/driver_cpu.c:361
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f84ea7fc700 (LWP 88353)):
#0  0x00007f8664d6f96d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f8665fade7e in starpu_sleep (nb_sec=nb_sec@entry=1) at 
common/utils.c:401
#2  0x00007f8665fb31b4 in watchdog_func (arg=<optimized out>) at 
core/task.c:1203
#3  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f84e9ffb700 (LWP 88354)):
---Type <return> to continue, or q <return> to quit---
#0  get_cycles () at ../include/linux-x86_64/sysdep.h:43
#1  ips_ptl_poll (ptl=0x1ac2ce0, _ignored=0) at ptl.c:490
#2  0x00007f8659acd3e7 in __psmi_poll_internal (ep=0x1ac2088, 
poll_amsh=poll_amsh@entry=1) at psm.c:490
#3  0x00007f8659ad85f1 in ips_ptl_mq_rndv (req=req@entry=0x7f865a6a1338, 
mepaddr=mepaddr@entry=0x192cb8da8, ipsaddr=ipsaddr@entry=0x192cb8e88, 
    buf=buf@entry=0x22ac5f00, len=len@entry=17604160) at ips_proto_mq.c:366
#4  0x00007f8659ad9136 in ips_proto_mq_isend (mq=0x1a8b8f8, 
mepaddr=0x192cb8da8, flags=<optimized out>, tag=43, ubuf=0x22ac5f00, 
len=17604160, 
    context=0x7f84bc01c3b8, req_o=0x7f84bc01c3d0) at ips_proto_mq.c:542
#5  0x00007f8659acb82e in __psm_mq_isend (mq=<optimized out>, dest=<optimized 
out>, flags=<optimized out>, stag=<optimized out>, 
    buf=<optimized out>, len=<optimized out>, context=0x7f84bc01c3b8, 
req=0x7f84bc01c3d0) at psm_mq.c:386
#6  0x00007f8659cf3824 in ompi_mtl_psm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_mtl_psm.so
#7  0x00007f865a8ce1e4 in mca_pml_cm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_pml_cm.so
#8  0x00007f86690285d4 in PMPI_Isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/libmpi.so.20
#9  0x00007f86662d9139 in _starpu_mpi_isend_data_func 
(req=req@entry=0x1b4c4b450) at starpu_mpi.c:361
#10 0x00007f86662d94e7 in _starpu_mpi_isend_size_func (req=0x1b4c4b450) at 
starpu_mpi.c:448
#11 0x00007f86662dce06 in _starpu_mpi_progress_thread_func (arg=0x19130f5e0) 
at starpu_mpi.c:1380
#12 0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f8664a9628d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f86694c37c0 (LWP 88271)):
#0  0x00007f8664d6c6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007f8665fab82e in _starpu_barrier_counter_wait_for_empty_counter 
(barrier_c=barrier_c@entry=0x7f86662a7b20 <_starpu_config+95872>)
    at common/barrier_counter.c:44
#2  0x00007f8665fd509a in _starpu_wait_for_all_tasks_of_sched_ctx 
(sched_ctx_id=sched_ctx_id@entry=0) at core/sched_ctx.c:1203
#3  0x00007f8665fb2b85 in 
_starpu_task_wait_for_all_in_ctx_and_return_nb_waited_tasks 
(sched_ctx=sched_ctx@entry=0) at core/task.c:875
#4  0x00007f8665fb2c50 in 
_starpu_task_wait_for_all_and_return_nb_waited_tasks () at core/task.c:862
#5  0x00007f8665fb2cf9 in starpu_task_wait_for_all () at core/task.c:868
#6  0x000000000045733f in 
FGroupTaskStarPUImplicitAlgorithm<FGroupTree<double, FChebCellPOD<double, 6, 
1, 1, 1>, FBasicCellPOD, FChebCellPODPole<double, 6, 1, 1, 1>, 
FChebCellPODLocal<double, 6, 1, 1, 1>, FP2PGroupParticleContainer<double, 1, 
1, 1>, 1u, 4u, double>, FGroupOfCells<FChebCellPOD<double, 6, 1, 1, 1>, 
FBasicCellPOD, FChebCellPODPole<double, 6, 1, 1, 1>, 
FChebCellPODLocal<double, 6, 1, 1, 1> >, 
FStarPUAllCpuCapacities<FChebSymKernel<double, FChebCellPOD<double, 6, 1, 1, 
1>, FP2PGroupParticleContainer<double, 1, 1, 1>, 
FInterpMatrixKernelR<double>, 6, 1> >, FGroupOfParticles<double, 1u, 4u, 
double>, FStarPUCpuWrapper<FGroupOfCells<FChebCellPOD<double, 6, 1, 1, 1>, 
FBasicCellPOD, FChebCellPODPole<double, 6, 1, 1, 1>, 
FChebCellPODLocal<double, 6, 1, 1, 1> >, FChebCellPOD<double, 6, 1, 1, 1>, 
FStarPUAllCpuCapacities<FChebSymKernel<double, FChebCellPOD<double, 6, 1, 1, 
1>, FP2PGroupParticleContainer<double, 1, 1, 1>, 
FInterpMatrixKernelR<double>, 6, 1> >, FGroupOfParticles<double, 1u, 4u, 
double>, FP2PGroupParticleContainer<double, 1, 1, 1> > 
>::executeCore(unsigned int) ()
#7  0x0000000000416cde in main ()

[Switching to thread 2 (Thread 0x7f84e9ffb700 (LWP 88354))]
#0  get_cycles () at ../include/linux-x86_64/sysdep.h:43
43      ../include/linux-x86_64/sysdep.h: No such file or directory.
(gdb) bt
#0  get_cycles () at ../include/linux-x86_64/sysdep.h:43
#1  ips_ptl_poll (ptl=0x1ac2ce0, _ignored=0) at ptl.c:490
#2  0x00007f8659acd3e7 in __psmi_poll_internal (ep=0x1ac2088, 
poll_amsh=poll_amsh@entry=1) at psm.c:490
#3  0x00007f8659ad85f1 in ips_ptl_mq_rndv (req=req@entry=0x7f865a6a1338, 
mepaddr=mepaddr@entry=0x192cb8da8, ipsaddr=ipsaddr@entry=0x192cb8e88, 
    buf=buf@entry=0x22ac5f00, len=len@entry=17604160) at ips_proto_mq.c:366
#4  0x00007f8659ad9136 in ips_proto_mq_isend (mq=0x1a8b8f8, 
mepaddr=0x192cb8da8, flags=<optimized out>, tag=43, ubuf=0x22ac5f00, 
len=17604160, 
    context=0x7f84bc01c3b8, req_o=0x7f84bc01c3d0) at ips_proto_mq.c:542
#5  0x00007f8659acb82e in __psm_mq_isend (mq=<optimized out>, dest=<optimized 
out>, flags=<optimized out>, stag=<optimized out>, 
    buf=<optimized out>, len=<optimized out>, context=0x7f84bc01c3b8, 
req=0x7f84bc01c3d0) at psm_mq.c:386
#6  0x00007f8659cf3824 in ompi_mtl_psm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_mtl_psm.so
#7  0x00007f865a8ce1e4 in mca_pml_cm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_pml_cm.so
#8  0x00007f86690285d4 in PMPI_Isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/libmpi.so.20
#9  0x00007f86662d9139 in _starpu_mpi_isend_data_func 
(req=req@entry=0x1b4c4b450) at starpu_mpi.c:361
#10 0x00007f86662d94e7 in _starpu_mpi_isend_size_func (req=0x1b4c4b450) at 
starpu_mpi.c:448
#11 0x00007f86662dce06 in _starpu_mpi_progress_thread_func (arg=0x19130f5e0) 
at starpu_mpi.c:1380
#12 0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f8664a9628d in clone () from /lib64/libc.so.6
(gdb) finish 
Run till exit from #0  get_cycles () at ../include/linux-x86_64/sysdep.h:43
496     ptl.c: No such file or directory.
(gdb) bt
#0  ips_ptl_poll (ptl=0x1ac2ce0, _ignored=0) at ptl.c:496
#1  0x00007f8659acd3e7 in __psmi_poll_internal (ep=0x1ac2088, 
poll_amsh=poll_amsh@entry=1) at psm.c:490
#2  0x00007f8659ad85f1 in ips_ptl_mq_rndv (req=req@entry=0x7f865a6a1338, 
mepaddr=mepaddr@entry=0x192cb8da8, ipsaddr=ipsaddr@entry=0x192cb8e88, 
    buf=buf@entry=0x22ac5f00, len=len@entry=17604160) at ips_proto_mq.c:366
#3  0x00007f8659ad9136 in ips_proto_mq_isend (mq=0x1a8b8f8, 
mepaddr=0x192cb8da8, flags=<optimized out>, tag=43, ubuf=0x22ac5f00, 
len=17604160, 
    context=0x7f84bc01c3b8, req_o=0x7f84bc01c3d0) at ips_proto_mq.c:542
#4  0x00007f8659acb82e in __psm_mq_isend (mq=<optimized out>, dest=<optimized 
out>, flags=<optimized out>, stag=<optimized out>, 
    buf=<optimized out>, len=<optimized out>, context=0x7f84bc01c3b8, 
req=0x7f84bc01c3d0) at psm_mq.c:386
#5  0x00007f8659cf3824 in ompi_mtl_psm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_mtl_psm.so
#6  0x00007f865a8ce1e4 in mca_pml_cm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_pml_cm.so
#7  0x00007f86690285d4 in PMPI_Isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/libmpi.so.20
#8  0x00007f86662d9139 in _starpu_mpi_isend_data_func 
(req=req@entry=0x1b4c4b450) at starpu_mpi.c:361
#9  0x00007f86662d94e7 in _starpu_mpi_isend_size_func (req=0x1b4c4b450) at 
starpu_mpi.c:448
#10 0x00007f86662dce06 in _starpu_mpi_progress_thread_func (arg=0x19130f5e0) 
at starpu_mpi.c:1380
#11 0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f8664a9628d in clone () from /lib64/libc.so.6
(gdb) finish 
Run till exit from #0  ips_ptl_poll (ptl=0x1ac2ce0, _ignored=0) at ptl.c:496
__psmi_poll_internal (ep=0x1ac2088, poll_amsh=poll_amsh@entry=1) at psm.c:491
491     psm.c: No such file or directory.
Value returned is $1 = PSM_OK_NO_PROGRESS
(gdb) bt
#0  __psmi_poll_internal (ep=0x1ac2088, poll_amsh=poll_amsh@entry=1) at 
psm.c:491
#1  0x00007f8659ad85f1 in ips_ptl_mq_rndv (req=req@entry=0x7f865a6a1338, 
mepaddr=mepaddr@entry=0x192cb8da8, ipsaddr=ipsaddr@entry=0x192cb8e88, 
    buf=buf@entry=0x22ac5f00, len=len@entry=17604160) at ips_proto_mq.c:366
#2  0x00007f8659ad9136 in ips_proto_mq_isend (mq=0x1a8b8f8, 
mepaddr=0x192cb8da8, flags=<optimized out>, tag=43, ubuf=0x22ac5f00, 
len=17604160, 
    context=0x7f84bc01c3b8, req_o=0x7f84bc01c3d0) at ips_proto_mq.c:542
#3  0x00007f8659acb82e in __psm_mq_isend (mq=<optimized out>, dest=<optimized 
out>, flags=<optimized out>, stag=<optimized out>, 
    buf=<optimized out>, len=<optimized out>, context=0x7f84bc01c3b8, 
req=0x7f84bc01c3d0) at psm_mq.c:386
#4  0x00007f8659cf3824 in ompi_mtl_psm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_mtl_psm.so
#5  0x00007f865a8ce1e4 in mca_pml_cm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_pml_cm.so
#6  0x00007f86690285d4 in PMPI_Isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/libmpi.so.20
#7  0x00007f86662d9139 in _starpu_mpi_isend_data_func 
(req=req@entry=0x1b4c4b450) at starpu_mpi.c:361
#8  0x00007f86662d94e7 in _starpu_mpi_isend_size_func (req=0x1b4c4b450) at 
starpu_mpi.c:448
#9  0x00007f86662dce06 in _starpu_mpi_progress_thread_func (arg=0x19130f5e0) 
at starpu_mpi.c:1380
#10 0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f8664a9628d in clone () from /lib64/libc.so.6
(gdb) finish 
Run till exit from #0  __psmi_poll_internal (ep=0x1ac2088, 
poll_amsh=poll_amsh@entry=1) at psm.c:491
ips_ptl_mq_rndv (req=req@entry=0x7f865a6a1338, 
mepaddr=mepaddr@entry=0x192cb8da8, ipsaddr=ipsaddr@entry=0x192cb8e88, 
buf=buf@entry=0x22ac5f00, 
    len=len@entry=17604160) at ips_proto_mq.c:365
365     ips_proto_mq.c: No such file or directory.
Value returned is $2 = PSM_OK_NO_PROGRESS
(gdb) bt
#0  ips_ptl_mq_rndv (req=req@entry=0x7f865a6a1338, 
mepaddr=mepaddr@entry=0x192cb8da8, ipsaddr=ipsaddr@entry=0x192cb8e88, 
    buf=buf@entry=0x22ac5f00, len=len@entry=17604160) at ips_proto_mq.c:365
#1  0x00007f8659ad9136 in ips_proto_mq_isend (mq=0x1a8b8f8, 
mepaddr=0x192cb8da8, flags=<optimized out>, tag=43, ubuf=0x22ac5f00, 
len=17604160, 
    context=0x7f84bc01c3b8, req_o=0x7f84bc01c3d0) at ips_proto_mq.c:542
#2  0x00007f8659acb82e in __psm_mq_isend (mq=<optimized out>, dest=<optimized 
out>, flags=<optimized out>, stag=<optimized out>, 
    buf=<optimized out>, len=<optimized out>, context=0x7f84bc01c3b8, 
req=0x7f84bc01c3d0) at psm_mq.c:386
#3  0x00007f8659cf3824 in ompi_mtl_psm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_mtl_psm.so
#4  0x00007f865a8ce1e4 in mca_pml_cm_isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/openmpi/mca_pml_cm.so
#5  0x00007f86690285d4 in PMPI_Isend () from 
/cm/shared/apps/openmpi/gcc/64/2.0.0rc3-nomxm-prefix-default/lib/libmpi.so.20
#6  0x00007f86662d9139 in _starpu_mpi_isend_data_func 
(req=req@entry=0x1b4c4b450) at starpu_mpi.c:361
#7  0x00007f86662d94e7 in _starpu_mpi_isend_size_func (req=0x1b4c4b450) at 
starpu_mpi.c:448
#8  0x00007f86662dce06 in _starpu_mpi_progress_thread_func (arg=0x19130f5e0) 
at starpu_mpi.c:1380
#9  0x00007f8664d68dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8664a9628d in clone () from /lib64/libc.so.6
(gdb) finish 
Run till exit from #0  ips_ptl_mq_rndv (req=req@entry=0x7f865a6a1338, 
mepaddr=mepaddr@entry=0x192cb8da8, ipsaddr=ipsaddr@entry=0x192cb8e88, 
    buf=buf@entry=0x22ac5f00, len=len@entry=17604160) at ips_proto_mq.c:365
^C
Program received signal SIGINT, Interrupt.
[Switching to Thread 0x7f86694c37c0 (LWP 88271)]
0x00007f8664d6c6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0