Re: [Starpu-devel] starpu_data_unregister_submit() and MPI

[Benoît Lizé <benoit.lize@gmail.com>]

> I am having trouble using it in my HMatrix code with r12389 (trunk). In this
> code, I submit all the tasks whils StarPU is "paused" (using starpu_pause()
> before starting to submit any task), and starpu_data_unregister_submit() hangs
> here:

Could you also provide the backtraces of other threads?

Here it is (with STARPU_NCPUS=1):

 

(gdb) info threads

  Id   Target Id         Frame 

  4    Thread 0x7fffe1c22700 (LWP 17395) "AS_ELFIP_FMM" 0x00007fffede6af28 in mca_btl_sm_component_progress ()

   from /Users/lize/local/cruisecontrol/WORKDIR/openmpi/install/intelLinuxShared_O/lib/libmpi.so.0

  2    Thread 0x7fffe2c24700 (LWP 17393) "AS_ELFIP_FMM" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

* 1    Thread 0x7ffff7fb6780 (LWP 17384) "AS_ELFIP_FMM" pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:34

(gdb) bt

#0  pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:34

#1  0x00007ffff5fb64b9 in _starpu_spin_lock (lock=lock@entry=0x582d790) at common/starpu_spinlock.c:62

#2  0x00007ffff5fc3f40 in _starpu_attempt_to_submit_data_request (request_from_codelet=request_from_codelet@entry=0, handle=0x582d780, mode=STARPU_W, 

    callback=callback@entry=0x7ffff5fe4c70 <_starpu_data_acquire_continuation_non_blocking>, argcb=argcb@entry=0x5832f40, j=j@entry=0x0, 

    buffer_index=buffer_index@entry=0) at core/dependencies/data_concurrency.c:106

#3  0x00007ffff5fc40c0 in _starpu_attempt_to_submit_data_request_from_apps (handle=<optimized out>, mode=<optimized out>, 

    callback=callback@entry=0x7ffff5fe4c70 <_starpu_data_acquire_continuation_non_blocking>, argcb=argcb@entry=0x5832f40)

    at core/dependencies/data_concurrency.c:195

#4  0x00007ffff5fe4d29 in starpu_data_acquire_cb_pre_sync_callback (arg=0x5832f40) at datawizard/user_interactions.c:114

#5  0x00007ffff5fb793b in _starpu_handle_job_termination (j=j@entry=0x58333e0) at core/jobs.c:225

#6  0x00007ffff5fceee8 in _starpu_push_task (j=j@entry=0x58333e0) at core/sched_policy.c:372

#7  0x00007ffff5fb7fd3 in _starpu_enforce_deps_and_schedule (j=j@entry=0x58333e0) at core/jobs.c:416

#8  0x00007ffff5fb88e4 in _starpu_submit_job (j=j@entry=0x58333e0) at core/task.c:288

#9  0x00007ffff5fba129 in starpu_task_submit (task=0x5832fe0) at core/task.c:531

#10 0x00007ffff5fba3fc in _starpu_task_submit_internally (task=<optimized out>) at core/task.c:548

#11 0x00007ffff5fe4f75 in starpu_data_acquire_on_node_cb_sequential_consistency (handle=0x582d780, node=-1, mode=STARPU_W, callback=<optimized out>, 

    arg=<optimized out>, sequential_consistency=1) at datawizard/user_interactions.c:170

#12 0x00007ffff5d95132 in starpu_mpi_cache_flush (comm=0x582d790, data_handle=0x582d780) at starpu_mpi_cache.c:201

#13 0x00007ffff5fe8830 in _starpu_data_unregister (handle=0x582d780, coherent=<optimized out>) at datawizard/interfaces/data_interface.c:723

#14 0x00007ffff67f4ac5 in HMatrixHandle<std::complex<float> >::submitDelete (this=0x582d740) at /Users/lize/aseris_dev/mpf/hmat/src/starpu_operations.cpp:444

(gdb) t 2

[Switching to thread 2 (Thread 0x7fffe2c24700 (LWP 17393))]

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

185     ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.

(gdb) bt

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

#1  0x00007ffff5fbd33e in _starpu_machine_is_running () at core/workers.c:1194

#2  0x00007ffff5ffc56a in _starpu_cpu_worker (arg=0x7ffff621d480 <config+1184>) at drivers/cpu/driver_cpu.c:302

#3  0x00007ffff4152062 in start_thread (arg=0x7fffe2c24700) at pthread_create.c:312

#4  0x00007ffff3e86a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) t 4

[Switching to thread 4 (Thread 0x7fffe1c22700 (LWP 17395))]

#0  0x00007fffede6af28 in mca_btl_sm_component_progress () from /Users/lize/local/cruisecontrol/WORKDIR/openmpi/install/intelLinuxShared_O/lib/libmpi.so.0

(gdb) bt

#0  0x00007fffede6af28 in mca_btl_sm_component_progress () from /Users/lize/local/cruisecontrol/WORKDIR/openmpi/install/intelLinuxShared_O/lib/libmpi.so.0

#1  0x00007fffed887acb in opal_progress () from /Users/lize/local/cruisecontrol/WORKDIR/openmpi/install/intelLinuxShared_O/lib/libopen-pal.so.0

#2  0x00007fffeddf7e5e in ompi_request_default_test () from /Users/lize/local/cruisecontrol/WORKDIR/openmpi/install/intelLinuxShared_O/lib/libmpi.so.0

#3  0x00007fffede27bc3 in PMPI_Test () from /Users/lize/local/cruisecontrol/WORKDIR/openmpi/install/intelLinuxShared_O/lib/libmpi.so.0

#4  0x00007ffff5d8ca42 in _starpu_mpi_progress_thread_func (arg=0x0) at starpu_mpi.c:1337

#5  0x00007ffff4152062 in start_thread (arg=0x7fffe1c22700) at pthread_create.c:312

#6  0x00007ffff3e86a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111