[Starpu-devel] starpu_data_unregister_submit() and MPI

[Benoît Lizé <benoit.lize@gmail.com>]

Hello,

Is there any known issue between the MPI layer in StarPU and starpu_data_unregister_submit() ?

I am having trouble using it in my HMatrix code with r12389 (trunk). In this code, I submit all the tasks whils StarPU is "paused" (using starpu_pause() before starting to submit any task), and starpu_data_unregister_submit() hangs here:

#0  pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:33

#1  0x00007ffff5fb64b9 in _starpu_spin_lock (lock=lock@entry=0x59a7f10) at common/starpu_spinlock.c:62

#2  0x00007ffff5fc3f40 in _starpu_attempt_to_submit_data_request (request_from_codelet=request_from_codelet@entry=0, handle=0x59a7f00, mode=STARPU_W, 

    callback=callback@entry=0x7ffff5fe4c70 <_starpu_data_acquire_continuation_non_blocking>, argcb=argcb@entry=0x59ad8a0, j=j@entry=0x0, 

    buffer_index=buffer_index@entry=0) at core/dependencies/data_concurrency.c:106

#3  0x00007ffff5fc40c0 in _starpu_attempt_to_submit_data_request_from_apps (handle=<optimized out>, mode=<optimized out>, 

    callback=callback@entry=0x7ffff5fe4c70 <_starpu_data_acquire_continuation_non_blocking>, argcb=argcb@entry=0x59ad8a0)

    at core/dependencies/data_concurrency.c:195

#4  0x00007ffff5fe4d29 in starpu_data_acquire_cb_pre_sync_callback (arg=0x59ad8a0) at datawizard/user_interactions.c:114

#5  0x00007ffff5fb793b in _starpu_handle_job_termination (j=j@entry=0x59add40) at core/jobs.c:225

#6  0x00007ffff5fceee8 in _starpu_push_task (j=j@entry=0x59add40) at core/sched_policy.c:372

#7  0x00007ffff5fb7fd3 in _starpu_enforce_deps_and_schedule (j=j@entry=0x59add40) at core/jobs.c:416

#8  0x00007ffff5fb88e4 in _starpu_submit_job (j=j@entry=0x59add40) at core/task.c:288

#9  0x00007ffff5fba129 in starpu_task_submit (task=0x59ad940) at core/task.c:531

#10 0x00007ffff5fba3fc in _starpu_task_submit_internally (task=<optimized out>) at core/task.c:548

#11 0x00007ffff5fe4f75 in starpu_data_acquire_on_node_cb_sequential_consistency (handle=0x59a7f00, node=-1, mode=STARPU_W, callback=<optimized out>, 

    arg=<optimized out>, sequential_consistency=1) at datawizard/user_interactions.c:170

#12 0x00007ffff5d95132 in starpu_mpi_cache_flush (comm=0x59a7f10, data_handle=0x59a7f00) at starpu_mpi_cache.c:201

#13 0x00007ffff5fe8830 in _starpu_data_unregister (handle=0x59a7f00, coherent=<optimized out>) at datawizard/interfaces/data_interface.c:723

#14 0x00007ffff67f4a62 in HMatrixHandle<std::complex<float> >::submitDelete (this=0x59a7ec0) at /Users/lize/aseris_dev/mpf/hmat/src/starpu_operations.cpp:440

line 440 of my code (frame #14) is:

    starpu_data_unregister_submit(deps[i]->handle);

Is this behavior expected ?

If I do a starpu_resume() (or remove the starpu_pause()) before starting to submit the tasks, I have a crash (SIGSEGV) somewhere inside malloc(), which isn't very informative...

I'm calling starpu_data_unregister_submit() on all the nodes, even the ones that don't own the handle. Maybe this is not the way it should be done, but I couldn't find any reference to it in the documentation.

Is there something wrong in my use case ?

-- 

Benoit Lize