Re: [Starpu-devel] Problem with starpu_mpi_wait_for_all and starpu_mpi_barrier functions

[Mirko Myllykoski <mirkom@cs.umu.se>]

Hi,

On 2018-11-09 12:34, COUTEYEN, Jean-marie wrote:

> I used to have a problem with starpu_mpi_wait_for_all also. It was in
> the context of explicit communication. The fact is that this function
> may call _starpu_mpi_barrier a different number of times by processes
> since there is no guaranty that
> "_starpu_task_wait_for_all_and_return_nb_waited_tasks" will be
> non-zero on every processes. (I used to have one process for which the
> value returned was non-zero and this process then called
> starpu_mpi_barrier once more than the other processes, since it enters
> the "while" one more time).
> If you can attach with a debugguer, if it is the same problem as I
> used to have, you will have on process stuck in starpu_mpi_barrier
> while the others already passed the starpu_mpi_wait_for_all.

This seems to be situation. I also added a break point to the 
MPI_Barrier() function and it appears that sometimes nodes call the 
function different number of times.

> But you might not need this function :
> "starpu_mpi_barrier" is supposed to wait all tasks also before
> exiting. When I use it, I do not have any problems in my case.

Is this really true? The StarPU handbook states only the following: 
"Block the caller until all group members of the communicator comm have 
called it."

I have implemented a temporary fix that calls the starpu_mpi_barrier() 
function before the starpu_mpi_wait_for_all() function.

> However... based on your previous mails, it seems you are using
> different submitting contexts? Moreover, did you change the current
> submitting context (using starpu_sched_ctx_set_context) ?
> "starpu_task_wait_for_all" only waits for task of the current
> submitting context. Since starpu_mpi_barrier rely on this function, it
> might be the origin of the deadlock.

Some of my codes use different scheduling context, some don't. I will 
have to check this. Thanks for the tip.

However, it is a bit weird that the code I attached to my original email 
seems to reproduce the problem even though it does not use different 
scheduling contexts. The problem seems to be timing sensitive so the 
code does not always trigger it.

> Without modifying StarPU, I don't think it is possible to reset the
> current submitting context to none? If it is possible, I would try to
> replace the "starpu_mpi_wait_for_all" in your code by a
> starpu_mpi_barrier preceded by the reset of the current submitting
> context.

I don't quite understand. Do you mean that I should set the scheduling 
context back to the default scheduling context and then call the 
starpu_mpi_wait_for_all() function?

Best Regards,
Mirko Myllykoski