Re: [Starpu-devel] Problem with starpu_mpi_wait_for_all and starpu_mpi_barrier functions

["COUTEYEN, Jean-marie" <jean-marie.couteyen-carpaye@airbus.com>]

> Is this really true? The StarPU handbook states only the following:
> "Block the caller until all group members of the communicator comm have
> called it."

You are right, you should not rely on implementation but on documentation. :)

However, it is currently the case, at least in starpu-1.2.6. 
(starpu_mpi.c:879,starpu-1.2.6)
I quote the relevant comment : 
        /* First wait for *both* all tasks and MPI requests to finish, in case
         * some tasks generate MPI requests, MPI requests generate tasks, etc.
         */

That's why I think the bug come from scheduling contexts with 
starpu_mpi_barrier. Internally it uses "starpu_task_wait_for_all" 
(starpu_mpi.c:894) which waits only for tasks in the current scheduling 
context.

> However, it is a bit weird that the code I attached to my original email
> seems to reproduce the problem even though it does not use different
> scheduling contexts. The problem seems to be timing sensitive so the
> code does not always trigger it.

It is a race condition. The problem with "starpu_mpi_wait_for_all" is not 
related to scheduling contexts. 
(starpu_mpi.c:2051,starpu-1.2.6). The variables task and mpi may not have the 
same value in different processes, resulting in a different number of loops.

> I don't quite understand. Do you mean that I should set the scheduling
> context back to the default scheduling context and then call the
> starpu_mpi_wait_for_all() function?

No, not the "starpu_mpi_wait_fo_all" but the "starpu_mpi_barrier" function. 
Ideally starpu_mpi_wait_for_all should work but it is bogus. And according to 
the documentation you cannot rely on starpu_mpi_barrier to wait for all tasks.

Until there is a fix, if your issue with starpu_mpi_barrier implies 
scheduling contexts,"reset_default_scheduling_context (*) then 
starpu_mpi_barrier" should work and do what you need. 

(*) That function does not exist. Perhaps "starpu_sched_ctx_set_context(0)" 
does that, I didn't check.

________________________________________
De : Mirko Myllykoski <mirkom@cs.umu.se>
Envoyé : vendredi 9 novembre 2018 13:25
À : COUTEYEN, Jean-marie
Cc : Starpu Devel
Objet : Re: [Starpu-devel] Problem with starpu_mpi_wait_for_all and 
starpu_mpi_barrier functions

Hi,

On 2018-11-09 12:34, COUTEYEN, Jean-marie wrote:

> I used to have a problem with starpu_mpi_wait_for_all also. It was in
> the context of explicit communication. The fact is that this function
> may call _starpu_mpi_barrier a different number of times by processes
> since there is no guaranty that
> "_starpu_task_wait_for_all_and_return_nb_waited_tasks" will be
> non-zero on every processes. (I used to have one process for which the
> value returned was non-zero and this process then called
> starpu_mpi_barrier once more than the other processes, since it enters
> the "while" one more time).
> If you can attach with a debugguer, if it is the same problem as I
> used to have, you will have on process stuck in starpu_mpi_barrier
> while the others already passed the starpu_mpi_wait_for_all.

This seems to be situation. I also added a break point to the
MPI_Barrier() function and it appears that sometimes nodes call the
function different number of times.

> But you might not need this function :
> "starpu_mpi_barrier" is supposed to wait all tasks also before
> exiting. When I use it, I do not have any problems in my case.

Is this really true? The StarPU handbook states only the following:
"Block the caller until all group members of the communicator comm have
called it."

I have implemented a temporary fix that calls the starpu_mpi_barrier()
function before the starpu_mpi_wait_for_all() function.

> However... based on your previous mails, it seems you are using
> different submitting contexts? Moreover, did you change the current
> submitting context (using starpu_sched_ctx_set_context) ?
> "starpu_task_wait_for_all" only waits for task of the current
> submitting context. Since starpu_mpi_barrier rely on this function, it
> might be the origin of the deadlock.

Some of my codes use different scheduling context, some don't. I will
have to check this. Thanks for the tip.

However, it is a bit weird that the code I attached to my original email
seems to reproduce the problem even though it does not use different
scheduling contexts. The problem seems to be timing sensitive so the
code does not always trigger it.

> Without modifying StarPU, I don't think it is possible to reset the
> current submitting context to none? If it is possible, I would try to
> replace the "starpu_mpi_wait_for_all" in your code by a
> starpu_mpi_barrier preceded by the reset of the current submitting
> context.

I don't quite understand. Do you mean that I should set the scheduling
context back to the default scheduling context and then call the
starpu_mpi_wait_for_all() function?

Best Regards,
Mirko Myllykoski
The information in this e-mail is confidential. The contents may not be 
disclosed or used by anyone other than the addressee. Access to this e-mail 
by anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and 
delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of 
this e-mail as it has been sent over public networks. If you have any 
concerns over the content of this message or its Accuracy or Integrity, 
please contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus 
scanning software but you should take whatever measures you deem to be 
appropriate to ensure that this message and any attachments are virus free.