Re: [Starpu-devel] StarPU feature suggestion on a collective MPI operation

[Samuel Thibault <samuel.thibault@inria.fr>]

Hello,

Henrik Henriksson, le mar. 02 avril 2019 13:15:03 +0000, a ecrit:
> To formulate it differently, I want to copy the contents of multiple
> StarPU data handles into a single new continuous vector on all ranks. I
> understand that this is very similar to the idea of the data
> partitioning feature, but I haven't seen any mention of MPI together
> with data partitioning in the manual, so I assume that this isn't
> supported. If that assumption is wrong, I'd be very happy.

It is not integrated in the MPI interface, because I don't think it
needs to be :)

> void starpu_mpi_allgather_copy(starpu_data_handle_t *source_data_handles, 
> int handle_count, starpu_data_handle_t destination_handle, MPI_Comm comm);
> void starpu_mpi_gather_copy(starpu_data_handle_t *source_data_handles, int 
> handle_count, starpu_data_handle_t destination_handle, int root, MPI_Comm 
> comm);

But then the question is how the data should be combined, which is
exactly what data partitioning is about :)

What you can do is on the receiving node to register the big vector,
partition it into pieces, run starpu_mpi_recv_submit() for each
piece (and the send on the sending nodes), and unpartition the big
vector. Then you have your whole vector.  This can even be done
completely asynchronously with the partitioning planning.

Probably it'd be useful if you could contribute an example that we would
include in StarPU :)

Samuel