[Starpu-devel] A question regarding the MPI cache and starpu_mpi_cache_flush

[Mirko Myllykoski <mirkom@cs.umu.se>]

Hi,

I am developing a numerical software that deals with large dense 
matrices (size class 50 000 X 50 000). The matrices are divided into 
smaller blocks (maybe 128 X 128) which are then registered with StarPU. 
I am trying to reduce the overhead by making each MPI node register only 
those blocks that it actually needs. Some blocks are needed by multiple 
MPI node which is why I implemented a separate subsystem that will 
automatically register a "placeholder" data handle and call the 
starpu_mpi_data_register() subroutine when a MPI node that does not own 
the block requests a data handle to the block.

In a certain situation, I would like to flush a data handle from the MPI 
cache using the starpu_mpi_cache_flush() subroutine. However, the 
documentation states that the subroutine has to be called by all MPI 
nodes. If I understand everything correctly, this means that all MPI 
nodes should have a corresponding data handle (either a placeholder, a 
copy or the actual block).

Is this correct? Is there any way to make this work with my block 
management subsystem?

Related question: Is there a way to obtain a pointer to the data that is 
stored inside a temporary data handle (home_node -1)?

Best Regards,
Mirko Myllykoski