[Starpu-devel] Mixing starpu_task_insert() and starpu_mpi_task_insert()

[Mirko Myllykoski <mirkom@cs.umu.se>]

Hi,

Are there any dangers in mixing starpu_task_insert() and
starpu_mpi_task_insert() functions in a same code? I assumed that
this is ok as long as I make sure that a node that inserts a task
using the starpu_task_insert() function owns the related data
handles (or it at least has a valid local copy of the data).

I have developed a StarPU-based code that has a distributed memory
support through StarPU-MPI. I am mainly using the
starpu_mpi_task_insert() function, gather operations and the
starpu_mpi_get_data_on_all_nodes_detached() function.

I am also using the starpu_task_insert() function in some places.
As I mentioned above, I assumed that this is ok in certain
situations. However, I have recently encountered a problem where one
node is left with unused early data:

=============================== BEGIN ===============================

[starpu][_starpu_mpi_early_data_check_termination] Unexpected message 
with comm 94690090821856 source 2 tag 251
[starpu][_starpu_mpi_early_data_check_termination] Unexpected message 
with comm 94690090821856 source 2 tag 252
[starpu][_starpu_mpi_early_data_check_termination] Unexpected message 
with comm 94690090821856 source 2 tag 256
...
[starpu][_starpu_mpi_early_data_check_termination] Unexpected message 
with comm 94690090821856 source 1 tag 58
/home/mirkom/.starpu_install/starpu_1.2.4_debug/lib/libstarpumpi-1.2.so.3(_starpu_mpi_early_data_check_termination+0x11a)[0x7fe791e01fe4]
/home/mirkom/.starpu_install/starpu_1.2.4_debug/lib/libstarpumpi-1.2.so.3(+0x13d68)[0x7fe791de7d68]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fe791bbc6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fe78ea1488f]

[starpu][_starpu_mpi_early_data_check_termination][assert failure] 
Number of unexpected received messages left is not 0 (but 2), did you 
forget to post a receive corresponding to a send?

nlafet-test: starpu_mpi_early_data.c:52: 
_starpu_mpi_early_data_check_termination: Assertion 
`_starpu_mpi_early_data_handle_hashmap_count == 0' failed.

===============================  END  ===============================

Disabling or periodically flushing the MPI cache fixes the problem.
Other nodes finish without problems.

The problem seems to be somehow related to the starpu_task_insert()
function. My code has a special copy-codelet that is used to copy a
section of a (tiled/blocked) matrix. The related tasks are inserted
using the starpu_mpi_task_insert() function. The data is copied to
data handles that are owner by rank X. I insert the copy tasks in all
nodes. The node/rank X will then insert a set of tasks using the
starpu_task_insert() function. This process involves some additional
data handles that are local to node/rank X (starpu_mpi_data_register
function is not used). I then insert a set of copy tasks in all nodes
that copy the data back using the starpu_mpi_task_insert() function.
In the end, the node X is left with "unexpected messages".

In summary:
 Initial: Matrix A is distributed among all nodes
 Step 1. All nodes: "MPI insert" a set of copy task:
      B (rank X, RW) <- part of A (any rank, R)
 Step 2. Node X: Insert a set of compute tasks:
      part of B (RW) <- C_1 (R), ...
 Step 3. All nodes: "MPI insert" a set of copy task:
      part of A (any rank, RW) <- B (rank X, R)
 Outcome: Node X has "unexpected messages" that are part of A.

I do not understand how modifying B locally on node X could cause
the other nodes to send excess data to node X.

Best Regards,
Mirko Myllykoski