[Starpu-devel] StarPU on multiple communicator

[Xavier Lacoste <xavier.lacoste@inria.fr>]

Hello,

I am using PaStiX in Jorek (A Tokamak simulation code).

Jorek uses sub-communicators to factorize different matrices on different communicators (i.e. P0 and P1 factorize mat1 while P2 and P3 factorize mat2). The two matrices have exactly the same shape, so the same shape block columns (my data unit in starpu), identified with the same tags, but these data are in fact different as they belong to different matrices (mat1 and mat2).

As far as I understand this cannot work in StarPU today ( I saw Nathalie who agrees with me on that ;) ).

P0 and P2 (resp. P1 and P3) will register their data with exactly the same TAG. Thus we have two different data with the same TAG which could be a problem.

An other issue is that the rank of the data is set using the sub-communicator rank which may not be what is expected by StarPU.

All this results in StarPU hanging in a deadlock, certainly waiting for communications...

I hope I am not too confusing in my explanations.

I'll try a hack in my code to fix that: use global rank when registering data and retrieve the number of communicators (here my communicators have the same size hopefully) and add an offset to the TAGS.

I think supporting communicator in StarPU would be a great feature.

Regards,

XL.

----------------------------------------

Xavier Lacoste

xavier.lacoste@inria.fr

INRIA Bordeaux Sud-Ouest

200, avenue de la Vieille Tour

33405 Talence Cedex

Tél : +33 (0)5 24 57 40 69