Re: [starpu-devel] [StarPU-MPI: no dedicated thread for communication; function to set MPI_THREAD_MULTIPLE]

[Maxim Abalenkov <maxim.abalenkov@gmail.com>]

Hello Philippe et al.,

How are you? Thank you for your reply.

Have also a look on all possible environment variables to configure StarPU execution, especially STARPU_MPI_NOBIND=1 (cf https://files.inria.fr/starpu/testing/master/doc/html/ExecutionConfigurationThroughEnvironmentVariables.html#mpiConf).

I think STARPU_MPI_NOBIND=1 should help in my case. I expect to have very little communication in my code. That’s why I would like to avoid reserving a core for MPI communication only. I will try that.

But, IMO, StarPU+MPI on Mac will give you poor performance anyway, so it doesn't really matter if you have +/- 1 core used for computations (for prototyping, for instance)…

Do you know, what is the reason for that? Why does StarPU-MPI perform badly on a Mac? 

I have my code in StarPU for shared-memory and StarPU-MPI. I would like to run a simple performance comparison test between these two versions. Can you please confirm my logic? To run StarPU-MPI on a single node using only this node’s cores, I set the number of StarPU CPUs equal to the number of MPI. For example to have four MPI processes each running on a dedicated CPU core I will run my code as

  STARPU_NCUDA=0 STARPU_NOPENCL=0 STARPU_NCPU=4 mpiexec -np 4 ./code

Is that right? Thank you and have a great day ahead!

—

Best wishes,

Maxim

Maxim Abalenkov \\ maxim.abalenkov@gmail.com
+44 7 486 486 505 \\ www.maxim.abalenkov.uk

On 11 Sep 2024, at 15:36, Philippe SWARTVAGHER <philippe.swartvagher@inria.fr> wrote:

Hello,

Le 11/09/2024 à 13:28, Maxim Abalenkov a écrit :

1) I would like StarPU-MPI to use all available threads for computation (i.e. avoid to have a dedicated thread for MPI communication). To achieve that I create a StarPU configuration structure and set the variable ‘reserve_ncpu = 0’.
    // ensure StarPU does not reserse CPUs solely for MPI communication
    struct starpu_conf conf = {.reserve_ncpus = 0};
    int status = starpu_conf_init(&conf);
    STARPU_CHECK_RETURN_VALUE(status, "starpu_conf_init");
    // initialise StarPU MPI
    status = starpu_mpi_init_conf(&argc, &argv, 1, MPI_COMM_WORLD, &conf);
    STARPU_CHECK_RETURN_VALUE(status, "starpu_mpi_init_conf");
    starpu_mpi_comm_rank(MPI_COMM_WORLD, &rank);
    starpu_mpi_comm_size(MPI_COMM_WORLD, &np);
However, when I print out timing results for each CPU worker I see only seven workers:
Worker CPU 0…
…
Worker CPU 6…
Am I creating the StarPU configuration incorrectly or is this (the hardware) reality of the Apple M2 CPU?

Hmm, looking at code, everything happens here : https://gitlab.inria.fr/starpu/starpu/-/blob/master/mpi/src/starpu_mpi_init.c?ref_type=heads#L366

So to avoid reserving a core for the MPI thread, you should either:
- set on which core / hyperthread the MPI thread will be bound (but it will fight with a worker thread, unless you use STARPU_WORKERS_CPUID or STARPU_WORKERS_COREID to tell StarPU not to bind a worker on this core)
- set STARPU_MPI_DRIVER_CALL_FREQUENCY to a strictly positive value, but this will disable the MPI thread, and communication work will be done only between compute tasks (is it something you want?)

Have also a look on all possible environment variables to configure StarPU execution, especially STARPU_MPI_NOBIND=1 (cf https://files.inria.fr/starpu/testing/master/doc/html/ExecutionConfigurationThroughEnvironmentVariables.html#mpiConf).

But, IMO, StarPU+MPI on Mac will give you poor performance anyway, so it doesn't really matter if you have +/- 1 core used for computations (for prototyping, for instance)...

2) Many StarPU-MPI examples use MPI_THREAD_MULTIPLE to allow multiple simultaneous calls to MPI library. In the examples, the value is set by calling ‘MPI_Init_thread()’. As I understand it is defined in various ‘helper.h’ files in the ‘tests’ directory. Is there a convenient StarPU API function to set ‘MPI_THREAD_MULTIPLE'? I tried including these helper header files. But I immediately face the errors of being unable to find “common/config.h”.

I guess common/config.h is just available inside the StarPU sources, not when you link with StarPU as an external library.

I think the best way to initialize MPI through StarPU is to call starpu_mpi_init{_conf}() with the argument initialize_mpi=1 to let StarPU call MPI_Init_thread() itself. Again, have a look on the documentation: https://files.inria.fr/starpu/testing/master/doc/html/MPISupport.html#MPIInitialization


--
Philippe SWARTVAGHER
Assistant Professor @ ENSEIRB-MATMECA
Research team Topal @ Inria Bordeaux-Sud Ouest