Re: [Starpu-devel] Running Cholesky_implicit on 2 gpu nodes

[Yizhou Qian <adncat@stanford.edu>]

Thanks! I tried using 4 nodes with 8 and 24 cores on each node respectively with a matrix of size 1000*20. However the result shows that using 24 cores on each node is slower than using 8 cores on each node. Is this normal or I have missed something when running the code? The exact command is:

STARPU_SCHED=dmdas mpirun ./starpu-1.3.2/mpi/examples/matrix_decomposition/mpi_cholesky_distributed -size $((1000*20)) -nblocks 20

The outputs are shown below:

Result with 4 nodes and 8 cores per node:

[starpu][compare_value_and_recalibrate] Current configuration does not match the bus performance model (CPUS: (stored) 24 != (current) 8), recalibrating...

[starpu][compare_value_and_recalibrate] Current configuration does not match the bus performance model (CPUS: (stored) 24 != (current) 8), recalibrating...

[starpu][compare_value_and_recalibrate] ... done

[starpu][compare_value_and_recalibrate] ... done

[starpu][check_bus_config_file] No performance model for the bus, calibrating...

[starpu][check_bus_config_file] ... done

[starpu][check_bus_config_file] No performance model for the bus, calibrating...

[starpu][check_bus_config_file] ... done

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_11 is not calibrated enough for cpu0_impl0 (Comb0) size 4000000 footprint a45f87dc (only 0 m$[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_22 is not calibrated enough for cpu0_impl0 (Comb0) size 12000000 footprint 67bed6a9 (only 0 $[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_11 is not calibrated enough for cpu0_impl0 (Comb0) size 4000000 footprint a45f87dc (only 9 m$[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_21 is not calibrated enough for cpu0_impl0 (Comb0) size 8000000 footprint d4b37584 (only 0 m$[starpu][_starpu_update_perfmodel_history] Too big deviation for model chol_model_21 on cpu0_impl0 (Comb0): 80135.622000 vs average 47935.153600, 5 such errors against $[starpu][_starpu_update_perfmodel_history] Too big deviation for model chol_model_11 on cpu0_impl0 (Comb0): 2473644.139000 vs average 5214670.073000, 1 such errors agai$[starpu][_starpu_update_perfmodel_history] Too big deviation for model chol_model_11 on cpu0_impl0 (Comb0): 4453137.907000 vs average 1652110.597000, 1 such errors agai$[starpu][_starpu_update_perfmodel_history] Too big deviation for model chol_model_11 on cpu0_impl0 (Comb0): 4959215.396000 vs average 1844665.254000, 1 such errors agai$[1572985659.293172] [sh-107-41:205341:0]          mpool.c:38   UCX  WARN  object 0x19c52c0 was not returned to mpool ucp_requests

Computation time (in ms): 31129.52

Synthetic GFlops : 85.66

Results with 4 nodes and 24 cores per node:

[starpu][compare_value_and_recalibrate] Current configuration does not match the bus performance model (CPUS: (stored) 8 != (current) 24), recalibrating...

[starpu][compare_value_and_recalibrate] Current configuration does not match the bus performance model (CPUS: (stored) 8 != (current) 24), recalibrating...

[starpu][compare_value_and_recalibrate] ... done

[starpu][compare_value_and_recalibrate] ... done

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

[starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level = MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a time.

size: 20000 - nblocks: 20 - dblocksx: 2 - dblocksy: 2

[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_11 is not calibrated enough for cpu0_impl0 (Comb0) size 4000000 footprint a45f87dc (only 1 m$[starpu][_starpu_update_perfmodel_history] Too big deviation for model chol_model_11 on cpu0_impl0 (Comb0): 2187839.665000 vs average 209929.500000, 1 such errors again$[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_22 is not calibrated enough for cpu0_impl0 (Comb0) size 12000000 footprint 67bed6a9 (only 0 $[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_22 is not calibrated enough for cpu0_impl0 (Comb0) size 12000000 footprint 67bed6a9 (only 0 $[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_11 is not calibrated enough for cpu0_impl0 (Comb0) size 4000000 footprint a45f87dc (only 0 m$[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_21 is not calibrated enough for cpu0_impl0 (Comb0) size 8000000 footprint d4b37584 (only 0 m$[starpu][_starpu_history_based_job_expected_perf] Warning: model chol_model_21 is not calibrated enough for cpu0_impl0 (Comb0) size 8000000 footprint d4b37584 (only 0 m$[1572987914.829063] [sh-107-45:226943:0]          mpool.c:38   UCX  WARN  object 0x1274b80 was not returned to mpool ucp_requests

Computation time (in ms): 41708.27

Synthetic GFlops : 63.94

Thanks again!

Best,

Yizhou

---

From: Samuel Thibault <samuel.thibault@inria.fr>
Sent: Sunday, October 27, 2019 8:40 AM
To: Yizhou Qian <adncat@stanford.edu>
Subject: Re: [Starpu-devel] Running Cholesky_implicit on 2 gpu nodes

 

Hello,

Just FYI, if you want detailed response, you should mail starpu-devel
so Nathalie can also answer, I don't necessarily have the time to delve
into details. Also, that would help Nathalie to write useful
documentation for all users.

Yizhou Qian, le sam. 26 oct. 2019 23:15:51 +0000, a ecrit:
> However, somehow the program is in serialized state
> [starpu][_starpu_mpi_print_thread_level_support] MPI_Init_thread level =
> MPI_THREAD_SERIALIZED; Multiple threads may make MPI calls, but only one at a
> time.

This only means the MPI calls are serialized. Please read about
MPI_THREAD_SERIALIZED, that's a limitation of your MPI implementation.
StarPU uses serialized communications anyway, but it means you can't do
MPI yourself in your application.

> and there was some kind of out of memory issue?

It seems so:

> slurmstepd: error: Detected 11 oom-kill event(s) in step 53469501.batch cgroup.
> Some of your processes may have been killed by the cgroup out-of-memory
> handler.

I don't see why that could happen, 57600 is not particularly large. Just
in case, you could try smaller values.

> size: 57600 - nblocks: 60 - dblocksx: 8 - dblocksy: 6

Are you really running on 48 nodes?

Are you perhaps telling MPI to run one process per core? Then it's no
wonder it fills the memory: that version of MPI cholesky allocates the
whole matrix on each node. To make each node only allocate its piece of
the matrix, use mpi_cholesky_distributed. That being said, you really
do not want to run one process per core, that would completely kill the
usefulness of StarPU. Have MPI run one process per machine, so StarPU
can handle intra-machine parallelism.

Samuel