[Starpu-devel] Performance profiling of tile low rank cholesky (HiCMA) with StarPU on distributed memory systems

[Hatem Ltaief <hatem.ltaief@kaust.edu.sa>]

Dear StarPU Team,

We have performed some string scalability experiments of HiCMA on a Cray XC system (2 sockets 14-core Intel BDW).

We fixed basically the problem size and increase the number of nodes.

We set the number of threads per node to 27 to free the single remaining thread for StarPU to handle communication.

As shown in the attached pdf document, when we increase the number of nodes, we do not scale as expected.

The profiles indicate that most of the time is being spent into internal StarPU functions:

|| 42.6% | 79,366.5 | 851.5 | 1.1% | _starpu_get_worker_task

|| 2.2% | 4,111.3 | 114.7 | 2.8% | _starpu_handle_pending_node_data_requests

|| 1.9% | 3,523.9 | 145.1 | 4.1% | _starpu_cpu_worker

|| 1.9% | 3,507.2 | 130.8 | 3.7% | _starpu_cpu_driver_run_once

|| 1.9% | 3,469.6 | 128.4 | 3.7% | _starpu_may_pause

|| 1.7% | 3,180.7 | 220.3 | 6.7% | starpu_memchunk_tidy

|| 1.7% | 3,171.0 | 139.0 | 4.3% | _starpu_datawizard_progress

|| 1.1% | 2,113.8 | 94.2 | 4.4% | _starpu_handle_node_data_requests

We reduced then the number of worker threads per node to 16 and 8: we recover our parallel efficiency up to a decent level!

This is strong scaling, work may not be available, we would expect threads to stay idle and not consume resources. But apparently, from our experiments, they seem to do some internal StarPU-related work, which seem to prevent parallel performance.

Is this an expected behavior? Have you also observed something similar on your side? Any advices?

Thanks,

Hatem

---

This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete
 this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

Attachment: hicma-starpu-profiling.pdf
Description: hicma-starpu-profiling.pdf