Re: [Starpu-devel] bad performance scaling behavior of cholesky decomposition

[Mario Tacconi <mtacco@caspur.it>]

> Hello,
>
> Sorry for the long delay, May is actually a busy month for us teachers
> :)
No problem, thank you for the reply :)
As you suggest below the cause of the problematic scaling behavior was a
bogus performance model.
By resetting the performance model I was able to obtain a pretty decent
scaling over a wide range of GPU+CPU configurations, as the graphs in
attach show :)
I was able to obtain a good performance model by using 2 GPU, 2 CPU
worker, STARPU_CALIBRATE=1 and the
"ws" scheduling policy.

> That could be odd, or maybe not. Our cholesky factorization is really
> not advanced, and suffers from lack of parallelism, so it is not
> necessarily surprising that it does not scale ; maybe you could rather
> give a try at Magma 1.1 or later revision, which is the state of the
> art for cholesky, and supports using StarPU for exploiting hybrid
> machines.
I know the magma library. Actually I discovered the StarPU project trying
to compile their dynamically-scheduled version of the LU and Cholesky
factorization.
I have to admit that I could not succeed in obtaining working code froma
that compilation: I think the Magma are using
an old version of StarPU along with the StarPU's deprecated interface.
The magma guys
tell me that that part of their library (which has basically no
documentation)  has to be considered as "experimental" so
I decided to wait for a more stable (and documented!) version of the
StarPU-based Magma.
In the meantime I start to study the StarPU system and I found the
project extremely interesting.
Actually, I'm using the cholesky benchmarks I attached to this mail in a
internal seminar on StarPU:
I hope I can convince my collegues (and my boss...) to use StarPU in our
future projects.

Mario


>  Only using GPUs might also be a sign that data transfer are
> too costly compared to letting a lot of computations happen on the
> CPUs. Actually on my tests on our latest machine with 9 cores and 3
> gpus, no computation is scheduled on CPUs.
>
>> Clearly, I'm doing something wrong, but I can't figure out what could
>> be. Any help/suggestion will be greatly appreciated.
> What you have shown looks right. Maybe you could check whether
> the performance models look fine, using starpu_perfmodel_display
> or starpu_perfmodel_plot. Also, check the bandwidth shown by
> starpu_machine_display.
>
> Samuel

Attachment: chol_gflops_vs_ncpus.pdf
Description: Adobe PDF document

Attachment: chol_2gpu_gflops_vs_ncpu.pdf
Description: Adobe PDF document