RE: [starpu-devel] Late data transfers

[XAVIER LACOSTE <xavier.lacoste@eviden.com>]

Hello,

Thanks for your answer,

I answer in the text bellow.

I'll need to double check that there are no synchronisation during submission but there shouldn't be maybe some error on our side.

Regards,

XL.

---

De : Samuel Thibault <samuel.thibault@inria.fr>
Envoyé : jeudi 22 août 2024 15:13
À : XAVIER LACOSTE <xavier.lacoste@eviden.com>
Cc : starpu-devel@inria.fr <starpu-devel@inria.fr>
Objet : Re: [starpu-devel] Late data transfers

 

Hello,

It would be useful to provide the paje trace, they contain much more
information than what is displayed by default.

I think I overwrote the traces, I could try to reproduce them.

I moved to another approach (allocate larger tiles, and show subpart to starpu while gathering them in macro-tasks)

XAVIER LACOSTE, le mer. 07 août 2024 13:23:57 +0000, a ecrit:
> One the attached trace of a LU factorization,

Which implementation is this? Chameleon?

It's on another implementation than Chameleon.

> GETRF (black) and TRSMs (blue)
> are executed on CPU  except for the first ones in row and column and GEMMs are
> all batched on the GPU (except the first one wich is on the GPU but not batch).
> Here are some remarks I can't explain on the trace:
>
>   • The TRSMs on host wait until the block is sent to GPU to start (while they
>     could start just after it).
>   • data transfers from host to device do not start immediatly after GETRF

I'm not sure what exactly you refer to. Do you refer to the red piece
just after the black GETRF? I don't know what that is but that is
surprising indeed. That being said, I see at the bottom a lot of tasks
submission right there, and the number of submitted tasks only grow
there, so it looks like you have some spurious synchronization in your
submission thread?

I will double check but I though there was no control on the submission flow and all should be submitted at the beginning.

>     nor after each TRSM.

I also seems there that you have task submissions showing up at the
bottom, so probably more spurious synchronization?

>   • data transfer from device to host, before next GETRF do not occurs
>     immediatly after the tiny GEMM on the GPU (just after the TRSMs on the GPU)

Again, this looks to me like spurious synchronization in the submission
loop.

> I use 2 streams on the CUDA device because there are small idle time between
> each kernel....

Is there really, how do you see it? StarPU already pipelines kernel
submission, by submitting kernel number n+1 while kernel n is running.

There are tiny red pieces between each kernel that can be a little bigger, and using 2 streams increases the performance.

Samuel