Re: [starpu-devel] Suspicious behavior of StarPU dmdar scheduler

[Samuel Thibault <samuel.thibault@inria.fr>]

Hello,

David, le mer. 09 févr. 2022 18:57:33 +0100, a ecrit:
> The program is PCI-e bound, i.e. the data transfer time is one of the
> crucial parameters which has to be taken into account by the scheduler.

Indeed, it seems that most of the time is just spent on transferring
data. See the attached snap of the noGPUallCPU.trace file, the GPU is
almost always in the purple FetchingInput state, so it's just waiting
for data to come.

> 1. use case: single GPU, no CPU (oneGPUnoCPU):
> 
> execution time: 7.2s
> 
> 2. use case: use both GPUs, no CPU (twoGPUnoCPU):
> 
> execution time: 7.6s

I am not really surprised. Depending on the details of the actual
motherboard, data transfers may not be achievable in parallel at all,
and then no gain is to be expected :/

> Our questions are:
> 
> 1. Do you think that the dmdar scheduler is not able to correctly take into
> account memory transfers, and thus under-utilizes GPU 2 in case 2 and case
> 3, and causes prolonged computation in case 5?

dmdar guesstimates the amount of time that data transfers will take, the
actual time depends a lot on the CUDA state, transfers interferences,
etc. so it's really only a heuristic.  Here the transfers are really
the key problem, so I'm not surprised that dmdar is the best performing
scheduler, but I'm not that surprised either that it does not manage to
find a better way of scheduling tasks.

You can try to tune the dmdar behavior with the STARPU_SCHED_BETA
variable, possibly that would help.

> 2. How can we verify that it is the scheduler to be blamed, and not some
> problem in our code?

On 1gpu the vite trace shows that the gpu is kept busy transferring
data, so it's not really the way to submit tasks which is the problem,
but the sheer amount of data to transfer on the PCI bus. For such kind
of application you will really want an NVlink between your main memory
and the GPU. That means moving to the powerpc architecture since x86
doesn't allow such RAM/GPU NVlink.

Now on the 2gpu trace we'd expect starpu to be trying to execute
FFTStarPU in parallel on the two gpus and that's not the case, cuda0
is left idle (red). That is not expected. What I notice is that the
number of submitted tasks (the white line at the top of the trace) is
still increasing during the iteration, i.e. not all tasks are actually
submitted at this point, and indeed in the UserThread14597 bar at the
bottom there are "Submitting task" states showing up (one has to zoom in
to see them). So possibly StarPU just doesn't know yet that there are
other FFTStarPU tasks to do, since they're not submitted yet.

So probably there is a problem in your code, something that seems
to be synchronizing with starpu, or otherwise taking a lot of time,
thus delaying the task submission. For a start you can try to use
starpu_pause/resume() around your task submission, to make sure
submissions are all done before starpu starts executing tasks. Possibly
your program will then hang because it is trying to synchronize with the
execution of tasks, but then you'll see where it gets stuck, which is
where it shouldn't actually be trying to synchronize.

> 3. Does StarPU try to transfer data between multiple GPUs in parallel?

Yes, it tries to transfer everything it can in parallel.

Samuel

Attachment: trace.png
Description: PNG image