Re: [starpu-devel] Suspicious behavior of StarPU dmdar scheduler

[David <dstrelak@cnb.csic.es>]

Hi,


we are still experiencing some weird behavior with StarPU schedulers.

We would be interested in showing you our application and the issues 
we're encountering.

Could we meet online?

We can also provide you with access to our test machine.


We are looking forward to your answer.


Kind regards,


David Strelak


On 23/2/22 10:01, David wrote:
> Hi,
>
>
> We have tried to change the beta (2/100/1000), as well as wrapping the 
> task submissions loops with the starpu_pause/resume functions.
>
> However, we have seen little to no change.
>
>
> We have further analyzed our program using the task graph and tasks 
> details logs, as well as Nsight Systems and we did not find anything 
> obviously wrong.
>
> With other configurations (namely single GPU worker, GTX 1070, no CPU 
> worker), we have noticed another strange behavior - it seems like 
> StarPU is refusing to release memory from already finished tasks, 
> which results in memory trashing later on, and in some cases even 
> deadlock on the memory allocation. It is similar to the oneGPUAllCPU 
> case we have shown you previously.
>
>
> In case it's related, we're using StarPU 1.3.7, however, when we tried 
> StarPU master branch (from today), our program crashed with the 
> following message:
>
> ../../src/datawizard/coherency.c:326: _starpu_determine_request_path: 
> Assertion `src_node >= 0 && dst_node >= 0' failed.
>
>
> In any case, we wonder if you would be willing to spend some time on a 
> video call with us, and we would show you our findings.
>
> If yes, please propose a few one to two hours time slots.
>
>
> KR,
>
>
> David Strelak
>
>
>
> On 21/2/22 14:25, Samuel Thibault wrote:
> > Hello,
> >
> > David, le mer. 09 févr. 2022 18:57:33 +0100, a ecrit:
> > > The program is PCI-e bound, i.e. the data transfer time is one of the
> > > crucial parameters which has to be taken into account by the scheduler.
> > Indeed, it seems that most of the time is just spent on transferring
> > data. See the attached snap of the noGPUallCPU.trace file, the GPU is
> > almost always in the purple FetchingInput state, so it's just waiting
> > for data to come.
> >
> > > 1. use case: single GPU, no CPU (oneGPUnoCPU):
> > >
> > > execution time: 7.2s
> > >
> > > 2. use case: use both GPUs, no CPU (twoGPUnoCPU):
> > >
> > > execution time: 7.6s
> > I am not really surprised. Depending on the details of the actual
> > motherboard, data transfers may not be achievable in parallel at all,
> > and then no gain is to be expected :/
> >
> > > Our questions are:
> > >
> > > 1. Do you think that the dmdar scheduler is not able to correctly 
> > > take into
> > > account memory transfers, and thus under-utilizes GPU 2 in case 2 
> > > and case
> > > 3, and causes prolonged computation in case 5?
> > dmdar guesstimates the amount of time that data transfers will take, the
> > actual time depends a lot on the CUDA state, transfers interferences,
> > etc. so it's really only a heuristic.  Here the transfers are really
> > the key problem, so I'm not surprised that dmdar is the best performing
> > scheduler, but I'm not that surprised either that it does not manage to
> > find a better way of scheduling tasks.
> >
> > You can try to tune the dmdar behavior with the STARPU_SCHED_BETA
> > variable, possibly that would help.
> >
> > > 2. How can we verify that it is the scheduler to be blamed, and not 
> > > some
> > > problem in our code?
> > On 1gpu the vite trace shows that the gpu is kept busy transferring
> > data, so it's not really the way to submit tasks which is the problem,
> > but the sheer amount of data to transfer on the PCI bus. For such kind
> > of application you will really want an NVlink between your main memory
> > and the GPU. That means moving to the powerpc architecture since x86
> > doesn't allow such RAM/GPU NVlink.
> >
> > Now on the 2gpu trace we'd expect starpu to be trying to execute
> > FFTStarPU in parallel on the two gpus and that's not the case, cuda0
> > is left idle (red). That is not expected. What I notice is that the
> > number of submitted tasks (the white line at the top of the trace) is
> > still increasing during the iteration, i.e. not all tasks are actually
> > submitted at this point, and indeed in the UserThread14597 bar at the
> > bottom there are "Submitting task" states showing up (one has to zoom in
> > to see them). So possibly StarPU just doesn't know yet that there are
> > other FFTStarPU tasks to do, since they're not submitted yet.
> >
> > So probably there is a problem in your code, something that seems
> > to be synchronizing with starpu, or otherwise taking a lot of time,
> > thus delaying the task submission. For a start you can try to use
> > starpu_pause/resume() around your task submission, to make sure
> > submissions are all done before starpu starts executing tasks. Possibly
> > your program will then hang because it is trying to synchronize with the
> > execution of tasks, but then you'll see where it gets stuck, which is
> > where it shouldn't actually be trying to synchronize.
> >
> > > 3. Does StarPU try to transfer data between multiple GPUs in parallel?
> > Yes, it tries to transfer everything it can in parallel.
> >
> > Samuel