Re: [Starpu-devel] Performance decline from StarPU 1.2.3 to 1.2.4

[Mirko Myllykoski <mirkom@cs.umu.se>]

On 2018-10-02 22:35, Mirko Myllykoski wrote:
> On 2018-10-02 22:05, Mirko Myllykoski wrote:
> > On 2018-10-01 15:59, Samuel Thibault wrote:
> > > Mirko Myllykoski, le jeu. 27 sept. 2018 20:21:14 +0200, a ecrit:
> > > > On 2018-09-27 18:32, Samuel Thibault wrote:
> > > > > I'm afraid the issue you are getting is that main memory registration
> > > > > for CUDA DMA transfers is quite costly, and actually not worth the
> > > > > cost for the asynchronism benefit. Setting --disable-cuda-memcpy-peer
> > > > > by hand allows to disable asynchronism optimization and thus the data
> > > > > registration. Ideally registration would be cheap, or cached, to avoid
> > > > > the issue. If your temporary data pieces are small (less than 4MB), you
> > > > > could try the attached patch which will make StarPU use a suballocator
> > > > > not only for the CUDA memory but also for the main memory, which we
> > > > > didn't feel would be useful, but perhaps would be.
> > > >
> > > > Compiling StarPU 1.2.4 with this patch (without 
> > > > --disable-cuda-memcpy-peer)
> > > > fixes the problem.
> > >
> > > Ok, that's the best option: you keep memcpy-peer which avoids 
> > > spurious
> > > synchronization, but avoid paying the corresponding cost all the
> > > time. I have commited a more elaborated version of the the patch to 
> > > all
> > > branches.
> > >
> > > Thanks!
> > > Samuel
> >
> > Hi,
> >
> > I believe that the patch did not fix the problem completely. I am now
> > experiencing a similar problem with a different code. Again, the
> > performance declines when I move from StarPU 1.2.3 to 1.2.4. Compiling
> > StarPU with --disable-cuda-memcpy-peer fixes the problem but this time
> > the patch seems to do nothing (I tried the one you sent me earlier and
> > the commit 8c35781b0ac517304ac2f2461b243b4447c38ab3).
> >
> > The code uses two scheduling contexts:
> >  A) Uses peager or pheft. In this case pheft is used and the context
> > contains a single GPU (the context can also contain some CPU workers).
> >  B) Uses prio or dmdasd. In this case dmdasd is used and the context
> > contains five CPU workers.
> >
> > The code has three task types: process_panel, update_trail and update
> > right. The first two are inserted to A and the last one is inserted B.
> >
> > When compiled without --disable-cuda-memcpy-peer, the workers that
> > belong to B report a lot more overhead. This does not happen when
> > compiled with --disable-cuda-memcpy-peer. The code also runs slower
> > when compiled without --disable-cuda-memcpy-peer.
> >
> > Fxt trace with --disable-cuda-memcpy-peer:
> > https://drive.google.com/open?id=1DuzEvPVSzB9lc1IjLlN44gjcw-mc7oN5
> >
> > Fxt trace without --disable-cuda-memcpy-peer:
> > https://drive.google.com/open?id=1oMTqXb4kYLQifbFSF2w5BU0Ivuz8kzHF
> >
> >  - Mirko
>
> Hi,
>
> The previous email contained an incorrect link. Here are the correct 
> ones:
>
> Fxt trace with --disable-cuda-memcpy-peer:
> https://drive.google.com/open?id=1DuzEvPVSzB9lc1IjLlN44gjcw-mc7oN5
>
> Fxt trace without --disable-cuda-memcpy-peer:
> https://drive.google.com/open?id=1fwakuRneziUVwJZ8saeW3riVigPPy6eJ
>
>  - Mirko

Hi,

After analyzing the traces more carefully, I noticed that "MEMMANAGER0" 
gets stuck to state "Freeing" for several hundreds of milliseconds. GDB 
backtraces indicate that many CPU workers threads spend a lot of time 
inside the cudaFreeHost function. My interpretation is that this gets 
recorded as "Freeing" by FxT.

It appears that each CPU worker manages to execute one update_right task 
and then something gets freed using the cudaFreeHost function. The only 
thing that might get freed at this point is a scratch buffer 
(STARPU_SCRATCH). If that is the case, why would StarPU free this 
scratch buffer using the cudaFreeHost function? It is not used in any 
GPU related computations.

 - Mirko