Re: [Starpu-devel] Performance decline from StarPU 1.2.3 to 1.2.4

[Mirko Myllykoski <mirkom@cs.umu.se>]

On 2018-10-01 15:59, Samuel Thibault wrote:
> Mirko Myllykoski, le jeu. 27 sept. 2018 20:21:14 +0200, a ecrit:
> > On 2018-09-27 18:32, Samuel Thibault wrote:
> > > I'm afraid the issue you are getting is that main memory registration
> > > for CUDA DMA transfers is quite costly, and actually not worth the
> > > cost for the asynchronism benefit. Setting --disable-cuda-memcpy-peer
> > > by hand allows to disable asynchronism optimization and thus the data
> > > registration. Ideally registration would be cheap, or cached, to avoid
> > > the issue. If your temporary data pieces are small (less than 4MB), you
> > > could try the attached patch which will make StarPU use a suballocator
> > > not only for the CUDA memory but also for the main memory, which we
> > > didn't feel would be useful, but perhaps would be.
> >
> > Compiling StarPU 1.2.4 with this patch (without 
> > --disable-cuda-memcpy-peer)
> > fixes the problem.
>
> Ok, that's the best option: you keep memcpy-peer which avoids spurious
> synchronization, but avoid paying the corresponding cost all the
> time. I have commited a more elaborated version of the the patch to all
> branches.
>
> Thanks!
> Samuel

Hi,

I believe that the patch did not fix the problem completely. I am now 
experiencing a similar problem with a different code. Again, the 
performance declines when I move from StarPU 1.2.3 to 1.2.4. Compiling 
StarPU with --disable-cuda-memcpy-peer fixes the problem but this time 
the patch seems to do nothing (I tried the one you sent me earlier and 
the commit 8c35781b0ac517304ac2f2461b243b4447c38ab3).

The code uses two scheduling contexts:
 A) Uses peager or pheft. In this case pheft is used and the context 
contains a single GPU (the context can also contain some CPU workers).
 B) Uses prio or dmdasd. In this case dmdasd is used and the context 
contains five CPU workers.

The code has three task types: process_panel, update_trail and update 
right. The first two are inserted to A and the last one is inserted B.

When compiled without --disable-cuda-memcpy-peer, the workers that 
belong to B report a lot more overhead. This does not happen when 
compiled with --disable-cuda-memcpy-peer. The code also runs slower when 
compiled without --disable-cuda-memcpy-peer.

Fxt trace with --disable-cuda-memcpy-peer: 
https://drive.google.com/open?id=1DuzEvPVSzB9lc1IjLlN44gjcw-mc7oN5

Fxt trace without --disable-cuda-memcpy-peer: 
https://drive.google.com/open?id=1oMTqXb4kYLQifbFSF2w5BU0Ivuz8kzHF

 - Mirko