Re: [Starpu-devel] Performance decline from StarPU 1.2.3 to 1.2.4

[Mirko Myllykoski <mirkom@cs.umu.se>]

Hi,

On 2018-09-27 18:32, Samuel Thibault wrote:

> Also you could try configuring with --disable-cuda-memcpy-peer.

Compiling StarPU 1.2.4 with --disable-cuda-memcpy-peer fixes the 
problem.

> What does your data look like? I guess it is different allocation 
> sizes?
> How do you allocate them, with starpu_malloc? Do you use temporary
> buffers?

The original data is two matrices that are stored continuously in the 
main memory. The matrices are partitioned into square tiles without 
copying. Long time ago, I experimented with the idea of copying the data 
so that each tile would be stored continuously in memory. I also tried 
to pin the tiles. This did not have any significant effect back then.

Some intermediate results get stored to temporary buffers (home node = 
-1) and certain task use scratch buffers (those tasks that have CUDA 
implementation in particular).

> I'm afraid the issue you are getting is that main memory registration
> for CUDA DMA transfers is quite costly, and actually not worth the
> cost for the asynchronism benefit. Setting --disable-cuda-memcpy-peer
> by hand allows to disable asynchronism optimization and thus the data
> registration. Ideally registration would be cheap, or cached, to avoid
> the issue. If your temporary data pieces are small (less than 4MB), you
> could try the attached patch which will make StarPU use a suballocator
> not only for the CUDA memory but also for the main memory, which we
> didn't feel would be useful, but perhaps would be.

Compiling StarPU 1.2.4 with this patch (without 
--disable-cuda-memcpy-peer) fixes the problem.

 - Mirko