Re: [Starpu-devel] Performance with SOCL on multiple devices

[Malcolm Roberts <malcolm.i.w.roberts@gmail.com>]

Hello, Samuel.

Thanks for taking a look at the code!  How were you able to detect that 
multiple GPUs were being used?

I agree that the macrocell interface code needs to be optimised and run 
in parallel.  I'm currently working on that, as well as reducing the 
amount of memory that is transferred.

However, I still find it strange that SOCL is unable to parallelize the 
simulation.  In particular, using SOCL (with either one or several GPUs) 
is twice as slow as compared to using just one GPU with OpenCL directly.

The test case that I'm considering is

./schnaps -G ../geo/sphereincube.msh -n 3

(NB: one must construct the .msh from the .geo using gmsh before running 
this).  This is a geometry which gives about 3000 different buffers 
arranged in a 3D geometry.  The domain could be divided into different 
domains which would minimize the number of transfers between GPUs.  We 
can do this in pre-processing using gmsh, for example.  With SOCL or 
StarPU, StarPU would need to divide the data itself, which might either 
take a large number of iterations or some pre-processing which would be 
very specific to our problem.

For example, with a 1D geometry, using one GPU (A), we could imagine 
having the macrocells all in a line  as in

AAAAAAAA

Using two GPUs (A and B) the optimal parallelization would be

AAAABBBB

which would mean that just one cell needs to be transffered in each 
direction.  The worst-case scenario would be

ABABABAB

which would require a total transfer of all the data.  Similar cases 
exist for 2D and 3D grids.

Do you think that StarPU (either directly or via SOCL) will be able to 
get close to AAAABBBB?

Thanks again and have a good weekend,

~Malcolm

On 06/01/2016 17:16, Samuel Thibault wrote:
> Samuel Thibault, on Wed 06 Jan 2016 17:03:42 +0100, wrote:
> > With these changes, I do see kernels running on various GPUs. It does
> > not seem faster, but that is probably due to data transfers.
> And also serialized kernels
>
> > You will probably want to use FxT to dump traces and read them with
> > Vite.
> It notably seems that the DGMacroCellInterface kernels are completely
> serialized, thus awfully bad performance due to no parallelism and
> data transfers :) DGFlux does get parallelized, on the other hand. The
> duration is however quite tiny (60µs), so it's really not efficient
> compared with the runtime overhead.
>
> Samuel

--
http://malcolmiwroberts.com