Re: [Starpu-devel] Possible bug (scratch data + combined workers + version 1.3)

[Aleksandr Mikhalev <aleksandr.mikhalev@kaust.edu.sa>]

Indeed, per-worker or per-task allocation of a scratch is a nice question. 
However, StarPU already have mechanism to obtain per-worker temporary working 
space by calling proper codelet once for each worker. This method of 
per-worker temporary space allocation is not suitable for situations, when 
you need to generate data in one codelet and pass it to another. To pass data 
to another codelet one has to use per-task temporary data. So, my assumption 
is that STARPU_SCRATCH memory should be allocated on a per-task basis no 
matter single or combined worker and user has to be careful with simultaneous 
writes. Since presented test code works correctly with StarPU 1.2, I assume 
someone who coded scratch data for combined workers used the same logics.

In my application, I generate data in one codelet and pass it to another 
codelet, spoiling it. And I do it in a parallel way, since second codelet is 
optimised for batches of data. I mean second codelet does the same operation 
on a differently generated data. Data itself is spoiled after second codelet 
and is not used in any other computations, so, as I think, STARPU_SCRATCH 
modifier fits best.

In a few words, assume you need to do SVD of many independent small matrices 
on a hybrid CPU/GPU system/cluster. Since matrices are small, GPU hardware is 
not fully utilised when doing SVD of matrices one after another. One of 
possible solutions is to use batches of matrices instead of single matrices. 
And amount of matrices in each batch should be not less, than 200. Since 
corresponding GPU and CPU codelets must take the same input, we come to a 
problem of doing the same batches on each CPU worker separately. When there 
is only one batch of 200 matrices, only one worker does the job and it does 
it sequentially. This is where combined workers of StarPU look really 
promising. Since we have batch of input data, it is intuitive to support 
batch of scratch data for each given task. Which means STARPU_SCRATCH should 
be allocated on a per-task basis, no matter how many workers are going to 
work on that.

Regards,
Aleksandr.

> On 5 Sep 2018, at 17:09, Samuel Thibault <samuel.thibault@inria.fr> wrote:
> 
> Hello,
> 
> Aleksandr Mikhalev, le mer. 05 sept. 2018 12:23:15 +0300, a ecrit:
>> I think I discovered strange behaviour of StarPU 1.3,
> 
> Mmm, actually I don't think anybody ever thought about scratch +
> parallel tasks, so it's little wonder that it broke :)
> 
> I'm however wondering: in such case should we make all workers of the
> combined worker use the same scratch area, or should it be per-worker ?
> Mmm...
> 
> Samuel


-- 

This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.