Re: [Starpu-devel] Possible bug (scratch data + combined workers + version 1.3)

[Aleksandr Mikhalev <aleksandr.mikhalev@kaust.edu.sa>]

I just realised, that all my previous thoughts are wrong. Sorry for your 
time. I got hugely confused.

The truth is: scratch is never processed from one task to another. I 
experienced strange behaviour of STARPU_SCRATCH only because I glued two GPU 
codelets into one and marked buffer, that transferred data between them, as a 
scratch. This scratch buffer now holds temporary data for entire task. 
Assuming size of scratch buffer is known when data handle is registered, 
amount of workers in a combined worker is decided during runtime and StarPU 
cannot know how user wants to divide this scratch buffer between workers, I, 
personally, would prefer every worker to get pointer to the same scratch data 
and leave user with possible data race.

Regards,
Aleksandr.

> On 6 Sep 2018, at 14:33, Aleksandr Mikhalev 
> <aleksandr.mikhalev@kaust.edu.sa> wrote:
> 
> Indeed, per-worker or per-task allocation of a scratch is a nice question. 
> However, StarPU already have mechanism to obtain per-worker temporary 
> working space by calling proper codelet once for each worker. This method 
> of per-worker temporary space allocation is not suitable for situations, 
> when you need to generate data in one codelet and pass it to another. To 
> pass data to another codelet one has to use per-task temporary data. So, my 
> assumption is that STARPU_SCRATCH memory should be allocated on a per-task 
> basis no matter single or combined worker and user has to be careful with 
> simultaneous writes. Since presented test code works correctly with StarPU 
> 1.2, I assume someone who coded scratch data for combined workers used the 
> same logics.
> 
> In my application, I generate data in one codelet and pass it to another 
> codelet, spoiling it. And I do it in a parallel way, since second codelet 
> is optimised for batches of data. I mean second codelet does the same 
> operation on a differently generated data. Data itself is spoiled after 
> second codelet and is not used in any other computations, so, as I think, 
> STARPU_SCRATCH modifier fits best.
> 
> In a few words, assume you need to do SVD of many independent small 
> matrices on a hybrid CPU/GPU system/cluster. Since matrices are small, GPU 
> hardware is not fully utilised when doing SVD of matrices one after 
> another. One of possible solutions is to use batches of matrices instead of 
> single matrices. And amount of matrices in each batch should be not less, 
> than 200. Since corresponding GPU and CPU codelets must take the same 
> input, we come to a problem of doing the same batches on each CPU worker 
> separately. When there is only one batch of 200 matrices, only one worker 
> does the job and it does it sequentially. This is where combined workers of 
> StarPU look really promising. Since we have batch of input data, it is 
> intuitive to support batch of scratch data for each given task. Which means 
> STARPU_SCRATCH should be allocated on a per-task basis, no matter how many 
> workers are going to work on that.
> 
> Regards,
> Aleksandr.
> 
>> On 5 Sep 2018, at 17:09, Samuel Thibault <samuel.thibault@inria.fr> wrote:
>> 
>> Hello,
>> 
>> Aleksandr Mikhalev, le mer. 05 sept. 2018 12:23:15 +0300, a ecrit:
>>> I think I discovered strange behaviour of StarPU 1.3,
>> 
>> Mmm, actually I don't think anybody ever thought about scratch +
>> parallel tasks, so it's little wonder that it broke :)
>> 
>> I'm however wondering: in such case should we make all workers of the
>> combined worker use the same scratch area, or should it be per-worker ?
>> Mmm...
>> 
>> Samuel
> 


-- 

This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.