Re: [starpu-devel] [StarPU for stencil-based code: logic, starpu_shutdown error]

[Maxim Abalenkov <maxim.abalenkov@gmail.com>]

Dear all,

How are you? I hope all is well with you.  I understand that your resources are limited and as many of us you are working under pressure. But if possible, would you please find half an hour to look at my code and advise me on what am I doing wrong? Thank you for your help and have a great day ahead!

—

Best wishes,

Maxim

Maxim Abalenkov \\ maxim.abalenkov@gmail.com
+44 7 486 486 505 \\ www.maxim.abalenkov.uk

On 5 Nov 2024, at 17:22, Maxim Abalenkov <maxim.abalenkov@gmail.com> wrote:

Dear all,

I hope all is well with you. Let me please share the link to a GitHub repository with my code:

  https://github.com/IBM/ifm/tree/starpu

The wrapper routine Engine::profile_run calls the six phases of computation at each iteration of the time loop:

  https://github.com/IBM/ifm/blob/9da5c1918ca125f82a71756dd0ed31af6808600d/Core/engine.C#L155

1) Intercept phase:

  https://github.com/IBM/ifm/blob/9da5c1918ca125f82a71756dd0ed31af6808600d/Core/watershed.C#L318

2) Overland depth phase:

  https://github.com/IBM/ifm/blob/9da5c1918ca125f82a71756dd0ed31af6808600d/Core/watershed.C#L465

3) Infiltration phase:

  https://github.com/IBM/ifm/blob/9da5c1918ca125f82a71756dd0ed31af6808600d/Core/watershed.C#L647

4) Diffusive routing phase:

  https://github.com/IBM/ifm/blob/9da5c1918ca125f82a71756dd0ed31af6808600d/Core/watershed.C#L1052

5) Outlet + Storm phase  // currently sequential:

  https://github.com/IBM/ifm/blob/9da5c1918ca125f82a71756dd0ed31af6808600d/Core/watershed.C#L1809

If possible, can you please spend 15 minutes to review my StarPU code and provide insight on what I could do to improve parallel performance? Thank you very much for all your help so far! Have a wonderful evening ahead!

—

Best wishes,

Maxim

Maxim Abalenkov \\ maxim.abalenkov@gmail.com
+44 7 486 486 505 \\ www.maxim.abalenkov.uk

On 5 Nov 2024, at 12:33, Maxim Abalenkov <maxim.abalenkov@gmail.com> wrote:

Dear all,

Let me please describe my finite-difference code in more detail.

There are four computationally expensive phases of the code. Each of them is formulated as C function. These functions are called at every iteration of a time loop.

for t in 0..t_max {

    comp_intercept_starpu

    comp_overland_depth_starpu

    comp_infiltration_starpu

    comp_diffusive_routing_starpu

    comp_outlet_starpu

    comp_storm_starpu

}

The code uses multiple two-dimensional arrays with data values. These arrays (as common in C) are flattened into one dimension. A data array is an m x n  matrix, where m — no. of rows and n — no. of columns. When flattened an m * n matrix becomes a vector of length m * n. To divide the data into blocks I divide my arrays along the first (row) dimension only. The second (column) dimension remains constant and is equal to n. Please see an attached sketch for details on data arrangement in arrays.

The first three routines (intercept, overland_depth and infiltration) can benefit from applying StarPU’s block filter and submitting tasks on these blocks. This situation is depicted in the leftmost figure on the sketch. 

Diffusive routing routine is the most complex and needs to omit the last row and the last column from its calculation. To make it work I created my own data structure called a matrix block. I manually divide the data arrays into these matrix blocks, manually register them with StarPU and submit tasks. This access pattern is illustrated in the middle figure in the sketch.

Finally, there are two low effort routines: outlet and storm. Their data access pattern is shown in the rightmost figure on the sketch. It is probably not worthy of parallelisation. But my final goal is to enable distributed-memory parallelism via StarPU-MPI. It may be beneficial to register these data points with StarPU and compute all outlet and storm values on a single MPI rank. I hope this makes sense.

May I please ask a few more questions:

1) Is it normal to have multiple data views registered with StarPU (‘automatic’ blocks created with a StarPU vector filter), manual matrix blocks manually registered with StarPU and finally a handful of data points manually registered with StarPU? Will these multiple views conflict with each other and degrade the overall parallel performance?

2) Do I understand correctly that it is best for StarPU performance to create and register all of these data partitions before the main loop over time and unregistered after the main loop?

3) What else can I try to improve performance of my code? Currently, everything apart from ‘outlet’ and ‘storm’ routines is ported to StarPU. I’m using starpu_pause and starpu_resume before and after calling the ‘outlet’ and ‘storm’ routines.

I will share my code in a public GitHub repository and send you a link in a very near future. Thank you very much for your attention and help so far! Have a great day ahead!

—

Best wishes,

Maxim

<data_arrangement_access_patterns.jpg>

Maxim Abalenkov \\ maxim.abalenkov@gmail.com
+44 7 486 486 505 \\ www.maxim.abalenkov.uk

On 5 Nov 2024, at 09:46, Maxim Abalenkov <maxim.abalenkov@gmail.com> wrote:

Dear Samuel, Philippe et al.,

How are you? I hope all is well with you. I re-compiled StarPU and ran the ‘task_size_overhead.sh’ script. Please find its output attached. Does it mean that for my system task sizes of 512 elements and above will be beneficial? In my StarPU experiments I have been using tasks of size 488. However, the performance I see is far from optimal. Thank you for your help and have a great day ahead!

—

Best wishes,

Maxim

<tasks_size_overhead.pdf>

Maxim Abalenkov \\ maxim.abalenkov@gmail.com
+44 7 486 486 505 \\ www.maxim.abalenkov.uk

On 4 Nov 2024, at 14:03, Maxim Abalenkov <maxim.abalenkov@gmail.com> wrote:

Dear Samuel, Philippe et al.,

How are you? I hope all is well with you. Thank you very much for your suggestions. I tried:

a) using starpu_pause, starpu_resume around sequential part of the code

b) using starpu_data_acquire, starpu_data_release —//—

c) examining the task size overhead on my system with the ‘task_size_overhead.sh’ script

find -name tasks_size_overhead.sh

./tests/microbenchs/tasks_size_overhead.sh

./starpupy/benchmark/tasks_size_overhead.sh

./build/tests/microbenchs/tasks_size_overhead.sh

./build/starpupy/benchmark/tasks_size_overhead.sh

The a) and b) didn’t not bring any performance gain. I run my StarPU code with

  STARPU_NCUDA=0 STARPU_NOPENCL=0 STARPU_NCPU=8 STARPU_SCHED=dmda python ./app.py

Python script ‘app.py’ is a simple wrapper pre-processing the input data and launching the main C + StarPU executable.

I also fail to run the shell script c). It crashes because it is unable to find the executable ‘tasks_size_overhead’. I couldn’t find it either using a ‘find’ command. Do I need to configure and compile StarPU with some extra options to make the executable ‘tasks_size_overhead’?

I’m using StarPU 1.4.7 @ macOS Sonoma 14.7.1 with an M2 CPU.

I will share my source code in a public GitHub repository for you to see in the very near future. In the meantime I’m going to convert the final phase of my finite-difference code to StarPU and test performance again. Thank you for your help and have a wonderful day ahead!

—

Best wishes,

Maxim

Maxim Abalenkov \\ maxim.abalenkov@gmail.com
+44 7 486 486 505 \\ www.maxim.abalenkov.uk

On 11 Oct 2024, at 16:13, Philippe SWARTVAGHER <philippe.swartvagher@inria.fr> wrote:

Le 11/10/2024 à 14:43, Samuel Thibault a écrit :

No, that should not really be a problem. But how large your tasks
are? See the checklist item about task sizes:
https://files.inria.fr/starpu/doc/html/CheckListWhenPerformanceAreNotThere.html#CheckTaskSize
That being said, you will probably want to put starpu_pause() /
starpu_resume() around your not-yet-starpufied code, otherwise StarPU
monopolizes all CPU cores, which can thus degrade the performance of the
non-starpufied code.

If your non-StarPU code accesses data managed by StarPU, you should also call

starpu_data_acquire(data_handle, STARPU_R /* or STARPU_RW */)

before your non-StarPU code reads (or writes) the StarPU data. This makes sure the tasks which manipulate data_handle are finished. Then, to tell StarPU you are finished working with this memory and it can start other tasks using this handle, you have to call

starpu_data_release(data_handle)

Have a look for instance on this example: https://gitlab.inria.fr/starpu/starpu/-/blob/f3e318b326666c5b279680d5a57fc0468e8c1876/examples/filters/fvector_pick_variable.c

Or, instead, you can just call

starpu_task_wait_for_all()

but depending on what your application does, you may not need to wait for all tasks to finish to start reading/writing one piece of data.

Not sure if this will solve your problems, but it was missing from your description of how your program uses StarPU.

--
Philippe SWARTVAGHER
Assistant Professor @ ENSEIRB-MATMECA
Research team Topal @ Inria Bordeaux-Sud Ouest