Re: [starpu-devel] Question about Designing Custom Interface for a Custom Struct

[Xinbo Li <lix34545@myumanitoba.ca>]

Hi Samuel, 

Thank you for your support over the last communications. Please let me summarize from the fundamental design for clarity. 

I have a custom struct named rkmatrix, containing some metadata and data buffer pointers:

struct rkmatrix

{

    int k;      

    int kt;

    int rows;    

    int cols;

    double* a;  

    double* a_imag;

    double* b;  

    double* b_imag;

};

I already have a serial application that uses this struct to perform some calculations. I am aiming to refactor the code so that StarPU can help accelerate it. The ultimate goal is to have a StarPU-MPI application, but as a first phase, I am trying to adapt the serial code so that each calculation is encapsulated as a StarPU task, and the serial execution of the tasks produces the same result as the original code.

For the convenience of StarPU serial implementation, I would like to use a custom data interface to encapsulate the rkmatrix struct. I am debating whether to contain a replicate of rkmatrix in the data interface, i.e.,

struct starpu_rkmatrix_interface

{

    int k;      

    int kt;

    int rows;    

    int cols;

    double* a;  

    double* a_imag;

    double* b;  

    double* b_imag;

};

or to contain a pointer to rkmatrix in the data interface, i.e., 

struct starpu_rkmatrix_interface

{

    prkmatrix rkmat;

};

Currently I am using the latter, because I would need pointers to rkmatrix (type prkmatrix) in my task kernels, and with the latter data interface, I can easily implement the get_pointer method as  

prkmatrix starpu_rkmatrix_get_ptr(starpu_data_handle_t handle)

{

    struct starpu_rkmatrix_interface *rkmatrix_interface =

        (struct starpu_rkmatrix_interface *) starpu_data_get_interface_on_node(handle, STARPU_MAIN_RAM);

    return rkmatrix_interface->rkmat;

}

However, the design of the latter data interface seems to contradict the "Pointers inside the data interface" section https://files.inria.fr/starpu/doc/html/AdvancedDataManagement.html#DefiningANewDataInterface_pointers, because prkmatrix is neither purely a pointer to the metadata nor a pointer to data buffer. It points to some metadata: prkmatrix->rows, prkmatrix->cols, prkmatrix->k, prkmatrix->kt and pointers to data buffers: prkmatrix->a, prkmatrix->a_imag, prkmatrix->b, prkmatrix->b_imag. 

My question is, does this conflict mean that I cannot define my data interface the second way and I have to replicate rkmatrix members in my data interface? Do you have any recommendations?

Thank you,

Xinbo

 

---

From: Samuel Thibault
Sent: Tuesday, April 22, 2025 12:41 PM
To: Xinbo Li
Subject: Re: Question about Properly Unregister Data for a Custom Interface

Caution! This message was sent from outside the University of Manitoba.


Showing your code bit by bit is really not a way for me to understand
what you are doing.

Xinbo Li, le mar. 22 avril 2025 19:27:21 +0000, a ecrit:
> I do have an allocate_data_on_node method implemented as
> static starpu_ssize_t rkmatrix_allocate_data_on_node(void *data_interface,
> unsigned node)
> {

>     rkmat_new = (prkmatrix) starpu_malloc_on_node(node, rk_struct_size); // a
> new rkmatrix pointer
>     if (!rkmat_new)
>         goto failure;
>
>     *rkmat_new = *rkmatrix_interface->rkmat; // copy the meta data (data
> pointers will also be
>     // copied but will be reset shortly)
>
>     rkmatrix_interface->rkmat = rkmat_new;

So this is allocated in rkmatrix_allocate_data_on_node, and
not in rkmatric_data_register, so it should be released in
rkmatrix_released_data_on_node, and not in rkmatric_data_unregister.

>     /* Allocate memory for the data members in rkmat */
>     requested_memory_a = rkmatrix_interface->rkmat->rows * rkmatrix_interface->
> rkmat->k * sizeof(rkmatrix_interface->rkmat->a[0]);
>     requested_memory_b = rkmatrix_interface->rkmat->cols * rkmatrix_interface->
> rkmat->k * sizeof(rkmatrix_interface->rkmat->b[0]);
>
>     a_real = (double*) starpu_malloc_on_node(node, requested_memory_a);
>     if (!a_real)
>         goto failure;
>     a_imag = (double*) starpu_malloc_on_node(node, requested_memory_a);
>     if (!a_imag)
>         goto failure;
>     b_real = (double*) starpu_malloc_on_node(node, requested_memory_b);
>     if (!b_real)
>         goto failure;
>     b_imag = (double*) starpu_malloc_on_node(node, requested_memory_b);
>     if (!b_imag)
>         goto failure;
>
>     /* update the data properly in consequence */
>     rkmatrix_interface->rkmat->a = a_real;
>     rkmatrix_interface->rkmat->a_imag = a_imag;
>     rkmatrix_interface->rkmat->b = b_real;
>     rkmatrix_interface->rkmat->b_imag = b_imag;
>
>     return 2*(requested_memory_a+requested_memory_b) + rk_struct_size;
>
> failure:
>     return -ENOMEM;
> }
>
> Would this method be called on the nodes who do not have the data after the
> registration somehow?

StarPU calls it when it needs to have the data allocated somewhere, and
calls release when it's not useful any more, or on registration, before
calling the unregister_data_handle.

> Also, the reason I implemented an unregistration method and called it
> explicitly is because calling starpu_data_unregister will always seg fault for
> some reason,

Digging the reason is the right way to fix a bug. Doing random things
cannot fix bugs.

> so I thought it might be easier to have a explicit call to the
> unregistration method.

I don't understand what you mean. It's starpu_data_unregister
which shall call the unregister_data_handle method. Not calling
starpu_data_unregister means you'll not synchronize anything, leak all
starpu internal buffers etc.

Samuel