Re: [Starpu-devel] Fwd: Strong interest in contributions and integrations

[Fangli Pi <hpcfapix@hlrs.de>]

Hi Andra,

Thanks very much for your reply. Although I cannot answer all the questions 
right now, they are all very valuable and we will take time to try to 
implement towards the direction.

Following what I can answer now:

1. I did feed energy in Joule back to the profiling_info->power_consumed. 
Because in StarPU handbook, it is written that "double 
starpu_task_expected_power() returns expected power consumption in J", and I 
thought that they need to be consistent. (Am I right?)

2. I didn't use directly the time (start_time, submit_time, etc.) in the 
profiling_info because these times are relative to the initialization time of 
StarPU, however I need the absolute date and time. If StarPU could provide 
this, it would be great.

3, 4, 5, 6. It is helpful to get these reviews and I will make changes 
accordingly in soon.

7. The calls to sleep is due to the delay of communication and the database. 
I will try to reduce these by increase the update rate of the database.

8. The monitoring framework we use is a light-weight and near-real time tool. 
There is one paper introducing it and one paragraph about overhead measuring 
is included. Please find the paper in the attachment. Since the monitoring 
framework is still under development, its updated overhead (for papi-based 
plugin) is reduced from 2.0%+ to 1.7%+ for an update rate of 100ms, and from 
18.1% to 14.2% for an update rate of 10ms.

9. It is true that the energy measurement cannot separate for different CPU 
cores, so that if multiple tasks are executed in parallel, the finally 
retrieved energy data is the energy consumed for all CPU and GPU. One 
possible solution could be that a  task which power data are profiled and 
power model is calibrating is considered to be in a training mode rather than 
an executing mode, and the tasks can only be executed asynchronously. Once 
the tasks are calibrated, it is possible to execute them in parallel.

10. This is an interesting point, which we haven't thought before. Maybe we 
can have a study of starpu threads first and see how hard it is to modify the 
monitoring framework according. It would also be nice if you could provide 
more info in this direction.

All in all, I will update the repo according to your reviews and I will let 
you know once it is ready.

Best regards,
Fangli

----- Original Message -----
From: "Andra Hugo" <andra.hugo@inria.fr>
To: "Fangli Pi" <hpcfapix@hlrs.de>
Cc: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>, "Dennis Hoppe" 
<dennis.hoppe@hlrs.de>, "Dmitry Khabi" <khabi@hlrs.de>, "Michael Gienger" 
<gienger@hlrs.de>
Sent: Monday, May 23, 2016 12:07:12 PM
Subject: Re: [Starpu-devel] Fwd: Strong interest in contributions and   
integrations

Hi Fangli,

I looked into the code you provided and I have a few suggestions/questions. I 
will separate them in 2 parts. Let me know if it makes sense and if I can 
help with anything.

A. Code reviewing

1. Do you measure energy or power? Maybe the field names don't match 
properly(you store the energy value in the power field).
2. The task already has a structure called profiling_info that has different 
fields (start_time, submit_time, etc.), no need to add one per job
3. We have a source file with some general functions used by all the drivers 
driver_common.c, here we update all the profiling information, and it would 
be a more appropriate place to update the energy and avoid code replication.
4. It would be good to have a configure option that activates this 
measurements when the user requires them
5. It would be maybe better to integrate the code of ex_starpu_ini in 
starpu_profiling_init. I am however a little bit worried about interfering 
with the application's cmd line parameters. We usually use env variables for 
runtime options for starpu.
6. It might be unnecessary overhead to count all the sockets at each time you 
measure the power, they won't change their number at runtime (I don't think 
starpu deals with any fault tolerance problems). 
7. In starpu_ex.c there are some calls to sleep, are they really necessary?

B. Performance

8. Do you have some measurements or a paper that shows the overhead of your 
approach? The client-server communication seams a lot of overhead added 
exactly when we pop tasks which is the most performance sensitive part of 
starpu.

9. It seams that you measure the power of all CPU cores and GPUs after the 
execution of each task (ex_starpu_get_power(j->ex_start_ts, j->ex_end_ts, 
ALL_POWER))). Am I correct? How does this work for a parallel application, 
when anther processor executes another task?

10. This is more like a suggestion: In order to monitor the processors you 
create a set of threads to take care of this. StarPU also has its own set of 
threads. Leaving the OS deal with the context switch might affect 
performance. Did you consider maybe using starpu's threads instead?  I 
understand that your approach client server is definitely a better software 
engineering solution, and it can then be easily integrated in any software 
without modifying the code, but the overhead might be significant. 

That's all I can see for now. Please let me know if you have any questions or 
if I can help in any way. 

Best,
Andra

----- Mail original -----
> De: "Fangli Pi" <hpcfapix@hlrs.de>
> À: starpu-devel@lists.gforge.inria.fr
> Cc: "Dennis Hoppe" <dennis.hoppe@hlrs.de>, "Dmitry Khabi" <khabi@hlrs.de>, 
> "Michael Gienger" <gienger@hlrs.de>
> Envoyé: Lundi 9 Mai 2016 12:15:03
> Objet: [Starpu-devel] Fwd: Strong interest in contributions and       
> integrations
> 
> Dear StarPU devel-group,
> 
> Sorry for bothering you again with my mail sent two weeks ago.
> 
> Being afraid that there will be delays due to mail barriers or holidays
> during these two weeks, I would here restate my deep interest in
> contributing in StarPU by our project team.
> 
> In these two weeks we have succeed in integrating the energy measurements in
> StarPU (version 1.2.0rc5) for cpu drivers (first step towards the
> integration). Changes based on StarPU source can be found in the following
> link:
> http://gitlab.excess-project.eu/hpcfapix/starpu-ex-1-2-0rc5.git
> 
> We are now keeping the development private and it would be grateful to get
> your permission to make it public after all development is done. It is still
> of great interest by our side to merge the extension into StarPU and
> hopefully it would also be an usable feature for StarPU.
> 
> Sincerely looking forward to your reply.
> 
> Best regards,
> Fangli Pi
> 
> ----- Forwarded Message -----
> From: "Fangli Pi" <hpcfapix@hlrs.de>
> To: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>
> Cc: "Michael Gienger" <gienger@hlrs.de>, "Dennis Hoppe"
> <dennis.hoppe@hlrs.de>, "Dmitry Khabi" <khabi@hlrs.de>
> Sent: Monday, April 25, 2016 10:27:27 AM
> Subject: Strong interest in contributions and integrations
> 
> Dear StarPU devel-group,
> 
> I am now working on an EU project (EXCESS, http://excess-project.eu/) using
> StarPU as a runtime system. For the project's concentration on
> energy-efficient computation, we have tried to extend StarPU power model to
> retrieve energy and power measurements from one monitoring framework,
> developed by our team.
> 
> Following is the link of our StarPU power extension, including the source
> code, some examples and a basic introduction.
> https://github.com/excess-project/starpu-energy-aware-extension.git
> 
> As we are currently thinking about an integration with StarPU, it would be
> very beneficial to get your professional opinions and relevant supports upon
> the aspect of StarPU.
> 
> Looking forward to your reply and a possible cooperation in the future.
> 
> Best regards,
> Fangli Pi
> 
> --
> High Performance Computing Center (HLRS)
> University of Stuttgart
> Nobelstr. 19
> D-70569 Stuttgart, Germany
> 
> phone: ++49-711-685-60442
> --
> High Performance Computing Center (HLRS)
> University of Stuttgart
> Nobelstr. 19
> D-70569 Stuttgart, Germany
> 
> phone: ++49-711-685-60442
> _______________________________________________
> Starpu-devel mailing list
> Starpu-devel@lists.gforge.inria.fr
> http://lists.gforge.inria.fr/mailman/listinfo/starpu-devel
>
-- 
High Performance Computing Center (HLRS)   
University of Stuttgart                    
Nobelstr. 19                               
D-70569 Stuttgart, Germany

phone: ++49-711-685-60442

Attachment: feedback-monitoring-framework-2015.pdf
Description: Adobe PDF document