Re: [Starpu-devel] Suggestions about creating starpu threads

[Andra Hugo <andra.hugo@inria.fr>]

Yes that is great!

----- Mail original -----
> De: "Fangli Pi" <hpcfapix@hlrs.de>
> À: "Andra Hugo" <andra.hugo@inria.fr>
> Cc: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>
> Envoyé: Mardi 8 Novembre 2016 15:04:08
> Objet: Re: Suggestions about creating starpu threads
> 
> Hi Andra,
> 
> 4. Do you mean a stable package like xxx.tar.gz? where should I put it? 
> still
> on github? Does something like the following work for you?
> 
> https://github.com/excess-project/monitoring-agent/archive/16.8.1.tar.gz
> 
> If this is fine, I can make an extra version especially for starpu and will
> not change it later.
> 
> Best regards,
> Fangli
> 
> ----- Original Message -----
> From: "Andra Hugo" <andra.hugo@inria.fr>
> To: "Fangli Pi" <hpcfapix@hlrs.de>
> Cc: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>
> Sent: Tuesday, November 8, 2016 1:58:58 PM
> Subject: Re: Suggestions about creating starpu threads
> 
> Hi Fangli,
> 
> 2. It is doxygen. There is a chapter offline_performance_tools, you could 
> add
> one with offline_power_tools
> 3. Yes. You might need to change the Makefile.am too but with the values
> obtained from the config.
> 4. A stable package would be better I think, so that if you change the git 
> it
> won't break starpu.
> 
> Andra
> 
> ----- Mail original -----
> > De: "Fangli Pi" <hpcfapix@hlrs.de>
> > À: "Andra Hugo" <andra.hugo@inria.fr>
> > Cc: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>
> > Envoyé: Mardi 8 Novembre 2016 10:44:18
> > Objet: Re: Suggestions about creating starpu threads
> > 
> > Hi Andra,
> > 
> > Thanks very much for your suggestions.
> > 
> > 1. No problem, can be done today.
> > 2. I don't know how the doc is generated and where should I put the
> > instructions, could you give me an example?
> > 3. Do you mean to change configure.ac? Is it the only file to change?
> > 4. We have stable releases on github for the two external tools, but a
> > stable
> > package is not generated. Is a stable package would be better? I can try 
> > to
> > make it if it is necessary.
> > 
> > Best regards,
> > Fangli
> > 
> > ----- Original Message -----
> > From: "Andra Hugo" <andra.hugo@inria.fr>
> > To: "Fangli Pi" <hpcfapix@hlrs.de>
> > Cc: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>
> > Sent: Monday, November 7, 2016 9:25:53 PM
> > Subject: Re: Suggestions about creating starpu threads
> > 
> > Hi Fangli,
> > 
> > I am currently working on the power branch. I have a few things that you
> > may
> > probably be able to help if you have some time.
> > 
> > 1. I've seen that you have a folder demo, could you integrate that one in
> > starpu examples?
> > 2. You have a few instructions on how to use the tool, could you integrate
> > them in the starpu doc?
> > 3. You are updating the compile flags manually in src/Makefile.am, could
> > you
> > do it through the autotools? (we do this with basically all the external
> > libraries, you can take hwloc as an example)
> > 4. Will the 2 external tools, the server and the agent, still exist on
> > github
> > for some time now? Do you plan a release or a stable package for it?
> > 
> > Thank you,
> > Andra
> > 
> > ----- Mail original -----
> > > De: "Fangli Pi" <hpcfapix@hlrs.de>
> > > À: "Andra Hugo" <andra.hugo@inria.fr>
> > > Envoyé: Mercredi 19 Octobre 2016 11:59:59
> > > Objet: Re: Suggestions about creating starpu threads
> > > 
> > > Hi Andra,
> > > 
> > > That is totally fine for me. Thanks also for making it clear with me
> > > about
> > > the release process.
> > > 
> > > Please tell me what need to be adjusted when you use it for the use
> > > cases.
> > > 
> > > Besides, what can be done now from my side to assist you? If there are
> > > any,
> > > please feel free to contact me.
> > > 
> > > Best regards,
> > > Fangli
> > > 
> > > ----- Original Message -----
> > > From: "Andra Hugo" <andra.hugo@inria.fr>
> > > To: "Fangli Pi" <hpcfapix@hlrs.de>
> > > Cc: "Samuel Thibault" <samuel.thibault@inria.fr>, "starpu-devel"
> > > <starpu-devel@lists.gforge.inria.fr>, "dennis hoppe"
> > > <dennis.hoppe@hlrs.de>
> > > Sent: Wednesday, October 19, 2016 11:45:20 AM
> > > Subject: Re: Suggestions about creating starpu threads
> > > 
> > > Hi Fangli,
> > > 
> > > So sorry I got caught up with different deadlines.
> > > 
> > > I suggest integrating it as a branch of the trunk of starpu sources in
> > > order
> > > to pass it regularly through the starpu testing system, and then from
> > > there
> > > it will start its road towards the release.
> > > 
> > > We might need to adjust it on the way to the release in order to match
> > > the
> > > use cases we might have for power measurements. Would that work for you?
> > > 
> > > I can work on this during the week of 7-11 of November. Would that work
> > > for
> > > you if I indicate after the mearge how does the final setup looks like?
> > > Would that match your deadlines?
> > > 
> > > So sorry again for the delay and thank you for getting back to me on 
> > > this
> > > matter!
> > > Andra
> > > 
> > > ----- Mail original -----
> > > > De: "Fangli Pi" <hpcfapix@hlrs.de>
> > > > À: "Andra Hugo" <andra.hugo@inria.fr>
> > > > Cc: "Samuel Thibault" <samuel.thibault@inria.fr>, "starpu-devel"
> > > > <starpu-devel@lists.gforge.inria.fr>, "dennis hoppe"
> > > > <dennis.hoppe@hlrs.de>
> > > > Envoyé: Mercredi 19 Octobre 2016 11:10:29
> > > > Objet: Re: Suggestions about creating starpu threads
> > > > 
> > > > Hi Andra,
> > > > 
> > > > Sorry for bothering you again. But it has been months since your last
> > > > reply.
> > > > 
> > > > Now my EU project is ending, at this moment, it would be great to know
> > > > if
> > > > my
> > > > starpu extension implementation could be possibly accepted and merged
> > > > into
> > > > your next release sooner or later.
> > > > 
> > > > For your convenience, I give the link to my source repo here:
> > > > http://gitlab.excess-project.eu/hpcfapix/starpu-ex-1-2-0rc5
> > > > 
> > > > Please contact me in case of any related problems and questions.
> > > > 
> > > > Best regards,
> > > > Fangli
> > > > 
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > From: "Andra Hugo" <andra.hugo@inria.fr>
> > > > To: "Fangli Pi" <hpcfapix@hlrs.de>
> > > > Cc: "Samuel Thibault" <samuel.thibault@inria.fr>, "starpu-devel"
> > > > <starpu-devel@lists.gforge.inria.fr>
> > > > Sent: Friday, June 17, 2016 9:57:45 AM
> > > > Subject: Re: Suggestions about creating starpu threads
> > > > 
> > > > Hi Fangli,
> > > > 
> > > > Thank you! Yes will take a look!
> > > > 
> > > > Best,
> > > > Andra
> > > > 
> > > > ----- Mail original -----
> > > > > De: "Fangli Pi" <hpcfapix@hlrs.de>
> > > > > À: "Andra Hugo" <andra.hugo@inria.fr>
> > > > > Cc: "Samuel Thibault" <samuel.thibault@inria.fr>, "starpu-devel"
> > > > > <starpu-devel@lists.gforge.inria.fr>
> > > > > Envoyé: Jeudi 16 Juin 2016 16:19:01
> > > > > Objet: Re: Suggestions about creating starpu threads
> > > > > 
> > > > > Hi Andra,
> > > > > 
> > > > > During these two weeks I have updated my starpu extension part
> > > > > according
> > > > > to
> > > > > the reviews from you and Samuel and a preliminary assessment about
> > > > > our
> > > > > monitoring client-server communication's overhead is done.
> > > > > 
> > > > > Please check these updates in the following repo:
> > > > > http://gitlab.excess-project.eu/hpcfapix/starpu-ex-1-2-0rc5
> > > > > 
> > > > > There is one clarification file "EX_ChangLog.md" with more details
> > > > > and
> > > > > a
> > > > > figure showing the assessment result.
> > > > > 
> > > > > Please feel free to contact me in case of any possible comments.
> > > > > 
> > > > > Best regards,
> > > > > Fangli
> > > > > 
> > > > > ----- Original Message -----
> > > > > From: "Andra Hugo" <andra.hugo@inria.fr>
> > > > > To: "Fangli Pi" <hpcfapix@hlrs.de>
> > > > > Cc: "Samuel Thibault" <samuel.thibault@inria.fr>, "starpu-devel"
> > > > > <starpu-devel@lists.gforge.inria.fr>
> > > > > Sent: Monday, May 30, 2016 10:12:30 AM
> > > > > Subject: Re: Suggestions about creating starpu threads
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > That sounds good!
> > > > > 
> > > > > Andra
> > > > > 
> > > > > ----- Mail original -----
> > > > > > De: "Fangli Pi" <hpcfapix@hlrs.de>
> > > > > > À: "Andra Hugo" <andra.hugo@inria.fr>, "Samuel Thibault"
> > > > > > <samuel.thibault@inria.fr>
> > > > > > Cc: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>
> > > > > > Envoyé: Lundi 30 Mai 2016 09:04:22
> > > > > > Objet: Re: Suggestions about creating starpu threads
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Sorry for the delay of reply.
> > > > > > 
> > > > > > Fangli
> > > > > > 
> > > > > > ----- Original Message -----
> > > > > > From: "Andra Hugo" <andra.hugo@inria.fr>
> > > > > > To: "Fangli Pi" <hpcfapix@hlrs.de>
> > > > > > Cc: "starpu-devel" <starpu-devel@lists.gforge.inria.fr>, "Samuel
> > > > > > Thibault"
> > > > > > <samuel.thibault@inria.fr>
> > > > > > Sent: Wednesday, May 25, 2016 6:46:11 PM
> > > > > > Subject: Re: Suggestions about creating starpu threads
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Yes indeed we should first measure if this is actually a problem
> > > > > > before
> > > > > > going
> > > > > > into unnecessary modifications.
> > > > > > 
> > > > > > Sure, I will try to do the assessment of the effect on 
> > > > > > performance.
> > > > > > 
> > > > > > Of course as long as we use the tools just for training it should
> > > > > > not
> > > > > > be
> > > > > > a
> > > > > > problem. However, as far as I know RAPL bases their values on a
> > > > > > model
> > > > > > (IPC,
> > > > > > cache misses, frequency etc.) they don't do actual hardware
> > > > > > measurement
> > > > > > of
> > > > > > the voltage. So as Sam says, depending on what we executed before
> > > > > > hand
> > > > > > the
> > > > > > frequency may change and affect the measurements. But I suppose
> > > > > > this
> > > > > > can
> > > > > > happen anyway. Do you do any approximations for this?
> > > > > > 
> > > > > > Right now I don't have any approximations according to the
> > > > > > frequency
> > > > > > change,
> > > > > > but I can have a look at it and try if it is possible.
> > > > > > 
> > > > > > So these measurements should be desactivated if STARPU_CALIBRATE =
> > > > > > 0,
> > > > > > right?
> > > > > > 
> > > > > > According to one of your reviews, I have added one environment to
> > > > > > activate
> > > > > > and deactivate the power measurements, named as
> > > > > > STARPU_POWER_PROFILING,
> > > > > > which determines if ex_starpu_init() is called or not.
> > > > > >  
> > > > > > Andra
> > > > > > 
> > > > > > > ----- Mail original -----
> > > > > > > > De: "Samuel Thibault" <samuel.thibault@inria.fr>
> > > > > > > > À: "Fangli Pi" <hpcfapix@hlrs.de>
> > > > > > > > Cc: "Andra Hugo" <andra.hugo@inria.fr>, "starpu-devel"
> > > > > > > > <starpu-devel@lists.gforge.inria.fr>
> > > > > > > > Envoyé: Mercredi 25 Mai 2016 16:33:42
> > > > > > > > Objet: Re: Suggestions about creating starpu threads
> > > > > > > > 
> > > > > > > > Fangli Pi, on Wed 25 May 2016 16:30:23 +0200, wrote:
> > > > > > > > > After all, the performance during training is not so
> > > > > > > > > important,
> > > > > > > > > right?
> > > > > > > > 
> > > > > > > > Hurting the performance of the application is not very
> > > > > > > > important
> > > > > > > > indeed,
> > > > > > > > as long as the performance of the tasks is preserved.
> > > > > > > > 
> > > > > > > > That said, if it really hurts the performance of the
> > > > > > > > application,
> > > > > > > > that
> > > > > > > > could potentially change the energy consumption. If it's not
> > > > > > > > too
> > > > > > > > bad
> > > > > > > > it's fine, but it'd be useful to assess it.
> > > > > > > > 
> > > > > > > > Samuel
> > > > > > > > 
> > > > > > >
> > > > > > --
> > > > > > High Performance Computing Center (HLRS)
> > > > > > University of Stuttgart
> > > > > > Nobelstr. 19
> > > > > > D-70569 Stuttgart, Germany
> > > > > > 
> > > > > > phone: ++49-711-685-60442
> > > > > >
> > > > > --
> > > > > High Performance Computing Center (HLRS)
> > > > > University of Stuttgart
> > > > > Nobelstr. 19
> > > > > D-70569 Stuttgart, Germany
> > > > > 
> > > > > phone: ++49-711-685-60442
> > > > >
> > > > --
> > > > High Performance Computing Center (HLRS)
> > > > University of Stuttgart
> > > > Nobelstr. 19
> > > > D-70569 Stuttgart, Germany
> > > > 
> > > > phone: ++49-711-685-60442
> > > >
> > > --
> > > High Performance Computing Center (HLRS)
> > > University of Stuttgart
> > > Nobelstr. 19
> > > D-70569 Stuttgart, Germany
> > > 
> > > phone: ++49-711-685-60442
> > >
> > --
> > High Performance Computing Center (HLRS)
> > University of Stuttgart
> > Nobelstr. 19
> > D-70569 Stuttgart, Germany
> > 
> > phone: ++49-711-685-60442
> >
> --
> High Performance Computing Center (HLRS)
> University of Stuttgart
> Nobelstr. 19
> D-70569 Stuttgart, Germany
> 
> phone: ++49-711-685-60442
>