Re: [Starpu-devel] Problem with performance model calibration

[Mirko Myllykoski <mirkom@cs.umu.se>]

Hi,

Thank you for your reply!

On 2020-05-20 12:44, Samuel Thibault wrote:

> > The ***main problem*** is that the performance is very unpredictable.
>
> Did you have a look at the output of starpu_perfmodel_plot? I am getting
> something like the attached.
>
> For CPUs, it shows that the linear regression indeed has troubles
> catching anything for small sizes, but a non-linear regression should
> be able to catch it fine, it would be useful to see what it gets.

I pulled the latest version of StarPU from the GIT repository, changed 
the GEMM tasks to use to non-linear models (STARPU_NL_REGRESSION_BASED) 
and re-calibrated with STARPU_CALIBRATE=1 STARPU_SCHED=random. The 
calibration files can be found from [1]. The starpu_perfmodel_plot 
output can be found from [3]. More data but quite noisy. Weirdly, it 
appears that the code still default to the linear models...

I did not see any improvement in performance during my actual test runs.

> The variability of the measurements is quite high, though, like 10x
> difference for the same size :/ So it looks like there is some parameter
> other than just M and N that you would want to pass to the MRM.
>
> > The GEMM codelets use the flop count as a "size base".
>
> > It appears that MRMs perform even worse than the linear models.
>
> That's surprising, it should be at least as good. I see in your source
> code that in MRM you included only M^2*N and M*N. I guess your flop
> count formula uses only these factors? Does MRM not manage to get even
> close to the constants for these factors?

The flop count is 2 * M^2*N (M X M times M X N matrix multiplication). 
The M*N factor is related to copy operation. You can find MRM 
calibration files from [2]. The main problem with MRMs is that I get 
some speedup with linear models but MRM models seem to make the code 
slower when GPUs are used. Also, the models [2] look wrong (negative 
coefficients).

The MRM model for the CPU-only starneig_push_bulges tasks seems to work 
quite well:

_________________________________________________________________________
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv

$ df <- read.csv("starneig_push_bulges_pm.out")
$ model1 <- lm(data=df, Duration ~ I(N^2*S) + I(N*N))
$ summary(model1)

Call:
lm(formula = Duration ~ I(N^2 * S) + I(N * N), data = df)

Residuals:
    Min      1Q  Median      3Q     Max
-480430    -107     283     404 1748349

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept) -4.817e+02  2.979e+01  -16.17   <2e-16 ***
I(N^2 * S)   7.252e-04  1.159e-06  625.57   <2e-16 ***
I(N * N)     5.265e-02  6.787e-04   77.57   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 11820 on 172325 degrees of freedom
Multiple R-squared:  0.9813,    Adjusted R-squared:  0.9813
F-statistic: 4.516e+06 on 2 and 172325 DF,  p-value: < 2.2e-16

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

However, the MRM model for the starneig_left_gemm_update tasks seems to 
fail:

_________________________________________________________________________
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv

$ df <- read.csv("~/codelets/tmp/starneig_left_gemm_update_pm.out")
$ model1 <- lm(data=df, Duration ~ I(M^2*N) + I(M*N))
$ summary(model1)

Call:
lm(formula = Duration ~ I(M^2 * N) + I(M * N), data = df)

Residuals:
    Min      1Q  Median      3Q     Max
-740061    -965    -719     173  476977

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  1.109e+03  1.675e+01   66.19   <2e-16 ***
I(M^2 * N)   5.369e-05  8.652e-08  620.49   <2e-16 ***
I(M * N)    -2.907e-02  1.692e-04 -171.79   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 15180 on 921547 degrees of freedom
Multiple R-squared:  0.6984,    Adjusted R-squared:  0.6984
F-statistic: 1.067e+06 on 2 and 921547 DF,  p-value: < 2.2e-16

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I looks like both CPU and GPU measurement are mixed in the *.out 
files... Is this true? I am just looking at the plot(df) output [4].

> For GPUs, the story is very different. The variability is as high (10x
> again) but doesn't seem to depend very much on the data size. That's
> surprising and it could be useful to investigate it.

Yes, this is quite weird. The same pattern can be seen in the new 
calibration data [1]. I have also seen the variability in FxT traces. I 
don't see this behavior on my desktop machine but the V100s in our 
cluster seem to suffer from this problem. The situation was even worse 
with K80s. I could not get any reliable performance out of those GPUs.

> > even if I first calibrate the models by setting STARPU_CALIBRATE=1
>
> Which scheduler are you using?  When using a dmda scheduler, it will
> use the performance model to optimize execution as soon as the model
> provides information. For linear regression models, that can happen
> quite soon that the model has some estimation, and then the scheduler
> will take decisions accordingly. That could explain why on the attached
> graph only big sizes were run on the GPU, because the perfmodel already
> says it's better to do so. A way to avoid this is to just use an eager
> or random scheduler to let tasks just go without caring about the
> perfmodel.

If GPU(s) are detected, the code defaults to dmdas, otherwise, the code 
deaults to prio. As I said above, I rerun the calibration with 
STARPU_CALIBRATE=1 STARPU_SCHED=random. See [1].

 - Mirko

[1]: https://drive.google.com/open?id=1_BwOxTGo8xr9LyI4i9EUIPerjttiZMyj
[2]: https://drive.google.com/open?id=1wDCjFEteBGNQF2-MPgCjRpTaSdFrSAC1
[3]: https://drive.google.com/open?id=1FJqWMfgoQ5jt4DlPwFbNePSP-9CHGdlk
[4]: https://drive.google.com/open?id=1KV7uwE_E9TcRjlkBunRuP0Q-AXafMc7E

--
Mirko Myllykoski, Ph.D.
Senior research engineer, Department of Computing Science
Application expert, HPC2N
Umeå University

By sending an email to Umeå University, the University processes your 
personal data. For more information, please read: https://www.umu.se/en/gdpr