Discussion related to cado-nfs

[Emmanuel Thomé <emmanuel.thome@gmail.com>]

On Fri, Apr 12, 2013 at 9:56 AM, Junyi <9jhzguy@gmail.com> wrote:
> Interleaving works for the git version works, thanks!

Good. Nice to hear.

>
> "Scheduler is at work then, and timings don't really mean much in the end"
> -> Not quite sure what this means?
> Each of the nodes have 64 cores, and it does tax them quite a bit indeed
> (CPU utilization is at 9000%). This was done as we observed that running
> mpi=4x4 and mpi=6x6 yielded a longer ETA compared to 8x8, and it was set to
> 8x8 because we wanted to minimise the runtime. Is there some other issue we
> overlooked that is related to the scheduler? (e.g. time we're basing our
> decision on isn't real-time but user-time, etc?)

Timings displayed are real time. If you are oversubscribing threads
(here, 128 threads on a 64 thread machine), the time during which
threads will wait for other threads while still in the computation
phase, before actually entering the communication phase, will probably
count as communication. As a consequence, I wouldn't bet too much on
the meaning of these.

Also you will have better results if you avoid distinct mpi jobs on
the same node. I would rather suggest thr=8x8, given that you have 64
cores per node, and then mpi=2x2 if you happen to have four nodes (I
don't know). Whether or not your memory bus will sustain the bandwidth
for all these threads at the same time is not clear, though.

> Some other random observations and issues:
>
> Dispatch stage fires off many error/info message during the local matrix
> construction: e.g. Lsl0 cols 161739+8175Ss13: cols 97043+32348... [cannot be
> small2, beyond impl limits]. This does not stop krylov from running though.

It's info. Admittedly cryptic, especially given that it's interspersed.

> Also, performance at the krylov stage wasn't quite what I expected compared
> to the non-interleaved version (about 6s/iter vs 2s/iter), will play with
> the parameters for now.

Not good indeed.

> Incidentally, krylov-master doesn't seem to like krylov-1.1's balancing info
> even with the same params, and will restart matrix dispatching. krylov-1.1
> will restart from 0th iter thus killing previous progress, in case anyone's
> thinking of trying that.

This is possible.

> Hoping to restart using a better code-base, I re-ran bwc.pl on 2 nodes with
> IB using cado-master on the rsa640 matrix without interleaving, but was
> surprised that it seg-faulted after secure finished ("...saved C.1000") but
> before split started. Message was: "mpiexec noticed that process rank 47
> with PID 34600 on node t002 exited on signal 11 (Segmentation fault).",
> bwc.pl was running on the other node (t001). Not sure if this is one off (I
> previously ran bwc.pl on rsa640 for mpi=4x4, 6x6, 8x8 with interleaving with
> no issues), or something systemic.

Ouch. Problematic. What was your command line ?

> Terminating krylov-1.1 after a checkpoint and restarting usually reduces the
> time per iteration by about 10% (i.e. 2.29s/iter -> 1.85s/iter). This has
> happened for both the times I started krylov from scratch. Not sure if this
> indicates a potential hardware (cache?) or software (memleak?) issue, or
> just simply an averaging issue (i.e. more time spent earlier than later), or
> just old code.

With the same code base ? Will try that. Could very well be a timing oddity.

E.