Discussion related to cado-nfs

[Pierrick Gaudry <pierrick.gaudry@loria.fr>]

Hi,

First thing: for large factorizations, the factor.sh script should not be
used. It is better to use the cadofactor.py script directly.

The README at the top-level mentions this fact, and gives pointer to
further documentation:
  scripts/cadofactor/README
  scripts/cadofactor/parameters

I think that this will answer your question
  "Where can I find these files during execution?"
since you'll have to give explicitly the work directory (I call it $wdir
in this mail).
Be aware that some of the files might be in a subdirectory. For instance,
relation files are in $wdir/*.upload/.

Next question is how to parallelize with MPI.
The answer is probably: you don't want MPI parallelization. 
Indeed, all the steps but the linear algebra can be "trivially"
parallelized, meaning that there is no communication between jobs.
And unless you are dealing with very large factorizations (say, more than
180 digits), the linear algebra is still feasible on a single node, with
non-MPI parallelization.

So, how to parallelize the "trivially parallelized" steps ?

The python script shipped with 2.0 is vastly superior to the old perl
script with this respect: at startup it will start a server, listening at
a certain internet port (that you define in the configuration file).
Then, all you have to do is to start "clients" (again a python script) on
the nodes that you want to help the computation, with the appropriate
command-line parameters. When it starts the server, the python script
prints a copy-pastable command-line for the clients to help the user.

There are some complications if the binaries that you have compiled for
the main computer that runs the server are not compatible with the nodes
where you want to run clients, but this is also documented in the above
cited documentation. Also, we had to add some security things (with SSL,
SHA1 fingerprints, and so on) so that it is not too easy for an attacker
to run arbitrary code on the clients. Feel free to disable them if you
are in a safe environment.

In any case, this should provide very scalable parallelism for the "easy"
tasks.

For the linear algebra task, getting it running in parallel is a bit more
tricky. Please come back to this list if needed, and I'll try to write a
tutorial.

To summarize:
- Choose a machine with a lot of memory and disk. This will be called
  "the server".
- Compile cado-nfs on this machine.
- Edit a param file (inspired from the parameter files given in the
  params/ directory) with your setting, following instructions in
  scripts/cadofactor/parameters.
- Start the cadofactor.py script on the server.
- If you want a machine to contribute, run the
  script/cadofactor/wuclient2.py script on this machine, with the full
  command-line printed by the server.
- You are allowed to set hundreds of those clients, and to switch them on
  and off dynamically. The server will detect this and reschedule tasks
  that were not properly completed.
- To have clients contributing to a server that is far away (maybe behind
  a firewall), then you'll have to set up SSH tunnels or this kind of
  tricks that are more networking than factoring...

Regards,
Pierrick

On Sun, Mar 30, 2014 at 08:32:35PM -0400, Andrew Johnston wrote:
> Hello,
> 
> I'm looking to factor some rather large numbers with CADO-NFS but could not
> find sufficient documentation regarding staged execution and MPI
> capabilities.
> 
> I'd like to run different parts of GNFS on different computers, and from
> what I could find CADO saves its progress (e.g. the polynomial, relations,
> etc.) and I should be able to move these files to different computers and
> rerun. Where can I find these files during execution? Further, how can I
> instruct CADO only to do a specific stage (e.g. only do polynomial
> selection)?
> 
> Further, I've read that CADO supports MPI when run from a perl script,
> however I could not determine which script I would need to run to utilize
> MPI. How do I enable this functionality?
> 
> I apologize if I'm asking questions that can be answered in the
> documentation, but I could not find any resources on the subjects.
> 
> Best Regards,
> 
> Andrew H. Johnston

> _______________________________________________
> Cado-nfs-discuss mailing list
> Cado-nfs-discuss@lists.gforge.inria.fr
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/cado-nfs-discuss