Discussion related to cado-nfs

[Emmanuel Thomé <Emmanuel.Thome@inria.fr>]

Your call of filter/replay got killed by SIGKILL (-9), which is an
evident hint at the OOM killer having decided to act on your program.
Look at dmesg right away, grep for 10270, I bet you'll have it.

How much memory do you have on your machine? The replay program can be a
bit memory hungry. IIRC we talked about allowing it to produce the output
matrix in a segmented manner so as to save memory, but I don't know the
status of this. There's no obvious trace of this functionality in the
code.

Forget about antebuffer entirely. It's useless noise which can (should)
be deleted. The error messages about it are irrelevant.

E.

On Thu, Feb 03, 2022 at 08:19:33AM +1300, Ross Atkins wrote:
> Thank you so much for your help.
> 
> I found the relation files (there were more than 30,000 of them) and I can
> now successfully start the Linear Algebra Step importing the polynomial +
> relations. I have tried a couple of times now (on two different machines)
> but I am running into a new error. I think these are the relevant lines in
> the c175.log file:
> """
> PID8002 2022-02-02 18:40:27,654 Info:Filtering - Merging: Starting
> PID8002 2022-02-02 18:40:27,654 Debug:Filtering - Merging: merge.run():
> Task state: {}
> PID8002 2022-02-02 18:40:27,665 Command:Command:
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/cado-nfs-master/build/greenkoala/filter/merge
> -mat
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.purged.gz
> -out
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.history.gz
> -skip 32 -target_density 170.0 -t 12 >
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.merge.stdout.1
> 2>
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.merge.stderr.1
> PID8002 2022-02-02 18:53:36,043 Debug:Command: Process with PID 10168
> finished successfully
> PID8002 2022-02-02 18:53:36,047 Info:Filtering - Merging: Merged matrix has
> 15774474 rows and total weight 2688906190 (170.5 entries per row on average)
> PID8002 2022-02-02 18:53:36,050 Command:Command:
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/cado-nfs-master/build/greenkoala/filter/replay
> -purged
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.purged.gz
> -his
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.history.gz
> -index
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.index.gz
> -out
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.sparse.bin
> >
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.replay.stdout.2
> 2>
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.replay.stderr.2
> PID8002 2022-02-02 18:55:48,552 Warning:Command: Process with PID 10270
> finished with return code -9
> PID8002 2022-02-02 18:55:48,566 Error:Filtering - Merging: Program run on
> server failed with exit code -9
> PID8002 2022-02-02 18:55:48,566 Error:Filtering - Merging: Command line
> was:
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/cado-nfs-master/build/greenkoala/filter/replay
> -purged
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.purged.gz
> -his
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.history.gz
> -index
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.index.gz
> -out
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.sparse.bin
> >
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.replay.stdout.2
> 2>
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.replay.stderr.2
> PID8002 2022-02-02 18:55:48,567 Error:Filtering - Merging: Stderr output
> (last 10 lines only) follow (stored in file
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/homework3/c175.replay.stderr.2):
> PID8002 2022-02-02 18:55:48,567 Error:Filtering - Merging:      antebuffer
> set to
> /home/rosstherock/Documents/michael_linton/general_number_field_sieve/cado-nfs-master/build/greenkoala/utils/antebuffer
> PID8002 2022-02-02 18:55:48,567 Error:Filtering - Merging:
> """
> However I am not sure how to adjust the antebuffer setting. In fact I am
> not 100% sure what the antebuffer is. Do you know where I should look next?
> 
> Kind regards,
> ~Ross
> 
> p.s.
> Attached is the entire c175.log in case the context for the antebuffer
> error is helpful.
> 
>  c175.log
> <https://drive.google.com/file/d/1S9E-N6Sgytgl6a18pfsxK8kv99CpvjfH/view?usp=drive_web>
> 
> 
> On Tue, Feb 1, 2022 at 9:14 PM Paul Zimmermann <Paul.Zimmermann@inria.fr>
> wrote:
> 
> >        Dear Ross,
> >
> > > From: Ross Atkins <rosstherock@gmail.com>
> > > Date: Tue, 1 Feb 2022 20:44:24 +1300
> > >
> > > Thanks Paul,
> > >
> > > I have attached the log file, (sorry it is so large). The
> > > scripts/cadofactor/README.md file was very helpful, I have
> > > *
> > >  c175.log
> > > found the polynomial file (c175.poly) but I don't know which file is my
> > > relations file. The readme says:
> > > """
> > > If you want to import already computed relations, use:
> > > tasks.sieve.import=foo
> > > where `foo` is the name of the relation file to be imported (in CADO-NFS
> > > format).
> > > """
> > > Is it possible that the file called "c175.purged.gz" or
> > "c175.freerel.gz" is
> > > my relations file?
> >
> > the relations are in the c175.upload directory. They are in the files
> > c175.xxx.gz. You can do:
> >
> > $ ls DIR/c175*gz > foo
> >
> > then you can use the file 'foo' in tasks.sieve.import=foo.
> >
> > Make sure the path of the relation files is absolute in 'foo'.
> >
> > Best regards,
> > Paul
> >