Discussion related to cado-nfs

[Peter Zenin <wdmfrs@gmail.com>]

Good evening.  I started to get errors on duplicate removal step (or filtering) when factoring C>130-ish  after updating the CADO. The sieving process was interrupted using ctrl+c and then resumed. The same error occurred once while factoring a C134 number with no interruptions. The log attached is from cado-nfs commit 615337a3a (Dec 19). This behavior started at the time the CADO-NFS started to use flask. Also there are errors when resuming the sieving process (which has already been reported), but I noticed that 75% of the time factoring ends up correctly.

Environment:
Ubuntu 22.04.4 LTS through WSL2 (Windows 11)
Disk free space: 180GiB

Command used:
python3 cado-nfs.py <number> --workdir=/home/path/to/work/dir

Best regards,
Peter Zenin

Log:

python3 cado-nfs.py 1017709191581021487521458627507460367684725901811744946918948992617746465702359435950016751420548357199118806825157254347623249136365047 --workdir=/home/peter/c136.akbzhjbajh/
Info:root: Using default parameter file ./parameters/factor/params.c135
Info:root: No database exists yet
Info:Database: Database URI is db:sqlite3:///home/peter/c136.akbzhjbajh//c135.db
Info:Database: Opened connection to database /home/peter/c136.akbzhjbajh//c135.db
Info:root: Set tasks.linalg.bwc.threads=8 based on detected physical cores
Info:root: Set tasks.threads=16 based on detected logical cpus
Info:root: tasks.threads = 16 [via tasks.threads]
Info:root: tasks.polyselect.threads = 2 [via tasks.polyselect.threads]
Info:root: tasks.sieve.las.threads = 2 [via tasks.sieve.las.threads]
Info:root: tasks.linalg.bwc.threads = 8 [via tasks.linalg.bwc.threads]
Info:root: tasks.sqrt.threads = 8 [via tasks.sqrt.threads]
Info:root: slaves.scriptpath is /home/peter/cado-nfs/build/DESKTOP-TUFO8678
Info:root: Command line parameters: cado-nfs.py 1017709191581021487521458627507460367684725901811744946918948992617746465702359435950016751420548357199118806825157254347623249136365047 --workdir=/home/peter/c136.akbzhjbajh/
Info:root: If this computation gets interrupted, it can be resumed with cado-nfs.py /home/peter/c136.akbzhjbajh/c135.parameters_snapshot.4
Info:Server Launcher: Adding DESKTOP-TUFO8678 to whitelist to allow clients on localhost to connect
Info:API server: server whitelist is ['127.0.0.1/32', '127.0.1.1/32']
Info:API server: Running from werkzeug
Info:Lattice Sieving: param rels_wanted is 55000000
Info:Complete Factorization / Discrete logarithm: Factoring 1017709191581021487521458627507460367684725901811744946918948992617746465702359435950016751420548357199118806825157254347623249136365047
Info:API server: Running on https://localhost:40443 (Press CTRL+C to quit))
Info:API server: You can start additional cado-nfs-client.py scripts with parameters: --server=https://localhost:40443 --certsha1=a911f15cae13d62a97e1d553818e892e6fa00547
Info:API server: If you want to start additional clients, remember to add their hosts to server.whitelist
Info:Client Launcher: Starting client id localhost on host localhost
Info:Client Launcher: Starting client id localhost+2 on host localhost
Info:Client Launcher: Starting client id localhost+3 on host localhost
Info:Client Launcher: Starting client id localhost+4 on host localhost
Info:Client Launcher: Starting client id localhost+5 on host localhost
Info:Client Launcher: Starting client id localhost+6 on host localhost
Info:Client Launcher: Starting client id localhost+7 on host localhost
Info:Client Launcher: Starting client id localhost+8 on host localhost
Info:Client Launcher: Running clients: localhost (Host localhost, PID 738), localhost+2 (Host localhost, PID 747), localhost+3 (Host localhost, PID 760), localhost+4 (Host localhost, PID 772), localhost+5 (Host localhost, PID 785), localhost+6 (Host localhost, PID 798), localhost+7 (Host localhost, PID 811), localhost+8 (Host localhost, PID 824)
Info:Polynomial Selection (size optimized): Starting
Info:Polynomial Selection (size optimized): 48 polynomials in queue from previous run, worst exp_E 35.580000
Info:Polynomial Selection (size optimized): Already finished - nothing to do
Info:Complete Factorization / Discrete logarithm: Polynomial Selection (size optimized)
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 63638.2
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 60111/39.460/50.083/60.940/2.175
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 50615/38.700/43.661/53.230/1.222
Info:Polynomial Selection (size optimized): Total time: 14375.2
Info:Polynomial Selection (root optimized): Starting
Info:Polynomial Selection (root optimized): Best polynomial previously has Murphy_E = 1.694e-06
Info:Polynomial Selection (root optimized): Already finished - nothing to do
Info:Polynomial Selection (root optimized): Best overall polynomial was 14-th in list after size optimization
Info:Complete Factorization / Discrete logarithm: Polynomial Selection (root optimized)
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 1240.55
Info:Polynomial Selection (root optimized): Rootsieve time: 1246.09
Info:Generate Factor Base: Starting
Info:Complete Factorization / Discrete logarithm: Generate Factor Base
Info:Generate Factor Base: Total cpu/real time for makefb: 6.9/0.945065
Info:Generate Free Relations: Starting
Info:Complete Factorization / Discrete logarithm: Generate Free Relations
Info:Generate Free Relations: Total cpu/real time for freerel: 1176.38/75.1638
Info:Lattice Sieving: Starting
Info:Lattice Sieving: We want 55000000 relation(s)
Info:Lattice Sieving: Reached target of 55000000 relations, now have 55037485
Info:Complete Factorization / Discrete logarithm: Lattice Sieving
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 55037485
Info:Lattice Sieving: Average J: 3795.98 for 447958 special-q, max bucket fill -bkmult 1.0,1s:1.153940
Info:Lattice Sieving: Total time: 171801s
Info:Filtering - Duplicate Removal, splitting pass: Starting
Info:Filtering - Duplicate Removal, splitting pass: Splitting 686 new files
Warning:Command: Process with PID 834 finished with return code -6
Error:Filtering - Duplicate Removal, splitting pass: Program run on server failed with exit code -6
Error:Filtering - Duplicate Removal, splitting pass: Command line was: /home/peter/cado-nfs/build/DESKTOP-TUFO8678/filter/dup1 -prefix dup1.0 -out /home/peter/c136.akbzhjbajh/c135.dup1/ -n 1 -filelist /home/peter/c136.akbzhjbajh/c135.dup1.filelist.3 > /home/peter/c136.akbzhjbajh/c135.dup1.stdout.3 2> /home/peter/c136.akbzhjbajh/c135.dup1.stderr.3
Error:Filtering - Duplicate Removal, splitting pass: Stderr output (last 10 lines only) follows (stored in file /home/peter/c136.akbzhjbajh/c135.dup1.stderr.3):
Error:Filtering - Duplicate Removal, splitting pass:    antebuffer set to /home/peter/cado-nfs/build/DESKTOP-TUFO8678/utils/antebuffer
Error:Filtering - Duplicate Removal, splitting pass:    # Opening output file for slice 1 : /home/peter/c136.akbzhjbajh/c135.dup1//1/dup1.0.0000.gz
Error:Filtering - Duplicate Removal, splitting pass:    # Opening output file for slice 0 : /home/peter/c136.akbzhjbajh/c135.dup1//0/dup1.0.0000.gz
Error:Filtering - Duplicate Removal, splitting pass:
Error:Filtering - Duplicate Removal, splitting pass:    gzip: stdin: decompression OK, trailing garbage ignored
Error:Filtering - Duplicate Removal, splitting pass:    filter_rels_producer_thread: load error (Bad file descriptor) while closing
Error:Filtering - Duplicate Removal, splitting pass:    /home/peter/cado-nfs/build/DESKTOP-TUFO8678/utils/antebuffer 24 /home/peter/c136.akbzhjbajh/c135.upload/c135.1510000-1520000.gz /home/peter/c136.akbzhjbajh/c135.upload/c135.1520000-1530000.gz
<...>
/home/peter/c136.akbzhjbajh/c135.upload/c135.8490000-8500000.gz /home/peter/c136.akbzhjbajh/c135.upload/c135.8480000-8490000.gz /home/peter/c136.akbzhjbajh/c135.upload/c135.8510000-8520000.gz | gzip -dc -
Error:Filtering - Duplicate Removal, splitting pass:
Info:API server: Got notification to stop serving Workunits
Info:Generate Free Relations: Total cpu/real time for freerel: 1176.38/75.1638
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 63638.2
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 60111/39.460/50.083/60.940/2.175
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 50615/38.700/43.661/53.230/1.222
Info:Polynomial Selection (size optimized): Total time: 14375.2
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 55037485
Info:Lattice Sieving: Average J: 3795.98 for 447958 special-q, max bucket fill -bkmult 1.0,1s:1.153940
Info:Lattice Sieving: Total time: 171801s
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 1240.55
Info:Polynomial Selection (root optimized): Rootsieve time: 1246.09
Info:Generate Factor Base: Total cpu/real time for makefb: 6.9/0.945065
Info:Client Launcher: Stopped client localhost (Host localhost, PID 738)
Info:Client Launcher: Stopped client localhost+2 (Host localhost, PID 747)
Info:Client Launcher: Stopped client localhost+3 (Host localhost, PID 760)
Info:Client Launcher: Stopped client localhost+4 (Host localhost, PID 772)
Info:Client Launcher: Stopped client localhost+5 (Host localhost, PID 785)
Info:Client Launcher: Stopped client localhost+6 (Host localhost, PID 798)
Info:Client Launcher: Stopped client localhost+7 (Host localhost, PID 811)
Info:Client Launcher: Stopped client localhost+8 (Host localhost, PID 824)
Info:API server: Shutting down server on exception Program failed
Traceback (most recent call last):
  File "/home/peter/cado-nfs/cado-nfs.py", line 210, in <module>
    factors = factorjob.run()
  File "/home/peter/cado-nfs/./scripts/cadofactor/cadotask.py", line 7185, in run
    last_status = task.run()
  File "/home/peter/cado-nfs/./scripts/cadofactor/cadotask.py", line 4223, in run
    raise Exception("Program failed")
Exception: Program failed