Discussion related to cado-nfs

[Philip Pemberton <philpem@philpem.me.uk>]

Hi,

I've been running CADO-NFS on a distributed cluster where several 
machines have failed due to hardware issues. In summary, I've tried to 
add more during the CADO-NFS run and sadly some nodes have failed.

I now get this error about having too many failed workunits.

Is there any way to discard these failed/incomplete work units and pick 
up where things left off using the remaining nodes?

This has been a long run and I'd like to salvage as much as possible.

I've included the output from cado-nfs.py below.

Thanks,
Phil.


Info:Lattice Sieving: Starting
Info:Lattice Sieving: We want 43267462 relation(s)
Error:Lattice Sieving: Program run on mint.5425b52 failed with exit code 132
Error:Lattice Sieving: Stderr output (last 10 lines only) follow (stored 
in file /tmp/cado.p1h1zthl/c150.upload/c150_sieving_
0000-43630000.urxwfbs6.stderr0):
Error:Lattice Sieving:  # redoing q=43620013, rho=4777421 because 1s 
buckets are full
Error:Lattice Sieving:  # Fullest level-1s bucket #173, wrote 8282/8272
Error:Lattice Sieving:  Illegal instruction (core dumped)
Error:Lattice Sieving:
Error:Lattice Sieving: Exceeded maximum number of failed workunits, 
maxfailed=100
Traceback (most recent call last):
  File "./cado-nfs.py", line 122, in <module>
    factors = factorjob.run()
  File "./scripts/cadofactor/cadotask.py", line 5914, in run
    last_status, last_task = self.run_next_task()
  File "./scripts/cadofactor/cadotask.py", line 6006, in run_next_task
    return [task.run(), task.title]
  File "./scripts/cadofactor/cadotask.py", line 3241, in run
    self.submit_command(p, "%d-%d" % (q0, q1), commit=False)
  File "./scripts/cadofactor/cadotask.py", line 1551, in submit_command
    self.wait()
  File "./scripts/cadofactor/cadotask.py", line 1627, in wait
    if not self.send_request(Request.GET_WU_RESULT):
  File "./scripts/cadofactor/cadotask.py", line 1412, in send_request
    return super().send_request(request)
  File "./scripts/cadofactor/patterns.py", line 66, in send_request
    return self.__mediator.answer_request(request)
  File "./scripts/cadofactor/cadotask.py", line 6074, in answer_request
    result = self.request_map[key]()
  File "./scripts/cadofactor/wudb.py", line 1589, in send_result
    was_received = self.notifyObservers(message)
  File "./scripts/cadofactor/patterns.py", line 32, in notifyObservers
    if observer.updateObserver(message):
  File "./scripts/cadofactor/cadotask.py", line 3257, in updateObserver
    if self.handle_error_result(message):
  File "./scripts/cadofactor/cadotask.py", line 1701, in 
handle_error_result
    raise Exception("Too many failed work units")
Exception: Too many failed work units


--
Phil.
philpem@philpem.me.uk
http://www.philpem.me.uk