Discussion related to cado-nfs

[Philip Pemberton <philpem@philpem.me.uk>]

Had a read through the documentation, now it starts and quickly fails :(

Anyone know if this is recoverable? I don't mind rerunning a few WUs.

Info:Generate Factor Base: Starting 

Info:Generate Factor Base: Total cpu/real time for makefb: 44.99/17.1804 

Info:Generate Free Relations: Starting 

Info:Generate Free Relations: Total cpu/real time for freerel: 
687.31/206.264
Info:Lattice Sieving: Starting 

Info:Lattice Sieving: We want 43267462 relation(s) 

Info:Lattice Sieving: Found 13405 relations in 
'/tmp/cado.p1h1zthl/c150.upload/c150.19100000-19110000.wcylm29r.gz', 
total is now
34509044/43267462 

Traceback (most recent call last):
  File "./cado-nfs.py", line 122, in <module>
    factors = factorjob.run()
  File "./scripts/cadofactor/cadotask.py", line 5914, in run
    last_status, last_task = self.run_next_task()
  File "./scripts/cadofactor/cadotask.py", line 6006, in run_next_task
    return [task.run(), task.title]
  File "./scripts/cadofactor/cadotask.py", line 3241, in run
    self.submit_command(p, "%d-%d" % (q0, q1), commit=False)
  File "./scripts/cadofactor/cadotask.py", line 1551, in submit_command
    self.wait()
  File "./scripts/cadofactor/cadotask.py", line 1627, in wait
    if not self.send_request(Request.GET_WU_RESULT):
  File "./scripts/cadofactor/cadotask.py", line 1412, in send_request
    return super().send_request(request)
  File "./scripts/cadofactor/patterns.py", line 66, in send_request
    return self.__mediator.answer_request(request)
  File "./scripts/cadofactor/cadotask.py", line 6074, in answer_request
    result = self.request_map[key]()
  File "./scripts/cadofactor/wudb.py", line 1589, in send_result
    was_received = self.notifyObservers(message)
  File "./scripts/cadofactor/patterns.py", line 32, in notifyObservers
    if observer.updateObserver(message):
  File "./scripts/cadofactor/cadotask.py", line 3265, in updateObserver
    self.verification(message.get_wu_id(), ok, commit=True)
  File "./scripts/cadofactor/cadotask.py", line 1595, in verification
    assert self.get_number_outstanding_wus() >= 1
AssertionError


On 13/01/2020 01:40, Philip Pemberton wrote:
> Hi,
>
> I've been running CADO-NFS on a distributed cluster where several 
> machines have failed due to hardware issues. In summary, I've tried to 
> add more during the CADO-NFS run and sadly some nodes have failed.
>
> I now get this error about having too many failed workunits.
>
> Is there any way to discard these failed/incomplete work units and pick 
> up where things left off using the remaining nodes?
>
> This has been a long run and I'd like to salvage as much as possible.
>
> I've included the output from cado-nfs.py below.
>
> Thanks,
> Phil.
>
>
> Info:Lattice Sieving: Starting
> Info:Lattice Sieving: We want 43267462 relation(s)
> Error:Lattice Sieving: Program run on mint.5425b52 failed with exit code 
> 132
> Error:Lattice Sieving: Stderr output (last 10 lines only) follow (stored 
> in file /tmp/cado.p1h1zthl/c150.upload/c150_sieving_
> 0000-43630000.urxwfbs6.stderr0):
> Error:Lattice Sieving:  # redoing q=43620013, rho=4777421 because 1s 
> buckets are full
> Error:Lattice Sieving:  # Fullest level-1s bucket #173, wrote 8282/8272
> Error:Lattice Sieving:  Illegal instruction (core dumped)
> Error:Lattice Sieving:
> Error:Lattice Sieving: Exceeded maximum number of failed workunits, 
> maxfailed=100
> Traceback (most recent call last):
>    File "./cado-nfs.py", line 122, in <module>
>      factors = factorjob.run()
>    File "./scripts/cadofactor/cadotask.py", line 5914, in run
>      last_status, last_task = self.run_next_task()
>    File "./scripts/cadofactor/cadotask.py", line 6006, in run_next_task
>      return [task.run(), task.title]
>    File "./scripts/cadofactor/cadotask.py", line 3241, in run
>      self.submit_command(p, "%d-%d" % (q0, q1), commit=False)
>    File "./scripts/cadofactor/cadotask.py", line 1551, in submit_command
>      self.wait()
>    File "./scripts/cadofactor/cadotask.py", line 1627, in wait
>      if not self.send_request(Request.GET_WU_RESULT):
>    File "./scripts/cadofactor/cadotask.py", line 1412, in send_request
>      return super().send_request(request)
>    File "./scripts/cadofactor/patterns.py", line 66, in send_request
>      return self.__mediator.answer_request(request)
>    File "./scripts/cadofactor/cadotask.py", line 6074, in answer_request
>      result = self.request_map[key]()
>    File "./scripts/cadofactor/wudb.py", line 1589, in send_result
>      was_received = self.notifyObservers(message)
>    File "./scripts/cadofactor/patterns.py", line 32, in notifyObservers
>      if observer.updateObserver(message):
>    File "./scripts/cadofactor/cadotask.py", line 3257, in updateObserver
>      if self.handle_error_result(message):
>    File "./scripts/cadofactor/cadotask.py", line 1701, in 
> handle_error_result
>      raise Exception("Too many failed work units")
> Exception: Too many failed work units

--
Phil.
philpem@philpem.me.uk
http://www.philpem.me.uk