Discussion related to cado-nfs

[Sam Stoelinga <sammiestoel@gmail.com>]

It seems it's still working but only using 1 thread. Can I update the params so that it restarts while using 20 threads? Or would that interfere?

I tried to change the params to threads is 20 and clients is 1, but that would cause it to throw an exception saying Program failed.

On Sun, Mar 2, 2014 at 2:34 PM, Sam Stoelinga <sammiestoel@gmail.com> wrote:

Hi All,

Just wanted to update you, that the change suggested by Paul and Alexander is working as expected. It could now pass the duplicate checking! Thanks for the fast help.

But another issue arrised it just got stuck at the following sentence: Info:Linear Algebra: Starting and nothing is happening after that. See the full log below. 

Anybody ideas? I will try to just restart and see what happens.

This is the log:

Info:Lattice Sieving: Marking workunit c127_sieving_24370000-24373000 as ok

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24436000-24439000 to client localhost14

Info:Lattice Sieving: Adding workunit c127_sieving_24466000-24469000 to database

Info:Lattice Sieving: Found 2838 relations in '/tmp/cado.0ed8NN8FIA/c127.upload/c127.sieving.24376000-24379000.6w1kd6.gz', total is now 15502175/15500000

Info:Lattice Sieving: Marking workunit c127_sieving_24376000-24379000 as ok

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24439000-24442000 to client localhost18

Info:Lattice Sieving: Adding workunit c127_sieving_24469000-24472000 to database

Info:Lattice Sieving: Reached target of 15500000 relations, now have 15502175

Info:Filtering - Duplicate Removal, splitting pass: Starting

Info:Filtering - Duplicate Removal, splitting pass: Splitting 4793 new files

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24442000-24445000 to client localhost8

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24445000-24448000 to client localhost12

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24448000-24451000 to client localhost11

Info:Filtering - Duplicate Removal, splitting pass: Relations per slice: 0: 7748939, 1: 7753236

Info:Filtering - Duplicate Removal, removal pass: Starting

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24451000-24454000 to client localhost4

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24454000-24457000 to client localhost

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24457000-24460000 to client localhost15

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24460000-24463000 to client localhost3

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24463000-24466000 to client localhost10

Info:Filtering - Duplicate Removal, removal pass: 6473688 unique relations remain on slice 0

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24466000-24469000 to client localhost13

Info:HTTP server: 127.0.0.1 Sending workunit c127_sieving_24469000-24472000 to client localhost19

Traceback (most recent call last):

  File "/usr/libexec/cado-nfs/bin/wudb.py", line 1306, in run

    try: result_tuple[0] = getattr(wuar, fn_name)(*args, **kargs)

  File "/usr/libexec/cado-nfs/bin/wudb.py", line 954, in assign

    cursor.begin(EXCLUSIVE)

  File "/usr/libexec/cado-nfs/bin/wudb.py", line 223, in begin

    self._exec("BEGIN EXCLUSIVE")

  File "/usr/libexec/cado-nfs/bin/wudb.py", line 188, in _exec

    self.execute(command)

sqlite3.OperationalError: database is locked

Info:Filtering - Duplicate Removal, removal pass: 6475897 unique relations remain on slice 1

Info:Filtering - Duplicate Removal, removal pass: Of 15502175 newly added relations 12949585 were unique (ratio 0.835340)

Info:Filtering - Duplicate Removal, removal pass: 12949585 unique relations remain in total

Info:Filtering - Singleton removal: Starting

Info:Filtering - Singleton removal: Reading 12949585 unique and 63491 free relations, total 13013076

Info:Filtering - Singleton removal: After purge, 3709934 relations with 3709774 primes remain with weight 70418864 and excess 160

Info:Filtering - Singleton removal: Have enough relations

Info:Lattice Sieving: Cancelling remaining workunits

Info:Client Launcher: Stopped client localhost14 (Host localhost, PID 2730)

Info:Client Launcher: Stopped client localhost17 (Host localhost, PID 2751)

Info:Client Launcher: Stopped client localhost (Host localhost, PID 2640)

Info:Client Launcher: Stopped client localhost3 (Host localhost, PID 2653)

Info:Client Launcher: Stopped client localhost16 (Host localhost, PID 2744)

Info:Client Launcher: Stopped client localhost10 (Host localhost, PID 2703)

Info:Client Launcher: Stopped client localhost15 (Host localhost, PID 2738)

Info:Client Launcher: Stopped client localhost20 (Host localhost, PID 2772)

Info:Client Launcher: Stopped client localhost5 (Host localhost, PID 2667)

Info:Client Launcher: Stopped client localhost4 (Host localhost, PID 2660)

Info:Client Launcher: Stopped client localhost7 (Host localhost, PID 2682)

Info:Client Launcher: Stopped client localhost6 (Host localhost, PID 2675)

Info:Client Launcher: Stopped client localhost13 (Host localhost, PID 2723)

Info:Client Launcher: Stopped client localhost12 (Host localhost, PID 2717)

Info:Client Launcher: Stopped client localhost11 (Host localhost, PID 2709)

Info:Client Launcher: Stopped client localhost2 (Host localhost, PID 2647)

Info:Client Launcher: Stopped client localhost9 (Host localhost, PID 2696)

Info:Client Launcher: Stopped client localhost8 (Host localhost, PID 2688)

Info:Client Launcher: Stopped client localhost19 (Host localhost, PID 2766)

Info:Client Launcher: Stopped client localhost18 (Host localhost, PID 2758)

Info:Filtering - Merging: Starting

Info:Linear Algebra: Starting

On Sun, Mar 2, 2014 at 12:26 AM, Alexander Kruppa <alexander.kruppa@inria.fr> wrote:

----- Original Message -----
> From: "Sam Stoelinga" <sammiestoel@gmail.com>
> To: "paul leyland" <paul.leyland@gmail.com>
> Cc: cado-nfs-discuss@lists.gforge.inria.fr
> Sent: Saturday, 1 March, 2014 12:11:05 PM
> Subject: Re: [Cado-nfs-discuss] Cado-nfs fails on duplicate removal
>
> Hi Paul,
>
> I'm running 3.3.4 so is the bug also affecting me? I guess it is unless
> they fixed the bug in python, unlikely.
>
> Sam

There are no updates on the bug: http://bugs.python.org/issue19003
Thus I'm quite certain that the bug remains in Python 3.3.4. I this case, please change in wuclient2.py the list of affected versions from

BUGGY_MIMEENCODER2 = (
    (3,2,4), (3,2,5),
    (3,3,2)
)

to

BUGGY_MIMEENCODER2 = (
    (3,2,4), (3,2,5),
    (3,3,2), (3,3,3), (3,3,4)
)


Alex

>
>
> On Sat, Mar 1, 2014 at 6:58 PM, Paul Leyland <paul.leyland@gmail.com> wrote:
>
> > I had similar problems.  Alex and Paul (the other one) helped me by
> > pointing to a set of bugs in Python's MIME encoder.
> >
> > The workaround for me was to follow Alex' advice:
> >
> > > There is an entry on http://cado-nfs.gforge.inria.fr/ under Known Bugs;
> > > do you happen to use Python 3.3.3 on the slaves? If yes, does adding
> > > (3,3,3) (without quotes) to the BUGGY_MIMEENCODER2 list in wuclient2.py
> > > fix it?
> >
> > YMMV.
> >
> >
> > Paul (this one)
> >
> >
> > On Sat, 2014-03-01 at 18:48 +0800, Sam Stoelinga wrote:
> > > Hi Pierrick,
> > >
> > >
> > > Thanks for the fast reply!
> > >
> > >
> > > 2.0 Version. Actually there were like 100+ files unrecoverable it
> > > seems. I tried similar commands.
> > > I used the Archlinux cado-nfs package from the aur.
> > >
> > >
> > > I think the disk should be fine on that machine, although I can;t
> > > promise that. I had the same problem on Amazon EC2 tried it 2 times.
> > >
> > >
> > > This is the folder containing al related files:
> > > http://pan.baidu.com/s/1hqgJDGg (Sorry for Chinese mirror)
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Mar 1, 2014 at 3:40 PM, Pierrick Gaudry
> > > <pierrick.gaudry@loria.fr> wrote:
> > >         Hi,
> > >
> > >         Which version of cado-nfs did you use? The 2.0 or a recent git
> > >         version ?
> > >
> > >         At first sight, it seems that the error comes from a corrupted
> > >         relation
> > >         file. Maybe running
> > >           cd /tmp/cado.r1psyafiQA/c127.upload/
> > >           for i in c127.sieving.*; do gzip -t $i; done
> > >         will reveal a file that is not well formed.
> > >         If you put a tarball of your /tmp/cado.r1psyafiQA/ directory
> > >         somewhere for download, I can have a look.
> > >
> > >         You are not the first one who experienced problem with bad
> > >         relation
> > >         files. But it is hard to tell whether this is a bug in
> > >         Cado-nfs, or a
> > >         problem with disks...
> > >
> > >         Maybe we should add more checks that the files are correctly
> > >         written to
> > >         disk and that the compression did well.
> > >
> > >         Regards,
> > >         Pierrick
> > >
> > >
> > >         On Sat, Mar 01, 2014 at 10:29:43AM +0800, Sam Stoelinga wrote:
> > >         > Hi,
> > >         >
> > >         >
> > >         > I get the following error while trying to factor:
> > >         >
> > >         > gzip: stdin: invalid compressed data--format violated
> > >         > filter_rels_producer_thread: load error (Bad file
> > >         descriptor) from
> > >         >
> > >         > It tries to execute this command it seems:
> > >         > ......... many more filesl_2ed.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.19687000-19690000.yr6eid.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.15931000-15934000.s9b_08.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.20965000-20968000.zo24c6.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.10804000-10807000.jszv9k.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.15352000-15355000.bpjgj5.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.23434000-23437000.u7_2tw.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.22465000-22468000.c21z8k.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.10699000-10702000.gq01be.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.21277000-21280000.o38532.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.15889000-15892000.as2trp.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.10231000-10234000.1sv89b.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.19729000-19732000.li0pv7.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.22060000-22063000.3_pxqo.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.15025000-15028000.sptcun.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.15217000-15220000.s5pszl.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.16585000-16588000.0q5cw7.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.22057000-22060000.efhup3.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17077000-17080000.7jnsok.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.20683000-20686000.bdzp9e.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.18061000-18064000.4ybs_h.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.20785000-20788000.5ehwy1.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.18913000-18916000.xt3ama.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.15985000-15988000.ypj3cw.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.12790000-12793000.6lb_cc.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.10537000-10540000.3vbz_n.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.21955000-21958000.eyqdof.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.23818000-23821000.mht30j.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.16324000-16327000.4imyz7.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17479000-17482000.685wuc.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.16504000-16507000.9sosap.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.23845000-23848000.ze28d6.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.21811000-21814000.zwhazw.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.15631000-15634000.x887_u.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.16942000-16945000.6htns3.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.12475000-12478000.aphbum.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.23674000-23677000.9wx6nt.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.21424000-21427000.pfh974.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.21874000-21877000.2imoc1.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17326000-17329000.q4j94g.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.10894000-10897000.f0nc93.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17944000-17947000.v9uz6n.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.20047000-20050000.kyuqxn.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.19468000-19471000.on99v0.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.16534000-16537000.g273_e.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.18736000-18739000.wtama2.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17053000-17056000.eucwjm.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.22261000-22264000.m3mjz8.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17353000-17356000.krizj8.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.13408000-13411000.paz5qu.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17134000-17137000.3y1mfj.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.11803000-11806000.wlkoak.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.22570000-22573000.c43yts.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.14566000-14569000.gathsf.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.14575000-14578000.ff1a_4.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.19345000-19348000.gi8953.gz
> > >         >
> > /tmp/cado.r1psyafiQA/c127.upload/c127.sieving.17836000-17839000.f6cc26.gz |
> > >         > gzip -dc -
> > >         >
> > >         > Any help would be greatly appreciated! I can provide more
> > >         info as needed.
> > >         >
> > >         > Context: I'm using Cado-nfs to decrypt files which were
> > >         encrypted by some
> > >         > malware following this guide:
> > >         >
> > >
> > http://blog.cassidiancybersecurity.com/post/2014/02/Bitcrypt-broken
> > >         > I'm doing this for a friend.
> > >         >
> > >         > Params used:
> > >         > slaves.basepath = /tmp/cado.r1psyafiQA/client/
> > >         > slaves.hostnames = localhost
> > >         > slaves.nrclients = 20
> > >         > slaves.niceness = 10
> > >         > slaves.scriptpath = /usr/libexec/cado-nfs/bin
> > >         > name = c127
> > >         > rlim = 5000000
> > >         > server.address = localhost
> > >         > N =
> > >         >
> > >
> > 6197414403066148465767611830477972406781444626901692539671142017622145058977671120729249030073292775561014115285376040234795763
> > >         > alim = 10000000
> > >         > lpba = 27
> > >         > tasks.merge.coverNmax = 100
> > >         > tasks.execpath = /usr/libexec/cado-nfs/bin
> > >         > tasks.sieve.rels_wanted = 15500000
> > >         > tasks.sieve.mfba = 81
> > >         > tasks.sieve.mfbr = 54
> > >         > tasks.sieve.rlambda = 2.1
> > >         > tasks.sieve.qrange = 3000
> > >         > tasks.sieve.qmin = 10000000
> > >         > tasks.sieve.alambda = 2.8
> > >         > tasks.sieve.I = 12
> > >         > tasks.linalg.bwc.interval = 1000
> > >         > tasks.linalg.bwc.interleaving = 0
> > >         > tasks.linalg.bwc.mn = 64
> > >         > tasks.linalg.bwc.shuffled_product = 1
> > >         > tasks.linalg.characters.nchar = 50
> > >         > tasks.filter.merge.forbw = 3
> > >         > tasks.filter.purge.keep = 160
> > >         > tasks.filter.ratio = 1.1
> > >         > tasks.filter.maxlevel = 20
> > >         > tasks.threads = 1
> > >         > tasks.polyselect.incr = 60
> > >         > tasks.polyselect.maxnorm = 41.7
> > >         > tasks.polyselect.incr = 60
> > >         > tasks.polyselect.maxnorm = 41.7
> > >         > tasks.polyselect.admax = 2000000
> > >         > tasks.polyselect.degree = 5
> > >         > tasks.polyselect.adrange = 20000
> > >         > tasks.polyselect.P = 50000
> > >         > tasks.polyselect.nq = 1000
> > >         > tasks.workdir = /tmp/cado.r1psyafiQA
> > >         > lpbr = 27
> > >
> > >
> > >         > _______________________________________________
> > >         > Cado-nfs-discuss mailing list
> > >         > Cado-nfs-discuss@lists.gforge.inria.fr
> > >         >
> > >
> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/cado-nfs-discuss
> > >
> > >
> > >
> > > _______________________________________________
> > > Cado-nfs-discuss mailing list
> > > Cado-nfs-discuss@lists.gforge.inria.fr
> > > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/cado-nfs-discuss
> >
> >
> >
>
> _______________________________________________
> Cado-nfs-discuss mailing list
> Cado-nfs-discuss@lists.gforge.inria.fr
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/cado-nfs-discuss
>