ó î–+[c@sdZddlZddlZddlZddlmZddlZddlZddlZddl Z yddl Z Wnddl Z nXe ej jddƒƒp®dZeräyddlZWqäek ràdZqäXner6yejƒZ[Wq6eefk r2ZdZejdefƒq6XnddlmZmZddlmZmZdd lmZmZd „Z d „Z!d e"fd „ƒYZ#de$fd„ƒYZ%d„Z&de$fd„ƒYZ'de$fd„ƒYZ(defd„ƒYZ)dS(s+ Helpers for embarrassingly parallel code. iÿÿÿÿN(tsqrttJOBLIB_MULTIPROCESSINGis'%s. joblib will operate in serial mode(t format_exctformat_outer_frames(tLoggertshort_format_time(tTransportableExceptiont _mk_exceptioncCstdkrdStjƒS(s Return the number of CPUs. iN(tmultiprocessingtNonet cpu_count(((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR 0s cCsv|s tS|dkrtS|dkr*tSdd|d}t||ƒ}t|d|ƒ}t|ƒt|ƒkS(s¤ Returns False for indices increasingly appart, the distance depending on the value of verbose. We use a lag increasing as the square of index i igà?i ii(tTruetFalseRtint(tindextverbosetscalet next_scale((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyt_verbosity_filter;s  tWorkerInterruptcBseZdZRS(sa An exception that is not KeyboardInterrupt to allow subprocesses to be interrupted. (t__name__t __module__t__doc__(((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyRNst SafeFunctioncBs eZdZd„Zd„ZRS(sÍ Wraps a function to make it exception with full traceback in their representation. Useful for parallel computing with multiprocessing, for which exceptions cannot be captured. cCs ||_dS(N(tfunc(tselfR((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyt__init__\scOs}y|j||ŽSWnbtk r0tƒ‚nItjƒ\}}}t|||ddddƒ}t||ƒ‚nXdS(Ntcontexti t tb_offseti(RtKeyboardInterruptRtsystexc_infoRR(Rtargstkwargste_typete_valuete_tbttext((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyt__call___s   (RRRRR&(((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyRVs csMtjˆƒ‡fd†}ytjˆƒ|ƒ}Wntk rHnX|S(s< Decorator used to capture the arguments of a function. cs ˆ||fS(N((R R!(tfunction(sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pytdelayed_functionvs(tpickletdumpst functoolstwrapstAttributeError(R'R(((R'sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pytdelayedos  tImmediateApplycBs eZdZd„Zd„ZRS(s# A non-delayed apply function. cCs|||Ž|_dS(N(tresults(RRR R!((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyRƒscCs|jS(N(R0(R((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pytgetˆs(RRRRR1(((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR/€s tCallBackcBs eZdZd„Zd„ZRS(sk Callback used by parallel: it is used for progress reporting, and to add data to be processed cCs||_||_dS(N(tparallelR(RRR3((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR‘s cCs3|jj|jƒ|jjr/|jjƒndS(N(R3tprint_progressRt _iterablet dispatch_next(Rtout((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR&•s (RRRRR&(((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR2s tParallelcBs_eZdZdddd„Zd„Zd„Zd„Zd„Zd „Zd „Z d „Z RS( sÖ Helper class for readable parallel mapping. Parameters ----------- n_jobs: int The number of jobs to use for the computation. If -1 all CPUs are used. If 1 is given, no parallel computing code is used at all, which is useful for debuging. For n_jobs below -1, (n_cpus + 1 - n_jobs) are used. Thus for n_jobs = -2, all CPUs but one are used. verbose: int, optional The verbosity level: if non zero, progress messages are printed. Above 50, the output is sent to stdout. The frequency of the messages increases with the verbosity level. If it more than 10, all iterations are reported. pre_dispatch: {'all', integer, or expression, as in '3*n_jobs'} The amount of jobs to be pre-dispatched. Default is 'all', but it may be memory consuming, for instance if each job involves a lot of a data. Notes ----- This object uses the multiprocessing module to compute in parallel the application of a function to many different arguments. The main functionality it brings in addition to using the raw multiprocessing API are (see examples for details): * More readable code, in particular since it avoids constructing list of arguments. * Easier debuging: - informative tracebacks even when the error happens on the client side - using 'n_jobs=1' enables to turn off parallel computing for debuging without changing the codepath - early capture of pickling errors * An optional progress meter. * Interruption of multiprocesses jobs with 'Ctrl-C' Examples -------- A simple example: >>> from math import sqrt >>> from joblib import Parallel, delayed >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10)) [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0] Reshaping the output when the function has several return values: >>> from math import modf >>> from joblib import Parallel, delayed >>> r = Parallel(n_jobs=1)(delayed(modf)(i/2.) for i in range(10)) >>> res, i = zip(*r) >>> res (0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5) >>> i (0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0) The progress meter: the higher the value of `verbose`, the more messages:: >>> from time import sleep >>> from joblib import Parallel, delayed >>> r = Parallel(n_jobs=2, verbose=5)(delayed(sleep)(.1) for _ in range(10)) #doctest: +SKIP [Parallel(n_jobs=2)]: Done 1 out of 10 | elapsed: 0.1s remaining: 0.9s [Parallel(n_jobs=2)]: Done 3 out of 10 | elapsed: 0.2s remaining: 0.5s [Parallel(n_jobs=2)]: Done 6 out of 10 | elapsed: 0.3s remaining: 0.2s [Parallel(n_jobs=2)]: Done 9 out of 10 | elapsed: 0.5s remaining: 0.1s [Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 0.5s finished Traceback example, note how the line of the error is indicated as well as the values of the parameter passed to the function that triggered the exception, even though the traceback happens in the child process:: >>> from string import atoi >>> from joblib import Parallel, delayed >>> Parallel(n_jobs=2)(delayed(atoi)(n) for n in ('1', '300', 30)) #doctest: +SKIP #... --------------------------------------------------------------------------- Sub-process traceback: --------------------------------------------------------------------------- TypeError Fri Jul 2 20:32:05 2010 PID: 4151 Python 2.6.5: /usr/bin/python ........................................................................... /usr/lib/python2.6/string.pyc in atoi(s=30, base=10) 398 is chosen from the leading characters of s, 0 for octal, 0x or 399 0X for hexadecimal. If base is 16, a preceding 0x or 0X is 400 accepted. 401 402 """ --> 403 return _int(s, base) 404 405 406 # Convert string to long integer 407 def atol(s, base=10): TypeError: int() can't convert non-string with explicit base ___________________________________________________________________________ Using pre_dispatch in a producer/consumer situation, where the data is generated on the fly. Note how the producer is first called a 3 times before the parallel loop is initiated, and then called to generate new data on the fly. In this case the total number of iterations cannot be reported in the progress messages:: >>> from math import sqrt >>> from joblib import Parallel, delayed >>> def producer(): ... for i in range(6): ... print 'Produced %s' % i ... yield i >>> out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')( ... delayed(sqrt)(i) for i in producer()) #doctest: +SKIP Produced 0 Produced 1 Produced 2 [Parallel(n_jobs=2)]: Done 1 jobs | elapsed: 0.0s Produced 3 [Parallel(n_jobs=2)]: Done 2 jobs | elapsed: 0.0s Produced 4 [Parallel(n_jobs=2)]: Done 3 jobs | elapsed: 0.0s Produced 5 [Parallel(n_jobs=2)]: Done 4 jobs | elapsed: 0.0s [Parallel(n_jobs=2)]: Done 5 out of 6 | elapsed: 0.0s remaining: 0.0s [Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s finished iitallcCs=||_||_||_d|_d|_tƒ|_dS(N(Rtn_jobst pre_dispatchR t_poolt_outputtlistt_jobs(RR:RR;((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR$s      cCs&|jdkr”t|||ƒ}t|jƒ}t||jƒsr|jd|dtt j ƒ|j ƒfƒn|jj |ƒ|j d7_ nŽ|j jƒzpyS|jjt|ƒ||dt|j |ƒƒ}|jj |ƒ|j d7_ Wntk rdGHnXWd|j jƒXdS(sK Queue the function for computing, with or without multiprocessing s!Done %3i jobs | elapsed: %sitcallbacks[Parallel] Pool seems closedN(R<R R/tlenR?RRt_printRttimet _start_timetappendt n_dispatchedt_locktacquiret apply_asyncRR2tAssertionErrortrelease(RRR R!tjobR((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pytdispatch.s&     cCs|jd7_xy|jrŠy>|jjƒ\}}}|j|||ƒ|jd8_Wqtk rlqtk r†d|_dSXqWdS(s4 Dispatch more data for parallel processing iN(t_dispatch_amountR5tnextRMt ValueErrort StopIterationR (RRR R!((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR6Hs    cCsY|js dS|jdkr+tjj}n tjj}||}|d||fƒdS(sG Display the message on stout or stderr depending on verbosity Ni2s [%s]: %s (RRtstderrtwritetstdout(Rtmsgtmsg_argstwriter((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyRB[s   cCs|js dStjƒ|j}|jrbt||jƒr?dS|jd|dt|ƒfƒn«|j}|dksÄ||d|j}||jd}|d|k}|s½||rÄdSn||d|j|d}|jd|d|t|ƒt|ƒfƒdS(suDisplay the process of the parallel execution only a fraction of time, controled by self.verbose. Ns!Done %3i jobs | elapsed: %siigð?s/Done %3i out of %3i | elapsed: %s remaining: %s( RRCRDR5RRBRRFt_pre_dispatch_amount(RRt elapsed_timet queue_lengthtcursort frequencyt is_last_itemtremaining_time((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR4is.        cCsRtƒ|_x?|jrMt|dƒr7|jjƒn|jjdƒ}t|dƒrh|jjƒny|jj|j ƒƒWqt |j ƒk rI}t |t tfƒrçt|dƒrÞ|jjƒ|jjƒn|‚nYt |tƒr@tddddƒ}d||jf}t|jƒd}||ƒ‚n|‚qXqWdS( NRGiR<Ri t stack_startisÏMultiprocessing exception: %s --------------------------------------------------------------------------- Sub-process traceback: --------------------------------------------------------------------------- %s(R>R=R?thasattrRGRHtpopRKRER1ttuplet exceptionst isinstanceRRR<tcloset terminateRRtmessageRtetype(RRLt exceptiont this_reporttreporttexception_type((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pytretrieves2       c Csz|jrtdƒ‚n|j}|dkrYtdk rYttjƒd|dƒ}ntg|_|dks‰tdks‰|dkr›d}d|_ nktj ƒj rÏd}d|_ t j dddƒn7tj|ƒ|_ tjƒ|_|jjttgƒ|jdks!|dkr6d|_d|_n^||_d|_|j}t|dƒrot|ƒ}nt|ƒ|_}tj||ƒ}tjƒ|_d|_ zx*|D]"\}}}|j!|||ƒq¶W|j"ƒtjƒ|j}|j#d t$|j%ƒt$|j%ƒt&|ƒfƒWd|dkrW|j j'ƒ|j j(ƒnt)ƒ|_X|j%}d|_%|S( Ns)This Parallel instance is already runningiis1Parallel loops cannot be nested, setting n_jobs=1t stackleveliR9tendswiths*Done %3i out of %3i | elapsed: %s finished(*R?RPR:RR tmaxR RRcR<tcurrent_processt _daemonictwarningstwarntPoolt threadingtLockRGtextendRRR;R5RXRNR`tevalR t itertoolstisliceRCRDRFRMRmRBRAR=RRetjoinR>( RtiterableR:R;R'R R!RYtoutput((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR&¸s\    $                  cCsd|jj|jfS(Ns %s(n_jobs=%s)(t __class__RR:(R((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyt__repr__÷s( RRRRRMR6RBR4RmR&R€(((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pyR8œs‡    ' ( ?(*RtosRRstmathRR+RCRvRztcPickleR)R tenvironR1R Rt ImportErrort Semaphoret_semtOSErrorteRtt format_stackRRtloggerRRt my_exceptionsRRR Rt ExceptionRtobjectRR.R/R2R8(((sW/home/psgendb/BIRCHDEV/install/quast-5.2.0/quast_libs/site_packages/joblib2/parallel.pytsH       !