3 QfXD@s6dZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ej ddkZerejZejZddZnejZejZeZd d Zd$Zed Zeed%ZddZddZd&ddZGdddeZGdddeZGdddeZ GdddeZ!Gddde!Z"d'd d!Z#d(d"d#Z$dS))zH Utilities for fast persistence of big data, with optional compression. N) _basestring)BytesIOcCst|tr|S|jdS)Nlatin1) isinstancebytesencode)sr Z/home/psgendb/BIRCHDEV/pkg/SPAdes-3.15.4/linux-x86_64/share/spades/joblib3/numpy_pickle.pyasbytess r cCs dj|S)z-Converts an int to an hexadecimal string z{0:#x})format)Zan_intr r r hex_str&srZZF@cCs|jtt}|jd|S)zV Utility to check the magic signature of a file identifying it as a Zfile r)readlen _ZFILE_PREFIXseek) file_handlemagicr r r _read_magic8s rcCs|jdt|tkstdttt}|j|}|ttd}t|d}|jd}|dkrj|j|tj |jd|}t||kstd||S) zRead the z-file and return the content as a string Z-files are raw data compressed with zlib used internally by joblib for persistence. Backward compatibility is not guaranteed. Do not use for external purposes. rz"File does not have the right magicNr zIIncorrect data length while decompressing %s.The file could be corrupted.) rrrAssertionErrorr_MAX_LENrintzlib decompress)rZ header_lengthlengthZ next_bytedatar r r read_zfileBs      r$cCsD|jttt|}|jt|jt|jtjt||dS)zWrite the data in the given file as a Z-file. Z-files are raw data compressed with zlib used internally by joblib for persistence. Backward compatibility is not guarantied. Do not use for external purposes. N) writerrrr ljustrr compress)rr#r'r"r r r write_zfilees  r(c@s"eZdZdZdddZddZdS) NDArrayWrapperz An object to be persisted instead of numpy arrays. The only thing this object does, is to carry the filename in which the array has been persisted, and the array subclass. TcCs||_||_||_dS)z&Store the useful information for laterN)filenamesubclass allow_mmap)selfr*r+r,r r r __init__|szNDArrayWrapper.__init__cCstjj|j|j}dd|jjjddddD}t|dd}|sHind|j i}|jj |f|}t |d r|j |jj |jjfkr|jjjj|j d d }|j||}|S) zReconstruct the arraycSsg|] }t|qSr )r).0xr r r sz'NDArrayWrapper.read...rNr,T mmap_mode__array_prepare__rb)r)ospathjoin_dirnamer*np __version__splitgetattrr3loadhasattrr+ndarraymemmapcore multiarray _reconstructr4)r- unpicklerr*Znp_verr,Z memmap_kwargsarrayZ new_arrayr r r rs"        zNDArrayWrapper.readN)T)__name__ __module__ __qualname____doc__r.rr r r r r)vs r)c@s eZdZdZddZddZdS)ZNDArrayWrapperaVAn object to be persisted instead of numpy arrays. This object store the Zfile filename in which the data array has been persisted, and the meta information to retrieve it. The reason that we store the raw buffer data of the array and the meta information, rather than array representation routine (tostring) is that it enables us to use completely the strided model to avoid memory copies (a and a.T store as fast). In addition saving the heavy information separately can avoid creating large temporary buffers when unpickling data with large arrays. cCs||_||_||_dS)z&Store the useful information for laterN)r*state init_args)r-r*rMrLr r r r.szZNDArrayWrapper.__init__c Cs\tjj|j|j}|jjjj|j }t |d}t |}WdQRX|j |f}|j ||S)z>Reconstruct the array from the meta-information and the z-filerbN)r6r7r8r9r*r:rBrCrDrMopenr$rL __setstate__)r-rEr*rFfr#rLr r r rs   zZNDArrayWrapper.readN)rGrHrIrJr.rr r r r rKsrKc@s<eZdZdZejjZd ddZddZd d Z d d Z dS) NumpyPicklera%A pickler to persist of big data efficiently. The main features of this object are: * persistence of numpy arrays in separate .npy files, for which I/O is fast. * optional compression using Zlib, with a special care on avoid temporaries. r Nc Cs||_|g|_||_||_|js.t|d|_nt|_d|_|dkrTtrNt j nt j }t j ||j|dy ddl}Wntk rd}YnX||_dS)Nwbr)protocol) _filename _filenames cache_sizer'rOfiler _npy_counterPY3pickleDEFAULT_PROTOCOLHIGHEST_PROTOCOLPicklerr.numpy ImportErrorr:)r-r*r'rXrUr:r r r r.s$    zNumpyPickler.__init__c Cs|js:|jj|||jj }ttjj|t ||d}n`|d7}|j \}}}t |d}t ||d|jdWdQRX|dd}t tjj|||}||fS)N)r,z.zrTr)r'rb)r'r:savedtype hasobjectr)r6r7basenametype __reduce__rOr(rK) r-rFr*r, container_rMrLzfiler r r _write_arrays      zNumpyPickler._write_arrayc Cs|jdk rt||jj|jj|jjfkr|j|j}|jrp||jt krpt||jjkrd|jj |}t j ||S|j d7_ y0d|j|j f}|j||\}}|jj|Wn2|j d8_ tdt|tjfYnXt j ||S)z Subclass the save method, to save ndarray subclasses in npy files, rather than pickling them. Of course, this is a total abuse of the Pickler class. Nrz %s_%02i.npyz"Failed to save %s to .npy file: %s)r:rgr@ZmatrixrAsizeitemsizer'rX_MEGAZasarrayr_rcrZrVrlrWappendprint traceback format_exc)r-objrmr*r r r rcs&    zNumpyPickler.savec Cs6|jr2t|jd}t||jj|jWdQRXdS)NrT)r'rOrVr(rYgetvalue)r-rkr r r closeszNumpyPickler.close)rrSN) rGrHrIrJr_dispatchcopyr.rlrcrvr r r r rRs    rRc@sReZdZdZejjZd ddZddZddZ e rDe ee j d <n e ee j <dS) NumpyUnpicklerz?A subclass of the Unpickler to unpickle our numpy pickles. Nc Csltjj||_tjj||_||_|j||_t j ||jy ddl }Wnt k r`d}YnX||_ dS)Nr)r6r7rfrVdirnamer9r3 _open_pickler Unpicklerr.r`rar:)r-r*rr3r:r r r r.(s   zNumpyUnpickler.__init__cCs|S)Nr )r-rr r r r{4szNumpyUnpickler._open_picklecCsPtj|t|jdtrL|jdkr,td|jj}|j|}|jj |dS)a This method is called to set the state of a newly created object. We capture it to replace our place-holder objects, NDArrayWrapper, by the array we are interested in. We replace them directly in the stack of pickler. rNz@Trying to unpickle an ndarray, but numpy didn't import correctlyrb) r| load_buildrstackr)r:rapoprrp)r-Znd_array_wrapperrFr r r r}7s    zNumpyUnpickler.load_buildr)N) rGrHrIrJr|rwrxr.r{r}r[r\BUILDr r r r ry#s  ryc@s eZdZdZddZddZdS)ZipNumpyUnpicklerzOA subclass of our Unpickler to unpickle on the fly from compressed storage.cCstj|||dddS)N)r3)ryr.)r-r*rr r r r.SszZipNumpyUnpickler.__init__cCs tt|S)N)rr$)r-rr r r r{XszZipNumpyUnpickler._open_pickleN)rGrHrIrJr.r{r r r r rOsrdc Cs|dkr d}t|ts*td|t|fz&t||||d}|j||jWddtkrzt|drz|j j |j jX|j S)aFast persistence of an arbitrary Python object into one or multiple files, with dedicated storage for numpy arrays. Parameters ----------- value: any Python object The object to store to disk filename: string The name of the file in which it is to be stored compress: integer for 0 to 9, optional Optional compression level for the data. 0 is no compression. Higher means more compression, but also slower read and write times. Using a value of 3 is often a good compromise. See the notes for more details. cache_size: positive number, optional Fixes the order of magnitude (in megabytes) of the cache used for in-memory compression. Note that this is just an order of magnitude estimate and that for big arrays, the code will go over this value at dump and at load time. protocol: positive int Pickle protocol, see pickle.dump documentation for more details. Returns ------- filenames: list of strings The list of file names in which the data is stored. If compress is false, each array is stored in a different file. See Also -------- joblib.load : corresponding loader Notes ----- Memmapping on load cannot be used for compressed files. Thus using compression can significantly slow down loading. In addition, compressed files take extra extra memory during dump and load. TrzUnicodeDecodeErrorr[r __cause__r?rrv)r*r3rrErtexcnew_excr r r r>s&     r>il)r)rrN)N)%rJr\rrsysr6r rstructcodecs_compatrior version_infor[ _Unpicklerr|_Picklerr_r strrrorrrrr$r(objectr)rKrRryrrr>r r r r s@     # (%`, @