3 :>"^4@s\dZddlmZddlZddlZddlZddlZddlZddlZddl m Z yddl m Z Wn ek r|ddl mZ YnXejdksejd ksejdkrejd kreZn dd l mZy(dd lmZddlmZddlmZWnek rdZYnXejdddZddZGdddeZ GdddeZ!Gddde Z"Gddde"Z#dS) aCode for more fancy file handles. Classes: - UndoHandle File object decorator with support for undo-like operations. Additional private classes used in Bio.SeqIO and Bio.SearchIO for indexing files are also defined under Bio.File but these are not intended for direct use. )print_functionN) basestring)UserDict) DictMixinZPyPyZCPython) OrderedDict)dbapi2)IntegrityError)OperationalErrorrcksttdr.ytj|}Wntk r,YnXt|trtjddkrZd|krZ|jdd}d|krt j ||f| }|VWdQRXqt ||f| }|VWdQRXn|VdS)aContext manager to ensure we are using a handle. Context manager for arguments that can be passed to SeqIO and AlignIO read, write, and parse methods: either file objects or path-like objects (strings, pathlib.Path instances, or more generally, anything that can be handled by the builtin 'open' function). When given a path-like object, returns an open file handle to that path, with provided mode, which will be closed when the manager exits. All other inputs are returned, and are *not* closed. Arguments: - handleish - Either a file handle or path-like object (anything which can be passed to the builtin 'open' function: str, bytes, and under Python >= 3.6, pathlib.Path, os.DirEntry) - mode - Mode to open handleish (used only if handleish is a string) - kwargs - Further arguments to pass to open(...) Examples -------- >>> from Bio import File >>> import os >>> with File.as_handle('seqs.fasta', 'w') as fp: ... # Python2/3 docstring workaround, revise for 'Python 3 only'. ... _ = fp.write('>test\nACGT') ... >>> fp.closed True >>> handle = open('seqs.fasta', 'w') >>> with File.as_handle(handle) as fp: ... # Python 2/3 docstring workaround, revise for 'Python 3 only'. ... _ = fp.write('>test\nACGT') ... >>> fp.closed False >>> fp.close() >>> os.remove("seqs.fasta") # tidy up Note that if the mode argument includes U (for universal new lines) this will be removed under Python 3 where is is redundant and has been deprecated (this happens automatically in text mode). fspathrrUencodingN) hasattrosr TypeError isinstancersys version_inforeplacecodecsopen)Z handleishmodekwargsfprX/home/psgendb/BIRCHDEV/lib-linux-x86_64/python/lib64/python3.5/site-packages/Bio/File.py as_handle9s1   r cCst|d}|jd}|jd|dkrddlm}y|jd|dStk r}z&dt|ksbt|j td WYd d }~XnX|S) zOpen a file in binary mode, spot if it is BGZF format etc (PRIVATE). This functionality is used by the Bio.SeqIO and Bio.SearchIO index and index_db functions. If the file is gzipped but not BGZF, a specific ValueError is raised. Zrbrs)bgzf)rZfileobjZBGZFz[Gzipped files are not suitable for indexing, please use BGZF (blocked gzip format) instead.N) rreadseekrr#Z BgzfReader ValueErrorstrAssertionErrorclose)filenamehandlemagicr#errr_open_for_random_access~s    r.c@seZdZdZddZddZddZejdd kr:d d Z d d Z ddZ d#ddZ ddZ ddZddZddZddZddZdd Zd!S)$ UndoHandlezA Python handle that adds functionality for saving lines. Saves lines in a LIFO fashion. Added methods: - saveline Save a line to be returned next time. - peekline Peek at the next line without consuming it. c Cs2||_g|_y |j|_Wntk r,YnXdS)zInitialize the class.N)_handle_savedZurlAttributeError)selfr+rrr__init__s  zUndoHandle.__init__cCs|S)z#Iterate over the lines in the File.r)r3rrr__iter__szUndoHandle.__iter__cCs|j}|st|S)zReturn the next line.)readline StopIteration)r3nextrrr__next__szUndoHandle.__next__rrcCs|jS)z8Python 2 style alias for Python 3 style __next__ method.)r9)r3rrrr8szUndoHandle.nextcOs|j|jj||}g|_|S)z6Read all the lines from the file as a list of strings.)r1r0 readlines)r3argskeywdslinesrrrr:szUndoHandle.readlinescOs&|jr|jjd}n|jj||}|S)z+Read the next line from the file as string.r)r1popr0r6)r3r;r<linerrrr6szUndoHandle.readliner"cCs|dkr$dj|j}g|jdd<nd}xz|dkr|jrt|jd|krn|t|jd}||jjd}q*||jdd|}|jd|d|jd<d}q*W||jj|S)zRead the File.r"rNr)joinr1lenr>r0r$)r3sizeZsavedrrrr$s zUndoHandle.readcCs|r|g|j|_dS)zStore a line in the cache memory for later use. This acts to undo a readline, reflecting the name of the class: UndoHandle. N)r1)r3r?rrrsavelineszUndoHandle.savelinecCs*|jr|jd}n|jj}|j||S)zJReturn the next line in the file, but do not move forward though the file.r)r1r0r6rD)r3r?rrrpeeklines    zUndoHandle.peeklinecCs|jjtdd|jDS)zKReturn the current position of the file read/write pointer within the File.css|]}t|VqdS)N)rB).0r?rrr sz"UndoHandle.tell..)r0tellsumr1)r3rrrrHszUndoHandle.tellcGsg|_|jj|dS)z1Set the current position at the offset specified.N)r1r0r%)r3r;rrrr%szUndoHandle.seekcCs t|j|S)zReturn File attribute.)getattrr0)r3attrrrr __getattr__szUndoHandle.__getattr__cCs|S)zACall special method when opening the file using a with-statement.r)r3rrr __enter__szUndoHandle.__enter__cCs|jjdS)zACall special method when closing the file using a with-statement.N)r0r))r3typevalue tracebackrrr__exit__szUndoHandle.__exit__Nr@)r@)__name__ __module__ __qualname____doc__r4r5r9rrr8r:r6r$rDrErHr%rLrMrQrrrrr/s     r/c@s(eZdZdZddZddZddZdS) _IndexedSeqFileProxya3Base class for file format specific random access (PRIVATE). This is subclasses in both Bio.SeqIO for indexing as SeqRecord objects, and in Bio.SearchIO for indexing QueryResult objects. Subclasses for each file format should define '__iter__', 'get' and optionally 'get_raw' methods. cCs tddS)zReturn (identifier, offset, length in bytes) tuples. The length can be zero where it is not implemented or not possible for a particular file format. zSubclass should implement thisN)NotImplementedError)r3rrrr5sz_IndexedSeqFileProxy.__iter__cCs tddS)z$Return parsed object for this entry.zSubclass should implement thisN)rW)r3offsetrrrgetsz_IndexedSeqFileProxy.getcCs tddS)zReturn the raw record from the file as a bytes string (if implemented). If the key is not found, a KeyError exception is raised. This may not have been implemented for all file formats. z#Not available for this file format.N)rW)r3rXrrrget_raw sz_IndexedSeqFileProxy.get_rawN)rRrSrTrUr5rYrZrrrrrVsrVc@seZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ e e drnddZddZddZddZddZd2ddZd d!Zd"d#Zd$d%Zd3d&d'Zd(d)Zd*d+Zd4d,d-Zd.d/Zd0d1ZdS)5_IndexedSeqFileDicta5Read only dictionary interface to a sequential record file. This code is used in both Bio.SeqIO for indexing as SeqRecord objects, and in Bio.SearchIO for indexing QueryResult objects. Keeps the keys and associated file offsets in memory, reads the file to access entries as objects parsing them on demand. This approach is memory limited, but will work even with millions of records. Note duplicate keys are not allowed. If this happens, a ValueError exception is raised. As used in Bio.SeqIO, by default the SeqRecord's id string is used as the dictionary key. In Bio.SearchIO, the query's id string is used. This can be changed by supplying an optional key_function, a callback function which will be given the record id and must return the desired key. For example, this allows you to parse NCBI style FASTA identifiers, and extract the GI number to use as the dictionary key. Note that this dictionary is essentially read only. You cannot add or change values, pop values, nor clear the dictionary. c s||_|_||_||_r0fdd|D}n|}t}x<|D]4\}}} ||krl|jjjtd|q@|||<q@W||_dS)zInitialize the class.c3s"|]\}}}|||fVqdS)Nr)rFkol) key_functionrrrGNsz/_IndexedSeqFileDict.__init__..zDuplicate key '%s'N) _proxy _key_function_repr _obj_repr_dictr0r)r&_offsets) r3random_access_proxyr_reprZobj_repr offset_iterZoffsetskeyrXlengthr)r_rr4Ds   z_IndexedSeqFileDict.__init__cCs|jS)z2Return a string representation of the File object.)rb)r3rrr__repr__csz_IndexedSeqFileDict.__repr__cCs&|rdt|jd|jfSdSdS)z2Create a string representation of the File object.z{%r : %s(...), ...}rz{}N)listkeysrc)r3rrr__str__gsz_IndexedSeqFileDict.__str__cCs ||jkS)z2Return key if contained in the offsets dictionary.)re)r3rirrr __contains__osz _IndexedSeqFileDict.__contains__cCs t|jS)zReturn the number of records.)rBre)r3rrr__len__ssz_IndexedSeqFileDict.__len__ccs&x |jD]}||j|fVq WdS)zIterate over the (key, SeqRecord) items. This tries to act like a Python 3 dictionary, and does not return a list of (key, value) pairs due to memory concerns. N)r5 __getitem__)r3rirrritemswsz_IndexedSeqFileDict.itemsccs"x|jD]}|j|Vq WdS)zIterate over the SeqRecord items. This tries to act like a Python 3 dictionary, and does not return a list of value due to memory concerns. N)r5rq)r3rirrrvaluessz_IndexedSeqFileDict.valuescCs|jS)zIterate over the keys. This tries to act like a Python 3 dictionary, and does not return a list of keys due to memory concerns. )r5)r3rrrrmsz_IndexedSeqFileDict.keys iteritemsccs"x|jD]}|j|Vq WdS)z"Iterate over the SeqRecord) items.N)r5rq)r3rirrr itervaluessz_IndexedSeqFileDict.itervaluesccs&x |jD]}||j|fVq WdS)z(Iterate over the (key, SeqRecord) items.N)r5rq)r3rirrrrtsz_IndexedSeqFileDict.iteritemscCs|jS)zIterate over the keys.)r5)r3rrriterkeyssz_IndexedSeqFileDict.iterkeyscCs t|jS)zIterate over the keys.)iterre)r3rrrr5sz_IndexedSeqFileDict.__iter__cCsH|jj|j|}|jr&|j|j}n|j}||krDtd||f|S)z$Return record for the specified key.zKey did not match (%s vs %s))r`rYreraidr&)r3rirecordkey2rrrrqsz_IndexedSeqFileDict.__getitem__Nc Cs$y |j|Stk r|SXdS)zReturn the value in the dictionary. If the key (k) is not found, this returns None unless a default (d) is specified. N)rqKeyError)r3r\drrrrYs z_IndexedSeqFileDict.getcCs|jj|j|S)zReturn the raw record from the file as a bytes string. If the key is not found, a KeyError exception is raised. )r`rZre)r3rirrrrZsz_IndexedSeqFileDict.get_rawcCs tddS)zWould allow setting or replacing records, but not implemented. Python dictionaries provide this method for modifying data in the dictionary. This class mimics the dictionary interface but is read only. z(An indexed a sequence file is read only.N)rW)r3rirOrrr __setitem__sz_IndexedSeqFileDict.__setitem__cOs tddS)zWould allow adding more values, but not implemented. Python dictionaries provide this method for modifying data in the dictionary. This class mimics the dictionary interface but is read only. z(An indexed a sequence file is read only.N)rW)r3r;rrrrupdatesz_IndexedSeqFileDict.updatecCs tddS)zWould remove specified record, but not implemented. Python dictionaries provide this method for modifying data in the dictionary. This class mimics the dictionary interface but is read only. z(An indexed a sequence file is read only.N)rW)r3ridefaultrrrr>sz_IndexedSeqFileDict.popcCs tddS)zWould remove and return a SeqRecord, but not implemented. Python dictionaries provide this method for modifying data in the dictionary. This class mimics the dictionary interface but is read only. z(An indexed a sequence file is read only.N)rW)r3rrrpopitemsz_IndexedSeqFileDict.popitemcCs tddS)zWould clear dictionary, but not implemented. Python dictionaries provide this method for modifying data in the dictionary. This class mimics the dictionary interface but is read only. z(An indexed a sequence file is read only.N)rW)r3rrrclearsz_IndexedSeqFileDict.clearcCs tddS)zWould return a new dictionary with keys and values, but not implemented. Python dictionaries provide this method for modifying data in the dictionary. This class mimics the dictionary interface but is read only. z0An indexed a sequence file doesn't support this.N)rW)r3rmrOrrrfromkeyssz_IndexedSeqFileDict.fromkeyscCs tddS)zWould copy a dictionary, but not implemented. Python dictionaries provide this method for modifying data in the dictionary. This class mimics the dictionary interface but is read only. z0An indexed a sequence file doesn't support this.N)rW)r3rrrcopysz_IndexedSeqFileDict.copycCs|jjjdS)a?Close the file handle being used to read the data. Once called, further use of the index won't work. The sole purpose of this method is to allow explicit handle closure - for example if you wish to delete the file, on Windows you must first close all open handles to that file. N)r`r0r))r3rrrr)sz_IndexedSeqFileDict.close)N)N)N)rRrSrTrUr4rkrnrorprrrsrmrdictrurtrvr5rqrYrZr}r~r>rrrrr)rrrrr[+s2       r[c@s~eZdZdZdddZddZddZd d Zd d Zd dZ ddZ e e drXddZ ddZdddZddZddZdS)_SQLiteManySeqFilesDictaKRead only dictionary interface to many sequential record files. This code is used in both Bio.SeqIO for indexing as SeqRecord objects, and in Bio.SearchIO for indexing QueryResult objects. Keeps the keys, file-numbers and offsets in an SQLite database. To access a record by key, reads from the offset in the appropriate file and then parses the record into an object. There are OS limits on the number of files that can be open at once, so a pool are kept. If a record is required from a closed file, then one of the open handles is closed first. c Cstsddlm}|d|dk r(t|}||_||_||_||_||_||_ ||_ i|_ t j jt j j||_t j j|r|jn|jdS)zInitialize the class.r)MissingPythonDependencyErrorz/Requires sqlite3, which is included Python 2.5+N)_sqliteBiorrl_index_filename _filenames_formatra_proxy_factoryrb _max_open_proxiesrpathabspathdirname_relative_pathisfile _load_index _build_index) r3index_filename filenames proxy_factoryformatr_rgmax_openrrrrr4s"    z _SQLiteManySeqFilesDict.__init__cCs|j}|j}|j}|j}|j}tj|dd}||_yP|jddj \}t ||_ |j dkrl|j t d|jdj \}|j t |kr|j t dt ||j f|jddj \|_|r||jkr|j t d |j|fy"|jddj \}|jd k}Wntk r d}YnXd d|jdjD|_|rtjjtjj|}g} xJ|jD]@} tjj| r| j| n | jtjj|| jdtjjqbW| |_~ |rt|t|jkr|j t dt|jt|f|r||jkrxt|j|D]t\} } tjj| tjj| kr |j |r\t dtjj| tjj| fn t dtjj| tjj| fq WWn8tk r} z|j t d| WYdd} ~ XnX||js|j t d|jdS)z9Call from __init__ to re-use an existing index (PRIVATE).F)Zcheck_same_threadz(SELECT value FROM meta_data WHERE key=?;countr"zUnfinished/partial databasez%SELECT MAX(_ROWID_) FROM offset_data;z#Corrupt database? %i entries not %irz!Index file says format %s, not %sfilenames_relative_to_indexZTRUEcSsg|] }|dqS)rr)rFrowrrr isz7_SQLiteManySeqFilesDict._load_index..z0SELECT name FROM file_data ORDER BY file_number;/z Index file says %i files, not %iz1Index file has different filenames, e.g. %r != %rzIndex file has different filenames [This is an old index where any relative paths were relative to the original working directory]. e.g. %r != %rz"Not a Biopython index database? %sNzUnsupported format '%s')rr@)r)r)rrrrrrconnect_conexecutefetchoneint_lengthr)r&upperrfetchallrrrrisabsappendrArseprBzip_OperationalError)r3r relative_pathrrrconrrZtmpfoldnewerrrrrr<s        && z#_SQLiteManySeqFilesDict._load_indexcs|j}|j}|j}|j}|j|j}|j}|j}| s<| rHtd|||s\td|t j |}||_ |j d|j d|j d|j dd|j dd |f|j dd|j d |j d d} xdt |D]V\} tjj| } tjj|  r(tjj| r(tjj| |jtjjd} n\tjjtjj| tjjj|tjjrtjj| |jtjjd} | jd st| |j d| f||| } rfdd| D} nfdd| D} x>ttj| d}|sP|jd||j| t|7} qWt||kr$| |<q| jjqW| |_ y|j dWnFt!k r}z(||_|j|jtd|WYdd}~XnX|j d|j d| df|jdS)z3Call from __init__ to create a new index (PRIVATE).z2Filenames to index and format required to build %rzUnsupported format '%s'zPRAGMA synchronous=OFFzPRAGMA locking_mode=EXCLUSIVEz.CREATE TABLE meta_data (key TEXT, value TEXT);z0INSERT INTO meta_data (key, value) VALUES (?,?);rr"rrTruez8CREATE TABLE file_data (file_number INTEGER, name TEXT);zYCREATE TABLE offset_data (key TEXT, file_number INTEGER, offset INTEGER, length INTEGER);rrz../z7INSERT INTO file_data (file_number, name) VALUES (?,?);c3s$|]\}}}|||fVqdS)Nr)rFr\r]r^)ir_rrrGsz7_SQLiteManySeqFilesDict._build_index..c3s |]\}}}|||fVqdS)Nr)rFr\r]r^)rrrrGsdzIINSERT INTO offset_data (key,file_number,offset,length) VALUES (?,?,?,?);z@CREATE UNIQUE INDEX IF NOT EXISTS key_index ON offset_data(key);zDuplicate key? %sNzPRAGMA locking_mode=NORMALz-UPDATE meta_data SET value = ? WHERE key = ?;r@)rr@)rr)"rrrrrarrrr&rrrr enumeraterrrrrelpathrrr startswithr(rl itertoolsZisliceZ executemanyZcommitrBr0r)r_IntegrityError)r3rrrrrrZrandom_access_proxiesrrr*rrfrhZbatchrr)rr_rrs                   z$_SQLiteManySeqFilesDict._build_indexcCs|jS)N)rb)r3rrrrksz _SQLiteManySeqFilesDict.__repr__cCst|jjd|fjS)Nz(SELECT key FROM offset_data WHERE key=?;)boolrrr)r3rirrrrosz$_SQLiteManySeqFilesDict.__contains__cCs|jS)z%Return the number of records indexed.)r)r3rrrrpsz_SQLiteManySeqFilesDict.__len__ccs(x"|jjdD]}t|dVqWdS)zIterate over the keys.z9SELECT key FROM offset_data ORDER BY file_number, offset;rN)rrr')r3rrrrr5sz _SQLiteManySeqFilesDict.__iter__rtcCsdd|jjdjDS)zIterate over the keys. This tries to act like a Python 3 dictionary, and does not return a list of keys due to memory concerns. cSsg|]}t|dqS)r)r')rFrrrrrsz0_SQLiteManySeqFilesDict.keys..zSELECT key FROM offset_data;)rrr)r3rrrrm sz_SQLiteManySeqFilesDict.keysc Cs|jjd|fj}|st|\}}|j}||krB||j|}nFt||jkrb|jdj j |j |j |j |}|j|}|||<|jr|j|j}n|j}||krtd||f|S)z$Return record for the specified key.z8SELECT file_number, offset FROM offset_data WHERE key=?;r"zKey did not match (%s vs %s))rrrr{rrYrBrrr0r)rrrrarxr&) r3rir file_numberrXproxiesryproxyrzrrrrqs(  z#_SQLiteManySeqFilesDict.__getitem__Nc Cs$y |j|Stk r|SXdS)zReturn the value in the dictionary. If the key (k) is not found, this returns None unless a default (d) is specified. N)rqr{)r3r\r|rrrrY/s z_SQLiteManySeqFilesDict.getc Cs|jjd|fj}|st|\}}}|j}||krf|rV||j}|j||j|S||j|Sndt ||j kr|j djj |j |j|j|}|||<|r|j}|j||j|S|j|SdS)zReturn the raw record from the file as a bytes string. If the key is not found, a KeyError exception is raised. z@SELECT file_number, offset, length FROM offset_data WHERE key=?;r"N)rrrr{rr0r%r$rZrBrrr)rrr) r3rirrrXrjrhrrrrrZ:s,       z_SQLiteManySeqFilesDict.get_rawcCs&|j}x|r |jdjjqWdS)zClose any open file handles.r"N)rrr0r))r3rrrrr)_sz_SQLiteManySeqFilesDict.close)r)N)rRrSrTrUr4rrrkrorpr5rrrmrqrYrZr)rrrrr s  "Xa   %r)rr)rr)r )$rU __future__rrrrZ contextlibrplatformZ Bio._py3krZ collectionsrZ _dict_base ImportErrorrrZpython_implementationrrdr Zsqlite3r rr rr rZcontextmanagerr r.objectr/rVr[rrrrrsB          Dn#_