3 [/@s`ddlmZddlZejeeeedddlmZddlmZm Z m Z eZ GdddeZ dS) )absolute_importN)BOLEOLEOF NOT_FOUND)Errors)rrrc@sdeZdZdZdddZddZdd Zd d Zd d ZddZ ddZ ddZ dddZ ddZ dS)Scannera A Scanner is used to read tokens from a stream of characters using the token set specified by a Plex.Lexicon. Constructor: Scanner(lexicon, stream, name = '') See the docstring of the __init__ method for details. Methods: See the docstrings of the individual methods for more information. read() --> (value, text) Reads the next lexical token from the stream. position() --> (name, line, col) Returns the position of the last token read using the read() method. begin(state_name) Causes scanner to change state. produce(value [, text]) Causes return of a token value to the caller of the Scanner. NcCsd|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ ||_ ||_ ||_ g|_d|_|jdd|_d|_d|_t|_d|_|dk r|d|d |_|_dS)a Scanner(lexicon, stream, name = '') |lexicon| is a Plex.Lexicon instance specifying the lexical tokens to be recognised. |stream| can be a file object or anything which implements a compatible read() method. |name| is optional, and may be the name of the file being scanned or any other identifying string. rr rN)tracebuffer buf_start_posnext_poscur_poscur_line start_pos start_line start_coltext state_namelexiconstreamnamequeue initial_statebegincur_line_startrcur_char input_state)selfrrr initial_posr"d/home/psgendb/BIRCHDEV/lib-linux-x86_64/python/lib64/python3.6/site-packages/Cython/Plex/Scanners.py__init__Is0  zScanner.__init__cCsl|j}xR|sX|j\|_}|dkr6|jd|jq|j||j}|dk r|j|qW|d}|d=|S)a" Read the next lexical token from the stream and return a tuple (value, text), where |value| is the value associated with the token as specified by the Lexicon, and |text| is the actual string read from the stream. Returns (None, '') on end of file. Nr)r scan_a_tokenrproduceeofperform)r ractionvalueresultr"r"r#readqs  z Scanner.readcCs|j|_|j|_|j|j|_|j}|dk rp|jrJtd||j|jf|j |j|j |j|j }||fS|j|jkr|j t kr|j |j dks|j tkrdStj||jdS)z Read the next input sequence recognised by the machine and return (text, action). Returns ('', None) on end of file. Nz"Scanner: read: Performing %s %d:%dr )r N)rrrrrrrun_machine_inlinedr printr rrr next_charrrUnrecognizedInputr)r r)rr"r"r#r%s$   zScanner.scan_a_tokencCs|j}|j}|j}|j}|j}|j}|j}|j}|j} t |} d\} } } }}}}|j }x|rzt d|d||t |f|d}|dk r|||||||f\} } } }}}}|}|j |t}|tkr|o|j d}|r|rt d|d|}|d kr|}|| }|| kr ||}|d 7}nh|j| }|jjd }|j|d|}||_| |7} | |_t |} ||8}|r||}|d 7}nd}|d krt}d }n|st}d }n|}nP|d krd }d}n<|dkr|d 7}|}}t}d }n|d krt}d}nd}qZ|rt d| dk r>| | | ||||f\}}}}}}}nd}PqZW||_||_||_||_||_||_|r|dk rt d||S)z; Inlined version of run_machine for speed. Nrr zState %d, %d/%d:%s -->numberr)elsezState %dri r blockedzDoing %s)Nrrrr rr)rrrrrrrr rlenr r.reprgetrrrr,rrr)r staterrrrrrr rbuf_lenb_action b_cur_pos b_cur_lineb_cur_line_start b_cur_char b_input_state b_next_posr r)c new_state buf_indexdiscarddatar"r"r#r-s               zScanner.run_machine_inlinedcCs|j}|jr tdd ||jf|dkrh|j|_|j}|dkrNt|_d|_q|s`t|_d|_q||_nd|dkr~d|_d|_nN|dkr|jd7_|j|_ |_t |_d|_n|dkrt |_d |_nd |_|jrtd ||j|jfdS) NzScanner: next: %s [%d] %d rr3r r4r5r6r z--> [%d] %d %rz ) rr r.rr read_charrrrrrr)r rrDr"r"r#r/ s6zScanner.next_charcCs|j|j|jfS)am Return a tuple (name, line, col) representing the location of the last token read using the read() method. |name| is the name that was provided to the Scanner constructor; |line| is the line number in the stream (1-based); |col| is the position within the line of the first character of the token (0-based). )rrr)r r"r"r#position)s zScanner.positioncCs|jS)zOPython accessible wrapper around position(), only for error reporting. )rL)r r"r"r# get_position4szScanner.get_positioncCs|jj||_||_dS)z8Set the current state of the scanner to the named state.N)rget_initial_staterr)r rr"r"r#r9sz Scanner.begincCs"|dkr|j}|jj||fdS)a Called from an action procedure, causes |value| to be returned as the token value from read(). If |text| is supplied, it is returned in place of the scanned text. produce() can be called more than once during a single call to an action procedure, in which case the tokens are queued up and returned one at a time by subsequent calls to read(), until the queue is empty, whereupon scanning resumes. N)rrappend)r r*rr"r"r#r&?s zScanner.producecCsdS)z_ Override this method if you want something to be done at end of file. Nr")r r"r"r#r'Nsz Scanner.eof)r N)N)__name__ __module__ __qualname____doc__r$r,r%r-r/rLrMrr&r'r"r"r"r#r s (j  r ) __future__rcythondeclareobjectr rRegexpsrrrrr r"r"r"r# s