fuzzy_dl_owl2.fuzzydl.parser.tokenizer.tokenizer_handler

Tokenizer backends and registry for the fuzzy-DL parser.

All tokenizers produce the same (kind, value, lower/raw, offset) 4-tuple format with a trailing T_EOF sentinel, so the recursive-descent parser is agnostic to which backend ran.

Usage:

from tokenizer_handler import TokenizerHandler
handler = TokenizerHandler()
tokens = handler.tokenize(source_string)

Manages a registry of tokenizer backends for the fuzzy-DL parser, providing a unified interface that abstracts over pure-Python and compiled C implementations.

Description

The software provides a flexible tokenization system designed to feed a recursive-descent parser with a consistent stream of 4-tuples containing token kind, value, normalized text, and offset. It implements a strategy pattern where a central handler inspects the runtime environment to select the most efficient backend, prioritizing compiled C extensions for speed while falling back to a pure Python regex implementation if necessary. To support the specific syntax of fuzzy-DL, the logic includes specialized handling for splitting compound identifiers on arithmetic operators and converts numeric literals into appropriate Python types. For large inputs, the architecture supports streaming by breaking source files into chunks based on top-level form boundaries, ensuring that memory usage remains bounded even when processing multi-gigabyte files.

Attributes

Token
_FDL_CODE_TO_TK
_HANDLER
_PROTECTED_KW
_SPLIT_CHARS
_SPLIT_NUM_RE
_STREAM_THRESHOLD
_TOKEN_RE
_fdl_ok

Classes

BaseTokenizer	Abstract base for all fuzzy-DL tokenizers.
FdlFileTokenizer	re2c/flex backend for files (uses mmap + streaming).
FdlStringTokenizer	re2c/flex backend for in-memory strings. Fastest when built.
PythonRegexTokenizer	Pure-Python regex tokenizer — always available, the ultimate fallback.
TokenizerHandler	Selects and dispatches to the fastest available tokenizer backend.

Functions

_emit_split_ident(→ None)	Splits a compound identifier on * and + operators and appends the
_iter_form_chunks(→ Iterator[str])	Yields substrings of src, each a run of whole top-level forms bounded by
_tokenize_best(→ List[Token])	Returns the parser's 4-tuple token stream using the fastest available

Module Contents

UML Class Diagram for BaseTokenizer

class BaseTokenizer

Abstract base for all fuzzy-DL tokenizers.

available() → bool

Returns whether this tokenizer backend can be used in the current process (e.g. its compiled extension is importable).

Returns:

True if the backend is usable, False otherwise.

Return type:

bool

abstractmethod tokenize(source: str) → List[Token]

Tokenizes the source string into the parser’s 4-tuple stream. Each concrete backend must implement this method.

Parameters:

source (str) – The fuzzy-DL source text to tokenize.

Raises:

NotImplementedError – always, because this is an abstract base method.

Returns:

The parser’s 4-tuple token stream, terminated by an EOF token.

Return type:

List[Token]

property name: str

Returns a human-readable name for this tokenizer backend. The base implementation falls back to the concrete class name; subclasses override it with a stable short identifier used in logging and backend selection.

Returns:

The backend’s display name.

Return type:

str

UML Class Diagram for FdlFileTokenizer

class FdlFileTokenizer

Bases: BaseTokenizer

re2c/flex backend for files (uses mmap + streaming).

static _form_batches(toks: fuzzy_dl_owl2.fuzzydl.parser.tokenizer.tokens.Tokens, batch_min: int = _STREAM_THRESHOLD)

Yields (lo, hi) index ranges covering whole top-level forms within the token array. Parenthesis depth is tracked so each batch ends at a form boundary; batches are yielded only once they reach at least batch_min tokens.

Parameters:

• toks (Tokens) – The token array to partition into form batches.

• batch_min (int) – Minimum token count before a batch is yielded.

available() → bool

Returns whether this backend can be used, i.e. whether at least one of its compiled entry points (the Cython streaming scanner or the CFFI mmap tokenizer) was successfully imported at construction.

Returns:

True if a compiled file tokenizer is available, False otherwise.

Return type:

bool

tokenize_file(path: str) → Iterator[List[Token]]

Tokenizes a file into batched 4-tuple token streams, streaming for large files so peak memory stays bounded. When the Cython FdlScan is available it reads the whole file and yields scan batches; otherwise the CFFI mmap-backed tokenizer is used and its raw token arrays are converted form-by-form via _form_batches().

Parameters:

path (str) – Path to the fuzzy-DL source file to tokenize.

Raises:

RuntimeError – if no compiled file tokenizer backend is available.

Returns:

An iterator yielding batches of parser 4-tuples.

Return type:

Iterator[List[Token]]

_FdlScan: Any | None = None

_available: bool = False

_get_tokens: Callable | None = None

property name: str

Returns the stable identifier "fdl-file" for this file-based re2c/flex backend.

Returns:

The backend’s display name.

Return type:

str

UML Class Diagram for FdlStringTokenizer

class FdlStringTokenizer

Bases: BaseTokenizer

re2c/flex backend for in-memory strings. Fastest when built.

static _tokens_to_parser(toks: fuzzy_dl_owl2.fuzzydl.parser.tokenizer.tokens.Tokens, src_len: int) → List[Token]

Converts a Tokens object (the CFFI int32-array output) into the parser’s 4-tuple stream. Each token’s raw source text is lazily decoded from the backing buffer via the span offsets stored in the token arrays. Keyword codes are remapped to T_IDENT with their lowercase spelling; structural codes are remapped via _FDL_CODE_TO_TK; numbers are parsed into int or float. A trailing EOF token is appended.

Parameters:

• toks (Tokens) – The CFFI token array produced by the re2c/flex backend.

• src_len (int) – The total byte length of the original source, used as the EOF token’s offset.

Returns:

The parser’s 4-tuple token stream, terminated by an EOF token.

Return type:

List[Token]

available() → bool

Returns whether this backend can be used, i.e. whether at least one of its compiled entry points (the Cython tuple builder or the CFFI byte tokenizer) was successfully imported at construction.

Returns:

True if a compiled string tokenizer is available, False otherwise.

Return type:

bool

tokenize(source: str) → List[Token]

Tokenizes the source string into the parser’s 4-tuple stream using the fastest available compiled backend. The text is UTF-8 encoded and, when present, handed to the Cython tokenize_tuples (which builds the tuples in C and the timing is logged); otherwise the CFFI byte tokenizer is used and its raw token arrays are converted with _tokens_to_parser before the underlying buffer is released. A RuntimeError is raised if neither entry point was built.

Parameters:

source (str) – The fuzzy-DL source text to tokenize.

Raises:

RuntimeError – if no compiled string tokenizer backend is available.

Returns:

The parser’s 4-tuple token stream.

Return type:

List[Token]

_available: bool = False

_get_tokens_from_bytes: Callable[[bytes], Any] | None = None

_tokenize_tuples: Callable[[bytes], List[Token]] | None = None

property name: str

Returns the stable identifier "fdl-string" for this in-memory re2c/flex backend.

Returns:

The backend’s display name.

Return type:

str

UML Class Diagram for PythonRegexTokenizer

class PythonRegexTokenizer

Bases: BaseTokenizer

Pure-Python regex tokenizer — always available, the ultimate fallback.

tokenize(source: str) → List[Token]

Tokenizes the source string into the parser’s 4-tuple stream using the pure-Python master regex. Each match is classified into identifiers, numbers, brackets/braces, and quoted strings; identifiers that contain split characters and are not protected keywords are further broken up via _emit_split_ident, while numbers are converted to int or float. A terminating EOF token is appended. This backend is always available and serves as the ultimate fallback when the compiled scanners are absent.

Parameters:

source (str) – The fuzzy-DL source text to tokenize.

Returns:

The parser’s 4-tuple token stream, terminated by an EOF token.

Return type:

List[Token]

property name: str

Returns the stable identifier "python-regex" for this pure-Python fallback backend.

Returns:

The backend’s display name.

Return type:

str

UML Class Diagram for TokenizerHandler

class TokenizerHandler

Selects and dispatches to the fastest available tokenizer backend.

Preference order (fastest first):

1. FdlStringTokenizer – re2c/flex C backend

2. CRegexTokenizer – hand-written C tokenizer

3. PythonRegexTokenizer – pure-Python fallback (always works)

tokenize(source: str) → List[Token]

Tokenizes an in-memory source string by delegating to the active backend’s tokenize method, returning the parser’s 4-tuple token stream.

Parameters:

source (str) – The fuzzy-DL source text to tokenize.

Returns:

The parser’s 4-tuple token stream.

Return type:

List[Token]

tokenize_file(path: str) → Iterator[List[Token]]

File path entry point. Uses FdlFileTokenizer when its compiled backend is available (streaming for large files); otherwise reads the whole file into a string and delegates to the active in-memory backend.

Parameters:

path (str) – Path to the fuzzy-DL source file.

Returns:

An iterator yielding batches of parser 4-tuples.

Return type:

Iterator[List[Token]]

_backends: List[BaseTokenizer]

_best: BaseTokenizer

property best: BaseTokenizer

Returns the active tokenizer backend chosen at construction, i.e. the fastest one that reported itself available.

Returns:

The selected tokenizer backend.

Return type:

BaseTokenizer

property best_name: str

Returns the display name of the active tokenizer backend, useful for logging which scanner the parser actually selected.

Returns:

The name of the selected backend.

Return type:

str

_emit_split_ident(raw: str, offset: int, out: List[Token]) → None

Splits a compound identifier on * and + operators and appends the resulting tokens to the output list. Numeric substrings are converted to int or float; alphabetic substrings become identifier tokens; the operators themselves are emitted as separate T_IDENT tokens.

Parameters:

• raw (str) – The compound identifier string to split.

• offset (int) – The byte offset of the original identifier in the source.

• out (List[Token]) – The token list to append split results to.

_iter_form_chunks(src: str, chunk_bytes: int = 4 * 1024 * 1024) → Iterator[str]

Yields substrings of src, each a run of whole top-level forms bounded by matching parentheses. Strings and line comments are skipped during parenthesis counting. Chunks are yielded once they reach at least chunk_bytes in size so the parser can stream large files without materialising the entire source at once.

Parameters:

• src (str) – The fuzzy-DL source text to chunk.

• chunk_bytes (int) – Minimum byte size of a chunk before it is yielded.

Returns:

An iterator over whole-form source chunks.

Return type:

Iterator[str]

_tokenize_best(source: str) → List[Token]

Returns the parser’s 4-tuple token stream using the fastest available backend, via the module-level TokenizerHandler singleton.

Parameters:

source (str) – The fuzzy-DL source text to tokenize.

Returns:

The parser’s 4-tuple token stream.

Return type:

List[Token]

Token

_FDL_CODE_TO_TK: Dict[int, int | None]

_HANDLER

_PROTECTED_KW: FrozenSet[str]

_SPLIT_CHARS: str = '*+'

_SPLIT_NUM_RE

_STREAM_THRESHOLD: int = 1048576

_TOKEN_RE

_fdl_ok: bool