fuzzy_dl_owl2.fuzzydl.parser.tokenizer.tokens

A high-performance tokenizer for the FuzzyDL language that bridges Python and a C-based lexer using CFFI and memory mapping.

Description

The software implements a bridge between Python and a low-level C lexer, specifically designed to parse FuzzyDL syntax efficiently. By utilizing C Foreign Function Interface (CFFI), it delegates the heavy lifting of lexical analysis to C functions while managing memory and data structures within Python. A two-pass tokenization strategy is employed where the first pass counts the tokens to allocate exact-sized NumPy arrays, and the second pass populates these arrays with token types, start offsets, and lengths. This design supports both memory-mapped file inputs for large ontologies and in-memory byte strings, ensuring that the underlying C backend receives NUL-terminated buffers regardless of the source. The resulting token stream is encapsulated in a class that allows lazy decoding of source text and provides context management to safely release system resources.

Attributes

AVAILABLE_LEXER
AVAILABLE_LEXER
COMMA
EQ
ERROR
FADD
FDIV
FIRST_KEYWORD
FMUL
FSUB
GE
IDENT
LBRACE
LBRACK
LE
LPAREN
MINUS
NUMBER
PLUS
RBRACE
RBRACK
RPAREN
STAR
TOK_NAMES
T_EOF
T_IDENT
_PAGE

Classes

Tokens	Flat token stream. Holds the buffer so span offsets stay valid.
_Buffer	Read-only file view with a guaranteed NUL at index size (needed by the
_BytesBuffer	In-memory counterpart of _Buffer for tokenizing a source string

Functions

_tokenize_buffer(→ Tokens)	Two-pass tokenization over any buffer exposing cdata / size.
get_tokens(→ Tokens)	Tokenizes the file at path via mmap. No whole-file Python str is
get_tokens_from_bytes(→ Tokens)	Tokenizes an in-memory source already decoded to UTF-8 bytes (e.g. a

Module Contents

UML Class Diagram for Tokens

class Tokens(types, starts, lens, buf)

Flat token stream. Holds the buffer so span offsets stay valid.

__enter__()

Enters a context manager over the token stream and returns it, so the stream can be used in a with block that guarantees the backing buffer is released on exit.

Returns:

This token stream instance.

Return type:

Tokens

__exit__(*exc)

Exits the context manager, releasing the backing buffer via close() regardless of whether the with block exited normally or because of an exception. Any exception is allowed to propagate.

Parameters:

exc (Any) – The exception type, value, and traceback (unused), present to satisfy the context-manager protocol.

__len__()

Returns the number of tokens in the stream, i.e. the length of the underlying types array.

Returns:

The token count.

Return type:

int

close()

Releases the source buffer backing this token stream by delegating to its close method. After this call the lazy text lookups are no longer valid, since the byte storage the span offsets point into has been released.

is_keyword(i)

Returns whether token i is a keyword. This is a pure integer code comparison, so no string work is performed.

Parameters:

i (int) – Index of the token in the stream.

Returns:

True if the token is a keyword, False otherwise.

Return type:

bool

is_name(i)

Returns whether token i is a user-defined bare identifier (as opposed to a keyword or structural token).

Parameters:

i (int) – Index of the token in the stream.

Returns:

True if the token is a bare identifier, False otherwise.

Return type:

bool

name(i)

Returns the display name of token i: the structural token name for punctuation, or the keyword spelling itself for keyword tokens.

Parameters:

i (int) – Index of the token in the stream.

Returns:

The display name of the token.

Return type:

str

text(i)

Returns the raw source text of token i, lazily decoded from the backing buffer via the span offset and length. Only the requested token is decoded, so repeated lookups are cheap.

Parameters:

i (int) – Index of the token in the stream.

Returns:

The UTF-8 decoded source text of the token.

Return type:

str

__slots__ = ('types', 'starts', 'lens', '_buf')

_buf

lens

starts

types

UML Class Diagram for _Buffer

class _Buffer(path)

Read-only file view with a guaranteed NUL at index size (needed by the re2c backend; harmless for flex). mmap zero-fills the final partial page for free; only a page-multiple file size needs a one-off copy with a NUL.

close()

Releases the resources held by this buffer. The exported memoryview is released and the CFFI pointer is dropped before the underlying mmap (if any) is closed, ensuring no live pointer references the memory map when it is unmapped. Safe to call once after tokenization is complete.

__slots__ = ('size', 'obj', 'cdata', 'view', '_mm')

cdata

view

UML Class Diagram for _BytesBuffer

class _BytesBuffer(data: bytes)

In-memory counterpart of _Buffer for tokenizing a source string that is already resident (a parser chunk). Appends the NUL sentinel the re2c backend needs; exposes the same cdata / view / size / close surface as _Buffer so Tokens is agnostic to the source.

close()

Releases the resources held by this in-memory buffer. The exported memoryview is released and the CFFI pointer is dropped so no live pointer references the byte string afterwards. Safe to call once after tokenization is complete.

__slots__ = ('size', 'obj', 'cdata', 'view')

cdata

obj

size

view

_tokenize_buffer(buf) → Tokens

Two-pass tokenization over any buffer exposing cdata / size. Pass 1 counts tokens in C so the parallel int32 arrays can be sized exactly; Pass 2 fills them. Shared by get_tokens() (mmap) and get_tokens_from_bytes() (in-memory) so the count/fill logic lives in one place.

Parameters:

buf (Any) – A buffer object (_Buffer or _BytesBuffer) exposing cdata and size for the C scanner.

Returns:

A Tokens object holding the parallel token arrays.

Return type:

Tokens

get_tokens(path: str) → Tokens

Tokenizes the file at path via mmap. No whole-file Python str is built, so this is the fast / low-memory entry point for large ontologies. The returned Tokens owns the mapping; call .close() (or use it as a context manager) to release it.

Parameters:

path (str) – Filesystem path of the file to tokenize.

Returns:

The token stream backed by a memory-mapped buffer.

Return type:

Tokens

get_tokens_from_bytes(data: bytes) → Tokens

Tokenizes an in-memory source already decoded to UTF-8 bytes (e.g. a parser chunk). Same Tokens output as get_tokens(), without a file or mmap.

Parameters:

data (bytes) – The UTF-8 encoded source bytes to tokenize.

Returns:

The token stream backed by an in-memory buffer.

Return type:

Tokens

AVAILABLE_LEXER: bool = False

AVAILABLE_LEXER = True

COMMA = 7

EQ = 126

ERROR = 10

FADD = 93

FDIV = 96

FIRST_KEYWORD = 57

FMUL = 95

FSUB = 94

GE = 125

IDENT = 9

LBRACE = 5

LBRACK = 3

LE = 124

LPAREN = 1

MINUS = 122

NUMBER = 8

PLUS = 121

RBRACE = 6

RBRACK = 4

RPAREN = 2

STAR = 123

TOK_NAMES

T_EOF = 131

T_IDENT = 9

_PAGE = 4096