estnltk-rs

EstNLTK regex subsystems rewritten in Rust using the resharp DFA regex engine, exposed to Python via PyO3.

• resharp provides guaranteed linear-time matching, native lookaround, and boolean operations (intersection, complement)
• The RegexTagger is the most self-contained regex-dependent component in EstNLTK — pattern matching → conflict resolution → annotation assembly

The project is a Cargo workspace of focused crates:

Pure-Rust users can depend on individual crates (e.g., estnltk-core + estnltk-taggers) without PyO3. Python users get everything through the estnltk-python crate, which exposes the estnltk_regex_rs module.

# Install maturin
pip install maturin

# Build and install the Python module
cd estnltk-regex-rs
maturin develop

from estnltk_regex_rs import RsRegexTagger, rs_regex_tag

# Class-based API
tagger = RsRegexTagger(
    patterns=[
        {"pattern": r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+",
         "attributes": {"type": "email"}, "group": 0, "priority": 0},
    ],
    output_layer="regexes",
    output_attributes=["type"],
    conflict_resolver="KEEP_MAXIMAL",  # KEEP_ALL, KEEP_MINIMAL, *_EXCEPT_PRIORITY variants
    lowercase_text=False,
)
result = tagger.tag("Contact bla@bla.ee for info")
# {'name': 'regexes', 'attributes': ['type'], 'ambiguous': True,
#  'spans': [{'base_span': (8, 18), 'annotations': [{'type': 'email'}]}]}

# Convenience function
spans = rs_regex_tag("Hello 123", [{"pattern": r"[0-9]+", "attributes": {"type": "number"}}])
# [{'base_span': (6, 9), 'annotations': [{'type': 'number'}]}]

# Rust unit + integration tests (all workspace crates)
cargo test --workspace

# Cross-implementation tests (requires estnltk installed)
pytest cross_tests/ -v

All benchmarks verify output parity — both implementations produce identical spans.

Notes:

• Speedup is end-to-end including PyO3 serialization overhead (Python dict construction on return). Pure Rust throughput is higher.
• The RegexTagger speedup is modest at medium scale because resharp DFA compilation is a one-time cost amortized over matching, and the Python regex library is itself a C extension. The gap widens at larger scales.
• SubstringTagger shows consistently higher speedup (3.6–3.7x) because the Rust Aho-Corasick implementation has lower per-match overhead than Python's.
• Benchmarks run with cargo bench (Criterion) and python benchmarks/rust_vs_python/bench_*.py.

Both implementations wrap the same C++ Vabamorf library. The benchmark measures binding overhead (PyO3 vs SWIG) and data marshalling differences.

Performance is near-identical since both call the same C++ code. The Rust port's value is not speed but integration: morphological expansion feeds directly into SubstringTagger without crossing the Python boundary.

• Tagger decorators — RegexTagger, SubstringTagger, SpanTagger, and PhraseTagger produce static annotations only. The grammar tagger supports decorators, validators, and scoring callbacks natively. For the other taggers, decorators can be applied Python-side on the output.
• Grammar tagger Python bindings — The estnltk-grammar crate is fully functional from Rust but not yet exposed via PyO3 bindings.
• Leftmost-longest semantics — resharp uses leftmost-longest (not leftmost-first). cat|catfish matches "catfish" in resharp vs "cat" in Python re.
• No lazy quantifiers — .*? not supported. Use character class negation or lookahead instead.