Skip to content

Tags: bettongia/lexical

Tags

0.1.0-dev.2

Toggle 0.1.0-dev.2's commit message

Verified

This commit was signed with the committer’s verified signature.
dedickinson Duncan Dickinson
Extend Stemmer to all 28 Snowball languages; re-export OffsetTokenizer

Stemmer previously wired up only English, throwing ArgumentError for
every other locale, even though the underlying snowball_stemmer
dependency already implements 28 languages. Extend the factory to map
all of them (excluding the generic `porter` variant, not a distinct
language). Each new mapping was verified against the actual stemmer
output for a representative inflected word per language, not just
type-checked.

Also bump the betto_icu dependency to 0.1.0-dev.2 and re-export its
new OffsetTokenizer/TokenSpan types, needed by kmdb's vault chunker
for position-aware tokenisation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

0.1.0-dev.1

Toggle 0.1.0-dev.1's commit message

Verified

This commit was signed with the committer’s verified signature.
dedickinson Duncan Dickinson
Update CLAUDE.md: correct language count and add web test to layout

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>