Self-contained Japanese Morphological Analyzer written in pure Go

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

A multilingual command line sentence tokenizer in Golang

Lex machinary for go.

NLP tokenizers written in Go language

Tokenizer (lexer) for golang

Tokenizers and lemmatizers for Go

OpenShield is a new generation security layer for AI models

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes.

Object mapping for golang.

XML Tokenizer is a low-memory high performance non-namespace parser library for parsing simple XML 1.0.

SQL like 'select' interface for files

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

✂️ OpenAI's tiktoken tokenizer written in Go

Package assocentity returns the mean distance from tokens to an entity and its synonyms

Smart code context extractor for AI assistants

The shamoji (杓文字) is a word filtering package