NLP tokenizers written in Go language

Self-contained Japanese Morphological Analyzer written in pure Go

A multilingual command line sentence tokenizer in Golang

Lex machinary for go.

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

Object mapping for golang.

Tokenizer (lexer) for golang

OpenShield is a new generation security layer for AI models

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

XML Tokenizer is a low-memory high performance non-namespace parser library for parsing simple XML 1.0.

SQL like 'select' interface for files

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

Tokenizers and lemmatizers for Go

Meet Programming Language Interpreter

A Text Tokenizer library for Golang

The shamoji (杓文字) is a word filtering package

✂️ OpenAI's tiktoken tokenizer written in Go

🙅 Go package for detecting and removing stopwords from text.