Lib.rs

› Keywords #bpe #llm #openai #tokenize #python-bindings #hugging-face

#tiktoken

  1. llm-tokenizer

    LLM tokenizer library with caching and chat template support

    v1.3.2 169K #chat-template #llm #hugging-face #tiktoken

  2. tiktoken

    A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer

    v3.1.4 1.7K #bpe #openai #tokenizer

  3. miktik

    A unified, multi-backend tokenizer library for LLMs

    v0.2.0 460 #tiktoken #hugging-face #llm #rust #tokenize #tokenizer

  4. bamboo-compression

    Compression utilities for Bamboo sessions and memory workflows

    v2026.5.17 120 #artificial-intelligence #token-budget #llm #bamboo #workflow #model-context #tiktoken #hybrid #summarization #heuristics

  5. riptoken

    Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken

    v0.3.0 #tiktoken #bpe #llm #nlp

  6. mkcontext

    that provides functionality for creating context

    v0.7.3 800 #context #globs #run-command #limit #exclude #tiktoken #glob-pattern #chatgpt #clipboard #command-line-tool

  7. skimtoken

    Fast token count estimation library

    v0.2.2 180 #token-count #python-bindings #estimation #parameters #memory #startup-time #statistical-model #tiktoken

  8. blazen-llm

    LLM provider abstraction layer for the Blazen workflow engine

    v0.5.3 100 #json-schema #llm-provider #model-provider #workflow-engine #blazen #agentic #openai #fal-ai #anthropic #tiktoken

  9. tiktoken-wasm

    WASM bindings for the tiktoken BPE tokenizer

    v3.2.4 #tiktoken #bpe #wasm-bindings #model-info #model-name #string-encoding #model-provider #wasm-memory #pricing

  10. toktrie_tiktoken

    tiktoken (OpenAI BPE) library support for toktrie and llguidance

    v1.7.4 #structured-output #tiktoken #openai #tokenize #llguidance #bpe #toktrie #json-schema #context-free-grammar #llama

  11. toklab-core

    Pure-Rust core for toklab: bulk tokenizer + counter for OpenAI BPE encodings

    v0.1.1 #bpe #openai #tiktoken #llm

  12. loctok

    Count LOC (lines of code) & TOK (LLM tokens), fast

    v0.2.0 #loc #llm #tiktoken #lines-of-code #token

  13. wordchipper-cli-util

    Wordchipper CLI

    v0.9.1 #tokenize #wordchipper #lexer #cli #python-bindings #bpe #tiktoken #openai #gpt-2

  14. rustbpe

    A BPE (Byte Pair Encoding) tokenizer written in Rust with Python bindings

    v0.1.0 #bpe #python-bindings #tiktoken #training #gpt-4 #tokenizer

  15. tiktokenx

    A high-performance Rust implementation of OpenAI's tiktoken library

    v0.1.0 #tiktoken #openai #bpe

  16. tokenmonster

    Greedy tiktoken-like tokenizer with embedded vocabulary (cl100k-base approximator)

    v0.1.0 #tokenize #tiktoken #nlp #tokenizer

  17. chat-splitter

    Never exceed OpenAI's chat models' maximum number of tokens when using the async_openai Rust crate

    v0.1.1 #chat #openai #chat-completion #max #split #exceed #tiktoken

    18. Try searching with DuckDuckGo.

  18. tokin

    Experimental fast tokenizer

    v0.1.0 #word-piece #tokenize #nlp #tiktoken #bpe #tokenizer