Commits

Branch selector

 main

User selector

All users

All time

Commit History

Commits on Jun 10, 2025

• chore: bump rust version in release dockerfile

  daulet

  committed
  b262445

  Copy full SHA for b262445

• chore: cargo update

  daulet

  committed
  ea08c40

  Copy full SHA for ea08c40

• chore: bump version of the lib

  daulet

  committed
  0f43680

  Copy full SHA for 0f43680

• fix: prefix symbols with tokenizers_

  daulet

  committed
  c645dab

  Copy full SHA for c645dab

Commits on Mar 25, 2025

• Migrating to MODULE.bazel to make it easier for other bazel projects to import. (#33)

  Show description for 452e497
  

  mattlgy

  and

  daulet

  authored
  452e497

  Copy full SHA for 452e497

• fix: gh actions build (#34)

  daulet

  authored
  d177abc

  Copy full SHA for d177abc

Commits on Nov 7, 2024

• fix: update example README

  daulet

  committed
  8c32990

  Copy full SHA for 8c32990

• fix: update example deps

  daulet

  committed
  8e366e1

  Copy full SHA for 8e366e1

• fix: bazel build

  daulet

  committed
  be6cf35

  Copy full SHA for be6cf35

• fix: README example

  daulet

  committed
  5fa02af

  Copy full SHA for 5fa02af

• chore: bump libc dep

  daulet

  committed
  e97da0c

  Copy full SHA for e97da0c

• feat: FromPretrained to load tokenizer directly from HF (#27)

  Show description for 9c972d9
  

  3 peopleauthored
  9c972d9

  Copy full SHA for 9c972d9

Commits on Nov 5, 2024

• fix: bump versions everywhere

  daulet

  committed
  0d469f8

  Copy full SHA for 0d469f8

• fix: bump lib version otherwise bazel gets it confused with hf's tokenizers

  daulet

  committed
  37d1c00

  Copy full SHA for 37d1c00

• feat: add better message when version mismatch detected

  daulet

  committed
  fb1262d

  Copy full SHA for fb1262d

• chore: update tokenizers version

  daulet

  committed
  6457cc5

  Copy full SHA for 6457cc5

Commits on Aug 9, 2024

• fix: update example to latest release

  daulet

  committed
  eafe2ce

  Copy full SHA for eafe2ce

• fix: release image

  daulet

  committed
  f5ef49c

  Copy full SHA for f5ef49c

• Update to huggingface/tokenizers v0.20.0 (#23)

  Show description for 89e504d
  daulet

  authored
  89e504d

  Copy full SHA for 89e504d

• feat: add option to retrieve offsets from tokenizer (#21)

  Show description for 7bb47dd
  riccardopinosio

  authored
  7bb47dd

  Copy full SHA for 7bb47dd

Commits on Jul 14, 2024

• Add tiktoken vs tokenizers bench

  

  Daulet Zhanguzin

  committed
  d9aff87

  Copy full SHA for d9aff87

Commits on Jul 10, 2024

• improve rust bench to exclude drop time

  

  Daulet Zhanguzin

  committed
  fce4681

  Copy full SHA for fce4681

• fix Bazel build

  

  Daulet Zhanguzin

  committed
  14835c0

  Copy full SHA for 14835c0

• add decode benchmarks for Rust piece to measure CGO overhead

  

  Daulet Zhanguzin

  committed
  21d1792

  Copy full SHA for 21d1792

Commits on Jul 9, 2024

• record bench results

  

  Daulet Zhanguzin

  committed
  8b4f0ce

  Copy full SHA for 8b4f0ce

• upgrade to 0.19.1

  

  Daulet Zhanguzin

  committed
  217df12

  Copy full SHA for 217df12

• update benchmarks before upgrade to 0.19.1

  

  Daulet Zhanguzin

  committed
  a735742

  Copy full SHA for a735742

• update README with linker param requirement

  

  Daulet Zhanguzin

  committed
  ba0db9d

  Copy full SHA for ba0db9d

Commits on Jun 12, 2024

• Update to allow for platform dependent libs in CGO (#18)

  Show description for d503b5b
  jmoney

  authored
  d503b5b

  Copy full SHA for d503b5b

Commits on Apr 10, 2024

• expose encodeSpecialTokens functionality

  

  Daulet Zhanguzin

  committed
  3a615e6

  Copy full SHA for 3a615e6

• update to tokenizers v0.15.2

  

  Daulet Zhanguzin

  committed
  b14c306

  Copy full SHA for b14c306

Commits on Jan 7, 2024

• exclude release from bazel build as it is not compatible

  

  Daulet Zhanguzin

  committed
  8531b2b

  Copy full SHA for 8531b2b

• fix cross compilation: libc::c_char is platform dependent

  

  Daulet Zhanguzin

  committed
  6e58579

  Copy full SHA for 6e58579

Commits on Nov 19, 2023

• add contribution README

  

  Daulet Zhanguzin

  committed
  4fd5a66

  Copy full SHA for 4fd5a66

Commits on Nov 15, 2023

• support more attributes from the Encoding structure (#5)

  Show description for 38a9a14
  

  clems4ever

  and

  Clément Michaud

  authored
  38a9a14

  Copy full SHA for 38a9a14