Skip to content

Tags: huggingface/tokenizers

Tags

v0.23.1

Toggle v0.23.1's commit message
Bump version to 0.23.1

v0.23.1rc0

Toggle v0.23.1rc0's commit message
ci(node-release): create missing npm/<triple> dirs before artifacts

`yarn artifacts` (napi-cli v3 `napi artifacts`) writes each per-target
.node file into `npm/<triple>/`. The repo has hand-written templates
for most triples but darwin-universal was added to `napi.targets`
without one, so the publish step failed with:

  ENOENT: no such file or directory, open
  '.../bindings/node/npm/darwin-universal/tokenizers.darwin-universal.node'

Add `napi create-npm-dirs` before `yarn artifacts` — it scans
`napi.targets` and creates any missing dirs (with a generated
package.json) from scratch. Idempotent for triples that already have
hand-written templates.

node-v0.23.1

Toggle node-v0.23.1's commit message
node: scope platform packages under @huggingface/

Set napi.packageName so per-platform sub-packages publish as
`@huggingface/tokenizers-<triple>` (e.g. @huggingface/tokenizers-darwin-x64)
instead of unscoped `tokenizers-<triple>`.

The unscoped names on npm are owned by `n1t0` (Anthony Moi) — the org
CI npm token has no publish access there, which is why the 0.23.1
release attempt 404'd on the very first sub-package. The unscoped
sub-packages were never successfully published from CI; the only
versions on the registry (0.13.4-rc1/rc2) were hand-published by
n1t0 and the 0.13.3 stable parent shipped with optionalDependencies
that 404 at install (npm tolerates it for optionalDependencies).

Parent package stays `tokenizers` — public-facing import name is
unchanged. napi regenerates the runtime loader to require the
scoped names.

Also delete the 13 checked-in `bindings/node/npm/<triple>/` template
files. They hardcoded the old unscoped name and version 0.13.4-rc1
and were misleading as source. CI regenerates them via `napi
create-npm-dirs` (already in node-release.yml line 213).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v0.23.0rc0

Toggle v0.23.0rc0's commit message
ci(node-release): drop removed --name flag from napi universal

@napi-rs/cli removed the --name option; the name is now read from the
napi.name field in package.json. Replace with --dir ./artifacts so the
command picks up the downloaded per-target .node files.

Also pins the second PyO3/maturin-action reference (inside the build
job) that was missed by the earlier action-modernization commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v0.22.2

Toggle v0.22.2's commit message
fix max build?

v0.22.2rc0

Toggle v0.22.2rc0's commit message
add .lock and v0.22.2

v0.22.1

Toggle v0.22.1's commit message
push the minor

v0.22.1rc0

Toggle v0.22.1rc0's commit message
update version to 0.22.1.rc0

v0.22.0rc0

Toggle v0.22.0rc0's commit message
push cargos