vad

Embeddable realtime voice activity detection and end-of-turn detection for Node and Bun.

A small, dependency-light TypeScript library that wraps two ONNX models:

• VAD — stream audio frames and get per-input speaking / not-speaking state. Powered by Silero VAD v5.
• End-of-Turn (EOT) — given a short chat transcript, get the probability the user has finished their turn. Powered by the LiveKit turn-detector.

The library does not bundle model weights. You point each detector at local ONNX files that you've downloaded ahead of time — no network access is required at runtime when the paths are local.

npm i @toma.com/vad
# or
bun add @toma.com/vad

Requires Node ≥ 18 or Bun ≥ 1.3.

Download once from HuggingFace (or vendor them into your repo / container image):

# Silero VAD v5
curl -L -o silero_vad.onnx \
  https://huggingface.co/onnx-community/silero-vad/resolve/main/onnx/model.onnx

# LiveKit turn-detector (model + tokenizer)
mkdir -p turn-detector
curl -L -o turn-detector/model_quantized.onnx \
  https://huggingface.co/livekit/turn-detector/resolve/main/model_quantized.onnx
curl -L -o turn-detector/tokenizer.json \
  https://huggingface.co/livekit/turn-detector/resolve/main/tokenizer.json
curl -L -o turn-detector/tokenizer_config.json \
  https://huggingface.co/livekit/turn-detector/resolve/main/tokenizer_config.json
curl -L -o turn-detector/special_tokens_map.json \
  https://huggingface.co/livekit/turn-detector/resolve/main/special_tokens_map.json

import { RealtimeVAD } from "@toma.com/vad";

const vad = new RealtimeVAD({
  modelPath: "/abs/path/to/silero_vad.onnx",
  format: "pcm_16000",        // or "mulaw_8000"
  frameDurationMs: 80,
  inputs: [
    {
      label: "default",
      positiveSpeechThreshold: 0.5,
      negativeSpeechThreshold: 0.2,
      lookbackFrames: 5,
    },
  ],
});

// Feed raw PCM (or mu-law) frames as Buffers as they arrive.
const result = await vad.onMessage(audioBuffer);
if (result) {
  for (const out of result.outputs) {
    console.log(out.label, out.isSpeaking);
  }
}

await vad.destroy();

You can also drive the loop yourself with push(buffer, timestamp) followed by await run() if you want to decouple ingestion from inference.

Define multiple entries in inputs to get independent detectors over the same audio (e.g. a sensitive detector and a strict detector running in parallel).

{
  start: 1717000000000,   // earliest sample timestamp processed in this batch
  end:   1717000000123,   // wall-clock time when processing finished
  latency: 7,             // ms spent in inference
  outputs: [{ label: "default", isSpeaking: true }],
}

import { EndOfTurnDetector } from "@toma.com/vad";

const eot = new EndOfTurnDetector({
  modelPath: "/abs/path/to/turn-detector/model_quantized.onnx",
  tokenizerPath: "/abs/path/to/turn-detector", // directory with tokenizer.json
  queueDepth: 3,                                // optional load-shedding limit
});

const { endOfTurnProbability } = await eot.detect({
  context: [
    { role: "user",      content: "what time do you open" },
    { role: "assistant", content: "we open at 9am"        },
    { role: "user",      content: "and on weekends"       },
  ],
});

await eot.destroy();

A value near 1 means the user has likely finished their turn; near 0 means they are likely to continue. If queueDepth is set and the inference queue is saturated, detect() sheds load and returns a neutral 0.5 without blocking.

Methods:

• onMessage(buffer: Buffer): Promise<VADOutputType | undefined> — push audio + run inference in one call. Convenience entry-point. Resolves to undefined when there aren't enough buffered samples to fill a frame yet.
• push(buffer: Buffer, originTimestamp: number): void — append audio to the internal buffer without running inference.
• run(): Promise<Array<{ label: string; isSpeaking: boolean; originTimestamp: number }>> — slice all buffered audio into frames and emit per-label state.
• destroy(): Promise<void> — release the ONNX session and clear internal buffers.

Methods:

• detect(input: { context: ChatMessage[] }, signal?: AbortSignal): Promise<{ endOfTurnProbability: number }> — returns the EOT probability. Returns a neutral 0.5 if signal aborts or if queueDepth is exceeded.
• destroy(): Promise<void> — release the ONNX session.

• src/services/realtime.ts buffers incoming audio, slices it into fixed-duration frames, runs each frame through Silero, applies per-label thresholds with a lookback window, and emits speaking / not-speaking state.
• src/services/endOfTurn.ts folds and normalizes the chat context, applies the turn-detector chat template, runs the ONNX model, and softmaxes the final token's logits to read off the end-of-turn token probability.
• src/utils/models/base.ts lazily creates one shared InferenceSession per model behind a semaphore (onnxruntime sessions are not safe to run concurrently), with optional bounded-queue load shedding.

This library does not bundle or redistribute any model weights. You bring your own copy of each model and it remains under its original license. All credit for the models goes to their authors.

The original implementation was written by hand by the maintainers and later refined with the help of AI tooling — refactoring into a reusable library, tightening tests, and improving the documentation.

bun install
bun run typecheck
bun run lint
bun test
bun run build

Maintainers: @anthonykrivonos (lead), @ray-cj-huang.

MIT © Toma