toon-token-diff

Estimate how much converting JSON payloads to TOON format saves tokens across multiple LLM tokenizers. This package provides a reusable core library, an MCP tool wrapper, and a convenient CLI.

• Core library – estimateTokenDiff(jsonText, models) calculates token savings per supported model.
• Logging utility – estimateAndLog(jsonText, options) estimates and logs token savings to stdout/file for validation.
• MCP tool – estimate_toon_token_savings exposes the functionality inside a Model Context Protocol server.
• CLI – toon-token-diff lets you gather quick stats from the terminal.

The project consumes the official tokenizer implementations shipped on npm:

• @toon-format/toon for TOON serialization
• tiktoken for GPT-family token counts
• @anthropic-ai/tokenizer for Claude token counts

Refer to the source files in src/ for API documentation and usage examples.

• examples/libraryUsage.ts – load a JSON payload and compute token savings with the core library.
• examples/mcpUsage.ts – register the MCP tool and invoke it in-memory to gather token statistics.

Use estimateAndLog() to log token savings during MCP server operations or AI agent tool calls:

import { estimateAndLog } from 'toon-token-diff/core';

// Log to stdout only
await estimateAndLog(JSON.stringify(toolArgs), {
  models: ['claude'],
  stdout: true,
  label: 'tool_get_weather'
});

// Log to file in JSONL format
await estimateAndLog(JSON.stringify(toolArgs), {
  models: ['openai', 'claude'],
  file: './token-logs.jsonl',
  format: 'json',
  label: 'mcp_tool_call'
});

// Log to both stdout and file
await estimateAndLog(JSON.stringify(toolArgs), {
  models: ['claude'],
  stdout: true,
  file: './validation.log',
  format: 'text',
  label: 'agent_step_1'
});