A local-first app for evaluating document → JSON, text → JSON, and tool-call proposals with OpenAI-compatible providers or OpenRouter. Inspect inputs, outputs, scores, and execution logs side by side.
Requires Node.js 22+. From the repository directory:
npm install
npm run devOpen localhost:4173. Configure a provider, import a dataset, then launch an evaluation from Setup. Data is stored locally in .localevals/.
For an offline demo with synthetic data and mock model responses, run npm run demo and open localhost:4180.
Run CLI commands from the repository directory:
npm run localevals -- --help
npm run localevals -- run sample-data/manifest.jsonl sample-data/config.json --db .localevals/example.db
npm run localevals -- inspect --db .localevals/example.dbFor the synthetic evaluation above, start npm run mock in another terminal first.
The CLI command is now npm run localevals -- (formerly npm run evalforge --).
- Back up and move app data: portable backup, restore, and migration recovery.
- Prepare datasets — images, text, tool-call expectations, and sample data.
- Configure providers and models — local endpoints, OpenRouter, and model capabilities.
- Set up and run evaluations — prompts, schemas, grading, and CLI usage.
- Compare evaluation results — compare runs on the same dataset, inspect regressions, and export results.