A benchmark that tests the local agentic and roleplay (persona) capabilities of small (V)LLMs.
As Waifmark benchmarks small, locally-hosted models, cost is not a major concern. Instead, Waifmark 2's scores are evaluated from 0.00 to 100.00 against time per response (total toks ouput / avg. tok/s).
Waifmark has now been updated to v2! A lot of things have been changed from v1, and so their scores are not directly comparable (see Leaderboard).
- 4x the question bank compared to v1
- Full automated benchmarking process (that launches vllm/llama.cpp server, serves the local model and benchmarks it in the same control center)
- Ability to test models consecutively, saving setup time
- LLM judge that auto-scores tasks, saving review time. Low confidence evals will still be flagged for a human to audit.
v2 consists of two main types of questions:
- Agentic (i.e. short-context toolcalling, strict formatted JSON output, lookup tables, safe shell cmds, code correction)
- Persona (i.e. multi-turn conversational context, personality persistance, safety, 中/EN vocabulary and expression)
Note: Waifmark's exact test bank is proprietary and is not released for the public.
Download the latest waifmark-macos.zip from Releases, unzip, and run:
./waifmark/waifmark # launches frontend at http://127.0.0.1:8001
./waifmark/waifmark --stop # shut downTo shut down:
waifmark --stopcp .env.example .env
# then, nano into the file or open a code editor to enter your
# OPENROUTER_API_KEY=sk-or-…, HF_TOKEN=hf_…If for example you would like to run consecutive benchmarks, or don't want to use the frontend UI:
python main.py --config config.yaml --test-bank data/test_bank.jsonThen add --mode pipeline to start/stop the backend (vLLM, llama.cpp) which serves the model.
To speed up the scoring process, you would need to set an LLM judge.
By default, this is done through Openrouter:
- get an api key from Openrouter
- set
modelinconfig.yamlunderjudgesto the judge model you want (e.g.openrouter/free) - point
model_under_testat your model or GGUF file.
Waifmark first puts the model in a sandbox to perform agentic tasks, recording and checking each step's tool call. There are three components that make up the final score:
score_100 = 100 × ( tool_syntax_accuracy × 0.3
+ goal_completion × 0.5
+ error_recovery × 0.2 )
tool_syntax_accuracy— the percentage of toolcalls whose JSON was validgoal_completion— a score derived from the final-answer hit rate (70%) and required-tool usage (30%).error_recovery— a corrected toolcall after the previous one failed.
Note: Small models are sometimes sensitive to chat templates. If your benchmark scores look unusually bad, check your vLLM
--chat-templateandconfig.yamlchat_template.
Then, the model is evaluated based on 4 components:
character_consistency, instruction_following, trap_resistance, conversational_quality.
To determine the final Roleplay score. Responses may be flagged for human review on judge disagreement, low confidence (<0.55), judge errors, etc.
The agentic testing is conducted through a harness, not a sandbox. Tasks are run locally on your machine, and the model-under-test can only invoke allowed shell commands. This includes
python3by default, so setagentic.allow_python3_tool: falseif you don't trust it.