Skip to content
qubixalPublic

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

25 Commits

Folders and files

Repository files navigation

WAIFMARK 2 Benchmarking Suite

Banner

PyPI Python Version

A benchmark that tests the local agentic and roleplay (persona) capabilities of small (V)LLMs.

As Waifmark benchmarks small, locally-hosted models, cost is not a major concern. Instead, Waifmark 2's scores are evaluated from 0.00 to 100.00 against time per response (total toks ouput / avg. tok/s).

Waifmark has now been updated to v2! A lot of things have been changed from v1, and so their scores are not directly comparable (see Leaderboard).

Why v2?

  • 4x the question bank compared to v1
  • Full automated benchmarking process (that launches vllm/llama.cpp server, serves the local model and benchmarks it in the same control center)
  • Ability to test models consecutively, saving setup time
  • LLM judge that auto-scores tasks, saving review time. Low confidence evals will still be flagged for a human to audit.

v2 consists of two main types of questions:

  • Agentic (i.e. short-context toolcalling, strict formatted JSON output, lookup tables, safe shell cmds, code correction)
  • Persona (i.e. multi-turn conversational context, personality persistance, safety, 中/EN vocabulary and expression)

Note: Waifmark's exact test bank is proprietary and is not released for the public.

Install & Boot

Download the latest waifmark-macos.zip from Releases, unzip, and run:

./waifmark/waifmark              # launches frontend at http://127.0.0.1:8001
./waifmark/waifmark --stop       # shut down

To shut down:

waifmark --stop

Env for API keys

cp .env.example .env
# then, nano into the file or open a code editor to enter your
# OPENROUTER_API_KEY=sk-or-…, HF_TOKEN=hf_…

Headless CLI

If for example you would like to run consecutive benchmarks, or don't want to use the frontend UI:

python main.py --config config.yaml --test-bank data/test_bank.json

Then add --mode pipeline to start/stop the backend (vLLM, llama.cpp) which serves the model.

Scoring Judge

To speed up the scoring process, you would need to set an LLM judge.

By default, this is done through Openrouter:

  1. get an api key from Openrouter
  2. set model in config.yaml under judges to the judge model you want (e.g. openrouter/free)
  3. point model_under_test at your model or GGUF file.

Scoring (v2)

Agentic (1/3 of test bank)

Waifmark first puts the model in a sandbox to perform agentic tasks, recording and checking each step's tool call. There are three components that make up the final score:

score_100 = 100 × ( tool_syntax_accuracy × 0.3
                  + goal_completion        × 0.5
                  + error_recovery         × 0.2 )
  • tool_syntax_accuracy — the percentage of toolcalls whose JSON was valid
  • goal_completion — a score derived from the final-answer hit rate (70%) and required-tool usage (30%).
  • error_recovery — a corrected toolcall after the previous one failed.

Note: Small models are sometimes sensitive to chat templates. If your benchmark scores look unusually bad, check your vLLM --chat-template and config.yaml chat_template.

Roleplay (2/3 of test bank)

Then, the model is evaluated based on 4 components:

character_consistency, instruction_following, trap_resistance, conversational_quality.

To determine the final Roleplay score. Responses may be flagged for human review on judge disagreement, low confidence (<0.55), judge errors, etc.

Note:

The agentic testing is conducted through a harness, not a sandbox. Tasks are run locally on your machine, and the model-under-test can only invoke allowed shell commands. This includes python3 by default, so set agentic.allow_python3_tool: false if you don't trust it.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages