Skip to content
intelPublic

About

Dokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

37 stars

Watchers

1 watching

Forks

Dokimi — A/B testing for agent context

docs codecov OpenSSF Scorecard

Find out whether your agent's skills, documentation and repository instructions improve its work. Dokimi runs the same questions or coding tasks under different context conditions, compares the outcomes, and helps you investigate the difference.

The same eval set runs on agent A as it is and on agent B with one change; the same grader scores both, and the difference with its p-value decides

Read the measured delta, uncertainty and task-level evidence together; one passing example does not establish a general benefit.

Assess with static checks, diagnose with novelty, reach and routing, change the skill or guidance, re-measure on the same tasks; keep a better result, reject none or worse with numbers, explain an unclear one, and reassess for a new model or harness

What can you find out?

Your question What to inspect
Does this context help? Controlled arms, models, harnesses and cost
Which parts of a skill matter? Knowledge probes, annotated passages, bundles and shortening
Will the agent use it? Discovery, routing, tool preference and instruction loading
Does repository guidance work? Static claims, scopes, executable tasks and rule evidence
Do the tests cover the work? Task creation, coverage and grader controls
Can the grades be trusted? Scoring methods, judge checks and grounding
What did the agent do? Context use, execution paths and process contracts
Will it keep working? Reports, baselines, CI and development directions

Full feature tree: 39 individual cards · Documentation

Each card states what works today and where it stops. Open PRs, partial workflows and research directions are included, with links, so they are discoverable without being advertised as released functionality.

Which parts of a skill help: novelty. Dokimi asks the model about each claim in a skill twice, without the skill and with it, and a different model grades both answers. Every claim lands in one of four outcomes, per model (more):

A 2x2 grid. Wrong without, right with: added, keep it. Right without and with: redundant, cut it for this model. Wrong both times: still missing, rewrite it. Right without, wrong with: degraded, fix it first

Choose your starting point

Skill author or reviewer · Repository maintainer · Evaluator or researcher · Agent / CI operator · Contributor

Start with a free check

From a checkout, with Python 3.11 or newer:

cd dokimi
pip install -e .
dokimi repo scan /path/to/your/repository

Replace the path with the repository you want to inspect. This static scan does not call an LLM. To check a skill collection instead:

dokimi catalog check --root /path/to/your/catalog --kind skills

Then follow reviewing a skill or repository for the next step and its cost. Agent experiments need the selected provider or logged-in harness; executable coding tasks also need their supported container environment. Check harness support before choosing a runner.

Read the evidence

A skill can teach something new without improving a task, and an instruction can exist without being loaded. Dokimi separates these questions. Knowledge novelty suggests edits to test; it does not mean familiar content is safe to delete. Grader controls help expose blind spots; they do not prove a verifier is infallible.

For methods and recorded studies, see what Dokimi measures, methodology and skill novelty evidence. Results belong to the evaluated artifact, task set, model, harness and grader.

Coding agents: use reviewing.md to evaluate content, and AGENTS.md to change Dokimi. See also contributing, the CLI reference and the project vision.

License

Licensed under the Apache License, Version 2.0; see NOTICE for attribution.

About

Dokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

37 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages