jev-spec

🌐 日本語 | 简体中文 | 한국어

Catch spec drift on every commit. jev-spec checks your code against the requirements in your Markdown specification and fails the build when they drift apart. It asks TypeSafe AI's Jev model one focused question per requirement, gets a probability back, and compares it with a threshold you set. That makes it small enough for a pre-commit hook and strict enough for a CI gate.

$ npx jev-spec check

=== jev-spec Check Report ===

Target: auth [✖ FAILED]
  Spec files: docs/specs/auth.md
  Code files: src/auth/session.ts
  Model: jev-1.13.0
    ✔ REQ-AUTH-01: probability: 0.97
    ✖ REQ-AUTH-02: probability: 0.08
       └─ Violation: Probability 0.08 is below minimum threshold 0.85
    ✔ introducesUnspecifiedBehavior: probability: 0.03

Overall: ✖ CHECKS FAILED

$ echo $?
1

Example report. The layout is what the CLI prints (timing and cost lines omitted); the probabilities are illustrative.

jev-spec is an independent open-source project. It is not affiliated with or endorsed by TypeSafe AI. Please report problems with jev-spec in this repository, not to TypeSafe.

Linters and schema checks can tell you that REQ-AUTH-02 exists, is well formed and is linked from the right place. They cannot tell you whether the code does what REQ-AUTH-02 says. With AI-assisted development that gap widens, because the code changes faster than anyone rereads the specification:

• Does src/auth/session.ts still satisfy REQ-AUTH-02?
• Did the assistant quietly add a bypass header or an endpoint that nobody specified?
• Is this pull request a complete implementation, or a stub with optimistic comments?

You can put these questions to a general-purpose LLM in a prompt. Then you parse prose, the shape of the answer changes between runs, and you pay for every generated token. That is an awkward thing to gate a build on.

1. You write questions, not prompts. Each requirement gets one yes/no question (noul). Categorical (choice) and ordinal (score) rubrics are available when a decision depends on them. They live in a typed jev-spec.config.ts.
2. Jev answers with numbers. Jev is a System One model: it does not generate text. It reads the specification and the code once and returns a probability for every question in the same request. TypeSafe trains it for calibrated probabilities, which is what gives a threshold its meaning.
3. Thresholds decide. minProbability: 0.85, maxProbability: 0.15, allowedChoices, minScore: plain comparisons and standard exit codes (0 passed, 1 failed, 2 broken setup).
4. It is cheap enough for every commit. Jev is priced on input tokens only, $0.042 per million, and output is free, so checking a target costs a fraction of a cent. Every report prints the estimate. TypeSafe publishes its own speed and cost comparison with general-purpose LLMs.

• A probability is not a proof. jev-spec tells you that code has probably drifted from a requirement. It complements tests and review and replaces neither. Tune the thresholds on your own code before you trust them.
• Keep targets small. A target is sent in one request, and a target that does not fit (more than 500 files, a file over 2 MiB, or more than 120,000 characters of code) stops the run with exit code 2 instead of being cut. Make it one domain, not the whole src/ tree: Jev gets less accurate as unrelated content grows (see its known limitations).
• Ask narrow, literal questions. One behaviour per question, asked directly ("Is a token rejected when …?"), and say which part of the code you mean when two are alike. A requirement restated as a claim, multi-part questions, counting and stacked negations are answered less reliably.
• English works best. Jev's primary training language is English. Other languages, CJK scripts included, are accepted but less accurate, and TypeSafe advises testing on your own content first. With specs in another language, tune the thresholds on your own documents before you gate on them.
• Markdown tables in a specification are not sent to the model yet. Restate the rows in the question, or write the requirement as a list.
• --staged and --diff choose the targets, not what the model sees. A target none of whose files changed is skipped, and a touched target is checked in full. --staged reads what is staged, not the working tree. A change to a spec alone selects nothing, so keep a full run in CI.
• Real checks need a TypeSafe API key. jev-spec check --dry-run validates your setup without one.

jev-spec has specs of its own and is checked against them. docs/specs/ states the requirements, jev-spec.config.ts pairs each group with the one to three files that implement it, and test/probes/ holds, for every requirement, a patch that breaks it on purpose. A rubric belongs in the gate only when it passes on the intact code and fails on the broken copy. docs/probe-results.md records the latest run: 29 of 29 probes caught with jev-1.13.0. A full check of the 9 targets and 29 rubrics takes under four seconds and costs an estimated $0.001. The first run did not look like that: the questions, not the thresholds, were what had to change, and what we learned is in the header of the configuration.

Specification (Markdown) ─────────┐
                                  ├─► [jev-spec Engine] ─► Jev (System One) ─► Probabilities ─► Assertions
Implementation (Code / Git Diff) ─┘   (Root Jail + Boundary Isolation)                          (exit code 0 / 1 / 2)

1. Context Extraction: Parses markdown specifications using mdast (filtering by heading, tag, or requirement ID) and reads the source files of every target; in a diff run, only of the targets that the change touches.
2. Security Isolation: Enforces workspace root jails, symlink escape checks, git revision argument sanitization, and delimiting tags around the untrusted text sent to the model (a mitigation, not a guarantee).
3. One Request per Target: Sends the specification, the code and every rubric of a target to Jev in a single request.
4. Assertion Evaluation: Compares the returned probabilities and scores with your thresholds and exits with 0, 1 or 2.

AI-assisted setup. For Claude Code, Cursor, Codex, Gemini CLI, GitHub Copilot and any other Agent Skills-compatible client (requires GitHub CLI v2.90+):

gh skill install nozomi-koborinai/jev-spec jev-spec-init
gh skill install nozomi-koborinai/jev-spec jev-spec-fix

Then ask your agent to "set up jev-spec". jev-spec-init maps your specifications to code, writes one focused question per requirement, validates the wiring offline and reports which requirements are not covered. jev-spec-fix works through a failing check and reports what was and was not checked.

Manual setup. Get up and running with jev-spec in three steps:

Install jev-spec as a development dependency using your package manager of choice:

# Bun (recommended for local developer loops)
bun add -d jev-spec

# npm
npm install -D jev-spec

# pnpm
pnpm add -D jev-spec

Or run directly without local installation via bunx jev-spec or npx jev-spec.

A target pairs one part of your spec with the code that implements it. It has its own rubrics and assertions and is checked as a unit.

Create jev-spec.config.ts in your repository root:

import { defineConfig, noul, choice, score } from 'jev-spec';

export default defineConfig({
  targets: {
    auth: {
      description: 'Authentication session token verification',
      specPath: 'docs/specs/auth-requirements.md',
      codePaths: ['src/auth/**/*.ts', '!src/auth/**/*.test.ts'],
      specFilter: {
        requirementPrefix: 'REQ-AUTH-',
      },
      rubrics: {
        'REQ-AUTH-01': noul(
          'Is the signature of a session token checked before access to a protected resource is granted?'
        ),
        'REQ-AUTH-02': noul('Is a token rejected when its ID is on the revocation list?'),
        introducesUnspecifiedBehavior: noul(
          'Does the code add a way to reach a protected resource that the spec does not describe?'
        ),
        securityPosture: choice('Security posture of session management', {
          secure: 'Proper signature validation and revocation checks present',
          insecure: 'Missing verification, weak crypto, or tokens logged',
        }),
        implementationCompleteness: score('Degree of completeness', [
          'Stub: Empty function signatures or TODO comments',
          'Partial: Happy path implemented, error handling missing',
          'Feature Complete: Complete implementation meeting all criteria',
        ]),
      },
      assertions: {
        'REQ-AUTH-01': { minProbability: 0.85 },
        'REQ-AUTH-02': { minProbability: 0.85 },
        introducesUnspecifiedBehavior: { maxProbability: 0.15 },
        securityPosture: { allowedChoices: ['secure'], minConfidence: 0.75 },
        implementationCompleteness: { minScore: 1.8 },
      },
    },
  },
});

Name each rubric after the requirement it checks, and ask one direct, literal question about what the code must do. The name of a rubric is never sent to the model, and the report prints it, so a failure names its requirement. Keep the ID out of the question: in our own measurements, "Does the code satisfy REQ-AUTH-01: …?" was answered "yes" for code that had been broken on purpose, and the same question asked directly was not. A question that joins several requirements cannot tell you which one failed.

Create an API key in the TypeSafe console, set it, and run the check:

export TYPESAFE_AI_API_KEY="your-typesafe-api-key"

# Fast execution with Bun
bunx jev-spec check

# Or with Node.js npx
npx jev-spec check

(No API key yet? jev-spec check --dry-run validates the configuration, spec parsing and file matching without evaluating anything. --mock runs the offline mock evaluator instead: its results are placeholders, and every report is labelled MOCK MODE.)

jev-spec configurations use the defineConfig(...) helper for full TypeScript type inference and auto-completion.

Every configuration is validated before anything is sent to Jev. Assertion keys that match no rubric, options that do not fit the rubric type, unknown choice keys and out-of-range thresholds (for example maxProbability: 15 instead of 0.15) and client options of the wrong type (for example allowCustomBaseUrl: "false", a string) stop the run with exit code 2 instead of silently producing a check that can never fail. With defineConfig(...), TypeScript reports the same mistakes in your editor.

A noul is a calibrated boolean proposition evaluated in the range [0, 1]. Jev estimates the empirical probability that the statement is true given the specification context and implementation code.

rubrics: {
  implementsRateLimit: noul('Does the rate limiter enforce a 60 req/min bucket?'),
  hasBypassHeader: noul('Does the implementation permit any unauthenticated bypass header?'),
},
assertions: {
  implementsRateLimit: { minProbability: 0.85 },
  hasBypassHeader: { maxProbability: 0.10 },
}

• Assertion Options:
  • minProbability?: number: Minimum acceptable probability threshold ([0, 1]).
  • maxProbability?: number: Maximum tolerated probability threshold ([0, 1]).

A choice rubric represents a discrete categorical distribution across mutually exclusive options.

rubrics: {
  architecturePattern: choice('Architectural pattern applied', {
    hexagonal: 'Domain logic isolated with ports and adapters',
    layered: 'Classic controller-service-repository layered pattern',
    spaghetti: 'Tightly coupled concerns without distinct abstraction boundaries',
  }),
},
assertions: {
  architecturePattern: {
    allowedChoices: ['hexagonal', 'layered'],
    blockedChoices: ['spaghetti'],
    minConfidence: 0.80,
  },
}

• Assertion Options:

  • allowedChoices?: readonly T[]: Permitted choice keys.
  • blockedChoices?: readonly T[]: Forbidden choice keys (fails if selected).
  • minConfidence?: number: Minimum required confidence score for the selected choice ([0, 1]).

  An answer whose choice is not one of the rubric's keys fails every one of these assertions.

A score rubric maps evaluation onto an ordinal scale between 2 and 10 levels. Returns a continuous score, level probabilities, and confidence.

rubrics: {
  implementationCompleteness: score('Implementation depth relative to specification', [
    'Stub: Function signatures with empty bodies or TODO comments',
    'Partial: Happy path implemented, error handling or edge cases missing',
    'Feature Complete: Happy path and error branches fully handled',
    'Production Ready: Complete implementation with validation and defensive bounds',
  ]),
},
assertions: {
  implementationCompleteness: {
    minScore: 2.0, // Minimum fractional level index (0 = Stub, 3 = Production Ready)
    minConfidence: 0.70,
  },
}

• Assertion Options:
  • minScore?: number: Minimum fractional score index.
  • maxScore?: number: Maximum fractional score index.
  • minConfidence?: number: Minimum confidence metric ([0, 1]).

export interface TargetConfig<R extends Record<string, AnyRubric> = Record<string, AnyRubric>> {
  /** Optional human-readable description */
  readonly description?: string;

  /** Relative path to markdown/MDX specification inside workspace */
  readonly specPath: string;

  /** Relative file paths or glob patterns for implementation files */
  readonly codePaths: readonly string[];

  /** Spec section filtering rules */
  readonly specFilter?: {
    readonly headings?: readonly string[];       // Filter by heading titles
    readonly requirementPrefix?: string;         // e.g. 'REQ-AUTH-' or 'AC-'
    readonly tags?: readonly string[];           // Filter by hashtag labels e.g. ['auth']
  };

  /** Declared Jev evaluation rubrics */
  readonly rubrics: R;

  /** Assertions checked against Jev results */
  readonly assertions: AssertionMap<R>;
}

specFilter keeps every section that satisfies all of the given criteria, together with its nested subsections, so details written under deeper headings stay part of the requirement. A filter that matches no section is treated as a configuration error (exit code 2) rather than silently sending the whole document.

export default defineConfig({
  client: { model: 'jev-1.13.0' },
  targets: {
    // …
  },
});

Without client.model, jev-spec asks jev-latest, an alias that TypeSafe moves to a newer model with every release, so a result can change without any change in your repository. Once you have tuned your thresholds, pin the versioned model ID they were tuned against (TYPESAFE_DEFAULT_MODEL works too). Every report prints the model that answered (Model: jev-1.13.0), and --dry-run warns while the model is not pinned.

Check every target declared in your configuration:

# Check via Bun
bunx jev-spec check

# Check via Node.js
npx jev-spec check

Check one target by name:

bunx jev-spec check --target auth

Check only the targets whose code changed. Each of them is checked in full:

# Check the targets that the staged changes touch (ideal for pre-commit git hooks)
bunx jev-spec check --staged

# Check the targets that a branch range touches (for local runs, or repositories too large for a full run)
bunx jev-spec check --diff origin/main...HEAD

Targets whose codePaths match none of the changed files are reported as SKIPPED: they are not sent to Jev and do not affect the exit code, so a pre-commit hook never blocks a commit that does not touch a target.

# Validate the setup: config, spec parsing, file matching. Evaluates nothing, needs no API key, still executes the config
npx jev-spec check --dry-run

# Offline mock evaluator (placeholder results, labelled MOCK MODE in every report)
npx jev-spec check --mock

# Usage and version (no configuration file required)
npx jev-spec --help
npx jev-spec --version

A dry run prints, for every target, the specification sections and requirement IDs it found, the code files it matched, the rubrics it would ask and the estimated cost. It also warns about requirement IDs that no rubric mentions and codePaths that match no file. Code that does not fit in one request stops it with 2, as it stops a real run. It exits with 0 when the setup is valid and 2 when it is not; it never exits with 1, because nothing is checked. It is not a static check: the configuration is TypeScript or JavaScript, and a dry run executes it like any other run (see the CI threat model below).

Unknown commands, unknown options, missing option values and unsupported --format values are rejected with exit code 2.

# Formatted terminal report (default)
npx jev-spec check --format terminal

# Markdown report (ideal for GitHub Step Summaries and PR comments)
npx jev-spec check --format markdown --output jev-spec-report.md

# Machine-readable JSON output (for custom reporting pipelines)
npx jev-spec check --format json --output result.json

When a run exits with 2, --format json and --format markdown still write a report. It has "passed": false, the message in error, and in uncheckedTargets each target whose code could not be checked, with its reasons (an empty list for any other error). The error is printed to standard error as well. The terminal format prints the error only.

--output only accepts paths inside the project root. To publish the report to the GitHub Actions step summary (which lives outside the workspace), redirect stdout instead:

npx jev-spec check --format markdown >> "$GITHUB_STEP_SUMMARY"

• 0: All targets and assertions passed.
• 1: A check failed (one or more assertions violated).
• 2: Configuration or runtime error (missing file, invalid argument, missing API key, code that does not fit in one request, a code file that cannot be read).

jev-spec provides first-class dual-runtime support across modern Node.js and Bun. Every pull request is tested in automated CI on Node.js 22, Node.js 24 and the latest Bun release.

Either. Measured on the maintainer's laptop against this repository (Node 22.23, Bun 1.4): the CLI starts in about 0.1 s on Node and about 0.05 s on Bun, and a dry run of this repository at 0.3.0 took 0.15 s and 0.08 s. A live check of one target takes 0.3 to 0.9 s, so the request to the model, not the runtime, decides how long a hook takes:

• What a hook costs: jev-spec check --staged checks only the targets that the commit touches, each in one request. A commit that touches no target sends nothing and needs no API key.
• bunx runs Node by default: the CLI has a #!/usr/bin/env node shebang, and bunx jev-spec honours it. Use bunx --bun jev-spec to run it on Bun.
• TypeScript configuration: jev-spec.config.ts is loaded directly on both runtimes.

jev-spec is engineered for safe execution in automated CI/CD environments and developer workstations.

1. Untrusted Code Risk: A pull request can modify jev-spec.config.ts, specifications, or code. The configuration is TypeScript or JavaScript, and jev-spec executes it when it loads it, in every run including --dry-run. Executing untrusted code with access to sensitive credentials introduces secret exfiltration vectors.
2. Recommended Defense-in-Depth Patterns:
   • Dry Run for Fork PRs: Run fork pull requests as a dry run (jev-spec check --dry-run). It validates the configuration, spec parsing and glob matching without an API key, but it still executes the configuration of the pull request, and with it that pull request's code. Run it in a job without secrets and with read-only permissions (permissions: contents: read), and never under pull_request_target.
   • Environment Protection: For live checks on external PRs, use GitHub Actions Environment Approvals so maintainers review the diff before secrets are unlocked.
   • Main Branch Checks: Run live checks on push to main and trusted internal release branches.

name: Specification Semantic Gate

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  check-specs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm

      - name: Install Dependencies
        run: npm ci

      - name: Run jev-spec (Same-Repository Pull Request / Full Run)
        if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
        env:
          TYPESAFE_AI_API_KEY: ${{ secrets.TYPESAFE_AI_API_KEY }}
        run: npx jev-spec check --format markdown >> "$GITHUB_STEP_SUMMARY"

      - name: Run jev-spec (Push to Main / Full Run)
        if: github.event_name == 'push'
        env:
          TYPESAFE_AI_API_KEY: ${{ secrets.TYPESAFE_AI_API_KEY }}
        run: npx jev-spec check --format markdown >> "$GITHUB_STEP_SUMMARY"

      - name: Run jev-spec (External Fork / Dry Run, No Secrets)
        if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository
        run: npx jev-spec check --dry-run

Both live steps are full runs on purpose: a diff run skips every target whose code did not change, so a pull request that changes only a specification would pass without anything being checked, and on main the range origin/main...HEAD is empty. This repository's own workflow (.github/workflows/jev-spec.yml) runs the live check only on pushes to main and a dry run on every pull request, because its job is advisory and the repository has a single maintainer.

jev-spec implements defensive security controls that protect developer machines and CI runners:

Contributions are welcome! Please ensure all tests and linter checks pass before submitting a pull request:

npm run check      # Biome (lint + format check), typecheck and tests
npm run lint:fix   # apply Biome formatting and safe lint fixes

Released under the MIT License.

Support ongoing work on jev-spec through GitHub Sponsors.