WILL IT JEV?

An in-process, high-throughput request routing engine written in Rust. It classifies incoming prompts and routes them to discriminative (Jev) or generative (LLM) execution pathways with a p99 decision latency under 5 ms for the overwhelming majority of traffic.

The engine is a five-tier deterministic fallback cascade. Each tier either resolves the request with sufficient confidence or passes it to the next, more expensive tier. Cheap tiers absorb most traffic, so the expensive tiers only see the ambiguous remainder.

1. Design Goals
2. Cascade Semantics
3. Tier Reference
4. Latency Budget and Target Coverage
5. Repository Layout
6. Runtime and Performance Model
7. Configuration
8. Building
9. Model Artifacts
10. Testing
11. Benchmarking
12. Observability
13. Extending the Router
14. Failure Modes and Fallback Behavior
15. Known Constraints
16. Contributing
17. License

• Bounded latency. Every tier has an explicit p99 budget. Total decision latency for tiers 1 to 3 stays within the 5 ms SLA.
• Early exit. A request leaves the cascade at the first tier whose confidence meets its configured threshold.
• No GC pauses, minimal allocation. The hot path is native Rust with C++ bindings, a replaced global allocator, and zero-copy ingress for the cheapest tiers.
• Deterministic behavior. Given the same input, model artifacts, and configuration, the router returns the same decision.
• Pluggable evaluators. Tier 2 and tier 3 engines sit behind a common trait, so models can be swapped through configuration without touching the dispatcher.

A request is a prompt payload. The dispatcher in src/router/cascade.rs runs tiers in order and stops at the first tier that returns a resolved outcome.

Each tier returns one of two outcomes:

• Resolved: the tier produced a routing decision whose confidence meets its threshold. The cascade terminates and the decision is returned.
• PassThrough: the tier could not decide with enough confidence (or is disabled). The payload moves to the next tier.

Tier 5 is the terminal tier. It always produces a decision, so the cascade is total: every request receives a routing outcome.

Disabled tiers are skipped without cost. Tiers are always evaluated in numeric order, and a tier is never re-entered once passed.

• Low-Latency Deterministic Pipeline: Replaces conventional regex guardrails with Aho-Corasick automata and Damerau-Levenshtein fuzzy matching, scanning the immutable payload through exact positional, fuzzy-token, imperative-zone, prefix, and polarity checks.
• Ambiguity as a First-Class Output: The policy explicitly models competing intent. When strong LLM generative cues conflict with Jev discriminative signals, Tier 1 enters an explicit conflict state and defers to Tier 2 rather than forcing a premature guess.
• Deterministic Query Fast Paths: Recognizes requests whose answers can be resolved through closed-form computation or other deterministic operations, bypassing semantic model inference entirely. Math dependent requests are the simplest case; the same layer is designed to absorb quantitative and other structured queries as additional fast paths.

• Subword Route Classification: FastText provides the first learned routing layer, using subword features to classify prompts into Jev or generative execution paths. The classifier exposes the full probability distribution rather than discarding the model's raw evidence after selecting the top class. n-gram embeddings evaluated via in-process static C++ bindings (libfasttext).

• Confidence-Gated Escalation: Tier 2 does not treat the top prediction as sufficient by itself. Its calibrated score is compared against a configurable confidence threshold; high-confidence Jev predictions terminate the cascade, while uncertain cases pass through to the next tier. The threshold therefore controls the tradeoff between Jev coverage and false-Jev commits. The classifier produces the top-1 probability P1 and top-2 probability P2. A margin-scaled confidence score is computed as:

  S = P1 * (1.0 + (P1 - P2))

  This tier resolves when S >= confidence_threshold; otherwise it returns PassThrough. A wide margin between the top two classes raises confidence, and a narrow margin lowers it, even when P1 alone is high.

• Empirical Threshold Calibration: Thresholds are evaluated against held-out labeled data rather than chosen by hand. The evaluation harness sweeps the actual score boundaries produced by the model, reports Jev precision and coverage with 95% Wilson confidence intervals, and rejects evaluation sets that overlap normalized training prompts. Source-held-out experiments provide an additional test of cross-source generalization.

• Robust to typos, inflections, and fuzzy keyword variation, because subword features generalize across spellings.

• Artifact: quantized binary router_fastpath.ftz.

• Exemplars at Runtime: Tier 3 does not use a separately trained routing model. It loads the current labeled exemplar set at runtime and evaluates the incoming request directly against those examples, keeping the routing knowledge explicit and updateable without retraining.
• Pluggable Semantic Engines: Harrier is the default evaluator, with ModernBERT as an upcoming alternative and Oryn available as an optional engine. The exemplar set remains constant while the semantic engine can change independently.
• Example-Driven Resolution: Rather than learning a permanent decision boundary from the exemplars, Tier 3 uses them directly to determine whether the request is sufficiently supported by the observed Jev or LLM examples. Weak, conflicting, or out-of-distribution evidence passes through to Tier 4.

• Capability-Level Judgment: Tier 4 uses Jev itself to determine whether Jev can handle the incoming request. The system therefore moves from predicting the route to asking the destination capability whether the request belongs within its executable domain.
• Final Discriminative Gate: Tier 4 provides the last opportunity to resolve a request without invoking a generative model. A positive Jev determination terminates the cascade; uncertainty or a negative determination escalates to Tier 5.
• Recursive Architecture: The cascade progressively asks increasingly capable systems whether generation is actually necessary, culminating in the unusual case where Jev evaluates whether Jev can solve the problem.

• Open-Ended Generation: Tier 5 is the terminal generative pathway for requests that survive every discriminative tier or inherently require open-ended language generation, synthesis, or reasoning.
• Guaranteed Termination: Tier 5 always produces the final routing outcome, making the cascade total even when every cheaper evaluator abstains or fails, or are not unavailable.
• Expensive by Design: The generative model is deliberately reserved for the residual traffic that earlier tiers cannot confidently establish as JEV-capable, minimizing unnecessary token consumption while preserving full LLM capability for genuinely generative requests.

Latency figures are per-tier p99 budgets. Coverage figures are design targets for the share of traffic each tier should resolve. Actual values depend on your traffic mix and must be measured (see Benchmarking and Observability).

Note that the 5 ms SLA holds for traffic that exits at tiers 1 to 3 within their lower budget range. The upper end of tier 3 (8 ms), tier 4, and tier 5 intentionally exceed it, which is why keeping their coverage small matters.

src/
  config/
    mod.rs            Configuration loader with environment overrides
    schema.rs         Serde schemas for tier thresholds, model paths, flags
  logging/
    mod.rs            Low-overhead tracing and telemetry initialization
  router/
    cascade.rs        Pipeline dispatcher and lifecycle orchestration
    tier1.rs          Static hash, length bound, and regex evaluator
    tier2.rs          FastText subword classifier runner
    tier3.rs          In-process micro-transformer runner
    tier4.rs          Heavy fallback model runner
    tier5.rs          Frontier LLM escalation runner
  tier3/
    calibration.rs    Margin calculation and probability calibration
    trait.rs          Core evaluator trait and contract definitions
    inprocess/
      fasttext.rs     FFI wrapper for libfasttext
      modernbert.rs   ONNX Runtime implementation for ModernBERT
      harrier.rs      ONNX Runtime implementation for Harrier
      oryn.rs         Oryn in-process engine
    remote/
      client.rs       Asynchronous gRPC client
      proto.rs        Generated Protocol Buffer bindings

• Global allocator: the system allocator is replaced with mimalloc via #[global_allocator] to avoid heap lock contention across worker threads.
• Threading: a fixed worker pool sized by router.workers. Pinning workers to physical cores is recommended for stable tail latency.
• Zero-copy ingress: tiers 1 and 2 receive the payload as &str, so no copy or allocation is made during pre-filtering.
• Quantization: tier 3 models run Int8 to fit the latency budget on CPU.
• Static linking: libfasttext is linked statically to avoid dynamic loading overhead and deployment drift.

The router reads config/router.toml. Any key can be overridden by an environment variable with the ROUTER__ prefix and double-underscore nesting, for example ROUTER__TIER2__CONFIDENCE_THRESHOLD=0.9.

[router]
workers = 16
max_payload_bytes = 1048576

[router.tier1]
enabled = true
regex_patterns_path = "config/rules.regex"

[router.tier2]
enabled = true
model_path = "models/router_fastpath.ftz"
confidence_threshold = 0.85

[router.tier3]
enabled = true
engine = "modernbert"            # modernbert | harrier | oryn
model_path = "models/tier3_quantized.onnx"
tokenizer_path = "models/tokenizer.json"
centroids_path = "models/centroids.bin"
confidence_threshold = 0.80

[router.tier4]
enabled = true
timeout_ms = 50

[router.tier5]
enabled = true
endpoint_url = "https://api.frontier.internal/v1/classify"

The values above are illustrative defaults. Thresholds must be calibrated on your own validation data before production use.

• A stable Rust toolchain
• A C++ toolchain and clang, required to build the static FastText library
• pkg-config
• Python 3 (for model training, exemplar dataset loading/serving, and calibration)

# Debian / Ubuntu
sudo apt-get install build-essential clang pkg-config

# macOS
xcode-select --install

cargo build --release

For best SIMD performance, build with native CPU features enabled:

RUSTFLAGS="-C target-cpu=native" cargo build --release

Binaries built with target-cpu=native are not portable across CPU generations. Build on, or for, the deployment hardware.

The router expects these artifacts at the paths configured above. They are not checked into the repository.

python3 scripts/train_fasttext.py \
  --input data/training_prompts.txt \
  --output models/router_fastpath.ftz

After training, recalibrate tier2.confidence_threshold against a held-out set. A retrained model invalidates the previous threshold.

cargo test --all-targets
cargo run --example eval_tier1
cargo run --example eval_tier2

The Tier 1 eval reads evals/tier1/cases.jsonl and reports Jev precision, coverage, LLM-verdict precision, and deferred cases. Review false Jev commits before changing the Tier 1 thresholds or pattern set.

The Tier 2 eval reads calibration/ft_valid.txt, the held-out split produced from the labeled calibration exemplars, and sweeps Jev precision and coverage across confidence thresholds. It checks for normalized prompt overlap with calibration/ft_train.txt, reports Wilson confidence intervals for precision, evaluates each distinct Jev score boundary as well as 0 and 1, and lists false Jev commits at the current 0.95 threshold. To evaluate an independent labeled set, pass it with --cases and provide its training source with --training-cases. Use a separate representative validation set before changing the production threshold.

For a source-held-out stress test, train without one exemplar file and write all generated artifacts outside the repository:

python3 calibration/make_fasttext_train.py \
  --holdout-source calibration/exemplars_ac-ct.jsonl \
  --output-dir /tmp/tier2-source-holdout
cargo run --example eval_tier2 -- \
  --model /tmp/tier2-source-holdout/tier2.bin \
  --cases /tmp/tier2-source-holdout/ft_holdout.txt \
  --training-cases /tmp/tier2-source-holdout/ft_train.txt

This checks cross-source generalization, not independent production performance; the held-out prompts come from the same labeled exemplar collection.

To see tier-level output while debugging:

cargo test --all-targets -- --nocapture

The suite should cover, at minimum:

• Per-tier resolve and pass-through behavior at threshold boundaries
• End-to-end cascade ordering and short-circuiting
• Disabled-tier skipping
• Payload bound rejection at tier 1
• Calibration math in calibration.rs

Micro-benchmarks use Criterion.

cargo bench

When validating the SLA:

• Benchmark each tier in isolation first, then the full cascade.
• Report p50, p99, and p99.9, not means.
• Run on the target deployment hardware with the production allocator and build flags.
• Use a representative traffic mix. Coverage targets in this document are only meaningful against realistic input distributions.

Telemetry is driven by tracing, initialized in src/main.rs. Every classification request generates a structured span containing feature variables, execution path outcomes, and timing metadata, exported to OpenTelemetry/Tempo for distributed tracing and Loki for log aggregation.

The Tier1Engine::classify method instruments spans with individual feature metrics and routing decisions:

• Exact & Fuzzy Feature Counters: matches_llm_prefix, starts_polar, jev_strong_hits, jev_mod_hits, llm_strong_hits
• Fast-Path Markers: is_binary_question
• Decision State: confidence, routing_reason

Tracking resolved-at-tier distribution over time is the primary health signal. A shift of traffic toward Tiers 4 and 5 indicates threshold, model, or prompt-distribution drift, causing tail latency and operational cost to rise.

Distributed tracing and log streaming are configured via default local endpoints or environment variables:

• OTLP Traces (Tempo): Sent via opentelemetry-otlp using SdkTracerProvider. Defaults to http://localhost:4317.
• Loki Events: Streamed via tracing-loki under the service label cascade-router. Defaults to http://localhost:3100.

Verbosity is controlled via RUST_LOG. The default level is info. RUST_LOG=info cargo run --release Recommended production configuration: RUST_LOG=info,cascade_router=debug

Tracking resolved-at-tier distribution over time is the most important signal. A shift of traffic toward tiers 4 and 5 means thresholds, models, or the input distribution have drifted, and both cost and tail latency will rise.

Each completed route is sent through a Tokio channel to a background writer, which appends one JSON object per line to logs/routing_capture.jsonl. The log directory is created at startup. The record includes a UUID v4 request ID, the request-start UTC timestamp, the raw prompt, SHA-256 hashes of the Tier 1 TOML and Tier 2 model files, the outcomes from tiers that ran, and the final backend, tier, and latency in microseconds. Tier 2 records its raw probability vector; Tier 3 records its score. The sampling probability currently defaults to 1.0.

The raw prompt is stored verbatim and may contain sensitive data. Restrict access to the capture file and define appropriate retention and cleanup; the writer appends indefinitely and does not rotate the file.

Log verbosity is controlled with the RUST_LOG environment variable. The default level is info. Invalid RUST_LOG values cause startup to fail with a configuration error.

Set RUST_LOG=debug for more detail:

RUST_LOG=info cargo run --release

Common settings:

Trace-level logging adds overhead and can distort the latency figures in this document. Benchmark and deploy with info or higher.

Tier 2 and tier 3 engines implement a common evaluator contract defined in src/tier2/trait.rs. To add an engine:

1. Add an implementation under src/tier2/inprocess/ (or a remote one under src/tier2/remote/).
2. Implement the evaluator trait, returning a resolved decision with a confidence score, or a pass-through.
3. Register the engine name in the configuration schema in src/config/schema.rs.
4. Wire selection in the corresponding tier runner in src/router/.
5. Add threshold-boundary tests and a Criterion benchmark.

Engines must not allocate on the per-request hot path where avoidable, and must keep their documented p99 within the tier's budget.

• Tier pass-through: low confidence is not an error. It is the normal escalation path.
• Model load failure at startup: the router should fail fast rather than silently run with a tier missing. Disabling a tier must be an explicit configuration choice.
• Tier 4 timeout: when timeout_ms elapses, the request escalates to tier 5 rather than blocking.
• Tier 5 unavailable: the terminal tier is an external dependency. Define and test your degraded-mode policy (default route, error, or retry) before production.
• Oversize or malformed payloads: rejected at tier 1 based on max_payload_bytes.

• The 5 ms p99 SLA applies to traffic resolved within tiers 1 to 3 at the lower end of tier 3's range. Tiers 4 and 5 exceed it by design.
• Coverage percentages are targets, not guarantees.
• AVX-512 and NEON acceleration depend on the host CPU. Throughput and latency on other hardware will differ.
• Thresholds are coupled to specific model artifacts. Swapping or retraining a model requires recalibration.

1. Open an issue describing the change before starting large work.
2. Keep changes to tiers 1 to 3 allocation-free on the hot path.
3. Include tests and, for any hot-path change, Criterion benchmark results before and after.
4. Run cargo fmt, cargo clippy --all-targets, and cargo test --all-targets before opening a pull request.

See the LICENSE file in the repository root.