for the engineers who want the details
How it works
NoGPU trains a small model on the answers your LLM already gave, measures it on rows it never saw, and sets a confidence gate: the model answers only when it is sure, and everything else goes to your LLM as before.
How we measured this
The $99 model's numbers come from the live sample report. The research benchmark: a bge-small-en-v1.5 text encoder (33M parameters, int8 ONNX) that learned from an LLM's answers, tested on 3,080 held-out Banking77 messages, with the confidence cutoff fixed beforehand on a separate dev split at a 90% bar. Latency: one request at a time on a Google Cloud e2-standard-4 (AMD EPYC 7B12, 4 vCPU, 16 GiB), 2 threads, 1,000 requests. Cost counts CPU time only, at Google's us-central1 list price ($0.134 an hour); LLM calls on the questions handed back are extra.
- The model your report measured: a fast gradient-boosted tree model (LightGBM) on your fields and text, yours to keep.
- The confidence gate, set so the model answers only above the bar your report certified.
serve.py: reads one question per line and answers, offline, with open-source libraries only.- The report it was built from, so anyone on your team can check the numbers.
$ echo '{"text": "where is my card"}' | python serve.py
{"label": "card_arrival", "confidence": 0.97, "gate": "answer"}
$ echo '{"text": "two cards, one dispute, wrong name"}' | python serve.py
{"label": null, "confidence": 0.54, "gate": "cascade"} # ask your LLM
# or in Python:
from serve import load, decide
model = load()
out = decide(*model, {"text": message})
label = out["label"] if out["gate"] == "answer" else ask_your_llm(message)
Logs in OpenAI, Anthropic, Langfuse, LangSmith or Helicone?
Convert them on your own machine first. The script uses only the Python standard library, sends nothing anywhere, and keeps just the last thing the user asked and the answer your model gave. System prompts and earlier messages are dropped.
It reads OpenAI chat completions (stored completions joined with their messages, fine-tune
JSONL, the Responses API), Anthropic Messages replies (tool_use included), Langfuse
observations, traces and API pages, LangSmith runs, and Helicone exports made with
--include-body.
curl -sO https://nogpu.dev/static/nogpu_export.py
python3 nogpu_export.py your_logs.jsonl > nogpu.jsonl
# custom shape? point at the fields:
python3 nogpu_export.py logs.json --input-field input.query --label-field output.route
What does "at least, with 95% confidence" mean?
We test the model on rows it never saw while training. On the Banking77 sample, when the $99 model answered, it matched the label 94.1% of the time. The lower figure (91.4%) comes from a standard statistical test (Clopper–Pearson, one-sided, 95% confidence): the true rate is at least that high, as long as your live traffic looks like the data you uploaded. Your report shows the same numbers for your data: how often it matches your LLM.
Which LLMs can it learn from?
Any of them: a frontier API model, your own fine-tuned model, or answers your team reviewed
by hand. If your logs include the model's probabilities as teacher_probs, the
report also shows how the LLM did on the questions the CPU model handed back.
What hardware does it need?
A normal CPU. On the Banking77 sample the $99 model takes about 5 ms per decision on our server; your report shows its speed on your data. The research benchmark ran on a 4-core Google Cloud server ($0.13 an hour) using 2 threads, also at about 5 ms per answer.