feat(evals): route TypeSafe decision-model connections through OpenRouter or Vercel AI Gateway - #17798

🟡 Live preview: pr-17798 app preview

What does this PR do?

Lets a TypeSafe decision-model connection pick where Jev is served from instead of exposing a free-form base URL. The connection form gains an Upstream select with three presets:

• TypeSafe (https://api.typesafe.ai/v1, models jev-1.13.0, jev-latest)
• Vercel AI Gateway (https://ai-gateway.vercel.sh/typesafe/v1, models typesafe-ai/jev, typesafe-ai/jev-latest)
• OpenRouter (https://openrouter.ai/api/v1, models jev-1.13, jev-latest)

Both gateways ship a TypeSafe-compatible POST …/v1/systemone endpoint that accepts TypeSafe's request shape (model, state, questions with noul/choice/score) and returns TypeSafe's response shape (model, answers, usage), so the existing @ai-sdk/typesafe-ai adapter only needs its baseURL swapped. See OpenRouter: TypeSafe SDK and Vercel: TypeSafe API with AI Gateway.

Changes

• packages/shared: TypeSafeConfigSchema ({ upstream }) joins the connection config union; TYPESAFE_UPSTREAM_DEFINITIONS holds label, base URL, default model IDs, and docs link per upstream; resolveTypeSafeUpstream / getDecisionModelDefaultModels read them. createTypeSafeDecisionModelClient takes an upstream and passes the matching baseURL to the AI SDK provider. Connections created before this change have no config and keep talking to TypeSafe directly.
• web: the LLM connection form renders the upstream select for the TypeSafe adapter and labels the key field per upstream. The router stores the upstream in config, still rejects a custom base URL and extra headers for decision models, rejects non-TypeSafe configs, and requires a new secret key when the upstream changes (a TypeSafe key does not work at OpenRouter). The connection test and the evaluator test call use the stored upstream. The decision-model picker lists per-upstream default model IDs.
• worker: the eval execution path passes the stored upstream to the client. The Jev pricing matchPattern additionally accepts OpenRouter's dated response slugs (typesafe/jev-1.13-20260917) and the ~typesafe/ alias so gateway-routed evaluations still get a cost.

Test this

https://pr-17798.preview.langfuse.com/project/<id>/settings/llm-connections (the typesafe adapter is behind the decisionModelEvaluators flag: Langfuse admins or LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=true):

1. + Add LLM Connection → adapter typesafe (experimental).
2. A new Upstream select appears; pick OpenRouter — the key field label becomes "OpenRouter API key" and the default models switch to jev-1.13, jev-latest.
3. Paste an OpenRouter key and create; the connection test runs one noul question against https://openrouter.ai/api/v1/systemone.
4. Edit the connection, switch upstream to Vercel AI Gateway without entering a key → blocked with "Enter the API key for the new upstream."
5. Under Evaluators → decision model, the model list for that connection shows the upstream's IDs.

Data: no seed needed; the settings page is enough. No gateway keys were available in the agent environment, so the authenticated round-trip against OpenRouter / Vercel (step 3) still needs a human with a key.

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update (the LLM connections docs should mention the upstream presets; follow-up in langfuse-docs)

Mandatory Tasks

• Make sure you have self-reviewed the code. A decent size PR without self-review might be rejected.

Verification

• pnpm --filter @langfuse/shared run test src/server/llm/typesafe/typeSafeDecisionModelClient.test.ts — Tests 6 passed (6) (new: per-upstream URL routing, tolerance of OpenRouter's extra response fields)
• pnpm --filter web run test src/__tests__/server/llm-api-key.servertest.ts — Tests 35 passed (35) (new: upstream persisted in config, base URL / foreign config rejected, upstream change requires a new key, legacy connection without config resolves to TypeSafe)
• pnpm --filter worker run test src/features/evaluation/decisionModel/runDecisionModelEvaluation.test.ts — Tests 3 passed (3)
• pnpm exec turbo run typecheck lint --force --filter=web --filter=@langfuse/shared — Tasks: 7 successful, 7 total, Cached: 0 cached, 7 total
• pnpm --filter worker run typecheck and pnpm --filter worker run lint — both exit 0 (run directly; the turbo path needs the Rust toolchain for @langfuse/native, which is not installed here)
• pnpm exec knip — exit 0
• node .agents/skills/add-model-price/scripts/test-match-pattern.mjs --model jev … — 15 accept/reject samples pass; validate-pricing-file.mjs validates 172 entries
• Unauthenticated POST probes of all three /v1/systemone endpoints return 401 with a JSON error the adapter parses (paths exist)
• Browser check on the rebuilt local Cloud stack (video above): upstream select, per-upstream default models, docs link and key label, and the empty-key validation error

  

Confidence Score: 4/5

The PR should not merge until upstream changes handle existing custom model IDs, because stale identifiers can prevent otherwise valid gateway migrations.

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Form[LLM connection form] -->|stores config.upstream| DB[(LLM connection)]
  DB --> Selector[Decision model selector]
  DB --> WebTest[Evaluator test]
  DB --> Worker[Evaluation worker]
  Selector -->|upstream-specific model IDs| Eval[Evaluator definition]
  WebTest --> Resolve[Resolve TypeSafe upstream]
  Worker --> Resolve
  Resolve --> TS[TypeSafe API]
  Resolve --> Vercel[Vercel AI Gateway]
  Resolve --> OR[OpenRouter]
  TS --> Price[Jev pricing match]
  Vercel --> Price
  OR --> Price

_{Reviews (1) · Last reviewed commit: "feat(evals): route TypeSafe decision-mod..."}