Arize AI

Agents changed the stack. The stack changed agents. Old software had one path, written in the files. Agents have many possible trajectories. The model plans, memory has to come back at inference time, and the path is not knowable until runtime. This Thursday at the Google Chicago office we'll walk that as one system: Gemini thinks, MongoDB remembers and acts, Arize shows the traces. Space is limited: https://lnkd.in/gmN8MxyC

Does your agent need retrieval on every turn? At Upstart, unnecessary retrieval API calls were costing real money at scale. In Arize AX, Signal surfaced the pattern automatically. For developers, it's important to evaluate whether a tool call needed to happen + whether it succeeded. Signal reviews production traces and groups recurring issues with supporting evidence, so engineers can investigate what to change. See how to find those patterns in your own agent: https://lnkd.in/efVcUjAA

For years the story has been that models are superhuman at chat, so where is all the automation? TypeSafe's answer, out of two years of stealth this week, is that we've been asking the wrong kind of model. Meet Jev, the first public System One Model. Founder Diogo Almeida, who worked on the instruction-following research behind ChatGPT at OpenAI, built a new class of frontier model that doesn't generate prose at all. You hand it unstructured state and typed questions. It hands back decisions your code can act on directly: a yes/no probability (Noul), a label with a probability for every option (Choice), or an expected value on a rubric you define (Score). Giving up strings buys a lot. Zero type errors, by construction. Calibrated probabilities instead of vibes, from a training method TypeSafe calls RLCD, reinforcement learning for calibrated decisions. And speed: 70 to 500 ms end to end, which TypeSafe puts at 40 to 200x faster than frontier LLMs at the same intelligence, at $0.042 per million input tokens with output free. Their workflow evals go as high as 193x faster and 444x cheaper. That's not a cheaper LLM. That's the Jevons paradox TypeSafe named it after: drop the cost of a decision by two orders of magnitude and every conditional in your software becomes a place you can afford to put intelligence. Routing, guardrails, classification, verification of LLM output, map-reduce over a data lake. Things you would never wait three seconds or pay a cent per call for. Which is exactly why the decisions need to be observable. When a 142 ms judge is gating what your agent sends to customers, you want to see the judgment where it happened: inside the agent's trace, next to the tool call and the draft it was judging. So we shipped OpenInference instrumentation for the TypeSafe SDK, Python and JS. Every Jev call is an LLM span in Phoenix with the state, the questions, the chosen answers, the full probability distribution, the confidence, the model version, and the token counts. Plain OpenTelemetry, so it goes to Phoenix, Arize AX, or any OTLP collector you already run. No vendor SDK wrapping your client. Screenshot below: Jev reading a support draft that promised free shipping for life off a stale playbook. unsafe_or_invented, confidence 1.0, blocked before it reached the customer. Jev is in early access at https://typesafe.ai. The instrumentation is open source today: Python: pip install openinference-instrumentation-typesafe https://lnkd.in/gSv_hzY7 JS: npm install @arizeai/openinference-instrumentation-typesafe https://lnkd.in/gfyh26B8 Source and the support-agent example: https://lnkd.in/g2B4pJTV https://lnkd.in/grNE9CcU

For all you Codex fans out there, we've just added support for Codex as the harness for your managed agents. Configure your managed agents with Codex as the harness, select your models, skills, attach repos, and define your task. Then let your managed agent rip analyzing your traces and working with them.

A whole new type of model may make us reevaluate how we do evals:

“Did the agent actually do what the user asked?” 👀 That is the question behind a lot of agent reviews. A trace can look good at first glance. The answer may still miss part of the request. The retrieved context may be real, but it may not help the task. The response may add a detail that never appeared in the conversation. The trace may also include personal data that needs review. Phoenix now has detailed docs for several built-in evals. ✅ Completeness checks whether active user requests were finished. 🔎 Retrieval relevance checks whether retrieved information served the request. 🧭 Hallucination checks whether the response is grounded in the conversation. 🛡️ Toxicity checks harmful or abusive text. 🔒 PII detection checks conversation records for personal data. The pages show the needed input fields and examples in Python and TypeScript. They also explain the score output so you know what the result means. For teams reviewing traces or comparing experiments, this is a practical starting point before writing custom judges. Start here: https://lnkd.in/g36GTc5z Direct links for each eval are in the first comment.

What happens when your evals are passing, but your agent is still failing? A polished final answer can hide a bad trajectory. The agent may have called the wrong tool, retried without making progress, or claimed an action succeeded when it never happened. Known failures are relatively easy to evaluate once you can define them. But it's a lot harder to find recurring production failures you haven’t learned to name yet. That’s the idea behind Signal in Arize AX. It automatically reviews production trajectories across runs, surfaces recurring patterns, and proposes fixes (and pushes a PR if you give it repo access). We break down the full debugging workflow here: https://lnkd.in/gUffbkrp

https://lnkd.in/gpMGrtVK