From the team behind Gatsby, Mastra is a framework for building AI-powered applications and agents with a modern TypeScript stack.

AI Observability & Evaluation

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

AI observability platform for production LLM and agent systems.

Evaluation and Tracking for LLM Experiments and AI Agents

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

Harbor is a framework for running agent evaluations and creating and using RL environments.

Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.

AI system design guide for engineers building production AI systems and evals.

Evaluate your LLM-powered apps with TypeScript

OpenSource Production ready Customer service with built in Evals and monitoring

🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

Build AI workflows by prompt or visual canvas. Heym is source-available and self-hosted, with agents, RAG, MCP, HITL, observability, evals, token cost tracking and more.

Cloud-synced dashboards for OpenCode and Claude Code. Track sessions, search with semantic lookup, export eval datasets.

Agent trace and tool-use safety evaluation lab.

Test Generation for Prompts

[NeurIPS 2024] Official code for HourVideo: 1-Hour Video Language Understanding

Vivaria is METR's tool for running evaluations and conducting agent elicitation research.