Official Python SDK for the Agent2Agent (A2A) Protocol

The LLM Evaluation Framework

Open Source TypeScript AI Agent Framework

Validation Tools for A2A Agents

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

AI Observability & Evaluation

🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with OpenTelemetry, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

Evaluation and Tracking for LLM Experiments and AI Agents

Building a Secure and Interoperable Future for AI-Driven Payments.

🐢 Open-Source Evaluation & Testing library for LLM Agents

🪢 Langfuse Python SDK - Instrument your LLM app with decorators or low-level SDK and get detailed tracing/observability. Works with any LLM or framework

Build production-ready AI agents in both Python and Typescript.

Label Studio is a multi-type data labeling and annotation tool with standardized output format

Open-source observability for your LLM application, based on OpenTelemetry

Inspect: A framework for large language model evaluations

Examples and guides for using the OpenAI API

LlamaIndex is the leading framework for building LLM-powered agents over your data.

AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data…

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]

Introduction to LLM Evaluations

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and…

Adding guardrails to large language models.

Understanding Deep Learning - Simon J.D. Prince

Verifiers for LLM Reinforcement Learning

AG2 (formerly AutoGen): The Open-Source AgentOS. Join us at: https://discord.gg/pAbnFJrkgZ

The AI Toolkit for TypeScript. From the creators of Next.js, the AI SDK is a free open-source library for building AI-powered applications and agents

Supercharge Your LLM Application Evaluations 🚀

Simple Agentic AI system that leverages Semantic Kernel to orchestrate end-to-end workflow between user (human) and an agent equipped with tools. Evaluate the system through built-in evaluation met…

Build AI Agents, Visually