Build software better, together

Arize-ai / phoenix

AI Observability & Evaluation

openai datasets agents ai-monitoring ai-observability prompt-engineering llms langchain llmops anthropic llamaindex llm-eval evals llm-evaluation aiengineering smolagents

Updated Sep 20, 2026
Python

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

benchmark evaluation llm large-language-model llm-eval

Updated Sep 21, 2026
Python

Giskard-AI / giskard-oss

Sponsor

Star

🐢 Open-Source Evaluation & Testing library for LLM Agents

ai-security mlops fairness-ai responsible-ai ml-validation red-team-tools trustworthy-ai ml-testing llm ai-red-team ai-testing llmops llm-security llm-eval llm-evaluation rag-evaluation agent-evaluation

Updated Sep 21, 2026
Python

truera / trulens

Star

Evaluation and Tracking for LLM Experiments and AI Agents

machine-learning neural-networks ai-agents explainable-ml agentops ai-monitoring ai-observability llms llmops llm-eval evals llm-evaluation agent-evaluation

Updated Sep 21, 2026
Python

uptrain-ai / uptrain

Star

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

machine-learning monitoring evaluation experimentation jailbreak-detection autoevaluation root-cause-analysis prompt-engineering llmops openai-evals llm-prompting llm-eval llm-test hallucination-detection

Updated Aug 18, 2024
Python

athina-ai / athina-evals

Star

Python SDK for running evaluations on LLM generated responses

evaluation evaluation-metrics evaluation-framework llmops llm-eval llm-ops llm-evaluation llm-evaluation-toolkit

Updated Jun 6, 2025
Python

Re-Align / just-eval

Star

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

evaluation gpt4 llm llm-eval llm-evaluation llm-evaluation-toolkit

Updated Jan 29, 2024
Python

parea-ai / parea-sdk-py

Star

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework

Updated Feb 13, 2025
Python

whitecircle / circle-guard-bench

Star

First-of-its-kind AI benchmark for evaluating the protection capabilities of large language model (LLM) guard systems (guardrails and safeguards)

benchmarking benchmark ai jailbreak safeguard guardrail guardrails large-language-models llm large-language-model llm-security llm-eval llm-evaluation llm-as-a-judge llm-jailbreaks

Updated Mar 7, 2026
Python

grigio / llm-eval-simple

Star

llm-eval-simple is a simple LLM evaluation framework with intermediate actions and prompt pattern selection

llm llm-eval llm-evaluation-benchmark

Updated Feb 28, 2026
Python

izam-mohammed / ragrank

Sponsor

Star

🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.

machine-learning evaluation language-model rag llm prompt-engineering llmops llm-eval

Updated Aug 29, 2026
Python

multinear / multinear

Star

Develop reliable AI apps

reliability evaluation llm llms llm-eval llm-evaluation llms-benchmarking llm-evaluation-framework

Updated Sep 2, 2025
Python

Striveworks / valor

Star

Valor is a lightweight, numpy-based library designed for fast and seamless evaluation of machine learning models.

nlp computer-vision evaluation text-generation classification object-detection image-segmentation evaluation-metrics model-evaluation mlops llm-eval

Updated Sep 18, 2026
Python

harnexa / nexa-gauge

Star

An graph-eval framework for LLM's

rouge-metric ranking-algorithm redteam grounding llm blue-score llm-eval llm-evaluation-framework llm-judge relevance-scoring geval

Updated Aug 3, 2026
Python

alan-turing-institute / prompto

Star

An open source library for asynchronous querying of LLM endpoints

python nlp machine-learning natural-language-processing deep-learning transformers transformer hut23 large-language-models llms llm-eval llm-evaluation

Updated Jul 18, 2025
Python

paradime-io / dbt-llm-evals

Star

The warehouse-native LLM evaluation package for dbt™ - monitor AI quality without data egress

bigquery ai snowflake gemini openai dbt databricks dbt-packages vertex-ai llm dbt-core anthropic llm-eval llm-evaluation

Updated Feb 10, 2026
Python

genia-dev / vibraniumdome

Star

LLM Security Platform.

security openai prompts adversarial-attacks llm prompt-engineering chatgpt llmops large-language-model prompt-injection llm-serving llm-agent llm-security llm-inference llm-eval llm-framework prompt-injection-tool llm-evaluation llm-firewall

Updated Oct 28, 2024
Python

thedataquarry / structured-outputs

Star

Structured output benchmarks comparing DSPy and BAML with different LLMs

information-extraction structured-evaluation structured-output baml dspy llm llm-eval llm-evaluation

Updated Dec 23, 2025
Python

HumphreySun98 / repoagentbench

Star

SWE-bench for your codebase — mine your merged PRs into local, contamination-free coding-agent benchmarks. Adapters: claude-code, aider (Opus 4.7 / GPT-5.5 / Sonnet 4.6 / Gemini 3.1 Pro).

benchmark developer-tools ai-agents aider llm-eval coding-agents agent-evals swe-bench gemini-3-1-pro claude-opus-4-7 gpt-5-5

Updated Apr 30, 2026
Python

multivon-ai / multivon-eval

Star

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

python mlops ai-evaluation prompt-engineering llmops llm-eval evals llm-evaluation rag-evaluation hallucination-detection llm-as-judge agent-evaluation

Updated Sep 17, 2026
Python

Navigation Menu

llm-eval

Here are 119 public repositories matching this topic...

Arize-ai / phoenix

open-compass / opencompass

Giskard-AI / giskard-oss

truera / trulens

uptrain-ai / uptrain

athina-ai / athina-evals

Re-Align / just-eval

parea-ai / parea-sdk-py

whitecircle / circle-guard-bench

grigio / llm-eval-simple

izam-mohammed / ragrank

multinear / multinear

Striveworks / valor

harnexa / nexa-gauge

alan-turing-institute / prompto

paradime-io / dbt-llm-evals

genia-dev / vibraniumdome

thedataquarry / structured-outputs

HumphreySun98 / repoagentbench

multivon-ai / multivon-eval

Add this topic to your repo