The LLM Evaluation Framework

[NeurIPS D&B '25] The one-stop repository for LLM unlearning

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models

Run a prompt against all, or some, of your models running on Ollama. Creates web pages with the output, performance statistics and model info. All in a single Bash shell script.

Measure of estimated confidence for non-hallucinative nature of outputs generated by Transformer-based Language Models.

Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.

Tools for systematic large language model evaluations

insideLLMs is a Python library and CLI for comparing LLM behaviour across models using shared probes and datasets. The harness is deterministic by design, so you can store run artefacts and reliably diff behaviour in CI.

VerifyAI is a simple UI application to test GenAI outputs

In this we evaluate the LLM responses and find accuracy

Your prompts need tests too. Run prompts against real datasets, score outputs with LLM judges, version everything, and compare runs to see what got better.

Deepeval AI context pack for Claude Code, Codex, Cursor, and Aider: AGENTS.md, CLAUDE.md, prompts, evals, pitfalls, and verification notes for confident-ai/deepeval.

Offical implementation for Spectral Scaling Laws (EMNLP 2025)

This repo is for an streamlit application that provides a user-friendly interface for evaluating large language models (LLMs) using the beyondllm package.

Benchmark LLM accuracy, latency, cost, and hallucination rates across models with this open-source evaluation suite.