TrainLoop Evals is a comprehensive LLM evaluation framework designed for developers who need simple, vendor-independent evaluation tools.

• Simplicity First – One environment variable, one function call, one folder of JSON files
• Vendor Independence – Everything stored as newline-delimited JSON; no databases required
• Type-Safe & Extensible – In-code tests with full TypeScript support and composable system
• Meet Developers Where They Are – Works with existing workflows and bespoke loops

# Install the CLI
pipx install trainloop-cli

# Create a workspace
trainloop init

# Set your data path
export TRAINLOOP_DATA_FOLDER=/path/to/data

# Run evaluations
trainloop eval

# View results
trainloop studio

For comprehensive documentation, installation guides, tutorials, and API reference:

👉 evals.docs.trainloop.ai

👉 DeepWiki - lets you chat directly with this codebase rather than wading through documentation. It's the purest form of talking to your docs.

• Getting Started - Installation and setup
• Quick Start Guide - Complete walkthrough
• SDK Guides - Python, TypeScript, and Go integration
• CLI Reference - Complete command documentation
• Contributing - Development setup and guidelines

• Demo Repository: chat-ui-demo
• Live Demo: evals.trainloop.ai

• GitHub Issues - Bug reports and feature requests
• GitHub Discussions - Community support and questions
• Documentation - Comprehensive guides and tutorials

MIT

Need help? Check out our comprehensive documentation or open an issue.