ai-metrics

Open measurement infrastructure for AI agents. Our audit of 124 usage tools found 45+ verified billing bugs — 19 fixes landed upstream. Conformance fixtures for token accounting: PASS / FAIL / UNPROVABLE in CI.

WakaTime plugin for GitHub Copilot CLI - Track your AI usage, lines of code generated, and time spent prompting Copilot AI

VerifyAI is a verification harness for testing, auditing, and evaluating responses from AI assistants and coding agents.

Honest metrics for AI tools and features. Agent skills for Claude Code, Codex, Cursor, Copilot, Gemini CLI and 30+ others: designs outcome metrics from your business context, audits dashboards against 20 named vanity-metric anti-patterns, generates instrumentation.

A simple Spring Boot application demonstrating AI chat client observability with token usage metrics, prompt logging, and performance monitoring.

Modern data exploration interface for analyzing LLM response patterns and performance metrics - Built with React, TypeScript, and TanStack Router

A posthog wrapper for laravel with support for: product analytics, Feature flags, Error tracking, LLM analytics & Events

AI observability Grafana plugin tracking real-time LLM metrics — latency, cost, tokens, insights, and provider performance (Cerebras, LLAMA, MCP) with live 5s updates.

为 AI 产品设计事件模型、转化漏斗、质量成本指标与数据看板。

Measure whether an AI deployment improves the operation it joined.

Agentic Workflow Evaluation: Text Summarization Agent. This project includes an AI agent evaluation workflow using a text summarization model with OpenAI API and Transformers library. It follows an iterative approach: generate summaries, analyze metrics, adjust parameters, and retest to refine AI agents for accuracy, readability, and performance.

Measurement method for AI Foundations source-line fidelity, drift, override, and non-merge testing.

Multi-model LLM benchmarking and evaluation toolkit

Interactive dashboard with visualizations across AI development, geographic distribution, innovation, investment, and public perception.

Open framework for measuring AI value across engineering, support, analytics, ops, and internal copilots.

Experimental framework for testing and measuring AI system capabilities, reasoning patterns, and emergent behaviors

📊 Ultra-fast lock-free token, latency, cost tracking and evaluation engine for LLMs and AI agents.

AI Security Metrics - Accuracy Monitoring and Data Drift Detection

Weekly Reading Group in AI measurement science