MLflow Skills for Coding Agents

Turn your favorite coding agent into an LLMOps expert with MLflow skills.

Build, debug, and evaluate GenAI applications with confidence. These skills give your AI coding assistant deep knowledge of MLflow's tracing, evaluation, and observability features.

Works with any coding agent that support Skills, including Claude Code, Cursor, Codex CLI, Gemini CLI, and OpenCode.

Building production-ready AI agents is hard. You need observability to understand what your agent is doing, evaluation to measure quality, and debugging tools when things go wrong. MLflow provides SDKs and best practices for all of these operations, and with skills we bring them directly into the environment where LLM agent development happens. Now you can go to your favorite coding agent and just ask:

• "Add tracing to my LangChain app" → Instruments your code automatically
• "Why did this trace fail?" → Analyzes spans, finds root causes, suggests fixes
• "Evaluate my agent's accuracy" → Sets up datasets, scorers, and runs evaluation
• "Improve my agent and verify your work" → Gives the coding agent reproducible and verifiable mechanism with MLflow eval to hill climb on quality
• "Show me token usage trends" → Queries metrics and analyze trends

npx skills add mlflow/skills

git clone https://github.com/mlflow/skills.git
cp -r mlflow-skills/* ~/.claude/skills/

Change the ~/.claude/skills/ directory to the appropriate location for your coding agent, e.g., ~/.codex/skills/ for Codex.

Add skills to your project for team sharing:

cd your-project
git clone https://github.com/mlflow/skills.git .skills/mlflow
# Or as a submodule:
git submodule add https://github.com/mlflow/skills.git .skills/mlflow

The hooks/ directory contains a UserPromptSubmit hook that automatically detects MLflow-related patterns in your prompts and surfaces the right skill before the agent responds — no need to remember which skill does what.

Step 1: Copy the hook somewhere permanent:

cp hooks/mlflow-suggest-hook.py ~/.claude/hooks/mlflow-suggest-hook.py

Step 2: Add it to your Claude Code settings (~/.claude/settings.json):

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "type": "command",
        "command": "python3 ~/.claude/hooks/mlflow-suggest-hook.py"
      }
    ]
  }
}

Step 3: Start a new session. When you ask something like "Add tracing to my app", you'll see:

💡 Use the `mlflow-trace-instrumentation` skill to add MLflow tracing.

See hooks/README.md for the full keyword-to-skill mapping and troubleshooting.

> Add MLflow tracing to my OpenAI app

The coding agent will:
1. Detect your LLM framework
2. Add the right autolog call
3. Configure experiment tracking
4. Verify traces are being captured

> Analyze trace tr-abc123 — why did it return the wrong answer?

The coding agent will:
1. Fetch the trace and examine all spans
2. Check assessments for quality signals
3. Walk the span tree to find the failure point
4. Correlate with your codebase to identify root cause
5. Suggest specific fixes

> Set up evaluation for my customer support agent

The coding agent will:
1. Discover existing evaluation datasets
2. Help create test cases if needed
3. Select appropriate scorers (correctness, relevance, etc.)
4. Run evaluation and analyze results
5. Generate improvement recommendations

> Show me token usage trends for the last 7 days

The coding agent will:
1. Query the MLflow metrics API
2. Generate time-series breakdowns
3. Identify cost spikes or anomalies
4. Suggest optimization opportunities

• MLflow 3.8+ (Instructions and code examples are tested with MLflow 3.8)

We welcome contributions! Here's how to help:

1. Report issues — Found a skill giving incorrect advice? Open an issue
2. Suggest improvements — Have ideas for better workflows? We'd love to hear them
3. Add new skills — See skill authoring best practices and file a pull request

The test code has two layers: live scenarios exercise skills through Claude; local verifier tests check that the graders accept valid work and reject broken work.

tests/
  configs/
    setup/       # Onboarding, docs, backend prerequisites
    tracing/     # Instrumentation, retrieval, diagnosis, metrics
    evals/       # Evaluation, scorers, alignment, verified repair
  judges/        # Graders used by live scenarios
  scripts/       # Scenario setup and seeded application data
  verifiers/     # Local regression tests for graders and harness
  test_skill.py  # Existing YAML scenario runner

Config paths for setup scripts and judges remain relative to the repository root. Run live scenarios from the repository root using their grouped paths below; the former flat config paths have moved.

The YAML scenarios in tests/configs/ run a real Claude Code participant in a fresh project and Claude home, install the selected repository skills plus the MLflow tracing plugin, and verify its work against a real MLflow server. Each scenario checks a successful native skill invocation or complete skill read, plus an observable task outcome. Task-only prompts test discovery among the skills installed for each scenario. No prompt names a skill or instructs the agent to load one. The dispatcher currently disables automatic invocation, so its required-loading check can expose that discovery limitation. Merely mentioning an API or reporting success does not pass.

Validated with MLflow 3.17.1. Requires an authenticated claude CLI, Git, Node/npm (for Claude tracing and TypeScript), and uv. Run a scenario:

uv run --with mlflow==3.17.1 --with litellm --with pytest --with langgraph --with langchain-core \
  python tests/test_skill.py tests/configs/evals/evaluation_nested.yaml

The original general evaluation scenario remains agent_evaluation.yaml: it clones REPO_URL and checks datasets, registered scorers, evaluation results, application traces, and both evaluation/tracing skill reads. Run it with REPO_URL=<repository-url> as a CLI override. The separate evaluation_nested.yaml is a deterministic local regression fixture.

Default timeouts, keep_workdir: true, and test_runs_dir: /tmp need not be repeated. Fixtures contain application context, data, and actual deliverable interfaces; prompts are short user requests. Conversational answers are checked by an MLflow judge using JUDGE_MODEL (or its default), alongside backend and tool-evidence checks; participants need not write grading JSON. Specialized regression cases still have explicit operational constraints, such as approved smoke inputs and an unavailable backend.

Model-backed scenarios require working provider credentials. Set TEST_JUDGE_MODEL for scorer_model_criteria; for scorer_memalign, also set TEST_EMBEDDING_MODEL and add --with dspy. The repair scenario's existing LLM judge accepts JUDGE_MODEL. No credentials or private endpoint names belong in scenario files. Each run retains environment.json, skill-inventory.json, verification.json, participant transcripts, generated files, and the local MLflow store in its printed work directory. Setup failures and skill failures both return nonzero, with different exit codes.

To run the local scenarios sequentially, excluding those needing a model provider:

for config in tests/configs/*/*.yaml; do
  case "$config" in
    *scorer_model_criteria*|*scorer_memalign*|*agent_debug_trace_first*|*agent_evaluation*) continue ;;
  esac
  uv run --with mlflow==3.17.1 --with litellm --with pytest --with langgraph --with langchain-core \
    python tests/test_skill.py "$config" || exit 1
done

Supporting verifier regression checks run without Claude or model calls:

uv run --with mlflow==3.17.1 --with pytest --with langgraph --with langchain-core --with dspy \
  pytest -q tests/verifiers

Expected answers are stored outside the project but remain accessible on the same machine; this is not an adversarial sandbox. Investigation grading also requires observed MLflow retrieval. These scenarios are behavioral regression coverage, not a claim of exhaustive platform coverage or a baseline-versus-treatment comparison. Live SageMaker connection, Databricks UC storage/Review App/online monitoring, GEPA prompt optimization, distributed tracing, and every supported framework still require dedicated environments and scenarios. A successful local run does not verify those integrations. The repository currently has no CI workflow that launches the paid Claude scenarios automatically.

We recommend using the skill-creator from Anthropic's skills repository as a starting point.

Apache License 2.0 — see LICENSE for details.

• 📖 Read MLflow Documentation
• 📖 Read MLflow Skills Blog)
• ⭐️ Star MLflow on GitHub

_{Built with care for the MLflow community}