CALT — Copilot Agent Lint Tool

Lint, validate, and analyze Microsoft 365 Copilot Agent configurations — locally or directly from your tenant.

CALT helps Makers and Pro Developers check the quality of their Declarative Agent, Copilot Studio Agent, and Custom Engine Agent configurations before they deploy.

Note: Scanning Agent Builder agents from your tenant uses a beta Microsoft Graph API endpoint (/beta/copilot/admin/catalog/packages). This endpoint may not yet be available in all tenants or regions. Local file scanning works independently of the Graph API.

Minimal Path to Awesome

npm install -g calt-cli          # 1. Install
calt scan ./declarativeAgent.json # 2. Scan — done!

Quick Start

npm install -g calt-cli

Local scanning (no setup required)

Scan local Declarative Agent manifests right away — no authentication or configuration needed:

# Scan a manifest file
calt scan ./declarativeAgent.json

# Scan a project folder (auto-detects appPackage/ etc.)
calt scan ./my-agent-project

# Scan with auto-fix
calt scan --fix ./declarativeAgent.json

Tenant scanning (requires setup)

To scan agents deployed in your M365 tenant (Agent Builder + Copilot Studio), you need to register an Entra App first.

Note: The Graph API endpoint for tenant scanning (/beta/copilot/admin/catalog/packages) requires an Entra admin role — AI Administrator (recommended) or Global Administrator. Regular user accounts will get a 403 Forbidden even with admin-consented permissions. This is a Microsoft Graph API limitation — there is currently no user-scoped endpoint for reading declarative agent manifests.

Alternative for non-admin users: Export your agent from Agent Builder and scan it locally with calt scan ./declarativeAgent.json.

There are two ways to set up:

Option A: Automated setup (requires Azure CLI)

az login                    # Login to Azure CLI first
calt setup --login          # Creates Entra App, config, and logs in

Option B: Manual setup (no Azure CLI needed)

Azure Portal → Entra ID → App registrations → New registration
Authentication → Add platform → Mobile and desktop applications → enable https://login.microsoftonline.com/common/oauth2/nativeclient
Authentication → Advanced settings → Allow public client flows → Yes (required for Device Code Flow)
API permissions → Add → Microsoft Graph → Delegated → CopilotPackages.Read.All → click Grant admin consent
For Copilot Studio: also add Dynamics CRM → Delegated → user_impersonation
Run:

calt init
calt login --client-id <YOUR_APP_ID> --tenant <YOUR_TENANT_ID>

Once authenticated, you can scan your tenant:

# List all agents in your tenant
calt fetch --list

# Scan all Agent Builder agents
calt scan --remote --all --type agent-builder

# Scan all Copilot Studio agents
calt scan --remote --all --type copilot-studio

# Scan a specific agent by ID
calt scan --remote --id T_cebfd158-7116-1e34-27f5-0efca5f046f0

# Download all agents locally
calt fetch --all --output ./agents/

CLI Commands

Scan

calt scan ./declarativeAgent.json              # Scan a local manifest
calt scan ./my-agent-project                   # Scan a project folder
calt scan                                      # Scan current directory
calt scan --format json                        # JSON output (for CI/CD)
calt scan --format sarif                       # SARIF output (for GitHub Code Scanning)
calt scan --verbose                            # Show all passed checks
calt scan --fix ./declarativeAgent.json        # Auto-fix issues
calt scan --remote --all                       # Scan all tenant agents
calt scan --remote --all --type copilot-studio # Scan Copilot Studio agents only
calt scan --remote --all --type agent-builder  # Scan Agent Builder agents only

Lint — instruction quality only

calt lint ./declarativeAgent.json
calt lint ./declarativeAgent.json --format json

Validate — schema only

calt validate ./declarativeAgent.json

Fix — auto-fix issues

calt fix ./declarativeAgent.json               # Apply fixes
calt fix --dry-run ./declarativeAgent.json     # Preview without modifying

Diff — compare two agents

calt diff ./agentA.json ./agentB.json          # Local vs local
calt diff ./local.json T_cebfd158-...          # Local vs remote

Fetch — download agents from tenant

calt fetch --list                              # List all agents
calt fetch --list --type copilot-studio        # List Copilot Studio agents
calt fetch --list --type agent-builder         # List Agent Builder agents
calt fetch --id T_cebfd158-... --output ./     # Download specific agent
calt fetch --all --output ./agents/            # Download all agents

Login / Logout

calt login                                     # Interactive login (Device Code Flow)
calt login --client-id <ID> --tenant <TID>     # With specific app
calt login --status                            # Check login status
calt login --status --verbose                  # Show token details
calt logout                                    # Clear cached tokens

Setup / Init

calt setup                                     # Automated Entra App registration (needs Azure CLI)
calt setup --login                             # Setup + login in one step
calt init                                      # Create .caltrc.json config only

# Scaffold a new agent manifest from template
calt init --template basic                     # Standard agent with instructions + starters
calt init --template enterprise                # Enterprise agent with SharePoint, WebSearch, security
calt init --template minimal                   # Bare minimum valid manifest

# Custom output path
calt init --template basic --output agents/my-agent.json

Report — export results

calt report ./declarativeAgent.json --format json
calt report ./declarativeAgent.json --format markdown
calt report ./declarativeAgent.json --format sarif
calt report ./declarativeAgent.json --format html --output report.html
calt report --remote --all --format markdown

Watch — re-scan on file changes

calt watch ./declarativeAgent.json             # Watch and re-scan on changes
calt watch ./my-agent-project                  # Watch a project directory
calt watch --format json                       # Watch with JSON output
calt watch --verbose                           # Watch with detailed output

Eval — Copilot Studio evaluation test cases

Manage Copilot Studio agent evaluation test sets as code: author them locally as YAML/JSON, push them straight to Copilot Studio, trigger eval runs, and stream results to the CLI.

Experimental: calt eval push and calt eval run use undocumented Copilot Studio APIs (discovered via the Copilot Studio web client). They require the --experimental-direct-api flag and may break without notice. The local commands (init, validate, export, import) are stable.

Auth: A single calt login is enough — CALT pre-warms Graph, Dataverse, Power Platform (PVA) and BAP tokens in one device-code flow. Your Entra app must have Power Virtual Agents Service (user_impersonation) added; calt setup adds it automatically.

# 1. Author a test suite locally (creates .calt/evals/sample.eval.yaml)
calt eval init

# 2. Validate schema and field lengths
calt eval validate ./.calt/evals/sample.eval.yaml

# 3a. Local CSV export (works with the Copilot Studio import UI)
calt eval export ./.calt/evals/sample.eval.yaml --output ./out/testset.csv

# 3b. Direct push to Copilot Studio (experimental)
calt eval push ./.calt/evals/sample.eval.yaml \
  --bot-id <BOT_ID> --env-id <ENV_ID> \
  --experimental-direct-api                   # dry-run by default
calt eval push ./.calt/evals/sample.eval.yaml \
  --bot-id <BOT_ID> --env-id <ENV_ID> \
  --experimental-direct-api --apply           # actually create/update test cases
calt eval push ./.calt/evals/sample.eval.yaml \
  --bot-id <BOT_ID> --env-id <ENV_ID> \
  --experimental-direct-api --apply --prune   # also delete remote-only cases

# 4. Trigger an eval run and stream results
calt eval run ./.calt/evals/sample.eval.yaml \
  --bot-id <BOT_ID> --env-id <ENV_ID> \
  --experimental-direct-api
# → resolves test set by suite name, reuses last run's mcsConnectionId,
#   polls until Completed/Failed/Cancelled and prints PASS/FAIL per test case

Suite file format (YAML or JSON):

name: Weather Agent – Core Scenarios
description: Smoke tests for the weather info agent
testCases:
  - id: TC-001
    question: What's the weather in Vienna right now?
    expectedResponse: Provides current temperature, conditions and wind for Vienna.
    testingMethod: General quality       # or "Compare meaning" | "Similarity"

The regional PowerVA gateway host is auto-discovered via BAP; override with --gateway-host if needed. Use --test-set-id, --mcs-connection-id, --run-name, --poll-ms, --timeout-ms to override defaults.

Exit Codes

Code	Meaning
`0`	All checks passed successfully
`1`	Lint or validation errors found in agent configuration
`2`	CLI error (network failure, file not found, invalid arguments)

Configuration

Run calt init to create a .caltrc.json:

{
  "rules": {},
  "instruction_min_length": 200,
  "instruction_ideal_range": [500, 4000],
  "custom_blocked_phrases": [],
  "require_conversation_starters_min": 2,
  "schema_version_target": "v1.6",
  "graph_api": {
    "client_id": "",
    "tenant_id": ""
  },
  "dataverse": {
    "org_urls": []
  }
}

Override rule severities or turn rules off:

{
  "rules": {
    "INST-005": "off",
    "INST-010": "warning"
  }
}

Lint Rules Reference

Instruction Rules

Rule ID	Name	Severity	What it checks
`INST-001`	instructions-not-empty	error	Instructions must not be empty
`INST-002`	instructions-max-length	error	Max 8,000 characters
`INST-003`	instructions-min-length	warning	Fewer than 200 chars is probably too short
`INST-004`	has-purpose-section	warning	Should have a clear objective/purpose
`INST-005`	has-markdown-structure	info	Recommends Markdown headers and lists
`INST-006`	uses-actionable-verbs	warning	Should use specific verbs (search, create, display...)
`INST-007`	avoids-vague-language	warning	Flags "maybe", "try to", "if possible", "sometimes"
`INST-008`	has-workflow-steps	info	Recommends numbered steps or a workflow section
`INST-009`	references-capabilities	warning	Configured capabilities should be mentioned in instructions
`INST-010`	has-examples	info	Recommends few-shot examples
`INST-011`	has-error-handling	info	Recommends error/fallback guidance
`INST-012`	has-output-format-rules	info	Recommends response formatting rules
`INST-013`	no-conflicting-instructions	warning	Detects "always X" vs "never X" contradictions
`INST-014`	avoids-negative-framing	info	Prefers "Do X" over "Don't do Y"
`INST-015`	conversation-starters-match	warning	Starters should be relevant to instruction content

Schema Rules

Rule ID	Name	Severity
`SCHEMA-001`	valid-schema	error
`SCHEMA-002`	schema-version	info
`SCHEMA-003`	required-fields	error
`SCHEMA-004`	name-length	error
`SCHEMA-005`	description-length	error

Knowledge Source Rules

Rule ID	Name	Severity
`KNOW-001`	sharepoint-url-valid	error
`KNOW-002`	sharepoint-ids-present	info
`KNOW-003`	websearch-site-limit	error
`KNOW-004`	websearch-url-valid	error
`KNOW-005`	connector-id-present	error

Action Rules

Rule ID	Name	Severity
`ACT-001`	action-file-exists	error
`ACT-002`	action-id-unique	error

Conversation Starter Rules

Rule ID	Name	Severity
`CS-001`	starter-min-count	warning
`CS-002`	starter-max-count	error
`CS-003`	starter-has-text	error
`CS-004`	starter-no-duplicates	warning

GitHub Action

Use CALT directly in your GitHub Actions workflow to lint agent manifests on every PR.

Quick Start

# .github/workflows/agent-lint.yml
name: Lint Copilot Agents
on: [push, pull_request]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: stephanbisser/calt@main
        with:
          path: ./agents/

With SARIF Upload (GitHub Code Scanning)

jobs:
  lint:
    runs-on: ubuntu-latest
    permissions:
      security-events: write
    steps:
      - uses: actions/checkout@v4
      - uses: stephanbisser/calt@main
        with:
          path: ./agents/
          sarif-upload: "true"

Inputs

Input	Description	Default
`path`	Path to agent manifest or directory	`.`
`format`	Output format (`terminal`, `json`, `sarif`)	`terminal`
`fail-on`	Minimum severity to fail (`error`, `warning`, `off`)	`error`
`config`	Path to `.caltrc.json`	—
`sarif-upload`	Upload SARIF to GitHub Code Scanning	`false`

Outputs

Output	Description
`exit-code`	Exit code (0=pass, 1=errors, 2=crash)
`sarif-file`	Path to SARIF file (if generated)

CI/CD Integration

CALT exits with code 1 when errors are found, making it usable as a CI gate:

# GitHub Actions example
- name: Lint Copilot Agents
  run: npx calt-cli scan ./agents/ --format json

SARIF Integration

Generate SARIF output for integration with GitHub Code Scanning, Azure DevOps, or VS Code:

calt scan ./agent.json --format sarif > results.sarif

Upload to GitHub Code Scanning via the github/codeql-action/upload-sarif action:

- name: Scan Copilot Agent
  run: npx calt-cli scan ./agents/ --format sarif > results.sarif
  continue-on-error: true
- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: results.sarif

Project Structure

calt/
├── src/
│   ├── index.ts                    # CLI entry point (commander)
│   ├── commands/                   # scan, lint, validate, fix, diff, fetch, login, init, setup, report, watch
│   ├── core/
│   │   ├── types.ts                # All TypeScript interfaces
│   │   ├── index.ts                # Public API (for VS Code extension reuse)
│   │   ├── config-loader.ts        # .caltrc.json
│   │   ├── project-detector.ts     # Auto-detect project type
│   │   └── manifest-loader.ts      # Load from file or Graph API
│   ├── graph/
│   │   ├── auth.ts                 # MSAL Device Code Flow + token cache
│   │   ├── client.ts               # Graph API client with pagination
│   │   └── transform.ts            # Graph response → manifest
│   ├── dataverse/
│   │   ├── auth.ts                 # Dataverse authentication
│   │   ├── client.ts               # Dataverse API client
│   │   └── transform.ts            # Bot → manifest transform
│   ├── rules/
│   │   ├── rule-engine.ts          # Orchestrates all checks
│   │   ├── schema/                 # AJV validation + embedded JSON schemas
│   │   ├── instructions/           # INST-001 through INST-015
│   │   ├── knowledge/              # SharePoint, WebSearch, Connector rules
│   │   ├── actions/                # Plugin file checks
│   │   ├── conversation-starters/  # Starter validation
│   │   └── security/               # OWASP LLM Top 10 rules
│   ├── formatters/                 # terminal (chalk), json, markdown, html, sarif
│   └── utils/                      # URL validator, Markdown parser
├── schemas/
│   └── config.schema.json          # JSON Schema for .caltrc.json
├── tests/
│   ├── fixtures/                   # Sample manifests + Graph API response
│   ├── core/                       # project-detector, config-loader, markdown-parser, manifest-loader
│   ├── graph/                      # transform tests
│   ├── rules/                      # All rule tests + rule-engine integration
│   └── utils/                      # URL validator tests
├── package.json
├── tsconfig.json
└── vitest.config.ts

Development

# Clone and install
git clone https://github.com/stephanbisser/calt.git
cd calt
npm install

# Run in dev mode (no build step, uses tsx)
npm run dev -- scan ./tests/fixtures/valid-manifest.json

# Build to dist/
npm run build

# Run tests
npm test

# Lint
npm run lint

Token Security

Authentication tokens are stored in ~/.calt/token-cache.json with file permissions 0600 (owner-only read/write). Do not commit this directory to version control.

Programmatic API

CALT can be used as a library (e.g., from a VS Code extension or custom tooling).

Installation

npm install calt-cli

Import

import {
  loadConfig,
  loadFromFile,
  runFullScan,
  runInstructionLint,
  runSchemaValidation,
  applyFixes,
  diffManifests,
  formatAsJson,
  formatAsMarkdown,
  formatAsHtml,
  formatAsSarif,
  detectProject,
} from "calt-cli";

Exported Functions

Function	Description
`loadConfig(dir?)`	Load and merge `.caltrc.json` with defaults
`loadFromFile(path)`	Load an agent manifest from a local file
`loadFromRemote(id, token)`	Fetch a single agent from Microsoft Graph API
`loadAllFromRemote(token)`	Fetch all agents from a tenant via Graph API
`loadFromDataverse(orgUrl, token)`	Load a Copilot Studio agent from Dataverse
`listDataverseBots(orgUrl, token)`	List all Copilot Studio bots in a Dataverse org
`runFullScan(agent, config)`	Run all checks (schema + instructions + knowledge + actions + starters)
`runInstructionLint(agent, config)`	Run instruction-quality checks only
`runSchemaValidation(agent, config)`	Run schema validation only
`applyFixes(agent, descriptors)`	Auto-fix detected issues
`diffManifests(a, b)`	Structured diff between two agent manifests
`detectProject(dir)`	Auto-detect project type (Teams Toolkit, Agents Toolkit, etc.)

Example

import { loadConfig, loadFromFile, runFullScan, formatAsJson } from "calt-cli";

const config = await loadConfig();
const agent = await loadFromFile("./appPackage/declarativeAgent.json");
const report = await runFullScan(agent, config);

if (report.error_count > 0) {
  console.error(formatAsJson(report));
  process.exit(1);
}
console.log(`✅ ${agent.metadata.name}: no errors found`);

Types such as AgentLensConfig, ScanReport, RuleResult, LoadedAgent, FixDescriptor, DiffReport, and others are also exported for full type safety.

Contributing

See CONTRIBUTING.md for guidelines.

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
.github		.github
schemas		schemas
src		src
tests		tests
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
ROADMAP.md		ROADMAP.md
SECURITY.md		SECURITY.md
action.yml		action.yml
eslint.config.js		eslint.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Folders and files

Latest commit

History

Repository files navigation

CALT — Copilot Agent Lint Tool

Minimal Path to Awesome

Quick Start

Local scanning (no setup required)

Tenant scanning (requires setup)

CLI Commands

Scan

Lint — instruction quality only

Validate — schema only

Fix — auto-fix issues

Diff — compare two agents

Fetch — download agents from tenant

Login / Logout

Setup / Init

Report — export results

Watch — re-scan on file changes

Eval — Copilot Studio evaluation test cases

Exit Codes

Configuration

Lint Rules Reference

Instruction Rules

Schema Rules

Knowledge Source Rules

Action Rules

Conversation Starter Rules

GitHub Action

Quick Start

With SARIF Upload (GitHub Code Scanning)

Inputs

Outputs

CI/CD Integration

SARIF Integration

Project Structure

Development

Token Security

Programmatic API

Installation

Import

Exported Functions

Example

Contributing

License

About

Resources

License

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages