Add skill-coverage PR workflow with slash command support#537
Add skill-coverage PR workflow with slash command support#537Evangelink wants to merge 1 commit into
Conversation
- Auto-triggers on PRs that change plugins/** or tests/** - Detects specific plugin/skill pairs from the PR diff - Runs Measure-SkillCoverage.ps1 and posts a coverage report as a PR comment - Supports /skill-coverage slash command on PR comments: /skill-coverage (runs on changed skills in the PR) /skill-coverage <plugin> (runs on all skills in the plugin) /skill-coverage <plugin> <skill> (runs on a specific skill)
|
Note This PR is from a fork and modifies infrastructure files ( Changes to infrastructure typically need to be submitted from a branch in Please consider recreating this PR from an upstream branch. If you don't have push access to |
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow to measure per-skill “teaching point” coverage for PRs, with both automatic PR triggers (scoped by changed files) and an on-demand /skill-coverage slash command that posts results back to the PR.
Changes:
- Introduces
.github/workflows/skill-coverage.ymlto detect affected plugin/skill pairs from PR diffs and runeng/skill-coverage/Measure-SkillCoverage.ps1. - Adds slash-command parsing and PR comment posting/updating with a markdown summary + uncovered details.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/skill-coverage.yml | New workflow to compute skill coverage on PRs and via /skill-coverage, and post/update a PR comment with results. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (5)
.github/workflows/skill-coverage.yml:27
- The
startsWith(github.event.comment.body, '/skill-coverage')predicate matches/skill-coverageX(and the bash parsing will accept it). If you intend to only accept/skill-coveragefollowed by whitespace or end-of-line, tighten the check (e.g., regex match) to avoid accidental triggers.
if: >-
github.event_name == 'pull_request' ||
(github.event_name == 'issue_comment' &&
github.event.issue.pull_request &&
startsWith(github.event.comment.body, '/skill-coverage'))
runs-on: ubuntu-latest
.github/workflows/skill-coverage.yml:179
$anyFailedis initialized but never used, which makes the step harder to maintain and suggests error handling may be incomplete. Either remove it or use it to control the step’s final exit code if you want the workflow to fail when any target can’t be analyzed.
$targets = $env:TARGETS_JSON | ConvertFrom-Json
$allJson = @()
$anyFailed = $false
$summary = @()
.github/workflows/skill-coverage.yml:18
- The workflow uses the Issues API (
reactions.createForIssueComment,issues.createComment,issues.updateComment,issues.listComments), but the job permissions only grantpull-requests: write. This commonly fails with 403 because PR comments are issue comments under the hood; addissues: write(andreactions: writeif you want to scope narrowly) to the workflow/job permissions.
permissions:
contents: read
pull-requests: write
.github/workflows/skill-coverage.yml:90
- For
issue_commenttriggers this checkout usesref: <branch-name>in the base repository. That won’t work for fork PRs (the branch lives in the fork), so/skill-coveragewill fail to check out the PR head. Use the PR head SHA with therefs/pull/<num>/headfetch pattern (as inevaluation.yml), or setrepository:to the PR head repo when checking out.
- name: Get PR ref for comment trigger
id: pr-ref
if: github.event_name == 'issue_comment'
uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9
with:
script: |
const pr = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number
});
core.setOutput('ref', pr.data.head.ref);
core.setOutput('sha', pr.data.head.sha);
core.setOutput('base_sha', pr.data.base.sha);
core.setOutput('pr_number', context.issue.number);
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ github.event_name == 'issue_comment' && steps.pr-ref.outputs.ref || '' }}
fetch-depth: 0
persist-credentials: false
.github/workflows/skill-coverage.yml:193
Measure-SkillCoverage.ps1 -Format Jsonemits pretty-printed multi-line JSON (ConvertTo-Jsonwithout-Compress). Capturing the script output into$resultand piping$result | ConvertFrom-Jsonwill treat each line as a separate pipeline input and typically fails to parse. Join the output into a single string (e.g.,Out-String) beforeConvertFrom-Json, or update the script/workflow so JSON output is compressed/single-line.
try {
$result = & "$env:GITHUB_WORKSPACE/eng/skill-coverage/Measure-SkillCoverage.ps1" @args
$parsed = $result | ConvertFrom-Json
# Handle single or array results
$reports = if ($parsed -is [array]) { $parsed } else { @($parsed) }
- Files reviewed: 1/1 changed files
- Comments generated: 1
Summary
Adds a GitHub Actions workflow that runs
Measure-SkillCoverage.ps1on PRs and posts a coverage report as a PR comment.Trigger modes
Automatic on PR — triggers when files under
plugins/**ortests/**change. Detects the specific plugin/skill pairs affected by the diff and runs coverage analysis only for those.Slash command — comment
/skill-coverageon a PR:/skill-coverage/skill-coverage <plugin>/skill-coverage <plugin> <skill>What the workflow does
eng/skill-coverage/Measure-SkillCoverage.ps1with JSON output for each target