Skip to content

Add skill-coverage PR workflow with slash command support#537

Closed
Evangelink wants to merge 1 commit into
dotnet:mainfrom
Evangelink:skill-coverage-pr-workflow
Closed

Add skill-coverage PR workflow with slash command support#537
Evangelink wants to merge 1 commit into
dotnet:mainfrom
Evangelink:skill-coverage-pr-workflow

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Summary

Adds a GitHub Actions workflow that runs Measure-SkillCoverage.ps1 on PRs and posts a coverage report as a PR comment.

Trigger modes

Automatic on PR — triggers when files under plugins/** or tests/** change. Detects the specific plugin/skill pairs affected by the diff and runs coverage analysis only for those.

Slash command — comment /skill-coverage on a PR:

Command Behavior
/skill-coverage Runs on changed plugins/skills detected from the PR diff
/skill-coverage <plugin> Runs on all skills in the specified plugin
/skill-coverage <plugin> <skill> Runs on a specific skill

What the workflow does

  1. Detects changed plugin/skill pairs from the PR diff (or uses explicit arguments from the slash command)
  2. Runs eng/skill-coverage/Measure-SkillCoverage.ps1 with JSON output for each target
  3. Posts (or updates) a summary table as a PR comment with per-skill coverage percentages
  4. Includes expandable details for uncovered teaching points
  5. Writes the report to GitHub Actions step summary

- Auto-triggers on PRs that change plugins/** or tests/**
- Detects specific plugin/skill pairs from the PR diff
- Runs Measure-SkillCoverage.ps1 and posts a coverage report as a PR comment
- Supports /skill-coverage slash command on PR comments:
  /skill-coverage              (runs on changed skills in the PR)
  /skill-coverage <plugin>     (runs on all skills in the plugin)
  /skill-coverage <plugin> <skill>  (runs on a specific skill)
Copilot AI review requested due to automatic review settings April 17, 2026 09:00
@github-actions

Copy link
Copy Markdown
Contributor

Note

This PR is from a fork and modifies infrastructure files (eng/ or .github/).

Changes to infrastructure typically need to be submitted from a branch in dotnet/skills (not a fork) so that CI workflows run with the correct permissions and secrets.

Please consider recreating this PR from an upstream branch. If you don't have push access to dotnet/skills, ask a maintainer to push your branch for you.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new GitHub Actions workflow to measure per-skill “teaching point” coverage for PRs, with both automatic PR triggers (scoped by changed files) and an on-demand /skill-coverage slash command that posts results back to the PR.

Changes:

  • Introduces .github/workflows/skill-coverage.yml to detect affected plugin/skill pairs from PR diffs and run eng/skill-coverage/Measure-SkillCoverage.ps1.
  • Adds slash-command parsing and PR comment posting/updating with a markdown summary + uncovered details.
Show a summary per file
File Description
.github/workflows/skill-coverage.yml New workflow to compute skill coverage on PRs and via /skill-coverage, and post/update a PR comment with results.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (5)

.github/workflows/skill-coverage.yml:27

  • The startsWith(github.event.comment.body, '/skill-coverage') predicate matches /skill-coverageX (and the bash parsing will accept it). If you intend to only accept /skill-coverage followed by whitespace or end-of-line, tighten the check (e.g., regex match) to avoid accidental triggers.
    if: >-
      github.event_name == 'pull_request' ||
      (github.event_name == 'issue_comment' &&
       github.event.issue.pull_request &&
       startsWith(github.event.comment.body, '/skill-coverage'))
    runs-on: ubuntu-latest

.github/workflows/skill-coverage.yml:179

  • $anyFailed is initialized but never used, which makes the step harder to maintain and suggests error handling may be incomplete. Either remove it or use it to control the step’s final exit code if you want the workflow to fail when any target can’t be analyzed.
          $targets = $env:TARGETS_JSON | ConvertFrom-Json
          $allJson = @()
          $anyFailed = $false
          $summary = @()

.github/workflows/skill-coverage.yml:18

  • The workflow uses the Issues API (reactions.createForIssueComment, issues.createComment, issues.updateComment, issues.listComments), but the job permissions only grant pull-requests: write. This commonly fails with 403 because PR comments are issue comments under the hood; add issues: write (and reactions: write if you want to scope narrowly) to the workflow/job permissions.
permissions:
  contents: read
  pull-requests: write

.github/workflows/skill-coverage.yml:90

  • For issue_comment triggers this checkout uses ref: <branch-name> in the base repository. That won’t work for fork PRs (the branch lives in the fork), so /skill-coverage will fail to check out the PR head. Use the PR head SHA with the refs/pull/<num>/head fetch pattern (as in evaluation.yml), or set repository: to the PR head repo when checking out.
      - name: Get PR ref for comment trigger
        id: pr-ref
        if: github.event_name == 'issue_comment'
        uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9
        with:
          script: |
            const pr = await github.rest.pulls.get({
              owner: context.repo.owner,
              repo: context.repo.repo,
              pull_number: context.issue.number
            });
            core.setOutput('ref', pr.data.head.ref);
            core.setOutput('sha', pr.data.head.sha);
            core.setOutput('base_sha', pr.data.base.sha);
            core.setOutput('pr_number', context.issue.number);

      - name: Checkout repository
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
        with:
          ref: ${{ github.event_name == 'issue_comment' && steps.pr-ref.outputs.ref || '' }}
          fetch-depth: 0
          persist-credentials: false

.github/workflows/skill-coverage.yml:193

  • Measure-SkillCoverage.ps1 -Format Json emits pretty-printed multi-line JSON (ConvertTo-Json without -Compress). Capturing the script output into $result and piping $result | ConvertFrom-Json will treat each line as a separate pipeline input and typically fails to parse. Join the output into a single string (e.g., Out-String) before ConvertFrom-Json, or update the script/workflow so JSON output is compressed/single-line.
            try {
              $result = & "$env:GITHUB_WORKSPACE/eng/skill-coverage/Measure-SkillCoverage.ps1" @args
              $parsed = $result | ConvertFrom-Json

              # Handle single or array results
              $reports = if ($parsed -is [array]) { $parsed } else { @($parsed) }
  • Files reviewed: 1/1 changed files
  • Comments generated: 1

Comment thread .github/workflows/skill-coverage.yml
@Evangelink Evangelink closed this Apr 17, 2026
@github-actions github-actions Bot mentioned this pull request Jun 4, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants