Skip to content

feat(workflows): standardize reusable workflows as drop-in CI for all lgtm-hq repos #168

Open
11 of 11 issues completed
Open
feat(workflows): standardize reusable workflows as drop-in CI for all lgtm-hq repos#168
11 of 11 issues completed
Assignees
TurboCoder13
Labels
ciCI/CD infrastructurecriticalCritical priority - must fix immediatelyenhancementNew feature or request
@TurboCoder13

Description

@TurboCoder13
TurboCoder13
opened on May 15, 2026

Summary

Make lgtm-ci the CI brain for lgtm-hq repositories. Consumer repositories should keep only thin workflow callers plus genuinely repo-specific scripts; generic CI behavior belongs in lgtm-ci as reusable workflows, composite actions, and shell libraries.

This issue is the master umbrella for every adoption shortfall not fully owned by the narrower issues below:

Compatibility policy

We do not need to preserve backwards compatibility for consumer repositories. Consumers pin a specific lgtm-ci SHA; when they adopt a newer lgtm-ci version, they must adapt their workflows to the new contract.

  • lgtm-ci may rename or remove inputs, change artifact names/layouts, and restructure jobs without deprecation cycles.
  • Do not add aliases, compatibility modes, or fallback behavior solely to keep older consumer workflows working.
  • Prefer the clean reusable contract, then update py-lintro, Rustume, turbo-themes, and other consumers when they repin.
  • This issue tracks the target architecture, not shims for historical consumer workflow shapes.

Current adoption snapshot

py-lintro

py-lintro is the furthest along. It pins lgtm-ci v0.24.0 and uses many thin reusable callers, but still carries meaningful local CI logic.

  • Roughly 12 of 22 workflows are thin lgtm-ci callers.
  • 7 workflows remain fully local.
  • 3 workflows are substantial hybrids: test-ci.yml, publish-pypi-on-tag.yml / publish-testpypi.yml, and docker-ci.yml.
  • Major local debt: PyPI OIDC upload job orchestration, Docker CI fork fallback / publish paths, Homebrew binary/formula automation, org-ruleset check-name shims, local maintenance jobs, and duplicated harden-runner allowlists.

Rustume

Rustume main still pins lgtm-ci v0.19.2 and has only partial adoption. The Rustume 263-pages-model-b-coverage worktree moves to v0.24.0 and proves the Model B direction, but does so with local workarounds.

  • Main uses reusable deploy-pages and reusable-docker, but most workflows remain local.
  • The Model B worktree uses reusable-deploy-site-with-reports, reusable-test-node, and reusable-artifact-pr-comment.
  • Major local debt: Rust coverage HTML generation, web coverage artifact repack, local Rust coverage job mirroring lgtm-ci, local site-quality logic, local post-pr-comment composite in security workflow, and old comments blocking Node delegation from v0.19.2-era bugs.
  • feat(ci): Pages Model B coverage HTML artifacts for Rust and Node reusables #251 owns the Rust + Node coverage artifact cleanup.

turbo-themes

turbo-themes has not adopted lgtm-ci reusable workflows yet.

  • 0 workflows call lgtm-hq/lgtm-ci/.github/workflows/reusable-*.yml.
  • CI is implemented locally across quality, coverage, examples, E2E, Lighthouse, Pages deploy, publishing, release, security, and maintenance.
  • A canonical examples/bundle-manifest-turbo-themes.json exists in lgtm-ci, but the consumer repo currently implements equivalent Model B artifact bundling imperatively via scripts.
  • Major local debt: local SBOM reusable, local deploy/report bundling, local Vitest/E2E/Lighthouse/Python/Ruby/Swift coverage, local npm/PyPI/RubyGems publish flows, local release train, and unused local reusable-quality.yml / reusable-build.yml drift.

Target architecture

Consumer repos should converge on this shape:

jobs:
  quality:
    uses: lgtm-hq/lgtm-ci/.github/workflows/reusable-quality-lint.yml@<commit-sha>
    with:
      tooling-ref: <same-commit-sha>
      # repo-specific inputs only

Allowed consumer-local logic:

  • Thin workflow files that call lgtm-ci reusables.
  • Small repo-specific scripts passed as inputs, for genuinely repo-specific behavior.
  • Repo-specific build/test commands where the generic reusable already owns checkout, setup, hardening, artifact shape, comments, and reporting.

Not allowed long-term:

  • Consumer-local reusable-*.yml duplicating lgtm-ci patterns.
  • Inline shell in workflow YAML for generic CI behavior.
  • Consumer-local artifact repack jobs that only compensate for lgtm-ci artifact layout.
  • Per-repo copies of harden-runner allowlists, PR comment plumbing, Pages bundling, package publish wrappers, or fork-PR Docker fallback logic.

Workstreams and sub-tasks

1. Pinning, version drift, and release reliability

  • Document and automate the correct commit-SHA pinning model for workflow_call consumers; annotated tag object SHAs are not valid substitutes for the commit SHA.
  • Add Renovate guidance/snippets that update lgtm-ci pins with the commit SHA and version comment together.
  • Link or extend feat(ci): implement comprehensive version-drift prevention #98 for cross-repo version-drift prevention.
  • Link or extend feat(release): surface reusable release workflow failures #207 so reusable release workflow failures are surfaced clearly to consumers.
  • Ensure every reusable accepts and forwards tooling-ref consistently.
  • After lgtm-ci releases, consumer repos repin and adapt workflows; no backwards-compatibility shims.

2. Composite action local-tooling pattern

3. Egress, hardening, and permissions

  • Link or extend feat(ci): default reusable workflows to block egress with documented allowlist presets #204 to make block-mode egress practical by default.
  • Add documented harden-runner allowlist presets so consumers do not duplicate long allowed-endpoints blocks.
  • Upstream the useful pattern from py-lintro's unused harden-runner-preset action.
  • Document permission profiles by workflow mode: lint/test, PR comment, Pages, release, PyPI OIDC, GitHub Release, package publish.

4. Check-name and org-ruleset compatibility — DONE

Status: completed in lgtm-ci via PR #264 (merged 2026-05-30). Shared gate reusable-required-check.yml, assert-required-check.sh, docs, and BATS contract tests. Consumer shim removal and org ruleset repin remain follow-ups (py-lintro + org admin).

Progress (§12): Done — #263 merged (hybrid job display names, Node split, static matrix names, job-name on always-run jobs).

Summary

Org branch-protection rulesets often require fixed check display names. Reusable workflows report checks as caller_job_id / inner_job_name, which may not match legacy ruleset labels. Consumers (for example py-lintro) add local shim jobs that only re-assert upstream success under the ruleset name.

Problem

  • py-lintro shims: test-suite-coverage (🧪 Test Suite & Coverage), lintro-code-quality (🛠️ Lintro Code Quality) while work runs in reusable-test-python / reusable-quality-lint with different job-name values.
  • Hand-rolled shim YAML duplicates egress, hardening, and failure logic per repo.
  • §12 fixed skipped-job UI noise but does not resolve ruleset vs reusable name mismatch.

Proposed Solution

  • Decision: Do not add per-consumer job-name aliases inside work reusables. Prefer ruleset updates to match caller_id / job-name; until then use a shared gate reusable.
  • Add reusable-required-check.yml + assert-required-check.sh (aggregate-status gate).
  • Document org ruleset naming in docs/workflow-contract.md and docs/reusable-workflows.md.
  • Remove py-lintro shims after org rulesets accept repinned check paths (consumer + org admin).
  • Do not add legacy per-consumer job-name aliases beyond the chosen contract.

Implementation Notes

  • Gate reusable: job-name, upstream-result, optional passed-output / status-output, draft-pr-skip, drop-in tooling-ref + harden-runner.
  • BATS contract tests for workflow YAML and assert script.
  • §12 guardrail: block-scalar job.name continuation detection in validate-static-job-names.sh.

Benefits

  • Consumers drop copy-pasted shim jobs; one tested lgtm-ci contract.
  • Rulesets can migrate incrementally without forking work reusables.

References

  • §4 gate reusable: #264
  • §12 job display names: #263
  • py-lintro shim example: test-ci.yml, docker-ci.yml
  • Quality / test reusables: reusable-quality-lint.yml, reusable-test-python.yml

5. PyPI and Python release publishing — DONE

Status: completed in lgtm-ci via PR #266 (merged 2026-05-31), building on #232, #248, #249, #252.
OIDC upload must stay in a caller-local job (PyPI trusted publishing constraint);
there is no cross-repo reusable-publish-pypi.yml.

Summary

Provide a drop-in Python release contract: build sdist/wheel in a reusable, upload via
OIDC in a caller job, attach assets with reusable-github-release.yml.

Problem

  • Consumers duplicated build/upload/release glue and egress allowlists.
  • A monolithic publish reusable cannot perform PyPI OIDC from workflow_call
    (warehouse#11096).
  • Umbrella §5 originally targeted reusable-publish-pypi.yml; that path was superseded.

Proposed Solution

  • reusable-build-python-dist.yml — build-only reusable (preflight, uv build, twine check, artifact).
  • upload-pypi-oidc composite — caller-job upload with strict validate, OIDC, best-effort provenance.
  • reusable-github-release.yml — GitHub Release assets from the same artifact.
  • examples/publish-python-release.yml — canonical caller template.
  • Docs: docs/python-release-publish.md, docs/workflow-contract.md (split egress).
  • BATS contract tests for build, upload action, and github-release.
  • Consumer repin/migration (tracked separately; out of scope for lgtm-ci platform PRs).

Implementation Notes

  • Callers compose: build reusable → local pypi-upload job (harden-runner + upload-pypi-oidc) → github-release reusable.
  • Upload validation uses VALIDATE_STRICT=true when validate: true (default).
  • Product-specific jobs (Homebrew, Docker, SBOM gates) remain in consumer workflows.

Benefits

  • Single maintained build/upload/release contract across lgtm-hq Python packages.
  • Least-privilege OIDC boundary preserved; no false drop-in publish reusable.
  • Copy-paste example reduces caller drift.

References

6. Pages Model B and coverage/report artifacts

  • Complete feat(ci): Pages Model B coverage HTML artifacts for Rust and Node reusables #251 for Rust + Node flat HTML coverage artifacts.
  • Add or document flat artifact contracts for Python, Ruby, and Swift coverage HTML used by turbo-themes.
  • Move bundle-manifest-turbo-themes.json into turbo-themes when it adopts reusable-deploy-site-with-reports.
  • Ensure Model B manifests resolve lgtm-ci-produced artifact names, not repo-local repack artifacts.
  • Add integration tests proving bundle-workflow-artifacts copies each artifact into the expected site dest with index.html present.

7. Docker CI and GHCR maintenance

  • Link or extend feat(workflows): add reusable Docker health check testing between build and publish #65 for reusable Docker health/smoke checks between build and publish.
  • Upstream py-lintro's generic fork-PR Docker pattern: fork detection, disk cleanup, image tarball save/load, and artifact fallback.
  • Consolidate py-lintro's duplicate main-branch Docker publish path into reusable-docker.
  • Parameterize GHCR cleanup so py-lintro and Rustume can use reusable-ghcr-cleanup instead of local cleanup workflows/scripts.
  • Preserve only repo-specific smoke-test commands in consumers.

8. Security, vulnerability, and maintenance workflows

  • Make vuln-suppression checks reusable enough for py-lintro, Rustume, and turbo-themes.
  • Add a Renovate reusable workflow or document why Renovate remains repo-local.
  • Standardize PR comment cleanup / marker behavior through lgtm-ci actions rather than consumer-local scripts.
  • Ensure CodeQL, dependency review, scorecards, SBOM, action-pinning validation, semantic PR title, PR labeler, and auto-assign are documented as thin callers.

9. Node, E2E, Lighthouse, and visual regression

  • Make reusable-test-node fully suitable for turbo-themes and Rustume site-quality jobs after feat(ci): Pages Model B coverage HTML artifacts for Rust and Node reusables #251 (Vitest via reusable-test-node.yml; custom commands via reusable-test-node-custom.yml).
  • Add a dedicated Lighthouse reusable workflow; lgtm-ci currently has Lighthouse actions but no complete reusable workflow.
  • Extend Playwright reusables for visual-regression workflows and snapshot update/report contracts.
  • Ensure Playwright report artifacts have a stable flat index.html contract for Model B.
  • Document skipped-job behavior when custom test commands bypass default Vitest jobs (superseded by Node workflow split in §12 — no skipped Vitest/custom siblings).

10. Swift, Jekyll, and multi-package release trains

  • Add Swift test/coverage reusable support; reusable-release-version-pr knows about Swift, but there is no Swift CI/coverage reusable.
  • Decide whether Swift package publishing belongs in lgtm-ci or remains repo-specific.
  • Add a Jekyll build/test reusable or generic static-site build mode that handles turbo-themes examples.
  • Add multi-package release-train orchestration for repos publishing npm + PyPI + RubyGems + optional Swift from one release.
  • Link or extend feat(workflows): add reusable Rust binary cross-compilation workflow #69 for reusable Rust binary cross-compilation and release assets.
  • Link or extend feat(actions): add monorepo support with path-based change detection #36 for monorepo path-based change detection so multi-language workflows do not run unnecessarily.

Per-consumer migration tracks

py-lintro

  • Replace caller-local PyPI upload jobs with reusable-publish-pypi.yml once available.
  • Adopt reusable-publish-homebrew or split generic Homebrew pieces into lgtm-ci.
  • Collapse Docker CI onto reusable-docker + reusable fork fallback.
  • Remove org-ruleset shim jobs: adopt reusable-required-check.yml or repin org rulesets to real check paths (#264 §4).
  • Replace duplicated harden-runner allowlists with presets.
  • Revisit local harden-runner-preset, extract-version, and egress-audit-lite; upstream or delete.
  • Keep only genuinely py-lintro-specific logic: bundled tools image behavior, lintro dogfooding, and package-specific verification.

Rustume

  • Adopt feat(ci): Pages Model B coverage HTML artifacts for Rust and Node reusables #251 outputs and delete local Rust coverage HTML generation.
  • Delete web-coverage-pages-artifact once reusable-test-node emits flat Pages coverage artifacts.
  • Use lgtm-ci Rust coverage/build reusables once checkout order and HTML outputs are fixed.
  • Re-test site-quality delegation on current lgtm-ci; existing blocker comments cite v0.19.2 behavior.
  • Replace local post-pr-comment composite in security workflow with lgtm-ci.
  • Keep repo-specific scripts for Typst/PDF, WASM, Docker smoke checks, and Astro build details where needed.

turbo-themes

  • Replace local SBOM reusable with reusable-sbom.
  • Adopt reusable-codeql, reusable-scorecards, reusable-dependency-review, reusable-pr-labeler, reusable-pr-auto-assign, and reusable-validate-action-pinning.
  • Adopt reusable-deploy-site-with-reports and move the bundle manifest into turbo-themes.
  • Adopt Node/Vitest, Playwright/E2E, Lighthouse, and multi-language coverage reusables as they become sufficient.
  • Adopt reusable-publish-npm, reusable-publish-pypi, and reusable-publish-gem with monorepo build hooks.
  • Retire unused local reusable-quality.yml and reusable-build.yml.
  • Keep genuinely turbo-themes-specific maintenance bots: theme sync, snapshot refresh policy, generated token checks, and design-token release details.

Definition of Done

References

### 11. Coverage/test PR comment correctness (empty/misleading comments) — WON'T DO (superseded by §13)

Status: won't do as a standalone point fix. The narrow fix (PR #260, closed won't-merge) patched only the generate-coverage-comment missing-file path.
While scoping it we concluded the real problem is the divergent comment architecture:
Rust/Node use a separate generate-coverage-comment (multi-metric, file-driven) while Python/Node-default use generate-test-comment (tests + single coverage line), and Rust runs cargo test and cargo llvm-cov as two checks (duplicate test runs).
The correctness goals below are carried forward and resolved inside §13 as part of unifying the comment contract — not via a band-aid on the legacy split.

Original observation (kept for context): on Rustume PR #264 the lgtm-ci coverage workflow posted a Code Coverage Report comment showing MISSING / 0% / "No coverage data found" even though the run's actual coverage jobs succeeded. In coverage run 26652567453, web-coverage / comment-pr and Node coverage PR comment were skipped, yet a separate empty MISSING / 0% coverage comment was still posted.

Correctness requirements (resolved in §13 / PR #261 for Rust):

  • A coverage/test PR comment must not render a misleading MISSING / 0% state. When coverage is intentionally not produced on the event, suppress the comment or post an explicit "coverage not run on this event" state instead of 0% MISSING.
  • Populate test counts / coverage percent from structured output; do not emit zeros on a parse miss (fail loudly instead).
  • Exactly one coverage/test comment source posts per suite.
  • Regression tests for: success-with-data, success-but-no-data-on-this-event (no misleading comment), and parse-failure (explicit failure, not silent zeros).
  • Remove Rustume's local coverage workaround once the reusable reports correctly.

12. Job display names — hybrid split + static names (org-wide) — DONE

Status: completed in lgtm-ci via PR #263 (merged). Hybrid split (reusable-test-node / reusable-test-node-custom), static matrix job names, job-name on always-run jobs, and validate-static-job-names.sh guardrail.

Skipped or mutually exclusive jobs can render unevaluated job.name expressions in the GitHub checks UI (for example inputs.node-versions == '' && inputs.job-name || format('{0} ({1})', ...) or Verify ${{ matrix.platform }}). Interim static names shipped for reusable-test-node in #256; this item completes the org-wide contract with a hybrid approach.

Strategy

  • Split consumer-facing modes into separate reusables (eliminates skipped siblings; restores job-name for check names on always-run jobs).
  • Static inner job names where split is wrong (internal implementation paths, multi-version matrix within one mode).
  • Lint guard + docs so the contract stays enforced.

Split (consumer-facing modes)

  • Split Node Vitest vs custom command: reusable-test-node.yml (Vitest) + new reusable-test-node-custom.yml; remove test-command branching and mutual if: skip from the combined shape. Hard cutover per compatibility policy.
  • Migrate lgtm-ci dogfood callers and document consumer migration (Renovate repin + caller refactor).

Static inner names (internal / matrix-within-mode)

  • reusable-test-python.yml — static test job name (e.g. Python tests); matrix legs keep GitHub (version) suffix.
  • reusable-docker.yml — static names for build-per-platform and verify-per-platform (internal multi-arch path; not consumer-split).
  • reusable-test-e2e-matrix.yml — static test job name (e.g. E2E tests).

Explicitly not split

Guardrails and docs

  • Add workflow contract lint (BATS + scripts/ci/quality/validate-static-job-names.sh) forbidding matrix. / format( / ternary expressions in job.name on jobs with if:.
  • Document job display name policy in docs/workflow-contract.md and docs/reusable-workflows.md: when job-name drives check names vs PR comments, caller wrapper naming, branch protection updates. Include Node split (supersedes feat(ci): Pages Model B coverage HTML artifacts for Rust and Node reusables #256 static-name interim) and static-name reusables.

References

13. Unify Rust test/coverage and align the PR comment contract with Python/Node — DONE (merged PR #261)~

Status: completed in lgtm-ci via PR #261 (merged). Unified reusable-rust-test.yml with coverage: true|false, cargo-nextest / cargo llvm-cov nextest, JUnit + LCOV parsing, and Rust PR comments via reusable-test-pr-comment / generate-test-comment.sh. Legacy split Rust workflows and cargo test parsers were removed (hard cutover, no shim).

Supersedes §11. Closed PR #260 (the §11 point fix) and the §11 standalone item are folded here.

Summary

Collapse Rust testing into a single flag-driven reusable that mirrors the Python/Node model (coverage: true|false), switch the Rust runner to cargo-nextest, and converge every stack on one PR-comment contract that never renders misleading MISSING / 0%.

Problem

  • Rust ships two checks — reusable-rust-test.yml (cargo test) and reusable-rust-coverage.yml (cargo llvm-cov) — plus the legacy reusable-test-rust.yml orchestrator. cargo llvm-cov already runs the same test binaries under instrumentation, so wiring both into one pipeline runs the suite twice.
  • Two PR-comment generators diverge: generate-coverage-comment (Rust/Node rich, file-driven, multi-metric) vs generate-test-comment (Python/Node-default, tests + single coverage line). Behavior on missing data differs — Python explains it, Rust/Node can go silent or (pre-fix(coverage): suppress misleading MISSING PR comments (#168) #260) emit fake 0% MISSING.
  • cargo test results are scraped from text (parse-cargo-test-results.sh) rather than structured output.

Proposed Solution

  • Replace cargo test with cargo-nextest as the canonical Rust runner.
    • coverage: false → cargo nextest run --profile ci (uninstrumented, fast).
    • coverage: true → cargo llvm-cov nextest --profile ci --lcov ... (one instrumented run yielding both JUnit results and LCOV).
  • Emit JUnit XML from nextest (.config/nextest.toml [profile.ci.junit]) and parse it with the existing scripts/ci/lib/testing/parse/junit.sh; retire the text-scraping parse-cargo-test-results.sh.
  • Collapse to a single Rust reusable with a coverage flag; never run both tools in one pipeline.
  • Converge all stacks on one PR-comment generator that takes optional test counts and an optional coverage file (tests + coverage when present, tests-only otherwise), with explicit "coverage not collected on this event" messaging and no MISSING / 0%. Deprecate generate-coverage-comment once callers move.
  • Hard cutover: delete legacy reusable-test-rust.yml, reusable-test-rust-test.yml, reusable-test-rust-coverage.yml, reusable-rust-test.yml, and reusable-rust-coverage.yml in favor of the unified reusable (no compatibility shim, per the umbrella compatibility policy).
  • Update org rulesets / branch-protection required-check names for the new unified job names (org admin owns this; no per-consumer alias jobs).
  • Carry over all §11 correctness requirements and their regression tests (success-with-data, no-data-on-event, parse-failure).

Implementation Notes

  • cargo-llvm-cov integrates first-class with nextest (cargo llvm-cov nextest); pin the tool versions with Renovate version comments.
  • Reuse the existing JUnit parser and coverage extraction libs; avoid new bespoke parsers.
  • All shell logic stays in scripts/ci/**/*.sh with set -euo pipefail; BATS contract tests for the unified reusable and the nextest/JUnit + llvm-cov paths.
  • Update docs (docs/rust-testing.md, docs/workflow-contract.md, docs/reusable-workflows.md) and CHANGELOG.md.

Benefits

  • One test execution per Rust pipeline (no double runs); faster, structured results.
  • A single, predictable PR-comment contract across Python, Node, and Rust.
  • Misleading empty coverage comments are eliminated by construction.

References

Consumer migration

  • Rustume repins to the unified Rust reusable and drops local Rust coverage/test workarounds (tracked also under the Rustume migration track).

Metadata

Metadata

Assignees

Labels

ciCI/CD infrastructurecriticalCritical priority - must fix immediatelyenhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions