Skip to content

Release self-heal is skipped when Semantic Release fails after pushing the release commit #1773

Description

@rainer-friederich

Summary

When Semantic Release publishes a package, pushes its release commit, and then fails, the root package-lock.json is left drifted and the step that exists to fix it never runs. Every open PR then fails the Test job at the Verify package-lock.json is in sync step with a diff it did not author, until someone files a lockfile-only PR by hand.

This is the "main-wide breakage that first surfaces on unrelated PRs" the drift guard warns about, and it is not the mechanism docs/RELEASE-RUNBOOK.md Failure mode 7 documents. FM-7 covers "the sync step itself fails (red ✗)"; here the sync step never runs at all.

The design gap

In .github/workflows/main.yaml, the Release job:

  • Semantic Release (~line 225) has no continue-on-error, so its failure fails the job and short-circuits later if: success() steps.
  • Sync package-lock.json after release (~line 230) is gated if: success() with continue-on-error: true.

The continue-on-error: true protects only against the sync step's own failure. When the failure is upstream, if: success() simply skips the sync. And because semantic-release pushes the release commit — the thing that drifts the lock — and can fail afterwards, the one scenario that most needs the re-sync is exactly the one where it is skipped.

The step's own comment already names the backstop it falls back on: "the Test-job drift guard is the backstop … if the push loses a race it is left for the next PR's drift guard to surface." That is the right posture for a lost race. It is the wrong posture for a guaranteed skip, because it converts a release-job failure into a main-wide contributor tax.

Observed instance

The 4.0.0 release of spacecat-shared-data-access (release commit a7b72cc7, run 28669792110):

  • Semantic Releasefailure; Sync package-lock.json after releaseskipped; Surface failure pointer to release runbook → success.
  • It had already published data-access@4.0.0 — OIDC token exchange succeeded, GitHub release created, tag pushed — then aborted with:
    ✘ An error occurred while running semantic-release: TypeError: Cannot read properties of undefined (reading 'map')
    AggregateError:
        SemanticReleaseError: No npm token specified.
    TypeError: Cannot read properties of undefined (reading 'map')
    
  • package.json said 4.0.0; the lock still pinned 3.81.0. Every open PR went red, e.g. feat(project-engine-client): re-vendor Semrush swagger; retire CR9/CR10, slim CR16 #1772, which worked around it with a per-branch lock-sync commit.

That specific drift was cleared by hand and main is in sync today, so nothing is currently broken — but the mechanism is unchanged and will produce the same outcome on the next partial release failure.

Work

1. Make the self-heal resilient to a partial Semantic Release failure. Two options to evaluate:

  • Change the sync step's gate from if: success() to if: always() (or if: success() || failure()), keeping continue-on-error: true, so a partial failure still triggers the re-sync. It already does a git diff --quiet check, so it commits only when the lock actually drifted.
  • Or add the lockfile to the @semantic-release/git assets list, so the lock is committed atomically with the version bump in the same release commit. This removes the separate-sync-step failure surface entirely. Assess why it was kept out originally — the workflow comment says the assets list is package.json + CHANGELOG.md only.

2. Diagnose the underlying Semantic Release failure so releases stop half-failing. data-access@4.0.0 published fine via OIDC, yet the run still aggregated SemanticReleaseError: No npm token specified. plus the reading 'map' TypeError. Determine which plugin or package instance raised "No npm token specified" after a successful OIDC publish — only data-access had commits to release that run, every other package reported "no release" — and whether the TypeError is a secondary crash in semantic-release-monorepo's error handling masking the real cause. See RELEASE-RUNBOOK for the OIDC publish path.

3. Document this failure mode. Add the "SR fails after pushing the release commit → sync step skipped → main drifted" variant to docs/RELEASE-RUNBOOK.md alongside FM-7, with its recovery, so on-call recognizes it. The recovery is a lockfile-only PR, no republish:

git checkout main && git pull
npm install --package-lock-only --ignore-scripts
git checkout -b fix/sync-package-lock
git add package-lock.json
git commit -m "fix: sync package-lock.json with released workspace versions"
# open PR; the drift guard on the PR confirms the fix

References

  • Workflow: .github/workflows/main.yaml (drift guard ~line 120; Release job sync step ~line 230)
  • Runbook: docs/RELEASE-RUNBOOK.md Failure mode 7

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions