Skip to content

perf(catalog): back off persistently failing stitches - #36061

Open
freben wants to merge 4 commits into
masterfrom
freben/catalog-stitch-failure-backoff
Open

freben wants to merge 4 commits into
masterfrom
freben/catalog-stitch-failure-backoff

Conversation

@freben

@freben freben commented Oct 7, 2026

Copy link
Copy Markdown
Member

Hey, I just made a Pull Request!

Persistently failing stitching requests currently retry at a fixed rate indefinitely. Add a failure counter and quadratic backoff on acquisition, capped at one hour before up to 10% positive jitter. Initial claims keep their configured lease, and longer configured leases remain the minimum.

Failed work remains recoverable. New requests reset the counter while preserving existing leases; failure updates are fenced by ticket and exact lease so stale attempts cannot affect successors. This is an independent resilience improvement.

Validation: PostgreSQL 16 and SQLite queue/publication/processing tests (139 passed), then 87 stitching tests including migration up/down and reset by entity ID. Root lint and yarn build:api-reports passed. MySQL execution is left to CI.

✔️ Checklist

  • A changeset describing the change and affected packages. (more info)
  • Added or updated documentation
  • Tests for new functionality and regression tests for bug fixes
  • Screenshots attached (for UI changes)
  • All your commits have a Signed-off-by line in the message. (more info)

Signed-off-by: Fredrik Adelöw <freben@gmail.com>
@freben
freben requested review from a team as code owners October 7, 2026 13:42
Copilot AI balanced review requested due to automatic review settings October 7, 2026 13:42
@github-actions github-actions Bot added area:documentation Improvements or additions to documentation area:catalog Related to the Catalog Project Area labels Oct 7, 2026
@backstage-goalie

Copy link
Copy Markdown
Contributor

Changed Packages

Package Name Package Path Changeset Bump Current Version
@backstage/plugin-catalog-backend plugins/catalog-backend patch v4.0.1-next.2

@backstage-goalie backstage-goalie Bot added size:medium Medium-sized pull requests receive no change in priority for reviews. waiting-for:review The PR needs a review and will be visible in the review queue unless already assigned an owner. labels Oct 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Concurrency-sensitive lease fencing needs human review, with MySQL validation pending and a stale-failure assertion needing strengthening.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds retry backoff to the catalog stitching queue to reduce repeated work on persistently failing entities.

Changes:

  • Tracks failures and applies capped quadratic backoff with jitter.
  • Resets counters for new requests and fences failure updates against stale claims.
  • Adds migration coverage, regression tests, and documentation.
File Description
plugins/​catalog-backend/​src/​database/​tables.ts Types the failure counter.
plugins/​catalog-backend/​src/​database/​operations/​stitcher/​performStitching.ts Records failed attempts.
plugins/​catalog-backend/​src/​database/​operations/​stitcher/​performStitching.test.ts Tests failure recording and recovery.
plugins/​catalog-backend/​src/​database/​operations/​stitcher/​markForStitching.ts Resets counters on new requests.
plugins/​catalog-backend/​src/​database/​operations/​stitcher/​markDeferredStitchFailed.ts Adds fenced failure updates.
plugins/​catalog-backend/​src/​database/​operations/​stitcher/​getDeferredStitchableEntities.ts Applies per-request backoff.
plugins/​catalog-backend/​src/​database/​operations/​stitcher/​getDeferredStitchableEntities.test.ts Tests backoff, resets, and migration preservation.
plugins/​catalog-backend/​report.sql.md Documents the schema addition.
plugins/​catalog-backend/​migrations/​20261007000000_stitch_failure_count.js Adds the failure counter column.
docs/​features/​software-catalog/​configuration.md Explains retry behavior.
.changeset/​catalog-stitch-failure-backoff.md Adds the release entry.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread plugins/catalog-backend/src/database/operations/stitcher/performStitching.test.ts Outdated
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 14:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-database lease fencing and migration behavior warrant human review, particularly with MySQL validation still pending.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 15:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-database lease fencing needs human review with MySQL validation pending, and completion errors still bypass failure recording.

0 open findings

Previously missed (1)

In code that hasn't changed since last review

Medium severity Record failures when deferred stitch completion errors

plugins/​catalog-backend/​src/​database/​operations/​stitcher/​performStitching.ts:336

Errors from markDeferredStitchCompleted in finally bypass this failure-recording call. If completion repeatedly fails while the claimed queue row remains, DefaultStitcher reports failed attempts but failure_count stays unchanged, so these retries receive no additional backoff. Handle completion errors with the same fenced, best-effort failure recording and rethrow the original error. Add a regression test where completion rejects while the queue row remains.

🧠 Review effort: Balanced

Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 19:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-database lease fencing warrants final human review, particularly with MySQL validation deferred to CI.

0 open findings

🧠 Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:catalog Related to the Catalog Project Area area:documentation Improvements or additions to documentation size:medium Medium-sized pull requests receive no change in priority for reviews. waiting-for:review The PR needs a review and will be visible in the review queue unless already assigned an owner.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants