Skip to content

Error breakdown: error call paths keyed by RPC status (Python + Go) - #71

Merged
lollllcat merged 1 commit into
mainfrom
go-port/error-breakdown
Sep 24, 2026
Merged

lollllcat merged 1 commit into
mainfrom
go-port/error-breakdown

Conversation

@lollllcat

@lollllcat lollllcat commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Error breakdown: error call paths keyed by RPC status (Python + Go)

Adds an error breakdown to both implementations: which call paths end in an error, keyed by RPC protocol and status code, with counts and exemplars.

crisp-trace --file t.json -s svc -a op --lightMode --errorBreakdown origins     # or propToRoot
crisp-go    --file t.json -s svc -a op --lightMode --errorBreakdown origins --errorBreakdownRoot trace

Writes error-breakdown.json next to the trace. The full spec is in the new CONFORMANCE.md section "Error breakdown".

Semantics

  • Modes. origins reports every erroring span with no erroring child (it traverses successful spans too). propToRoot only reports errors that reach the root: a successful RPC span stops the walk. This is the existing Graph.computePropToRootGraph walk.
  • Roots. trace (the default) starts at the trace's own root span. If no span lacks a parent, a virtual [INCOMPLETE_TRACE]virtual_root node parents the orphaned spans. analysis starts at the root chosen for -s/-a.
  • Detection. A span errors if parseForErrorReturn flags it or its status code signals failure (HTTP >= 400, otherwise non-zero). Protocol and status come from standard tags (http.response.status_code, http.status_code, grpc.status_code, rpc.grpc.status_code, rpc.yarpc.status_code + rpc.transport, ...). A gRPC status name such as grpc.status = "UNAVAILABLE" sets only the protocol.
  • Deployment hints. Hints specific to one deployment are new span_utils lists, all empty by default: HTTP_COMPONENTS, TCHANNEL_MARKER_TAGS, TCHANNEL_STATUS_TAGS, YARPC_STATUS_TAGS, IGNORED_ROOT_OPS.
  • Tree used. The walk runs during Graph construction, before timeline sanitization drops out-of-range (clock-skewed) spans. Critical-path outputs are unchanged, and every existing conformance.* golden is byte-identical.

Fix

process() passed {} to getMetrics, so the --errorAnalysis flame graph of errors propagated to the root was always empty. It now passes computePropToRootGraph(), which also gains tests. --errorBreakdown outside light mode now logs a warning and is ignored, instead of being computed and never written.

Tests

  • Goldens. Every conformance fixture gets error-breakdown-{origins,propToRoot}.json. Seven new test_cases/error_breakdown/ fixtures (status codes, orphans, multiple roots, a sanitized child, propagation cases) cover all four mode/root combinations. scripts/generate_goldens.py produces both sets and checks determinism.
  • Parity. Python and Go both check the same goldens byte-for-byte. difftest -strict also compares error-breakdown.json when both templates pass the flag: 40/40 in both modes.
  • Unit tests. Status-code extraction edge cases, the deployment lists, root selection, exemplar merging and counting, and option validation.
  • Housekeeping. difftest -mode golden now checks only the conformance fixtures (as generate_goldens.py does). Gazelle added the missing light_test.go to go/crisp:crisp_test.

Validation: scripts/ci-local.sh (837 passed, test.sh clean), bazel test //... (37/37), go vet ./..., and difftest in golden mode (33/33) and strict corpus mode in both breakdown modes (40/40).

About 7k of the added lines are golden JSON.

Add --errorBreakdown {origins,propToRoot} and --errorBreakdownRoot
{trace,analysis}: light mode writes error-breakdown.json, the call paths
ending in an error, keyed by RPC protocol and status code, with counts and
exemplars.

- origins reports every erroring span with no erroring child; propToRoot
  only errors that propagate to the root (the computePropToRootGraph walk).
- trace root is the trace's own root span, or a virtual
  [INCOMPLETE_TRACE]virtual_root over orphaned spans; analysis root is the
  one chosen for -s/-a.
- A span errors if parseForErrorReturn flags it or its status code
  signals failure (HTTP >= 400, else non-zero). Protocol/status come from
  standard tags; deployment-specific hints are empty-by-default span_utils
  lists (HTTP_COMPONENTS, TCHANNEL_MARKER_TAGS, TCHANNEL_STATUS_TAGS,
  YARPC_STATUS_TAGS, IGNORED_ROOT_OPS).
- The walk runs during Graph construction, before sanitization drops
  out-of-range spans. Critical-path outputs are unchanged.

The Go port matches byte-for-byte: goldens for every conformance fixture
(both modes) and new test_cases/error_breakdown/ fixtures (all four
mode/root combinations); difftest -strict also compares the file.

Fix: process() passed {} to getMetrics, so the --errorAnalysis
errors-propagated-to-root flame graph was always empty; it now uses
computePropToRootGraph(), which gains tests.

difftest golden mode now checks only the conformance fixtures, like
generate_goldens.py. Gazelle adds the missing light_test.go to
go/crisp:crisp_test.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lollllcat
lollllcat force-pushed the go-port/error-breakdown branch from 18b3eab to b6b967a Compare September 24, 2026 21:30
@lollllcat
lollllcat merged commit 56c9a9a into main Sep 24, 2026
2 checks passed
@lollllcat
lollllcat deleted the go-port/error-breakdown branch September 24, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant