Tags: uber/tango
Tags
TGB (tango graph binary) implementation
feat(storage): use gob instead of JSON for graph/changed-targets stre… …ams (#276) ## Summary - JSON-encoded target-graph and changed-targets blobs are ~1.8x the size of the prior proto-delimited format (900MB vs 500MB), pushing storage upload latency from ~15s to ~60s. - Proto-delimited encoding isn't an option per prior feedback that proto types shouldn't leak into internal storage implementation. - Switches `core/storage/graphwriter.go` and `core/storage/reader.go` to `encoding/gob`, since the streamed `entity.GetTargetGraphResponse` / `entity.GetChangedTargetsResponse` types are already plain Go structs with no proto dependency. - Updates test fixtures across `controller` and `orchestrator` packages that hand-built JSON bytes to simulate cached storage blobs. ## Measured impact Ran a real ~950MB JSON target-graph blob (134 streamed chunks, ~2.75M targets, ~3.6M metadata entries) through both encoders to compare actual output size rather than estimate it: | Format | Size | Ratio vs JSON | |---|---|---| | JSON (current) | 950,694,991 bytes (~950MB) | 1.00 | | gob (this PR) | 557,124,529 bytes (~557MB) | 0.586 | | proto-delimited (prior) | ~500MB | ~0.53 | gob lands within ~11% of the old proto-delimited size on this sample, recovering essentially all of the size regression that caused upload latency to jump from ~15s to ~60s — without reintroducing proto types into the storage layer. ## Test plan - [x] `go test ./...` (excluding `integration`, which requires `TANGO_REPO_REMOTE` and fails identically on `main`) - [x] `make gazelle` (no BUILD.bazel changes needed — both packages are stdlib)
fix(controller): use a custom cancel cause for canceling graph fetch … …jobs (#274) Cancelling a job's context on sibling failure now carries a distinct cause instead of the default context.Canceled, so it's clear the cancellation was self-inflicted rather than a client disconnect. ## Test Plan <!-- How did you test this? Provide evidence (screenshots, logs, or steps to reproduce). --> unit test ## Issue <!-- Link the issue here. - Use 'Closes #123' if this is the final fix. - Use 'Part of #123' or just '#123' if the feature is still in progress. -->
config: update fields, defaults, and required checks to match README (#… …190) ## Summary Aligns config field names, defaults, and required checks with config/README.md from #180. - Renamed fields: `worker_pool_size` → `max_worker_pool_size`, `repo_manager_clone_path` → `workspaces_root`, `chunking` → `streaming` (with sub-fields renamed to `max_num_targets`/`max_num_changed_targets`/`max_num_metadata_entries`), `query_timeout` → `query_timeout_seconds`. - `config.Parse` now enforces `service.workspaces_root` as required and applies documented defaults: `bzlmod_enabled` defaults to `true`, `query_timeout_seconds` defaults to `600`, streaming fields default to `250`/`125`/`50000`. - Added Go doc comments for config fields documented in the README. - Updated all callsites and YAML fixtures for the renames. - Added `config/config_test.go` covering new validation and defaults. ## Test plan - [x] `make build` - [x] `make test` ## Stack 1. #180 2. @ #190 3. #191 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
fix(errors): classify appCtx cancelled as infra, not user-cancelled (#… …247) GetErrorCode previously mapped every context.Canceled to ErrorCancelled, so a request aborted by appCtx firing on SIGTERM/SIGINT looked identical to an ordinary client disconnect. linkRequestCtx now uses WithCancelCause and sets a non-context.Canceled cause when appCtx fires, and call sites read context.Cause(ctx) instead of the generic ctx.Err() so the distinction survives into the returned error. ## Test Plan <!-- How did you test this? Provide evidence (screenshots, logs, or steps to reproduce). --> unit tests ## Issue <!-- Link the issue here. - Use 'Closes #123' if this is the final fix. - Use 'Part of #123' or just '#123' if the feature is still in progress. -->
fix(bazel): classify bazelisk download network failures as retryable (#… …236) ## Summary Intent: - A bazel client creation failure previously returned a plain error that fell through to the ErrorInfra default in tangoerrors.GetErrorCode -- correct by accident, not by design. A transient network failure while downloading bazelisk is retryable and shouldn't be lumped in with permanent infra failures (e.g. a bad BazelCommand path). Changes: - Add bazel.ErrDownloadBazeliskNetwork, wrapped when downloading bazelisk fails with a net.Error. - Add orchestrator.classifyBazelClientError, classifying ErrDownloadBazeliskNetwork as tangoerrors.ErrorInfraRetryable and everything else as tangoerrors.ErrorInfra; wired into nativeOrchestrator. ## Test Plan unit test ## Revert Plan Revert this PR; classification falls back to the existing ErrorInfra default. ## Issues --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: xiao <48696694+xytan0056@users.noreply.github.com>
refactor(controller): fold compare_target_graphs metrics into get_cha… …nged_targets (#242) Route `compareTargetGraphs`'s metrics under `opGetChangedTargets` (it's a private sub-step of GetChangedTargets, not its own RPC) and replace its Begin/Complete lifecycle with a single `compare_duration` histogram, since a failure there already surfaces on `get_changed_targets.finish` and was being double-counted. **Breaking:** `controller.compare_target_graphs.*` is removed; use `controller.get_changed_targets.{compare_duration,decode_duration,diff_duration}` instead.
feat(controller): emit result-tagged cache-lookup counters (#241) Replace the hit-only `cache_hit` counters with `result`-tagged cache-lookup counters at every controller cache-lookup site, so a per-layer hit rate (`hit / (hit + miss)`) is derivable for each cache. Three counters, each emitted under its parent RPC op with a `result=hit|miss` tag: | metric | emitted from | |---|---| | `treehash_cache_lookup` | `get_target_graph`, `get_changed_targets` | | `graph_cache_lookup` | `get_target_graph` | | `compared_targets_cache_lookup` | `get_changed_targets` | ## Classification The result is classified by the lookup error's **origin**: a not-found is a genuine miss; an infra failure is *not* a cache miss (it's already tracked by the failure metric) and is left out so it can't skew the hit rate. | site | not-found | infra error | |---|---|---| | treehash | miss | nothing (request fails) | | graph | miss | nothing (request fails) | | compared-targets | miss | nothing (recomputes) | **Breaking:** the bare `controller.get_target_graph.cache_hit` and `controller.get_changed_targets.cache_hit` counters are removed. Stacked on `refactor/share-metrics-buckets` (#240) — review/merge that first.
feat(repomanager): emit lease metrics for health and slot-wait latency ( #232) ## What Instruments `RepoManager.Lease` with the `observability/metrics` emitter, using the same Scope-in pattern as the orchestrator and graphrunner. - `lease.start` counter + result-tagged (`success`/`failure`/`cancelled`) `lease.finish` histogram — pool health. - Per-step latency histograms: `ensure_origin_duration`, `wait_slot_duration` (the worker-slot wait — the saturation signal), `create_worker_duration`. Repo-tagged; buckets are exponential 1ms..~1.3h. The example server leaves the scope no-op (as the orchestrator/graphrunner are wired there today); production wiring supplies a reporter-backed scope. Also folds nil-scope handling into `metrics.New` (nil → no-op, no error) and adds `Emitter.SubScope`, removing the repeated `if scope == nil` / `if err != nil { Nop() }` boilerplate across all call sites (repomanager, controller, graphrunner, orchestrator). ## Test `make build` / `make test` / `make gazelle` pass.
refactor(cachekey): move internal/cachekey to core/cachekey to make i… …t importable (#213) The cache key helpers are used in orchestrator implementations, so users implementing their own orchestrator cannot import cachekey because it's under internal. Move the package to core/ to make it importable. ## Test Plan <!-- How did you test this? Provide evidence (screenshots, logs, or steps to reproduce). --> CI ## Issue <!-- Link the issue here. - Use 'Closes #123' if this is the final fix. - Use 'Part of #123' or just '#123' if the feature is still in progress. -->
PreviousNext