Repository navigation
Tags
Tags
Update Go to 1.27.0 (#350) Go 1.25.x branch is archived and no longer receiving updates. Since we are updating, update to the latest major version. Bump rules_go and gazelle accordingly. --------- Co-authored-by: João Guerra <joca-bt@users.noreply.github.com> Co-authored-by: Justin Won <justinwon777@gmail.com>
feat(config): default the graph format to tgb (#348) This change makes tgb the default graph format. All production repositories use the TGB format without issue. There's no reason to use gob graph format anymore. - config: Parse defaults the graph section to tgb and rejects "gob". - orchestrator tests: the tests parse a config without a graph section, so they now use TGB fixtures. - Update the example config and the README, which still described service.graph_format. ## Test Plan <!-- How did you test this? Provide evidence (screenshots, logs, or steps to reproduce). --> CI ## Issue <!-- Link the issue here. - Use 'Closes #123' if this is the final fix. - Use 'Part of #123' or just '#123' if the feature is still in progress. --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fix(bazel): run output_base and repo mapping with the query's options (… …#342) ## Intent `OutputBase` and `RepoMapping` ran bare `bazel info` and `bazel mod`, ignoring the startup options and extra args the query used. A custom `--output_base` or `--output_user_root` made the marker read look in a directory the query never wrote to, and module-resolution flags such as `--registry` were not applied when dumping the repo mapping. ## Changes `OutputBase` and `RepoMapping` now take the startup options the query ran with, so they use the same output base and Bazel server. `RepoMapping` also takes the query's extra args and forwards only an allowlist of them to `bazel mod`, via `ModuleResolutionArgs`, because `bazel mod` rejects query-only flags such as `--order_output` or `--keep_going`. `ReadRepoMarkerHashes` takes the startup options and the native graph runner passes the repo config's values. The allowlist holds the flags that change the mapping (`--registry`, `--override_module`, `--inject_repository`) and the flags that say where downloads come from, so `bazel mod` reuses what the query fetched instead of going to the network (`--repository_cache`, `--repo_contents_cache`, `--distdir`, `--vendor_dir`, `--downloader_config`, `--experimental_remote_downloader`, `--credential_helper`, plus the path-, URL- and env-valued `--override_repository`, `--module_mirrors` and `--repo_env`). These are command options, so they belong in `bazel_extra_args`, not `bazel_startup_options`. `--config` is not forwarded. `query-bench` gains repeatable `-startup-opt` and `-bazel-arg` flags that are applied the same way as in the native runner, so a custom `--output_base` or `--repository_cache` can be exercised. The exported signatures of `OutputBase`, `RepoMapping` and `ReadRepoMarkerHashes` changed, so out-of-tree callers need to pass the new arguments. ## Testing Unit tests in `core/bazel` cover `OutputBase`, `RepoMapping` and `ModuleResolutionArgs`, including argument order and that query-only flags are dropped. Integration, using `query-bench` against a large internal Bzlmod monorepo with a 575-target query. On this repo, with a cold output base and a cold cache, `bazel mod` fetched 157 files into the repository cache. Running `query-bench` with a cold custom `--output_base` and `--repository_cache` read 81 markers from the custom directory, wrote 204 files to the custom cache, and produced the same graph as the default run. --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
fix(tgb): drop per-column zstd expansion cap (#332) ## Summary - Valid graphs with no tags are rejected as "corrupt TGB graph blob", failing GetChangedTargets and GetChangedTargetsAndEdges for every commit. - TAG_DEG is one byte per node; with no tags it is all zeros and compresses to ~100 bytes regardless of node count, so its ratio (~7,600x) exceeds the 4096x cap once a graph passes ~414k nodes. - Remove the per-column maxZstdExpansion check from validateColumns. A compression ratio is not a memory bound for constant-valued per-node columns. - Keep the real bounds: maxColumnRawSize, the total-raw budget against file size, and the per-node checks against nodeCount. ## Test Plan - Add a regression test that round-trips a 700k-node tagless graph and asserts TAG_DEG exceeds the old 4096x ratio. ## Issues
fix(targethasher): only hash local FILE lines in bzlmod repo markers (#… …330) ## Summary **Why?** Some of the skipped `FILE:` lines are generated during the fetch (gazelle's go.env and the gazelle and fetch_repo binaries) and contain the Bazel output_base path, which depends on the workspace location. The same commit computed in two different workspaces got different hashes, so the whole repo and everything depending on it showed up as changed. **What?** Now it only hashes the first line of the marker (the repo rule's inputs) plus the inputs that live in the main repo (FILE:, DIRTREE:, DIRENTS: with an @@// path, which covers patches). Everything else is skipped: ENV:, REPO_MAPPING:, inputs from other repos, and absolute paths. Patch edits and version, URL or integrity changes still change the hash. The marker file hashing is only supported since bazel doesn't support `bazel mod show_repo --all_repos` until later versions, and internally the repo is on bazel 8.1.1 with bzlmod. On Bazel below 8.6.0, markers are how we hash these repos, one hash per repo rather than per file. Hashing external repos via `bazel mod show_repo --all_repos` will be supported in follow up PRs. **Test Plan** New markers_test.go covers the cases above, bazel test //core/targethasher:targethasher_test passes, and on two checkouts of the same commit in different directories none of the 28 real markers differ anymore. ## Test Plan Unit test Tested on internal monorepo to ensure the external hash did not differ across revisions ## Issues --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(core/errors): add NewCancelled constructor (#321) Jobs can be cancelled downstream such as in orchestrator implementation when one get target graph job fails and the other is cancelled. In those cases, the cancelled job should carry a cancelled error, but currently they are infra. Create an exported func to allow users to classify as cancelled. <!-- Provide a summary in the Title above. Example: "Feature: add user authentication" --> ## Test Plan <!-- How did you test this? Provide evidence (screenshots, logs, or steps to reproduce). --> unit test ## Issue <!-- Link the issue here. - Use 'Closes #123' if this is the final fix. - Use 'Part of #123' or just '#123' if the feature is still in progress. --> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
feat(targethasher): collapse bzlmod external targets to repo-level ha… …sh (#312) ## Summary For legacy WORKSPACE repos, tango collapses external source/generated files to a single `//external:repo` hash so `HashRecursively` doesn't visit each file individually. Bzlmod repos (`@@repo+...//...`) don't have `//external:` rules, so this collapse was skipped — every external source file got hashed one by one during the DFS. On a python bzlmod repo (2.49M targets, 82% external pip wheels), this made `HashRecursively` take ~15 minutes single-threaded. This PR adds the equivalent collapse for bzlmod using Bazel's marker file content hashes from `$(output_base)/external/@repo.marker`. Each marker file contains a hash of the repository rule's inputs (URL, sha256, patches, etc.) that changes on any version bump — even for repos like `protobuf+` whose canonical name stays the same across versions. ### How it works 1. After bazel query, read marker files from `$(output_base)/external/` 2. For each bzlmod external source/generated file, look up its repo's marker hash 3. If found, pre-hash the target with `sha1(marker_content_hash)` before the DFS 4. If no marker exists, skip collapsing — let `HashRecursively` hash the actual file content ### Why marker files instead of repo name hashing Bzlmod canonical repo names fall into two categories: - **With content hash suffix** (e.g. `rules_python++pip+..._a6ebbe51`): 10,585 repos, 1.87M files — repo name changes on upgrade - **Without content hash suffix** (e.g. `protobuf+`, `grpc+`, `zlib+`): 455 repos, 104K files — repo name stays the same on upgrade Hashing just the repo name string would miss version bumps for the second category. Marker files solve this because they always reflect the repository rule's inputs. ## Benchmark (2.49M targets) | Metric | Before | After | |---|---|---| | HashRecursively | 899s | ~120s | | Total compute | 969s (16.2 min) | ~182s (3 min) | ## Test plan - [ ] Unit tests for `bzlmodRepoName`, `collapseBzlmodExternalTargets` (marker lookup, no-marker fallback, excluded regex, fullHashRepos skip) - [ ] End-to-end test: `fromProto` with excluded bzlmod targets gets empty hash, not marker hash - [ ] Benchmarked on uber-one with `--bypass-cache` - [ ] Verified target set is identical (2,493,050 targets) between original and collapsed - [ ] Repos without markers fall through to `HashRecursively` for file content hashing --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
feat(config)!: make graph_format and shadow_compare per-repository (#308 ) ## Summary - Move `graph_format` and `shadow_compare` from the service-wide `ServiceConfig` to a new top-level `graph` YAML map keyed by short repository name (e.g. `uber/go-code`) - A `"default"` entry provides the fallback for repos without an explicit override; requests for unconfigured repos with no default are rejected as user errors - Enables gradual per-repo TGB rollout without affecting other repositories served by the same Tango instance **BREAKING CHANGE:** `service.graph_format` and `service.shadow_compare` YAML fields are removed. Omitting the `graph` section entirely preserves the old default (gob, no shadow compare). ### New YAML shape ```yaml graph: default: format: gob uber/go-code: format: tgb shadow_compare: true ``` ### Key design decisions - New `GraphConfigProvider` interface (mirrors existing `RepositoryConfigProvider` pattern) keeps controller testable - Controller resolves format/shadow per-request via `graphFormatFor(remote)` / `shadowCompareFor(remote)` - Orchestrator resolves via `config.GetGraphConfig(build.Remote)` per-request - Short remote extraction inlined in `config` package to avoid `config → internal/url` dependency ## Test plan - [x] `make build` — compiles - [x] `make test` — all 29 unit tests pass - CI
Include label name in source file hash and handle missing files grace… …fully (#290) HashSourceFile fails hard when os.Stat returned error. Sometimes bazel rules can generate labels for paths that aren't real files (stale exports_files entries, test fixture jars, and pseudo-labels from string attributes like resource_strip_prefix). Changes can add more labels like these without build failures, which means tango can easily break. Instead of failing, produce a deterministic hash for missing files. Hash the label name so different missing files get distinct hashes, and a file appearing/disappearing changes the hash. ## Test Plan <!-- How did you test this? Provide evidence (screenshots, logs, or steps to reproduce). --> unit test ## Issue <!-- Link the issue here. - Use 'Closes #123' if this is the final fix. - Use 'Part of #123' or just '#123' if the feature is still in progress. --> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop doubled-package pseudo-labels from RuleInput before hashing (#288) Bazel query can render a workspace-relative string attribute (e.g. resource_strip_prefix, used by kt_jvm_library/java_library for resource stripping) as a pseudo-label. The result is a RuleInput entry like //pkg:pkg/resources that never corresponds to a real file, and HashRecursively's fallback for unresolved labels fails trying to stat the doubled path. Filter such labels out of a rule's deps at proto-to-Target conversion time, since it's not a real label. Real example: `tooling/program-analysis/detekt/BUILD.bazel` in Java monorepo: ``` uber_kt_library( name = "src_main", resource_strip_prefix = package_name() + "/src/main/resources", ... ) ``` bazel query adds resource_strip_prefix as a label-shaped RuleInput entry, formatted as `//<current-package>:<raw-string>`. Since the raw string is already workspace-rooted, the result doubles the package. The actual resource file is already part of the inputs `bazel query 'deps(//tooling/program-analysis/detekt:src_main, 1)' --output=label_kind`: ``` source file //tooling/program-analysis/detekt:src/main/resources/META-INF/services/io.gitlab.arturbosch.detekt.api.RuleSetProvider source file //tooling/program-analysis/detekt:tooling/program-analysis/detekt/src/main/resources ``` ## Test Plan <!-- How did you test this? Provide evidence (screenshots, logs, or steps to reproduce). --> unit test ## Issue <!-- Link the issue here. - Use 'Closes #123' if this is the final fix. - Use 'Part of #123' or just '#123' if the feature is still in progress. --> https://linear.app/uber/issue/TANGO-33/fix-hasing-labels-with-package-prefixed Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
PreviousNext