Repository navigation
Tags
Tags
chore(protos): pin vendored Agent protos to the 7.85.x commit with pr… …ocess_instance_id (backport 1.7.x) (#3060) ## Summary Re-pins the vendored Datadog Agent protos on `releases/1.7.x` from the `main` commit `b13554d0c66` to `c00253cf015`, the commit on the Agent's `7.85.x` release branch that added the optional `process_instance_id` to remote-agent registration (DataDog/datadog-agent#59686, a backport of #58951). #3018 pinned a `main` commit as a stopgap, because no Agent release branch had the field yet; the `Makefile` comment said to move it once one did. Agent 7.85.x ships this branch's Agent Data Plane, so the vendored protos should match it. The pin is the commit SHA rather than the branch name `7.85.x`. A branch name would move, and every proto check would then compare against whatever `7.85.x` looks like that day. What changes in the vendored files: `kubemetadata`, `private_actions`, `process_manager` and `workloadmeta` lose definitions for Agent features newer than 7.85.x (for example `KAIPodGroup` and `ConnectionTokenV2`). Agent Data Plane doesn't reference them. `remoteagent.proto`, which carries `process_instance_id`, is unchanged. 1.8.x needs no equivalent change: its vendored protos already match Agent 7.86.x. ## Change Type - [ ] Bug fix - [ ] New feature - [x] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? On top of `releases/1.7.x`: - `make update-protos` regenerates the vendored files from the pinned commit; the diff is the 6 files above. - `cargo check --workspace --tests`, `make check-fmt`, `make check-clippy` and `make check-release-notes` pass. - `cargo nextest run -p datadog-protos -p saluki-core -p saluki-components -p prometheus-exposition -p agent-data-plane -p datadog-agent-commons`: 1947 passed, 3 skipped. ## References - DataDog/datadog-agent#59686 (7.85.x backport of #58951) - #3018 (introduced the temporary `main` pin), #3003 (the 1.8.x equivalent)
fix(metrics): follow-ups from review of the expvar telemetry changes … …(backport 1.8.x) (#3043) ## Summary Backport of #3031 to `releases/1.8.x`, cherry-picked with `-x` from 863a687. - **Deleted-counter queue** (`fix(metrics)`): the deleted-counter queue is now a set, so a counter deleted and re-registered repeatedly between two flushes is queued once. - **API key validation in RAR telemetry** (`fix(metrics)`): series tagged `purpose:api_key_validation` are no longer remapped into the forwarder transaction telemetry that ADP reports to the Agent, so a failed validation request no longer counts in `transactions.errors`. - **Encoder flush telemetry** (`fix(encoders)`): the Events, Service Checks and Logs encoders now report the events they dispatched, including those sent before a flush fails part-way, and count the events in a discarded payload as dropped. The patch is identical to `main`'s, except for one file. `lib/saluki-components/src/forwarders/mod.rs` conflicted: on 1.8.x it lacks the config, endpoint, proxy and retry re-export block that `main` has, so only the new `API_KEY_VALIDATION_PURPOSE_TAG` re-export is added there. ## Change Type - [x] Bug fix - [ ] New feature - [ ] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? On top of `releases/1.8.x`: - `cargo nextest run -p saluki-core -p saluki-components -p agent-data-plane`: 2051 passed, 3 skipped. - `cargo check --workspace --tests`, `make check-fmt`, `make check-clippy`, `make generate-api-docs`, `make check-docs` and `make check-release-notes` pass. ## References - Original PR: #3031 - 1.7.x equivalent: opened alongside this PR.
chore(dev): ignore `rustls` advisories pending FIPS validation (#2828) <!-- dd-meta {"pullId":"367183fc-2677-4966-ba15-166273897db8","source":"chat","resourceId":"247d8b08-a5fd-4dc8-a98c-5c444ff4b84c","workflowId":"6fea2280-9411-402c-8c48-653d0ddbcebd","codeChangeId":"6fea2280-9411-402c-8c48-653d0ddbcebd","sourceType":"slack"} --> ## Summary This PR adds three new RustSec advisories for `rustls` to the `cargo-deny` ignore list. We can't yet upgrade our version of `rustls`, as doing so would pull in an updated version of AWS-LC which is not yet FIPS validated: `aws-lc-rs` 1.18, which translates to AWS-LC 4 for FIPS purposes. We're simply ignoring these for now as we've deemed them to not materially affect Saluki/ADP at the present time. ## Change Type - [ ] Bug fix - [ ] New feature - [x] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? Made sure `make check-deny` passes. ## References AGENTDP-7 Co-authored-by: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
chore(dev): ignore `rustls` advisories pending FIPS validation (#2828) <!-- dd-meta {"pullId":"367183fc-2677-4966-ba15-166273897db8","source":"chat","resourceId":"247d8b08-a5fd-4dc8-a98c-5c444ff4b84c","workflowId":"6fea2280-9411-402c-8c48-653d0ddbcebd","codeChangeId":"6fea2280-9411-402c-8c48-653d0ddbcebd","sourceType":"slack"} --> This PR adds three new RustSec advisories for `rustls` to the `cargo-deny` ignore list. We can't yet upgrade our version of `rustls`, as doing so would pull in an updated version of AWS-LC which is not yet FIPS validated: `aws-lc-rs` 1.18, which translates to AWS-LC 4 for FIPS purposes. We're simply ignoring these for now as we've deemed them to not materially affect Saluki/ADP at the present time. - [ ] Bug fix - [ ] New feature - [x] Non-functional (chore, refactoring, docs) - [ ] Performance Made sure `make check-deny` passes. AGENTDP-7 Co-authored-by: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
chore(agent-data-plane): bump version to 1.6.3 (#2742) ## Summary This PR bumps the Agent Data Plane version to 1.6.3 for the next patch release. It replicates what the [Bump ADP Version](.github/workflows/bump-adp-version.yml) workflow should have done on the `1.6.2` tag push. That run failed after 1.6.2 was re-cut, and the original bump PR (#2702) was closed because it was based on the pre-re-cut commit. ### Details of change - **Previous version:** 1.6.2 - **New version:** 1.6.3 ## Change Type - [ ] Bug fix - [ ] New feature - [x] Non-functional (chore, refactoring, docs) - [ ] Performance Co-authored-by: jesse.szwedko <jesse.szwedko@datadoghq.com>
fix(config): decode map settings from agent stream (backport 1.5.x) (#… …2743) ## Summary Backport of #2701 (itself a backport of #2700) to ADP 1.5.x. When the new config stream is enabled, the Agent streams map-typed settings set via environment variables, like `DD_ADDITIONAL_ENDPOINTS`, as the raw JSON string. 1.5.x rejects that string at the strict startup gate, so ADP fails to start and restarts in a loop. This lets those settings accept either a map or a JSON-encoded map string. ```mermaid flowchart LR A["DD_ADDITIONAL_ENDPOINTS='{...}'"] --> B[Agent config stream<br/>sends raw string] B --> C{translate_strict} C -- before --> D["'expected a map'<br/>startup fails, restart loop"] C -- after --> E[decoded map<br/>ADP starts] ``` Differences from #2701, because 1.5.x has diverged: - `cast_de.rs` doesn't exist on 1.5.x, so `deserialize_map_or_json_string` / `deserialize_json_object_or_string` live in `list_de.rs`. - The 1.6.x-only JSON-array decoder and the unrelated V3 series mode test are left out. - The integration case asserts `shared.endpoints.additional_endpoints` via `/config/runtime` rather than the `configured_endpoint` debug log, which 1.5.x doesn't emit. The unrelated `DD_SITE` assertions are also left out. The CLI that reads that view scrubs API keys, so the expected value is the scrubbed key. This PR also includes a Windows CI fix that isn't part of #2701. `unit-tests-windows-amd64` was failing before compiling anything: `rustup toolchain install` updated rustup itself, and the next `rustup` call then failed with `rustup.exe is not a valid subcommand`. The release zip jobs share the same setup, so they would hit the same failure. `windows-rust-env.psm1` now passes `--no-self-update`, since rustup comes from the build image. Main and 1.6.x use the same script and likely need the same change. ## Test plan - [x] New `connected_startup_accepts_json_encoded_map_settings` regression test in `agent-data-plane-config-system` (verified it fails without the fix) - [x] New `list_de` tests for JSON-encoded string maps and free-form objects - [x] `adp-config-stream` integration case (updated to set `DD_ADDITIONAL_ENDPOINTS`) passes on Linux CI - [ ] `unit-tests-windows-amd64` gets past rustup setup with `--no-self-update` References: DADP-204 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Toby Lawrence <tobz@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
fix(aggregate): answer liveness probes during flush (#2726) ## Human Summary If the encoder downstream is applying backpressure, the upstream aggregator flush can wait a long time in a CPU starvation scenario. This adds a `select!` loop inside that flush, one branch of which is the flush future, so that a `/live` probe can respond during that await. Note: if this seems like a good approach, I suppose all components should probably get it, and that makes me wonder if there is something more fundamental here that could be done with the component traits themselves. But this is a quick fix for an issue we just saw in production, and I'd say I'm about 67% convinced that it will help. ## AI Summary On CPU-starved internal pods, `/live` can return 503 while the DogStatsD aggregator flushes. The flush runs inside one `select!` arm, so a slow downstream send prevents that loop from answering health probes until the entire flush finishes. Keep the flush future pinned and poll it alongside `health.live()`, with probes checked first. This leaves the aggregation state, output order, and memory use unchanged; it does not make the aggregator consume input during a flush. **Limit:** This keeps *the aggregator* live, but it is not a complete liveness fix. Upstream components can still block sending into it and fail their own probes. It does not fix CPU starvation or sender-side packet drops. The other DogStatsD transforms that await dispatch inside their own loops need separate investigation. ## Change Type - [x] Bug fix ## How did you test this PR? - `make fmt`, `cargo check --workspace`, `make check-docs`, `make check-clippy`, and `cargo nextest run -p saluki-components aggregate` (29 tests passed). - Exploratory local harness (not included in the PR): a one-thread Tokio topology and a separate one-thread health checker inside Lima Docker, pinned to one CPU with two CPU-burning neighbours. A synthetic slow consumer behind a bounded channel models downstream work; this is **not** a production or full-pipeline reproduction. - At 205k contexts and 25k updates/s, with a consumer that could not keep up, the aggregator's worst probe response fell from ~17 s to ~0.07 s; its failed probes fell to zero. Total `/live` failures fell from 39/87 to 24/87 one-second polls, but the remaining failures were from the upstream source. With a consumer that kept up, neither version failed `/live` in the local test. ## References None. Co-authored-by: matt.briggs <matt.briggs@datadoghq.com>
chore(ci): update images trigger project to ddoghq/images (#2543) ## Summary The `DataDog/images` GitLab project is moving to the `ddoghq` org next Tuesday, and any hardcoded references need to be updated beforehand so our integration doesn't break after the migration. This updates the pipeline trigger references in `.gitlab/build.yml` to point at `ddoghq/images` instead. Do not merge until the `images` repo migration cuts over. ## Test plan - [ ] N/A — config-only change, verified no other `DataDog/images` references remain in the repo Co-authored-by: jesse.szwedko <jesse.szwedko@datadoghq.com>
PreviousNext