feat(datadog_agent): support v3 series metrics intake - #26264
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Datadog Agent v3 series metrics intake support to the datadog_agent source so Vector can ingest metrics from Agent versions that default to the v3 endpoint, and updates tests to exercise the new path.
Changes:
- Add Warp routing + decoding for
/api/intake/metrics/v3/seriesand/api/intake/metrics/v3beta/series, translating v3 payloads into Vector metrics. - Add unit test coverage for decoding a v3 series payload.
- Restore the floating Datadog Agent v7 leg in the Datadog metrics E2E test matrix and add the v3 proto to codegen.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/e2e/datadog-metrics/config/test.yaml | Unpins the v7 agent matrix leg to exercise current Agent behavior (v3 default). |
| src/sources/datadog_agent/tests.rs | Adds a v3 endpoint constant, a raw-bytes sender helper, and a v3 decode test. |
| src/sources/datadog_agent/mod.rs | Wires in the generated v3 metrics protobuf module. |
| src/sources/datadog_agent/metrics.rs | Adds v3 series route and decoding/translation logic. |
| proto/vector/dd_metric_v3.proto | Introduces the v3 intake proto definition used for decoding. |
| changelog.d/25790_datadog_agent_v3_series.feature.md | Adds a changelog fragment documenting the user-facing feature. |
| build.rs | Adds the v3 proto to the build-time proto compilation inputs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5330fba0d3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8be2d900a4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 584d5bcbee
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| series.push(super::ddmetric_proto::metric_payload::MetricSeries { | ||
| resources, | ||
| metric: names[name_ref as usize].clone(), | ||
| tags: tagsets[tagset_ref as usize].clone(), |
There was a problem hiding this comment.
Account for tags cloned into every series
When many series reference the same nontrivial tagset, this clone materializes the complete tagset once per series before any events are emitted. For example, a roughly 64 KiB tagset plus 100,000 zero-point series rows can fit in a small request while allocating multiple gigabytes here. The fresh issue after the nested-expansion fix is that the 16 MiB accounting ends when the dictionary is built and does not include these per-series clones, so the endpoint remains vulnerable to memory exhaustion; bound total expanded series data or avoid cloning shared dictionaries.
Useful? React with 👍 / 👎.
| for set in &mut resources { | ||
| for pair in metadata.resources.chunks_exact(2) { | ||
| set.push((pair[0].clone(), pair[1].clone())); |
There was a problem hiding this comment.
Bound global-resource replication across resource sets
When metadata contains global resources, every entry in dict_resource_len causes all of those strings to be cloned into another set, even if that entry has length zero and is never referenced by a series. An untrusted payload can therefore combine thousands of one-byte zero lengths with a modest resource list and expand a sub-megabyte request into gigabytes of allocations before validation completes. Apply an expanded-resource budget analogous to the tagset limit, or defer applying global resources until a set is actually used.
Useful? React with 👍 / 👎.
| name_ref += data.name_refs[index]; | ||
| tagset_ref += data.tagset_refs[index]; | ||
| resources_ref += data.resources_refs[index]; | ||
| source_type_ref += data.source_type_name_refs[index]; | ||
| origin_ref += data.origin_info_refs[index]; |
There was a problem hiding this comment.
Check overflow while accumulating series references
When an untrusted payload supplies a valid positive reference followed by a delta such as i64::MAX, these additions overflow before the subsequent range checks execute. Rust builds with overflow checks enabled, including normal development builds and any hardened production profile, therefore panic the request task instead of returning 422; use checked_add for each reference accumulator, including the unit accumulator below.
Useful? React with 👍 / 👎.
Summary
Adds support for the Datadog Agent v3 series metrics intake format used by default by Datadog Agent 7.81.0 and newer.
The
datadog_agentsource now accepts the dictionary-encoded protobuf payload at/api/intake/metrics/v3/seriesand/api/intake/metrics/v3beta/series, translates it into Vector metrics, and preserves metric names, tags, resources, units, intervals, timestamps, values, and origin metadata.The Datadog metrics e2e matrix also restores the floating Agent
7test leg so current Agent versions exercise the v3 path.References
Closes: #25790
Vector configuration
No new configuration is required. Existing
datadog_agentsource configurations accept the new intake endpoint automatically.How did you test this PR?
cargo check --lib --no-default-features --features sources-datadog_agentcargo test --lib --no-default-features --features sources-datadog_agent datadog_agent::cargo clippy --lib --no-default-features --features sources-datadog_agent -- -D warningscargo fmt --all -- --checkIs this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.