feat(llmobs): add tool_definitions support to Tagger#8082
Conversation
Overall package sizeSelf size: 5.68 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: b452a1f | Docs | Datadog PR Page | Give us feedback! |
abcf58c to
6f9ab4d
Compare
6f9ab4d to
21f766e
Compare
BenchmarksBenchmark execution time: 2026-05-04 06:45:26 Comparing candidate commit b452a1f in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1347 metrics, 97 unstable metrics. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 21f766e1cd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
sabrenner
left a comment
There was a problem hiding this comment.
one comment that can be for a follow up but lgtm!!
Introduces `_ml_obs.meta.tool_definitions` for LLM spans, mirroring the
shape already produced by dd-trace-py. Each definition is
`{ name, description, schema }`.
- `constants/tags.js`: new `TOOL_DEFINITIONS` tag key.
- `tagger.js`: new `tagToolDefinitions(span, toolDefinitions)` that
validates entries and stores them in the registry. Matches the
`#filterToolCalls` validation style.
- `span_processor.js`: forwards the registry entry to
`meta.tool_definitions` on LLM spans.
- `test/llmobs/util.js`: assertion support for a new
`toolDefinitions` expected field.
No integration plugin emits tool definitions yet; this unblocks
Bedrock Converse (see follow-up PR) and opens the door for
parity follow-ups on openai/anthropic/langchain/ai-sdk.
Route `tool_definitions` through `#addObject` before assigning to `meta`, matching the existing `metadata` path. Direct assignment bypassed the circular-ref / BigInt substitution done via `UNSERIALIZABLE_VALUE_TEXT`, so a user-provided schema with a cycle or unserializable value would make `JSON.stringify(event)` throw in the span writer and the whole LLMObs span would get dropped. This is a generic-infra safety fix: the current Bedrock caller is safe by construction (the AWS SDK JSON.stringifies the request before send), but `tagToolDefinitions` is meant to be reused by other plugins and eventually the public `llmobs.annotate()` surface where the upstream-already-serialized guarantee doesn't hold.
Add an `else` branch that calls `#handleFailure` for non-array / empty input. Matches the validation pattern used by other tagger methods. Flagged on review.
Adds a `tagToolDefinitions` describe block in the tagger spec mirroring the `tagMetrics` style: one happy-path test asserting the tag is set, and one negative test asserting the malformed-input branch (non-array, empty array, undefined) routes through `#handleFailure`.
1cd9ac5 to
1451e6c
Compare
When a ConverseCommand or ConverseStreamCommand request includes a
`toolConfig.tools` array, map each `toolSpec` to LLMObs'
`{ name, description, schema }` shape and attach it to the span via
the new `tagToolDefinitions` API (see parent PR #8082).
- utils.js: new `extractConverseToolDefinitions` helper.
- llmobs/plugins/bedrockruntime.js: call it on Converse requests.
- spec: assert on the `toolDefinitions` round-trip.
Brings Node parity with dd-trace-py's Bedrock Converse integration
for the "Available Tools" UI section.
Ref: MLOB-3509
…sent `tagToolDefinitions` now logs a failure for non-array / empty input (see #8082). The Bedrock plugin previously called it unconditionally on every Converse request — `extractConverseToolDefinitions` returns `[]` when there's no `toolConfig`, which would now produce noisy `invalid_tool_definitions` logs on every tool-less Converse call. Gate the call on `length > 0`.
When a ConverseCommand or ConverseStreamCommand request includes a
`toolConfig.tools` array, map each `toolSpec` to LLMObs'
`{ name, description, schema }` shape and attach it to the span via
the new `tagToolDefinitions` API (see parent PR #8082).
- utils.js: new `extractConverseToolDefinitions` helper.
- llmobs/plugins/bedrockruntime.js: call it on Converse requests.
- spec: assert on the `toolDefinitions` round-trip.
Brings Node parity with dd-trace-py's Bedrock Converse integration
for the "Available Tools" UI section.
Ref: MLOB-3509
…sent `tagToolDefinitions` now logs a failure for non-array / empty input (see #8082). The Bedrock plugin previously called it unconditionally on every Converse request — `extractConverseToolDefinitions` returns `[]` when there's no `toolConfig`, which would now produce noisy `invalid_tool_definitions` logs on every tool-less Converse call. Gate the call on `length > 0`.
When a ConverseCommand or ConverseStreamCommand request includes a
`toolConfig.tools` array, map each `toolSpec` to LLMObs'
`{ name, description, schema }` shape and attach it to the span via
the new `tagToolDefinitions` API (see parent PR #8082).
- utils.js: new `extractConverseToolDefinitions` helper.
- llmobs/plugins/bedrockruntime.js: call it on Converse requests.
- spec: assert on the `toolDefinitions` round-trip.
Brings Node parity with dd-trace-py's Bedrock Converse integration
for the "Available Tools" UI section.
Ref: MLOB-3509
…sent `tagToolDefinitions` now logs a failure for non-array / empty input (see #8082). The Bedrock plugin previously called it unconditionally on every Converse request — `extractConverseToolDefinitions` returns `[]` when there's no `toolConfig`, which would now produce noisy `invalid_tool_definitions` logs on every tool-less Converse call. Gate the call on `length > 0`.
When a ConverseCommand or ConverseStreamCommand request includes a
`toolConfig.tools` array, map each `toolSpec` to LLMObs'
`{ name, description, schema }` shape and attach it to the span via
the new `tagToolDefinitions` API (see parent PR #8082).
- utils.js: new `extractConverseToolDefinitions` helper.
- llmobs/plugins/bedrockruntime.js: call it on Converse requests.
- spec: assert on the `toolDefinitions` round-trip.
Brings Node parity with dd-trace-py's Bedrock Converse integration
for the "Available Tools" UI section.
Ref: MLOB-3509
…sent `tagToolDefinitions` now logs a failure for non-array / empty input (see #8082). The Bedrock plugin previously called it unconditionally on every Converse request — `extractConverseToolDefinitions` returns `[]` when there's no `toolConfig`, which would now produce noisy `invalid_tool_definitions` logs on every tool-less Converse call. Gate the call on `length > 0`.
When a ConverseCommand or ConverseStreamCommand request includes a
`toolConfig.tools` array, map each `toolSpec` to LLMObs'
`{ name, description, schema }` shape and attach it to the span via
the new `tagToolDefinitions` API (see parent PR #8082).
- utils.js: new `extractConverseToolDefinitions` helper.
- llmobs/plugins/bedrockruntime.js: call it on Converse requests.
- spec: assert on the `toolDefinitions` round-trip.
Brings Node parity with dd-trace-py's Bedrock Converse integration
for the "Available Tools" UI section.
Ref: MLOB-3509
…sent `tagToolDefinitions` now logs a failure for non-array / empty input (see #8082). The Bedrock plugin previously called it unconditionally on every Converse request — `extractConverseToolDefinitions` returns `[]` when there's no `toolConfig`, which would now produce noisy `invalid_tool_definitions` logs on every tool-less Converse call. Gate the call on `length > 0`.
* feat(llmobs): add tool_definitions support to Tagger
Introduces `_ml_obs.meta.tool_definitions` for LLM spans, mirroring the
shape already produced by dd-trace-py. Each definition is
`{ name, description, schema }`.
- `constants/tags.js`: new `TOOL_DEFINITIONS` tag key.
- `tagger.js`: new `tagToolDefinitions(span, toolDefinitions)` that
validates entries and stores them in the registry. Matches the
`#filterToolCalls` validation style.
- `span_processor.js`: forwards the registry entry to
`meta.tool_definitions` on LLM spans.
- `test/llmobs/util.js`: assertion support for a new
`toolDefinitions` expected field.
No integration plugin emits tool definitions yet; this unblocks
Bedrock Converse (see follow-up PR) and opens the door for
parity follow-ups on openai/anthropic/langchain/ai-sdk.
* fix(llmobs): sanitize tool_definitions through serialization guard
Route `tool_definitions` through `#addObject` before assigning to
`meta`, matching the existing `metadata` path. Direct assignment
bypassed the circular-ref / BigInt substitution done via
`UNSERIALIZABLE_VALUE_TEXT`, so a user-provided schema with a cycle
or unserializable value would make `JSON.stringify(event)` throw in
the span writer and the whole LLMObs span would get dropped.
This is a generic-infra safety fix: the current Bedrock caller is
safe by construction (the AWS SDK JSON.stringifies the request
before send), but `tagToolDefinitions` is meant to be reused by
other plugins and eventually the public `llmobs.annotate()` surface
where the upstream-already-serialized guarantee doesn't hold.
* fix(llmobs): handleFailure in tagToolDefinitions for malformed input
Add an `else` branch that calls `#handleFailure` for non-array /
empty input. Matches the validation pattern used by other tagger
methods. Flagged on review.
* test(llmobs): cover tagToolDefinitions happy path and malformed input
Adds a `tagToolDefinitions` describe block in the tagger spec mirroring
the `tagMetrics` style: one happy-path test asserting the tag is set,
and one negative test asserting the malformed-input branch (non-array,
empty array, undefined) routes through `#handleFailure`.
* test(llmobs): cover tool_definitions sanitization in span processor
* test(llmobs): clarify intent of nested cycle in tool_definitions test
* test(llmobs): simplify tool_definitions test to forwarding only
What does this PR do?
Introduces
_ml_obs.meta.tool_definitionsfor LLMObs spans, mirroring the shape dd-trace-py's infra PR #14159 already ships. Each definition is{ name, description, schema }.This PR is infrastructure only, same scope as the Python counterpart.
Motivation
Prerequisite for bringing Node's Bedrock Converse spans (#8079) to parity with Python, which emits
tool_definitionsfromtoolConfig.tools. Opens the door for parity follow-ups on openai / anthropic / langchain / ai-sdk too.Test plan
npm run linton touched files — cleantoolDefinitionsround-trip end-to-end.