feat: Add tokio-metrics runtime metrics integration by yulnr · Pull Request #256 · awslabs/metrique

yulnr · 2026-03-24T17:41:53Z

Closes #248

Depends on

tokio-rs/tokio-metrics#121 - once merged and released, will update metrique-util/Cargo.toml from git dep to crates.io version. The [patch.crates-io] section in the workspace Cargo.toml should needs to be removed.

Summary

Adds a new feature tokio-metrics-bridge to metrique-util which adds a bridge for tokio-metrics.

Main Changes

metrique-writer-core: Refactors join in AttachHandle into a shutdown registry. Background tasks register cleanup functions via register_shutdown_fn that run (in LIFO order) before the sink detaches on drop.
metrique-util: New tokio-metrics-bridge feature with AttachGlobalEntrySinkTokioMetricsExt::subscribe_tokio_runtime_metrics. Spawns a reporter task that appends RuntimeMetrics snapshots at a configurable interval, automatically aborted on handle drop.

Usage

use metrique_util::{AttachGlobalEntrySinkTokioMetricsExt, TokioRuntimeMetricsConfig};

// After attaching a sink:
let _handle = ServiceMetrics::attach_to_stream(emf.output_to(std::io::stderr()));

// Subscribe: reporter runs in the background, aborted when _handle drops.
let config = TokioRuntimeMetricsConfig::default().with_interval(Duration::from_secs(30));
ServiceMetrics::subscribe_tokio_runtime_metrics(config);

See metrique/examples/tokio-metrics.rs for a better example.

How it works

The RuntimeMetrics struct in tokio-metrics derives metrique_writer::Entry (behind the metrique-integration feature flag), so each snapshot is appended directly to the sink as a structured metric record with all runtime fields (worker utilization, park counts, queue depths, steal counts, poll durations, etc.).

Open question: duplicate subscriptions

Right now nothing prevents calling subscribe_tokio_runtime_metrics twice, which would just spawn two reporter tasks appending duplicate metrics. It won't break anything, but it's almost certainly users won't intend that to happen.

I left it unguarded for now since the options each have tradeoffs and I'd rather get some input:

Panic on duplicate: track whether a sink type has already subscribed and panic on the second call. Needs a per-type flag.
Just log a warning
Do nothing

Happy to add a guard if there is a preferred approach, otherwise I'm leaning towards having at least a warning.

Update: this now warns of duplicate subscriptions

Testing

Adds some shutdown registry tests to metrique-writer-core
Adds tests to metrique-util for the tokio-metrics-feature

🔏 By submitting this pull request

I confirm that I've made a best effort attempt to update all relevant documentation.
I confirm that my contribution is made under the terms of the Apache 2.0 license.

jlizen

Right now nothing prevents calling subscribe_tokio_runtime_metrics twice, which would just spawn two reporter tasks appending duplicate metrics. It won't break anything, but it's almost certainly users won't intend that to happen.

Logging a warning seems fine to me for now

jlizen · 2026-03-25T15:42:09Z

+/// Configuration for Tokio runtime metrics bridge subscriptions.
+#[derive(Debug, Clone, Copy)]
+#[non_exhaustive]
+pub struct TokioRuntimeMetricsConfig {


This API to emit on an interval makes sense to me.
But, would it be worth having a different option, to fold the current runtime metrics into a given metrics struct? Some sort of #[metrics(with_tokio_metrics)] macro support?

Definitely not something to impl in this PR, but wondering if it merits a backlog (any opinion @rcoh ?)

I think the cleanest thing would be to have EmbeddedTokioMetrics that had a ::default() impl so you could embed it as a field (rather than needing to drop into having macro support)

yulnr · 2026-03-31T18:01:46Z

Following up on tokio-rs/tokio-metrics#121 (comment)

With tokio-metrics using #[metrics(subfield)] without rename_all, I'm looking into how to let callers pick a name style.

InflectableEntry is generic over NS: NameStyle, so to expose this to callers we have some options:

Generic method

// Default (Identity — snake_case):
ServiceMetrics::subscribe_tokio_runtime_metrics(config);

// PascalCase:
ServiceMetrics::subscribe_tokio_runtime_metrics::<PascalCase>(config);

Generic config

let config = TokioRuntimeMetricsConfig::default()
    .with_name_style::<PascalCase>();

Config becomes TokioRuntimeMetricsConfig<PascalCase>

Now If we introduce a new public enum we could also do:

Runtime enum on config

let config = TokioRuntimeMetricsConfig::default()
    .with_name_style(MetricNameStyle::PascalCase);

(dispatches via match internally)
I think much better for users, only downside is we'd have a mirror of NameStyle enum from metrique-macro, but that should be easy to keep in sync.

If we support folding via #[metrics] with automatic appending

Here @jlizen mentioned some sort of macro support for folding RuntimeMetrics into another metrics struct (I'm assuming that would automatically pull in the RuntimeMetrics data). If we go that route then we could solve this in multiple ways at the struct level rather than subscription config level, one example: #[metrics(rename_all = "PascalCase")]

What already works as this stands now: since RuntimeMetrics is a subfield, users can flatten it into their own #[metrics(rename_all = "PascalCase", with_tokio_metrics)] struct and the naming is inherited. Though this is less likely to be used that way because users would need to write the monitor loop and append to sink themselves.

Do we have any preferences here, or other approaches I'm missing?

cc @rcoh

rcoh · 2026-03-31T21:53:03Z

I think I have a slight preference for the enum. No issues with mirroring it. We could eventually have it set on the sink too for runtime-cased metrics.

Overall approach seems good to me

rcoh

Nice!

rcoh · 2026-04-10T14:55:50Z

+/// Configuration for Tokio runtime metrics bridge subscriptions.
+#[derive(Debug, Clone, Copy)]
+#[non_exhaustive]
+pub struct TokioRuntimeMetricsConfig {


I think the cleanest thing would be to have EmbeddedTokioMetrics that had a ::default() impl so you could embed it as a field (rather than needing to drop into having macro support)

rcoh · 2026-04-10T14:59:19Z

+) -> (tokio::task::AbortHandle, JoinHandle<()>) {
+    let interval = config.interval;
+    let name_style = config.name_style;
+    let worker = tokio::spawn(async move {


we will eventually (but can probably punt) need to provide an overridable executor

Good idea, would you be ok with doing this as a follow up? I can follow up with some work that does it for metrique in general, we do some hardcoded spawning in metrics-rs integration and in some of the internals as well.
I'm also curious as to how to catch this on CI, tests or even dial9 warnings, I have some ideas I'd like to try out later

rcoh · 2026-04-10T15:07:38Z

+#[doc(hidden)]
+pub type ShutdownFn = Box<dyn FnOnce() + Send>;
+
+#[doc(hidden)]
+pub type ShutdownRegistry = Mutex<Vec<ShutdownFn>>;


what are we attempting to achieve by making these doc hidden? register_shutdown_fn is not doc hidden an in practice, we would't be able to change these types. Probably worth just making newtypes for them for backwards compat and not trying to hide them in the API.

rcoh · 2026-04-10T15:08:49Z

+        tracing::debug!("tokio runtime metrics reporter stopped");
+    });
+    let worker_abort = worker.abort_handle();
+    let monitor = tokio::spawn(async move {


do we actually need two separate tasks here? (Maybe?) Is this a general pattern that all bridge-reporters are going to need?

I added this earlier when Jess asked about guarding against task panics and having an error. Is not truly needed contract-wise, I now simplified to use a single abort handle so we only pass one in the abort closure. Other bridges could opt not to do this, but perhaps it's preferred that we catch these and use tracing to log errors rather than having a single task panic and go into stderr?

rcoh · 2026-04-11T00:23:00Z

 toml = "0.9"
 walkdir = "2"
+
+[patch.crates-io]


Still needed?

I'm looking into this, because without it I get compilation errors:, note the there are multiple different versions of crate metrique_core in the dependency graph:

error[E0599]: no method named `close` found for struct `tokio_metrics::RuntimeMetrics` in the current scope --> metrique-util/src/tokio_metrics_reporter.rs:126:33 | 126 | entry: snapshot.close(), | ^^^^^ | note: there are multiple different versions of crate `metrique_core` in the dependency graph --> /Users/joulei/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/metrique-core-0.1.18/src/lib.rs:131:1 | 131 | pub trait CloseValue { | ^^^^^^^^^^^^^^^^^^^^ this is the expected trait | ::: metrique-core/src/lib.rs:131:1 | 131 | pub trait CloseValue { | -------------------- this is the trait that was imported = help: you can use `cargo tree` to explore your dependency tree help: some of the expressions' fields have a method of the same name

cargo is pulling metrique-core from crates.io for tokio-metrics, and from path for metrique.

So at least we'll need it for local development until release. But I'm not sure yet if once this is released in crates.io, cargo will know how to unify the crates.io metrique deps from tokio-metrics, with the path deps that metrique has between crates. I'm trying to find docs on this or what others libs are doing (both serde and tokio have similar patches)

Got it ok makes sense. Just add a comment

Adds a tokio-metrics-bridge feature to metrique-util that periodically appends tokio RuntimeMetrics snapshots to a global entry sink. This is powered by a new shutdown registry on AttachHandle. Background tasks register cleanup functions via register_shutdown_fn, which run in LIFO order before the sink detaches. Under tokio_unstable, poll_time_histogram bucket counts are paired with their ranges from the runtime handle and emitted as a separate distribution metric entry.

…y alive

…d yet

jlizen reviewed Mar 25, 2026

View reviewed changes

yulnr force-pushed the tokio-metrics-integration branch 4 times, most recently from c70838c to ef8db10 Compare March 30, 2026 11:06

yulnr mentioned this pull request Mar 30, 2026

Breaking: Add optional metrique Entry derive for RuntimeMetrics tokio-rs/tokio-metrics#121

Merged

yulnr commented Mar 30, 2026

View reviewed changes

Comment thread metrique-util/src/tokio_metrics_reporter.rs Outdated

yulnr force-pushed the tokio-metrics-integration branch 2 times, most recently from d227e56 to dab8555 Compare April 6, 2026 11:12

yulnr marked this pull request as ready for review April 10, 2026 14:51

rcoh reviewed Apr 10, 2026

View reviewed changes

rcoh approved these changes Apr 11, 2026

View reviewed changes

yulnr added 16 commits April 13, 2026 12:59

fix: use mem::forget in AttachHandle::forget to keep shutdown registr…

cf2c429

…y alive

Improve tests and docs

e49923b

Use reference-style link in metrique-util README

c22dc77

docs: Switch to doc_auto_cfg in metrique-util and metrique-writer-core

3f39320

docs: Add tokio-metrics usage docs and sinks guide breadcrumb

e05e56a

Add must_use to TokioRuntimeMetricsConfig

bb7867b

Refactor with_interval to take self

e559b26

Clarifiy subscribe_tokio_runtime_metrics panics if no sink is attache…

c0404b8

…d yet

Monitor tokio metrics reporter task for panics

084ff73

Clean up shutdown registry in AttachHandle::forget

477249c

refactor: simplify tokio-metrics bridge

547b2ce

Add a warning for duplicate subscriptions

e4a8cd5

feat: add configurable MetricNameStyle for tokio metrics bridge

2216edd

Bump tokio-metrics git dep (temporary)

06c2d05

Use metrique dep and extract DynamicInflectionEntry

ba1bdb1

yulnr added 10 commits April 13, 2026 13:01

Fix docs link to AttachHandle

d1f043e

Improve tokio-metrics reporter tests

c676889

Remove duplicate subscription guard

eca1132

Use lazy sink instead of Self::sink()

d5cf3db

Clarify usage of tokio_unstable

052bb19

Add DynamicInflectionEntry prefix behavior tests

8285ad2

Use assert2 on tests

a803c93

Fix format

f8e4274

Replace type aliases with newtypes for ShutdownFn and ShutdownRegistry

256cb98

Simplify spawn_tokio_runtime_metrics_task to return a single AbortHandle

22c41be

yulnr force-pushed the tokio-metrics-integration branch from 3573c8d to 22c41be Compare April 13, 2026 11:17

yulnr added 2 commits April 13, 2026 13:39

Fix clippy warning

275b576

Fix format

2a56a2f

rcoh approved these changes Apr 13, 2026

View reviewed changes

Comment thread metrique-writer-core/src/global.rs Outdated

Comment thread metrique-writer-core/src/global.rs Outdated

yulnr added 4 commits April 13, 2026 14:58

Comment metrique deps patch

3272f87

Append and drain on reverse order

d13f8ce

Make drain_and_run consume self

ce5d169

Fix format

2045776

rcoh approved these changes Apr 13, 2026

View reviewed changes

Merge branch 'main' into tokio-metrics-integration

18863a0

This was referenced Apr 13, 2026

feat: Add the ability to provide an overridable executor for spawned tasks #268

Open

feat: Support folding RuntimeMetrics (tokio-metrics) into other structs #269

Open

rcoh merged commit 5110172 into awslabs:main Apr 13, 2026
10 checks passed

jlizen mentioned this pull request Apr 20, 2026

feat: Support "folding" tokio metrics into other structs #278

Closed

fmzbl mentioned this pull request May 4, 2026

sysinfo bridge #283

Open

2 tasks

Conversation

yulnr commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Depends on

Summary

Usage

How it works

Open question: duplicate subscriptions

Testing

Uh oh!

jlizen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yulnr commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rcoh commented Mar 31, 2026

Uh oh!

rcoh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yulnr Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yulnr Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yulnr commented Mar 24, 2026 •

edited

Loading

yulnr commented Mar 31, 2026 •

edited

Loading

yulnr Apr 13, 2026 •

edited

Loading

yulnr Apr 13, 2026 •

edited

Loading