feat(kvp): add diagnostics layer over KvpPoolStore by peytonr18 · Pull Request #313 · Azure/azure-init

peytonr18 · 2026-07-09T23:35:44Z

Summary

This PR adds a DiagnosticsKvp layer that centralizes azure-init's KVP telemetry behavior, including event key formatting, message chunking, and message reassembly. This provides a single tested implementation that can be reused by the follow-up wiring PR and simplifies the existing logic in kvp.rs and logging.rs.

It also extends the existing dump command with a --parse-diagnostics mode for decoding and viewing KVP telemetry as readable events, making provisioning diagnostics easier to inspect and troubleshoot.

Specifically, this PR adds:

DiagnosticsKvp, a typed view over KvpPoolStore that implements telemetry-specific behavior while keeping the underlying store format-agnostic:
- Event keys formatted and parsed as <prefix>|<vm_id>|<level>|<name>|<event_id>.
- Message chunking at UTF-8 boundaries for values larger than a single KVP record, written via append_multiple under a single lock. Each chunk gets a unique |<subevent_index>-suffixed key so the Hyper-V host (which keeps one record per key) retains every fragment, and the chunks are reassembled when read.
- Classification of records as diagnostic events, raw records such as PROVISIONING_REPORT, or malformed event keys.
Diagnostics folded into the existing raw KVP commands via flags rather than a separate command tree:
- dump --parse-diagnostics reassembles chunked events and decodes each record. Raw output remains the default when the flag is absent.
- --include-raw also prints raw (non-event) records; --level, --name, and -n/--tail narrow the output to a filtered, events-only view.
- The filter flags (--level, --name, -n/--tail) require --parse-diagnostics, and --include-raw is mutually exclusive with them — it applies only to the unfiltered view, where raw records can appear.
- clear --diagnostics removes every diagnostic key (events and malformed event keys) while leaving raw records such as PROVISIONING_REPORT intact.

Example

A long event is stored as multiple records, each under a unique |<subevent_index> key. The raw view shows the fragments; --parse-diagnostics decodes and reassembles them into a single event:

$ libazureinit-kvp dump
azure-init-1.0.0|3f25…|INFO|user:create_user|8f3e…|0=Time: … | Event: creating
azure-init-1.0.0|3f25…|INFO|user:create_user|8f3e…|1= azureuser (uid 1000) …
PROVISIONING_REPORT=result=success|…

$ libazureinit-kvp dump --parse-diagnostics --name user:create_user
event level=INFO name=user:create_user event_id=8f3e… message=Time: … | Event: creating azureuser (uid 1000) …

$ libazureinit-kvp dump --parse-diagnostics --include-raw
event level=INFO name=user:create_user event_id=8f3e… chunks=2 message=Time: … | Event: creating azureuser (uid 1000) …
raw key=PROVISIONING_REPORT value=result=success|…

$ libazureinit-kvp dump --parse-diagnostics --include-raw --json
[{"chunks":2,"event_id":"8f3e…","kind":"event","level":"INFO","message":"…","name":"user:create_user"},{"key":"PROVISIONING_REPORT","kind":"raw","value":"result=success|…"}]

codecov-commenter · 2026-07-09T23:38:15Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.94%. Comparing base (02fc780) to head (7caa370).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #313      +/-   ##
==========================================
+ Coverage   95.70%   95.94%   +0.24%     
==========================================
  Files          23       24       +1     
  Lines        7705     8167     +462     
==========================================
+ Hits         7374     7836     +462     
  Misses        331      331

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR introduces a DiagnosticsKvp layer in libazureinit-kvp to provide a typed, reusable view over KvpPoolStore for diagnostic telemetry (event key formatting/parsing, UTF-8-safe chunking, and chunk reassembly), and adds new diag CLI subcommands to inspect this decoded event view.

Changes:

Added DiagnosticsKvp, DiagnosticEvent, and DiagnosticRecord with chunking/reassembly and record classification logic.
Added diag CLI subcommands (dump, events, tail, clear) for decoding and filtering diagnostic events (including JSON output support).
Added integration and CLI tests covering round-trips, chunking, classification, and clear scoping; enabled uuid v4 generation via crate features.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
libazureinit-kvp/tests/diagnostics.rs	New integration tests for diagnostics emit/read, chunking, classification, scoped clear, and concurrency behavior.
libazureinit-kvp/tests/cli.rs	New CLI tests validating `diag` command behaviors for text/JSON output, filtering, tailing, and clear confirmation.
libazureinit-kvp/src/lib.rs	Exposes the new diagnostics module and re-exports its public types/constants.
libazureinit-kvp/src/error.rs	Adds a dedicated error for rejecting `
libazureinit-kvp/src/diagnostics.rs	Implements the diagnostics layer: event key format/parse, UTF-8 chunking, reassembly, classification, and scoped clearing.
libazureinit-kvp/src/cli.rs	Adds `diag` subcommands and renders diagnostics records/events in text and JSON.
libazureinit-kvp/Cargo.toml	Enables `uuid` v4 feature required for `Uuid::new_v4()`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cjp256 · 2026-07-17T18:29:44Z

+        event_prefix: String,
+        /// Confirm the removal (required).
+        #[arg(long)]
+        yes: bool,


yes is overkill i think. It's volatile data and likely past the window of collection if someone is using it. Would suggest removing it.

Removed this

cjp256 · 2026-07-17T18:30:55Z

+    Clear {
+        /// VM identifier whose events to remove.
+        #[arg(long)]
+        vm_id: String,


this seems more like "delete" than "clear" to me.

I would expect clear to nuke all diagnostics-based keys.

I've adjusted this so clear gets rid of all diagnostic keys!

Append a chunk index to diagnostic event keys so the Hyper-V host retains all chunks. Reassembly and cleanup now operate on the base event key to correctly reconstruct and delete multi-record events.

…d clear --diagnostics

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

libazureinit-kvp/src/diagnostics.rs:455

reassemble() currently concatenates any consecutive records whose base_event_key() matches, which also merges adjacent raw records that happen to share the same key (e.g., two consecutive PROVISIONING_REPORT records appended via KvpPoolStore::append). That loses record boundaries and contradicts the doc comment that grouping is for event-chunk reassembly.

        while dumped
            .peek()
            .is_some_and(|(next, _)| base_event_key(next) == base)
        {

feat(kvp): add diagnostics layer over KvpPoolStore

603d113

peytonr18 mentioned this pull request Jul 10, 2026

[RFE] Support compression/decompression for large diagnostic KVP payloads (e.g. dmesg) #314

Open

cadejacobson approved these changes Jul 15, 2026

View reviewed changes

cjp256 and others added 2 commits July 17, 2026 13:03

Merge branch 'main' into probertson-diganostics-kvp

9a3da33

Merge branch 'main' into probertson-diganostics-kvp

76af3e6

Copilot AI review requested due to automatic review settings July 21, 2026 16:39

Copilot started reviewing on behalf of peytonr18 July 21, 2026 16:39 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread libazureinit-kvp/src/cli.rs

Comment thread libazureinit-kvp/src/diagnostics.rs

Comment thread libazureinit-kvp/src/cli.rs

cjp256 reviewed Jul 21, 2026

View reviewed changes

peytonr18 added 2 commits July 22, 2026 16:52

fix(kvp): preserve chunked diagnostic events

a13158c

Append a chunk index to diagnostic event keys so the Hyper-V host retains all chunks. Reassembly and cleanup now operate on the base event key to correctly reconstruct and delete multi-record events.

refactor(kvp): fold diag subcommands into dump --parse-diagnostics an…

7caa370

…d clear --diagnostics

Copilot AI review requested due to automatic review settings July 23, 2026 18:01

Copilot started reviewing on behalf of peytonr18 July 23, 2026 18:02 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread libazureinit-kvp/src/cli.rs

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(kvp): add diagnostics layer over KvpPoolStore#313

feat(kvp): add diagnostics layer over KvpPoolStore#313
peytonr18 wants to merge 5 commits into
Azure:mainfrom
peytonr18:probertson-diganostics-kvp

peytonr18 commented Jul 9, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjp256 Jul 17, 2026

Uh oh!

peytonr18 Jul 23, 2026

Uh oh!

cjp256 Jul 17, 2026

Uh oh!

peytonr18 Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

peytonr18 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example

Uh oh!

codecov-commenter commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjp256 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

peytonr18 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

cjp256 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

peytonr18 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

peytonr18 commented Jul 9, 2026 •

edited

Loading

codecov-commenter commented Jul 9, 2026 •

edited

Loading