chore(tests): antithesis harness, durability harness scenario by blt · Pull Request #25562 · vectordotdev/vector

blt · 2026-06-02T21:10:25Z

Summary

This PR introduces a new antithesis harness for Vector, similar to the work I have done recently for saluki. The goal here is to provide a stable foundation for this project to add deterministic simulation testing. Vector project has a good practice of internal property testing and the like but it's tricky to demonstrate good behavior across multiple, networked Vector instances.

To that end and to provide a demonstration about what we can do with Antithesis I've put together a test scenario vector_to_vector_e2e_disk that is meant to demonstrate a Vector -> Vector maintains conservation of input and liveness. There's more detail in that scenario about how it functions, see tests/antithesis/scenarios/. The scenario is not rigged up into CI in this PR. Submission requires Antithesis credentials and is done now manually.

Finally, I have added antithesis_sdk assertions through the code. I don't yet have a House Style for these that I like yet, open to feedback here.

Vector configuration

N/A

How did you test this PR?

The change is itself test code.

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

blt · 2026-06-02T21:10:41Z

chore(tests): antithesis harness, durability harness scenario #25562 : 2 dependent PRs (#25525 , #25571 ) 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copilot

Pull request overview

This PR adds an Antithesis-based deterministic simulation testing harness to Vector, centered on a new multi-node “vector → vector” end-to-end scenario for exercising disk_v2 durability/conservation and liveness under injected faults. It also checks in a large “scratchbook” (property/evidence catalog) and introduces Antithesis SDK instrumentation behind a new antithesis Cargo feature.

Changes:

Add Antithesis scratchbook documentation (semantic claims ledger + property/evidence catalog) under tests/antithesis/scratchbook/.
Add a runnable Antithesis scenario vector_to_vector_e2e_disk (compose config, workload/oracle binaries, launch script, reload fault helper).
Add antithesis feature wiring + SDK-based assertions/instrumentation in lib/vector-buffers (disk_v2 reader/writer/ledger), plus workspace and dependency updates.

Reviewed changes

Copilot reviewed 76 out of 77 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/antithesis/scratchbook/semantic-claims-ledger.md	Semantics-first claims vs code reality ledger for the disk buffer e2e scenario.
tests/antithesis/scratchbook/properties/throughput-progresses-under-contention.md	Property/evidence note describing throughput under contention.
tests/antithesis/scratchbook/properties/sink-failure-not-silently-acked.md	Property/evidence note for sink-error ack semantics.
tests/antithesis/scratchbook/properties/recovery-completes-after-crash.md	Property/evidence note for crash recovery completion.
tests/antithesis/scratchbook/properties/record-id-monotonicity-holds.md	Property/evidence note for record-id monotonicity invariants.
tests/antithesis/scratchbook/properties/overflow-chain-no-unaccounted-gap.md	Property/evidence note for overflow buffer chain behavior.
tests/antithesis/scratchbook/properties/no-corrupted-record-delivered.md	Property/evidence note for corruption validation and delivery gating.
tests/antithesis/scratchbook/properties/multi-hop-conservation-no-loss.md	Property/evidence note for N-hop conservation expectations.
tests/antithesis/scratchbook/properties/fsync-window-bounded-under-clock-jitter.md	Property/evidence note for fsync window and clock jitter faults.
tests/antithesis/scratchbook/properties/foreign-data-file-no-writer-stall.md	Property/evidence note for foreign `.dat` file impact.
tests/antithesis/scratchbook/properties/file-id-rollover-stays-coordinated.md	Property/evidence note for file-id rollover coordination.
tests/antithesis/scratchbook/properties/every-written-event-eventually-delivered.md	Property/evidence note for at-least-once delivery across crashes.
tests/antithesis/scratchbook/properties/durable-unacked-events-survive-crash.md	Property/evidence note for durability of fsync’d-but-unacked events.
tests/antithesis/scratchbook/properties/dropped-events-are-counted.md	Property/evidence note for drop accounting metrics expectations.
tests/antithesis/scratchbook/properties/delivery-is-at-least-once-not-exactly-once.md	Clarification note: duplicates are expected (anti-vacuity).
tests/antithesis/scratchbook/properties/corruption-skip-record-id-accounting-consistent.md	Evidence note linking corruption skip to accounting/monotonicity.
tests/antithesis/scratchbook/properties/corruption-skip-loss-is-counted.md	Evidence note for corruption skip loss accounting/metrics.
tests/antithesis/scratchbook/properties/corruption-skip-loss-bounded.md	Evidence note for bounding loss on corruption skip behavior.
tests/antithesis/scratchbook/properties/corruption-is-detected-and-recovered.md	Reachability note for corruption detection/recovery execution.
tests/antithesis/scratchbook/properties/ack-is-per-hop-not-transitive.md	Clarification note about per-hop vs transitive acks.
tests/antithesis/scratchbook/properties/ack-does-not-imply-durability.md	Evidence note about 200/ack not implying fsync durability.
tests/antithesis/scratchbook/grind-plan.md	Execution plan for running/triaging Antithesis “shots”.
tests/antithesis/scratchbook/existing-assertions.md	Inventory of committed Antithesis assertions and harness checks.
tests/antithesis/scratchbook/.markdownlint.jsonc	Scratchbook-scoped markdownlint relaxations.
tests/antithesis/scratchbook/_external-references-digest.md	Digest of in-repo/external references and known bug targets.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/tail.yaml	Tail node Vector config for scenario.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/src/bin/parallel_driver_produce.rs	Workload producer command: claim ID, POST to head, report ack.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/src/bin/oracle.rs	Oracle service: issues IDs, validates delivery + payload integrity.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/src/bin/eventually_conservation.rs	Eventually-phase judge: drain + conservation + liveness probe.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/README.md	Scenario documentation.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/launch.sh	Pinned `snouty launch` wrapper with fault profile.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/head.yaml	Head node Vector config for scenario.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/head.b.yaml	Alternate head config swapped by reload fault.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/Dockerfile	Builds SUT Vector + workload binaries with Antithesis coverage flags.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/docker-compose.yaml	Compose topology for head/tail/oracle + volumes/healthchecks.
tests/antithesis/scenarios/vector_to_vector_e2e_disk/Cargo.toml	Scenario crate manifest (workload/oracle binaries).
tests/antithesis/scenarios/vector_to_vector_e2e_disk/anytime_reload.sh	In-container config swap + SIGHUP reload helper.
tests/antithesis/README.md	Top-level Antithesis test directory README.
tests/antithesis/harness/src/lib.rs	Shared harness helpers (payload generation/validation).
tests/antithesis/harness/Cargo.toml	Harness crate manifest.
tests/antithesis/harness/.gitignore	Harness build artifacts ignore.
tests/antithesis/AGENTS.md	Agent guidance for running Antithesis scenarios and launching shots.
LICENSE-3rdparty.csv	Adds Antithesis crates to license inventory.
lib/vector-lib/Cargo.toml	Adds `antithesis` feature forwarding to `vector-buffers/antithesis`.
lib/vector-buffers/src/variants/disk_v2/writer.rs	Adds Antithesis assertions around file size, blocking, rollovers, large records.
lib/vector-buffers/src/variants/disk_v2/reader.rs	Adds Antithesis assertions for monotonicity, bad read roll, delivery invariants.
lib/vector-buffers/src/variants/disk_v2/mod.rs	Exposes `common` module + antithesis env override for max data file size.
lib/vector-buffers/src/variants/disk_v2/ledger.rs	Adds Antithesis assertions for underflow detection and reopen reachability.
lib/vector-buffers/src/lib.rs	Links Antithesis instrumentation + re-exports write buffer size under feature.
lib/vector-buffers/Cargo.toml	Adds optional Antithesis deps + `antithesis` feature definition.
Cargo.toml	Adds workspace members + Antithesis deps + top-level `antithesis` feature wiring.
Cargo.lock	Locks new crates and dependencies.
.github/actions/spelling/excludes.txt	Excludes `tests/antithesis/` from spelling checks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fe00bcb6c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 18fa9da292

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3614c5a966

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5209ae6df4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24d8e0f1ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Antithesis research artifacts for disk buffer v2: SUT analysis, 29-property catalog (incl. the silent data-loss cluster), property relationships, evaluation synthesis, per-property evidence files, and the run log. Internal references genericized.

Self-driving exercisers (disk_v2_antithesis: #21683 accounting cluster; disk_v2_lossfinder: 7-scenario data-loss fault menu with a no-silent-loss oracle) plus the full Antithesis harness: Dockerfiles, docker-compose configs (block / drop variants / direct / lossfinder), the Vector-SUT workload, and the test templates under tests/antithesis/test/v1. Registers the examples + serde_json dev-dep.

Adds antithesis_sdk + antithesis-instrumentation deps and three #[inline] assert_always! guards that fold away in production builds (no-ops outside Antithesis) at the unchecked-arithmetic sites: ledger.rs decrement_total_buffer_size and get_total_records (#21683 / 0-1 underflow), and reader.rs:524 data-file size-delta. These guide Antithesis to the accounting-underflow cluster (organically reproduced #21683 in run D0).

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e79a45c3a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…sis/

Gate antithesis-instrumentation (ELF .init_array, Linux-only) behind cfg(target_os = "linux") in the harness Cargo.toml and extern crate declarations. antithesis_sdk with `full` compiles on all platforms so it stays unconditional. Removes WORKSPACE_EXCLUDE workaround from unit_windows.yml.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da4714b31f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-10T15:40:41Z

@@ -0,0 +1,5 @@
+# Per-scenario settings for ../launch.sh. Only the bits that differ from every
+# other scenario live here; the launcher supplies the rest.
+SCENARIO_TEST_NAME=vector_e2e


Add the missing vector_e2e compose file

When this newly added scenario config is selected with the shared launcher (./launch.sh vector_e2e), the launcher exits before build/submission because tests/antithesis/scenarios/vector_e2e/docker-compose.yaml is absent while scenarios/launch.sh hard-requires one. I checked the scenario tree with find tests/antithesis/scenarios -maxdepth 2 -type f, and only this launch.env exists under vector_e2e, so the memory-buffer scenario advertised here cannot be launched until its compose/config files are added or this incomplete launch.env is removed.

Useful? React with 👍 / 👎.

…ess-first-scenario # Conflicts: # .github/actions/spelling/excludes.txt

pront

/ci-run-unit-windows

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85656409e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…fg gates

github-actions Bot added the domain: ci Anything related to Vector's CI environment label Jun 2, 2026

blt mentioned this pull request Jun 2, 2026

chore(unit tests): failing disk_v2 bug demonstrations #25525

Closed

9 tasks

blt changed the title ~~docs(buffers): disk_v2 data-loss research scratchbook~~ chore(tests): antithesis harness, durability harness scenario Jun 2, 2026

blt force-pushed the blt/antithesis-harness-first-scenario branch from ec466b3 to 8fe00bc Compare June 3, 2026 13:17

blt requested a review from Copilot June 3, 2026 17:59

blt marked this pull request as ready for review June 3, 2026 17:59

blt requested a review from a team as a code owner June 3, 2026 17:59

Copilot started reviewing on behalf of blt June 3, 2026 17:59 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jun 3, 2026

View reviewed changes

Comment thread tests/antithesis/scenarios/vector_to_vector_e2e_disk/head.yaml Outdated

Comment thread tests/antithesis/harness/src/lib.rs

blt force-pushed the blt/antithesis-harness-first-scenario branch 2 times, most recently from 615dda3 to 18fa9da Compare June 3, 2026 18:44

chatgpt-codex-connector Bot reviewed Jun 3, 2026

View reviewed changes

blt force-pushed the blt/antithesis-harness-first-scenario branch 2 times, most recently from 14630a6 to 3614c5a Compare June 3, 2026 20:10

chatgpt-codex-connector Bot reviewed Jun 3, 2026

View reviewed changes

Comment thread tests/antithesis/scenarios/vector_to_vector_e2e_disk/anytime_reload.sh Outdated

Comment thread lib/vector-buffers/Cargo.toml Outdated

Comment thread tests/antithesis/scenarios/vector_to_vector_e2e_disk/src/bin/eventually_conservation.rs Outdated

blt mentioned this pull request Jun 3, 2026

fix(buffers): disk buffers v2 reopen buffer after restart crash loop #25570

Draft

9 tasks

blt force-pushed the blt/antithesis-harness-first-scenario branch from 3614c5a to 5209ae6 Compare June 4, 2026 00:25

chatgpt-codex-connector Bot reviewed Jun 4, 2026

View reviewed changes

Comment thread tests/antithesis/scenarios/vector_to_vector_e2e_disk/launch.sh Outdated

Comment thread tests/antithesis/scenarios/vector_to_vector_e2e_disk/launch.sh Outdated

Comment thread tests/antithesis/scenarios/vector_to_vector_e2e_disk/Dockerfile Outdated

blt mentioned this pull request Jun 4, 2026

chore(tests): Antithesis Vector end-to-end ack scenario #25571

Open

9 tasks

blt force-pushed the blt/antithesis-harness-first-scenario branch from 5209ae6 to 24d8e0f Compare June 4, 2026 13:17

chatgpt-codex-connector Bot reviewed Jun 4, 2026

View reviewed changes

Comment thread tests/antithesis/scenarios/launch.sh

Comment thread tests/antithesis/harness/src/bin/oracle.rs

blt force-pushed the blt/antithesis-harness-first-scenario branch from 24d8e0f to 2445cb8 Compare June 4, 2026 14:57

github-actions Bot added the domain: vdev Anything related to the vdev tooling label Jun 4, 2026

blt force-pushed the blt/antithesis-harness-first-scenario branch from 2445cb8 to 5e79a45 Compare June 4, 2026 15:05

blt added 3 commits June 4, 2026 15:19

chatgpt-codex-connector Bot reviewed Jun 4, 2026

View reviewed changes

Comment thread tests/antithesis/AGENTS.md Outdated

Comment thread tests/antithesis/harness/src/bin/oracle.rs

blt force-pushed the blt/antithesis-harness-first-scenario branch 2 times, most recently from 487ed7e to 1dbdb4e Compare June 4, 2026 16:32

compress scratchbook

b2c70cd

blt force-pushed the blt/antithesis-harness-first-scenario branch from 1dbdb4e to b2c70cd Compare June 4, 2026 17:08

blt added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Jun 4, 2026

pront added 2 commits June 10, 2026 11:06

chore: add single-machine-performance as codeowners for tests/antithe…

f783e9a

…sis/

chatgpt-codex-connector Bot reviewed Jun 10, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into blt/antithesis-harn…

3fb5f27

…ess-first-scenario # Conflicts: # .github/actions/spelling/excludes.txt

github-actions Bot removed the domain: ci Anything related to Vector's CI environment label Jun 10, 2026

pront reviewed Jun 10, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jun 10, 2026

View reviewed changes

Comment thread vdev/src/commands/check/rust.rs Outdated

pront force-pushed the blt/antithesis-harness-first-scenario branch from 8565640 to 58c9222 Compare June 10, 2026 17:11

chore(tests): rename antithesis-harness, drop WORKSPACE_EXCLUDE via c…

d3094d8

…fg gates

pront force-pushed the blt/antithesis-harness-first-scenario branch from 58c9222 to d3094d8 Compare June 10, 2026 17:15

cargo fmt

68acbd8

github-actions Bot removed the domain: vdev Anything related to the vdev tooling label Jun 10, 2026

Conversation

blt commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

blt commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

blt commented Jun 2, 2026 •

edited

Loading

blt commented Jun 2, 2026 •

edited

Loading