Skip to content

refactor(deja): move HashMap and HashSet onto a facade whose hasher follows the request - #14368

Open
maverox wants to merge 14 commits into
work/deja-pin-to-main-and-miss-seamsfrom
work/deja-collections-facade
Open

maverox wants to merge 14 commits into
work/deja-pin-to-main-and-miss-seamsfrom
work/deja-collections-facade

Conversation

@maverox

@maverox maverox commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Note

Stacked on #14311 and merges after it. The facade calls deja::try_current_correlation_id, which exists only at the deja revision #14311 pins. This PR's diff is the facade alone.

Type of Change

  • Refactoring

Description

std::collections::HashMap and HashSet default to RandomState, whose keys come from per-process entropy, so two runs of one binary iterate the same map in different orders. That is invisible in ordinary use. It matters to record and replay: when a map's order reaches a response body, a query bind or an outgoing request, the replay differs from the recording and the difference is charged to the candidate rather than to the hasher.

This moves the workspace onto common_utils::collections::{HashMap, HashSet}. They are repr(transparent) wrappers over the std types that change one thing, the default hasher:

  • Without the deja feature the default is RandomState, exactly as before.
  • With deja, the default is CorrelationHasher. Inside a request that deja is recording or replaying, it derives its keys from the correlation id, so the same request iterates the same way in both. Everywhere else, including every request in a build where deja is compiled in but idle, it draws fresh random keys per map, as RandomState::new() does. That keeps hash-flooding resistance where nothing is being compared.

The wrappers define new, with_capacity and From<[_; N]> themselves, because std defines those only for RandomState. Everything else is std's, reached through Deref, plus forwarding impls of Default, Clone, Debug, PartialEq/Eq, FromIterator, Extend, IntoIterator (value and both references), Index, From both ways, into_inner, and serde. So the change at a call site is its import.

Commits, in order:

  • the facade itself;
  • its hasher;
  • two fields moved first because they were measured diverging: pre_routing_results and allowed_domains;
  • the wrapper types;
  • a disallowed-types lint, plus disallowed-methods for std's new/with_capacity, since the type lint does not see a fully qualified constructor path;
  • generated code: Smithy's derive is allowed inside its impls, and FlatStruct now returns the facade's map;
  • the named exemptions;
  • the migration: 2,139 uses in 311 files across 32 crates, with about 317 of them generated code;
  • completion under every feature set CI lints.

Each commit message carries its own detail.

What stays on std, each with a named allow and its reason:

  • common_enums and router_env: common_utils depends on them.
  • router_derive and smithy: proc-macros whose maps exist only while compiling.
  • smithy-core, events, currency_conversion, euclid_wasm, card_metadata and config_importer: no dependency on common_utils.
  • Maps whose type belongs to an API we do not own:
    • prost-generated gRPC fields;
    • redis FromRedisValue, which is implemented for std's map only;
    • the events crate's send_message metadata;
    • currency_conversion's types;
    • the Superposition SDK.

Known limits:

  • The keys are fixed when a collection is built. A collection built with no correlation in scope, such as settings deserialized at startup, keeps one random order for the life of the process.
  • Equal keys give equal order only when the insertion sequence also repeats. So a collection rebuilt inside a request from a source that iterates randomly (a startup-built set, or a std map read from redis or prost) is re-keyed but not reproducibly ordered.

Neither limit is a regression: those orders were random before. Where such an order leaves as a list or a string, the fix is to sort at that point.

How did you test it?

Short answer: nothing changes but the hasher. Every forwarded impl matches std byte for byte under a shared hasher, and every change outside the facade module reduces to the collections path swap, a lint allow, or one of 14 hand-read edits that are all behaviour-neutral.

  • The API surface is unchanged. The OpenAPI v1 and v2 specs regenerated after the migration are byte-identical to before (generation checked to have actually written, via a sentinel), and CI's spec job regenerates both on the combined tree and finds no difference.
  • The wrappers behave as std does. A differential test drives the facade and std with the same input under the same hasher value, so both hold identical tables. It asserts that every observable matches exactly: return values, iteration order, {:?}/{:#?} text, serde_json bytes, capacity, equality, Index panics and Deserialize errors.
    • It covers 22 sizes up to 1000 entries, randomized operation sequences, and both the RandomState and fixed-hasher settings.
    • It also probes the trait surface at compile time. The only difference is that std's &HashSet operators (|, &, ^, -) are not forwarded, so code using them does not compile rather than behaving differently.
    • Nine deliberate breakages of the forwarding impls are each caught.
    • The test file is to be added to this branch.
  • Every other hunk was checked mechanically. Every change outside the facade module was classified at the token level, with comments, formatting, lint allows and the collections path itself normalized away and imports compared as sets. That leaves 14 files of residue, each read and found neutral: a type alias, empty constructors written as Default::default(), a test-only into_inner(), and std maps collected into the facade with the same contents.
  • Lints and builds: clippy under all four CI configurations (v1 and v2, redis-rs and fred, -D warnings), a workspace clippy without deja, the router's test targets compiled with and without deja, and the release image's feature line all pass.
  • Router test suite: run with --features v1,deja against the merge base, it has the same failures on both sides by name, so this introduces none.
  • Not yet covered by a test: the decision in CorrelationHasher::default() between derived and random keys. No existing test goes through Default inside a correlation, and nothing yet asserts that record mode registers a decision for a request the sampler skips, which that decision relies on.

One commit here (the last) carries co-author trailers from earlier work; they are left as they are at the author's discretion.

Additional Changes

  • This PR modifies the API contract
  • This PR modifies the database schema
  • This PR modifies application configuration/environment variables

maverox and others added 14 commits September 22, 2026 18:36
WHY. `common_utils::collections` gives `HashMap`/`HashSet` a
correlation-derived hash seed, so a set's iteration order is the same in a
replay as in the recording. It does not compile against deja main at all:
`Default for CorrelationHasher` runs on every `HashMap::new()`, destructors
included, and `current_correlation_id` reads its thread-local with a plain
`with`/`borrow`, which ABORTS the process there rather than failing a request.
#171 adds the fallible twin the facade calls; the pin below this change
provides it.

WHAT IT DOES NOT DO. Nothing calls the facade. It is the mechanism, not the
adoption — the constructors exist and no collection uses them yet. The target is
`payment_methods_enabled`, built at
`crates/router/src/core/payment_methods/cards.rs:5215` by iterating nested
HashMaps into Vecs; ordering there is 26 of 27 blocking body mismatches on the
run we care about. That construction path is verified. The causal link to those
26 is strongly indicated and NOT proven against a failing scorecard, which is
exactly what a build carrying this is for.

CHECKED. `common_utils` at --features deja and `router` at
--features v1,deja,release, both clean. The common_utils green was verified
capable of failing first: the seeded hasher sits behind
`#[cfg(feature = "deja")]`, so a run without it compiles none of the file and
reports success anyway. A `compile_error!` inside `Default for
CorrelationHasher` fired under this feature set.
`BuildHasher` was imported into `digest` alongside `Hasher` and never named
there, which is a warning and so an error under the workspace's deny setting.

It was not visible when the facade was written because the file compiles only
under `--features deja`: a check without it compiles none of the module and
reports success either way. The formatter bot's pass over this file on the
18th did not catch it, because an unused import is not a formatting question.
…played

A HashMap iterates in an order drawn from per-process entropy, so a replay
renders the same map in a different order than its recording did. Where that
order reaches a value deja compares, the difference is charged to the
candidate although the candidate did nothing.

The facade's HashMap and HashSet now default, under the deja feature, to a
hasher whose keys are derived from the current correlation id. The recording
and the replay both know that id, so they agree on the order and nothing has
to be recorded. The default is what carries it, because a serde-deserialized
field constructs its map through Default and no call site exists there to
name a seed. Without the feature the aliases are std's, unchanged.

Derived keys are predictable to whoever can choose the correlation id, and
the feature is compiled into release builds. So the keys are derived only
while deja is active and the request was not sampled out; every other map
keeps random keys. Both checks read process-wide state rather than a
thread-local, since Default also runs inside destructors.

SipHash-1-3 comes from the siphasher crate rather than from DefaultHasher,
whose algorithm std does not promise across toolchains, and the recorder and
a candidate are different builds. One test pins the derived keys to fixed
values: it is the only assertion that notices the derivation changing between
two builds. Another renders equal maps under two random key pairs and under
one correlation, which is the divergence and its absence.

The named-seed helpers are removed. They could not reach a deserialized
field, and nothing used them.
A same-image self-replay of one day of sandbox traffic (442 correlations)
blocked on 17 correlations for one reason: a collection iterated in hash
order on its way into a SQL bind. In 13 it was the pre_routing_results map
written to payment_attempt; in 4 it was the allowed_domains set inside a
profile's payment link config. The statement executed and returned the same
row both times. Only the rendering of its operands differed, and it differed
in order alone.

Both now use the facade's collections, so under the deja feature their order
follows the correlation and a replay renders them as the recording did.
pre_routing_results gets one alias, PreRoutingResults, used at every
signature that named the map. allowed_domains is a field of seven structs
that convert into one another, so all seven move together, along with the
validator that consumes it and the constant and constructors that build an
empty one. Constructors become Default, which holds for either hasher.

Without the deja feature these are the std types they were.
The guard reads an absent recording decision as "no sampler is engaged". deja
does not promise that: it returns the same answer for a request whose decision
was never set. The reading is sound only because record mode sets a decision
for every request, a skipped one included, before the correlation is entered,
and that happens in router_env, a different crate, with nothing tying the two
together. The comment now names that dependency, so a change there is known to
reach here, and records the one window that remains at request teardown.
The facade aliased std's HashMap and HashSet with a different default hasher.
That leaves every call site of new(), with_capacity() and From<[_; N]> unable
to compile, because std defines those only for its own default hasher and an
alias cannot add them: about 590 sites in this tree.

They are now thin wrappers, repr(transparent) over the std types, that define
those constructors themselves and reach everything else through Deref. The
forwarding impls are the ones the std types have: Default, Clone, Debug,
PartialEq, Eq, FromIterator, Extend, IntoIterator for the value and both
references, Index, conversions to and from the std type, and serde. A call
site then changes nothing but its import.

Serialization is tested against std's own output rather than against itself,
since a wrapper that serialized differently would still round-trip. What does
change is a collection handed by value to an API that names the std type,
which takes into_inner().
A std collection iterates in an order drawn from per-process entropy, so a
replay renders it differently than its recording did. The facade's types fix
that under the deja feature, but only where they are used, and nothing stops a
new module importing std's. This makes that an error with the replacement
named.

The lint resolves types rather than text, so it sees an import nested inside
use std::{..} and stays quiet on the facade's own types. It is red until the
migration that follows is complete.
Deref gives the wrapper std's methods through method-call syntax only. A path
such as HashMap::is_empty, which two serde skip_serializing_if attributes use,
resolves against the wrapper itself and found nothing. Both types now define
is_empty and len inherently. A search of the tree finds no other method named
by path.
Two derive macros write std::collections::HashMap into what they expand to,
and because they expand with call-site spans the disallowed-types lint charges
it to every crate that derives them.

SmithyModel accounts for about 317 of the 2,139 sites the lint reported: 290
in api_models and 27 in common_types, none of them code anyone wrote. The model
is assembled at compile time by smithy_core, which does not use the facade, so
the five impls the macro generates now carry their own allow.

FlatStruct returns the map it builds to its caller, and one caller is on a
request path, in payouts. It now returns the facade's map. Both crates that
derive it already depend on common_utils. The order of that map was never
rendered, it is read by key, so this changes no recording; it removes an
exemption that would otherwise have had to be written down.

Both proc-macro crates are themselves exempt from the lint, since their own
maps exist only while compiling.
The facade lives in common_utils, so it can serve only what depends on
common_utils, and a map whose type belongs to someone else has to stay that
type. Each exemption from the disallowed-types lint is written where it
applies, with its reason, so the set is stated rather than discovered.

Whole crates:
  common_enums, router_env        common_utils depends on them
  router_derive, smithy           proc-macros (allowed in the previous commit)
  smithy-core                     compile-time model generation, no dependency
  events, currency_conversion,
  euclid_wasm, card_metadata      no dependency on common_utils

Build scripts: smithy-generator and payment_link. A build script cannot see
common_utils and its maps exist only while compiling.

Files whose maps belong to an external API: the redis_interface tests, where
redis implements FromRedisValue for std's map only, and the Superposition
client, whose maps are the SDK's and open_feature's.

euclid_wasm also builds structs defined downstream whose fields are now the
facade's. It constructs those with Default, which names no type.
With the lint on, clippy reported 2,139 uses of std's HashMap or HashSet in
311 files across 32 crates. About 317 were generated code and 133 sit in
crates that cannot use the facade; both are dealt with in the two commits
before this one. The rest move here, in 22 crates, and the change at each is
its import. The facade's types define new(), with_capacity() and From<[_; N]>
themselves, so no constructor had to change.

The list of files came from clippy rather than from a search, because the lint
resolves types: it finds an import nested inside use std::{..}, which a search
for the path misses, and it stays quiet on the facade's own types.

After the imports moved the workspace had 38 compile errors, every one of them
where a map of ours meets a type we do not own:

  - a gRPC-generated field: the map is collected into std's type going in, and
    into ours coming out
  - the events crate's send_message trait, which fixes a std map parameter;
    the three implementations keep it, allowed on the function
  - currency_conversion's types, so router's currency module keeps std's map
  - a set returned by common_enums, collected into ours before it is compared
  - the settings tests, which hand a map to serde's IntoDeserializer by value

One import form was not covered by the rewrite and was moved by hand:
std::collections::hash_map::HashMap, in api_models.

Under the deja feature a map's order now follows the request in all of these
crates rather than in the two fields moved earlier. Without the feature the
hasher is std's, as before.
The migration before this was driven by a clippy queue generated with
--features v1,deja, and gated the same way. It was complete for that feature
set and not for the workspace: code behind any other feature was never
compiled, so it was never linted and never moved. CI lints every feature, and
its four configurations found what was left. This moves it.

What had been missed, all behind features the first queue did not enable:
  - dynamic_routing: the gRPC health-check client and a routing helper's map
  - hashicorp-vault: the map a secret is read into
  - fred: two local annotations in the fred backend that still named std's map
    after the alias they feed had moved
  - v2: a customers type wrapping a std map, the revenue-recovery and
    account-updater modules, and six imports written on one line as
    use std::{collections::HashMap, ..}
  - euclid's ast_parser: three constructor calls, fixed by the integrator

Kept on std, each allowed where it applies with its reason: the prost-generated
decider module and a conversion that fills one of its fields, since prost
writes std's map for every proto map field; two redis hash reads, since redis
implements FromRedisValue for std's map only and the orphan rule lets neither
crate implement it for ours; and config_importer, a standalone tool with no
dependency on common_utils.

A second lint is added, disallowed-methods for std's HashMap and HashSet new
and with_capacity. disallowed-types sees a type position and not a constructor
path, so std::collections::HashMap::new() written in full was invisible to it,
which is how the euclid sites went unseen. The exemptions now name both lints.
The Smithy derive accounted for about 350 of the new lint's first 373 hits; it
is allowed inside the macro's generated impl, not at its users.

Verified, each run to completion and read from build-finished rather than from
an exit code, with both lints on and zero warnings:
  v1 every feature, redis-rs        v1 every feature, fred
  v2 every feature, redis-rs        v2 every feature, fred   (CI's flags, -D warnings)
  workspace --features v1, no deja, -D warnings
  the image's line: --no-default-features --features release,v1,redis-rs

release enables deja, and clippy_v2 drops deja from its list but keeps release,
so five of those six compile the deja arm. The no-deja run is the only one that
lints the other arm, which is one line. It follows that anything built with
release ships the correlation hasher, and what protects such a build is the
hasher's own guard, not the feature flag.

Four ways a green result here can mean less than it says, all met while doing
this: the lint sees type positions and not constructor paths; a queue generated
by clippy covers only the features it was generated under; a run that stops on
a compile error has linted nothing above the failure, so its count is a floor;
and just clippy does not deny warnings, so its exit code says nothing about
hits.
…ack is random

Squashed from two review follow-ups by the author: the first names which use of a
collection's order is observable, the second records why a fixed fallback would
give up hash-flooding resistance for maps that are never replayed. Both touch the
same comment block, so they read better as one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JsLHFnDDnUZ4soNmF2Zf1h
@maverox
maverox requested review from a team as code owners September 22, 2026 13:30
@semanticdiff-com

semanticdiff-com Bot commented Sep 22, 2026

Copy link
Copy Markdown

Review changes with  SemanticDiff

@maverox maverox left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These findings move here from the review of #14311 (#14311 (review)). Two of them concern code that now lives only in this PR, and the third was raised since. Rechecked on this tip (da41601263). The other two findings from that review stay on #14311: the lock edges are fixed there, and the fork-lint gap is being closed there.

  1. The hash-key decision in CorrelationHasher::default() is untested in both directions. Forcing it to always derive, or to never derive, or dropping the runtime-disabled check, leaves the whole common_utils --features deja suite green. No test in the workspace enters a correlation and builds a map through Default. Nothing asserts that record mode registers Skip for a request the sampler declines (router_env/src/request_id.rs), which the guard's own doc says it depends on.

  2. Equal keys reproduce the order only when the insertion order also repeats (measured: with one fixed pair of keys, 30 elements inserted in 200 shuffled orders iterate in 190 distinct orders). So the remedy suggested in the doc at collections.rs:54-56 gives a collection the request's keys, but not a reproducible order when its source iterates randomly. The PR body now states this as a known limit; the doc sentence should say keys, not order.

  3. Nine runtime exemptions silence more than std's constructors. disallowed_methods carries the whole ban list: clock, entropy, host identity, into_boxed and spawn, as well as the new HashMap/HashSet new/with_capacity. The crate-wide allows in router_env, common_enums, events, card_metadata, currency_conversion and euclid_wasm, and the file-wide ones in router/src/utils/currency.rs, external_services/src/superposition.rs and redis_interface/src/test.rs, therefore also silence the clock and entropy bans in those scopes.

    • Nothing is hidden today. Narrowing them surfaces 15 calls, all std map/set constructors, and every clock and entropy call in those scopes carries its own local allow.
    • A raw clock read added to router_env later would pass lint, though. Suggest per-site allows on the 15 constructor calls, or a stated reason for keeping them scope-wide.

Not findings: the forwarding impls were checked differentially against std under a shared hasher (identical output, and 9 of 9 deliberate breakages caught), and every change outside the facade module reduces to a path swap, a lint allow, or a behaviour-neutral edit.

// included, and a plain `with` on a destroyed thread-local panics
// inside a drop, which aborts the process.
let (k0, k1) = match deja::try_current_correlation_id() {
Some(correlation) if order_is_replayed(&correlation) => {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finding 1: this arm, and the random fallback below it, are reached by no test. Always-derive, never-derive, and dropping !deja::runtime_mode_is_disabled() each leave the suite green. A test binary with a replay hook installed (two maps in one correlation render identically), one with no hook (they differ), and a router_env test asserting recording_decision(id) == Some(Skip) for a declined request would pin both directions.

/// process. That matters only where the order leaves the collection: deja
/// reads a JSON object by key, so what is exposed is a list or a string
/// built by iterating it. Nothing here addresses that. Rebuilding the
/// collection inside the request, by collecting it, gives it the request's

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finding 2: collecting gives the request's keys, but the order still follows insertion order, so a collection collected from a startup-built set is re-keyed and still not reproducibly ordered (190 distinct orders out of 200 shuffled insertions of 30 elements under one key pair). Suggest saying keys rather than order here, and pointing at sorting where the order leaves the collection.

@@ -3,6 +3,8 @@
//! Environment of payment router: logger, basic config, its environment awareness.

#![doc = include_str!(concat!(env!("CARGO_MANIFEST_DIR" ), "/", "README.md"))]
// common_utils depends on this crate, so it cannot use the facade.
#![allow(clippy::disallowed_types, clippy::disallowed_methods)]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finding 3: disallowed_methods is the whole ban list, so this crate-wide allow also silences the clock, entropy and spawn bans across router_env, not only std's map constructors. Nothing is hidden today (the local allows on the formatter and the Uuid::now_v7 seams do the work), but a new raw clock read here would pass. The same applies to eight other scope-wide exemptions; per-site allows on the 15 constructor calls would keep the other bans live.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant