Conversation
WHY. `common_utils::collections` gives `HashMap`/`HashSet` a correlation-derived hash seed, so a set's iteration order is the same in a replay as in the recording. It does not compile against deja main at all: `Default for CorrelationHasher` runs on every `HashMap::new()`, destructors included, and `current_correlation_id` reads its thread-local with a plain `with`/`borrow`, which ABORTS the process there rather than failing a request. #171 adds the fallible twin the facade calls; the pin below this change provides it. WHAT IT DOES NOT DO. Nothing calls the facade. It is the mechanism, not the adoption — the constructors exist and no collection uses them yet. The target is `payment_methods_enabled`, built at `crates/router/src/core/payment_methods/cards.rs:5215` by iterating nested HashMaps into Vecs; ordering there is 26 of 27 blocking body mismatches on the run we care about. That construction path is verified. The causal link to those 26 is strongly indicated and NOT proven against a failing scorecard, which is exactly what a build carrying this is for. CHECKED. `common_utils` at --features deja and `router` at --features v1,deja,release, both clean. The common_utils green was verified capable of failing first: the seeded hasher sits behind `#[cfg(feature = "deja")]`, so a run without it compiles none of the file and reports success anyway. A `compile_error!` inside `Default for CorrelationHasher` fired under this feature set.
`BuildHasher` was imported into `digest` alongside `Hasher` and never named there, which is a warning and so an error under the workspace's deny setting. It was not visible when the facade was written because the file compiles only under `--features deja`: a check without it compiles none of the module and reports success either way. The formatter bot's pass over this file on the 18th did not catch it, because an unused import is not a formatting question.
…played A HashMap iterates in an order drawn from per-process entropy, so a replay renders the same map in a different order than its recording did. Where that order reaches a value deja compares, the difference is charged to the candidate although the candidate did nothing. The facade's HashMap and HashSet now default, under the deja feature, to a hasher whose keys are derived from the current correlation id. The recording and the replay both know that id, so they agree on the order and nothing has to be recorded. The default is what carries it, because a serde-deserialized field constructs its map through Default and no call site exists there to name a seed. Without the feature the aliases are std's, unchanged. Derived keys are predictable to whoever can choose the correlation id, and the feature is compiled into release builds. So the keys are derived only while deja is active and the request was not sampled out; every other map keeps random keys. Both checks read process-wide state rather than a thread-local, since Default also runs inside destructors. SipHash-1-3 comes from the siphasher crate rather than from DefaultHasher, whose algorithm std does not promise across toolchains, and the recorder and a candidate are different builds. One test pins the derived keys to fixed values: it is the only assertion that notices the derivation changing between two builds. Another renders equal maps under two random key pairs and under one correlation, which is the divergence and its absence. The named-seed helpers are removed. They could not reach a deserialized field, and nothing used them.
A same-image self-replay of one day of sandbox traffic (442 correlations) blocked on 17 correlations for one reason: a collection iterated in hash order on its way into a SQL bind. In 13 it was the pre_routing_results map written to payment_attempt; in 4 it was the allowed_domains set inside a profile's payment link config. The statement executed and returned the same row both times. Only the rendering of its operands differed, and it differed in order alone. Both now use the facade's collections, so under the deja feature their order follows the correlation and a replay renders them as the recording did. pre_routing_results gets one alias, PreRoutingResults, used at every signature that named the map. allowed_domains is a field of seven structs that convert into one another, so all seven move together, along with the validator that consumes it and the constant and constructors that build an empty one. Constructors become Default, which holds for either hasher. Without the deja feature these are the std types they were.
The guard reads an absent recording decision as "no sampler is engaged". deja does not promise that: it returns the same answer for a request whose decision was never set. The reading is sound only because record mode sets a decision for every request, a skipped one included, before the correlation is entered, and that happens in router_env, a different crate, with nothing tying the two together. The comment now names that dependency, so a change there is known to reach here, and records the one window that remains at request teardown.
The facade aliased std's HashMap and HashSet with a different default hasher. That leaves every call site of new(), with_capacity() and From<[_; N]> unable to compile, because std defines those only for its own default hasher and an alias cannot add them: about 590 sites in this tree. They are now thin wrappers, repr(transparent) over the std types, that define those constructors themselves and reach everything else through Deref. The forwarding impls are the ones the std types have: Default, Clone, Debug, PartialEq, Eq, FromIterator, Extend, IntoIterator for the value and both references, Index, conversions to and from the std type, and serde. A call site then changes nothing but its import. Serialization is tested against std's own output rather than against itself, since a wrapper that serialized differently would still round-trip. What does change is a collection handed by value to an API that names the std type, which takes into_inner().
A std collection iterates in an order drawn from per-process entropy, so a
replay renders it differently than its recording did. The facade's types fix
that under the deja feature, but only where they are used, and nothing stops a
new module importing std's. This makes that an error with the replacement
named.
The lint resolves types rather than text, so it sees an import nested inside
use std::{..} and stays quiet on the facade's own types. It is red until the
migration that follows is complete.
Deref gives the wrapper std's methods through method-call syntax only. A path such as HashMap::is_empty, which two serde skip_serializing_if attributes use, resolves against the wrapper itself and found nothing. Both types now define is_empty and len inherently. A search of the tree finds no other method named by path.
Two derive macros write std::collections::HashMap into what they expand to, and because they expand with call-site spans the disallowed-types lint charges it to every crate that derives them. SmithyModel accounts for about 317 of the 2,139 sites the lint reported: 290 in api_models and 27 in common_types, none of them code anyone wrote. The model is assembled at compile time by smithy_core, which does not use the facade, so the five impls the macro generates now carry their own allow. FlatStruct returns the map it builds to its caller, and one caller is on a request path, in payouts. It now returns the facade's map. Both crates that derive it already depend on common_utils. The order of that map was never rendered, it is read by key, so this changes no recording; it removes an exemption that would otherwise have had to be written down. Both proc-macro crates are themselves exempt from the lint, since their own maps exist only while compiling.
The facade lives in common_utils, so it can serve only what depends on common_utils, and a map whose type belongs to someone else has to stay that type. Each exemption from the disallowed-types lint is written where it applies, with its reason, so the set is stated rather than discovered. Whole crates: common_enums, router_env common_utils depends on them router_derive, smithy proc-macros (allowed in the previous commit) smithy-core compile-time model generation, no dependency events, currency_conversion, euclid_wasm, card_metadata no dependency on common_utils Build scripts: smithy-generator and payment_link. A build script cannot see common_utils and its maps exist only while compiling. Files whose maps belong to an external API: the redis_interface tests, where redis implements FromRedisValue for std's map only, and the Superposition client, whose maps are the SDK's and open_feature's. euclid_wasm also builds structs defined downstream whose fields are now the facade's. It constructs those with Default, which names no type.
With the lint on, clippy reported 2,139 uses of std's HashMap or HashSet in
311 files across 32 crates. About 317 were generated code and 133 sit in
crates that cannot use the facade; both are dealt with in the two commits
before this one. The rest move here, in 22 crates, and the change at each is
its import. The facade's types define new(), with_capacity() and From<[_; N]>
themselves, so no constructor had to change.
The list of files came from clippy rather than from a search, because the lint
resolves types: it finds an import nested inside use std::{..}, which a search
for the path misses, and it stays quiet on the facade's own types.
After the imports moved the workspace had 38 compile errors, every one of them
where a map of ours meets a type we do not own:
- a gRPC-generated field: the map is collected into std's type going in, and
into ours coming out
- the events crate's send_message trait, which fixes a std map parameter;
the three implementations keep it, allowed on the function
- currency_conversion's types, so router's currency module keeps std's map
- a set returned by common_enums, collected into ours before it is compared
- the settings tests, which hand a map to serde's IntoDeserializer by value
One import form was not covered by the rewrite and was moved by hand:
std::collections::hash_map::HashMap, in api_models.
Under the deja feature a map's order now follows the request in all of these
crates rather than in the two fields moved earlier. Without the feature the
hasher is std's, as before.
The migration before this was driven by a clippy queue generated with
--features v1,deja, and gated the same way. It was complete for that feature
set and not for the workspace: code behind any other feature was never
compiled, so it was never linted and never moved. CI lints every feature, and
its four configurations found what was left. This moves it.
What had been missed, all behind features the first queue did not enable:
- dynamic_routing: the gRPC health-check client and a routing helper's map
- hashicorp-vault: the map a secret is read into
- fred: two local annotations in the fred backend that still named std's map
after the alias they feed had moved
- v2: a customers type wrapping a std map, the revenue-recovery and
account-updater modules, and six imports written on one line as
use std::{collections::HashMap, ..}
- euclid's ast_parser: three constructor calls, fixed by the integrator
Kept on std, each allowed where it applies with its reason: the prost-generated
decider module and a conversion that fills one of its fields, since prost
writes std's map for every proto map field; two redis hash reads, since redis
implements FromRedisValue for std's map only and the orphan rule lets neither
crate implement it for ours; and config_importer, a standalone tool with no
dependency on common_utils.
A second lint is added, disallowed-methods for std's HashMap and HashSet new
and with_capacity. disallowed-types sees a type position and not a constructor
path, so std::collections::HashMap::new() written in full was invisible to it,
which is how the euclid sites went unseen. The exemptions now name both lints.
The Smithy derive accounted for about 350 of the new lint's first 373 hits; it
is allowed inside the macro's generated impl, not at its users.
Verified, each run to completion and read from build-finished rather than from
an exit code, with both lints on and zero warnings:
v1 every feature, redis-rs v1 every feature, fred
v2 every feature, redis-rs v2 every feature, fred (CI's flags, -D warnings)
workspace --features v1, no deja, -D warnings
the image's line: --no-default-features --features release,v1,redis-rs
release enables deja, and clippy_v2 drops deja from its list but keeps release,
so five of those six compile the deja arm. The no-deja run is the only one that
lints the other arm, which is one line. It follows that anything built with
release ships the correlation hasher, and what protects such a build is the
hasher's own guard, not the feature flag.
Four ways a green result here can mean less than it says, all met while doing
this: the lint sees type positions and not constructor paths; a queue generated
by clippy covers only the features it was generated under; a run that stops on
a compile error has linted nothing above the failure, so its count is a floor;
and just clippy does not deny warnings, so its exit code says nothing about
hits.
…ack is random Squashed from two review follow-ups by the author: the first names which use of a collection's order is observable, the second records why a fixed fallback would give up hash-flooding resistance for maps that are never replayed. Both touch the same comment block, so they read better as one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JsLHFnDDnUZ4soNmF2Zf1h
maverox
left a comment
There was a problem hiding this comment.
These findings move here from the review of #14311 (#14311 (review)). Two of them concern code that now lives only in this PR, and the third was raised since. Rechecked on this tip (da41601263). The other two findings from that review stay on #14311: the lock edges are fixed there, and the fork-lint gap is being closed there.
-
The hash-key decision in
CorrelationHasher::default()is untested in both directions. Forcing it to always derive, or to never derive, or dropping the runtime-disabled check, leaves the wholecommon_utils --features dejasuite green. No test in the workspace enters a correlation and builds a map throughDefault. Nothing asserts that record mode registersSkipfor a request the sampler declines (router_env/src/request_id.rs), which the guard's own doc says it depends on. -
Equal keys reproduce the order only when the insertion order also repeats (measured: with one fixed pair of keys, 30 elements inserted in 200 shuffled orders iterate in 190 distinct orders). So the remedy suggested in the doc at
collections.rs:54-56gives a collection the request's keys, but not a reproducible order when its source iterates randomly. The PR body now states this as a known limit; the doc sentence should say keys, not order. -
Nine runtime exemptions silence more than std's constructors.
disallowed_methodscarries the whole ban list: clock, entropy, host identity,into_boxedand spawn, as well as the newHashMap/HashSetnew/with_capacity. The crate-wide allows inrouter_env,common_enums,events,card_metadata,currency_conversionandeuclid_wasm, and the file-wide ones inrouter/src/utils/currency.rs,external_services/src/superposition.rsandredis_interface/src/test.rs, therefore also silence the clock and entropy bans in those scopes.- Nothing is hidden today. Narrowing them surfaces 15 calls, all std map/set constructors, and every clock and entropy call in those scopes carries its own local allow.
- A raw clock read added to
router_envlater would pass lint, though. Suggest per-site allows on the 15 constructor calls, or a stated reason for keeping them scope-wide.
Not findings: the forwarding impls were checked differentially against std under a shared hasher (identical output, and 9 of 9 deliberate breakages caught), and every change outside the facade module reduces to a path swap, a lint allow, or a behaviour-neutral edit.
| // included, and a plain `with` on a destroyed thread-local panics | ||
| // inside a drop, which aborts the process. | ||
| let (k0, k1) = match deja::try_current_correlation_id() { | ||
| Some(correlation) if order_is_replayed(&correlation) => { |
There was a problem hiding this comment.
Finding 1: this arm, and the random fallback below it, are reached by no test. Always-derive, never-derive, and dropping !deja::runtime_mode_is_disabled() each leave the suite green. A test binary with a replay hook installed (two maps in one correlation render identically), one with no hook (they differ), and a router_env test asserting recording_decision(id) == Some(Skip) for a declined request would pin both directions.
| /// process. That matters only where the order leaves the collection: deja | ||
| /// reads a JSON object by key, so what is exposed is a list or a string | ||
| /// built by iterating it. Nothing here addresses that. Rebuilding the | ||
| /// collection inside the request, by collecting it, gives it the request's |
There was a problem hiding this comment.
Finding 2: collecting gives the request's keys, but the order still follows insertion order, so a collection collected from a startup-built set is re-keyed and still not reproducibly ordered (190 distinct orders out of 200 shuffled insertions of 30 elements under one key pair). Suggest saying keys rather than order here, and pointing at sorting where the order leaves the collection.
| @@ -3,6 +3,8 @@ | |||
| //! Environment of payment router: logger, basic config, its environment awareness. | |||
|
|
|||
| #![doc = include_str!(concat!(env!("CARGO_MANIFEST_DIR" ), "/", "README.md"))] | |||
| // common_utils depends on this crate, so it cannot use the facade. | |||
| #![allow(clippy::disallowed_types, clippy::disallowed_methods)] | |||
There was a problem hiding this comment.
Finding 3: disallowed_methods is the whole ban list, so this crate-wide allow also silences the clock, entropy and spawn bans across router_env, not only std's map constructors. Nothing is hidden today (the local allows on the formatter and the Uuid::now_v7 seams do the work), but a new raw clock read here would pass. The same applies to eight other scope-wide exemptions; per-site allows on the 15 constructor calls would keep the other bans live.
Note
Stacked on #14311 and merges after it. The facade calls
deja::try_current_correlation_id, which exists only at the deja revision #14311 pins. This PR's diff is the facade alone.Type of Change
Description
std::collections::HashMapandHashSetdefault toRandomState, whose keys come from per-process entropy, so two runs of one binary iterate the same map in different orders. That is invisible in ordinary use. It matters to record and replay: when a map's order reaches a response body, a query bind or an outgoing request, the replay differs from the recording and the difference is charged to the candidate rather than to the hasher.This moves the workspace onto
common_utils::collections::{HashMap, HashSet}. They arerepr(transparent)wrappers over the std types that change one thing, the default hasher:dejafeature the default isRandomState, exactly as before.deja, the default isCorrelationHasher. Inside a request that deja is recording or replaying, it derives its keys from the correlation id, so the same request iterates the same way in both. Everywhere else, including every request in a build where deja is compiled in but idle, it draws fresh random keys per map, asRandomState::new()does. That keeps hash-flooding resistance where nothing is being compared.The wrappers define
new,with_capacityandFrom<[_; N]>themselves, because std defines those only forRandomState. Everything else is std's, reached throughDeref, plus forwarding impls ofDefault,Clone,Debug,PartialEq/Eq,FromIterator,Extend,IntoIterator(value and both references),Index,Fromboth ways,into_inner, and serde. So the change at a call site is its import.Commits, in order:
pre_routing_resultsandallowed_domains;disallowed-typeslint, plusdisallowed-methodsfor std'snew/with_capacity, since the type lint does not see a fully qualified constructor path;FlatStructnow returns the facade's map;Each commit message carries its own detail.
What stays on std, each with a named allow and its reason:
common_enumsandrouter_env:common_utilsdepends on them.router_deriveandsmithy: proc-macros whose maps exist only while compiling.smithy-core,events,currency_conversion,euclid_wasm,card_metadataandconfig_importer: no dependency oncommon_utils.FromRedisValue, which is implemented for std's map only;eventscrate'ssend_messagemetadata;currency_conversion's types;Known limits:
Neither limit is a regression: those orders were random before. Where such an order leaves as a list or a string, the fix is to sort at that point.
How did you test it?
Short answer: nothing changes but the hasher. Every forwarded impl matches std byte for byte under a shared hasher, and every change outside the facade module reduces to the collections path swap, a lint allow, or one of 14 hand-read edits that are all behaviour-neutral.
{:?}/{:#?}text, serde_json bytes, capacity, equality,Indexpanics andDeserializeerrors.RandomStateand fixed-hasher settings.&HashSetoperators (|,&,^,-) are not forwarded, so code using them does not compile rather than behaving differently.Default::default(), a test-onlyinto_inner(), and std maps collected into the facade with the same contents.-D warnings), a workspace clippy withoutdeja, the router's test targets compiled with and withoutdeja, and the release image's feature line all pass.--features v1,dejaagainst the merge base, it has the same failures on both sides by name, so this introduces none.CorrelationHasher::default()between derived and random keys. No existing test goes throughDefaultinside a correlation, and nothing yet asserts that record mode registers a decision for a request the sampler skips, which that decision relies on.One commit here (the last) carries co-author trailers from earlier work; they are left as they are at the author's discretion.
Additional Changes