fix(engine): parse_date_ms accepts no-colon numeric zone offsets - #91
Conversation
`parse_date_ms` is the single, shared date-to-epoch-ms conversion used throughout xerj-engine — aggregations (min/max, date_histogram, terms script buckets), range queries, and sorting all go through it. It only tried `chrono::DateTime::parse_from_rfc3339`, which requires a colon in the zone offset per the RFC3339 spec (`+00:00`) — but ES's own `strict_date_time`/`XX`-style date formats also accept the no-colon numeric form (`+0000`), which real clients emit (e.g. Java's default date formatters). No other branch in the function handled it either, so it fell through to the final "parse as raw integer" fallback and returned `None`. Found live: a document with `"timestamp": "2025-01-24T07:31:52.102+0000"` was validly mapped and successfully ingested (not malformed, not `_ignored`) but silently vanished from every date aggregation and range query — a dashboard's fixed time-range filter excluded it, and a min/max aggregation over the field returned null, even though the raw value was sitting right there in `_source`. Fixed by trying `%Y-%m-%dT%H:%M:%S%.f%z` (chrono's RFC 2822-style numeric-offset parser) as a fallback before the no-timezone formats. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL
|
Housekeeping, not a review comment. We've just added a Contributor License Agreement to the project (#93). Until now contributions here were covered only by Apache-2.0 §5's inbound=outbound clause, which gives no explicit patent grant — worth tightening for a project this size. Once #93 merges, a
That PR is the signature — it comes from your own account, so the commit history is the record. To be explicit about something: we did not add anyone to the signed list on their behalf, including you. That file asserts a person has signed, and that's not ours to assert for someone else — hence the ask rather than a quiet edit. Sorry for the extra step on work that's already in flight. Thanks for the contributions. |
…nd (#119) * perf(aggs): guard the no-colon zone-offset parse instead of reordering it (#96) PR #91 taught parse_date_ms to accept `+HHMM` zone offsets, but put the new `%z` try immediately after the RFC3339 fast path. parse_date_ms runs per date field per document on ingest, and offsetless ISO values — the overwhelmingly common shape, and one no zone-offset pattern can ever match — reach neither RFC3339 nor `%z`, so every one of them started paying for a doomed offset parse before reaching the branch that does match. Measured ~1.9x on that path. Moving the `%z` try to the end of the chain fixes offsetless values but hands the bill to `+HHMM`, the class #91 exists to serve: it then falls through a dozen doomed parses first, 4.7x slower than before. Neither order is free. So keep the original order and gate the parse on a cheap shape check instead. chrono's `%z` needs an explicit sign (`+`, `-`, or MINUS SIGN U+2212), it cannot appear before byte 11 (the shortest datetime chrono will accept in front of it is `5-1-2T3:4:5`), and it needs `HHMM` behind it, so it cannot sit in the last four bytes. Canonical `YYYY-MM-DDTHH:MM:SS` separators pin the field widths and raise that floor to byte 18 — 18 and not 19, because one-digit seconds are legal and `2025-01-02T03:04:5+00:00` really does parse. Finding a sign in that window is a necessary condition for the parse, so a value that fails the check cannot have parsed. The check over-matches in every judgement call: a false positive only costs the parse that would have run anyway, a false negative would silently drop a date out of every aggregation. Release-mode ns/call, minimum over four interleaved runs pinned to one core, this box being shared (identical-code classes below show the noise floor): #91 %z-last guarded offsetless ISO 314.9 161.0 161.3 <- 1.95x, the hot path +HH:MM (RFC3339) 30.8 30.8 30.8 +HHMM (no colon) 206.2 962.4 205.9 <- 4.68x vs %z-last Z-suffixed 27.7 27.3 27.2 epoch integer 360.0 352.0 323.2 date only 272.0 195.9 196.4 space separated 670.6 595.2 604.3 garbage 339.1 332.7 302.3 epoch JSON number 3.7 3.6 3.6 No class is slower than #91; the two the guard cannot spare (`+HHMM`, which must still parse, and the space-separated form, whose head is not canonical) pay a bounded 3-9ns for the check itself. Behaviour is unchanged. A differential harness carrying verbatim copies of all three versions ran 213,559 inputs — every offset spelling, chrono's one-digit field widths, colon/space runs inside the offset, U+2212, signed and wide years, single-edit mutations of six skeletons at every position, and 200k pseudo-random strings over a date alphabet — and the three output dumps are byte-identical (sha256 b5bb3e0b18e4...). 4,800 of those inputs are `%z`- parseable, with signs as early as byte 11 and 1,827 of them before byte 18, so both lower bounds are exercised rather than assumed; forcing the canonical floor to 19 makes the harness fail on `2025-01-02T03:04:5+00:00`. In-tree, `parse_date_ms_golden_matrix` pins 74 shapes to the values the pre-guard code produced, `zone_offset_guard_never_false_negatives` re-asserts the one-sidedness over ~11k generated inputs, and `nocolon_offset_parse_is_guarded` counts the parses that actually ran — placement is invisible to results, so the saving has to be asserted as work skipped rather than as a value returned. * perf(aggs): bound the zone-offset probe to a six-byte window at the end The guard added for issue #96 was correct but paid for itself with a worse regression: it hunted for the offset's sign by scanning `[11, len - 5)`, which is O(len). Any value whose window holds no `+`, `,`, `-` or non-ASCII byte — prose, digits, anything long — ran that scan to the end, where before the guard `parse_date_ms` failed in O(1)-ish time. Measured, per-pass paired timings pinned to one core, ns/op before the guard -> with it: 512 B 214 -> 326, 4 KB prose 214 -> 1139, 4 KB digits 254 -> 1214, 64 KB 804 -> 15587. That is a 19x regression on the same per-document path the guard exists to make cheaper. A `%z` offset cannot be anywhere but the end: it closes the pattern and `parse_from_str` insists the whole value be consumed, so an accepted value ends with the offset's two minute digits and the sign sits a fixed distance in front of them — `len - 5` for `±HHMM`, `len - 6` for `±HH:MM` and `±HH MM`. So the probe now reads a six-byte tail (sign + `HH` + one separator + `MM`, named and derived from those spellings) and nothing else. No scan, no loop, no dependence on length. chrono's separator scanner does accept a longer run — `+00 : 00` really does parse — which walks the sign out of any constant window. Chasing that run would put the O(len) back, so it is over-matched from the separators it leaves behind instead: the probe stays one-sided, and an over-match only pays for the parse that would have run anyway. Now best-or-tied on every class, against both origin/main and the scanning guard (ns/op, 12 pinned runs x 41 interleaved passes, median of per-pass ratios; 1.00x here is the noise floor, established by the two classes that return before the probe is even reached): class main prev now vs main vs prev offsetless ISO 211.8 107.2 108.2 0.513x 1.001x +HH:MM 19.8 19.8 19.8 1.000x 1.000x +HHMM 136.8 136.1 135.9 0.997x 0.999x Z-suffixed 18.0 17.9 17.9 1.000x 1.001x epoch integer 241.8 219.6 219.8 0.910x 0.998x 128 B sign-free 215.3 218.2 197.0 0.905x 0.897x 512 B sign-free 213.5 325.9 195.1 0.906x 0.593x 4 KB prose 213.9 1139.1 197.4 0.920x 0.169x 4 KB digits 254.1 1213.7 232.1 0.912x 0.192x 64 KB sign-free 804.3 15587.1 774.2 0.974x 0.048x Output is unchanged: 16315 values — every zone spelling including `+0000`, `+00:00`, `+00 00`, `+00:::00`, `Z`, `z` and U+2212, offsetless ISO, epoch integers, fractional seconds of every width, garbage and multi-kilobyte blobs — parse byte-identically under origin/main, the scanning guard and this probe. `zone_offset_probe_window_is_bounded` pins the property the scan lost: the probe's answer is fixed by the last six bytes plus the length, so filler in front of them cannot change it or add work — including filler made entirely of the bytes the old scan was hunting for, at 64 B through 64 KB. * test(aggs): make the bounded-window test actually catch an unbounded scan `zone_offset_probe_window_is_bounded` is true but does not discriminate: it grows a HOMOGENEOUS filler and asserts the answer does not change, and an O(len) scan's answer is invariant under exactly that transformation — a `"+"` filler contains a `+` at every length and an `"a"` filler contains none at any length. Verified by mutation: replacing the probe's body with the previous unbounded scan leaves that test green, so nothing in the suite would have caught the regression coming back. The property that characterises the fix is that the answer depends on `(len, last ZONE_OFFSET_PROBE_BYTES bytes)` and nothing else. The new test holds BOTH the length and the window fixed and perturbs every byte strictly below the window, over 8 tail shapes x 5 lengths x 9 single-byte edits. Mutation-checked both ways on this tree: bounded probe (as shipped) -> passes, 3 168 checks unbounded scan restored -> FAILS at "editing byte 14 of 22 ... to 0x3a", while the old scaling test still passes
Summary
parse_date_msis the single, shared date-to-epoch-ms conversion used throughoutxerj-engine— aggregations (min/max, date_histogram, terms script buckets), range queries, and sorting all go through it (verified:index.rshas no separate date parser, every call site delegates to this one function).It only tried
chrono::DateTime::parse_from_rfc3339, which requires a colon in the zone offset per the RFC3339 spec (+00:00) — but ES's ownstrict_date_time/XX-style date formats also accept the no-colon numeric form (+0000), which real clients emit (e.g. Java's default date formatters). No other branch in the function handled it either, so it fell through to the final "parse as raw integer" fallback and returnedNone.How this was found
While doing an end-to-end test of the UBI sample dashboards (unrelated to this fix — see #86/#87/#88/#89), one dashboard panel ("Most Common Search Result") showed "No results found". Using OSD's own "Inspect → Requests" panel to see the exact query, then tracing it: a
min/maxaggregation overtimestampfor the subset of documents with"timestamp": "2025-01-24T07:31:52.102+0000"-shaped values returnednull— the field was validly mapped (strict_date_time), successfully ingested (not_ignored), sitting right there in_source, but invisible to every date-based query or aggregation.Important caveat, so the fix isn't over-claimed: fixing the parser does NOT make that specific panel populate — verified live before and after. The documents carrying this timestamp shape are genuinely dated January–March 2025, while the dashboard's saved view has a fixed December-2024-only time range (
timeRestore: true). Once the panel's time filter is widened to include that range, the aggregation returns real buckets (confirmed:B013UFPODY: 20, B07XLFYN6S: 20, ...) — so the panel's emptiness within its saved December-only window is correct behavior, matching what real ES would also do. What the fix actually corrects: before it,min/maxon the full index silently reported2024-12-15as the latest timestamp; after it, it correctly reports2025-03-03— i.e. every date-aware read path was silently blind to any document using this (valid, common) offset spelling, regardless of dashboard time filters.Fix
Added
%Y-%m-%dT%H:%M:%S%.f%z(chrono's RFC 2822-style numeric-offset parser) as a fallback tried right after the RFC3339 attempt.Test plan
cargo fmt --checkcargo clippy -p xerj-engine -p xerj-query -p xerj-api --all-targets -- -D warningscargo test -p xerj-engine -p xerj-query -p xerj-api --lib(356 passed; only the 2 pre-existing, unrelatedsnapshot_path_security_testsfailures)min/maxover the real live index before/after —max_tswent from2024-12-15(silently truncated) to the correct2025-03-03🤖 Generated with Claude Code
https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL