Skip to content

fix(engine): parse_date_ms accepts no-colon numeric zone offsets - #91

Merged
xerj-org merged 1 commit into
xerj-org:mainfrom
Vinz2168:fix/parse-date-ms-numeric-offset
Aug 1, 2026
Merged

fix(engine): parse_date_ms accepts no-colon numeric zone offsets#91
xerj-org merged 1 commit into
xerj-org:mainfrom
Vinz2168:fix/parse-date-ms-numeric-offset

Conversation

@Vinz2168

@Vinz2168 Vinz2168 commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

parse_date_ms is the single, shared date-to-epoch-ms conversion used throughout xerj-engine — aggregations (min/max, date_histogram, terms script buckets), range queries, and sorting all go through it (verified: index.rs has no separate date parser, every call site delegates to this one function).

It only tried chrono::DateTime::parse_from_rfc3339, which requires a colon in the zone offset per the RFC3339 spec (+00:00) — but ES's own strict_date_time/XX-style date formats also accept the no-colon numeric form (+0000), which real clients emit (e.g. Java's default date formatters). No other branch in the function handled it either, so it fell through to the final "parse as raw integer" fallback and returned None.

How this was found

While doing an end-to-end test of the UBI sample dashboards (unrelated to this fix — see #86/#87/#88/#89), one dashboard panel ("Most Common Search Result") showed "No results found". Using OSD's own "Inspect → Requests" panel to see the exact query, then tracing it: a min/max aggregation over timestamp for the subset of documents with "timestamp": "2025-01-24T07:31:52.102+0000"-shaped values returned null — the field was validly mapped (strict_date_time), successfully ingested (not _ignored), sitting right there in _source, but invisible to every date-based query or aggregation.

Important caveat, so the fix isn't over-claimed: fixing the parser does NOT make that specific panel populate — verified live before and after. The documents carrying this timestamp shape are genuinely dated January–March 2025, while the dashboard's saved view has a fixed December-2024-only time range (timeRestore: true). Once the panel's time filter is widened to include that range, the aggregation returns real buckets (confirmed: B013UFPODY: 20, B07XLFYN6S: 20, ...) — so the panel's emptiness within its saved December-only window is correct behavior, matching what real ES would also do. What the fix actually corrects: before it, min/max on the full index silently reported 2024-12-15 as the latest timestamp; after it, it correctly reports 2025-03-03 — i.e. every date-aware read path was silently blind to any document using this (valid, common) offset spelling, regardless of dashboard time filters.

Fix

Added %Y-%m-%dT%H:%M:%S%.f%z (chrono's RFC 2822-style numeric-offset parser) as a fallback tried right after the RFC3339 attempt.

Test plan

  • cargo fmt --check
  • cargo clippy -p xerj-engine -p xerj-query -p xerj-api --all-targets -- -D warnings
  • cargo test -p xerj-engine -p xerj-query -p xerj-api --lib (356 passed; only the 2 pre-existing, unrelated snapshot_path_security_tests failures)
  • 5 new regression tests, confirmed to fail without the fix (temporarily reverted, reran, restored)
  • Verified live: rebuilt with the fix, re-ran min/max over the real live index before/after — max_ts went from 2024-12-15 (silently truncated) to the correct 2025-03-03

🤖 Generated with Claude Code

https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL

`parse_date_ms` is the single, shared date-to-epoch-ms conversion used
throughout xerj-engine — aggregations (min/max, date_histogram, terms
script buckets), range queries, and sorting all go through it. It only
tried `chrono::DateTime::parse_from_rfc3339`, which requires a colon
in the zone offset per the RFC3339 spec (`+00:00`) — but ES's own
`strict_date_time`/`XX`-style date formats also accept the no-colon
numeric form (`+0000`), which real clients emit (e.g. Java's default
date formatters). No other branch in the function handled it either,
so it fell through to the final "parse as raw integer" fallback and
returned `None`.

Found live: a document with `"timestamp": "2025-01-24T07:31:52.102+0000"`
was validly mapped and successfully ingested (not malformed, not
`_ignored`) but silently vanished from every date aggregation and
range query — a dashboard's fixed time-range filter excluded it, and
a min/max aggregation over the field returned null, even though the
raw value was sitting right there in `_source`.

Fixed by trying `%Y-%m-%dT%H:%M:%S%.f%z` (chrono's RFC 2822-style
numeric-offset parser) as a fallback before the no-timezone formats.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL
@xerj-org

xerj-org commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Housekeeping, not a review comment.

We've just added a Contributor License Agreement to the project (#93). Until now contributions here were covered only by Apache-2.0 §5's inbound=outbound clause, which gives no explicit patent grant — worth tightening for a project this size.

Once #93 merges, a cla-bot check will appear on this PR and it will be red until you're covered. Signing is one small pull request, once per contributor — not per PR:

  1. Read CLA.md
  2. Open a PR adding your GitHub username to .contributors
  3. Comment @cla-bot check back here and it turns green

That PR is the signature — it comes from your own account, so the commit history is the record.

To be explicit about something: we did not add anyone to the signed list on their behalf, including you. That file asserts a person has signed, and that's not ours to assert for someone else — hence the ask rather than a quiet edit.

Sorry for the extra step on work that's already in flight. Thanks for the contributions.

@xerj-org
xerj-org merged commit 4376b54 into xerj-org:main Aug 1, 2026
9 checks passed
xerj-org added a commit that referenced this pull request Aug 1, 2026
…nd (#119)

* perf(aggs): guard the no-colon zone-offset parse instead of reordering it (#96)

PR #91 taught parse_date_ms to accept `+HHMM` zone offsets, but put the new
`%z` try immediately after the RFC3339 fast path. parse_date_ms runs per date
field per document on ingest, and offsetless ISO values — the overwhelmingly
common shape, and one no zone-offset pattern can ever match — reach neither
RFC3339 nor `%z`, so every one of them started paying for a doomed offset
parse before reaching the branch that does match. Measured ~1.9x on that path.

Moving the `%z` try to the end of the chain fixes offsetless values but hands
the bill to `+HHMM`, the class #91 exists to serve: it then falls through a
dozen doomed parses first, 4.7x slower than before. Neither order is free.

So keep the original order and gate the parse on a cheap shape check instead.
chrono's `%z` needs an explicit sign (`+`, `-`, or MINUS SIGN U+2212), it
cannot appear before byte 11 (the shortest datetime chrono will accept in
front of it is `5-1-2T3:4:5`), and it needs `HHMM` behind it, so it cannot sit
in the last four bytes. Canonical `YYYY-MM-DDTHH:MM:SS` separators pin the
field widths and raise that floor to byte 18 — 18 and not 19, because
one-digit seconds are legal and `2025-01-02T03:04:5+00:00` really does parse.
Finding a sign in that window is a necessary condition for the parse, so a
value that fails the check cannot have parsed. The check over-matches in every
judgement call: a false positive only costs the parse that would have run
anyway, a false negative would silently drop a date out of every aggregation.

Release-mode ns/call, minimum over four interleaved runs pinned to one core,
this box being shared (identical-code classes below show the noise floor):

                        #91    %z-last   guarded
  offsetless ISO       314.9    161.0     161.3   <- 1.95x, the hot path
  +HH:MM (RFC3339)      30.8     30.8      30.8
  +HHMM (no colon)     206.2    962.4     205.9   <- 4.68x vs %z-last
  Z-suffixed            27.7     27.3      27.2
  epoch integer        360.0    352.0     323.2
  date only            272.0    195.9     196.4
  space separated      670.6    595.2     604.3
  garbage              339.1    332.7     302.3
  epoch JSON number      3.7      3.6       3.6

No class is slower than #91; the two the guard cannot spare (`+HHMM`, which
must still parse, and the space-separated form, whose head is not canonical)
pay a bounded 3-9ns for the check itself.

Behaviour is unchanged. A differential harness carrying verbatim copies of all
three versions ran 213,559 inputs — every offset spelling, chrono's one-digit
field widths, colon/space runs inside the offset, U+2212, signed and wide
years, single-edit mutations of six skeletons at every position, and 200k
pseudo-random strings over a date alphabet — and the three output dumps are
byte-identical (sha256 b5bb3e0b18e4...). 4,800 of those inputs are `%z`-
parseable, with signs as early as byte 11 and 1,827 of them before byte 18, so
both lower bounds are exercised rather than assumed; forcing the canonical
floor to 19 makes the harness fail on `2025-01-02T03:04:5+00:00`.

In-tree, `parse_date_ms_golden_matrix` pins 74 shapes to the values the
pre-guard code produced, `zone_offset_guard_never_false_negatives` re-asserts
the one-sidedness over ~11k generated inputs, and `nocolon_offset_parse_is_guarded`
counts the parses that actually ran — placement is invisible to results, so
the saving has to be asserted as work skipped rather than as a value returned.

* perf(aggs): bound the zone-offset probe to a six-byte window at the end

The guard added for issue #96 was correct but paid for itself with a worse
regression: it hunted for the offset's sign by scanning `[11, len - 5)`, which
is O(len). Any value whose window holds no `+`, `,`, `-` or non-ASCII byte —
prose, digits, anything long — ran that scan to the end, where before the
guard `parse_date_ms` failed in O(1)-ish time. Measured, per-pass paired
timings pinned to one core, ns/op before the guard -> with it: 512 B 214 ->
326, 4 KB prose 214 -> 1139, 4 KB digits 254 -> 1214, 64 KB 804 -> 15587.
That is a 19x regression on the same per-document path the guard exists to
make cheaper.

A `%z` offset cannot be anywhere but the end: it closes the pattern and
`parse_from_str` insists the whole value be consumed, so an accepted value
ends with the offset's two minute digits and the sign sits a fixed distance
in front of them — `len - 5` for `±HHMM`, `len - 6` for `±HH:MM` and
`±HH MM`. So the probe now reads a six-byte tail (sign + `HH` + one separator
+ `MM`, named and derived from those spellings) and nothing else. No scan, no
loop, no dependence on length.

chrono's separator scanner does accept a longer run — `+00 : 00` really does
parse — which walks the sign out of any constant window. Chasing that run
would put the O(len) back, so it is over-matched from the separators it
leaves behind instead: the probe stays one-sided, and an over-match only
pays for the parse that would have run anyway.

Now best-or-tied on every class, against both origin/main and the scanning
guard (ns/op, 12 pinned runs x 41 interleaved passes, median of per-pass
ratios; 1.00x here is the noise floor, established by the two classes that
return before the probe is even reached):

  class                  main       prev        now   vs main    vs prev
  offsetless ISO        211.8      107.2      108.2    0.513x     1.001x
  +HH:MM                 19.8       19.8       19.8    1.000x     1.000x
  +HHMM                 136.8      136.1      135.9    0.997x     0.999x
  Z-suffixed             18.0       17.9       17.9    1.000x     1.001x
  epoch integer         241.8      219.6      219.8    0.910x     0.998x
  128 B sign-free       215.3      218.2      197.0    0.905x     0.897x
  512 B sign-free       213.5      325.9      195.1    0.906x     0.593x
  4 KB prose            213.9     1139.1      197.4    0.920x     0.169x
  4 KB digits           254.1     1213.7      232.1    0.912x     0.192x
  64 KB sign-free       804.3    15587.1      774.2    0.974x     0.048x

Output is unchanged: 16315 values — every zone spelling including `+0000`,
`+00:00`, `+00 00`, `+00:::00`, `Z`, `z` and U+2212, offsetless ISO, epoch
integers, fractional seconds of every width, garbage and multi-kilobyte
blobs — parse byte-identically under origin/main, the scanning guard and
this probe.

`zone_offset_probe_window_is_bounded` pins the property the scan lost: the
probe's answer is fixed by the last six bytes plus the length, so filler in
front of them cannot change it or add work — including filler made entirely
of the bytes the old scan was hunting for, at 64 B through 64 KB.

* test(aggs): make the bounded-window test actually catch an unbounded scan

`zone_offset_probe_window_is_bounded` is true but does not discriminate: it
grows a HOMOGENEOUS filler and asserts the answer does not change, and an
O(len) scan's answer is invariant under exactly that transformation — a `"+"`
filler contains a `+` at every length and an `"a"` filler contains none at any
length. Verified by mutation: replacing the probe's body with the previous
unbounded scan leaves that test green, so nothing in the suite would have
caught the regression coming back.

The property that characterises the fix is that the answer depends on
`(len, last ZONE_OFFSET_PROBE_BYTES bytes)` and nothing else. The new test
holds BOTH the length and the window fixed and perturbs every byte strictly
below the window, over 8 tail shapes x 5 lengths x 9 single-byte edits.

Mutation-checked both ways on this tree:
  bounded probe (as shipped)  -> passes, 3 168 checks
  unbounded scan restored     -> FAILS at "editing byte 14 of 22 ... to 0x3a",
                                 while the old scaling test still passes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants