fix(api): resolve aliases on single-index endpoints; persist aliases across restart#30
Open
Vinz2168 wants to merge 1 commit into
Open
fix(api): resolve aliases on single-index endpoints; persist aliases across restart#30Vinz2168 wants to merge 1 commit into
Vinz2168 wants to merge 1 commit into
Conversation
…across restart
Root-caused a real OpenSearch Dashboards bug where a FRESH OSD
container, pointed at an xerj instance that already has a fully
migrated `.kibana` index, gets stuck forever on:
"Another OpenSearch Dashboards instance appears to be migrating
the index. Waiting for that migration to complete."
...even though there is only ever one real instance. Traced via a
raw byte-level TCP relay between a real OSD 3.6.0 container and
xerj (same technique as the earlier _cat/templates fix) to two
distinct, compounding bugs:
1. GET /{alias} and GET /{alias}/_mapping never resolved an alias to
its underlying real index. `.kibana` is ALWAYS an alias, never a
bare index (OSD/Kibana create `.kibana_1` then alias `.kibana` to
it) -- so `GET /.kibana` 404'd outright, and `GET /.kibana/_mapping`
returned a schema-derived reconstruction missing
`mappings._meta.migrationMappingPropertyHashes`, the exact field
OSD's savedobjects migrator reads to decide "already migrated,
adopt it" vs "someone else is migrating, wait". Fixed by resolving
aliases in `get_index_inner`'s inline selector and in the shared
`resolve_index_selector` (used by get_mapping/get_settings/etc.),
mirroring how ES/OpenSearch accept a real index name OR an alias
name interchangeably on every single-index endpoint.
2. The actual root cause: `engine.aliases` was NEVER persisted to
disk. Index DATA survives a restart (confirmed working all
night), but the alias pointing at it didn't -- so even after (1)
was fixed and an alias was manually recreated, a REAL restart of
xerj still wiped it, and the very next OSD container hit the
identical stuck state again. Added `aliases.json` persistence
(atomic temp+rename, same pattern already used for
`es_mapping.json` per-index and `api_keys.json` globally):
flushed on every mutation (`add_alias`, `remove_alias`,
`delete_index`'s alias cleanup, data-stream rollover/delete), and
loaded once at `Engine::new()` startup, before the server accepts
requests.
Verified end-to-end against a real OpenSearch Dashboards 3.6.0
container, repeated across THREE separate xerj restarts in a row: no
manual alias recreation, no lock warning, straight to "Server
running" / `status.overall.state: "green"` every time -- previously
reproduced needing manual `_aliases` intervention after every single
restart.
Full ES-compat YAML conformance suite: 1360 passed, 0 failed, 3
skipped -- no regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016XJaZygeuRfZfUg2B8tPKU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #28/#29. Root-caused a real, reproducible OpenSearch Dashboards bug: a fresh OSD container, pointed at an xerj instance that already has a fully migrated
.kibanaindex, gets stuck indefinitely on:— even though there is only ever one real instance. This blocks OSD from starting at all on any second/subsequent run against existing data.
Root cause — two compounding bugs, found via a raw byte-level TCP relay between a real OSD 3.6.0 container and xerj (same technique used for the earlier
_cat/templatesfix)1. Single-index endpoints never resolved an alias to its underlying real index.
.kibanais always an alias, never a bare index — OSD/Kibana create.kibana_1thenPOST _aliasesto point.kibanaat it. So:GET /.kibana→ 404index_not_found_exception, even though.kibanawas a registered alias.GET /.kibana/_mapping→ 200, but missingmappings._meta.migrationMappingPropertyHashesentirely — silently falling back to a schema-derived reconstruction instead of the actually-stored mapping blob._meta.migrationMappingPropertyHashesis exactly the field OSD's savedobjects migrator reads to decide "already migrated, adopt it" vs "someone else is migrating, wait." Both gaps meant it could never see a completed migration. Fixed by resolving aliases inget_index_inner's inline selector and the sharedresolve_index_selector(used byget_mapping/get_settings/etc.) — matching real ES/OpenSearch, which accept a real index name or an alias name interchangeably on every single-index endpoint.2. The actual root cause:
engine.aliaseswas never persisted to disk. Index data survives an xerj restart (confirmed working all night); the alias pointing at it didn't. So even after (1) was fixed and an alias was manually recreated via_aliases, a real restart of xerj wiped it again, and the next OSD container hit the identical stuck state. Addedaliases.jsonpersistence — atomic temp+rename, the same pattern already used fores_mapping.json(per-index) andapi_keys.json(global): flushed on every mutation (add_alias,remove_alias,delete_index's cleanup, data-stream rollover/delete), loaded once atEngine::new()before the server accepts requests.Verified end-to-end
Real OpenSearch Dashboards 3.6.0 container, repeated across three separate xerj restarts in a row: no manual alias recreation, no lock warning, straight to
Server running/status.overall.state: "green"every single time. Previously this needed manual_aliasesAPI intervention after every restart, indefinitely.Test plan
cargo build --release -p xerj-engine -p xerj-api -p xerj-servercargo fmt --check/cargo clippy --no-deps— cleanGET /.kibanaandGET /.kibana/_mappingcorrectly resolve through the alias,_metaintact_cat/aliases,GET /.kibanaboth correct) with zero manual intervention🤖 Generated with Claude Code